Title: VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video

Multimodal Large Language Model

Video MLLM

Facial Analysis

Video Emotion Analysis
Abstract: Understanding and predicting emotions from videos has gathered significant attention in recent studies, driven by advancements in video large language models (VideoLLMs). While advanced methods have made progress in video emotion analysis, the intrinsic nature of emotions—characterized by their open-set, dynamic, and context-dependent properties—poses challenge in understanding complex and evolving emotional states with reasonable rationale. To tackle these challenges, we propose a novel affective cues-guided reasoning framework that unifies fundamental attribute perception, expression analysis, and high-level emotional understanding in a stage-wise manner. At the core of our approach is a family of video emotion foundation models (VidEmo), specifically designed for emotion reasoning and instruction-following. These models undergo a two-stage tuning process: first, curriculum emotion learning for injecting emotion knowledge, followed by affective-tree reinforcement learning for emotion reasoning. Moreover, we establish a foundational data infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG) consisting of 2.1M diverse instruction-based samples. Emo-CFG includes explainable emotional question-answering, fine-grained captions, and associated rationales, providing essential resources for advancing emotion understanding tasks. Experimental results demonstrate that our approach achieves competitive performance, setting a new milestone across 15 face perception tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VidEmo, a family of emotion-centric video foundation models trained with a novel two-stage training pipeline: Curriculum Emotion Learning and Affective-Tree Reinforcement Learning. It also presents a new large-scale benchmark dataset, Emo-CFG, containing 2.1M annotated samples with fine-grained emotional reasoning, captions, and rationales. The approach outperforms state-of-the-art models on 15 facial perception and emotion tasks, including closed and open-source baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The affective-tree structure elegantly decomposes emotional understanding into interpretable stages.\n- The post-hoc reinforcement learning (via GRPO) is a very sensible option that proves to bring additional gains.\n- Emo-CFG is a strong contribution: well-annotated and validated using committee-based LLM feedback.\n- The system is designed to provide high interpretability, increasing transparency in a typically opaque task.\n- The experiments show superior performance over 18 strong baselines.\n\nWeaknesses:\n- As acknowledged by the authors, their model uses only on visual data. The integration of audio or textual context (e.g., speech, subtitles), which is left as future work, would certainly bring better and more robust predictions, as emotional expression often depends on these.\n- An ablation study proving the improvements of the curriculum learning (vs. regular random training) and only post-training without curriculum learning is missing. It would be great if the authors could add this study to the appendices.\n- Although the committee-based verification helps, reliance on LLM-generated labels and validation could encode biases from those models. The authors should add this limitation/risk to the appropriate section.\n\nQUESTIONS:\nSection 4.2 is not very clear about how the affective tree is parsed from the generated captions. Could you elaborate more on this aspect?\n\nLIMITATIONS:\nLimitations are appropriately acknowledged. I suggested one more is added in the \"Weaknesses\" section of my review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel video emotion base model called VidEmo, which enables fine-grained and interpretable understanding of complex emotions in dynamic videos by combining curriculum emotion learning and affective-tree reasoning frameworks. The paper also constructs an emotion-centric fine-grained dataset, Emo-CFG, containing 2.1 million diverse instruction-based samples, which provides an important resource for emotion understanding tasks. Experimental results show that VidEmo outperforms existing open- and closed-source models in 15 face perception tasks, with advantages in emotion understanding and reasoning capabilities in particular. The authors also point out the limitations of the model in terms of possible generation of false content and multimodal integration, which provide directions for future research.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. A new video emotion base model (VidEmo) is proposed, combining curriculum emotion learning (phased injection of emotion knowledge) and affective-tree reinforcement learning (hierarchical reasoning) to enable fine-grained analysis and interpretable reasoning about dynamic and complex emotions in videos. The models derived from the base model achieve better performance on 15 facial perception tasks (e.g., attribute recognition, expression analysis, emotion understanding) compared to existing VideoLLMs.\n2. A new emotion-centric fine-grained dataset (Emo-CFG) is constructed. It covers detailed annotations across attributes, expressions and emotions, and supports full chain learning from low-level attributes to high-level emotions. The data quality is ensured through a committee voting-based verification strategy, which addresses the inherent ambiguities in emotional data due to its subjective nature.\n\nWeaknesses:\n\n1. The source data of the proposed Emo-CFG dataset is from 17 existing datasets. The paper does not give an in-depth analysis and discussion on the question of whether the possible bias in sample labeling and distribution between different source datasets affects the quality of Emo-CFG.\n2. The generation of some of the annotations in the constructed dataset relies on GPT-4o and Gemini 2.0, which may introduce biases or errors inherent to large models, and the paper does not analyze in detail the impact of differences between synthetic data and manual annotations.\n3. Affective-tree rewards may have limitations. The construction of ground-truth tree may be subjectively influenced by the annotator. The consistency of reasoning paths with human intuition was not quantitatively analyzed in the paper.\n4. The authors acknowledge that the model may generate \"false narratives\" or \"emotionally inconsistent descriptions\" but do not propose specific mitigating measures (e.g., post-processing filtering or uncertainty calibration) that may affect the reliability of actual deployments of the proposed models.\n5. The evaluation is insufficient. The proposed model is not compared with the latest dedicated expression/emotion analysis models (e.g., Emo-LLaMA, Exp-CLIP, ExpLLM) on the same dataset, and open-source model comparisons are focused on generic VideoLLMs, which makes it difficult to prove that it outperforms the optimal models in the field of expression/emotion analysis.\n\nQUESTIONS:\nSee Weaknesses 1~5\n\nLIMITATIONS:\nThe proposed models of this paper may be used for emotion monitoring or privacy invasion (e.g., inferring mental states through micro-expression analysis), but the paper only briefly mentions \"limitations\" without discussing specific safeguards (e.g., data desensitization or usage restrictions).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces a novel fine-grained, large-scale emotion understanding dataset by aggregating 17 existing datasets, designed to comprehensively capture emotional expressions in complex scenarios. To address the limitations of existing methods in handling such challenging contexts, the authors propose two innovative approaches: Curriculum Emotion Learning and Affective-Tree Reasoning. These methods are integrated into a newly designed base model, which demonstrates significantly superior performance compared to existing approaches across multiple evaluation tasks on the proposed dataset, highlighting its strong generalization ability and practical applicability.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	A large-scale (2.1M) emotion understanding dataset, Emo-CFG, is constructed.

2.	Two promising methods are proposed: Curriculum Emotion Learning and Affective-Tree Reasoning.

3.	The methods show strong performance, outperforming existing LLMs (1-8B).

Weaknesses:

1.	Emo-CFG is reviewed only by three VideoLLMs without human validation or user study, raising concerns about its reliability.

2.	Method descriptions lack clarity and theoretical grounding; the discussion in Section 5.3 could be moved to 4.1 for better structure.

3.	Experiments are limited to Emo-CFG, without cross-dataset evaluation to test generalization.

4.	Key information in the supplementary material is not explicitly referenced in the main text. For example, the additional details in Section 4.1 should refer to Table 7 in the supplementary C.2.

QUESTIONS:
1. Please see Weakness.
2. Is the Emo-CFG dataset planned to be released publicly?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper proposes VidEmo, a video MLLM for facial and emotional analysis. It integrates curriculum emotion learning with the affective-tree reasoning paradigm. Besides, a large-scale emotion dataset, Emo-CFG, is curated, covering question-answering, fine-grained captions, and associated rationales. Experiments show that Emo-CFG achieves competitive performance across 15 tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper applies the novel reasoning technology to the emotion tasks and proposes a task-specific variant, which advances the research progress of the emotion-centric problems.
2. A new dataset, Emo-CFG, is constructed, which is useful for training and benchmarking emotion models.
3. The experiments comprehensively compare with multiple types of baseline approaches on multiple tasks and metrics.

Weaknesses:
1. This paper aims to develop an emotion foundation model. However, it mainly focuses on face perception tasks. Non-facial video emotion tasks remain underexplored [r1, r2].
2. In Table 2, VidEmo-Base-7B performs worse than VidEmo-Base-3B. VidEmo-T1-7B achieves marginal improvements over VidEmo-Base-3B.
3. The case of VidEmo-T1-3B is not considered.

[r1] Mazeika, Mantas, et al. "How would the viewer feel? Estimating wellbeing from video scenarios." NeurIPS, 2022.

[r2] Guo, Yuxiang, et al. "Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models." IJCV, 2025.

QUESTIONS:
1. What is the backbone MLLM of VidEmo?
2. What GPUs and how many GPU-hours are used for the experiments?

LIMITATIONS:
See weaknesses
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-07