VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
Authors: Zhicheng Zhang
Weicheng Wang
Yongjie Zhu
Wenyu Qin
Pengfei Wan
Di ZHANG
Jufeng Yang
Venue: NeurIPS 2025 poster ()
Keywords: Video
Multimodal Large Language Model
Video MLLM
Facial Analysis
Video Emotion Analysis
Primary area: deep_learning
TL;DR: We present an emotion-centric video foundation model trained with fine-grained captions and rationales via affective-tree reasoning guidance, achieving high-level emotional intelligence for video understanding.

Abstract: Understanding and predicting emotions from videos has gathered significant attention in recent studies, driven by advancements in video large language models (VideoLLMs). While advanced methods have made progress in video emotion analysis, the intrinsic nature of emotions—characterized by their open-set, dynamic, and context-dependent properties—poses challenge in understanding complex and evolving emotional states with reasonable rationale. To tackle these challenges, we propose a novel affective cues-guided reasoning framework that unifies fundamental attribute perception, expression analysis, and high-level emotional understanding in a stage-wise manner. At the core of our approach is a family of video emotion foundation models (VidEmo), specifically designed for emotion reasoning and instruction-following. These models undergo a two-stage tuning process: first, curriculum emotion learning for injecting emotion knowledge, followed by affective-tree reinforcement learning for emotion reasoning. Moreover, we establish a foundational data infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG) consisting of 2.1M diverse instruction-based samples. Emo-CFG includes explainable emotional question-answering, fine-grained captions, and associated rationales, providing essential resources for advancing emotion understanding tasks. Experimental results demonstrate that our approach achieves competitive performance, setting a new milestone across 15 face perception tasks.

VidEmo: Affective-Tree Reasoning for Emotion-CCetric Video Foundation Models
Zhicheng Zhang Weicheng Wang', Yongjie Zhu Wenyu Qin", Pengfei Wan", Di Zhang", Jufeng Yang" 1243 Nankai University ² Pengchenn Laboratory Kuaishou Technology Nankai International Advanced Research Institute (SHENZHEN:FUTIAN)
gl gloryzzc60sina com, 21202306390mall. nankai. edu cn (ok / com, yangj uf infennenankai.. edu...eduuomm cn https://zzcheegg.tt//Vdddii
(zhuyongjie, qinweny,, wanpengfei
Attribute Perception
Appearance
Face Identity
Classify
Head Pose
Caption
Open Attribute Eye, Nose, Mouth, Chin, Shape, Hair, Skin, Action, Feature,
Action
Classify
Caption
Expression Analysis
Emotion
3 Affective Cues
Classify, Multiclass
Micro-expression Action Unit
Fine-grained
Sentiment
Complex Scenario
Classify Fine-grained
Emotion Caption
Conversation
Emotional Intelligent: Fine-grained Caption & Rationale Reasoning
Back
Four young girs ane sitting on the floor front of gray sofa with multigle pattereed rushion, all gazing foowrd. seamingly screen. The gir on the left has brown, mavy, long hair big almond shaped brown thick
Persow 1
ground
eyebrows,
App-
Act
opens has brown, curly, long big simond shaped brown eyes, thick brown eyebrows, nose with a high bridge, and thick lps, She wears a light pink tank top and gray pants and bracelets on both her wrists the sits with her arms clsppe sround her lines, the is pleased with show that indicited
bright
initialy,
slight goop.
Person
The gir
Emo
App
Time
Acf
Person
Emo
blonde eyebrows, small desed,
shirt white her exoression briefly shows concern; seems shocked by something the screen at throw out the popcom, and the expression turns fear. The fourth stts the for right, with blonde, strightt, long hat: big almond-oi shaped bue eyes, narrow blonde eyebrows, small nose with high bridge, thick lps. wearing light purale shirt with white pink polko pants. From the mid of video, she's chewing poprorn with her mouth closed and she seen nating fair smooth
by smie. The girl the immediate right has Blonde, straight, long hair, big almond-shaped blue eyes, narrow thick
App
Time
hoiding
Act
Person 4
popcor
mid,
Emo
App
Time
Act
Scenario
Emo
focused forward.
with
Camera
Time
white The camera angle focing
no camera movement
Lighting
Figure 1: Selected examples of inputs and outputs obtained from VidEmo. Apart from providing toolkits for basic attribute perception and expression analysis (top), VidEmo extends the cognitive capacity and is able to generate fine-grained emotional captions with explainable rationale (bottom).
Abstract
Understanding and predicting emotion from videos has gathered significant at- tention in recent studies, driven by advancements in video large language models (VideoLL.Ms). While advanced methods have made progress in video emotion analysis, the intrinsic nature of emotions poses significant challenges. Emotions
Equal Contribution. Project Leader. t: Corresponding Author,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are characterized by dynamic and cues-dependent properties, making it difficult to understand complex and evolving emotional states with reasonable rationale. To tackle these challenges, we propose novel affective cues-guided reasoning framework that unifies fundamental attribute perception, expression analysis, and high-level emotional understanding in a stage-wise manner. At the core of our approach is a family of video emotion foundation models (VidEmo), specifi- cally designed for emotion reasoning and instruction-following These models undergo two-stage tuning process: first, curriculum emotion learning for in- jecting emotion knowledge, followed by affective-tree reinforcement learning for emotion reasoning. Moreover, we establish a foundational data infrastructure and introduce amotion-centric tine-grained dataset (Emo-CFG) consisting of 2.1M diverse instruction-based samples. Emo-CFG includes explainable emo- tional question-answerring, fine-grained captions, and associated rationales, pro- viding essential resources for advancing emotion understanding tasks. Experi- mental results demonstrate that our approach achieves competitive performance, setting a new milestone across 15 face perception tasks.
Introduction
Understanding and predicting human emotions from dynamic videos is an increasingly vital chal- lenge in computer vision, with far-reaching applications in human-computer interaction, surveil- lance, and healthcare 55, 58]. Despite the success of advanced methods [37, 74, 75], particularly in classifying basic emotional expressions, the ability to predict about complex, evolving emotional states with reasonable rationale remains limited. This largely due the dynamic and context- dependent nature of emotions [98, 35], which require models capable of providing both high-level emotional intelligence and rational, explainable outputs [68, 39j. Recently, the emergence of Vide- oLLMs [8. 65, 81, 86] has provided a promising baseline as a pathway. However, these foundational models often struggle with high-level emotional understanding, as they lack the ability to effectively combine basic facial attributes into representations of complex emotion. Even the cutting-edge milestone, Gemini 2.0 [62], achieves only an accuracy of 26.3% in fine- fine-grained sentiment analysis, highlighting the gap in performance and the need for further innovation in this domain.
To address these challenges, we introduce VidEmo, novel affective cues-guided reasoning frame- work based on tree-structure that integrates three core components: fundamental attribute percep- tion, expression analysis, and high-level emotional understanding (see Fig. 1). Across 15 face per- ception tasks, VidEmo outperforms all existing open open-source VideoLLMs, surpassing the previous -tate-of-the-art benchmark, i.e., Gemini 2.0, as shown in Fig.2. To achieve this, Our VidEmo draws inspiration from recent work on reasoning models (R1), which excel at providing explainable ra- tionales [18, 52, 64, 73]. These models solve complex tasks by incorporating a thinking process alongside the model's operation. Our finding demonstrates that this same reasoning process can be applied to high-level emotion understanding by introducing stage-wise thinking, structured around attribute perception [102, 5], expression analysis [4, 61]. and emotion understanding [92, 70]. To be specific, we equip VidEmo with curriculum emotion learning and affective-tree reasoning, which inject emotion reasoning pathways during both the pre-train and post-train stages, respectively. In the pre-train stage, curriculum emotion learning progressively tunes the model from basic facial at- tributes to more complex emotional states. In the post-train stage, affective-tree reasoning helps the model refine its emotional understanding by using a hierarchical structure, ensuring that emotional responses are both accurate and interpretable. This two-stage process enables VidEmo to effectively analyze and reason about emotions in dynamic video data.
Relalivee Performance
Curting- Edigg Milezton
150%
Open-Sourced SOTAs
Attribute Perception
Exaression Analyss
Emotion Intellgent
152
125%
10054
SU
94
152
91
1:66
91
100
97
106
97
107
110
80
101
100
104
201
105
105
75%
Action Open Attribute
Verfccation
Head Pase
Appearance
Affective Cless
Comptex
Emotion
Sentiment
Ruency
Amuraay
Cbe Overlap
Label Overiap
Aatherance
Relavance
Figure 2: Results Overview. Our best model, VidEmo-TI, shows superior performance across 15
face perception tasks, surpassing advanced milestone (Gemini 2.0: 5th Feb, 2025) on 14 of 15 tasks.

---

To support our approach, we construct an amotion-centric fine- the-grained dataset, Emo-CFG, specif- ically designed serve as the foundational data infrastructure for emotion understanding. Emo- CFG is a large-scale dataset consisting of 2.1 million diverse, characterized by emotion-centric labels, rigorous data verification, and high diversity, ensuring comprehensive and reliable annota- tions across a wide range of emotional contexts. By offering rich annotations and wide variety of emotional contexts, Emo-CFG empowers VidEmo to effectively learn fine- ane-grained emotion under- standing from the emotion reasoning pathway.
Our contributions are two-fold: (I) We propose VidEmo, novel affective cues-guided reasoning framework that combines curriculum emotion learning and affective-tree reasoning, enabling fine- grained and interpretable emotion understanding from dynamic video data. Experimental results show that VidEmo achieves over a 16.3% and 14.2% improvement compared to existing open-source VideoL.LMs across 15 facial perception tasks on 1-3B and 7-8B scales. (2) We present Emo-CFG. a large-scale, emotion-centric dataset comprising MM diverse samples with detailed annotations across attributes, expressions, and emotions, serving as comprehensive data infrastructure for ad- vancing emotion-centric video analysis.
2 Related Work
Facial Video Analysis. Face video analysis is long standing problem towards high-level hu- man understanding which involves various tasks, including attribute perception [102, 51, expression analysis [4, 61], and emotion understanding [92, 70, 26]. Various face perception models leverages strong backbone power for constructing multi-task framework [59]. Going forward to high-level emotion understanding [95, 98, 94], recent methods embrace MLLM [9, 33, 34, 93] for their strong zero-shot perception capacity [59, 88, 14]. EmotionI.LaMA [10] introduces an emotion dataset in- cluding 28K coarse-grained and 4K fine-grained annotated datasets. OmniEmotion [80] proposes explicitly integrate facial and audio modeling for emotion recognition. However, existing ap- proaches are often constrained to limited set of emotion categories or rely on static attribution per- ception. To advance cognitive human emotion understanding, we propose fine-grained emotion- centric model empowered by dynamic attribution perception and emotion reasoning.
Reasoning Model in MILM. With the blossom of series of recent models such as DeepSeek- RI and OpenAl o-series [52, 18], various works probe into integrating MLLMs with reasoning capacity [2]. Multimodal chain-of Mfaii--of-thouhh (MCoT) prompting [27, 66, 46] offers ste-bb-step reasoning trajectory when MILM faces hard questions including detail grounding [72, 42], agent planing [27], etc. Specifically, MCoT decomposes the question into group of reasoning steps and builds a chain to guide the model to generate the results of complex problems step-by-step [73, 64, 89]. For instance, LLaVA-CoT [76] prompts MLLMs reasoning steps into the summary, caption, reasoning, and conclusion stages and proposes stage-level beam search strategy to further enhance reasoning capacity. In this paper, we propose affective cues-based rationale tree as intermediate bridge to meet the gap between abstract emotion and basic attribute.
3 VidEmo: Video Emotion Foundation Models
To develop family of emotion-centric video foundation models, we propose a comprehensive set of toolkits designed for the pre-training, post-training, and reasoning stages, as illustrated in Fig. 3. Through a structured pre-training process, emotion knowledge is injected, followed by post-training to enhance the model's reasoning capabilities. Finally, the reasoning stage allows the model to effectively generate emotional outputs, leveraging learned attributes, expressions, and emotions.
3.1 Pre-training: Curriculum Emotion Learning
To inject emotion knowledge into the foundation model, we employ curriculum emotion learning to progressively tuning our base model. The training is structured into three stages: I) Attribute Tuning, II) Expression Tuning, and III) Emotion Tuning. The pre-training focuses on curating data that balances the difficulty of emotion tasks while addressing perplexity. At each stage, we care- fully curate the data to ensure that the emotion-related tasks gradually increase in complexity. By starting with simpler attributes and progressively moving towards more complex expressions and emotions, we ensure that the model builds strong foundational understanding of emotion, which
3

---

(a) Training
(b) Reasoning
2e1
Curriculum Emotion Learning
GRPO with Mixed Affactive- Tree Reward
Policy
Class Attribute Tuning
Rule based Reward
Affective-Tree Tree based
Model
Grai--r Caption Reward
Sempling
Class II: Expression Tuning
Attribute
Attribute
Attribute
Class DI: Emotion Tuning
Best-of-Atribute
Sompling
Shared Parameter
Model based
Expression
Expression
Expression
Initialization
Caption Reward
Best-oo-isgeesion
Reference Model
KL
Policy Model
Reward Feedback
Sumpling
Penalty
Emotion
Emotion
Best-of-Emmtton
Figure 3: Pipeline of VidEmo. (a) Training:
The model is trained using curriculum emotion learn- ing, divided into three stages: attribute, expression, and emotion tuning. A reference model provides initial parameters, and a policy model is trained with reward feedback. (b) Reasoning: The policy model performs hierarchical reasoning by sampling from the best attributes, expressions, and emo- tions to generate the final emotional output.
facilitates smoother emotion knowledge injection throughout the process. Figure 4 presents the vi- sualization results of our model across three key aspects: Attribute Perception, Expression Analysis, and Emotion Understanding.
The model accurately identifies facial attributes, such as hair color, length, and presence of bangs, with the ground truth comparison clearly shown for validation. For instance, the model correctly identifies a person's hair as blonde and hhoulder-enngh,, while also distinguish- ing the presence absence of bangs.
Attribute Perception:
Expression Analysis: The model analyzes subtle facial expressions, identifying features such as downward-tilted eyes and posture. These features, as seen in the second part of the figure, provide insight into the emotional states of the person, such as sadness or introspection, based on facial and contextual cues, like lighting and body movements.
Emotion Understanding: By combining the insights from facial features and contextual cues, the model provides a detailed interpretation of the emotional state. For example, in the final part of the figure, the model identifies contemplative emotion, indicated by the subject's slightly tilted head, furrowed brows, and subtle eye movements.
3.2 Post-training: GRPO via Mixed Affective- Tree Reward
Building on emotion-kunn -oowledge-injeetteeeee base models, we proceed post-training to explore the emotional reasoning pathway. Recent reinforcement learning (RL) techniques [73] have demon- strated strong capabilities in reasoning, and GRPO [18] has garnered significant attention due to its simplicity and effectiveness. This makes GRPO an ideal starting point for our work.
Formally, let q be query. GRPO samples group of outputs {or with the number of G from the old policy model me... and train a policy model by maximizing the following objective:
lo Σ min (r:(9)A,,,
(o) E.g.(o.)
-To || Te))
T
where A is the advantage based on relative rewards in group, and 3 are coefficient of KL penalty and clip threshold, and no; Te Tee are current, old, and reference policy models, respectively.
Rule based QA Reward. The model is evaluated on its ability to respond to emotion-related queries using predefined rules of Acc and Fl score. The evaluation tasks include classification (single-label, multi-label), fine-grained classification, micro-expression detection, and action unit (AU) detection.
Model based Caption Reward. For the short caption of action, appearance, and emotion, we use generative reward model to score the quality of captions generated by the model.
Affective-Tres based Fine-Grained Caption Reward. To assess the model's capacity for struc- tured emotional reasoning, we introduce a reward mechanism based on a hierarchical affective tree constructed from fine-grained captions. Given generated caption ó, we first parse it into set of aspect-item pairs at three semantic levels: attribute (A), expression (E), and emotion (M). These elements organized into three-level affective tree T pred. where each node represents extracted

---

Phose the video and ianntls the actoro perfernet herratiog papposen
langle
thes valo Selcct appl cabée adtaon that oocur and hore cleevve
Ground Truth: close rpes, gare, tal
Ground Truth: the hair Ninnde, shoulteeinngth and she thout bangs
Quen2.55r S-ViL dose frowm, fane, nod Vidémo:
Quenn2S--i. The han blonde hair that shouldeeieggh with X
blande, shouldon without
(a) Attribute Perception
pase wotch dioecl and idwatty the sxprrdtiee perserts fore Hapay 90 foeir) Reuersiitt Angr 15 Surpeise Dieuurt feor
Vidtimo B
Q-P pase anclyte witch zarts the endrommett lggting bedggnard, Ausal anri noc, etr, can pxpiln matces for your sutare
Vidémo: The oumination indiiduals in this video commences with central character: features, particularly slighll downtumedd eye corners and mouth corners indicat common physical manifestation aligns fcelings sadness distess. These subtlc crucial fedal cues suggest
cmotional state rother than neutrality, which often involves neutral and
corners.
might support an narrative recent physical backdrop observed subdued demeanor, Additonnlly
Injuries wounds
providing on civironmental and slightly downward, Indicating withdrswn posture, impresdon introspection dejection than engagement stmosphedic components, highlgghting
gase just
lightiig
states aligned sadness, Smultaniously suggests
patentially
emotional
pesition sigitikaat body movements suggest stllnees-- typical response not assoclated energetic or poshilee emotions anger or surprise.
posture contestual lighting interpretation Considering observation the 13-second span, elements eyes typically happler
emphositing
upiiting expressions dynamic
(b) Expression Analysis
bangs young woman with long. stright styled swept eyelids high-bridged
large, almond-haped eyes
dangling
pink purple
patterned eiihtly right, appears
engaged
expressing
Centrully part. young dangling long, black styled
Extitty Purrnedd Brrwnt
everlager.
large,
eyes
varry High-bridged slightly
GREH DENoce
faintly
direction, angled slighll downward
though looking something suggeting empathy,
conseys quiet contemplation
syebrows. upper shible,
throughout, intially
less background difuseed lighting,
Contenn fetter
The right Eurrewiig syshoww
while maintaining
perspective
(c) Emotion Understanding
Figure 4: Visualization on attribute perception, expression analysis, and emotion understanding.
item and directed edges encode rationale-based dependencies:
rationale E rationale M.
(1)
We compare the predicted tree Tpeed with a ground-truth tree Tg. parsed from human-annotated captions, using the tree edit distance [87] Edit(Tg,, Tpea),, which quantifies the minimal number of edit operations (insertions, deletions, substitutions) required to transform one tree into the other.
The final reward R is computed using exponential decay over the tree distance:
R = exp (-X Edit(Tg., Tpea)),
where A > O is a scaling factor controlling the reward sensitivity to tree differences. This for- mulation encourages the model to generate captions that are not only accurate in content but also structurally explainable, aligning with human reasoning patterns over emotional understanding.
3.3 Inference: Reasoning for High-level Emotion Understanding
Our VidEmo facilitates stage-wise training can be smoothly integrated with search- based reasoning strategy. Specifically, we adopt hierarchical, search-based reasoning approach that decomposes
5

---

Data Source Meta
(b) Labelling
Data Verification
Causal Affective Reasoning
Voting by Committee
MOS!
TERVERA
DVEW
HELD dataset
Cobll Ni
Emo-CFG
SAP
RAVEESS
Expression
Attribute
S soodd Eiper Bast Por
QA/Capetom Ratimale
Coptiom
QA
Aspact
PI-GA
GA
AF
Misiing HAr Mosel
AfRi
in dis opdter Coanto dan
Copelcnn
#N--AA
Emotion
Aspecity
Figure 5: Data Curation Pipeline of the Emo-CFG dataset. (a) The source of data from 17 datasets, (b) The illustration of data labeling steps. (c) The illustration of data verification loop.
emotional understanding into three levels: attribute perception, expression analysis, and emotion inference. At each level, the policy model samples multiple candidate outputs and selects the best one via soward-guided scoring mechanism, forming bottom-up reasoning trajectory. notice that we disable ER when comparing with other SOTA methods, for a fair comparison setting with only one model response are sampled.
Emo-CFG: Emotion-Centrie Fine-Grained Video Dataset
The Emo-CFG dataset is designed to advance the understanding of emotional dynamics in the video. Motivated by the need for high-quality, emotion-centric data train emotion reasoning models, Emo-CFG addresses key challenges of diverse emotional contexts, reliable annotations, and rigorous verification. We illustrate the data curation pipeline and statisties of Emo-CFG in Fig. 5&Fig. 6.
Data Source & Meta Information. Our data collecting starts from high-quality video datasets. The data source include 17 datasets from head, avatar, and full-body avatar. By utilizing multiple data types, we ensure a holistic perspective to understanding the nuances of visual and emotional data. Further, we maintain the meta information of each video, including the face bounding box offset, video duration, video resolution, video fps.
Caption & QA Instruction Data Labeling. We utilize two primary data sources for labeling: large-scale, unlabelled datasets for broad coverage, and scall-scale, fully labeled datasets for preci- sion. For the labeled datasets, instruction pairs generated using GPT-4o, which creates multiple templates, including multiple-choice questions, open-ended questions, and short captions. For the unlabelled datasets, apply causal affective reasoning strategy to generate labels in a sequential, stage-by-stage manner. Specifically, given a video, we first leverage the state-of-the-art Gemini 2.0 model, prompting it generate fine-grained Caption data, focusing on attributes, expressions, and emotions in sequence. Subsequently, QA pairs are generated using GPT-4o, tailored to different aspects of the video. By combining these attribute and expression labels, the underlying emotion is accurately inferred, enabling a detailed and nuanced understanding of emotional states.
Caption R & QA R Rationale Data Labeling. Building upon the instruction data, we fur- ther explore the relationship between lowelevel attributes and high-level emotions. We prompt the
advanced VideoLLM to conduct self-refiection on the rationale behind the emotional cues R A and Q R2 Caption. This step not only enhances the model's interpretability by offering insights behind emotional expressions, but also serves crucial stage for enabling reasoning capacity.
Critic Data Verification by Committee Voting. To address the inherent ambiguities in emotional data, which arise from its subjective nature, we implement committee voting-based verification strategy. We use three heterogeneous VideoLLMMs as a committee to verify the correctness of the data and output Crilic items, including incorrect answers and suggested corrections. Verified data is retained, while data that does not pass verification is sent back for rewriting based on the suggested corrections. Additionally, we extract different aspects of the caption data and separate them into multiple QA pairs to ensure alignment with the QA process.
Data Statistics. Fig. 6 provides key statistics of the Emo-CFG dataset. In (a), the data taxonomy organizes the dataset into three primary face perception tasks: Emotion Intelligence, Expression Analysis, and Attribution Perception, covering a wide range of facial features and emotional at- tributes. (b) The data distribution plots show the relative face area and video duration across different datasets, illustrating the diversity and variety of video data present in Emo-CFG. (c) The annotation distribution includes the breakdown of facial views (head, half, full) and video length, accompa- nied by a word cloud highlighting the most frequently annotated terms, such as "neutral", "face",

---

(a) Data Taxonomy
(b) Data Distribution Relative Face Area
Video Duration
CASME
Caeey Text
oree
CHOMM5
Come
DFEV
Generaf
PERK
Emotion
AFEW
Intalligence
Attribution Perception
eieett
(c) Annotation Distribution (d) Data Statistics
Expression
Head
conversation
and
Analysis
Open
Valeacoe
130
208
X
neutral mouth corner face head expression
Sharo
Vidao
4BC
X
X
X
7:00 13M
X
X
X
X
115C
X
Word boon Ter
4.5
X
X
Length
Figure 6: Data Statisties of our Emo-CFG dataset. (a) The data taxonomy from three types of
face perception tasks. (b) The temporal and spatial distribution of video data. (c) The data label distribution and examples. (d) The comparison with other emotion and video datasets.
Table 1: Comparison with 18 leading VideoLLMs on 14 face attribute perception tasks of Emo- CFG, including 6 closed-set attribute perception tasks and 12 open attribute perception tasks. Cls: classification, Cap: caption, ID: identity verification, Pose: head pose estimation, AVG: average.
Model
Size Appearanee
Action
ID
Head
AVG
Open Attribute
AVG
Cls
Cup
Cls
Cap
Veri
Pose
Eye
Mout Hair Chin Shap. Feut.
Acce. Age
Gend. Skin Act.
Closed MELM API
Gemini 2.0 [62]
42.2
5644
41.4
62.2
86.5
25.8
52.
72.1
72.9
87.3
87.3
64.1
61.9
72.1
80.0
78.7
94.3
81.5
61.6
76.2
Clande 3.5 Sonnet [3]
39.1
58.8
35.7
61.1
63.6
22,0
46.7
70,8
67.6
34.4
77.8
68.9
56.1
54.7
77.3
79.5
93.3
48.9
61.4
67.6
Qwen-V1. VI.MMXX [63]
41.1
54.3
89.7
34.5
52.0
71.9
69.6
84.4
64.0
71.5
64.4
78.9
9
74.0
60.9
73.7
GPT-4o [51]
29.3
51.5
05.1
40.7
79.0
27.6
38.8
$1.5
45.7
62.9
77.1
48.5
46.4
54.5
78.1
68.5
86.3
75.5
52.8
62.3
GPT-4o mini [51]
20.5
55.8
04.1
54.0
69.3
27.8
38.6
43.2
44.5
52.0
52.4
31.6
43.5
40.1
45.0
41.0
69.6
45.9
30.7
44,9
Open-sourced I:BB Video MLLM
LLLVA-OV [25]
1B
06.3
34.9
00.2
47.6
50.3
14.4
25.6
41.0
49.7
36.0
50.9
46.7
32.0
39.0
48
29.8
87.6
20.1
61.9
45.2
Vntem
28
17.7
46.2
47.1
04.7
17.3
24
53.2
38.6
80.7
54.6
Videol [86]
2B
00.3
36.8
05.2
48.5
89.2
20.6
33.4
55.9
51.5
52.0
73.7
45.2
36.7
47.1
52.7
55.0
85.4
59.3
52.1
55.5
mPLUG-OWWB [81]
28
16.0
45.4
13.8
52.3
76.4
07.7
35.3
54
614
55.4
71.4
43.7
45.
52.3
60.8
39.8
91.6
58.7
48.5
56.9
Qwen2.5-VL
43.6
40.8
93.5
VidEmo- Base
3B
57.0
67.9
37.7
47.9
100
90.7
66.9
84.9
82.7
94,0
85.2
75.9
80.8
78.0
83.4
84,0
95.0
88.8
61.1
82.8
Open-sowrced Video
SharGPT4Vileo (7)
80
10.3
7
13.7
51.6
03.0
17.1
22.4
53.9
54.7
37.7
74.8
13.1
28.6
46.7
45.1
45.2
51.6
57.9
53.1
46.9
IntemmV1.2.5 [8]
8B
36.9
38.7
17.4
49.8
61.2
15.5
36.6
56.3
59.0
55.0
72.
52.4
36.4
52.7
61.5
60.0
76.9
57.6
59.3
58.3
LLAVA-N. [41]
16.9
34.6
20.5
51.7
05.8
29.8
42.6
67.5
18.3
49,9
48.6
LLaVA-OV [25]
78
05.6
37.3
12.2
46.6
97.2
19.8
36.4
53.0
47.6
504
64.0
35.3
32.2
49.9
55.8
72.9
94.6
47.5
59.0
55.2
VadeoLLaMA3 [86]
7B
28.3
33.5
15.8
48.7
89.2
1644
38.6
54.4
56.7
55.5
71.
40.5
36.6
50.6
61.
60.4
84.7
65.6
64.8
58.6
LLVVA-Video [91]
7B
14.5
38.2
14.1
46.0
8
20.3
37.0
65.0
57.0
66.2
72.6
21.3
29,4
59.3
68.3
79.4
93.0
62.5
63.8
61.5
mPLUG-DW1B [81]
7B
34.3
41.6
21.3
55,1
66.4
21.1
40.0
55.0
56.2
48.0
70.5
39,9
48.1
50.5
58.8
61.
89.7
62.0
60.8
58.4
Qwen2.5.VL. [65]
7B
44.7
45.2
21.0
52.3
99.7
22.6
47.6
68.6
70.5
83.2
74.7
66.4
55.6
73.8
71.2
94.0
76.2
64.0
72.1
VidEmo Base
7B
60.3
72,9
38.4
5
99.8
93.4
69.2
86.4
85.5
95.1
85.
77.3
81.6
78.7
85.0
85.6
95.0
89.5
71.7
84.7
VidEmmo-T Tl
78
64.8
73,1
41.4
57.4
99.7
96.7
72.1
88.2
87,8
95.6
87.8
79,2
82.0
80,8
85.7
86.9
97.0
90.4
74,3
86.3
and "expression". (d) Data statistics compares Emo-CFG with other emotion and video datasets, showing that Emo-CFG provides a richer set of annotations and label types, including fine- -ra-nrai emotion, rationales. and comprehensive video data, making it unique and valuable resource for emotion-centric research. More details can be refer to Sec. B of appendix.
5 Experiment
As shown in Tab. 1 and 2, we conduct experiments to verify the effectiveness of VidEmo on three types of tasks: attribute perception, expression analysis, and emotion understanding. We compare VidEmo with 5 closed MLLM APIs and 13 open-sourced VideoLLMs with scales ranging from IB to 8B. More training details (Sec. A) and evaluation settings (Sec. D) please refer to appendix.
5.1 SOTA Comparison
We benchmark VidEmo on metrics, spanning 6 closed-set attribute tasks, 12 open-set attribute tasks, 9 expression tasks, and 6 high-level emotion understanding tasks.
7

---

Table 2: Comparison with 18 leading VideoLLMMs on 11 expression analysis tasks and 6 fine- grained emotion understanding tasks of Emo-CFG. Sin: single-label classification, Mul: multi- label classification, Fine: fine-grained classification, Mic: micro-expression detection, AU: action unit detection, Cap: caption, Conv: conversation emotion analysis, VTR: video-text relevance, Flu: fluency, RA: response accuracy, IA: instruction adherence, Clu: clue overlap, Lab: label overlap,
AVG: average.
Model
Size
Emotion
AVG
Sontiment
AVG
Cues
AVG
Complex AVG
Emotion Understanding VTR
AVG
Sin Mul Fine
Mic AU
Cap Conv
Lab
Clu
Cloved MLLM API
Gemini 2.0 [62]
45.3
42.2
23.6
37.0
45.0
26.0
35.5
16.2
36.1
26.1
42.0
50.6
46.3
51.0
52.6
58.2
60.2
66.8
92.1 63.5
Claude Sonnel [3]
26.6
17.6
16.6
20.3
37.6
23.3
30.4
29.2
37.1
33.2
50.3
36.3
43.3
47.5
49.5
57.6
54.9
62.0
92.5 60.7
Owen-VL-MM [63]
29.3
25.8
24.6
26.6
32.6
21.3
27.0
11.7
24.0
17.9
46.4
49.3
47.8
47,9
50.7
57.4
56.6
63.6
92.0
61.4
GPT-4a [51]
20.6
19:0
04.3
14.6
29.6
18.0
23.8
29.6
02.3
16.0
38
43.6
40.8
30.0
35.4
40.8
43.2
50,9
87.4 48.0
GPT-4o mini [51]
19.6
20.2
07.3
15.7
29.9
16.6
23.3
21.9
08.9
15.4
40.7
47.6
44.2
35.5
39.0
46.2
46.7
54.4
91.0
52.1
Open-sowrced I-38 Video MLLM
LLaVA-OV [25]
18
28.3 17.5
05.3
17.0
36.3
15.0
25.6
13.4
15
14.6
40.2
29.6
34,9
17.2
18.2
28.6
23.5
28.8
89.0
34.2
L
23.0
16.1 08.3
09.7
07.4
08.5
34.8
40.1
90.8
55,2
Lade [86]
2B 29.3
18.3
11.6
19.7
34.3
16.0
25
06.5
06.5
06.5
20.1
31.3
25.7
24.0
22.4
29.5
33.4
40.3
86.8
39.4
mPLUG-Ow[3 [81]
32.3
184
05.0
21.
28.0
22.7
25.3
27.8
27.9
89.9
42.9
Qwen2.5-VL [65]
3B
36.0
23.7
09.3
23.0
36.6
20.3
28.5
12.1
23.0
17.6
44.2
40.3
42.2
38.2
40.9
49.7
49.2
56.3
91.4
54.3
VidEmo- Base
46.0
38.0
26.0
36.6
40.3
36.5
22.3
32.1
27.2
48.1
45.0
57.3
59.6
70.7
93.1
68.6
Open souree 7B+ Video MT..M
SlareGPTaVidea [7]
8B 07.6 06.0
04.6
06.1
38.0
14.3
26.1
09.7
01.
05.6
46.
32.3
39.2
16.1
18.8
34,4
21.8
26.0
91.1
34.7
V1. [8]
8R 28.0 26.2 09.0
21.0
29.3
18.3
23.8
16.2
12.8
14.5
40.8
35.0
37.9
52.3
53.4
61.5
59.8
66.1
92.4
64.2
LLaVA-N-Viders [41]
7B 24.3 23.7
10.6
19.5
39.0
14.0
26.5
10.9
13.3
12
44.1
39.0
41.5
33.7
33
43.3
38.8
45.2
90.9
47.5
HaVA-OV [25]
7B 31.6 22.7
10.3
21.5
36.0
20.0
28.0
11.7
15.5
13.6
42.
43.0
42.6
36.5
39.3
49.5
46.
91.0
52.6
LaM3 [86]
7B 27.6
23.9
10.6
20.7
31.0
19.3
25.1
08.9
108
09.8
33
36.0
34.6
42.
44.
53
52.9
59.5
89.5
56.9
HLVA [91]
7B 32.6
22.9
09.3
21.6
35.3
20.6
28.0
12
02.5
07.3
45.8
42.0
43.9
38.8
40.9
50.1
47.8
54.6
91.2 53.9
mPLUG-OwL3 [81]
7B 30.0
22.2
10.
21.0
29.3
15.6
22,5
21.5
25.1
23.3
47.
33.6
40.4
36.5
35.6
44.2
43.8
50.3
90.8
50.2
Qwen2.5.VI. [65]
7B
38.6
27.0
12.3
26.0
22.3
26.1
10.5
14.
12.5
46.
44.3
45.2
50.7
52.1
60.0
59.7
66.3
92.7
63.6
VidEmmo Base
7B
47.3
37.6
34.6
39.8
42.3
36.0
39.1
18.2
34.2
26.2
50.0
48.6
49.3
55.9
57.4
67.9
62.6
68.3
92.8
67.5
VidEsmo-1
38.8
35.6
39,9
20.4
34.1
27.3
92.6
69.3
Scale: Our models significantly outperform existing closed and open-source VideoLLMs across all metrics from IB to 8B scales. At the 1-3B / 7-8B scale, our VidEmo-Base model (3B/7B) achieves an overall average accuracy of 62.4%//4.11 1,, outperforming the strongest baseline, Qwen2.5-VV 5L at 1%%51.7% by a margin of + +16.3%/+111 12.4%. The consistent improvement across scales demon- strate the effectiveness of our proposed curriculum learning as well as affective-tree reward in scaling up our foundation model.
Attribute & Expression & Emotion Tasks: We conduct comprehensive analysis of VidEmo across three core task categories in the Emo-CFG benchmark: attribute perception, expres- sion analysis, and emotion understanding. In attribute tasks, which include both closed-set (c.g., identity, head pose) and open-set (e.g., hair type, age, skin tone) recognition, VidEmo achieves an average score of 86.3%, surpassing all baselines including Qwen2.5-VL. (7B) at 80.6%, yielding a +5.7% improvement. Particularly, our model achieves 99.7% on identity verification, 95.6% on facial shape, and 97.0% on gender prediction, reflecting strong generalization on fine-grained visual perception. In expression analysis, covering single-label, multi-label, and fine-grained classification, as well as micro-expression and AU detection, VidEmo delivers an average of 39.9%, outperform- ing Qwen2.5-VI. (7B) by +6.8%. Notably, VidEmo leads in fine-grained expression classification (35.6% vs. 29.7%) and micro-expression detection (20.4% vs. 13.6%). demonstrating its sensitiv- ity to subtle and transient affective cues. In emotion understanding tasks-spanning spanning instruction- following, fluency, response accuracy, and video-text relevance- VidEmoo achieves 69.3% on aver- age, outperforming all prior models, including Gemini 2.0 (63.5%) and Qwen2.5-VL. (7B) (63.6%), with improvements of over +5%. It sets new benchmarks on tasks like instruction adherence (68.1%) fluency (69.1%), and video-text relevance (69.3%), showcasing its capacity for coherent, explainable, and semantically grounded emotional inference. We also notice that for the attribute tasks achieves an higher averaged performance. This finding also aligns with the dynamics we ob- served during training, that the perplexity of model increase with a sequential order of attribute, expression, and emotion.
8

---

5.2 Discussion
Open-sourced Models and Closed Models: We evaluate both open- sourced and closed multi- modal large models (MLLMs) on the Emo-CFG benchmark. Closed models, including Gemini 2.0, Claude 3 Sonnet, GPT-40, GPT-44 mini, and Liven-VL-MAX, typically operate as APIs with proprietary architectures and in-distribution training data. In contrast, open-sourced models span both small and large-scale variants (1-7B parameters), including LLaVA-OV, VL2.5, Vide- oLLaMA3, mPLUG-OWl, Qwen-VL, and our own VidEmo series. Across all three evaluation cat- open-sourced
egories-attributt perception, expression analysis, and emotion understanding VidEmo-TI (7B) outperforms all closed-source models. For instance, VidEmo-TI achieves 86.3% in attribute perception, surpassing Gemini 2.0 by +9.8%, and obtains 39.9% expression tasks, outperforming Claude by +16.6%. Notably, in high-level emotion understanding, VidEmo-TT reaches 69.3%, exceeding GPT-4o's ' 48.0% by margin of +21.3%.
Base Model and Reasoning Model We further compare our base model (VidEmo-Bass 7B) with the reasoning-enhancee model (VidEmo-TI) to assess the effectiveness of affective-tree reason- ing. In attribute perception, VidEmo-TI improves the average performance from 84.7% to 86.3%, with notable gains in key tasks such as head pose estimation (+1.9% from 93.4 to 96.7), fa- cial feature prediction (+1.3% from 85.6 to 86.9), and gender recognition (+1.9% from 89.5 to 90.4). In expression analysis, VidEmo-TI shows consistent improvements across all sub- tasks. The average increases from 39.1% to 41.3%, with gains in micro-expression detection (+2.2%) and fine-grained expression recognition (+1.0%). Most notably, in emotion understand- ing, VidEmo-T TI achieves a substantial improvement from 67.5% to 69.3%, with strong gains in flu- ency (+2.3%, from 67.9 to 70.2), video-text relevance (+2.6%), and instruction adherence (+1.9%).
Downstream Tasks. For downstream emotion- related tasks, we evaluate the capacity of VidEmo on facial expression recognition using the DFEW and MAFW datasets. When fine-tuned for specific tasks, VidEmo consistently outperforms both tradi- tional video-based expression recognition methods and zero-shot emotion-oriented CLIP approaches. As shown in Table 3, VidEmo achieves a perfor- mance gains compared to state-of-the-ar baselines (EMO-LLaMA), with an average improvement of 9.4% over the previous best results across both datasets. Notably, VidEmo achieves relative im- provements of 7.8% in UAR and 10.9% in WAR on DFEW, and 5.9% in UAR and 12.8% in WAR on MAFW, demonstrating its superior effectiveness for facial expression recognition tasks.
Table 3: Downstream tasks on DFEW and MAFW datasets.
DFEW
MAFW
Model
UAR
WAR
UAR
WAR
EmotionCLIP [90]
13.77 19.89
9.20
11.65
Exp-CLIP [99]
24.25
25.87
17.53
20.27
EmoCLIP [15]
36.76 46.27
25.86
33.49
EmoCapCLIP [60]
42.19 43.99
30.85
34.50
13D [6]
46.52 58.27
F-DFER [100]
53.69 65.70
EST [45]
53.43 65.85
IAL [28]
55.71 69.24
CLIPER [29]
57.56 70.84
DFER-CLIT CLIP [101]
59.61 71.25 38.89
52.55
AMO-LLaMA [75]
60.23
65.89
41.57
48.63
VidEmo (Ours)
64.92
73.10
44.02 54.86
5.3 Ablation Study
To investigate the effectiveness of curriculum emotion learning (CEL), affective-tree reward (ATR), and emotion reasoning (ER), we conduct a component-wise ablation study, as presented in Tab. and more in-depth amalysis in Tab. 8, Tab. 9 and Tab. 10 of appendix. In this study, assess the contribution of each component by analyzing the performance of the model under different configurations, removing one or more of the components. With the ablation studies conducted, we find four interesting observations:
When none of the components are used, the model achieves an average performance of 51.4. This baseline highlights the importance of incorporating these components into the model for improved performance.
With the inclusion of CEL alone, the model performance increases to an average 61.9, demon- strating the positive impact of curriculum emotion learning on the model's ability to handle emotional contexts. Specifically, we observe improvements in the emotion-related metrics, par- ticularly in the expression and emotion attributes.

---

Table 4: Ablation study on the
Table 5: User study across three dimensions between Emo-
proposed components for our
CFG and Cextb We evaluate the label quality on pre- cision, rationality, and complementary through pairwise com-
VidEmo. CEL = curriculum emo-
tion learning, ATR affective-tree
parison with 50 videos and 25 users. All three dimensions
reward, ER emotion reasoning.
show statistically significant preference for Emo-CFG.
Components
Pairwise
Att
Exp
Emo Avg
Dimension
#Vid #Usr
Prefer p p-value
CEL ATR ER
Win
Tie
Loss
63.5
27.3
63.6
51.4
Precision
964 204
82
50
25
95.5%
0.00021
79.5
38.7
67.5
61.9
Rationality
1082
92.
0.00015
81.3
40.1
69.3
63.6
Complementary
1172 23
55 50
25
93.0%
0.00008
84,5
43.8
72.9
67.0
Introducing ATR alongside CEL further enhances the model's performance, with an average score of 63.6. The inclusion of ATR results more refined emotion handling, as seen in the improvements in the emotion and expression attributes.
The full model, with CEL, ATR, and ER, achieves the highest performance, with an average score of 67.0. This configuration benefits from the combined effects of all components, espe- cially in emotion reasoning, where the model shows notable improvements across all attributes, particularly in the expression and emotion metrics.
5.4 Dataset Verification
As data scale increases beyond 50K samples, maintaining consistent data quality becomes challeng- ing. Our data pipeline offers a systematic solution to this problem. To assess the quality of the generated expressions, we conducted a user study on a manually inspected subset of test samples to verify their alignment with the intended emotional semantics. Specifically, we compared Emo-CFG with CelebV-Text, the largest human-labeled video emotion dataset, across three key dimensions: precision, rationality, and complementarity. Preference rates for Emo-CFG across these dimen- sions reached 95%, 92%, and 93%, respectively, with statistically significant differences (Wilcoxon signed-rank test, p i 0.01). This result demonstrates that Emo-CFG provides more precise and ex- pressive emotional representations than existing benchmarks.
Conclusion
In this work, we introduced VidEmo, a family of video-based emotion foundation models designed to unify fine-grained facial attribute perception, expression analysis, and high-level emotion under- standing. Our framework integrates curriculum emotion learning with a novel affective-tree reason- ing paradigm, enabling interpretable and structured emotion inference. We further curated Emo- CFG, a large-scale, instruction-driven dataset with hierarchical annotations and rationale -enonnddggd data, which serves as foundamental data infrastructure for training and evaluation. Experimen- tal results on the Emo-CFG benchmark demonstrate that VidEmo consistently outperforms existing open- and closed-source VideoLLMs across 15 tasks, setting up perception, expression analysis, and emotion understanding tasks.
milestone in all the attribute
Limitations. While VidEmo exhibits strong generalization across diverse tasks, several limitations remain. First, like most existing VideoLLMs, VidEmo is susceptible to generating counterfactual content, which can lead to false narratives emotionally inconsistent descriptions. Second, emotion understanding is inherently multimodal; integrating additional modalities such as audio or contextual cues could significantly enrich affective reasoning and we view VidEmo strong foundation for future work in this direction, enabling the exploration of richer, more holistic emotion understanding.
Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 623B2056), the Natural Science Foundation of Tianjin, China (N0.2JJCZJCC00400 the Fundamental Research Funds for the Central Universities, the Supercomputing Center of Nankai University (NKSC). We sincerely thank the reviewer team (KQSY, 96WN, ntby, and JqQW) for their invaluable feedback to improve our manuscript.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Almman, Shyamal Anadkat, et al. Gpt-4 technical report. In arXiv, 2023. 33
[2] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In EACI,, 2024. 3,33
[3] Anthropic.
Introducing claude 3.5
sonnet.
https: attpp://wwi //www www.ropthopicccoo com/news/
claude-3- sonnnet 2024. 8.8
[4] Xianye Ben, Yi Ren, Junping Zhang, Su-Jing Wang, Kidiyo Kpalma, Weixiao Meng, and Yong-Jin Liu. Video- baseo-baeed facial micro-expresssion analysis: A survey of datascts, features and algorithms. TPAMI, (409)-582658846, 2021. 3,
[5] Zhixi Cai, Shreya Ghosh. Kalin Stefanov, Abhinav Dhall, Jianfei Cai. Hamid Rezatofighi, Reza Haffari, and Munawar Hayat. Marlin: Masked autoencoder for facial vidco representation learning. In CVPR, 2023. 2, 33
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 9
[7] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. In NeurlPS, 2024. 7, 31
[8] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfci Cui, Jinguo Zhu, Shenglong Yc, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. In arXiv, 2024. 3,
[9] Zhe Chen, Weiyun Wang, Hao Tian, Shengiong Ye, Zhangwei Gao, Erfei Cui, Weawen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far to gpt-4v? closing the gap to commercial multimodal models with open-souree suites, Science China Information Sciences, 67(12)220101, 2024. 3
[10] Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He. Kai Wang. Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G Hauptmann. LLaMA: Multimodal emotion recognition and reasoning with in- struction tuning. NewrIPS, 2024. 3,33
[11] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, al. Videollama 2: Advancing spatial-temporal modeling and audio understanding video-Ilms. In arXiv, 2024. 33
[12] Abhinav Dhall, Roland Goecke, Simon Lucey, and Tom Gedeon. Acted facial expressions in the wild database. 2011. 30
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, image is worth 16x words: Transformers for image recognition scale. In ICLR, 2021. 33
[14] Yasaman Etesam, Özge Nilay Yalçin, Chuxuan Zhang, and Angelica Lim. Contextual emotion recogni- tion using large vision language models. IROS, 2024. 3,
[15] Niki Maria Foteinopoulou and Joannis Patras. Emoclip: A vision-language method for zero-shot video facial expression recognition. In FG, 2024. 9
[16] Xun Gao, Yin Zhao, Jie Zhang, and Longjun Cai. Pairwise emotional relationship recognition in drama videos: Datasct and benchmark. In ACM MM, 2021. 30
[17] Mislav Grgic, Kresimir Delac, and Sonja Grgic. Scface-surveillance cameras face database. MTA,
51:863-879, 2011. 2
[18] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Decpscck-rl: Incentivizing reasoning capability in lmss via reinforce- ment learning. In Xiv,, 2025. 2, 4,
[19] Wenyi Hong. Weihan Wang. Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang. Yean Cheng, Shiyu Huang. Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. In arXiv, 2024. 33
11

---

[20] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Bootstrapping vision-language learning with de- coupled language pre-training. In NeurlPS, 2024. 33
[21] Xingxun Jiang. Yuan Zong, Wenming Zheng. Chuangao Tang. Wanchuang Xia. Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In ACM MM, 2020. 29, 30
[22] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björ Schuller, Irene Kotsia, and Stefanos Zafeiriou. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. IJC,, 127(6)907-922,, 2019. 30
[23] Xing Lan, Jian Xue, Ji Qi, Dongmei Jiang, Ke Lu, and Tat-Seng Chua. Expllm: Towards chain of thought for facial expression recognition. TMM, 2025. 33
[24] Jiyoung Lec, Scungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion
recognition networks. In ICCV, 2019. 30
[25] Bo Li, Yuanhan Zhang, Dong Guo, Rennui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. In arXiv, 2024. 7, 8, 31
[26] Chang Li, Yimeng Hou, Rencheng Song, Juan Cheng. Yu Liu, and Xun Chen. Multi-channel eeg-based emotion recognition in the presence of noisy labels. Science China Information Sciences, 65(4):140405, 2022. 3
[27] Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-o-tthuuggl In arXiv, 2025. 3, 33
[28] Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. Intensity-aware loss for dynamic facial expression recognition in the wild. In AAAI, 2023. 9
[29] Hanting Li, Hongjing Niu, Zhaoqing 7hu, and Feng Zhao. Cliper: A unified vision-language framework for in-the-wild facial expression recognition. ICME, 2024. 9
[30] Jingzhi Li, Changjiang Luo, Ruoyu Chen, Hua Zhang, Wengi Ren, Jianhou Gan, and Xiaochun Cao. Faceinsight: A multimodal large language model for face perception. In arXiv, 2025. 33
[31] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In ICMI, 2023. 33
[32] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. In arXiv, 2023. 33
[33] Yifan Li, Anh Dao. Wentao Bao, Zhen Tan, Tianlong Chen, Huan Liu, and Yu Kong, Facial affective behavior analysis with instruction tuning. In ECCV, 2025. 3,33
[34] Zheng Lian. Affectgpt-rl: Leveraging reinforeement learning for open vocabuluy emotion recognition. arXiv, 2025.
[35] Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. Ov-mer: Towards open-vocabulary multimodal emotion recognition. In ICML, 2025. 2
[36] Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi- supervupe learning. In ACM MM, 2023. 29, 30,33
[37] Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. Affectgpt: Datasct and framework for explainable multimodal emotion recognition. In arXiv, 2024.
[38] Zheng Lian, Licai Sun, Haoyu Chen, Zebang Cheng, Fan Zhang, Ziyu Jia, Ziyang Ma, Fei Ma, Xiaojiang Peng, and Jianhua Tao. Dmer-ranker: Learning to rank emotion descriptions the absence of ground truth. arXiv, 2025. 33
[39] Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. Explainable multimodal emotion reasoning. In arXiv, 2023. 2
[40] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-llvaa Learning united visual representation by alignment before projection. In EMNIP, 2024. 33
12

---

[41] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: A strong zero-shot video understanding model. htttp:://llaaa-a wwa.aaaaaaa.....iaii..... io/blog/ 2024-01-30- next 2024. 7, 8,
[42] Yangzhou Liu,, Yue Cao, Zhangwei Gao, Weiyun Wang. Zhe Chen, Wenhai Wang, Hao Tian, Lewei Lu, Xizhou Zhu, Tong Lu, et al. Mminstruct: A high-qualic multi-modal instruction tuning dataset with extensive diversity. Science China Information Sciences, 67(12):220103. 2024. 3
[43] Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang. Yuanzhe Qiu, Tie Cheng, aoteng Li, Hua Xu, and Kai Gao. Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av av-mixup consistent module. In ICMI, 2022. 30
[44] Yuanyuan Liu, Wei Dai, Chuanxu Feng, Weabin Wang. Guanghao Yin. Jiabei Zeng. and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. In ACM MM, 2022. 29,31 30
[45] Yuanyuan Liu, Wenbin Wang, Chuanxu Feng, Haoyu Zhang. Zhe Chen, and Yibing Zhan. Expression snippet transformer for robust video-based facial expression recognition. PR. 138:109368, 2023. 9
[46] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 61112)2200 2024.
[47] Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ee016331, 2018. 29, 30
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019. 24
[49] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv, 2024.
[50] Muhammad Maaz, Hanoona Rasheed. Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ACL, 2023. 33
[51] OpenAI. Gpt-4o system card. https: http:p///pppaaa...cccc///aeeelep 2024. 7,
[52] OpenAL. Openai 03-mini. httpp///gpeassscccm//eeee//ppass
2025. 2, 3, 33
[53] Xizheng Y. University of Wisconsin Madison Peiran L, Linbo T. L-svd: A comprehensive video dataset for emotion recognition. https://github wwt.ccomm//////eeeee http///eiraa..133mmmmmmmooommmm
2024. 30
[54] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mi- halcea. Meld: A multimodal multi-party datasct for emotion recognition in conversations. ACL, 2019. 30
[55] Jenny Preece, Yvonne Rogers, Helen Sharp. David Benyon, Simon Holland, and Tom Carey. Human- computer interaction. 1994. 2
[56] Fangbing Qu, Su-Jing Wang, Wen-Jing Yan, He Li, Shuhang Wu, and Xiaolan Fu. Cas (me) 2: database for spontaneous macro- exprrssiixessinnnnoo and micro excro-expression spotting and recognition. TAC, 9(4)422 436, 2017. 30
[57] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 33
[58] Alamgir Sardar, Saiyed Umer, Ranjeet Kr Rout, Shui Hua Wang, and M Tanveer. A secure face recog- nition for iot-cnabled healthcare system. 7SN, 1933 2023. 2
[59] Haomiao Sun, Mingjic He, Tianheng Lian, Hu Han, and Shiguang Shan. Face-mllm: A large face perception model. In arXiv, 2024. 3, 33
[60] Licai Sun, Xingxun Jiang. Haoyu Chen, Yante Li, Zheng Lian, Biu Liu. Yuan Zong Wenming Zheng, Jukka M Leppänen, and Guoying Zhao. Learning transferable facial cmotion representations from large- scale semantically rich captions. arXiv, 2025. 9
[61] Licai Sun. Zheng Lian, Bin Liu, and Jianhua Tao. Mae-dfer: Efficient masked autoencoder for self- supervised dynamic facial expression recognition. In ACM MM, 2023. 2,3,33 33
13

---

[62] Gemini Team, Petko Georgiev. Ving Ian Lei. Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2024. 7,8
[63]
Qwen Team. Introducing qwen-vl.
www.tt...//bbooeeeeeeo howblo////wnn-vl..r.../////// 2024. 7.8
[64] Qwen Team. Qvq: To see the world with wisdom. https://gwenls.githubbbb gVq-72b-previens 2024. 2, 3,
[65] Qwen Team. Qwen2.5 vi! qwen2.5 vi! qwen2.5 vi! 2025. 2, 7, 8, 31
[66] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethinking step-by-step visual reasoning in llms. In arXiv, 2025. 3,33
[67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. In arXiv. 2023. 33
[68] Vijaysrce Venkatraman. Where logic meets emotion. Science, 368:1072 0000007272222 2020. 2
[69] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran Hc, Yu Qiao, and Chen Change Loy. Mead: large-scale audio-visual dataset for emotional talking-face generation. In ECCV, 2020. 29.30
(70] Shangfei Wang and Qiang Ji. Video affective content analysis: A survey of state-of the-thr-rt methods. TAC, 6(4):410-436) 2015. 2, 3,33
[71] Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos. In CVPR, 2022. 30
[72] Penghao Wu and Saining Xie. VT: Guided visual search a core mechanism in multimodal Ilms. In CVPR, 2024. 3.33
[73] Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafacl Rafailov, nathan lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought. In arXiv, 2025. 2, 4, 33
[74] Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Heng-Jen Chen, Chan Hsu, Hong- Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruction tuning. In CVPR, 2024.
[75] Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xic, Huanjing Yue, Jingyu Yang. Heikki Kälviäinen. Emo-Ilama: Enhancing facial emotion understanding with instruction tuning. In arXiv, 2024. 2, 33
[76] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-b In arXiv, 2024. 3,
[77] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. In arXiv, 2024. 33
[78] Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, and Xiaolan Fu. Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS one, 9())e86041. 2014. 29
[79] Wen-Jing Yan. Qi Wu, Yong-Jin Jin Liu, Su-Jing Wang, and Xiaolan Fu. Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In FG, 2013.
[80] Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. Omni-emotion: Extending video milm with detailed face and audio modeling for multimodal emotion analysis. In arXiv, 2025. 3.
[81] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug- ow13: Towards long image- sequence understanding in multi modal large language models. In ICLR, 2025. 2, 7, 8, 31
14

---

[82] Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy. Weidong Cai, and Wayne Wu. Celebv-text: A large-scale facial text-video dataset. In CVPR, 2023. 29,30
[83]] Wenmeng Yu, Hua Xu, Fanyang Meng. Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with ane-grained annotation of modality. In ACL. 2020. 29,
[84] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. In arXiv, 2016. 29, 30
[85] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosed dataset and interpretable dynamic fusion graph. In ACL, 2018. 29, 30
[86] Boqiang Zhang. Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang. Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. In arXiv, 2025. 2, 7, 31
[87] Kaizhong Zhang and Dennis Shasha. Simple fast algorithms for the editing distance between trees and related problems. SIAM journal computing, (886))1245-1266 1989. 5
[88] Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, and Zhenyue Qin. Visual prompting in LLMs for enhancing emotion recognition. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, EMNLP, 2024. 3.
[89] Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang. Ruoming Pang. and Yiming Yang. Improve vision language model chain-of of-thought reasoning. In arXiv, 2024. 3.33
[90] Sitao Zhang, Yimu Pan, and James Wang. Learning emotion representations from verbal and nonverbal communication. In CVPR, 2023. 9
[91] Yuanhan Zhang. Jinming Wu, Wei Li. Bo Li. Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. In arXiv, 2024. 7, 31,33
[92] Zhicheng Zhang. Lijuan Wang, and Jufeng Yang. Weakly supervised video emotion detection and pre- diction via cross-modal temporal erasing network. In CVPR, 2023. 2, 3, 33
[93] Zhicheng Zhang, Wayou Xia, Chenxi Zhao, Zhou Yan, Xiaoqiang Liu, Yongjic Zhu, Wenyu Qin, Pengfei Wan, Zhang, and Jufeng Yang. Moda: Modular duplex attention for multimodal perception, cogni- tion, and emotion understanding. In ICML, 2025. 3
[94] Zhicheng Zhang and Jufeng Yang. Temporal sentiment lncalization: Listen and look untrimmed
videos. In ACM MM, 2022. 3
[95] Zhicheng Zhang, Pancheng Zhao, Eunil Park, and Jufeng Yang. Mart: Masked affective representation learning via masked temporal distribution distillation. In CVPR, 2024. 3,
[96] Jiaxing Zhao, Boyuan Sun, Xiang Chen, and Xihan Wei. Facial dynamics in vidco: Instruction tuning for improved facial expression perception and contextual awareness. In arXiv, 2025. 33
[97] Jiaxing Zhao, Qize Yang, Yixing Peng. Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. Humanomni: A large vision-speech language model for human-centric video understanding. In arXiv, 2025. 33
[98] Sicheng Zhao, Guoli Jia, Jufeng Yang, Guiguang Ding, and Kurt Keutzer. Emotion recognition from multiple modalities: Fundamentals and methodologies. SPM, 38(6):59-73, 2021. 3, 33
[99] Zengqun Zhao, Yu Cao, Shaogang Gong, and loannis Patras. Enhancing zero-shot facial expression recognition by Ilm knowledge transfer. In WACV, 2025. 9
[100] Zengqun Zhao and Qingshan Liu. Former-dfer: Dynamic facial expression recognition transformer. In ACM MM, 2021. 9
[101] Zengqun Zhao and loannis Patras. Prompting visual-language models for dynamic facial expression recognition. In arXiv, 2023.
[102] Xin Zheng, Yanqing Guo, Huaibo Huang, Yi Li, and Ran He. A survey of deep facial attribute analysis. IJCV, 128:2002-2034, 2020. 2,3,33
15

---

[103] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In ECCV, 2022. 29, 30
[104] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiso, Philippe Wansen-Fstrucee Licheng
Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, al. Apollo: An exploration of video understanding large multimodal models. In arXiv, 2024, 33
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4 and Section 3 describes the detailed data curation pipeline and methodology. Section validates the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section 5, Appendix D, and Appendix ??. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducíbility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g.. in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The code and the representative data samples are provided the supplemen- tary zip file. The full data will be publicly released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: ptettps//niiss cc/
cublice/giides/ooeSStiie oooiisoooossssssoocooooo for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g,, for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wwwpppblccggguiieeeeceeeoooeeeee Salees//ooeeubbiii ississsse ssi ossoiiyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section 5 and Appendix ??. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///nuriippppeee hwtp:///w.....icccccc.eeeeeeeeee
Answer: [Yes]
Guidelines: Please see Appendix ??.
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: Please see Appendix A, Appendix 4, Appendix D, and the code comments provided in the supplementary material file.
21

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions NeurlPS Code of Ethics and
and locations, and we expect authors to adhere
guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We propose -LaM- original method as described in Section Section and Appendix A.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
22

---

Please refer to our LLM policy for what should or should be described.
23

---

A Implementation and Training Details
Following the approach outlined by Qwen2.5-VL [65], we adopt Vision Transformer (ViT)-based architecture for the visual encoder and utilize an autoregressive model for the text encoder. For the foundational large language model (LLM), we select models within the 3B to 7B parameter range. VidEmo is pre-trained for 3 epochs with a batch size of 1024 and is subsequently post-trained for epoch with batch size of 128. We employ the AdamW W [48] optimizer with cosine learning rate schedule. The learning rate is set to 2e-5 for SFT stage and le-5 for RL stage, with warmup rate of 0.03. As shown in Table 6, we follow the existing MILMM training setting and use a two-stage tuning paradigm.
Backbone: We use the same backbone LLM and vision encoder as Qwen2.5-VL.
Data: We train VidEmo by our constructed Emo-CFG and textual knowledge dataset MAGPIE.
Hyperparameter: We follow the common setting and train the model by default learning rate, weight decay, and batch size.
Table 6: Implementation details and hyperparameters for our VidEmo family.
Backbone
Data
Hyperparameter
Model
LLM
Vision
SFT
RL
Ir
wd
bs
rollout
VidEmo-Base-33 Base-3B Qwen2,5-3B
VIT
Emo-CFG
2e-5
0
1024
VidEmo-BBi Base-7B
Qwen2.5-7B
VIT
Emo-CFG
2c-5
0
1024
VidEmo-TI-77B
Qwen2.5-7B
VIT
Emo-CFG
Emo-CFG
le-5
0
128
B Emo-CFG Dataset Details
B.1 Dataset Construction
In this section, we provide an overview of the training data used in our Emo-CFG dataset, which is sourced from multiple datasets address various tasks related emotion and attribute perception. The data is collected to support model training, and the details of each task are summarized in Table We illustrate the scenario, data source, task, data number, and ratio of our training data.
The attribute perception category includes tasks such as appearance recognition, action recogni- tion, and human identity recognition. These tasks are sourced from Cel, CelebV-Text, and MEAD, with multi-label question answering (QA) and caption generation tasks. The open attribute perception category involves tasks that focus on the recognition and analysis of open-ended at- tributes like eye, mouth, nose, shape, gender, and more. Important attributes such as age, gender, and accessories are covered in this category, and each task plays a significant role in identifying open-ended features that contribute to emotional understanding.
In the expression analysis category, focus tasks related to sentiment recognition (SR), emo- tion recognition (ER), affective cues detection, and complex scenario understanding. These tasks sourced from datasets like MOSEI, CHSIMSvI, and CASME and aim to capture fine-grained emotional expressions and actions. The analysis of micro-expressions and emotion-related cues in this category contributes to the detailed recognition of emotional states.
Lastly, the emotional intelligence category covers important capacities related to video-text rele- vance, fluency, and emotional reasoning. These capacities focus understanding the relationship between video and text for emotional intelligence applications. Notably, emotional reasoning tasks make up 60. 90% of the dataset in this category, highlighting the importance of reasoning based tasks in the overall dataset.
B.2 Dataset Statisties
Our constructed Emo-CFG richs in fine-grained caption for high-level emotion understanding. We illustrate the caption distribution as shown Figure 7. We displays the distribution of caption lengths across multiple sources, including CelebV-Text, MOSEI, RAVDESS, MELD, CelebV-HQ.
24

---

The overview of training data. All the data used for training sampled from the training or validation split of the source datasets. QA: Question Answering. OPEN: Open-ended Question Answering. SR: Sentiment Recognition. ER: Emotion Recognition.
Table 7:
Scenario
Data Source
Task
Number
Ratio
Attribute Perception
Appearance Recognition
CeIcbV-HQ
Multi-label QA
32010
8.16%
Appearance Caption
CelehV-Text
Caption
59879
15.26%
Action Recognition
CelebV-HQ
Multi-label QA
32010
8.16%
Action Caption
CelebV-Text
Caption
59797
15.24%
Human Identity
MEAD
QA
19998
5.10%
Head Pose
MEAD
QA
188590
48.10%
Open Attribute Perception
Eye
QA OPEN
72197
5.94%
Eyebrow
QA OPEN
71793
5,90%
Mouth
QA OPEN
94722
7.79%
Nose
QA OPEN
126248
10.38%
Hair
QA OPEN
154994
12.75%
Chin
QA OPEN
20559
1.69%
Shape
QA OPEN
78689
6,47%
Emo-CFG
Feature
QA OPEN
92830
7.63%
Accessory
QA OPEN
56835
4.67%
Age
QA & OPEN
145359
11.9%%
Gender
QA & OPEN
151901
12.49%
Skin
QA & OPEN
104005
8.55%
Body Action
QA & OPEN
16577
1.36%
Head Action
QA & OPEN
2658
0.22%
Face Action
QA & OPEN
26515
2.18%
Expression Analysis
Single-label SR
MOSELMOSI
QA
19710
5.66%
Fine-grained SR
CHSIMSv1
QA
1824
0.52%
Single-lahel ER
MARRDDFEW.MER202
QA
20060
5.76%
Multi-label ER
MAFW
Multi-label QA
7178
2.06%
Fine-grained ER
MEAD.RAVDESS
QA
198957
57.11%
Micro-expression Detection
CASME,CASME'
QA
515
0.15%
Action Unit Detection
AfTiild2
QA
2180
0.63%
Conversation Reasoning
PERR.MELD
QA
38153
10.95%
Expression Caption
Celt
Caption
59797
17.16%
Emotional Intelligent
Vidco-Tece Text Relevance
Fluency
Coherency
Fine-grained Caption
78072
39.10%
Response Accuracy
Emo-CFG
Cue Overlap
Label Overlap
Emotion Reasoning
Rationale
121618
60.90%
MOSI, AFEW, DFEW, CHSIMS, MAFW, PERR, and FERV39K. Each histogram represents the frequency of captions of varying lengths, ranging from 0 to 500 words. We can observe that the variability in caption length across these datasets, with some datasets exhibiting distribution (e.g.. CelebV-Text and MOSI) and others showing skewed distributions (c.g., MAFW and PERR).
uniform
We further illustrate the rationale distribution for high-level emotion understanding as shown in Figure 8. The rationale lengths exhibit distinct distributions across the datasets. Some sources, like CelebV-Text and MOSI, show more uniform distributions, while others, such as MAFW and FERV39K, present skewed distributions. These distributions are crucial as they reflect the varying complexity and the level of detail involved in the rationales used for emotion analysis and under- standing in multimodal tasks.
25

---

CelebV-Text
MOSEI
RAVDESS
200
300
40
do
20
330
4.0
380
20
20
MELD
CelebV-HQ
MOSI
AFEW
DFEW
CHSIMS
MAFW
PERR
FERV39K
300
430
200
350
4.0
300
Figure 7: Length Distribution of caption data from the sources of Emo-CFG.
CelebV-HQ
RAVDESS
MOSI
CelebV-Text
CHSIMS
CASME
DFEW
MAFW
MELD
AFEW
PERR
FERV39K
Figure Length Distribution of rationale data from the sources of Emo-CFG.
C More Experimental Results
We further explore the design of the proposed components and analyze their effects in detail.
Supervised Finetuning: Curriculum Emotion Learning. The effectiveness of our proposed Cur- riculum Emotion Learning (CEL) approach is demonstrated through an in-depth analysis of the pre-training stages, as outlined in Table 4.
For the single-stage pre-training, where each stage is isolated, we observe varying levels of perfor- mance across different emotion-related tasks. Specifically, when the model trained on attribute data alone, the average performance is lower due to the limited complexity of the tasks. Introducing expression data results in moderate improvements, particularly expression-related tasks, but overall performance remains relatively modest (42.2). The most substantial improvement occurs when emotion data is introduced, leading to a noticeable boost in emotion-relater tasks (53.9), but still lacking the holistic integration seen in multi-stage training.
26

---

For the our proposed multiple-stage pre-training. where the model is exposed to progressively more complex tasks across all three stages, the performance improves significantly. The introduction of both attribute and expression data at earlier stages (52.6 average) enables the model to better integrate and align emotion-related information. With the complete multi-stage pre-training, which includes all three data types, the model achieves robust performance (61.9 average), indicating that the curriculum learning strategy successfully enhances the model's understanding of emotional complexity. The analysis reveals that progressively increasing task difficulty facilitates the model's ability to learn emotion-based tasks more effectively, aligning with our goal of gradually injecting emotion knowledge into the base model.
Table 8: In-Depth Analysis on curriculum emotion learning with data curation for our VidEmo. Attr = Attribution Data, Exp Expression Data, Emo = Emotion Data.
CEL
Attribute Expression Emotion Average
Attr
Exp
Emo
Single Stage
63.5
27.3
63.6
51.4
65.7
24.1
32.4
40.7
52.9
31.6
42.1
42.2
64.3
30.2
67.3
53.9
Multiple Stage
77.2
35.1
45.5
52.6
79.5
38.7
67.5
61.9
Post-training with Affective- Tree Reward. The post-training phase with Affective- Tree Reward (ATR) is designed enhance the model's performance in emotional reasoning tasks. Table presents the results of integrating various components into the post-training process. Initially, when only the group relative policy optimization (GRPO) is applied, the model achieves average score of 51.2. This score serves as the baseline performance before incorporating the Affective Re- ward.
Upon adding the Affective- Tree Reward (ATR), the model's performance increases to an average of 61.4, indicating that the inclusion of this reward mechanism improves the model's ability generate emotion-related captions. The introduction model's performance, resulting in an average score of 63.6 This improvement is observed across the attribute, expression, and emotion tasks.
the Tree Edit Distance (ATR) further enhances the
These results demonstrate that the addition of the Affective-Ti Tree Reward and Tree Edit Distance en- hances the model's performance in emotional reasoning tasks by improving the accuracy of caption generation and ensuring structural alignment with human-annotated captions.
Table 9: In-Depth Analysis on affective tree reward for our VidEmo.
Attribute
Expression Emotion
Average
GRPO
75.7
33.8
44.2
51.2
+
Tree Reward
80,9
38.1
65.3
61.4
Tree Edit Distance (ATR)
81.3
40.1
69.3
63.6
Emotion Reasoning. The performance of the model in emotion reasoning tasks is evaluated by varying the number candidate responses sampled during inference. As shown in Table 10, the baseline model, which uses the Affective-Tree Reward (ATR) with a single output (n=l), achieves an average score of 63.6 across the attribute, expression, and emotion tasks. This baseline represents the basic reasoning model, which is trained with ATR and generates only response per query.
When the number of candidate outputs is increased, the model's performance improves. For in- stance, when two candidate responses (n=2) are sampled, the model achieves average score of 1, with notable improvements in expression and emotion tasks. Further increasing the number of candidate outputs to four (n=4) results in slight performance boost, bringing the average score to
27

---

66.2. This trend continues with the sampling of eight candidate responses (n=8), where the model achieves the highest average score of 67.0, along with consistent improvements across all tasks.
This analysis highlights the benefit of the search-based reasoning strategy in enhancing the model's performance. By sampling multiple candidate outputs and selecting the best one based reward- guided scoring mechanism, the model is able to refine its emotional reasoning process.
Table 10: In-Depth Analysis on emotion reasoning for our VidEmo. Baseline is the basic rea- soning model trained with ATR and only outputs one response (n=l).
Attribute
Expression Emotion Average
Baseline
81.3
40.1
69.3
63.6
+Emotion Reasoning (n=2)
82.9
42.3
70.1
65.1
+Emotion Reasoning (n=4)
84.2
43.2
71.2
66.2
+Emotion Reasoning (n=8)
84.5
43.8
72.9
67.0
28

---

Table 11: The overview of our evaluation benchmark. All the data used for evaluation sampled from the testing split of the source datasets. QA: Question Answering. OPEN: Open-ended Question Answering. SR: Sentiment Recognition. ER: Emotion Recognition.
Scenario
Data Source
Task
Number
Metric
Attribute Perception
Appearance Recognition
Ce
Multi-label QA
500
Fl
Appearance Caption
Text
OPEN
GPT score
Action Recognition
CelehV-HQ
Multi-lahel QA
500
FI
Action Cuption
CelebV-Text
OPEN
500
GPT
Human Identity
MEAD
QA
500
ACC
Heead Pose
MEAD
QA
500
ACC
Open Attribute Perception
Eye
QA & OPEN
300
ACC & GPT score
Byebrow
QA & OPEN
300
ACC & GPT score
Mouth
QA & OPEN
300
ACC & GPT score
Nose
QA & OPEN
300
ACC & GPT score
Hair
QA & OPEN
300
ACC & GPT score
Chin
QA & OPEN
300
ACC & GPT score
Shape
QA & OPEN
300
ACC & GPT score
Emo-CFG
Feature
QA & OPEN
300
ACC & GPT score
Accessory
QA & OPEN
300
ACC & GPT score
Age
QA & OPEN
300
ACC & GPT score
Gender
QA & OPEN
300
ACC & GPT score
Skin
QA & OPEN
300
ACC & GPT score
Body Action
QA & OPEN
107
ACC & GPT score
Head Action
QA & OPEN
ACC & GPT score
Face Action
QA & OPEN
189
ACC & GPT score
Expression Analysis
Single-label SR
MOSELMOSI
QA
ACC
Fine-grained SR
CHSIMSv1
QA
300
ACC
Single-label ER
MAFW.DFEW.MER2OD
QA
ACC
Multi-lahel FR
MAFW
Multi-lahel QA
300
F1
Fine-grained ER
MEVDDRAVVESS
QA
300
ACC
Micro-cxpression Detection
CASME II AffWild2
246
ACC
Action Unit Detection
Multi-labe QA
87
F1
Conversation Reasoning
PERR,MLLD
QA
300
ACC
Fmotion Caption
CelehV-Text
OPEN
300
GPT score
Emotional Intelligent
Video-Tex Text Relevance
GPT
Fluency
GPT score
Coherency
Vine-graing
GPT score
Emo-CFG
2600
Response Accuracy
Caption & Rationale
GPT score
Cue Overlap
GPT score
Label Overlap
GPT score
D Evaluation Settings
D.1 Task & Source
We outline the overview of our evaluation benchmark as shown in Table 11, including scenario, data source, task, data number, ratio and evaluation metrics.
Attribute Perception: CelebV-HQ [103] for appearance recognition and action recognition. Ceteb [82] for appearance caption and action caption. MEAD [69] for head pose estimation and human identity verification.
Expression Analysis: MOSEI [85] and MOSI [84] for single-label sentiment recognition. CH-
SIMSv1 [83] for fine-grained sentiment recognition. MAFW [44], DFEW [21] and MER2023 [36]
for single-label emotion recognition. MAFW [44] for multi-label emotion recognition. MEAD [69]
and REAVDESS [47] for fine-grained emotion recognition. CASME [79], CASME II [78] and
29

---

CASME² [56] for micro exprrssiixeesiinn detection. Afi-Wild [22] for AU detection. MELD [54] and
PERR [16] for Conversation Reasoning. CelebV-text [82] for emotion caption.
Emotion Understanding: The annotation for open attribute perception and fine-grained caption in
Emo-CFG from 17 source datasets: AFEW [12], CAER [24), CASME2 [56], CelebV-HQ [103],
CelebV-text [82], CHSIMSv1 [83], CHSIMSv2 [43], DFEW [21], FERV39K [71]. L-SVD [53],
MAFW [44], MELD [54], MER2023 [36], MOSEI [85], MOSI [84], PERR [16], REAVDESS [47],
Aff-Wild [22], MEAD [69].
D.2 Competitive Alternatives
Gemini 2.0 Gemini 2.0 is cutting-edge closed-source multimodal large language model devel- oped by Google Deep.Mind. is designed to handle both textual and visual inputs, excelling in tasks such video understanding, summarization, and generation.
Claude-3.5-Sonne Anthropic's Claude-3.5- is closed-source VideoLLM that builds on the Claude series with enhanced capabilities in video comprehension and interaction.
GPT-4o 4o mini GPT-4o and its lightweight variant, GPT-4o mini, are closed-souree Vide- oLLMs developed by OpenAL. These models are optimized for visual understanding tasks, of- fering balance between computational efficiency and performance.
Qwen-VL-Max Qwen-VL-Max designed process complex video content in conjunction with textual inputs, making it a versatile tool for video summarization, captioning, and question- answering tasks.
Qwen2.5-VL. Qwen2.5-VL. is an advanced open-source vision-language model that excels in multimodal tasks such as object localization, and long-video comprehension. Its innovative architecture enables efficient visual recognition and interaction with extended temporal video data.
VL2. Intern pushes the boundaries of open-source multimodal models by intro- ducing advanced scaling strategies across model architecture, diverse video-text datasets, and test-time optimization.
mPLUG-OWl3 mPLUG-OW13 is a cutting-edge multi-modal large language model designed excel in long image-sequence understanding, including tasks involving lengthy videos and interleaved image-text scenarios.
VideoLLaMA3 LaMA3 adopts vision-centric training paradigm, emphasizing the use of high-quality image-text data to improve multimodal capabilities.
LLaVA-OV By leveraging insights from data, models, and visual representations, VA-VA-OV achieves significant performance improvements in three major computer vision tasks while en- abling strong transfer learning across modalities.
ShareGPT4Video ShareGPT4Video introduces a framework for video understanding and gen- eration by leveraging high-quality captions generated through specific summary prompt. LLaVA-NeXT-Video LLaVA-NeXT-Video is advanced open-sourced large multimodal model designed for comprehensive video understanding, leveraging interleaved data formats enhance performance across multi-image, video, and 3D tasks.
LLaVA-Video LLaVA-Vides is a video understanding model that processes video sequences using a straightforward approach, supporting both fps and uniform frame sampling. It modular and scalable, allowing for efficient training and inference with limited resources, and achieves performance comparable to some 7B models on multiple benchmarks.
Table 12 provides model eards for different MI.LMs, including reference papers, parameter scale, and links to pre-trained weights.
D.3 Evaluation Metrics
We evaluate the performance of the model using various metrics (GPT score, Accurary, and score), depending the task. For text-oriented tasks such as Appearance Caption, Action Cap- tion, Emotion Caption, Fine-grained Caption, and Open-ended QA, we employ GPT scores, using GPT-4o to score the predictions based on labels and model responses. For recognition tasks that
30

---

Table 12: Model cards for Open-sourced MLLMs.
Model
Scale
Link
LLaVA-OV [25]
1B
https///hugiingface.
IntemVL22.5 [8]
2B
VidcoLLaMMA3 [86]
2B
httpe: tttge://uugginggftccco//// http:ggginnaaac.cooooMMe
mPLUG-OW13 [81]
2B
Qwen2.5VV. [65]
3B http:s///mnnggiiiisrrr..cc/eeeer
ShareGPT4Video [7]
IntemVL2..5
8B
LLaVA-NeXT-Vidoo [41]
7B
sttpe://bugginggfacc.
LLaVA-OV [25]
7B
httpe: http://gg.....eeeeoooooo////d/de
VideoLLaMA3 [86]
7B
https: htape://buggingface
LLaVA-Vidco [91]
7B
tttpe://buggingtacee htt.a//gg......oo...eaaaaaa/aala
mPLUG-OW13 [81]
7B
httpe: htspe://bugginggfcce http://wgggggggaeooooo/o/////ue/ 1ttp://///PP0000000033----11-110
Qwen2.5-VI. [65]
7B
hetps://huggingfan wtweeee.o///uueeeeeeeeeee ---...-----.---ntt-e--
involve classification, such as Human Identity, Head Pose, choice-based QA, Single-label Senti- ment Recognition (SR), Fine-grained SR, Single-label Emotion Recognition (ER), Fine-grained ER, Micro-expression Detection, and Conversation Reasoning, we use accuracy as the primary evalua- tion metric. Prior calculating accuracy, GPT-4o used to convert the model's ' responses into standardized labels to ensure consistency in the evaluation. For multi-label tasks, including Appear- ance Recognition, Action Recognition, Multi-label ER, and Action Unit Detection, we utilize the Fl-score to evaluate the model's performance, capturing both precision and recall in these multi- label settings.
D.4 Prompts used for Evaluation
We utilize three distinct prompts during the evaluation phase, each designed for specific aspects of model performance: F1 score, accuracy evaluation, and GPT score computation.
The score prompt, shown in Figure 9, used to convert the model's 's response into multiple choice options. The prompt requires the model extract relevant labels directly from the provided answer, ensuring that only valid and directly relevant labels are included. If the answer is invalid or none, the prompt instructs the model output "none." The answer options then concatenated together, and the model is instructed not to include any additional phrases such as "output:" or "Here is the output."
The accuracy evaluation prompt, depicted in Figure 10, is designed to assess the correctness of the model's response. It presents the model with question, the ground truth answer, and the model's own response. The model is tasked with judging whether the response matches the ground truth and is required to output simple "yes" "no" depending on whether the model's response correct.
The GPT score evaluation prompt, illustrated Figure 11, is employed to compute a numerical score reflecting the accuracy and the degree of match between the model's response and the ground truth. The score is provided as a number between 0 and 100, with no additional commentary or text. This prompt allows for more granular evaluation of the model's performance, especially in terms of accuracy and relevance.
These prompts are critical for evaluating different facets of model performance, ensuring that the model's s responses are both accurate and relevant.
31

---

Prompt for Label Extraction
Based on the question. please convert provided answer multiple choices connected by commas. output none the answer none or invalid. Choice candidates from: (Choices from Question
Question:
{Question]
Model's Response: (Model's Response]
[Requirement]
1. Directly output the converted phrases from the answer. Only output the phrases.
Don't say like "Here output" or "output:". Each option is directly concatenated together in the output.
[Output Example]
rescach, hairline
Figure 9: Prompt for extracting multiple phases and compute fl score from the response of models.
Prompt for Acc Evaluation
You are an expert evaluating vidco facial expression analysis.
Question: {Question}
Ground Truth: (ruth:(The The ground true answer)
Model's Response: (Model's Response]
Please judge the correctness of model's response according to the given answer and question.
[Requirement]
Directly output the judgement: yes or no
[Example]
Yes
Figure 10: Prompt for computing accuracy score.
Prompt for GPT Score Evaluation
You expert in evaluating the accuracy of video facial actions.
Question: [Question]
Ground Truth: (The ground true answer)
Model's Response: (Model's 's Response) }
Please score the model's prediction according the correctness and matching degree.
[Requirement]
1. Directly output the score number from 0 to 100.
No sentence word. [Example]
80
Figure 11: Prompt for computing GPT score.
32

---

E Full Related Work
E.1
Facial Video Analysis
Face video analysis is long standing problem towards high-level human understanding which in- volves various tasks, including attribute perception [102, 5], expression analysis [4, 61], and emo- tion understanding [92. 70J. Advanced models can be divided into those for attribution perception tasks [97, 30] and for high-level emotion understanding tasks [92, 70, 4, 36]. Various face percep- tion models leverages strong backbone power for constructing multi-task framework [59]. Pioneer- ing methods are developed solve pre-defined tasks, while MLLM -asedd method are proposed to enhance their zero-shot capacity. Going forward to high-level emotion understanding [95, 98, 38), recent methods embrace MLLM for their strong zero-shot perception capacity [59, 88, 33, 14]. EMO [75] firstly incorporates facial priors, including facial embeddings, landmarks, and age- gender-race attributes, through a mlp-based connnector to improve emotion understanding. Emo- tionLLaMA [10] introduces an emotion dataset including 28K coarse-grainet and 4K fine-grained annotated datasets. OmniEmotion [80] proposes to explicitly integrate facial and audio modeling for emotion recognition. FacialDynamic [96] construct existing largest human-labelled dataset with 5K samples. EXpLLM [23] recently explore to leverage thain-of-thoughe strategy to empower LLM with the reasoning capability. However, existing approaches often constrained to a limited set of emotion categories or rely on static attribution perception. To advance cognitive human emotion understanding, we propose fine-grained emotion-centric model empowered by dynamic attribution perception and emotion reasoning.
E.2
Video Extension in MILM
VideoLLMs [40, 50, 91, 32, 49] have gained significant attention in recent years by leveraging isting pre-trained foundational models, particularly powerful Large Language Models (LLMs), to enhance support for video inputs and outputs [11, 19, 104, 77]. The key components of VideoLLMs include: 1) a video encoder responsible for encoding inputs from different modalities into feature representations that the model can understand, e.g., ViT [13]. CLIP [57]: 2) an input projector to
align encoded spatiotemporal features from video with textual feature space in LLM. Input projec- tors be implemented by linear projection compressed- beseed projection such Q-Former [31] or P-Former [20]; and 3) LLM Backbone based on pre-trained models like GPT [I] or LLaMA [67], which processes representations from different modalities and performs semantic understanding.
E.3 Reasoning Model in MILM
With the blossom of a series of recent models such as DeepSeek-RI and OpenAl o-series [52, 18), various works probe into integrating MILMs with reasoning capacity [2]. Multimodal chain-of- thought (MCoT) prompting [27, 66) offers a step-by-ste reasoning trajectory when MI.LM faces hard questions including detail grounding [72], agent planing [27], etc. Specifically, MCoT decom- poses the question into group of reasoning steps and builds a chain to guide the model to generate the results of complex problems step-by-step [73, 64, 89]. Recent works have demonstrated that CoT prompting substantially improves the MLLM's capability on reasoning tasks. For instance, LLaVA-CoT [76] prompts MLLMs reasoning steps into the summary, caption, reasoning, and con- clusion stages and proposes a stage-level beam search strategy to further enhance reasoning capacity. establishing a new direction for structured prompting techniques. In this paper, we propose affective cues based rationale tree as intermediate bridge to meet the gap between abstract emotion and basic attribute.
LLaVA-Reasonee [89] pioneers the use of forced Thain-of-Thoughee
33

---

Visualization
In this section, we present visualization samples generated by our VidEmo model, as well those from our constructed Emo-CFG dataset. For more comprehensive visualization results, please refer the video demos provided in the attached zip file.
F.1 Results from our VidEmo model
Attribute Perception. We show the visualization comparison hetween our VidEmoand cutting- edge milestone Gemini 2.0 for appearance recognition and appearance caption in Figure 12, action recognition and action caption in Figure 13, head pose recognition and human identity recognition in Figure 14, open attribute perception in Figure 15.
Expression Analysis. We show the visualization comparison between our VidEmoand cutting-edge milestone Gemini 2.0 for single-label emotion recognition, fine-grained emotion recognition and multi-label emotion recognition in Figure 16, single-label sentiment recognition and fine- fine-graing grained sentiment recognition in Figure 17, micro-expression detection and action unit detection Fig- ure 18, conversation reasoning and emotion caption in Figure 19.
Emotion Understanding. We show the visualization comparison between VidEmoand cutting- edge milestone Gemini 2.0 for fine-grained video caption in Figure 20
F.2 Samples from our Emo-CFG Dataset
Attribute Perception. As shown in Figure 21. we provide samples from public face attribute datasets, and we convert their annotations into question-answer pairs. addition, we visualized the labeled face attribute samples in Figure 22.
Expression Analysis. As shown in Fig 23 and Fig 24, provide samples from public emotion recognition datasets and our fine-grained caption dataset.
Emotion Understanding, As shown in Fig 25, we provide samples for emotion reasoning from the provided dataset.
Meta Labels of Face Landmarks and Parsing Masks. As shown in Fig 26, we provide meta labels of face boxes, face landmarks and parsing masks.
34

---

outaut how mas ruzebraeen (harry wal jeand chully, sall men You Jleak swil Buol Troadi Juilled bald, targ: MuikJak blonk Buidduuui gwe Temen rmrigh
sigucleeiaennn Mouiinu Cucllasoo under foedhener: "unnl pengieeet oee.. wos wu.eennee gosiing.. peoco weringg rediaal anng.nult
Ground Trutts mais, meading jairiie, brown Joli no, baard
(maker /'habbyy /resading Jubiliee, brewn hat' 'no baard www.ing necksar
hadies,
Hamaan: Winch the
loau (ppascacc sacdind panua Passa the douuriicaaag aapalalaaaa Joeld Trcadi bald
binak ow her hemen res det Ves hat wetted ryttrmnn hovm everree Tep. pieer even "yedhoser sunglowet" surow mwdt.
obravedd Marry- meal)
west wwg.ereeeee... Seninng Lal Jea poac) wec.eliliee "eeating redte Net "waiing. PuA
Ground Trush male Meg beard wearing
diservedin
Vidtern mofeesyeplessss Hisoo messues beard searing
apuinecs
young
Homan Peaue des the appernece sithe peraaii in vides
Ground This person
yourg.
Gemvie Hore destripton of the persen's apeearance videa: person the video is older mas with shart, Right blonde
Viltisn Marry and young He hos Moxdl
Figure 12: Visualization comparison results for appearance recognition and appearance caption.
35

---

Humant Peass wash the video arovided and iuadily the sutiic parforeed by the person vicho Rised on lovingline of actos plase silec allipp sodeactiio Pat
Gerrin 2.0: Pall X
cour and eegut home fret chaeved 13 theal Moon, jeet) tough loy: 2000 lart Town. jpar ileel has veginng Nar Nugh: iaaan mark, lec, acundd
Ground Brutt chabe head, amile, 10
Vidtmn: chaba hand, unle, tuk
Humarc lass sas Dee thaud penznivide Adds.
This made talle while criling fora long
Gumini 2.D. The women in the viduo speaking and resting her dhis on her theniing She touches than poents
X
Vidtmn: The man talks while sriling for long time.
M
V
V
Maman Plaase dest the actore persee vievedee
Grousd This man talla frowning long
Gereie The mon in the video speaking dioutly to caneraa He gestures with kis intially and then continues calk appears
realsssiiing
Vidtiso The salks frowes same time for long tire.
Figure 13: Visualization comparison results for action recognition and action caption.
36

---

Iraet night degrece Ln't degres Left Anginet Hight deginss Dowc Su.y
Ground Trutts
Vistmo
Hamani Pasle andlee the oven velosued anterine the hear pese Dhocse the mocil inperplati cetionattiiiion Frome
Gonnnni 2.0 Here's analysdi of the video: 6. Front X
Ground Truth Wes
X
Vidtio:
Humann Plecse onterment
porple oight part wile sreeperiss
Gemil 2:2: X
Figure 14: Visualization comparison results for head pose estimation and identity verification.
37

---

HArTES dese le accsures and aubitanu husurs th price
Hamar: Deccribe Pocal atribanss person crear ofthe
pensen
Ground The persas has alang faoe and beanded.
headdress paldon
she wortin with baiittpl..
Vint glasses sangosse, neckte, and ainng coloree beadureee,
Vietim: has bearded
whan Strdiier Eisenazr Cyning Enw.diieeee
When eetere hrme??
Homar
oeder ofe Jualt tap. aithout
Grosed Track
Gruad
Geneni 2.3
X
Valteoa
Humann updarow color persen vess?
oystron calor
Geebie 2.8: The person vides has dirk trown syelrres X
Wallieo The epderos color black
Homan What lenal acters poron at thecentor vikoot
des siora and Cin
Ground Truit actton indiatt
Gonis 2.0. in vidao looking right
Viliies balling
throughour
Havan: pertra coneeet wirte sollding
Hamone thht dusihore thelchtiee
Ground trusts
Ground Trushs Hi bal brown, dhart, and without banggs
Geneai Thene mssagh information reale deternttation X
Comen12.5 The
video hai and X
Vatro
Marta:: Whor rychnow coler of person the cvenr offT adni?
Hareas Whot grater bol ahaneended Aome Altendd
Groand Trath
Ground Trush The
freshics
Garini 2.3 Okay, ban'' the descrptiin: The woman has Bail
X
Viltme:
the peronit dest
viser somedieen C.
Greand
X
Figure 15: Visualization comparison results for open attribute perception.
38

---

H4 bow the DJ serpor Routsal 1E Eegarr thai Proter Coreeeee Milaggr meginseen Deuppedi shett
Gernet 2.0.
Vidiiee
Human:: "aave viteo
grained with inseriity peoseboree
A0 dagymm sor on degg 10 Mouu to eneerne Aoouuni mit castout jei tig ingusted-nn Lnd bequrced st
Sed MeckuniPSed bedi---ooe String E4 Sararted--- Weel pottedddddd---. Studurr Sarpradd Stonn 1Vt hauteal
Ground Truth
Goniin
Hurmarc Fiesse ride di and derth panoni fra
2.3:A,
Vilter: BF
Figure 16: Visualization comparison results for single-label emotion recognition, multi-labed emo- tion recognition and fine-gras fine-grained emotion recognition.
Ground Trathe
Gemin
Hamano: Peaas
hagerive
Vidtmne
wethh ritee disses reniment
Nesooall Puskles
video dia,
Homan:: Prmas vide dusmn centrment punzt's Wy You oo Hegil Reggite Wolk (fg be (y) b1 pioutenn Proilue
Ground Truth
Vidtmee
Figure 17: Visualization comparison results for micro-expression detection and action unit detection.
39

---

Plsse witch vic audihorrh racrune penta lie sappiest Kepretiian E Ueguut Siraare tttex
Ground Truth:
present mout cloch sovociteed isted with Reprosioo
Gamiin 2.0 Based on the video dig, the micro sxprendio
X
Human Alase
Villinoe
Muman: Menusteen the video det Mdeeri the action unto performent thetee senon thendeeo Gtred onithe folow blactie din man cener auo celle Enmerr un ai.4 bnrau lounice dilv vitur un frew to waer d61t comeee pullee shop
AUlScip ip aame exprazzr Sigraaee #426-1 sumer AUGS
Gemini 2.0. AU1I
Visttre: AUS AUU.AUUT.OUUAA. AUZS
Plamaat Plaus winch the video proelded and idersh the as3i unita (A4)) perforrunt the samonn thennd vidio Gtand on follow dluccion oster
AMG lanar
MutSi p sortet exprataa Au torire M42611 vater
drop.
Gretn 2.5: AUG, AU12, RUZS
Viltnn: AU12
Figure 18: Visualization comparison results for single-label sentiment recognition and fine-grained
sentiment recognition.
40

---

Hurrars watch the idu amunional benuwan churacess Hourst
inteact au with affic ouste be musiee, rcludiig wnt.sttppppicci contact (0 lass Diugres with other under
Ground
Vistmer
watch menelyyh
Nertr nerma white Axmetiooe cippciti Hostrvare alastant with
enractine with affic ou vac incid sate ahydcll contact lmmer sagge with ot
Vistfiucc
EE
NE SEEE
NEE
E
Plaman: Pleas procide deraild saptir Emortinn www.eettioaaaa dant amotic nal changes dis the adivkial tmrcugh sut videa.
Ground Thir man begine with sngre and then basa curprie,, in the
surgrte..
Gemiin 2.0 Here't brookdzwnn ematione disptayed mas video:
Oreral the man mairranne greecally neutral to dlightty serieus demoonas but eutit X
he speaks appeare engaged and thoughttu theoughout
Pl
pu
pu
pu
pu
P
Maman detalled curion dest tingth smotce trenthving amotioall charge dhppryed indleduur througher ofetr
with Truihs woran begira face she eventaaly turna
Comiai 2.D: Hard's breakdown woman's emotional exprasioro: "iystal Ereotion":, The
of surgris astorialment, her mouth open wids. **Shitt to Uneau"": As the video prograse, ahe dap unease or diguutt X contuntion "Coughing" abroplyy cougls, consring mouth
Vidtimo:
finally tumn bace cudaace.
Figure 19: Visualization comparison results for conversation reasoning and emotion caption.
41

---

Groand Trasts woman with strught, dark brown black weating red ipriic posiberned on apturned in sumil she salls the
frome, young sdult, Mady ergaged,
directed loword fir, further emphaising her uctive partsipution the convervation. summnn's
pleasent cpprossioon, cordered ned with her smill and the ale angles her head toweard, the man, suggolt connersation a posilive nuturee the partiallyy wiil on right sode the furme hus features that are diffcell to dicem doe to the limited vine and ligtingg but partion of fase and shoulder are wistll, sugposiing heis lacing woman, lighting and stati ruture enhance the seroe intimecy and focus on sobtle romeriial caes ketween the two intivesall.
are
obstrunted
stilaityy restritt understanningg focesed soledy
outhange, the
Gumini Z.c: The women is standing indeors, The buckground is blurry out of fecas but &t appears office simillrr busines cnwiromennt The lighting dim, creating a semee of perissinesa or privecy She wearing a white jecket. nnd cating profouionalan uum or formelits Based on thos siaal ovo. the woman is Bialy empiged in serioss comerrstionn in proficaional lusting.
reppoion phytica atrritutee individani, nuch color and myle, txcal
miieed
though and other andrrmmennil alements noted in
dersile lese spaciti
the pround bouhh [Scondl 65
2. Hleency The resporse is giammnttic ally comnett and smoothly sriicllned with dear sentence continctionn iin that mukes the decorption eay to understord (Scarel as ancurately captures genenul serting domeerer indiviuall mises specific detab suth otfeesity detullcd
Resporns Accuricy
movement
instructionn Adveranss: response adheres the Intructionn descriiing individaaion in the video, but lacks the depit capturing fodtal ustritsutes, emotional cans, and actions. The response provides a general descrption rither than detailer snalysh.. [Scond 55
Label Deerlap: The madeef' desn poion of mon's enationaal state neutral and posoiilyy smorrive or austenulaaaiv connellut owertogs with the ground truth, cantreppaatinn
respers some aprassoee and lack
facial
specili lighting offecs noted ground bruth. [Scon]]
room, young camera, She white or bluse red ipuiid, long, straight
makuup Her beadla tamned siihtlly towards enother pointed poattioned consistently
whon
directed. Inttally weating neutral expresioon bon the bugins speskngg and subde smils grodually appears, uptaming the comeen of her morth, indicring enpagiment and pandilly anurrment bical wetsss primorlly leacke calinng and oviling both appearing pensine These are na dipiifieet body mosemseets implyinn the partblly restivvely dets Red decolption thirr fonure represtion The soh, din oe alubiiliyy their intenction
they eapaped the stalcc soens contributts the intinvate feel of
bveluution 1. Vidoo Text Rulovence: The mode's repponse is doody rekood to the vboal comtant dhoolbed in the ground brtts, capturing simler especto such the
vorean' sppenrance, expresie rxpression, and the ambiaanee of wetting (Srene
responte granmntically accurnely
providing
deccliptin
[Ecore]:
Accuney: The
the quastion prodlinn detailed decttptiin video contant, with particater attention the woman's
exprestion and the interiction (Scam):: 15
instruction ranee repasse adhares the instruttion by direclly descrbingg sujjett fadal attribates, emotions, actions, and andramment
wilhott additional explonations Hoseree,
included spec ii desoripiire to man's features
(howe))
aharas igrificnnt overaa with truth identiying fedal abributes actions talking swriingg anvirnmmnnaal such sa ligting and setting [Bong) 95
Clue Overlep: The
modievatlly setting teasage boy with Mack wearing light live sweetshtt, the facing -ppd wornnn on haad oriented towards suggaaing lntening and partidpation the comernation though the linited of face malus ciffccat ascertain the full resitt his enutiosall respense.
Opaaiate Nan, the wcomon, with shart, stright blic hair and far compledan, deosee in browrih- beige cardgga Her slkondlshaped, dari bross eyes, namoo and furmand brow. mainealles contact with the
pirk tip, ropressann serinaa cosseened, indicited dightly downtuuree
haad rolatindy still. dightly sowerds him, Hur fesueed gasx and ongaged pastun, along with vobile roda, The camera capture their insecction, crvating privsry focat or thatr sarbor diccurios. The din lightigg enhances an atrooophare of quaet comeempatiio, the
agramment undersianding
exstc camera observes exxted pal:, eegaged an intenss eachangs.
Gentri 2.l:
Based
video, sppeare that woman teelnng concemed, and past bit frumantdd The setting tooks the stmple, commethen esowded public
spars, which with sense urgance the converation. Her feoul oxpossioe serioss, and the way she pross the quesiion sugpesto she's looling for homest srower.
Exaluatoo:: 3. Mideo Relevance The model's repprrse is someater aggeed with vhual content the video, capturing the general itmosphore and the expreas
specitic
divintiiing relevance
contant. [Scone]
2. Planna: The porse is grimmatiiaaly correct and smoothly artec iited with dear sentarce structure [Scors]:33 33
Respoese Accurnng While the model capturec comber strospheee and reviussens talk to sccurately decalbe dessiled budal eepressions appessonce, and
Achunana: meedion and Einuirti
sppeasionn,
and lacko speallic emotional utate dusrriptom, i abe induska anditimall explanatory slemet that focued the direct of descrbing the video
[boog): ag
Overleg: There limited overlap umotionel stats ciptiora as both to eserioza inteructiorc howewer the model resporse ledk spec the focial
Clue Overtop: model Ideatifes teg/omale din Agnting but mirrre everl uiai detalle Bhe tacbi attribaeee, clettin and actions noted Ground frath Arownr Boorc) 35
brightl
courtee yourg
pullue fasaground partilly
comiiiently
week cartigan periweele meintaning exerios, kos and cassumed demanns, Her oydrross subilly furroeed, and her mouth comer om sighnty downoureed in convercaiin with the young man, the secadorally revsk her terth while guaddngg and her gane nemalic flned on him. The overall scane sugantte serices conernation with uoman appnaring concorted poterttady delivriig upaeting news or expresinng warry and the young man seemiingyy tibweing atbershwely
remran indistiee. The camera stry stati, emotlousl
backgrsund, another interaction between woman the young
Exsiaaton: 3. Motte-Tot telessnce The model's responte appsan panerally correpondd with the video desorittion socursttly focuting on the suppects and the interaction betweee them Hovemeey there are sight dicrepancis in the detall, sac os the setting descrpation Bbaard
Plaeno The raporns is grammsticall correct, and sentances wall di-uttttddddd providing nsmosth narrativ flom [Soon) Dusponos
light swoutahist repeatedd incicating
miner precaiion [Boan]] 30
6. Insbrction Achorans:: The model adherus to the instrction by druecly desorlinn the sutjects and thair entronment with extranoo commentary yet the oetting and enaloommenta desale vary dightyy camaariese with the graan (Kooe) as The
cigpuring key reperesioss Che OueiippTh medel cxptures the recestrry reaties- waated duas thial aepandone and owraie finer deraale speeeitic conditoro and attine deser show partial werkay [Scon]]
ground
[Sooo]]
Figure 20: Visualization comparison results for fine-graine emotion caption. We achieve competi- tive performance with Gemini 2.0 on six different metrics.
42

---

Humon: Poose warc the vóco provided and iecerrr the actions pe ford by theperson the video, Boood on the follwwing rg lst actiono aloas octer spoiiabie aetiene otter outper kone nazt obennent (hlow, cheer, zine sund toughh try, denaal lest, Troont "pare i gane hos"" ykay instrmmettt hexef hoad; Sight Singl Sloap/,
head_ mugging" Nis" Sottnn,,, musi, around) lwrep'
frowe, nod,
Human: Watch video and icentlyy oppserance rasocited person to bslosw and under ever, lepegtmoen; 'unngtonnn"" "samas, S. clock Shadow, pooted Trusiaded, Sikdumss' Thanyy makcuuy war carings: Swcaring sall Weering Fostikk woainng recklaee Weering recitee weaaing_mast mask]
young!
high creakbenec 'ig. jpr, idunne chei tho beard
gray. haii.eyoglasees.pccinn
necklace
Human: Please decri theactlors the vidri
Human: Moase desoliee the uppcara toe of the the widoo
Answer: She young. She has high checkbones. She wearing earrings.
pose. appereriiee option Down
dagres
Figure 21: Visualization samples for attribute perception in classification-yype tasks.
43

---

10
10
107
10
10
10
Humant
indieding
Answer:
blonds, past shoulders bangs
epetrount
Answer:
The person has thick eyebrows with black
color.
Human: Describe the aye atti perzon right
videe, indidding the rine.
eye eye cole, syctils, undir
Answer: Her eyes normal she and almond almond-shaped with brown color, She has double eyelids, without bags or dark circles.
Human: Desolibe the ovewaar of the person on the side vilio.
Answer: He wwering glasses.
Human: Does the perton i owthene ght of the viec howe dimplec?
Answer: The dimples absent.
Answer: The face shope is oval.
Figure 22: Visualization samples for attribute caption in caption-type tasks.
44

---

Humant Presse waich the video prow ded and iderritly the untion-nits (Al) performed by person the video, fooed on the Billowiing actionn and thei oca inlinattooo ploas select applicable astion unita that ocsus reply dinectly to rame caech us AUL, AUS and output None observed: comner publer
AIlI tiner time piser
hrow moier brme fowenee pressor part jaw crop
But? nigheeeer Aifid: uppect up
sorner depresser
light
watch identlly oxprosaionn person's
Angry Digguet
Angry Weok Angry Mediam String Conmeng Contenon ideaiim Contenpt String Digguted teek
Duguated Ned un Dlggaood Store foar Weat (8] Med four Strong happy 'Waat Happy Medium (0) Happy Strong Sed Weak (G) Medium bad Strong 50 Surprised Weak (f) Surprised Mesium Surpriedd Strong Noutral
Planra deuriba the esprecier person video.
happiness.
Human: flesse watch the cideo and iolenoffy the cmtetional rekaionnliis between the two main main dharactess in the video: Houtrall Neutmills, dalcgge ametionat stspention hills leertowrriing elenant allog in intimater lowing Antagoniatic
Disagres
Answer:
Figure 23: Visualization samples for expression analysis.
45

---

Caption: dimly Indoor setting, middle-aped Asian woman with short, straight
complesion stands
is siighlyy dowmwards, and somewhhst sad sppearance. Tears are visible on face she engages a serious conversation, her mouth moving the speaks to young adult fast Asian man who enters the frams on the right side. The young man, with short cropped black hair, is wearing a light blue over-ahit and simple white t-shirt. He listens attentively, his gane fixed on the woman, and his exgression ranges from neutral to slightly concerned, with furrowed eyebrows at times, He responds to the woman, spealking in reture. The scene is serious, centered on the interactoon between the woman and the man, with soft, diffused lighting casting subtle shadows on their faces. The bluree background hints traditional East Asian decar, The focuses their facms and bodies, emphasting their expresions and eochange. The woman's sadness expression
gaee directed off off-ccreen the right. Her downturned mouth and eyes, along with slightly furrowed syebrows, give pensive
the focus, while man's attenthvenes and diigtlly
he providing support comfort.
Caption: In static clos-up shot with saft, even lighting, two East Asian Ieely their young adulthood, are engaged in converatioon the left side of slightly
short black hair styled bowl rounded happy or amused
upwards appears clean shaven.
speaking, head widens.
initially downturned
talking, and this change expression directly related talking and smiling: Positioned on right of frame, other man short black hair in bowl cut and wears a similely colored shirt. He maintains relathvely posture with his head turned towards the man speaking the left, scemingly ottentive to the ongoing conversation. Initially appearing neutral perhaps penshre, his demeanor complements the static noture the scene, Due to the close framing of the shot, no body actions are visible spart from the subile head movements, and the scene remains static throughout with no apparent camera movement, giving the overall impression of a casual, friendly exchange between the two men.
Caption: A young woman with fair sMn, dark brown eyes, and shoulder-ength back hair tied In low ponytall stts on bench a sunlit park, wearing a light- rolored cable-knit sweater vest, small pearl earrings, and a watch. She holds a dark blue jacket with white stripes, soumingly a school uniform, and presents it to child sitting slightly lower on the berch. The child has fair skin and shoulderiangth black hair styled in low ponytall, sdomed with colorful hair tims, and wears plaid shirt and jeans, Initiall, both the women and the child dowmsards mouth upturning eyes crinkligg slightly, she lifts her head response,
jecket with neutral and focused expresions. The woman smiles affectionately the child, her engage with the child, gentle happy
she adjusts the child's clothing gass.
reflects
child's
the happy expression.
child maintains eye and connection between them, captured by static camera focusing solely on their Interaction.
widening
Caption: In dimiy setting featuring brick archway hanging light, a middle-aped man, pasitioned centrally frame, is the focel point. wears a suit and a black necitic, presenting a formal appesrance, His short hair is light brown, and has some stubble on his face. His sideburns are visible, and he is otherwise clean-shaven. He is not wearing glaises or any visible accessoris other than his tis. He maintairs a serious expression throughout, characterized by a slight furrow in his brows and downtumed mouth corners, His head is sightly titted downwards, and his gaze remains directed off screen, focused on something below his line of sight. He remains completely still, eshibiting body movement. facial expression, along with his fised and still posture, emphastees absorption whatever beyond frame. background remains constant, with
an seriousness and focused
attention. The man's movement and unchanging lighting arched passageway contribute the
somber serious
The features both scemingly East Adion, engaged serfous discussion uniform with dark
Indoor backdrop. adult posttioned hair
blue lips natural pinbtih towerds the man on the right, his head slightly tilted turned. Subtle movements of lips and jew indicate he's speaking, whis his serious, neutral espression and largely unchanged posture sugpest focused conversation.The The second young adult man appears more casually dressed dark jacket over a white collared shirt and dark necktie. His short, dark hair is styled with bangs that fall just above his eyebrows. His double eyclids are less prominent, and his nose features high bridge and pointed tip. His lips, like the fint man's, have a natural pinkash hus. He mainly lnstens attentioely, his head slightly turned towards the man in the police uniform. His downward gaze and serious expression Indicate contemplatton reserved reaction. Subtie lpp movements sugpest betef verbal responses as he primarily engages through active listening,Te static nature their Interaction emphasteed netther making significant movements; their bodies remain mostly undorfining focus
from forehead,
double eyelids, almond-chaped eyes, high bridge
gaze
exchange. Both
iluminated, highlighting their features expressions
Figure 24: Visualization samples for emotion understanding in fine-grained caption task.
46

---

Theme
(b) Surprise heutral (0)
feor Conteept
Human:
Please anahov which parts of the envinnment, lightigg bec secugrondd, visua atu butos ets, can caplan reason for your answes
recognition analysis begins by focusing on attributes relevant to expression of surprise, eye movements and changes the mouth's position, indicators commanly associated this umotion. situations involving surprise, the epen typicelly widen vidin, which corresponds with attentive readiness to unespected stimuli, However, the current information on eye features this analysts limited-the eye cornert are noted as neutral, and specifics about eye widening are not provided, potentially due lighting and image quality tsues. Nonetheless, the mouth's activity offers notable insight, it's initally slighty open, aligning well with surprised expression where the mouth often naturally opens of reflec unsopected events. The subequent closing of the mouth might by brief frame above the charecteristic surprise responss.
Reasoning:
the surpris resolved swiftly, observation supported
remains straight directed forward, further lending the While gaze Itself oxplicitly widened, direction suggests encounter something unexpected above usual sight,
engagement
surpriing stimulus, with
directed
Despite the neutral posttloning of the eyebrows, which may also raise slighlly surprise, they do not contradict the expression given their stabiitty within the fimited provided data.
finally, the lighting conditions (diffused and naturall imply that visibility adequste for genuine micro-esgresion to be captured, though some shadows unist. The presence of natural light without obscuring eccessorias indicatus authenticity in the individual's reaction. composite
attributes-the inittal mouth opening, the direct slightly elevated gron, contesttual clues around these markers guoe with positioning corroborates
suggests coordination
limited
widening
activity,
possibly surprise
Numan: Poase witch the video cii and ideralyy the expessionn on on theasssn's froe (Al Anditty (Bo Surpeise (5l Neutral (2) Oligaar (I] Secinsa (1) hear [0] Costampt 3r Anger 10 pep Duappo nomant IK) happiness
Answer:
Human: Mlsase anahos parts of the ervionment, lightngg bec sacugronnd
Remoning: The analysis of the video's emotional indicators begins with facial expressions that convey clear emotional The positioning of the emotions as
provides vital they sligtlly downturned the which associaed potentially reflecting disapproval unhoppiness. Comphementarily,
further
eychrows, however, focused, negative euskuation of something unplessant disspreable. head posture,
disustt dzasproval, as suggests significant claes-being sightly furrowed. action classic
slight downsard adds enother layer by potentiall indicating a reflective or Introspective stance, whith often accompanies emotional states involving judgment or disapproval.
The pace directio- locking downward and slightly to the left- can also imply an avoidance of direct eye contact, possibly supporting the notion feeling negatively about an unseen stimalus or interlocutor. Eyegiasses and low lighting might obscure finer details, but the analydss the above features provides cohwsiee understanding the emotional state.
Taken together, features-thh the downward tlited head, the slightly furrowed associated expression disgust. This combination sugaests
downtumed comers, and dowmeard gare-form evaluation charactoristic perceived
Figure 25: Visualization samples for emotion understanding in rationale analysis task.
47

---

Human: Dexcribe contered including evatrose thicosns
eyebrows
Humani
Answer: The face shape ovall.
Figure 26: Visualization samples of labels of face boxes, face landmarks and parsing masks.
48


Reviews:
[{"review_id": "2Cvsdyxxlc", "reviewer": "Reviewer_KQSY", "summary": "This paper proposes VidEmo, a video MLLM for facial and emotional analysis. It integrates curriculum emotion learning with the affective-tree reasoning paradigm. Besides, a large-scale emotion dataset, Emo-CFG, is curated, covering question-answering, fine-grained captions, and associated rationales. Experiments show that Emo-CFG achieves competitive performance across 15 tasks.", "questions": "1. What is the backbone MLLM of VidEmo?\n2. What GPUs and how many GPU-hours are used for the experiments?", "limitations": "See weaknesses", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper applies the novel reasoning technology to the emotion tasks and proposes a task-specific variant, which advances the research progress of the emotion-centric problems.\n2. A new dataset, Emo-CFG, is constructed, which is useful for training and benchmarking emotion models.\n3. The experiments comprehensively compare with multiple types of baseline approaches on multiple tasks and metrics.\n\nWeaknesses:\n1. This paper aims to develop an emotion foundation model. However, it mainly focuses on face perception tasks. Non-facial video emotion tasks remain underexplored [r1, r2].\n2. In Table 2, VidEmo-Base-7B performs worse than VidEmo-Base-3B. VidEmo-T1-7B achieves marginal improvements over VidEmo-Base-3B.\n3. The case of VidEmo-T1-3B is not considered.\n\n[r1] Mazeika, Mantas, et al. \"How would the viewer feel? Estimating wellbeing from video scenarios.\" NeurIPS, 2022.\n\n[r2] Guo, Yuxiang, et al. \"Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models.\" IJCV, 2025.", "was_revised": true, "final_justification": "My concerns have been addressed. I'm raising my rating to 5: Accept.", "rebuttal": "# General Response\n\nWe sincerely appreciate all the Reviewers and the Area Chair for their time and effort in reviewing our paper. We appreciate all four reviewers for acknowledging novel idea (Reviewer KQSY), interpretable and sensible methodology (Reviewer JqQW, 96WN), well-annotated large-scale dataset with validated quality (Reviewer KQSY, 96WN, ntby, JqQW), extensive experiments (Reviewer KQSY) and strong performance (Reviewer 96WN, ntby, JqQW). In response to the reviewers’ suggestions, we have provided the following **additional results and evidence** in the rebuttal:\n\n- We conducted new experiments on non-facial video emotion tasks, demonstrating VidEmo’s generalization capability on the VCE dataset. **[A1 for KQSY]**\n- We verified annotation fidelity by showing that the newly labeled data achieved a 90.3% consistency rate with the original human labels in CelebV-Text. **[A1 for 96WN; A2 for ntby; A3 for JqQW]**\n- We conducted a user study comparing Emo-CFG with CelebV-Text, where Emo-CFG was preferred in 93–95% of cases across precision, rationality, and complementarity dimensions, with statistically significant results confirmed by the Wilcoxon signed-rank test (*p* < 0.01). **[A1 for 96WN; A2 for ntby; A3 for JqQW]**\n- We clarified that our data source selection strategy as well as the ablation studies on dataset sources were already included in the original manuscript. **[A1 for ntby]**\n- We added comparative results against emotion-specific VideoLLMs (e.g., Emo-LLaMA) on the MAFW and DFEW benchmarks, showing consistent gains in both UAR and WAR metrics. **[A4 for ntby]**\n- We confirmed plans to release the Emo-CFG dataset, VidEmo models, and all related code to support community use and reproducibility. **[A5 for 96WN]**\n- We revised the manuscript structure for improved clarity and explicitly referenced key supplementary content within the main text. **[A2 & A4 for 96WN]**\n\n# Response to Reviewer KQSY\n\n**A1: Non-facial Video Emotion Tasks**\n\nThank you for highlighting this important aspect. (1) **Indeed, with the benefit of large-scale emotion-centric pretraining, VidEmo effectively generalizes to these tasks.** We conducted new experiments and clarified the generalization ability of VidEmo beyond facial perception. Following the protocol in StimuVAR [r2], we evaluate VidEmo on the large-scale VCE dataset [r1], which focuses on non-facial video emotion. VidEmo achieves strong baseline results, showing its broad emotional modeling capability. (2) **Our work can support these tasks from both the data and model perspectives.** Emo-CFG scales up the pretraining corpus from 60k (VCE) to 2.1M multimodal emotion-centric samples. The distilled emotion logits from VidEmo can also be used to supervise downstream models like StimuVAR, enhancing their affective reasoning. These results and analyses will be added to Sec. 5 (Experiments) and Sec. 2 (Related Work), respectively.\n\n| Method            | VCE (Top-3) |\n| ----------------- | ----------- |\n| Chat-UniVi        | 38.6        |\n| mPLUG-Owl         | 23.6        |\n| Qwen2.5-VL        | 41.3        |\n| StimuVAR [r2]     | 73.5        |\n| **VidEmo (Ours)** | **85.8**    |\n\n\n\n**A2: Performance**\n\n(1) **VidEmo-Base-7B vs. Base-3B:** Despite mixed trends in some metrics, Base-7B shows consistent gains on Emotion (+3.2), Sentiment (+2.6), and Complex (+4.3) tasks. These tasks involve more subtle and abstract understanding, where the 7B model excels.\n\n(2) **T1-7B vs. Base-3B:** VidEmo-T1-7B improves across all five sub-tasks: Emotion (+4.7), Sentiment (+3.4), Cues (+0.1), Complex (+6.8), and Understanding (+0.9). Especially for Complex, the T1-7B model shows better dialogue-text reasoning and is well-suited for RL-based training in the next stage.\n\n\n\n**A3: VidEmo-T1-3B**\n\nWe appreciate the reviewer’s insightful comment. We have carefully attempted to train VidEmo-T1 with the 3B-scale backbone, but encountered challenges during post-training. Specifically, the 3B-scale model was found to be prone to model collapse [r3] due to the inherent complexity and abstraction involved in emotion understanding [r4]. This task requires fine-grained emotional attributes and expression captions, which, as we have observed, exceed the capacity of the 3B model. Our experimental results further corroborate this. We noticed a sharp decline in the reward function during the middle of training, reflecting an inability of the 3B model to maintain stable performance as the training progressed. Given the importance of accurately modeling emotion at different levels of abstraction (from attributes to expressions to emotions), we recognize that scaling up the model is crucial. This observation has informed our approach, and we are actively exploring the use of larger model variants, such as VidEmo-Base-7B, to overcome these limitations and enhance the model's ability to handle complex emotional representations.\n\n\n\n**A4&A5: Training details, including MLLM backbone, GPU time, and hyperparameters**\n\n(1) **Common Setting as ms-swift:** To ensure reproducibility and fairness, we adopt the widely used Qwen2.5-VL series as the backbone MLLM for both 3B and 7B variants of VidEmo. Qwen2.5-VL offers robust visual-text alignment and is compatible with mainstream deployment frameworks such as vLLM and HuggingFace Transformers. This ensures ease of integration for downstream researchers and practical deployment.\n\n(2) **Training Infrastructure:** VidEmo was trained on 4× NVIDIA A800 nodes, with 2.8 days for SFT and 7.8 days for RL training. Training was conducted with a batch size of 1024 (SFT) and 128 (RL).\n\n(3) **Implementation Details:** We follow the common two-stage tuning paradigm (SFT + RL), using AdamW optimizer with cosine learning rate schedule. A ViT-based visual encoder is used alongside the autoregressive LLM. Our training pipeline closely follows standard MLLM configurations.\n\n| Model          | LLM        | Vision | LR   | WD   | BS   | Hardware | Time           |\n| -------------- | ---------- | ------ | ---- | ---- | ---- | -------- | -------------- |\n| VidEmo-Base-3B | Qwen2.5-3B | ViT    | 2e-5 | 0    | 1024 | 4× A800  | 1.7 days (SFT) |\n| VidEmo-Base-7B | Qwen2.5-7B | ViT    | 2e-5 | 0    | 1024 | 4× A800  | 2.8 days (SFT) |\n| VidEmo-T1-7B   | Qwen2.5-7B | ViT    | 1e-5 | 0    | 128  | 4× A800  | 6.3 days (RL)  |\n\nFor more details, please refer to Tab. 1 and Lines 1–13 of the supplementary material. Due to space limitations, we placed the implementation and training details in the supplementary. In the camera-ready version, we will move this content into the main paper to improve clarity and ensure all key components of our method are presented in the main body.\n\n\n\n[r1] Mazeika, Mantas, et al. \"How would the viewer feel? Estimating wellbeing from video scenarios.\" NeurIPS, 2022.\n\n[r2] Guo, Yuxiang, et al. \"Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models.\" IJCV, 2025.\n\n[r3] Cui, Ganqu, et al. \"The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\" arXiv:2505.22617, 2025.\n\n[r4] Zhao, Sicheng, et al. \"Emotion recognition from multiple modalities: Fundamentals and methodologies.\" SPM, 38(6): 59-73, 2021."}, {"review_id": "YNifSHCAGA", "reviewer": "Reviewer_96WN", "summary": "This paper introduces a novel fine-grained, large-scale emotion understanding dataset by aggregating 17 existing datasets, designed to comprehensively capture emotional expressions in complex scenarios. To address the limitations of existing methods in handling such challenging contexts, the authors propose two innovative approaches: Curriculum Emotion Learning and Affective-Tree Reasoning. These methods are integrated into a newly designed base model, which demonstrates significantly superior performance compared to existing approaches across multiple evaluation tasks on the proposed dataset, highlighting its strong generalization ability and practical applicability.", "questions": "1. Please see Weakness.\n2. Is the Emo-CFG dataset planned to be released publicly?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tA large-scale (2.1M) emotion understanding dataset, Emo-CFG, is constructed.\n\n2.\tTwo promising methods are proposed: Curriculum Emotion Learning and Affective-Tree Reasoning.\n\n3.\tThe methods show strong performance, outperforming existing LLMs (1-8B).\n\nWeaknesses:\n\n1.\tEmo-CFG is reviewed only by three VideoLLMs without human validation or user study, raising concerns about its reliability.\n\n2.\tMethod descriptions lack clarity and theoretical grounding; the discussion in Section 5.3 could be moved to 4.1 for better structure.\n\n3.\tExperiments are limited to Emo-CFG, without cross-dataset evaluation to test generalization.\n\n4.\tKey information in the supplementary material is not explicitly referenced in the main text. For example, the additional details in Section 4.1 should refer to Table 7 in the supplementary C.2.", "was_revised": true, "final_justification": "The inclusion of extensive cross-dataset experiments, as well as human validation and a user study, enhances the persuasiveness of the authors' work.", "rebuttal": "# Response to Reviewer 96WN\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results in our response to Reviewer KQSY.\n\n**A1: Human Validation and User Study**\n\nAs suggested, we conducted the human validation and user study and show that the reliability matches the previous largest existing dataset CelebV-Text.\n\n(1) **Human Label Verification**: As suggested, we further validated label fidelity by extracting fine-grained captions from the generated videos and comparing them with the original human-assigned labels of CelebV-Text. The consistency rate reached 90.3%.\n\n(2) **User Study Verification**: To assess the quality of the generated expressions, we conducted a user study on a manually inspected subset of test samples to verify their alignment with the intended emotional semantics. Specifically, we compared Emo-CFG with CelebV-Text, the largest human-labeled video emotion dataset, across three key dimensions: precision, rationality, and complementarity. Preference rates for Emo-CFG across these dimensions reached 95%, 92%, and 93%, respectively, with statistically significant differences (Wilcoxon signed-rank test, *p* < 0.01). This result demonstrates that Emo-CFG provides more precise and expressive emotional representations than existing benchmarks.\n\n| Dimension       | Win  | Tie  | Loss | # Video | # User | Prefer | p-value | Significance |\n| --------------- | ---- | ---- | ---- | ------- | ------ | ------ | ------- | ------------ |\n| Precision       | 964  | 204  | 82   | 50      | 25     | 95.52% | 0.00021 | Significant  |\n| Rationality     | 1082 | 87   | 81   | 50      | 25     | 92.16% | 0.00015 | Significant  |\n| Complementarity | 1172 | 23   | 55   | 50      | 25     | 93.04% | 0.00008 | Significant  |\n\n\n\n**A2 & A4: Method Description**\n\n(1) **Methodology:** High-level emotion understanding is a challenging task due to the abstract nature of emotion [r4]. Even existing state-of-the-art models (e.g., Gemini 2.0) can only achieve 26.3% accuracy. To address this challenge, our research is indeed motivated by cognitive principles and inspired by the human cognitive process [r5]. We propose an affective reasoning scheme and inject it into the SFT stage (Sec. 4.1), RL stage (Sec. 4.2), and inference stage (Sec. 4.3). As indicated, we added the above analysis as an overview paragraph in Sec. 4, presented in a structured manner to clarify the theoretical grounding.\n\n(2) **Re-structuring**: We agree with the suggestion and move the analysis in Section 5.3 to 4.1.\n\n(3) **Main Body and Supplementary**: To further strength the structure clarity, we have revised the manuscript to explicitly integrate key supplementary content into the corresponding sections of the main paper.\n\n| Main Paper            | Supplementary                | Description of Integrated Content                            |\n| --------------------- | ---------------------------- | ------------------------------------------------------------ |\n| Sec. 4.1 (SFT Stage)  | Supplementary C.2 (Tab. 2&7) | Added specific sample illustrations of attribute-expression-emotion triplets |\n| Sec. 4.1 (SFT Stage)  | Supplementary F              | Incorporated references to detailed visualization results.   |\n| Sec. 5 (Experiment)   | Supplementary A              | Incorporated references to detailed training settings.       |\n| Sec. 5.2 (Discussion) | Supplementary C.1 (Tab. 3-6) | Included ablation study results for clearer model comparison. |\n\n\n\n**A3: Cross-Dataset Evaluation** \n\n(1) Indeed, the evaluation benchmark is constructed by merging and cleaning a broad collection of open-source datasets, while strictly following their original training-testing splits to prevent data leakage. The evaluations for expression analysis and closed-set attribute recognition are conducted using open-source datasets.\n\n(2) Further, as suggested, we conducted cross-dataset evaluations on the video emotion benchmark VCE [r1], following the evaluation protocol in StimuVAR [r2]. VidEmo achieves strong baseline performance across all datasets and performs competitively with the domain-specialized method StimuVAR.\n\n| Method            | VCE (Top-3) |\n| ----------------- | ----------- |\n| Chat-UniVi        | 38.6        |\n| mPLUG-Owl         | 23.6        |\n| Qwen2.5-VL        | 41.3        |\n| StimuVAR [r2]     | 73.5        |\n| **VidEmo (Ours)** | **85.8**    |\n\n\n\n**A5: Open-Source** Yes, we plan to publicly release the Emo-CFG dataset, the VidEmo-3B/7B models, and all associated training and inference code upon acceptance of the paper. We will also provide support for easy deployment via Transformers and HuggingFace interfaces to facilitate community use and reproducibility.\n\n\n\n[r1] Mazeika, Mantas, et al. \"How would the viewer feel? Estimating wellbeing from video scenarios.\" NeurIPS, 2022.\n\n[r2] Guo, Yuxiang, et al. \"Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models.\" IJCV, 2025.\n\n[r3] Cui, Ganqu, et al. \"The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\" arXiv:2505.22617, 2025.\n\n[r4] Zhao, Sicheng, et al. \"Emotion recognition from multiple modalities: Fundamentals and methodologies.\" SPM, 38(6): 59-73, 2021.\n\n[r5] Grill-Spector, Kalanit and Malach, Rafael. \"The human visual cortex.\" Annual Review of Neuroscience, 27:649–677, 2004"}, {"review_id": "DixZRktkBu", "reviewer": "Reviewer_ntby", "summary": "This paper proposes a novel video emotion base model called VidEmo, which enables fine-grained and interpretable understanding of complex emotions in dynamic videos by combining curriculum emotion learning and affective-tree reasoning frameworks. The paper also constructs an emotion-centric fine-grained dataset, Emo-CFG, containing 2.1 million diverse instruction-based samples, which provides an important resource for emotion understanding tasks. Experimental results show that VidEmo outperforms existing open- and closed-source models in 15 face perception tasks, with advantages in emotion understanding and reasoning capabilities in particular. The authors also point out the limitations of the model in terms of possible generation of false content and multimodal integration, which provide directions for future research.", "questions": "See Weaknesses 1~5", "limitations": "The proposed models of this paper may be used for emotion monitoring or privacy invasion (e.g., inferring mental states through micro-expression analysis), but the paper only briefly mentions \"limitations\" without discussing specific safeguards (e.g., data desensitization or usage restrictions).", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. A new video emotion base model (VidEmo) is proposed, combining curriculum emotion learning (phased injection of emotion knowledge) and affective-tree reinforcement learning (hierarchical reasoning) to enable fine-grained analysis and interpretable reasoning about dynamic and complex emotions in videos. The models derived from the base model achieve better performance on 15 facial perception tasks (e.g., attribute recognition, expression analysis, emotion understanding) compared to existing VideoLLMs.\n2. A new emotion-centric fine-grained dataset (Emo-CFG) is constructed. It covers detailed annotations across attributes, expressions and emotions, and supports full chain learning from low-level attributes to high-level emotions. The data quality is ensured through a committee voting-based verification strategy, which addresses the inherent ambiguities in emotional data due to its subjective nature.\n\nWeaknesses:\n\n1. The source data of the proposed Emo-CFG dataset is from 17 existing datasets. The paper does not give an in-depth analysis and discussion on the question of whether the possible bias in sample labeling and distribution between different source datasets affects the quality of Emo-CFG.\n2. The generation of some of the annotations in the constructed dataset relies on GPT-4o and Gemini 2.0, which may introduce biases or errors inherent to large models, and the paper does not analyze in detail the impact of differences between synthetic data and manual annotations.\n3. Affective-tree rewards may have limitations. The construction of ground-truth tree may be subjectively influenced by the annotator. The consistency of reasoning paths with human intuition was not quantitatively analyzed in the paper.\n4. The authors acknowledge that the model may generate \"false narratives\" or \"emotionally inconsistent descriptions\" but do not propose specific mitigating measures (e.g., post-processing filtering or uncertainty calibration) that may affect the reliability of actual deployments of the proposed models.\n5. The evaluation is insufficient. The proposed model is not compared with the latest dedicated expression/emotion analysis models (e.g., Emo-LLaMA, Exp-CLIP, ExpLLM) on the same dataset, and open-source model comparisons are focused on generic VideoLLMs, which makes it difficult to prove that it outperforms the optimal models in the field of expression/emotion analysis.", "was_revised": true, "final_justification": "The authors gave detailed responses in their rebuttal, which have addressed most of my concerns. However, there is a certain gap between the author's rebuttal and the original manuscript. Much of the important information, as well as the related analysis and discussion, is disclosed in the rebuttal but is completely missing from the original manuscript. The rebuttal appears to be an extension of the original manuscript rather than an explanation of its unclear points. Frankly speaking, the prerequisite for this paper to be accepted for publication is that the manuscript undergoes thorough revision and improvement. Furthermore, in my opinion, when stripped of the trappings of large language models, the innovation and contribution of this paper are actually limited. There is still a certain gap compared to the high standards of NeurIPS. Therefore, given the author's careful and thoughtful response, I am willing to raise my score, but I still do not recommend acceptance.", "rebuttal": "# Response to Reviewer ntby\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results in our response to Reviewer KQSY.\n\n**A1: Dataset Source** \n\n(1) In fact, we conducted a comprehensive literature review and initially collected 108 video datasets following the emotion data taxonomy (Fig. 4a). To mitigate source bias, we applied a series of filtering rules to select 17 high-quality datasets that offer broad coverage across video length (Fig. 4b), visual clarity (Fig. 4b), and caption-related affective cues (Fig. 4c). Datasets exhibiting strong domain skew or poor annotation quality are deliberately excluded to ensure the reliability and representativeness of the final dataset composition.\n\n(2) To directly address concerns regarding source-induced bias, we have conducted an ablation study where samples from different data sources and emotional stages (attribute → expression → emotion) are progressively integrated (Tab. 7 of supplementary). The results demonstrate that multi-stage curriculum learning, which incorporates all data sources in a structured manner, significantly outperforms single-stage variants across all tasks.\n\n| Data Source      | Attribute | Expression | Emotion  | Average  |\n| ---------------- | --------- | ---------- | -------- | -------- |\n| Attr             | 65.7      | 24.1       | 32.4     | 40.7     |\n| Exp              | 52.9      | 31.6       | 42.1     | 42.2     |\n| Emo              | 64.3      | 30.2       | 67.3     | 53.9     |\n| Attr+Exp         | 77.2      | 35.1       | 45.5     | 52.6     |\n| **Attr+Exp+Emo** | **79.5**  | **38.7**   | **67.5** | **61.9** |\n\n\n\n**A2: Dataset Annotation**\n\n(1) **Cognition-inspired Human-led, Model-assisted Data Construction**: Indeed, Emo-CFG is not solely labeled by VideoLLMs, but  also constructed through a human-led, model-assisted pipeline that incorporates cognitive principles and human priors. Inspired by abstract human cognition process [r4,r5], we design a causal perception-to-emotion pathway that begins with stimulus perception (Attribute), proceeds through holistic expression organization (Expression), and culminates in emotion understanding (Emotion)—as detailed in Sec. 3. Further, we adopted a *human-led, model-assisted* pipeline during dataset construction [r6]. Human annotators first provide attribute and expression labels as semantic scaffolding, and models then generate fine-grained captions conditioned on these labels. This design ensures alignment with human cognitive logic while maintaining scalability and annotation efficiency.\n\n(2) **Human Label Verification**: As suggested, we further validated label fidelity by extracting fine-grained captions from the generated videos and comparing them with the original human-assigned labels of CelebV-Text. The consistency rate reached 90.3%.\n\n(3) **User Study Verification**: To assess the quality of the generated expressions, we conducted a user study on a manually inspected subset of test samples to verify their alignment with the intended emotional semantics. Specifically, we compared Emo-CFG with CelebV-Text, the largest human-labeled video emotion dataset, across three key dimensions: precision, rationality, and complementarity. Preference rates for Emo-CFG across these dimensions reached 95%, 92%, and 93%, respectively, with statistically significant differences (Wilcoxon signed-rank test, *p* < 0.01). This result demonstrates that Emo-CFG provides more precise and expressive emotional representations than existing benchmarks.\n\n| Dimension| Win  | Tie  | Loss | # Video | # User | Prefer | p-value | Significance |\n| - | - | - | - | - | - | - | - | - |\n| Precision       | 964  | 204| 82| 50| 25| 95.52% | 0.00021 | Significant  |\n| Rationality     | 1082 | 87   | 81| 50| 25| 92.16% | 0.00015 | Significant  |\n| Complementarity | 1172 | 23| 55| 50| 25| 93.04% | 0.00008 | Significant  |\n\n\n\n**A3: Affective Tree Rewards**\n\n(1) We fully agree that emotion understanding inherently involves subjective interpretation. To reduce subjective bias in constructing the affective tree, we employed a voting-by-committee strategy. Multiple annotators provide independent annotations and a majority vote is used to resolve disagreement, which is consistent with best practices in emotion datasets.\n\n(2) The results of the user study show that our affective trees are consistently preferred in terms of complementarity, precision, and rationality, with an especially high 93% preference rate in the rationality dimension for the reasoning path. This indicates that the reasoning paths in our dataset closely align with human affective interpretation. Moreover, incorporating the reasoning path as a reward signal leads to consistent improvements over the GRPO baseline across all sub-goals (attribute, expression, emotion). Further integrating tree edit distance (ATR) into the reward function provides additional gains, underscoring the effectiveness of structure-aware supervision.\n\n| Method| Attribute | Expression | Emotion  | Average  |\n| - | - | -| - | - |\n| GRPO (w/o Tree Reward)| 75.7| 33.8| 44.2| 51.2|\n| + Tree Reward| 80.9| 38.1| 65.3| 61.4|\n| + Tree Edit Distance (ATR) | **81.3**  | **40.1**   | **69.3** | **63.6** |\n\n\n\n**A4: Mitigating Hallucination** We thank the reviewer for recognizing our analysis of harmful false narratives. As described in Section 4.3 of the original manuscript, we propose an emotion-centric post-processing strategy to mitigate hallucinated outputs. Specifically, we sample multiple descriptions at each stage (attribute, expression, emotion) and select the one with the highest logit score. This filtering process effectively eliminates most false or irrational narratives. Our ablation study shows that increasing the number of sampled candidates improves performance. With higher sampling, the model achieves better results, yielding an average improvement of +3.4.\n\n| Method| Attribute | Expression | Emotion  | Average  |\n| -| - | - | - | - |\n| Baseline (n=1)| 81.3| 40.1| 69.3| 63.6|\n| + Emotion Reasoning (n=2) | 82.9| 42.3| 70.1| 65.1|\n| + Emotion Reasoning (n=4) | 84.2| 43.2| 71.2| 66.2|\n| + Emotion Reasoning (n=8) | **84.5**  | **43.8**   | **72.9** | **67.0** |\n\n\n\n**A5: Comparison with SOTA Expression/Emotion Analysis Models**\n\n(1) The three models mentioned (Emo-LLaMA, Exp-CLIP, and ExpLLM) are not taken into consideration for direct comparison in our setting due to the following reasons:\n\n- Emo-LLaMA is not open-sourced, making it impossible to reproduce results or conduct fair comparisons.\n- Exp-CLIP is designed for image/video-text retrieval tasks, and is therefore not directly comparable to our setup.\n- ExpLLM is a still-image-based model. Its public release was in April 2025, just one month before the NeurIPS deadline, violating the NeurIPS guideline that excludes concurrent work not published prior to submission.\n\n(2) As suggested, we directly compared VidEmo with existing emotion-specific models on the same benchmarks (MAFW and DFEW) using their reported numbers. With the affective tree reasoning, VidEmo achieves a relative improvement of +4.7 UAR / +7.2 WAR on DFEW and +5.9 UAR / +12.8 WAR on MAFW.\n\n| Model|DFEW UAR (%)|DFEW WAR (%)| MAFW UAR (%) | MAFW WAR (%) |\n| - | - | - | - | - |\n| EMO-LLAMA|60.23| 65.89| 41.57| 48.63|\n| **VidEmo (Ours)** |**64.92**| **73.10**| **44.02**| **54.86**|\n\n\n\n**A6: Significance and Limitation**\n\n(1) **Scientific Value and Significance**: Emotion understanding is a long-standing and impactful research direction across disciplines, including computer science, neuroscience, and psychology. In recent years, there has been an increasing trend of emotion-related topics in high-impact publications such as ***Nature***, ***Science***, and ***Cell***, with 21, 35, and 9 papers published in the last three years, respectively. Our work contributes to this broader scientific effort by modeling video emotion in a cognitively inspired and interpretable manner.\n\n(2) **Emerging Research Trend**: In the deep learning area, recent LLM-enhanced emotion works published in top-tier conferences such as ICML 2025 oral [r6], ICLR 2024 oral [r7], and NeurIPS 2023 spotlight [r8] demonstrate the field’s growing importance and wide recognition. Our work follows this emerging trend and focuses on understanding emotion in a responsible, transparent, and reproducible manner.\n\n(3) **Responsible Usage and Licensing**. All data used in our work is sourced from publicly available, open-source datasets. No private or sensitive data is involved, and we ensure compliance with the original licenses of these datasets. Additionally, we are committed to research-only usage. All models, datasets, and code will be released under the MIT license, with explicit restrictions stated for non-commercial and non-surveillance use.\n\n\n\n[r1] Mazeika, Mantas, et al. \"How would the viewer feel? Estimating wellbeing from video scenarios.\" NeurIPS, 2022.\n\n[r2] Guo, Yuxiang, et al. \"Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models.\" IJCV, 2025.\n\n[r3] Cui, Ganqu, et al. \"The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\" arXiv:2505.22617, 2025.\n\n[r4] Zhao, Sicheng, et al. \"Emotion recognition from multiple modalities: Fundamentals and methodologies.\" SPM, 38(6): 59-73, 2021.\n\n[r5] Grill-Spector, Kalanit and Malach, Rafael. \"The human visual cortex.\" Annual Review of Neuroscience, 27:649–677, 2004\n\n[r6] Lian, Zheng, et al. \"AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models.\" ICML, 2025.\n\n[r7] Huang, Jen-tse, et al. \"On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs.\" ICLR, 2024.\n\n[r8] Chandra, Kartik, et al. \"Inferring the Future by Imagining the Past.\" NeurIPS, 2023."}, {"review_id": "JBqOcxOiiM", "reviewer": "Reviewer_JqQW", "summary": "This paper introduces VidEmo, a family of emotion-centric video foundation models trained with a novel two-stage training pipeline: Curriculum Emotion Learning and Affective-Tree Reinforcement Learning. It also presents a new large-scale benchmark dataset, Emo-CFG, containing 2.1M annotated samples with fine-grained emotional reasoning, captions, and rationales. The approach outperforms state-of-the-art models on 15 facial perception and emotion tasks, including closed and open-source baselines.", "questions": "Section 4.2 is not very clear about how the affective tree is parsed from the generated captions. Could you elaborate more on this aspect?", "limitations": "Limitations are appropriately acknowledged. I suggested one more is added in the \"Weaknesses\" section of my review.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The affective-tree structure elegantly decomposes emotional understanding into interpretable stages.\n- The post-hoc reinforcement learning (via GRPO) is a very sensible option that proves to bring additional gains.\n- Emo-CFG is a strong contribution: well-annotated and validated using committee-based LLM feedback.\n- The system is designed to provide high interpretability, increasing transparency in a typically opaque task.\n- The experiments show superior performance over 18 strong baselines.\n\nWeaknesses:\n- As acknowledged by the authors, their model uses only on visual data. The integration of audio or textual context (e.g., speech, subtitles), which is left as future work, would certainly bring better and more robust predictions, as emotional expression often depends on these.\n- An ablation study proving the improvements of the curriculum learning (vs. regular random training) and only post-training without curriculum learning is missing. It would be great if the authors could add this study to the appendices.\n- Although the committee-based verification helps, reliance on LLM-generated labels and validation could encode biases from those models. The authors should add this limitation/risk to the appropriate section.", "was_revised": true, "final_justification": "The Authors addressed all my questions adequately. I particularly appreciated that they have decided to include the suggested ablation study, which confirms the effectiveness of the proposed curriculum learning.\n\nI have also read the remaining reviews and the Authors' responses to them. I believe they addressed the vast majority of the concerns of Reviewer ntby, who recommended rejection, with valid arguments.\n\nThus, I decided to keep my score (5), recommending the acceptance of this manuscript.", "rebuttal": "# Response to Reviewer JqQW\n\nThank you for your insightful comments and questions. For your reference, we summarized the main results in our response to Reviewer KQSY.\n\n**A1: Multimodal Support** We agree that multimodal support is essential for advancing emotion understanding and plays a key role in future research. Our work lays a strong foundation for future multimodal integration from two key perspectives. (1) **Pretraining Data**: VidEmo scales the pretraining dataset from 60K (e.g., VCE) to 2.1M multimodal emotion-centric samples, many of which include paired text and are suitable for audio-text expansion. (2) **Model Transferability**: VidEmo’s structured affective reasoning and distilled emotion logits can be readily used to supervise downstream multimodal models, thereby enhancing their emotional grounding and reasoning capabilities. To support follow-up research, we will release the model, dataset, and code upon acceptance.\n\n\n\n**A2: Ablation Studies** As suggested, we conducted an ablation study to verify the effectiveness of the proposed curriculum learning strategy.\n\n(1) Curriculum learning vs. Regular Training during the SFT stage: We compare our curriculum learning strategy with conventional random training during the supervised fine-tuning stage. As shown below, curriculum learning yields substantial gains across all tasks, improving the overall average by +10.5.\n\n| Method              | Attribute | Expression | Emotion  | Average  |\n| ------------------- | --------- | ---------- | -------- | -------- |\n| Regular Training    | 63.5      | 27.3       | 63.6     | 51.4     |\n| Curriculum learning | **79.5**  | **38.7**   | **67.5** | **61.9** |\n\n(2) Post-training with vs. without Curriculum Learning in the RL Stage: To further evaluate its impact, we compare post-training (via GRPO) initialized with and without curriculum learning. The results show that curriculum-informed initialization significantly boosts emotion understanding performance, leading to a +12.4 gain in overall average.\n\n| Method                                 | Attribute | Expression | Emotion  | Average  |\n| -------------------------------------- | --------- | ---------- | -------- | -------- |\n| Post-training w.o. Curriculum learning | 75.7      | 33.8       | 44.2     | 51.2     |\n| Post-training w. Curriculum learning   | **81.3**  | **40.1**   | **69.3** | **63.6** |\n\nFor more detailed results, please refer to Tables 3–6 in our supplementary materials.\n\n\n\n**A3: Limitation induced by LLM bias** As suggested, we investigated potential biases and discussed the findings. This discussion will be included in Section 3 as part of our limitations analysis.\n\n**(1) Human Label Verification**: As suggested, we further validated label fidelity by extracting fine-grained captions from the generated videos and comparing them with the original human-assigned labels of CelebV-Text. The consistency rate reached 90.3%.\n\n**(2) User Study Verification**: To assess the quality of the generated expressions, we conducted a user study on a manually inspected subset of test samples to verify their alignment with the intended emotional semantics. Specifically, we compared Emo-CFG with CelebV-Text, the largest human-labeled video emotion dataset, across three key dimensions: precision, rationality, and complementarity. Preference rates for Emo-CFG across these dimensions reached 95%, 92%, and 93%, respectively, with statistically significant differences (Wilcoxon signed-rank test, *p* < 0.01). This result demonstrates that Emo-CFG provides more precise and expressive emotional representations than existing benchmarks.\n\n| Dimension       | Win  | Tie  | Loss | # Video | # User | Prefer |\n| --------------- | ---- | ---- | ---- | ------- | ------ | ------ |\n| Precision       | 964  | 204  | 82   | 50      | 25     | 95.52% |\n| Rationality     | 1082 | 87   | 81   | 50      | 25     | 92.16% |\n| Complementarity | 1172 | 23   | 55   | 50      | 25     | 93.04% |\n\n\n\n**A4: Affective Tree Parsing** We appreciate the reminder. The affective tree parsing described in Section 4.2 is implemented via a hybrid pipeline combining rule-based extraction and LLM-based structure inference. First, we use a customized NER-style extractor built on the SpaCy framework to identify aspect–item pairs from the generated captions, covering three semantic levels: low-level attributes (e.g., “closed eyes”, “shoulder-length hair”), mid-level expressions (e.g., “frowning”, “gasping”), and high-level emotions (e.g., “fear”, “sadness”). Next, we use a few-shot prompted LLM (InternLM-XComposer 2.5) to construct the hierarchical reasoning structure from the generated caption. The model outputs a structured JSON-like affective tree that connects attribute → expression → emotion through rationale-informed links. This structured output forms the predicted tree $T{\\text{pred}}$, which is compared against the ground truth $T{\\text{gt}}$ for reward computation. An example of the parsed affective tree can be found in Fig. 6c of the manuscript."}]

Author rebuttal:

Meta-review: This paper proposes VidEmo, a family of video foundation models for emotion understanding, combining curriculum emotion learning and affective-tree reinforcement learning, along with a new large-scale dataset (Emo-CFG). The contributions are timely and significant, offering both methodological novelty and a valuable resource for the community.

Strengths include the interpretable reasoning framework, the scale and validation of Emo-CFG, and comprehensive experiments with cross-dataset evaluations. Weaknesses are mainly around clarity; several important details were clarified only in rebuttal, and limited discussion of safeguards for ethical risks. 

Reviewer consensus is positive, with three clear accept scores and one borderline reviewer raising their score after rebuttal. I recommend Accept, contingent on the authors incorporating rebuttal clarifications into the final version.
Decision: Accept (poster)