Human Behavior Atlas: Benchmarking Unified Psychological And Social Behavior Understanding
Authors: Keane Ong
Wei Dai
Carol Li
Dewei Feng
Hengzhi Li
Jingyao Wu
Jiaee Cheong
Rui Mao
Gianmarco Mengaldo
Erik Cambria
Paul Pu Liang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Learning
Unified Models
Benchmarking
Transfer Learning
Human Behavior
Primary area: datasets and benchmarks
TL;DR: Benchmark of psychological and social behavioral datasets for developing a multimodal unified model.

Abstract: Using intelligent systems to perceive psychological and social behaviors, that is, the underlying affective, cognitive, and pathological states that are manifested through observable behaviors and social interactions, remains a challenge due to their complex, multifaceted, and personalized nature. Existing work tackling these dimensions through specialized datasets and single-task systems often miss opportunities for scalability, cross-task transfer, and broader generalization. To address this gap, we curate Human Behavior Atlas, a unified benchmark of diverse behavioral tasks designed to support the development of foundation models for understanding psychological and social behaviors. Human Behavior Atlas comprises over 100,000 samples spanning text, audio, and visual modalities, covering tasks on *affective states*, *cognitive states*, *pathologies*, and *social processes*. Our unification efforts can reduce redundancy and cost, enable training to scale efficiently across tasks, and enhance generalization of behavioral features across domains. On Human Behavior Atlas, we train three models: Omnisapiens-7B SFT, Omnisapiens-7B BAM, and Omnisapiens-7B RL. We show that training on Human Behavior Atlas enables models to consistently outperform existing multimodal LLMs across diverse behavioral tasks. Pretraining on Human Behavior Atlas also improves transfer to novel behavioral datasets; with the targeted use of behavioral descriptors yielding meaningful performance gains. The benchmark, models, and codes can be found at: https://github.com/MIT-MI/human_behavior_atlas.

Published as a conference paper at ICLR 2026
HUMAN BEHAVIOR ATLAS: BENCHMARKING UNIFIED PSYCHOLOGICAL AND SOCIAL BEHAVIOR UNDERSTANDING
Keane Ong!2., Wei Dails, Carol Li', Dewei Feng', Hengzhi Li Jingyao Wu', Jiaee Cheong', Rui Mao', Gianmarco Mengaldo", Erik Cambria', Paul Pu Liang' 'IIT Nattional University of Singapore 'Imperial College London HHarvard University NNayynng Technological University
ABSTRACT
Using intelligent systems perceive psychological and social behaviors, that is, the underlying affective, cognitive, and pathological states that are manifested through observable behaviors and social interactions, remains challenge due to their complex, multifaceted, and personalized nature. Existing work tackling these dimensions through specialized datasets and single-task systems often miss opportunities for scalability, cross-task transfer, and broader generalization. To address this gap, we curate HUMAN BEHAVIOR ATLAS, a unified benchmark of diverse behavioral tasks designed to support the development of foundation models for understanding psychological and social behaviors. HUMAN BEHAVIOR ATLAS comprises over 100,000 samples spanning text, audio, and visual modalities, covering tasks on affective states, cognitive states, pathologies, and social processes. Our unification efforts can reduce redundancy and cost, enable training to scale efficiently across tasks, and enhance generalization of behavioral features across domains. On HUMAN BEHAVIOR ATLAS, we train three models: OMNISAPIENS- 7B SFT, OMNIS A APIENS-7B BAM, and OMNIS APIENS-7B RI. We show that training HUMAN BEHAVIOR ATI.AS enables models to consistently outperform existing multimodal LLMs across diverse behavioral tasks. Pretraining on HUMAN BEHAVIOR ATLAS also improves transfer to novel behavioral datasets; with the targeted use of behavioral descriptors yielding meaningful performance gains. The benchmark, models, and codes can be found at: https: /ttes://qitiee http::/gttthuccoooooo WITTMIIhuumnn behavior aaaevvioorrtllll atlas
INTRODUCTION
Developing AI systems perceive and understand psychological and social behaviors, that is, the underlying affective, cognitive, and pathological states that are manifested through observable behavior and social interactions, has long been a central goal of Al (Picard [0000) Breazeal [2000). However, these behaviors are complex, multifaceted, and personalized, which makes it challenging to capture their diverse dimensions (Pantic and Rothkrantz 2003 [nkpen et al., 2019 Mathur et al.
2024 Liang et al.) 2024b). Existing work typically tackles these complexities through specialized datasets and single-tasi task systems, e.g., sentiment analysis (Cambria 2016), depression detection (Yang et al. 2025), human action recognition (Shaikh et al.) 2024). However, despite the breadth of efforts, current models remain primarily specialized for the task they were trained on, rather than capturing holistic understanding of psychological and social behaviors. This constrains reuse extension beyond their originally intended domains, leading to inefficiency and costly duplication of effort. Each task typically demands bespoke architectures, fresh data collection, and separate training pipelines (Gao et al. 2024). Additionally, focusing on niche task-specific solutions misses opportunities for scalability, cross-task transfer, and broader generalization (Chen al., 2024a Dai et al. 2025).
Yet, training a foundation model that can process the full spectrum of psychological and social behaviors remains major challenge (Amin et al. 2023). Despite the volume of available datasets, the single-task specialization has led to highly disparate formats and evaluation protocols. On the one hand, datasets vary in their input representations. Some rely on pre-extracted features (Zadeh et al. 2016 RRineeaal et al., 2019 |iang et al. 2024a), while others operate on raw
longstanding emphasis
video and audio clips (Cao al. 2014
Livingstone and Russo 2018 Zadeh et al., 2018). On the
Equal contribution

---

Published as a conference paper at ICLR 2026
other hand, these approaches diverge in their output formats and evaluation practices, with subjective labeling schemes (Susanto et al., 2020) and inconsistent benchmarking conventions (Kollias 2022), hindering consistent evaluation across tasks. As result, the community has seen limited success in training foundation models for psychological and social behavior analysis.
To address these gaps, we build HUMAN BEHAVIOR ATLAS, benchmark support the development of foundation models for general psychological and social behavior understanding. HUMAN BELTAV- IOR ATLAS captures diverse range of datasets and tasks under a broad laxonomy of psychological and social behaviors (affective states, cognitive states, pathology, social processes), while standardiz- ing them into consistent format for training a foundation model. To achieve this, all datasets are converted into common schema by reformulating each sample into a prompt-target format, where prompts consistently reference or contain the available modalities (e.g., transcripts, audio, video) and targets are explicitly standardized into either free-text responses or discrete label sets. In addi- tion, evaluation metrics are standardized across datasets to ensure comparability while maintaining task-specific nuance. We further augment the benchmark with behavioral descriptors extracted from MediaPipe (Lugaresi et al., (219)) and OpenSMILE (Eyben et al. 2010), providing additional signals beyond raw video and audio inputs. The result is diverse large- seagesccal multimodal benchmark of psychological and social behavior data - 101,96 ,964 unified and standardized samples spanning 35046 videos, 10287 audio clips, 25385 transcripts, augmented with behavioral descriptors.
Leveraging HUMAN BEHAVIOR ATLAS, we train OMNISAPIENS-77 SFT, which applies supervised fine-tuning: OMNIS BAM, which integrates behavioral descriptors into the SFT model through a novel residual-style Behavioral Adapter Module (BAM); and OMNISAPIENS 7B RL, which applies GRPO-style reinforcement learning (Shao et al.. 2024). Our results show that (1) training on the diverse behavioral datasets in HUMAN BEHAVIOR ATLAS enables the three OMNISAPIENT 7B variants to consistently outperform existing multimodal LLMs across various behavioral tasks, with each variant demonstrating distinct advantages and trade-offs; (2) pretraining on HUMAN BEHAVIOR ATLAS can meaningfully improve transfer learning to novel behavioral datasets and tasks; (3) while single foundation model can be broadly effective, behavioral descriptors can be compatibly integrated into existing LLM architectures through BAM, enhancing performance on targeted behavioral tasks.
Research Contributions. 1. We develop and release HUMAN BEHAVIOR ATLAS, one of the first large-scale multimodal benchmarks support the development of foundation models for general understanding of psychological and social behaviors. 2. To support future work, we conduct experiments and analyses with three variants of OMNISAPIENS-7E SFT, BAM, and RL. 3. In developing this benehmark, we establish foundational practices for constructing human behavior atlases, from general domains like psychological and social to specialized areas such as autism. These include creating taxonomy of behavioral phenomena, standardizing datasets, defining unified evaluation metrics, and assessing models on the benchmark to guide future research. Together, these practices provide foundation for building large-scale, multimodal resources that support foundation behavioral models.
2 RELATED WORK
Psychological and social behavioral analysis is a field that develops systems to model human behaviors across various dimensions (Picard 2000 Liu et al. 22024). In affective states, methods include facial action unit detection (Ekman and Rosenberg 1997). valence-arousal modeling (Valenza al. 2011), and explainable emotion detection (Cambria et 2024). Work on cognitive states has examined mental attention (Kaushik et al. 2022), cognitive workload (Gerjets et al., 2014). and communicative intent (Li et al.) 2023ap. pathology, efforts include depression (aalstar et al. 2017). anxiety (Zhang et al. 2025), suicidal idcation detection (ji et al. (022). and PTSD detection (Sawadogo et al.) 2024). Finally, research on social processes has addressed humor (Haaan et al.] 2019), ner-verbal communication (lii et al.) 2025a), and sarcasm detection (Castro et al. 2019). Yet, despite the progress, there have been limited efforts to develop a unified system capable of understanding the aforementioned dimensions of affective states, cognitive states, pathology, and social processes in a unified manner.
Multimodal multitask pretraining involves training on multiple modalities and tasks to build general general-purpose purpose models (Liang et al., (022c)). PaLI (Chen et al. 2022) shows that scaling across languages and multimodal tasks yields strong generalization, BL.I (Li et al. 2022 2023b) illustrates
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with other large behavioral datasets, Raw Raw videos, audios, text: BD Behavioral Descriptors; Aud Audio: Vis Vision: Txt Text. All other acronyms for dimensions and tasks follow Sec. [. HUMAN BEHAVIOR ATLAS provides authentic human recordings that can often reflect the richness and variability of real-world human behavior more faithfully (Reddy ct al. 2023
Beuchmark
Samples
Purpose
Format
Modalities
Dimeensios
Bchavioral Tacks
Rone
SAR
ANX
DBP
PTSD
32
Convrrsaional
Lce et 2025
Emotion Recogni
Motiooss
27% Real
Shoct-ideo ema- bon analysis
K
Walet all 2055
MMHU
57% Real
Human driving understunding
G et al
X0256
TMD SUPERB
85% Synthetic
Spetch emotion recognition
Shao ct al. 2024
HumanDuuni
2400k Syathete
Human semne destannling
Zlan al
2055
Human Behuvior Alas
10lk Real
Prychological social behav- understanding
(Ours)
how lightweight multimodal modules in pretraining can enhance understanding and generalization, and Kosmos (Huang et al. 2023 Peng et al. [023) highlights the value of multimodal instruction fol- lowing across diverse tasks for reasoning. Together, they demonstrate that large-scale pretraining is promising approach for robust and generalizable systems. However, the capabilities and limitations of pretraining in the domain of psychological and social behaviors remain largely underexplored.
HUMAN BEHAVIOR ATLAS
To curate HUMAN BEHAVIOR ATLAS, we begin by defining a broad behavioral taxonomy, before collecting 13 publicly available multimodal datasets aligned with the taxonomy's dimensions. To facilitate the training of foundation model, we reformulate all samples in standardized prompt- target format, while introducing a unified evaluation framework ensure consistent performance comparability across datasets. Finally, we extract behavioral descriptors to enrich the benchmark. We summarize HUMAN BEHAVIOR ATLAS, its curation process and statistics with Fig.
3.1 BEHAVIORAL TAXONOMY AND DATASET COLLECTION
Psychological and So- cial Behavior Taxonomy. To capture behaviors that express internal psycho- logical states and encom- pass external social pro- cesses, we define four overarching behavioral di- mensions. (i) Affective states (Aff) encompass emotions and moods
Table 2: Datasets in HUMAN BEHAVIOR ATLAS. CLS = Classification (evaluated by direct label matching). TXTR Text- Rest-Res (evaluated by LLM judge).
Dataset
Dimension
Taskis)
Task Type Modalities Samples Eval Metric
CMU-MOSEI
Aff; Cog
EMO, SEN
CLS
T/AIV
31,454
Binary weighted Fl
(SEN), Meza weighted (FMO)
Soc: Cog
SEO.
CLS
TIAIY
27,412
Binzzy weighted
(SENS, Moan weighte
acc. (EMO)
TESS
Aft: Cog
CLS
T/AI-
2800
Mean weighted accuracy
CREMA-D
Alf
CLS
T/AI-
7.442
Mei weighted accuracy
CH-SIMSV2
Aff
SEN
CLS
T/A/V
4.483
Binary weighted F1
Social-1Q
Cog
TXTR
T/A/V
Accuncy (1iM-uulge)
IntenrQA
Soe: Cog
INT
TXTR
T/ATV
16,297
Ancuacy (1. (L..M-udge))
MimeQA
Sox:
NVC
TXTR
T/A/V
806
Accuncy (LIM-Julge)
ie.. short-term feelings (anger, joy) to longer- lasting sentiments. (ii)
UR-FUNNYV22
HUM
C1S
T/A/V
2.125
Weighted
MUSIARD
Soc
SAR
CLS
T/A/V
690
Weighed Fl
DARC-WOZ
Weighted
MMPs
Puth
DEP, ANX
ANX
CLS
T/-/-
1,275
Weighted F1
Cognitive states (Cog) iho--in-
Puth
PTSD
CLS
T/A/V
634
Weighted Fl
are internal mental processes ie.. attention, reasoning, surprise, decision decision-making: which are
often inferred through external cues. (iii) Pathology (Puth) covers psychological and psychiatric conditions i.e., depression, anxiety; where internal states are assessed via verbal or nonverbal indi- cators. Finally, (iv) social processes (Soc) describe social interaction and communicative behaviors i.c., humour, intent, and cooperation.
The defined behavioral dimensions map onto established tasks in the literature, providing the basis for collecting diverse set of datasets. In practice, task may align with multiple dimensions e.g., emotion recognition can involve understanding affect (sadness) and cognition (surprise). Our collated tasks include sentiment polarity (SEN): classifying attitudes as
Collecting Tasks and Datasets.
3

---

Published as a conference paper at ICLR 2026
(a) HUMAN BEHAVIOR ATLAS
Taxonomy
Annotation Standardization
Downstream task
(1 Affoctive
(D Cognitive
PIIQ-- Depression Depression
State
Feature Extraction
Modaa Fraiiing
Uniifid Metrics
OpenSMILE
Haggy
Working
Emnttin
Humor
Anxiety
(10) Pathology
"Haa asyanne
Social
Facial Landmarks Acoustic Features
rrund you nuticed you moniing or speaking Sinner than usual?"
Social judgement
Pase Keypoints
Transriptions
PTSD
Intention
Social
T
13 3..2%
T+A
3.26%
5-20s
29.2%
<5
52.9%
MELO-
T+A+V
S-10s
11.5%
83.6%
TAN
(c) Dist. of Data Modalities
(d) Dist. of Sample Durations
AWS
Souree Datercts
(b) Dataset Distribution by Task
(e) Source of Datasets
Figure Overview of HUMAN BEHAVIOR ATLAS. (a) Selection criteria and preprocessing pipeline datasets. (b) Dataset distribution across 10 behavior related tasks. Inner circle indicates the modality combination of the input data, where T=Text, A=Audio and V=Video, Middle ring describes the tasks of the dataset, as defined in Sec. 3.1 The outer ring and bars lists the datasets and its sample sizes respectively. (c) Distribution of data modalities. Our dataset has focus video understanding as comprises both vision and audio modalities, with 83.6% samples containing video data. (d) Distribution sample durations, Both short and long videos/audio tasks are covered, with 29.2% of video/audio clips lasting more than 20 seconds. (e) Source of datasets. Datasets are sourced from diverse geographic regions across North America, Europe and Asia.
positive, negative, neutral; emotion recognition (EMO): identifying emotions (anger, joy, sadness); social reasoning (SOC): understanding socially grounded judgments like empathy or appropriateness: intent recognition (INT): identifying the underlying purpose behind behavior; and non-verbal communication (NVC): interpreting gestures and facial expressions, They also cover humor detection (HUM), sareasm detection (SAR), anxiety defection (ANX), depression detection (DEP), and PTSD detection (PTSD). Table summarizes all datasets, tasks, and their behavioral dimensions. App. describes the datasets and our rationale for their behavioral dimension categorizations.
3.2 DATASET SCHEMA UNIFICATION
Standardizing Samples for Foundation Model Training. The collected behavioral datasets in
Table 3: Example of converting a MELD sample into unified prompt-target target format. More examples are in App. A Table 22
samples that involve varying formats, often requiring task-specific input processing and output heads. This makes it difficult to build foundation model which requires a standardized training interface. To resolve this heterogeneity, every benchmark sample is reformulated into standardized
Datsset
Owiginal pats / Outputs
In-
Reformulated Prompt
Target
Tisk
MELD
Vileo Audio Trancript (pew-
cridwcs <audio>
Discrete
Iabel
(SEN)
(positive,
da. Thatrs Monica wey missing, sen Likt, "Wacre"s nandwe 272
negarive,
ncutial)
The above is video and sodio recording from a conversation, along with the Lranseeipt ipt.. What 15 the sentiment at the speaker In this zocording? Choose from the fill lowinng went trens labelss Meatral.
sponse: Positive
Ansvent
Regell Liw..oww
Neutral
Focstivo,
ttompt-target format (Table 3). designed to align with recent multimodal LLMs, that currently underpin foundation model development, e.g., Qwen2.5-Ommi (Xu et al. 2025a). We handeraft and
4

---

Published as a conference paper at ICLR 2026
create new prompts that explicitly reference the available modalities and includes textual transcripts where available, while the target is standardized as free-text answers or a fixed discrete label set. To elaborate on standardizing the target format, continuous outputs (e.g., PHQ-9 scores in MMPsy) are discretized into categories based on their original paper guidelines, whereas categorical outputs (e.g., sentiment in MELD) retain their original form. Through this step, all outputs are cast into discrete categories, which can then be represented as free-text answers or elements of discrete label set. Conversely, tasks requiring open-ended inference via text generation (e.g., MimeQA) have their targets preserved as free-text responses. Depending on the model design, HUMAN BEHAVIOR ATLAS provides the choice of representing the target as free-text or as a discrete label set. For designs that rely solely on decoder head, the target is provided in free-text form. Alternatively, for designs that incorporate classifier heads, the target instead provided as a discrete label set. This enhances versatility and enables a broader range of models to be developed, acknowledging that classifier heads may offer more stability for classification tasks (Edwards and Mamacho-Colllddo 2024).
3.3 STANDARDIZED EVALUATION FRAMEWORK
Unitying Evaluation Metrics Across Datasets. Integrating diverse datasets is further complicated by how dataset evaluation metries differ, even when tasks overlap (i.e., emotion recognition CMU- MOSEI (Zadeh et al., 2018) uses weighted accuracy, while MELD (Poria et al. 2019) uses weighted F1). To enable explicit performance comparisons, we introduce standardized evaluation framework (Table 2
We assign each task single standardized metric and apply it uniformly across all datasets associated with that task. For sentiment polarity (SEN), we use binary weighted F1, computed over positive and negative labels, since sentiment datasets often differ in class schemes (c.g., three- or five-point scales, inclusion or exclusion of neutral (Poria et al. 2019 Yu et al., 2022)), ). and prior work highlights positive and negative sentiment as the most informative and foundational (Socher eet al. (213). For emotion recognition (EMO), where datasets and emotion categorization models often contain multiple often-imbalanced emotion classes (e.g.. Ekman's 's model has 4 of negative emotion categories), we adopt the mean of per-class weighted accuracies (Zadeh et al. 20188) to ensure balanced evaluation. Another important refinement for emotion recognition involves consolidating labels that represent the same underlying emotion (c.g., joining "joy" and 'happiness' labels) while differentiating those that are intrinsically distinct (e.g., disjoining 'surprise' into 'positive surprise' and 'negative surprise"). ). For other classification tasks (HUM, SAR, ANX, DEP, PTSD), we use weighted F1, which is well suited for binary detection tasks and mitigates the effects of label imbalance in their corresponding datasets. Finally, for free-text response tasks (SOC, INT, NVC), we use LLM-judge to evaluate whether the generated text output aligns with the ground truth, reporting accuracy as the proportion of responses judged TRUE. Full details evalaution metric formulas, LLM judging prompts can be found in App.
3.4 BEHAVIORAL DESCRIPTORS EXTRACTION
Behavioral Descriptors. To augment HUMAN BEHAVIOR ATLAS, we extract behavioral descriptors that capture fine- fine-graing grained affective and social signals. These descriptors may not be explicitly present raw video and audio, yet they are often critical for understanding psychological and social behaviors, encoding rich cues about prosody, facial expressions, and body language /Zhagg ctt al. 2022 Narayanan and Georgiou| 2013). For the visual modality, we use MediaPipe (Lugaresi et al. 2019) to extract facial landmarks and body pose keypoints, capturing muscle movements, expressions, posture, and gestures relevant to emotion and nen-verbal communication. For the audio modality, we extract acoustic features with OpenSMILE (Eyben et al. 2010), using the ComParE 2016 configuration (Schuller et al.) (0166) to capture prosodic cues (pitch, energy, spectral properties, and voice quality). Finally, where ground truth transcriptions are missing, we extract text transcriptions from audio content using Whisper v3 Large model (Radford et al. (023)..
3.5 OMNISAPIENS-7 7B SET
We train a multimodal foundation model for psychological and social behaviors, OMNISAPIENS-7B SFT on HUMAN BEHAVIOR ATLAS. Full model details can be found in App.
Backbone & Output Heads. We initialize a pretrained multimodal LLM (Qwen2.5-0mni- which includes audio and image encoder, alongside an LLM. We project raw audio, visual, and
textual inputs into a shared embedding space, enabling the LLM backbone to process multimodal benchmark samples. We take the pooled hidden representations Nyeeaate the penultimate layer
5

---

Published as a conference paper at ICLR 2026
immediately before the final hidden states, input to task-specific classifier heads, which produce categorical predictions (i.e., positive/ negative sentiment) for classification tasks. For tasks requiring open-ended inference via free-text responses, a separate decoder head generates outputs directly from the final hidden states after /ppullt Consistent with the standardized prompt-target format /Sec.3..2 the classifier heads operate targets represented discrete label sets, while the decoder head leverages free-text responses. We use cross-entropy loss and teacher forcing for the classifier and decoder respectively.
3.6 OMNISAPIENS- 7B BAM
We introduce simple variant, OMNISAPIENS- 7B BAM, by equipping OMNISAPIENS--77 7B SFT with a Behavioral Adapter Module (BAM) that incorporates behavioral descriptors, allowing us to examine the performance impact of integrating the behavioral descriptors (i.e., pose, prosody, and facial landmarks) into the LLM backhone. Behavioral analysis literature has sometimes framed end-to-end models trained raw signals and reature-based approaches trained on the descriptors as mutually exclusive choices (Li and Mahmoud 2025; Luo et al., 2025). We investigate the approaches interoperate non-invasively, allowing descriptors to augment end-to-end models without compromising backbone representations, which could otherwise cause destabilized features or catastrophic forgetting (Ramasesh et al.. 2021).
Post-Preed Post-Pretraining Residual Adapter. After full training, OMNIS APTENS-1 SFT is frozen and an adapter integrating behavioral descriptors is attached; with only the adapter and the classifier/decoder heads receiving gradient updates. Formally, we describe the time-dependent descriptors:
fraw E TR: T
where T is the temporal dimension and Draw the raw feature dimensionality (i.e., concatenated OpenSmile and/or MediaPipe streams). A temporal pooling operator computes both the mean and standard deviation across timestamps in a single audio or video clip:
My Σ
[a [t].
of
1 T
(f [t] - =,)) f [m[, of d R Dieat = 22Drass
The pooled behavioral descriptors f are normalized and regularized with dropout before being passed through a small feed-forward network:
f Dropout (rrpout(Norm(F)) (Norm(f)),
2 z (W1xz++b)) by).
Output 2 produces a residual update to the penultimate hidden state Apeault (defined in Sec. 3.5). scaled by a learnable scalar o; the adapted representation hadapt is fed into respective heads:
Ah a 2f,
hadage = Appemute + Ahj-
Because the update follows a residual design preserving /ppnitt and adding learned adjustment, it supplements rather than overwrites the original hidden state. Removing BAM reduces hadap back to hyeault. ensuring the backbone remains unaffected. We utilize lightweight feed-forward network to enable computationally efficient adaptation of the multimodal LLM using behavioral features (see computational study in App. D.6). The temporal descriptors are pooled via mean and standard deviation to capture the distributional characteristics of the behavioral signals within each video or audio clip, and maintain compatibility with the static text, audio, and video modality streams that are taken in by the multimodal LLM. We integrate the adapter at the penultimate layer rather than at the early fusion stage, as the early textualization and alignment of behavioral features would require substantially more training cost (Xu et al.) 2025b), which is contrary to the intended plug-and aogg-nnd-pay play nature of the adapter design.
3.7 OMNISAPIENS- 7B RL
SANISAPIENSS 7B RL reuses the same multimodal backbone as OMNISAPIENS-77 7 SFT, but handles all tasks within a single decoder head. OMNIŠAPIENS-77H RL does not make use of any classifier heads; all outputs are produced directly from single generative decoder. This preserves the full flexibility of the LLM decoding architecture while enabling a clean comparison against the hybrid setup used in OMNISA 7B SFT. We leverage the benchmark samples that have been standardized into a unified prompt- target format (Sec. 3.2). with all targets represented in free-text form. We then optimize OMnISAPIenS 7B RL with Group Relative Policy Optimization (GRPO) (Shao et al. 2024). Details of the training process and reward structures are in App. B.2

---

Published as a conference paper at ICLR 2026
Table 4: Results grouped behavioral tasks (headers) and relevant datasets (sub-headers). Best results bolded, second best are underlined. Following the unified metrics (Sec. (33) we use binary weighted for SEN; mean per-class weighted accuracy for EMO; weighted Fl for HUM, SAR, ANX, DEP, PTSD; and LLM-Judge accuracy for SOC, INT, NVC. *MMPSY uses text-only input and excludes BAM; as the backhone is preserved, results equivalent to OMNISAPIENS-7B SFT.
Model
EMO
HUM INT PTSD ANX
DEP
SEN
SAR SOC NVC
FUNNY
WTSD_WILD
AMMPS (A)
MMPSY (D)
DAIC-WOY DAIC-WOO WOZ.
CH-SIMSV2
Social-IQ 2.0
CREMA-D
MELD (B)
MOSEI (F)
IntenQQ
MELD (S)
MOSEL(S)
MUSIARD
MineQA
TESS
Gemma-3-48
.495
.642
.565
.499
.597
.227
.499
.601
.788
.137
.785
813
.617
.529
.191
.023
HumanOmmiVV-77
.560
.633
.558
.637
.638
.263
.824
.527
.672
.636
.768
8225
.633
.395
.282
.093
Qwen 2.5-Omni-7B
.521
.661
-580
-568
.543
.254
.760
.793
.791
.636
770
.714
.656
.254
.069
Qwen- V..7V
.501
.571
.592
.499
.583
.249
.755
.631
.653
.623
.674
.524
.317
.511
.231
.098
OMNISAPIENS-77 RL
.501
.699
-881
-500
.639
.968
.919
.844
.729
571
.33
.24
.647
.304
.133
OMNISAPIENS-76 SFT
.542
.709
.614
655
.532
.256
1.00
9009
.839
.626
746
813
744
.624
.257
.121
OMNISAPIENS-7B BAM
.48
.711
.077
.715
.644
.177
1.00
909
.83'
778
744
8837
.775
.795
.201
.162
4
EXPERIMENTS
To offer insights for future research on HUMAN BELAAIIOR ATLAS, we assess the three OMNISAPIENS-7B models trained on the benchmark against existing multimodal LLMs pretrained on general multimodal and numan-related data, with the latter comparison meant to contrast HUMAN BEHAVIOR ATLAS against other large scale human-data related benchmarks (i.e. HumanOmni). Full experimental details are in App.
4.1 MULTI-TASK TRAINING
In this section, we study the general understanding of psychological and social behaviors gained through pretraining on HUMAN BEHAVIOR ATLASS. Namely, we pretrain OMNISAPIENS-77 7B SFT, OMNISAPIENS-7B BAM and OMNISAPIENS-7B RL in a multi-task setup, where a single instance of the model trained jointly across all behavioral tasks in the benchmark.
General multimodal LLMs exhibit limited capabilties across the breadth of behav- ioral tasks. Conversely, we observe that OMNISAPIENS OMNISAPIENS-7B BAM, and OPNISAPIENS-7B RL, all pretrained on behavioral tasks in HUMAN BEHAVIOR ATLAS, consistently exceed the performance of these general multimodal LLMS (Qwen
Girmma-J
48
0.55
0.65
0.23
0:10
6:67
0.:06
0.59
0.11
0.07
HantoeChnni
0.00
0.64
0.62
0.52
0.74
0.03
SFT,
Onen 2.4
0.54
0.76
0.07
0.07
0.54
0.58
0.76
0.61
0.64
0.50
023
5.32
OmsaiSopiaas -78
-7t SFT
0.63
0.52
1:00
0:92
0.75
0.77
5.52
0.12
2.5-Omni-7B (Xu et al.) 2025a).
OrnaiSapieas Ham
Qwen 2.5-VL-7B (Bai et al. 2025). )
Gemma-3-4B (Team et al., 2025
IMD
HUM
JN1
F15D
ANN
D8P
S8N
SAK
500
NVC
From Figure 2 OMNIS APIENS-7B
Figure 2: Multitask results across tasks for each model. Each result reports the average score worst dark green + yellow + red. Upon training on HUMAN BEHAVIOR ATLAS, OMNISAPIENS-7E SFT & RL outperform existing pretrained models across most behavioral tasks.
SFT and OMNISAPIENS-7B BAM each outperform the general multi- modal LLMs on of 10 behavioral tasks, while OMNISAPIENS-70 RL does so on of 10. This demonstrates that a comprehensive understanding of psychological and social behaviors is better supported by specialized adaptation across behavioral tasks. Specifically, pretraining on HUMAN BEHAVIOR ATLAS can improve the capabilities of general multimodal LLMs on diverse behavioral tasks.
all datasets for that task. Best
Existing efforts to develop multimodal LLMs trained on large-scale human-related data remain few, and tend to focus understanding narrow range of behavioral phenomena. One such example, HumanOmmi V2 primarily targets human-centrie scenes involving emotion recognition and facial description (Zhao al. 2025). While this specialized focus allows it achieve competitive

---

Published as a conference paper at ICLR 2026
scores tasks such HUM, SEN and SOC, its performance does not generalize broadly. In contrast, our variants consistently outperform HumanOmmiV2-778 across the spectrum of behavioral tasks. OMNISAPIENS-7B SFT leads on 7 of 10 tasks, while both OMNISAPIENS-77 7B BAM and RL. lead on 8 of 10. This highlights that developing a foundation model for comprehensive understanding of psychological and social behaviors requires more than unstructured exposure to general human-related content - it demands systematic and deliberate adaptation across a diverse set of tasks pertaining to these behavioral dimensions.
While a foundation model improves performance across behavioral tasks, the best-performing variant depends on task type. For structured classification tasks (EMO, HUM, PTSD, ANX, DEP, SEN, SAR), both OMNISAPIENS- strongest performers, each surpassing existing multimodal LLM baselines. The only exceptions are SAR and HUM, where OMNISAPIENS- OMNISAPIENS- 7 RL demonstrates the most consistent strong performance on open-ended text generation tasks, achieving the strongest results on INT (0.49) and SOC (0.30), and the second-best on NVC (0.13). Taken together, these results highlight a contrast between supervised fine-tuning approaches, which dominate on structured classification benchmarks, and reinforcement learning. which is better suited to tasks requiring free-form reasoning and generation. Future work can focus on exploring hybrid training strategies, integrating RL's strengths reasoning with SFT reliability on structured recognition tasks, to achieve balanced performance across the full spectrum of tasks.
BAM and OMNISAPIENS 7B SFT emerge as the
SFT shows relatively weaker performance. In contrast,
4.2 TRANSFER LEARNING TO HELDOUT DATASETS
Besides multi-task training. we study the transfer capabilities enabled by pretraining HUMAN BEHAVIIR ATLAS. We withhold four datasets during multi-task pretraining of OMNIS SFT. These held-out datasets fall under two settings: (i) novel datasets for tasks already represented in pretraining - i.e. MOSEI as a heldout sentiment dataset, with sentiment still represented in the pretraining tasks by CH-SIMSv2, and (ii) a novel dataset introducing a new behavioral task not seen in pretraining. The pretrained OMNIS APIENS-BB SFT is then fine-tuned separately on each held-out dataset under fixed, minimal epoch budget. For comparison, Qwen 2.5-Omni-7B SFT which shares the same architecture as OmniSapiens-77 SFT, differing only in its absence of pretraining on HUMAN BEHAVIOR ATLAS, also fine-tuned on these held-out datasets under the same training conditions. This enables a clean isolation of the effect of training on our benchmark, ensuring that observed improvements can be attributed to the data and tasks within Human Behavior Atlas rather than to architectural changes.
Pretraining on HUMAN BEHAVIOR ATLAS enables
Table S: Transfer to held-out datasets after minimal epoch fine-tuning (1 epoch). Bold denotes best score. DAIC-WOZ" involves 2 epochs as it has only 107 train- ing samples. MUStARD' presents a novel behavioral task of SAR (sarcasm detection). $ Other held-out datasets that have tasks represented during pretraining.
SFT to adapt more effectively when transferring to new datasets that involve overlapping behavioral tasks (setting i). From Table the pretrained OMNISAPIENS- 7B SFT consistently outper- forms Qwen 2.5-Omni-78 SFT on held-out datasets MOSEI (SEN) +18.3% (+0.112), MELDD (EMO) +3.95% (+0.027), and DAIC- WOZ (DEP) +29.4% (+0.17). This highlights that pretraining diverse behavioral tasks in HUMAN BEHAVIOR ATLAS can yield transfer benefits to unseen datasets, provided that these datasets comprise tasks represented in pretraining. Therefore, pretraining on HUMAN BEHAVIOR ATLAS may hold promise for transfer to real-world applications where the underlying behavioral task remains consistent but the datasets domains vary - i.e sentiment analysis across product reviews or news settings, although real-world validation could offer further insight on this potential.
Datuset
OMNIS 7B SPT Qwen 2.5.0mni-7 7D SFT
MOSEI (SEN)
0.724
MELD' (UMO)
0.711
0.684
DAIC-WOZ" (DEP)
0.749
0.579
MUS:ARD (SAR)
0.658
0.473
Pretraining on HUMAN BEHAVIOR ATLAS enables OMNISAPIENS-PT 7B SFT to adapt more effectively when transferring to a new dataset that involves a novel behavioral task (setting ii). From Table 5 OMNISAPIENS-77 SFT outperforms Qwen 2.5-Omni-7B on MUStARD, a dataset for sarcasm detection (SAR), by 39.1%% % (+0.185), even though SAR was excluded from pretraining tasks. This suggests that pretraining may not only benefit transfer to tasks already represented in pretraining, but also provides a foundation for adapting to new behavioral tasks or phenomena. Thus, pretraining HUMAN BEHAVIOR ATLAS can hold potential for transfer real-world settings where
8

---

Published as a conference paper at ICLR 2026
new psychological social behavioral phenomena emerge, which are often understudied and have minimal available data i.e. evolving patterns online interaction (Oksanen et al. 2024). Yet, we qualify that the complexity of these behaviors means transfer performance may likely vary across behavioral phenomena. Therefore, understanding which novel behavioral tasks remain challenging to transfer to is research direction that future work on HUMAN BEHAVIOR ATLAS can explore.
Beyond the transfer improvements themselves, the comparison of OmniSapiens-79 SFT against the single-task trained Qwen 2.5-0mni-77 SFT, as shown in Table clarifies how the diversity of tasks in Human Behavior Atlas shapes the learned features while training on the benehmark. On represented tasks (SEN, EMO, DEP). OmniSapiens-7B SFT outperforms the single-task pretrained Qwen model even after minimal fine-tuning, indicating that exposure to multiple behavioral tasks enables the model to internalize shared behavioral features across heterogeneous tasks that generalize across datasets. Meanwhile, the fact that OmniSapiens-7B SFT also surpasses Qwen 2.5-Omni-7B on MUStARD, despite SAR being absent during pretraining, suggests that multi-task behavioral pretraining provides broader, more general behavioral patterns that may occasionally extend to novel behavioral phenomena. These transfer patterns, when contrasted against the single-task pretrained baseline of Qwen 2.5-Omni-7BB, suggests that the diverse and heterogeneous behavioral tasks in Human Behavior Atlas contribute complementary and synergistic behavioral features, rather than conflicting contradictory supervision.
Qualitative analysis suggests that pretraining on behavioral tasks in HUMAN BEHAVIOR ATLAS yields stronger pragmatic recognition capabilities. An interesting pattern observed in the held-out task of SAR that OMNISAPIENS: 7B SFT shows a greater ability to recognize pragmatic cues (i.e. contextual signals beyond literal word meaning) after extensive pretraining on behavioral datasets in HUMAN BEHAVIOR ATLAS. An example (Figure 3 from MUS- tARD, the heldout sarcasm detection (SAR) dataset, illustrates this. While OMNISAPIENS
SET
Cuiten Denoon y?
In Io rebody
May Tume put ROTH
Figure 3: Example from MUStARD where the speaker (Chandler, from Friends) sarcastically suggests putting up balcony lights. While Qwen2.5-Ommi-77 predicts no sarcasm, OMNISAPIENS-77 SFT correctly identifies the instance as sarcasm.
7B SFT correctly interprets Chandler's 's remark about putting up balcony lights as sarcasm rather than serious suggestion, Owen2.5-0mmii-77 without the same behavioral pretraining, defaults to a prediction of "no sarcasm" (a literal, face-value interpretation of Chandler's remark), even after full epoch of fine-tuning. This case is not isolated: across testing, Qwen2.5-0mmii778 tends to default to "no sarcasm" predictions for 93. 93.2% of samples, resulting in lower weighted recall of SFT achieves a higher weighted recall of 0.670 by more reliably identifying sarcastic instances, assigning the "sarcasm" label to 30.1 1..1 % of samples. These results suggest that pretraining on HUMAN BEHAVIOR ATLAS can improve the general capacity to capture the subtle, context-dependent nuances of psychological and social behaviors. Future research could explore whether this capacity is emergent and scales with model size, potentially enabling more reliable detection of behaviors that go beyond surface-level or literal cues.
0.534. contrast,
4.3 EFFECT OF BEHAVIORAL DESCRIPTORS
Table 6: A highlights the change in performance from OMNISAPIENS-78 SFT OMNIŠAPIENS-7I BAM. shown as percentage (%) and absolute (Abs). BAM provides notable performance gains for a considerable number of behavioral tasks, although its benefits are not consistent across all tasks.
Behavioral descriptors provide target- ted performance gains for specific behav- ioral tasks. We integrate behavioral descrip- tors through OMNISAPIENS-77 BAM, an ex- tension of OMNISAPIENS plies post-training residual updates through Behavioral Adapter Module (BAM) (Sec. (36). while leaving the backbone representations unchanged. From Table ( OMNISAPIENS-
SFT, that ap-
Task OWNISAPENS-778
A % (Abs)
NVC
0.12
0.16
-33,00 (+0.4)
SAR
0.62
+99.00 (10.18)
HUM
0.53
0.64
+21.00
DEP
0.73
0.79
48.21 (10.66)
7B BAM shows performance improvements
EMO
0.63
0.65
+3.17 (+0.02)
over OMNISAPIENS 7B SFT on a consider- able number of tasks NVC (+33,00%), SAR (429.00%), HUM (+21.00%). DEP (+8.21%) 1)), EMO (+3.17%) and SEN (+2.60%), but the
SEN
0.77
0.79
+2.60
PTSD
1.00
0.00 (40.00)
ANX
0.91
0.9
0.00 (+0.00)
SOC
0.26
0.20
-3.08 (-0.66)
INT
0.26
0.18
-5077 (40.8)

---

Published as a conference paper at ICLR 2026
same gains do not hold for PTSD, ANX, SOC and INT. Yet, because BAM is implemented in a modular, residual fashion, it can be flexibly applied where useful and omitted where not, without altering the backbone representation of the model. In this fashion, behavioral descriptors can be selectively applied to improve the performance of OMNIS APIENS-E SFT, or foundation models more broadly, on targeted tasks. Therefore, while existing literature may often frame ceaturebbased approaches and end-to-end models as mutually exclusive choices, our results highlight that descriptors can still be useful within end-to-end models - perhaps not as a blanket integration into the shared backbone, since not all behavioral tasks benefit equally, but as a means of targeted adaptation for tasks that benefit most.
Qualitative analysis suggests that behavioral descriptors can help improve foundation models through the detection of subtle be- havioral cues. In our experiments, we observe cases where subtle cues, such as fleeting smile in CH-SIMSv2 (Figure D. go undetected by OMNISAPIENS- SFT, which relies solely raw audio-visual signals, leading to an incor- rect prediction of negative sentiment. However, when behavioral descriptors are incorporated in OPIISAPIENS-77 BAM, the subtle smile ap- pears to register, with the prediction changing to positive. This suggests that subtle behavioral cues may be more effectively registered with the additional behavioral descriptors than with raw audio and video alone. A potential hypothesis is that coarser raw video and audio input streams may overshadow fine-grained behavioral expressions, while descriptors such as MediaPipe facial keypoints more explicit representations of these expressions. Yet, while we observe instances consistent with this hypothesis, further research can systematically pinpoint the data distributions and context where behavioral descriptors provide the most benefit for foundation models, thereby guiding their effective integration into such models.
Figure 4: Example from the CH-SIMSv2 dataset where the speaker briefly displays a split-second smile, signal- ing positive sentiment. While OMNISAPIENS-7D SFT misses the subtle cue and predicts negative sentiment, OMNISAPIENS-PT BB BAM correctly predicts positive.
5 CONCLUSION
In this work, we introduce HUMAN BELAAIIOR ATLAS and evaluated three model variants, OMNISAPIENS-7B SFT, OMNIS APIENS-7B BAM, OMNISAPIENS- RL, alongside other multi- modal LLMs. Our experiments demonstrate that training on this benchmark produces significant gains in both multi-task and transfer learning scenarios. Moreover, incorporating behavioral descriptors through BAM enables targeted improvements for specific domains, highlighting the importance of structured behavioral information. At the same time, our findings expose new research questions critical for advancing foundation models capable of comprehensively understanding psychological and social behaviors. To support the research community, we release HUMAN BEHAVIOR ATLAS, and its associated OMNIS 7B models, aiming to foster investigation into these challenges and contribute to the development of foundation models for psychological and social behavior analysis.
In constructing this benchmark, we propose set of foundational practices for developing human behavior atlases, designed to span the full spectrum of behavioral domains from broad areas such as psychological and social behaviors to specialized domains like autism. These practices establish a methodological framework that includes: (i) deriving a taxonomy that organizes diverse behavioral phenomena into a coherent structure, (ii) standardizing heterogeneous datasets into consistent format that supports the training of foundation models, (iii) defining robust and comparable evaluation metrics, (iv) refining label spaces by consolidating categories that capture the same underlying concept (e.g., merging semantically overlapping labels) while distinguishing those that are inherently different (c.g.. splitting broad or ambiguous categories into finer- greineg, meaningful sublabels), and (v) systematically evaluating foundation models on the benchmark to surface limitations and highlight directions for future research.
Together, these practices provide the building blocks for the next generation of large-scale, multimodal resources that enable the development of foundation models with comprehensive capabilities for analyzing human behavior. By offering both methodological guidance and an empirical testbed, this benchmark lays the groundwork for advancing the scientific study and computational modeling of psychological, social, and domain-specific behaviors.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research/project is supported by the Asian Institute of Digital Finance, National University of Singapore; Ministry of Education, Singapore under its MOE Academic Research Fund Tier (MOE-T2EP20123-0005
"Neurosymbolic AI for Commonsense- Multiple Domains"), MOE Tier 1 Award (MOE-T2EP5221-- -0028: "Discipline-Inform Informedd Neural Networks for Interpretable Time-Series Discovery"), and by the RIE2025 Industry Alignment Fund- Industry Collaboration Projects (1230150026: "Generative AI"), administered by S as well supported by Alibaba Group and NTU Singapore. The authors would also like to acknowledge Nvidia for their support through the provision of GPU and compute resources.
Question Answering in
ETHICS AND REPRODUCIBILITY STATEMENT
This research focuses on curating a benchmark for training foundation models for psychological and social behavior understanding. Leveraging this benehmark, we also developed foundation model, and experimented with different methods (SFT, BAM, RL). All datasets, and model architectures utilized have been properly cited, and are publicly available. Additionally, there are no discrimination, bias, or fairness issues that should arise in this paper. Furthermore, the models trained are not expected to generate harmful content. For reproducibility, we provide all experimental details in Section 3 and the corresponding appendices. We will release the benchmark, models and codes to support reproducibility.
REFERENCES
Mostafa Amin, Erik Cambria, and Bjöm Schuller. Will affective computing emerge from foundation models and General AI? A first evaluation ChatGPT. IEEE Intelligent Systems, 38(2):15-23, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Cynthia Lynn Breazeal. Sociable machines: Expressive social exchange between humans and robots. PhD thesis, Massachusetts Institute Technology, 2000.
Erik Cambria. Affective computing and sentiment analysis. IEEE Intelligent Systems, 31(2):102-077, 2016.
Erik Cambria, Xulang Zhang, Rui Mao, Melvin Chen, and Kenneth Kwok. SenticNet 8: Fusing emotion Al interpretable, trustworthy, and explainable affective computing. In International Conference on Human- Comopttr Interaction, pages 197-216. Springer, 2024.
and commonsense
Houwei Cao, David G Cooper, Michael Keutmann, Ruben Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390,
2014.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcca, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously. perffecteeeettteeeeleee paper). In ACL, 2019.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song. Bowen Chen, Andrew Zhang. Daniel Shao, Muhammad Shaban, et al. Towards general-purpose foundation model for computational pathology. Nature medicine, 30(3):50-862, 2024a.
Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony J Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: jointly-scaled multilingual language- image model. arXiv preprint arXiv:2209.06794 2022.
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use large language models for reference-free text quality evaluation: An empirical study, arXiv preprint 2rXiv:2304.00725. 2023.
Zhawnen Chen, Tianchun Wang. Yizhou Wang, Michal Kosinski, Xiang Zhang, Yun Fu, and Sheng Li. Through the theory of mind's eye: Reading minds with multimodal video large language models. arXiv preprint arX.v:2406.1337 2024b.
Wei Dai, Peilin Chen, Malinda Lu, Daniel Li. Haowen Wei, Hejie Cui, and Paul Pu Liang. Climb: Data foundations for large scale multimodal clinical foundation models. arXiv preprint arXiv:2503.0766 2025.
Aleksandra Edwards and Jose Kamacho-Collados Language models for text classification: Is in-context learning enough? arXiv preprint 2403 102..77661. 17661, 2024.
11

---

Published as a conference paper at ICLR 2026
Paul Ekman and Erika L Rosenberg. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA, 1997.
Florian Eyhen, Martin Wällmer, and Bjöm Schuller. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459-1462, 2010.
Shanghua Gao, Teddy Koker, Owen Queen, Tom Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. Units: A unified multi task time series model. Advances in Neural Information Processing Systems, 37: 100589-140611, 2024.
Peter Gerjets, Carina Walter, Wolfgang Rosenstiel, Martin Bogdan, and Thorsten O Zander. Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach. Frontiers neuroxcience, 8:385, 2014.
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, and Jianyuan Zhong. Ur- Ur-funny; A multimodal language dataset for understanding humor. In EMNIP--JCNIPP, pages 2046-2056, 2019.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal. Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning pereeption with language models. Advances in Neural Information Processing Systems, 00272299-72110 2023.
Kori Inkpen, Stevie Chancellor, Munmun De Choudhury, Michael Veale, and Eric PS Baumer. Where is the human? bridging the gap between ai and hei. In Extended abstracts of the 2019 chi conference on human factors computing systems, pages 1-9, 2019.
Shaoxiong Xue Li, Zi Huang, and Erik Cambria. Suicidal ideation and mental disorder detection with attentive relation networks. Neural Computing and Applications, 34:10309-1033 2022.
Pallavi Kaushik, Amir Moye, Marieke van Vugt. and Partha Pratim Roy. Decoding the cognitive states of attention and distraction in a real-life setting using eeg. Scientific Reports, 12(1):20649, 2022.
Dimitrios Kollias. Abaw: Valence-arousa. estimation, expression recognition, action unit detection & multi-
task learning challenges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2328-2336, 2022.
Min-Ho Lee. Adai Shomanov, Balgyn Begim. Zhuldyz Kabidenova, Aruna Nyssanbay. Adnan Yazici, and Seong-Whan Lee. Eav: Eeg-audio-video dataset for emotion recognition in conversational contexts. Scientific data, 11(1): 2024.
Hengzhi Li, Megan Tjandrasuwita, Yi R Fung, Armando Solar-Lezama, and Paul Pu Liang. Mimeqa: Towards socially-intelligett nonverhal foundation models, arXiv preprint arXiv:2502. 16671, 2025a.
Jiapeng Li. Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In
Proceedings of the IEEE/CVF international conference on computer vision, pages 11963-1199 11974, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730 19742. PMLR, 2023b.
Renjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang. Zhiwen Fan, Hezhen Hu, and Zhengzhong Tu. Mmhu: massive-scale multimodal benchmark for human behavior understanding. arXiv preprint arXiv:2507. 12463. 2025b.
Xinyu Li and Marwa Mahmoud. Interpretable concept- based deep learning framework for multimodal human behavior modeling. arXiv preprint arXiv:2502. 10145, 2025.
Bin Liang, Lin Gui, Yulan He, Erik Cambria, and Ruifeng Xu. Fusion and discrimination: A multimodal graph contrastive learning framework for multimodal sarcasm detection. IEEE Transactions on Affective Computing, 15(4):1874-1888 2024a.
Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion. In ACT,, 2018.
12

---

Published as a conference paper at ICLR 2026
Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Russ Salakhutdinov. and Louis- Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. The eight
Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, (6(10):1-30 42, 2024c.
Yuanyuan Liu, Ke Wang, Lin Wei, Jingying Chen, Yibing Zhan, Dapeng Tao, and Zhe Chen. Affective computing for healthcare: Recent trends, applications, challenges, and beyond. arXiv preprint arXiv:2402. 13589, 2024.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13 (5):c1193391, 2018.
Camillo Lugaresi, Jiugiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays. Fan Zhang, Chuo-Ling Chang, Ming Guang Yong. Juhyun Lee, al. Mediapipe: A framework for building perception pipelines. arXiv preprint (.X:::900.08//111 2019.
Miaosen Luo, Jiesen Long, Zequn Li, Yunying Yang. Yuncheng Jiang, and Sijie Mai. Multimodal large language models for end-to-end affective computing: Benchmarking and boosting with generative knowledge 2025.
prompting. arXiv preprint
Leena Mathur, Paul Pu Liang, and Louis-Philippee Morency. Advancing social intelligence in ai agents: Technical challenges and open questions. arXiv preprint arXiv:2404.1 1023,, 2024.
Christian Montag, Chengzhong Xu, Michiel Spapé, and Erik Cambria. The emotion labeling problem in affective computing research.
Shrikanth Narayanan and Panayiotis G Georgiou. Behavioral signal processing: Deriving human behavioral informatics from speech and language. Proceedings of the IEEE, 101(5):1203-11) 10330333,301111111111111113-1331 2013.
Atte Oksanen, Magdalena Celuch, Reetta Oksa, and lina Savolainen. Online communities come with real-world consequences for individuals and socicties. Communications Psychology, 2(1):71, 2024.
OpenAl. Gpt-5. https://oppeee http://aa..coggpptttettao gpt-5/ August 2025.
Maja Pantic and Leon JM Rothkrantz. Toward an affect--enssiive multimodal human-computer interaction. Proceedings of the IEEE, 910):1370-1390, 2003.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Rosalind W Picard. Affective computing. MIT press, 2000.
M. Kathleen Pichora-Fulle Fuller and Kate Dupuis. Toronto emotional speech (TESS). Borealis, 2020. doi: 10.5683/SP2/E8H2MF URL hhttts://///// /oo.. 0ttp//00..883//2/88811
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi party dataset for emotion recognition in conversations. In ACL, pages 536.536, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models unsupervised multitask learners. 2019.
Alee Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning. PMLR, 2023.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz. and Ethan Dyer. Effect of scale catastrophic forgetting in neural networks. In International conference on learning representations, 2021.
Arun V Reddy, Ketul Shah, William Paul, Rohita Mocharla, Judy Hoffman, Kapil Katyal, Dinesh Manocha, Celso M De Melo, and Rama Chellappa. Synthetic-to-real domain adaptation for action recognition: A dataset and baseline performances. arXiv preprint 2o0.xi2:33332220 2023.
Fabien Ringeval, Björ Schuller, Michel Valstar, Nicholas Cummins, Roddy Cowie, and Maja Pantic. Avec' 19: Audio/visual emotion challenge and workshop. In ICMI, pages 2718-2719, 2019.
13

---

Published as a conference paper at ICLR 2026
Moctar Abdoul Latif Sawadogo. Furkan Pala, Gurkirat Singh, Imen Selmi, Pauline Puteaux, and Alice Othmani. Ptsd the wild: vidco database for studying post-traumatic stress disorder recognition in unconstrained environments, Multimedia Tools and Applications, 83(14):42861 -2284428881228,,, 2024.
Björn Schuller, Stefan Steidl, Anton Batliner, Julia Hirschberg. Judee K Burgoon, Alice Baird, Aaron Elkins, Yue Zhang, Eduardo Coutinho, and Keelan Evanini. The interspeech 2016 computational paralinguistics challenge: Deception, sincerity & native language. In 17TH Annual Conference of the International Speech Communication Association (Interspeech 20/6), Vols 1-5, volume 8, pages 2001-2005. ISCA, 2016.
Muhammad Bilal Shaikh, Douglas Chai, Syed Muhammad Shamsul Islam, and Naveed Akhtar. From enns to transformers in multimodal human action recognition: A survey. ACM Transactions on Multimedia Computing, Communications and Applications, 20(8):1 100:1124,,000 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Suwon Shon Superb-sg: Enhanced speech processing universal performance benchmark with semantic and generative tasks. arXiv preprint arXiv:2409, 10762 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
Yosephine Susanto, Andrew Livingstone, Bee Chin Ng. and Erik Cambria. The Hourglass Model revisited. IEEE Intelligent Systems, 35(5):96 1026 2020.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin. Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint WwXiv:2503.19788 2025.
Gaetano Valenza, Antonio Lanata, and Enzo Pasquale Scilingo. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE transactions affective computing, 3(2):237-249, 2011.
M. Valstar et al. Avec 2017: Real-life depression, and mond recognition workshop and challenge. In Proceedings of the 7th International Workshop on Audio/Visual Emotion Challenge, pages 3-10, 2017.
Michel Valstar, Jonathan Gratch, Bjöm Schuller, Fabien Ringeval, Dennis Lalanne, Mercedes Torres Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. Avec 2016: Depression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pages 3-10. ACM, 2016.
Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Youssouf Kebe, Paul Pu Liang, and Louis-Philippe Morency. Social-iq 2.0 challenge: Benchmarking multimodal social understanding. https:///gith:t O-c.leeeeeeeneeeee 2023.
Xuecheng Wu, Dingkang Yang, Danlei Huang, Xinyi Yin, Yifan Wang, Jia Zhang, Jiayu Nie, Liangyu Fu, Yang Liu, Junxiao Xue, et al. emotions: A large-scale dataset and audio-visual fusion network for emotion analysis in short-form videos. arXiv preprint arXiv:2508.06902 2025.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answeringg explaining temporal actions. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
Baijun Xie and Chung Hyuk Park. Multi-modal correlated network with emotional reasoning knowledge for social intelligence question-answeringg In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3075-3081, 2023.
Jin Xu, Zhifang Guo, Jinzheng Hc, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2022 2002,,,,, 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509. 2025b.
Shanliang Yang. Shaojie Liu, Guangjun Nie, Lei Wang. Tao Wang, Jiebing You, and Erik Cambria. Fine-grained multimodal fusion for depression assisted recognition based on hierarchical knowledge-enhanced prompt learning. Expert Systems with Applications, 291:128532, 2025.
14

---

Published as a conference paper at ICLR 2026
J. Yu al. Ch-sims v2: A chinese multimodal sentiment analysis dataset. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1234-1243, 2022.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo. Yu Yue, Weinan Dai. Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source reinforcement learning system at seale. arXiv preprint arXiv:2503. 2025.
Amir Zadeh, Rowan Zellers, Eli Pincus, Louis- Phiis-Philippe Morency. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis online opinion videos. arXiv preprint arXiv:1606.0623 2016.
Amir Zadeh, Paul Pu Liang. Soujanya Poria, Erik Cambria, and Louis-Philige Philispp Morency. Multimodal language analysis the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In ACL, pages 2236-2246, 2018.
Wei Zhang, Feng Qiu, Suzhen Wang, Hao Zeng. Zhimeng Zhang. Rudong An, Bowen Ma, and Yu Ding. Transformer-based multimodal information fusion for facial expression analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2428-2431 2022.
X. Zhang et al. Mental-ppreeevee Multimodal mental health dataset and model. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun. Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. Humanomni: A large vision-speech language model for human-centric vidco understanding. arXiv preprint 1511: 2025.
Lianmin Zheng. Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-hench and chatbot Advances neural information processing systems, 36:46595-466 2023.
15

---

Published as a conference paper at ICLR 2026
A BENCHMARK DETAILS
In this appendix, we describe the corresponding details of our curated HUMAN BENIAVIR ATLAS benchmark.
A. DATASET DESCRIPTIONS
Below, we describe the datasets used in our benchmark and provide a concise rationale for their assigned behavioral dimensions.
CMU-MOSEI Zadeh et al. (2018) is one of the largest and most widely used multi-modal datasets for sentiment and emotion analysis, with over 30,000 samples. It provides a diverse set of real- world conversations in which speakers express opinions on various topics. Each video is annotated for sentiment polarity and emotion categories, enabling study on how speech, text, and facial expressions contribute to sentiment and emotional expression. This dataset reflects affective states through emotion and sentiment, and also relates cognitive states through categories such as surprise, which involve inference of internal mental processes beyond affect.
MELD (Poria et al., 2019) The Multimodal Emotionl Lines Dataset (MELD) contains a large collection of multi-pane party conversations drawn from 1,433 dialogues in the TV series Friends. Each utterance is annotated with emotion categories (neutral, joy, sadness, anger, surprise, fear, disgust) and sentiment polarity labels (positive, neutral, negative). This dataset reflects affective states through emotions and sentiment, relates to cognitive states through categories such as surprise and the need to reason over conversational context, and captures social processes through multi-party dialogue.
UR-FUNNYv2 (Haann al. 2019) is dataset of humor detection from more than 16,000 TED talk videas, with each short video clip annotated for humor indicators from words, gestures, and prosodic cues. This dataset reflects social processes through the recognition of humor in communication.
MUStARD (Castro et al. 2019) is a corpus of 690 videos for research in sarcasm detection from TV shows, such as Friends and the Big Bang Theory. It consists of audio-visual utterances annotated with sarcasm labels, with additional contextual information about the scenarios provided. This dataset reflects social processes through the interpretation of sarcasm in dialogue.
WOZ (Valstar et al. 2016) is designed for automatic depression diagnosis. The dataset consists of clinical interviews conducted with the help of a virtual agent (controlled by a human behind the scenes), with nearly 200 samples. Participants were asked mental health-related questions, and their responses were recorded for multi-modal analysis. This dataset reflects pathological dimensions by focusing on clinical depression.
CREMA-D (Cao al., 2014) is a dataset contains 7,442 audio files recorded by 91 actors (48 males and 43 females). Categorical emotion labels (the six basic emotions; anger, disgust, fear, happy, neutral, sad) for the perceived emotion displayed were collected via crowd-soureing from 2,443 raters. This dataset reflects affective states by providing categorical and intensity- -bsse emotional expressions.
CH-SIMS v2 (Yu et al. 2022) includes 4,403 samples with sentiment annotations. The dataset was collected from II different scenarios designed to simulate real-world human- with annotations based on sentiment scores from strongly negative to strongly positive. This dataset reflects affective states by capturing sentiment polarity and intensity.
interaction,
MMPsy (Zhang al. 2025) is dataset containing 1,275 real student records, including mental health scale scores, therapy interview audio recordings, and transcripts. The data were collected from over 15,000 middle school students in Guangdong Province, China. Annotations were obtained using standard psychological scales such as GAD-7 (Generalized Anxiety Disorder-7)
and MHT (Mental Health Test), collaboration with the local Department of Education. This dataset reflects pathological dimensions through the assessment of depression and anxicty.
PTSD in the Wild (Sawadogo et al. (024)) is a publicly available video database that provides a diverse range of real-world videos of individuals with PTSD symptoms. Videos were annotated as 'With PSTD' or Without PTSD'. This dataset reflects pathological dimensions through the diagnosis of post-traumatic stress disorder.
TESS (Pichora-Fuus Fuller and Dupuis 2020) The Toronto emotional Speech Set consists of 200 target words spoken by two actresses demonstrating seven emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral), with over 2000 samples. This dataset reflects affective states through explicit emotion portrayals, and also relates to cognitive states in that it includes
16

---

Published as a conference paper at ICLR 2026
categories such as surprise, which involve the inference of internal mental processes distinct from affect.
Social-IQ (Wilf et al. to evaluate Al's social intelligence by testing comprehension of human and behaviors in YouTube videos. This dataset reflects social processes through its focus understanding interactions, and also relates to cognitive states as it requires inferring internal mental processes such as intent and reasoning from observed behavior.
2023 is dataset containing over 1,000 videos and 6000 samples designed
IntentQA (Li et al. 2023a) is a video question-answering dataset built from the NExT-QA dataset Xiao al. (2021). ) comprising over 16000 samples. contains causal and temporal inference questions that require models to recognize the underlying intent behind human actions in everyday social scenarios. This dataset reflects social processes through its focus on intent in interaction, and also relates to cognitive states insofar as intent recognition involves inferring internal mental processes underlying human actions.
MimeQA [i et al. (2025a) is video question-answerte dataset derived from 8 hours of YouTube mime videos that focuses on nonverbal social reasoning with over 800 samples. It contains human pairs designed to evaluate Al models' ability to understand and interpret nonverbal gestures and movement without spoken language. This dataset reflects social processes by evaluating understanding of nonverbal communication.
annotated question-ansye
A.2 DETAILS OF BENCHMARK TRAINING SPLITS
We follow the official training splits of the original datasets whenever they are available. When such splits are unavailable, we create new splits by randomly partitioning the data at the level of separate video and audio clips. This approach ensures that samples from the same recording do not leak across train, validation, and test sets, while maintaining consistency with prior dataset usage. For transparency, the exact training, validation, and test counts for each dataset are reported in Table
Table 7: Train-validation-tee breakdown with per-dataset totals and the combined total across all datasets. Social-IQ 2.0 does not have public test set, therefore we utilize the validation set in line with existing literature Xie and Park (023): Chen et al. 2024b) ** "MimeQA has no validation set.
Dataset
Train
Val
Test
Total
CMU-MOSEI (SEN)
16326
1871
4659
22856
CMU-MOSEI (EMO)
6246
599
1753
8598
MELD (SEN)
9988
1108
2610
13706
MELD (EMO)
9988
1108
2610
13706
TESS
1960
420
420
2800
CREMA-D
5209
1116
1117
7442
CH-
2722
647
1034
4403
Social IQ 2.0"
5562
875
6437
IntentQA
12119
2044
2134
16297
MimeQA
633
173
806
UR-FUNNY
1709
192
224
2125
MUSIARD
483
104
103
690
DAIC-WOZ
189
MMPsy
893
191
191
1275
PTSD-in-the-Wi Wild
506
64
64
634
Combined Total
74451
9499
18014
101964
A.3 GEOGRAPHIC DISTRIBUTIONS OF DATASETS
We provide a breakdown of the geographic distribution of the datasets contained within our benchmark in Table 21
A.4 REFORMULATED PROMPT-TARGET PROMPTS
provide further examples of how reformulated datasets within our benchmark to standardized prompt-targe format that is compatible with foundation model training. To standardize the target format across datasets, we process them follows:
In Table 22
17

---

Published as a conference paper at ICLR 2026
MMPsy: continuous PHQ-9 scores discretized into categories following its paper guidelines. IntentQA, MimeQA, Sucial-IQ 2.0: targets are inherently open-ended and preserved as free-text responses, since they require inference through text generation.
All other datasets (e.g.. MELD, MOSEI, CH-SIMSv2, CREMA-D, TESS, UR-FUNNY, MUS- tARD, WTSD--nnthh-WWile DAIC-WOZ): contain discrete categorical outputs (e.g., sentiment, emotion, humor, sarcasm) and are retained in their original categorical form.
A.5 DETAILS OF BENCIMARK EVALUATION METRICS
In this section, we provide the full details of the evaluation metrics, We highlight the formulas used compute the various metrics reported in Table 2
For tasks HUM (Humour Detection), SAR (Sarcasm Detection), DEP (Depression Detection), ANX (Anxiety Detection), and PTSD (PTSD Detection), we compute the weighted FI score. The Fl score is defined as:
2- 2-Precision Recall
F1
Precision Recall
where
TP
TP TP FN
Precision
Recall
TP FP
The weighted Fl is then computed as:
Weighted-F1
Σ Fl ecC N
where is the number of true instances in class c, N is the total number of instances, and C is the set of classes.
For the SEN (Sentiment Detection) task, we use binary weighted F1, following Yu et al. (2022), which applies the same formula but only over the positive and negative sentiment classes. Since datasets such as MOSEI (Liang et al. 2018) may contain fine-grained sentiment scales (e.g., seven-point labels: strongly negative, negative, weakly negative, neutral, weakly positive, positive, strongly positive), we map all non-neutral predictions and labels into either negative positive classes.
For the EMO (Emotion Recognition) task, we compute the mean/ average weighted accuracy across all emotion classes (e.g., fear, surprise, joy), using the weighted accuracy formula, following Liang al. (2088)
TP P + 0.5
TN N
Weighted-AAcuracy = 0.5
where TP and TN are the number of true positives and negatives for the target class, and P and N denote the total number of positive and negative samples, respectively.
For free-text response tasks such as NVC (Non- Verbal Communication). INT (Intent Recognition), and SOC (Social Reasoning), we leverage an LLM judge to grade the generated responses. Specif- ically, we provide task-specific prompts to the LLM judge and record the proportion of responses marked as TRUE as an estimate of accuracy:
Accuracy number of TRUE responses
number of total responses
We utilize the following LLM grading prompts with the GPT-5-nano (OpenAl 2025) model. Evalua- tion Protocol for LLM-Judge Scoring. For all QA-style tasks requiring LLM-judge evaluation, we ensure strict isolation between the model under evaluation and the judging model. Each evaluation is performed in a fresh session, with shared history across queries. The LLM judge receives only the dataset-pre provvdded question and the model's 's generated answer, with no additional system context. All models are presented with identical prompts to maintain consistency and prevent evaluation bias. This protocol guarantees that scoring reflects the intrinsic quality of model responses rather than prompt design accumulated conversational context.
18

---

Published as a conference paper at ICLR 2026
L.M Judge Prompts
System Instructions
MimeQA:
Carefully consider the following question and answers regarding understanding of mime performance. You will be shown "gold-standard" answer from a human annotator, referred to as the "Reference Answer", and a "Candidate Answer". Your task to determine whether the candidate captures the core meaning of the reference answer using the following criteria:
The candidate must concisely state one primary answer in its first clause. Option lists without choosing one and deferrals are not allowed.
The candidate does not contain misleading information and does not hallucinate story plots not present in the reference answer.
Since the videos are mime performances, invisible actions, objects, or the mime actor portraying objects should be considered correct if and only if they are relevant to the question.
The candidate answer can be good answer in place of the reference answer long as they are the same ballpark. However, the candidate must not refer to a different subject object not supported by the question/reference. If the candidate's ' answer centers on different primary subject/object than the reference, it is incorrect. Social-IQ 2.0: Carefully consider the following question and answer regarding understanding of a video. You will be shown "gold-standard" answer from human annotators, referred to as the "Reference Answer". and a "Candidate Answer" Your task to judge whether the candidate captures the core meaning of the reference answer using the following criteria:
The candidate must concisely state one primary answer in its first clause. Option lists without choosing one and deferrals are not allowed.
The candidate's 's explanation is in the same ballpark as the reference and does not add claim that conflicts with it. The wording need not be the same; minor additions are allowed if they are consistent with the reference and the question.
The candidate should not assert conflicting explanation
introduce factually incompatible details. The candidate must not refer to a different subject or object not supported by the question/reffrrece. If the candidate's 's answer centers different primary subject/object than the reference, it is incorrect.
IntentQA: Carefully consider the question and answers about the intent behind actions in a video. You will be shown a "gold-standard" answer from human annotators, referred to as the "Reference Answer". and a "Candidate Answer". Your task to judge whether the candidate gives plausible interpretation of the intent that does not contradict the reference, using the following criteria:
The candidate must concisely state one primary answer in its first clause. Option lists without choosing one and deferrals are not allowed.
The candidate's explanation is in the same ballpark as the reference and does not add
claim that conflicts with The wording need not be the same; minor additions are allowed if they are consistent with the reference and the question.
The candidate should not assert a conflicting explanation, introduce factually incom- patible details, miss the core intent. The candidate must refer to different subject or object not supported by the question/referrece. If the candidate's 's answer centers different primary subject/object than the reference, it is incorrect.
User Instruction
Question: [question]
Candidate Answer: [candidate answer]
Reference Answer: [reference answer]
Please evaluate whether the candidate answer is correct according to the grading instructions.
19

---

Published as a conference paper at ICLR 2026
B MODEL IMPLEMENTATION DETAILS
In this appendix, provide further details the implementation of the models that trained our benchmark, OMNISAPIENS-7D SFT and OMNISAPIENS-77 RL, in appendices B. and B.2 respectively, while the implementation of OMNISAPIENS-77 BAM is detailed in Section 3.6
B.1 ARCHITECTURE AND TRAINING OBJECTIVES OF OMNISAPIENS-7E SFT
OMNISAPIENS-77 7B processes text, audio, and vision modalities from raw transcripts, audio clips, images, and videos by projecting their respective features into shared embedding space. In the first stage, audio and vision tokens are projected into the backbone LLM embedding space to align with textual embeddings. This allows the model perform cross-modal integration, where information from textual semantics, speech, facial expression, and broader scene context are fused to capture complex behavioral dynamics. Formally, let T denote the number of text tokens, and and the numbers of audio and visual feature frames, respectively. We obtain modality-speciff embeddings as:
Ttext Brexi W1:T) E TR
Zand nud Ennd (J ((AA IR RR
Zvis Pvis
ovis
R VxH
where Wir are text tokens embedded by Exext, and I.A and JLV are raw frame-level audio inputs and patch-level visual inputs, which first encoded by Eand and Evis and then projected into the shared hidden space R via Pand and Pvi. This allows audio and vision sequences to be aligned with text in the backbone transformer LLM embedding space.
The fused multimodal sequence:
Ztexti Zand; Evis J ((T+A+V)x
processed by the transformer LLM backbone F' with layers. We denote the penultimate layer (immediately before the final hidden layer) as:
Apenuut p(t-1) ER (T+A+++)x
To obtain fixed-size representation for classification, we apply masked mean pooling across the sequence dimension:
T+A my lppemali.t R
ET+ my
where m, {0,1) is an attention mask indicating valid tokens.
Classification head. Each task HUMAN BEHAVIOR ATLAS is assigned its own classifier head. For task with labels Y,, a linear head C. R - kil produces
4 = Ce(Pppenah)
Ú = softmax(fe),
where Y, represents the complete of unique labels for that task. When multiple datasets map to the same task, they share single classifier head, with Y, constructed as the union of the label sets across those datasets. We train with cross-entropy loss, with mini-batch size B:
1 B Yi
Cets
Σ Σ
1s
k] log Ú..
B
Decoder head. For open-ended generation, the sequence representation hppamlt continues into the language modeling head. conditional decoder G generates text autoregressively. where ViLL are target tokens and Vau is the output vocabulary size. At step
Or G (hppemult: Tee)= RR. Van.
P(Ue Tet, /pemtts softmax
20

---

Published as a conference paper at ICLR 2026
We train with the following teacher forcing loss:
qup
ΣΣ
( Di,ets hpeoutt.s).
B 1-1
For multi-task pretraining of OMNISAPIENS--P SFT, we jointly optimize both objectives, minimiz- ing the overall loss.
L Less Lagu-
B.2 DETAILS OF REINFORCEMENT LEARNING
Preliminaries. Group Relative Policy Optimization (GRPO) (Shao et al. 2024) is a reinforcement learning method designed specifically for fine-tuning language models. In standard reinforcement learning for LLMs, the model (policy) generates text responses to prompts, receives rewards based on response quality, and updates its parameters to maximize expected rewards. GRPO distinguishes itself from other methods like Proximal Policy Optimization (PPO) by eliminating the need for a separate value network estimate advantages, instead computing them directly from groups of sampled responses.
The core insight of GRPO is that for language generation tasks, the relative quality of responses to the same prompt provides sufficient signal for optimization. For a given prompt q at training iteration 4, GRPO samples multiple responses {o(g.i.t) [ from the current policy, forming group ((g,))- Each response ((g.i.)) (o(1, (g.i.)):: -... -((...))::) consists of sequence of n tokens. After generation, each response receives a scalar reward ((q.i.t) from a reward function that evaluates response quality according to task-specific criteria.
Rather than estimating the value function as in actor-critic methods, GRPO computes the advan- tage-a measure of how much better response is compared the expected performance-direde from the reward distribution within each group. The advantage for the i-th response is computed as:
A(gi.)) = T(g.k.t) (.G
(1)
oG +
where (.o and GG.,., are the empirical mean and standard deviation of rewards in the group, and is a small constant for numerical stability. This normalization ensures that advantages have zero mean and unit variance within each group, which stabilizes training across prompts with different reward scales and enables consistent gradient updates regardless of the absolute reward magnitudes.
The policy optimization follows trust region approach similar to PPO, updating the model parameters to improve responses while preventing excessive deviation from the current policy. The GRPO objective incorporates the normalized advantages with importance sampling and clipping:
((a),))(()) min (f (6) A clip (P),)))(()) -
(2)
To(o( (g.it))kk)
(()) Tead (o 9, (qi..)))))
where (( (0) represents the importance sampling ratio between the current policy and the old policy Tew, at token position k. The clipping operation with parameter prevents large policy updates that could destabilize training. The per-token formulation allows the model to learn which specific parts of the response contribute higher rewards.
The complete GRPO training objective combines the clipped advantage with KL divergence penalty:
Go
JoRPO(O) (0))==EDDD D.g-D..0
(O) DKDL (No (eer)
(G(goo)
Boca..
where D is the prompt distribution, Pagao is the length of response i, and 3 controls the strength of KI. regularization against reference policy mee (typically the initial supervised fine-tuned model).
21

---

Published as a conference paper at ICLR 2026
This regularization prevents the model from deviating too far from coherent language generation while optimizing for spsc-fpe rewards. The expectation is approximated through sampling during training, with gradient ascent performed batches of prompt-response pairs to maximize this objective.
Training Dataset. We train OMNIS 7B RL on the unified prompt-targer pairs from HUMAN BENIVVIR ATLAS, where all 101k samples have been standardized into a consistent format (Sec- tion 3.2). ). Each sample's s target represented as free-text to enable end-to-end optimization through the decoder head.
Prompt Structure. To enhance the model's reasoning capabilities, we augment each original prompt with explicit reasoning instructions. Specifically, for each behavioral analysis task with original prompt content, we construct the training prompt as:
[content]
You FIRST think about the reasoning process monologue and then provide the final answer The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in (boxed
internal
This structure encourages the model to generate intermediate reasoning steps before producing the final behavioral classification interpretation, improving both interpretability and accuracy.
Reward Architecture. We design composite reward function that evaluates three aspects of model outputs. Let ((g. denote the model's generated response for prompt y, and Yo the ground truth label. The total reward is:
[(a.,)) Tnce (g.,.)) ( Mq) + Aformat 'foomma (g))))))) Asim sim (g.s.)) 3a))
(5)
where:
Tace Taco (o. : Myy I (extract(o( = NyJ is the accuracy reward, which equals if the extracted answer from the boxed content matches the ground truth exactly iasseinneesiitive). and 0 otherwise.
Tiomal ())) haas think( )] [Kaas boxed(((a..1)) evaluates format compli- ance, awarding 0.5 points each for proper of <think> tags and boxed notation.
Tsim ( Vg = cos( (os(ee((g...))) ). e(y)) computes the cosine similarity between sentence embeddings of the predicted and ground truth labels, providing partial credit for semantically similar but not exact matches.
In our experiment, we set Aformat 0.2 and Agim 0.5.
C EXPERIMENTAL DETAILS & HYPERPARAMETERS
In this appendix, provide the full details of our training and inference in our experimental setups.
C.I EXPERIMENTAL PROCEDURES
In the multi-task experiments (Section general multimodal LLMs pretrained on broad multi- modal data (Qwen 2.5-0mni-7B, Qwen 2.5- VL-7B, Gemma-3-4B) and on human-related datasets (HumanOmmi-775 are taken directly from Hugging Face and evaluated in inference mode, without additional fine-tuning, We use the prompt-target samples formulated in Section 3.22 which contain free-text responses for all tasks and datasets, ensuring direct compatibility with the models' trained configurations.
In the transfer learning experiments (Section 4.2) we fine-tune both the pretrained OMNISAPIENS- 7B SFT and Qwen 2.5-Omni-7B separately held-out sets. For fair comparison, we attach classifier head to the penultimate layer of Qwen 2.5-Omni- Minni-BB's backbone, following the same design described in Appendix B. This setup helps control for design differences, making the primary factor of comparison the presence or absence of pretraining on the diverse behavioral tasks in HUMAN BEHAVIOR ATLAS.
22

---

Published as a conference paper at ICLR 2026
For training the Behavioral Adapter Module (BAM), we keep the backbone of the pretrained OMNISAPIENS-77 7B SFT fixed and integrate BAM as described in Section 3.6 Gradient updates applied only to the BAM and the classifier decoder heads. BAM is lightweight and additive, introducing minimal computational overhead and leaving the backhone representations intact, which makes it practical to train separate BAMs for each dataset. This approach accounts for variations in behavioral descriptors and label spaces across datasets, with datasct-specifie BAMs capturing these nuances while mitigating interference from the differing behavioral descriptor signals across datasets. For datasets that include both audio and visual modalities, we train two separate BAMs: one for the MediaPipe visual descriptors and one for the OpenSmile audio descriptors, each providing an additive residual update to /ppenutt. For datasets with only single modality, we train a BAM solely for the available set of behavioral descriptors.
C.2 HYPERPARAMETERS UTILIZED
For pretraining OMNISAPIENS-77 7B SFT on HUMAN BEHIAVIOR ATLAS, whether in the multitask setup (Section 4.1 or the transfer learning setup (Section 4.2 with 4 datasets held out from the pretraining corpus), we train for 5 epochs and select the checkpoint with the lowest cross-entropy loss on the validation set. We adopt an effective batch size of 512 (micro-batch size of 2 across 2 GPUs, with 128 gradient accumulation steps). Low-rank adaptation (LoRA) is applied in full float32 precision, with learning rate of 1 10 selected after parameter search between x 10-5 and 1 x 10-3 The LoRA configuration uses rank r = 32, scaling factor a = 64, dropout rate 0.05, and targets the modules [q-proj. k. k.proj. v v-proj, O o-proj. gate.proj, down proj. up-proj], We employ a cosine learning rate scheduler with 50 warmup steps to stabilize training during the initial phase.
For fine-tuning OMNISAPIENS-TE SFT and Qwen 2.5-Omni-7B on the separate held-out sets in the transfer learning experiments (Section 4.2). we adopt consistent configurations to ensure fair comparison. Specifically, we reuse the same LoRA parameters and learning rate search as described in the pretraining configuration above, while adjusting the effective batch sizes to account for the varying sizes of the held-out datasets (e.g.. DAIC-WOZ contains only ~ ~100 samples). The effective batch sizes are follows: 4 (micro-bater size of across 2 GPUs, gradient accumulation of 1) for 16 (micro-batch size of 2 across 2 GPUs, gradient accumulation of 4) for MUSIARD, 32 (miero-batte size of across GPUs, gradient accumulation of 8) for MELD (EMO) and MOSEI (SEN).
For training the Behavioral Adapter Module (BAM) (Section 3.6.. we first perform learning rate search. For the unfrozen classifier and decoder heads, we explore learning rate values between 1 x 10 and 5 10 selecting 1 10 as the optimal rate. For the BAM adapters, we search between x 10 and 8x 10-², selecting 5x 10-4, Each BAM is trained for four epochs, and the epoch achieving the lowest cross-entropy loss on the validation set is retained for testing. Consistent with the transfer learning experiments, we adjust effective batch sizes to account for datasct-specific sample sizes. The effective batch sizes are: 4 (micro- batch size of 2 across GPUs, gradient accumulation of 1) for DAIC-WOZ: 16 (micro-batch size of 2 across GPUs, gradient accumulation of 4) for MUStARD and MIMEQA; and 32 (micro-batch size of 2 across GPUs, gradient accumulation of 8) for the remaining datasets. Additionally, since BAM includes an intermediate hidden layer (Section 3.6). we set its hidden dimension to 256, while applying a dropout rate of 0.10.
SET and BAM training was done on 8x Nvidia H200 141GB GPUs, the Adam optimizer was utilized for all experiments. Additionally, while this was disable during transfer experiments and the training of the Behavioral Adapter Module. All experiments were conducted on the Pytorch and the Accelerate framework.
For reinforcement learning, we train OMNISAPIENS-77 RL using GRPO with the following key hyperparameters. The model is initialized from 1wee2.5-Ommii- and trained for 10 epochs with a batch size of 256. We use a learning rate of 5e - 7 with the Adam optimizer. For each prompt during rollout, we sample n = responses to form a group for advantage estimation. The maximum prompt and response lengths are both set to 4096 tokens. We process multimodal inputs (audio, video, images) with modality-awar batching to ensure efficient GPU utilization. The PPO mini-batch size is set to 128 with micro-batch size of 2 per GPU. We disable KL regularization in the reward (8 = 0 in the objective) following Yu et al. (025).. We perform validation every 5 steps to monitor convergence.
23

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
D.1 ZERO TRANSFER LEARNING
we conduct an additional transfer learn- ing experiment to test the zero-shot transfer of OMNIS APIENS- SeEiANs--BBLB 7BRL. Specifically, we trained OmniSapiens-7B RI on Human Behavior Atlas while holding out MOSEI (SEN), MELD (EMO), DAIC-WOZZ (DEP), and MUStARD (SAR) entirely from the training corpus. We then evaluated the model in strict zero-shot setting on these held-out datasets. For comparison, we used Qwen 2.5-Omni-7B, which shares the same underlying architecture as OmniSapiens-77 7B RL but is not pretrained Human Behavior Atlas. This enables clean attribution of the transfer gains solely from pretraining on HUMAN BEHAVIOR ATLAS, as opposed to any architectural modifications.
Following from the experimental setup in Section 4
Table 8: Zero-shot evaluation on held-out datasets. Bold denotes best score. Heldout datasets are defined the same way as in Table
Dataset
OMNISAPIENS-P RL Qwen 2.5-Omni--7
MOSEI (SEN)
0,247
0.201
MELD (EMO)
0,549
0,403
DAIC-WOZ' (DEP)
0.499
0.108
MUSIARD" (SAR)
0.59
0.445
From this table, we observe that pretraining the Human Behavior Atlas benchmark yields sub- stantial zero-shot gains on held-out datasets. OmmiSapiens-77 RL outperforms Qwen 2.5-Omni-7B across all four datasets, with improvements of +22.99% (+0.046) MOSEI (SEN), +36.2% (+0.1 on MELD (EMO), +362.04% (+0.391) on DAIC-WOZ. (DEP), and +33.9% (+0.151) on MUS- tARD, These findings suggest that Human Behavior Atlas pretraining provides strong transferable representations across diverse behavioral tasks, under zero-shot settings.
D.2 TRANSFER TO ADDITIONAL UNSEEN DATASET
Table 9: Transfer to IEMOCAP after minimal fine-tuning (1 epoch).
Dataset
OMNISAPIENS 7B SFT Qwen 2.5-Omni-7B
IEMOCAP
0.6213
0.5625
We further assess cross-dataset generalization by evaluating transfer to IEMOCAP, a large emotion- recognition benchmark that fully excluded from pretraining and differs considerably from Human Behavior Atlas datasets in domain, speaker context, and annotation style. As shown in Table g Under a one-epoch fine-tuning budget, OmniSapiens-7B SFT (pretrained on Human Behavior Atlas) achieves a +10.5% improvement (+0.059) over Qwen 2.5 5-0mmii-7B (without Human Behavior Atlas pretraining). This result suggests that the behavioral representations learned during multi-task pre- training may retain some adaptability to new emotion-recognitoo settings with distributional and contextual properties not present in Human Behavior Atlas. This offers an encouraging indication for transfer to new affective or emotion-recognitis benchmarks, where contextual and annota- tion variability often limits the generalization performance of models trained on existing emotion datasets (Montag al.).
D.3 BAM ABLATION OF RAW AUDIO AND VIDEO FEATURES
To analyze the isolated contribution of behavioral descriptors, we conducted an ablation in which we removed the raw audio and video features during the BAM model's training and inference. The resulting model, which we term OmniSapiens-775 BAM (ABL), is evaluated across all tasks. From Table we observe that removing the raw video and audio modalities leads to consistently weaker performance compared to the full BAM model, with OmniSapiens-7D BAM (ABL) underperforming BAM on most datasets. This suggests that behavioral descriptors provide complementary information to the raw audio and visual encoders, rather than serving as an effective standalone substitute. These findings offer practical guidance for deploying BAM within unified omni-mode architectures: behavioral descriptors are most beneficial when integrated jointly with raw multimodal signals
24

---

Published as a conference paper at ICLR 2026
Table 10: A highlights the change in performance from APIIS 7B SFT to OMNISAPIENS-T7 7 BAM and OMNISAPIENS- 7B BAM (ABL), shown as percentage (%) and absolute (Abs). Behavioral adapters (BAM.
ABL) provide notable performance gains for several behavioral tasks, although benefits are not consistent across all tasks.
Task
OMNISAPIENS-TB SFT
OMNISAPIENS-7B BAM
OMNISAPIENS-7B BAM (ABL)
ABAM % (Abs)
AABL (Abs)
NVC
0.12
0.16
0.06
+33.00 (+0.04)
-0.41 (-0.06)
SAR
0.62
0.80
0.75
+29.00 (40.18)
+20,97 (40.13)
HUM
0.53
0.64
0.64
+21.00 (+0.11)
+20.30 (+0.11)
DEP
0.73
0.79
0.75
+8.21 (+0.06)
+2.74 (+0.02)
EMO
0.59
(+0.02)
6,35 (-0.04)
SEN
0.77
0.76
+2.60 (+0.02)
-1.30 (-0.01)
PTSD
1.00
1.00
1.00
0.00 (+0.00)
0.00 (40.00)
ANX
0.91
0.91
0.909
0.00 (+0.00)
0.00 (+0.00)
SOC
0.26
0.20
0.22
23.08 (-0.06)
(-0.04)
0.18
-30.77 (-0.08)
-29.69 (-0.08)
D.4 BAM ABLATION OF HIDDEN DIMENSIONS
We evaluate more computationally expensive configuration of BAM using 512 hidden dimensions, from Table II From our results, we observe that increasing the hidden dimension does not necessarily yield better performance: out of 10 tasks show noticeably worse performance, while only 2 out of 10 tasks improve. This highlights that larger adapters do not consistently translate to performance gains, with lighter design of 256 hidden dimensions striking a more effective balance between efficiency and accuracy." We hope this clarifies the reviewer's 's question on the effect of varying the BAM design
Table 11: A highlights the change in performance from OPNISAPIENS-7N 7B SFT to OMNISAPIENS-7T 7B BAM (256) and OMNISAPIENS 7B BAM (512), shown as pereentage (%) and absolute (Abs). Behavioral adapters generally provide performance gains for several behavioral tasks, though not uniformly across all tasks.
Task
OMNISAPIENS-TB SFT
OXNISAPIENS-7B BAM (256)
OMNISAPIESS-TB BAM (512)
Apaez & (Abs)
AnAMon (Abs)
NVC
0.12
0.16
0.11
+33.00 (+0.04)
-9.09 (-0.01)
0.76
+29.00 (10.18)
122.58 (10.14)
HUM
0.53
0.64
0.67
+21.00 (40.11)
+25.94 (40.14)
DEP
0.73
0.79
0.77
+8.21 (+0.06)
+5.48 (-0.04)
EMO
0.63
0.65
0.64
+3.17 (+0.02)
+1,59 (-0.01)
SEN
0.77
0.79
0.77
12.60
(10,00)
PISD)
1,00
1,00
(40000)
(40.000
ANX
0.91
0.91
0.91
0.00
(+0.00)
0.15
-23.08 (40.06)
41.63 (-0.11)
INT
0.26
0.18
0,22
-30.77 (-0.08)
-14,06 (-D.45)
D.5 BAM ABLATION OF AUDIO AND VISUAL BENIAVIRAA DESCRIPTORS
Given that OMNISAPIENS-7B BAM integrates both the audio and visual behavioral descriptors, conduct additional experiment to ablate each visual and audio descriptor stream during OMNISAPIENS- 7B BAM'S 's training and inference. This allows us to analyze the contribution each visual and audio stream.
Table 12: A highlights the change in performance from OMNISAPIENS- 7B SFT to each BAM variant (with audio and visual bchavioral features ablated), shown as percentage (%) and absolute (Abs). Dash (-) highlights unavailable features due to the absence of modality etc.
ONeISSTIEES-188 OHNISANENS-TB BAM (AUTH ViS
ONENSATEENS-18 BAM-AUD)
OMRISATIENS-1P 1B BAM (VIS)
Sastrautt (ebb))
Aasureat (Abs)
Aeseco & (Abe)
1940
1663
1885
12.4
11.4
+3.07 (W1002)
(4ECC))
RTM
0.53
0:64
0.55
-31 1000000000 11)
#16 (40000)
+23 (5: 12:
INT
0/1
3.76
.30774005
P130
0.00002000
(-000-0000
030-42.000
074
(W115))
(vl EF)
SEN
1033
674
1178
47.00 (4117))
10 D15
0180
-35 0010000--00000 180
+1065 (40L19)
+30
-23.00-00361
From Table 12 the results indicate that while integrating behavioral features is generally beneficial, the synergistic effects of combining acoustic and visual descriptors are dask-dependent. For example, tasks such as EMO and SEN exhibit clear gains when both streams are used together: BAM (Aud+ Aud+Vis)
25

---

Published as a conference paper at ICLR 2026
achieves scores of 0.65 and 0.79, respectively, compared with 0.64 and 0.78 when using either BAM (Vis) or BAM (Aud) alone. However, for other tasks such as HUM and SAR. a single descriptor stream outperform the combined configuration. BAM (Vis) achieves 0.65 HUM and 0.81 on SAR, while BAM (Aud) reaches 0.81 on SAR. In contrast, BAM (Aud+ Vis) performs 0.64 on HUM and 0.80 on SAR. These mixed outcomes highlight that the utility of each behavioral stream varies across tasks. This reinforces the motivation for BAM's lightweight and computationally efficient design (two layers with 256 hidden dimensions). Namely, enables practitioners selectively discard behavioral descriptor stream (Vis/ Audio) and retrain BAM when particular descriptor stream does not provide meaningful benefit.
D.6 STUDY ON THE COMPUTATIONAL EFFICIENCY OF BAM
BAM is designed to be lightweight module (i.e. with small Feed-Forward Network) that can be integrated into multimodal LLMs with minimal additional computational cost. To quantify BAM's runtime and memory cost, we evaluate its impact latency and GPU memory consumption. Specifically We mea- sure the mean and standard deviation of latency, as well as peak VRAM usage, by rumning forward passes over 500 samples from the Human Behavior Atlas, All experiments use batch size 1, FP16 mixed precision, and are run on 4x NVIDIA H200 GPUs, with the first batch used for warm-up.
Table 13: Inference lateney and peak VRAM usage with and without the BAM adapter, evaluated 500 samples from the Human Behavior Atlas. experiments use FP16, batch and NVIDIA H200 GPUs.
Configuration
Mean Latency (S)
Std. I. Latency (s)
Peak VRAM (MB)
WITHOUT BAM
0.284
0.029
24,265
WITH BAM
0,300
0.047
24,291
BAM introduces only marginal increase in both latency and VRAM usage (e.g., +0.016s mean latency and +26 MB peak VRAM). These results highlight that BAM provides plug-and- and- play adaptation mechanism that adds negligible computational overhead
As shown in Table 13
D.7 ANALYSIS OF DATASET IMBALANCE
We conducted an analysis the potential impact of dataset imbalance performance in HUMAN BEHAVIOR ÁTLAS, as shown by Table 14 and Table 15) This includes computing the spearman correlation of performance against sample counts, and analyzing the performance rank of datasets with fewer samples.
Table 14: Analysis of dataset sample frequency versus model results.
Dataset
Sample Count
Sample Rank
SFT Score
SFT Rank
RL Score
RL Rank
BAM Score
BAM Rank
MOSEI (SEN)
31453
0.744
6
0.224
15
0,775
6
InteniQA (INT)
0.256
0.177
MELD (EMO)
13706
0.709
7
0.699
5
0.711
10
MELD (SEN)
13706
0.746
5
U.571
9
0.744
7
MOSEI (EMO)
0.614
I1
0,581
0.607
12
CREMA-D (EMO)
7442
0.542
0.501
11
0,548
Social-IQ (SOC)
6437
0.257
14
U,304
14
0.201
14
CH-SIMSv2 (SEN)
4403
0.813
0,393
13
0.837
4
TESS (EMO)
2800
0.658
8
0.510
10
0,715
9
UR-FUNNY (HUM)
2125
10
0.532
13
0.639
0,644 7
11
MMPsy (ANX)
1275
11
0.909
2
0.919
2
0.909
2
MMPsy (DFP)
1275
0.839
3
0,814
3
0.839
3
MimeQA (NVC)
0.121
16
0.133
0.162
MUSIARD (SAR)
0.624
0.647
0.795
Wild PTSD- Wild (PTSD)
634
15
1,000
0.968
1,000
DAIC-WOZ (DEP)
189
16
0.626
9
0.729
0.738
8
Our findings suggest that dataset imbalance has negligible impact on per-dataset performance. As shown in Table 14 and Table 15] we compare each dataset's sample count and sample-count rank against its downstream performance under SFT, RL, and BAM. Interestingly, several of the smallest datasets (i.e., those ranked in the bottom five by sample count) achieve performance in the top half of all datasets across training methods, with the only exception being MimeQA. To quantify this more formally, compute Spearman rank correlations between dataset sample count and model
26

---

Published as a conference paper at ICLR 2026
Table 13: Spearman correlations between dataset size and performance across SFT, RL, and BAM models.
Metric
Value
Spearman (SFT)
-0. -0.134554126
Spearman (RL)
-0.47971471
Spearman (BAM)
-0.342235494
performance. Across all three SFT, RL, BAM, training methods, the correlations are negative, with the magnitudes being weak non-existent, indicating consistent positive monotonic relationship between dataset size and task performance. In other words, having more samples for a given dataset does systematically correspond better performance the performance disparities across datasets are more likely driven by differences task difficulty and complexity rather than by dataset imbalance within the benchmark
that dataset. These results suggest that
D.8 TRANSFER COMPARISON
We contrast the SFT results (i.e. OMNISAPIENS-7D SFT and Qwen 2.5-Omni-7DB SFT) in our transfer experiments Section 4.2 to the models that have not been fine-tuned on any of the heldout datasets, with the Table 16
Table 16: Transfer to held-out datasets after minimal epoch fine-tuning (1 epoch). Bold denotes best score. DAIC-WOZ' uses 2 epochs due only 107 training samples. MUStARD" presents novel hehavioral task (sarcasm). & Other held-out datasets with tasks represented during pretraining.
OMNISAPIENS-T
Qwen
HumanOmmiV2
Qwen
Qwen
MOSEI' (SEN)
0.724
0.612
0.517
0.633
0.602
0,317
(EMO)
0.684
0.633
DANC-WOZ" (DEP)
0.749
0.579
0.137
0.666
0.636
0.623
MUSIARD'
0.473
0.395
0.666
D.9 ANALYSTS ON RETENTION OF GENERAL TEXT-GENERATION CAPABILITIES
We evaluate the free-form text responses generated by OmniSapiens-7B RI. and by Qwen 2.5-Omni- 7B, which shares the same architecture but lacks the additional Human Behavior Atlas training, across all question-answering tasks (IntentQA, MimeQA, Social-IQ 2.0) to analyze whether fine-tuning on Human Behavior Atlas influences general text generation quality. This involves computing perplexity over the generated outputs, and then evaluating fluency, coherence, and relevance using LLM-based scoring, following from established evaluation protocols for general text generation (Zheng et al.. 2023; Radford et al. 2019 Chen et al., 2023). Across the four metrics as shown in Table 17 we observe no meaningful differences between Qwen 2.5-Omni-7B and OmniSapiens-77 RI. These findings indicate that continued training on Human Behavior Atlas does not degrade the underlying text generation ability of the model.
Table 17: Human evaluation and perplexity results.
Model
Fluency
Coherence
Relevance
Perplexity
2.5-Omni-7B OmmiSapiens-7B RL
3.682
3.562 3.566
3.365
542.676
3.698
3.375
542.676
We highlight the LLM-judge prompt that we utilize for analyzing fluency, coherence, relevance the following.
27

---

Published as a conference paper at ICLR 2026
L.M Judge Prompts
System Instructions
You are an impartial evaluator of Al-generated answers to open-ended questions. You will be given:
user question A model answer evaluate
Your job is to rate the model_answe aanswrr three dimensions from to 5:
Poor noticeable errors very awkward phrasing; sometimes unclear. - 3: Fair understandable overall; a few minor errors or clunky phrases. Good clear and natural with only rare minor issues. 5: Excellent polished, natural, very casy to read.
1) Fluency: I: Very poor many grammatical errors, hard to understand.
Very poor disjointed, celf-contradictory. or very confusing. - 2: Poor some logical gaps or confusing wording; hard to follow. 3: Fair- mostly coherent but with minor confusion or awkward flow. 4: Good - ideas are easy to follow; no significant contradictions. Excellent very clear and internally consistent.
2) Coherence:
3) Relevance to the question: I: Very poor mostly irrelevant; barely addresses the question.
2: Poor partially on topic but misses the core of the question. 3: Fair addresses the general topic but may miss important aspects or be too generic. 4: Good directly answers the question and focuses on what was asked. 5: Excellent directly and specifically answers the question with minimal irrelevance.
Important: Ignore <think> tags and their content. Ignore follow-up questions. Do NOT penalize for brevity; short answers are acceptable if they satisfy the criteria above. Do NOT penalize for minor stylistic preferences. Do NOT judge factual correctness (accuracy is measured separately), except insofar obviously nonsensical statements hurt coherence.
User Instruction
Question:
{question}
Model Answer: (model answer)
D.10 LLM-JUDGE RELIABILITY
To assess the reliability of our LLM-judge for free-text response evaluation, we randomly sample 500 predictions from Qwen 2.5- 2.5-VL-7B across the SOC, INT, and NVC tasks. An independent human evaluator manually judges each prediction against the ground truth. We then compute Cohen's kappa between these human judgments and the original LLM-judge (GPT-S-nano) to quantify human-jin judge agreement. To further justify our choice a closed-source judge, we additionally evaluate Qwen3- 30B as open-source judge under the same protocol.
Table 18: Reliability and robustness of LLM-based judges for free-text evaluation. We report Cohen's 's between human annotations, GPT-5-nano, and Qwen3-30B under different judging setups. All results are computed over 500 randomly sampled predictions from Qwen 2.5-VL-7B across SOC, INT, and NVC tasks.
Judge
Compared Against
Prompt Setting
Cohen's
GPT-S-nano
Human
Original judge prompt
0.78
GPT-S-nano
Qwen3-30B
Original judge prompt
0.69
Qwen3-30B
Human
Original judge prompt
0.64
GPT-5-nano
Human
Paraphrased reordered prompt
0.75
find that GPT-5-nano exhibits strong agreement with human judgments, achieving a Cohen's s kappa of k = 0.78. It also shows reasonable inter-judge consistency with Qwen3-30B (== 0.69). Qwen3-30B proves to be weaker judge overall, with lower human-judge agreement of K= = 0.64.
From Table 18
To evaluate the robustness of LLM-judge performance to prompt variations, construct an al- ternative judging prompt by automatically paraphrasing the existing prompt using GPT-5-namo and randomly reordering rubric items. Under this altered prompt, GPT-5-nano maintains high hu-
28

---

Published as a conference paper at ICLR 2026
man-judge agreement (I 0.75), shown from Table I8 These findings collectively support the reliability and robustness of using LLM--ased judges for scoring free-text generation tasks. We highlight the paraphrased judge prompt in the following.
Paraphrased LLM Judge Prompts to Evaluate Prompt Sensitivity
System Instructions
MimeQA:
Carefully consider the following question and answers regarding understanding of mime performance. You will be shown "gold-standard" answer from a human annotator, referred to as the "Reference Answer", and a "Candidate Answer". Your task to determine whether the candidate captures the core meaning of the reference answer using the following criteria:
The candidate must avoid adding incorrect or invented details and should not intro-
duce narrative elements absent from the reference.
The candidate must provide one clear, primary answer and must not include option lists, hedging, or deferrals.
Because the videos depict mime performances, unseen actions, objects, or mimed elements are acceptable if and only if they are meaningfully relevant to the question.
The candidate answer should align closely enough with the reference answer that it could reasonably replace it, without shifting to different subject introducing unsupported entities.
Social-IQ 2.0: Carefully consider the following question and answer regarding understanding of video. You will be shown "gold-standard" answer from human annotators, referred to the "Reference Answer". and "Candidate Answer". Your task is judge whether the candidate captures the core meaning of the reference answer using the following criteria:
The candidate must offer one definite answer, without listing alternatives or postpon- ing judgment.
The candidate's explanation must remain within the conceptual scope of the reference answer and must not introduce claims that contradict it; minor consistent elaborations are allowed.
The candidate should not introduce distracting, inaccurate, or incompatible details and must focus on the same primary subject or object as the reference answer. IntentQA: Carefully consider the question and answers regarding the intent behind actions in a video. You will be shown a "gold-standard" answer from human annotators, referred to the "Reference Answer". and "Candidate Answer". Your task is judge whether the candidate presents a plausible interpretation of intent that does not contradict the reference answer using the following criteria:
The candidate must present a single, specific interpretation of intent and must not
provide alternative possibilities or noncommittal responses.
The explanation should remain compatible with the reference answer and aligned
with the question; minor additions are acceptable if they not introduce conflict.
The candidate must capture the core intent accurately without inserting misleading, irrelevant, or unsupported details and must not shift focus to a different subject.
User Instruction
Question: [question]
Candidate Answer: [candidate answer]
Reference Answer: [reference answer]
Please evaluate whether the candidate answer is correct according the grading instructions.
D.11 FULL RESULTS FOR SENTIMENT ANALYSIS
We provide the full results for sentiment analysis for CMU-MOSEI Zadeh et al. 2018), which involves 7-point sentiment labelling scale, in the following Table 19
29

---

Published as a conference paper at ICLR 2026
Table 19: class Results CMU-MOSEI emotion recognition.
Model
CMU-MOSEI
CH-SIMS v2
7-class F1
7-class Ace
7-class
7-class Acc
Gemma-3-4B
.166
.221
.439
.435
numanOmniV2-75
.10
.196
.432
.417
Qwen 2.5-Omni-7B
.199
.215
.397
.391
Qwen-2.5-VL-7B
.229
.202
.411
.361
OMNISAPIENS- 7B RL
.235
.270
.451
.434
OMNIS SFT
.308
.325
.470
.484
OMNISAPIENS-7B BAM
.310
.328
.484
.483
D.12 RL ABLATION
We provide an additional ablation of our RL experiments, by altering the different rollouts and learning rates, as shown Table 20 Beside showing the sensitivity of results to the different learning rates and rollouts, this highlights the rationale behind our choice of hyperparameters (rollouts of 5, learning rate of 5e-7), and learning algorithm (GRPO).
Table 20; Ablation studies number rollouts, learning rate, and training method. Best results are bolded. report binary weighted Fl for SEN; mean per-class weighted accuracy for EMO; weighted FI for HUM, SAR,
ANX, DEP, PTSD; and M-LM-ud accuracy for SOC, INT, NVC.
Model
EMO
HUM INT PTSD ANX DEP
SEN
SAR SOC NVC
UR-FUNNY
PTSD WILD
MMPS (D)
CH-SIMSV2
Social-1Q
CREMA-D
MELD (E)
MOSEI (E)
IntenQQA
MMPSY
MELD
MOSEL(S)
MUSIARD
MimeQA
TESS
Number of Rollouts (Ir = Se-7)
Rollouts =2
.500
.563
.576
499
.643
.422
.984
.669
.700
.588
333
.253
.670
.168
.162
Rollouts =
.501
.699
.581
.510
.639
486
.968
.919
814
.571
.393
.224
.647
.304
.133
Rollouts 20
.495
.522
.81
507
.600
.48
.944
882
.841
.577
.403
.252
.670
.179
.178
Learning Rate (rollouts
=== le-6
.96
531
.588
500
.640
.408
.968
881
.843
574
.371
224
640
.161
.150
Ir= 5c-7
.501
.699
.639
.486
.919
.224
.647
.304
.133
Ir lc-7
.501
.521
.508
.636
.755
.702
.211
.580
.161
Training Method (rollouts = 5, Ir= 5e-7)
RLOO
.501
.585
.573
.499
.613
.436
.984
.877
.852
.617
.404
.276
.675
.173
.170
GRPO
.501
.699
.581
.510
.639
.486
.968
.919
844
.571
.93
.224
.647
.304
.133
30

---

Published as a conference paper at ICLR 2026
Table 21: Multimodal datasets for behavioral analysis with their data and geographic locations. Bold locations indicate confirmed data collection sites; other locations are based on project authors' institutions.
Dataset
Source
Location
CMU-MOSEI
YouTube videos (1000+ speakers, various
Pittsburgh, PA, USA (CMU)
topics)
MELD
Friends series episodes
Singapore (SUTD)
TESS
Controlled lab recordings with profes- sional actresses
Toronto, Canada
CREMA-D
Controlled lab recordings with 91 profes- sional actors
Philadelphia, PA, USA (UPenn)
CH-SIMSv2
Chinese movies, TV series, multimedia content
Beijing, China (Tsinghua University)
Social-IQ 2.0
YouTube videos, movic clips, vehicle recordings
Pittsburgh, PA, USA (CMU)
IntentQA
Natural daily social activity videos (NExT- QA)
Xi' Xi'an, China (Xi Jiaotong Univ.)
MimeQA
YouTube mime performance videos TED talk videos (1741 speakers)
Cambridge, MA, USA (MIT)
UR-FUNNYV2
Rochester, NY. USA (Univ.
of
Rochester)
MUSIARD
TV shows (Friends, Big Bang Theory, etc.)
Ann Arbor, MI, USA (Univ. Michi- gan)
DAIC-WOZ
Clinical interviews with virtual inter- viewer "Ellie"
Los Angeles, CA, USA (data collection at USC) Guangdong Province, China
MMPsy
Web-based anxiety/depression interviews (Mandarin)
PTSD-in-the-Wild
Natural video recordings with varied ditions
Paris, France (Université de Paris)
31

---

Published as a conference paper at ICLR 2026
Table 22: Examples of converting heterogeneous datasets into unified prompt target format referencing including the modalities explicitly in the prompts, while prediction targets are standardized either in discrete label set for classification head setups, or a free-text response for decoder head setups. Reformulated prompts with targets are released as part of our benchmark.
prompt-target
Dataset
Original Inputs Outputs
Reformulated Prompt
Target
MMPsy
(De-
Text (Transcript) / PHQ-9 (0-27 nu-
4g
ii IX T G
2e I' TA
Diserete label set:
pression)
$
{Depression,
merical score)
T ffEl 3T T
No
X # 0.
depression]
Free-text
IX WE. #2 RR RE +
response:
My. NEV0, $1 F #
Answer:
T 18. 17 32 M
Depression
WE, AE TE
T
No depression
The above is an audic transcript of participant speaking in a elinical setting. Based on the speech, does the speaker show signs of depressicn? Respond with the follewing: Depression No Depression.
URFunny (Hu- mor)
Video Audio + Transcript / Humor detection false)
<video> <audio>
Discrete
label
set:
(True,
(true.
you know the story
Faise}
The above is a video and audio recording from conversation, along with transcript. Does this video segment contain a humorous punchline? Answer with one word: True or False.
Free-text
response:
Answer:
True / False
MOSEI (Emo-
Video + Audio + Transcript / Emo- tion recognition (jay, sadness, anger, neutral)
<video> <audio>
Discrete label
tion)
set: [HapPy,
had really long day work and I'm just exhausted, but seeing you made me smile The above is video and audio recording from conversation, along with the transcript. what emoticn is the speaker expressing? Choose from the follewing: Happy, Sadness, Anger, Surprise, Disgust, Fear.
Sadness,
Anger,
Surprise,
Disgust,
Fear)
Free-text
response:
Answer:
Happy /
Sadness
Anger
Surprise /
Disgust
Fear
CREMA-D (Emotion)
Audio + Transcript / Emotion recogni- tion (anger, disgust. fear, happy, neutral, sad)
<audio> It's 11 clock. The above a speech recording along with the transcript from a clinical context. What emotion is the speaker expressing? Answer with one word from the following: Anger, Disgust, Fear, Happy, Neutral, Sadness.
Discrete label
set: (Anger,
Disgust, Fear, Нарру,
Neutral,
Sadness}
Free-text
response:
Answer:
Anger /
Disgust
Fear / Happy
Neutral
Sadness
32


Reviews:
[{"review_id": "5cDaJq4zIQ", "reviewer": "Reviewer_PK4q", "summary": "This paper introduces HUMAN BEHAVIOR ATLAS, a large-scale multimodal benchmark with numerous tasks for general psychological and social behavior understanding. The core data contribution is curation and standardization of existing public datasets. Three variants of OMNISAPIENS-7B are evaluated to show the effectiveness of multi task training and transfer learning.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Benchmark, source code, models will be released.\n* Atlas is a large-scale multimodal benchmark with numerous tasks for general psychological and social behavior understanding.\n* Paper is well-written, and easy to follow.\n* Promising performances for multi-task training and transfer learning are shown.", "weaknesses": "* My major concern is the limited contributions\n  * The paper does not propose a new model architecture.\n  * The core data contribution is curation and standardization of existing public datasets instead of new data collection.\n* The comparison in Table 4 is unfair because the OMNISAPIENS-7B variants were trained directly on the HUMAN BEHAVIOR ATLAS data, while the general multimodal LLM baselines were evaluated in zero-shot inference mode without fine-tuning on this specific benchmark. The performance gain largely reflects the benefit of SFT or RL on the target tasks, not necessarily the inherent superiority of the model's architecture or the benchmark itself.\n* To demonstrate the superiority of the ATLAS for unified modeling, the authors should compare OMNISAPIENS-7B variants to models trained exclusively on another large, existing social behavior or affective computing dataset (e.g., a comprehensive version of CMU-MOSEI, MELD, or a large synthesized dataset like HumanOmni) and then test all models across the full range of ATLAS tasks.\n* A similar issue in the transfer learning experiment in Section 4.2 and Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8HnbZftLK", "reviewer": "Reviewer_aP6y", "summary": "The paper proposes HUMAN BEHAVIOR ATLAS, a unified multimodal benchmark focused on understanding psychological and social behaviors. The benchmark combines 13 existing datasets into a prompt-response instruction format across text, audio, and video modalities, totaling around 101k samples spanning 10 behavioral task categories. Three model variants are trained and evaluated: OMNISAPIENS-7B SFT, OMNISAPIENS-7B BAM, and OMNISAPIENS-7B RL. Results show that these models outperform general multimodal LLMs (Qwen 2.5, Gemma-3, HumanOmni) on both multi-task and transfer-learning evaluations.", "questions": "1. In Table 4, can you add results using only the behavioral descriptors (e.g., OpenSMILE, MediaPipe) to show how much these features alone contribute?\n2. Since the models are fine-tuned on the same datasets used for evaluation, have you checked whether fine-tuning affects general-purpose abilities (e.g. text generation)?\n3. Did you perform any true zero-shot evaluations to measure actual generalization?\n4. How were prompts generated - hand-written or LLM-generated?\n5. In Table 5, it would be interesting to add results before fine-tuning on the held-out datasets and also show results for the multimodal LLMs used in Table 4 for comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The benchmark is a useful contribution that could help move multimodal LLM research toward more holistic behavioral understanding. \n2. The experimental setup is thorough, covering 10 in-domain datasets and some \"transfer\" datasets. \n3. The benchmark and code will become publicly available.", "weaknesses": "1. No new data is collected - the benchmark repackages existing datasets into a unified format. While valuable, this limits novelty.\n2. The models are fully fine-tuned on the same datasets used for evaluation. In the transfer-learning section, the model is again fine-tuned for a few epochs on the \"held-out\" datasets, and only Qwen 2.5-Omni-7B is used for comparison. This setup mainly measures fine-tuning efficiency, not true zero-shot generalization.\n3. The construction of prompts is unclear. It’s not specified whether they were hand-crafted or automatically generated, nor whether prompt robustness was tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5N4zfbtTLQ", "reviewer": "Reviewer_3tAt", "summary": "The authors propose an unified benchmark of diverse behavioral tasks designed to support the development of unified models for understanding psychological and social behaviors. The aim is to take the opportunities for scalability, cross-task transfer, and broader generalization.", "questions": "What will be the restrictions about the use of this benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well organized and written\n-  The unified benchmark named HUMAN BEHAVIOR ATLAS comes in a timely manner for the field. It covers a large spectrum of situations.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KTehuVt3MV", "reviewer": "Reviewer_eg78", "summary": "The paper introduces **Human Behavior Atlas (HBA)**, a unified multimodal benchmark for psychological and social behavior understanding. Key contributions include:\n- A standardized **prompt–target interface** across 14 datasets and 10+ behavioral tasks.\n- A **Behavioral Adapter Module (BAM)** for injecting external behavioral descriptors (e.g., pose, acoustic features) into a frozen backbone.\n- Three training variants: **SFT, BAM, and RL (GRPO)**, evaluated comprehensively across tasks.", "questions": "1. **LLM-as-Judge Reliability**: Please report human–judge agreement (κ or ρ) on a 500-sample subset, add at least one open-source judge, and show sensitivity to prompt variations.\n2. **BAM & RL Ablations**: Systematically vary BAM’s injection layer, hidden size, and descriptor streams. For RL, sweep β, group size, and reward terms—report accuracy vs. compute and training stability.\n3. **Fine-Grained Evaluation**: Provide results on full label sets (e.g., 7-point sentiment) and justify binarization decisions with class-wise F1 and confusion matrices.\n4. **Metric Harmonization**: Propose a unified scoring scheme (e.g., normalized score per task family) and report confidence intervals over multiple runs.\n5. **Prompt and Protocol Transparency**: Document judge–model isolation measures and include a finalized, typo-free version of the judge rubric.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Unified benchmark** with broad task coverage and standardized metrics.\n- **Clean and modular design** of BAM, enabling non-invasive feature injection.\n- **Consistent empirical gains** of behavior-specialized models over general MLLMs.", "weaknesses": "#### **1. Insufficient Ablation Studies on BAM and RL**\n- **BAM Design Choices**: The paper does not justify key design decisions—e.g., why the adapter is inserted at the **penultimate layer**, why **mean and standard deviation pooling** is used for temporal descriptors, or why a **two-layer FFN** is chosen. A systematic sweep over injection points, hidden dimensions, and pooling strategies is missing.\n- **Descriptor Contribution**: The relative importance of **visual vs. acoustic descriptors** is not analyzed. Ablations disabling each stream would clarify their individual and synergistic effects.\n- **RL Hyperparameters**: The use of **β = 0 for KL regularization** is not motivated, and no sensitivity analysis is provided for group size, clipping epsilon, or reward scaling. Training stability and reward hacking risks are unexamined.\n\n#### **2. Evaluation Protocol Gaps**\n- **LLM-as-Judge Reliability**: The open-ended evaluation relies solely on a **single closed-source LLM judge** (GPT-5-nano). No human–judge agreement, inter-judge consistency, or prompt robustness tests are reported.\n- **Prompt Isolation & Contamination**: The paper does not clarify whether the judge and model prompts are isolated, whether sessions are reset, or whether tools are disabled—raising concerns about evaluation integrity.\n\n#### **3. Label Granularity Reduction**\n- Sentiment labels in datasets like **MOSEI (7-point scale)** are collapsed to **binary positive/negative**, discarding nuanced distinctions. This limits comparability with prior work and may mask model failures on fine-grained sentiment.\n\n#### **4. Metric Heterogeneity and Cross-Task Comparability**\n- Discrete tasks use **weighted F1 or weighted accuracy**, while open-ended tasks use **LLM-judged TRUE-rate**. This inconsistency complicates cross-task aggregation and model ranking.\n- No unified scoring mechanism (e.g., normalized score per task family) is proposed.\n\n#### **5. Lack of Computational Efficiency Analysis**\n- The computational cost of BAM (e.g., latency, memory overhead) is not reported, nor is its impact on inference speed or scalability—key for real-world deployment.\n\n#### **6. Limited Discussion on Multimodal Fusion Strategy**\n- The fusion of text, audio, and video is performed via **simple concatenation** in the embedding space. More advanced fusion mechanisms (e.g., cross-attention, gating) are not explored or motivated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTh7hRRpXa", "reviewer": "Reviewer_tGVo", "summary": "By integrating diverse modalities and heterogeneous datasets under a single LLM-based paradigm, it demonstrates the potential of large models to generalize across emotional, cognitive, and behavioral domains. While some architectural or methodological innovations are limited, the work’s scale, data diversity, and behavior-centric vision represent a meaningful contribution that could serve as a foundation for future research in multimodal affective computing.", "questions": "1. Although the authors devote substantial discussion to explaining how behavioral descriptors benefit model training (a point already well established in many multi-task affective models) it is unclear why the model itself is not designed to predict these descriptors directly. Doing so would align more closely with the paper’s title, “Human Behavior Atlas”, which implies an explicit mapping or prediction of behavioral factors.\n2. In the implementation section, the authors carefully adjust minibatch sizes across different tasks. However, it would be more informative if they could quantitatively present the impact of task diversity on training. Moreover, since the datasets vary greatly in size, it remains unclear how the authors address data imbalance during training or does imbalance affect the model performance.\n3. The comparison models were not trained on such a large and diverse dataset, which raises concerns about fairness. A more rigorous and convincing evaluation would involve cross-dataset validation, for example, testing affective models across IEMOCAP, Aff-Wild2, or AffectNet, to demonstrate the generalization ability of the proposed approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors successfully harmonize heterogeneous datasets and tasks (e.g., emotion recognition, personality, mental health indicators) under a single large language model (LLM)-based paradigm, which demonstrates strong potential for generalization across domains.\n2. Leverages an unprecedented scale of multimodal behavioral data, enabling the model to learn rich representations that reflect both emotional and cognitive dimensions of human behavior.\n3. Experimental results demonstrate solid transfer capabilities across a wide range of downstream behavioral and affective computing tasks", "weaknesses": "1. The authors designed two types of output: a dedicated classifier for categorical predictions, and a decoder that generates open-ended responses from the final hidden states. However, this classifier+decoder structure seems inconsistent with the logic of using large language models. Ideally, we expect an LLM to use a single decoder to produce all outputs, while the classifier approach merely leverages the model’s encoding capability, which deviates from the intended design philosophy.\n2. The main contribution lies in integrating various tasks and datasets into a unified LLM-based training paradigm. However, the authors did not introduce any specific architectural designs to enhance model capability, for example, handling behavioral temporal dynamics, or effectively leveraging facial landmarks, acoustic cues, or pose keypoints. Instead, these modalities are simply fed into the model as additional data for brute-force learning, which shows limited innovation.\n3. Currently, LLMs perform poorly on regression tasks, yet affective computing tasks such as valence-arousal estimation or depression assessment require fine-grained regression rather than simple binary classification. The authors’ choice to convert PHQ-9 into a binary classification problem thus oversimplifies the task and fails to reflect real-world application scenarios.\n4. A common issue in multi-task learning is negative transfer or gradient conflict, but the authors did not address whether their method incorporates any targeted solutions to mitigate these problems.\n5. The superior performance in transfer learning could be attributed to the use of large-scale and diverse multimodal data (visual/acoustic cues) rather than the proposed model architecture or data structure itself, the paper does not clarify this distinction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)