Read the Room: Video Social Reasoning with Mental-Physical Causal Chains
Authors: Lixing Niu
Jiapeng Li
Xingping Yu
Xinyi Dong
Shu Wang
Ruining Feng
Bo Wu
Ping Wei
Yisen Wang
Lifeng Fan
Venue: ICLR 2026 Poster (2026)
Keywords: video question answering
social reasoning
theory of mind
causal chains
read the room
Primary area: applications to neuroscience & cognitive science

Abstract: "Read the room", or the ability to infer others' mental states from subtle social cues, is a hallmark of human social intelligence, but remains a major challenge for current AI systems. Existing social reasoning datasets are limited in complexity, scale, and coverage of mental states, falling short of the rich causal dynamics found in real-life interactions. In this work, we introduce R$^3$-Bench, an evaluation benchmark with fine-grained annotations of belief, intent, desire, emotion, and their causal chains in complex scenarios. Furthermore, we introduce R$^3$-FDT, a large-scale training set generated through a novel automated pipeline with the same chain structure. We conduct a comprehensive evaluation of state-of-the-art (SOTA) large vision-language models (LVLMs) on R$^3$-Bench, revealing substantial deficiencies in consistent multi-step social reasoning. We also fine-tune a 7B model on R$^3$-FDT, achieving notable improvements across multiple relevant benchmarks. Our contributions are three-fold: (i) a novel benchmark with richly annotated, multi-step causal reasoning data; (ii) systematic evidence that SOTA LVLMs fall far short of human-level reasoning; (iii) a scalable training dataset that significantly enhances social reasoning performance. The datasets and code are available at: <https://github.com/LiXingNiu/Read-the-Room.git>.

Published as a conference paper at ICLR 2026
READ THE ROOM: VIDEO SOCIAL REASONING WITH
CAUSAL CHAINS
Lixing Niu
Jiapeng Li
Xingping Yu Xinyi Dong
Shu Wang"
Ruining Feng' 7 Bo Wu
Ping Wei Yisen Wang' Lifeng Fan 2
i Corresponding author.
School of Intelligence Science and Technology, Peking University
2 State Key Laboratory of General Artificial Intelligence, BIGAI
3 State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi an Jiaotong University
School of Psychological and Cognitive Sciences, Peking University
Yuanpei College, Peking University *University of California, Los Angeles
7 Department of Automation, Tsinghua University MIT-IBM Watson Al Lab iniuustu pku. edu. cn, Lifengfan@bigai..dt
ABSTRACT
"Read the room", the ability infer others' mental states from subtle social cues, is a hallmark of human social intelligence, but remains major challenge for current AI systems. Existing social reasoning datasets are limited in complexity, scale, and coverage of mental states, falling short of the rich causal dynamics found in real-life interactions. this work, we introduce R Re-Ben,h, an evaluation benchmark with fine-grained annotations of belief, intent, desire, emotion, and their causal chains in complex scenarios. Furthermore, we introduce R²-FDT, a large-scale training set generated through novel automated pipeline with the same chain structure. We conduct a comprehensive evaluation of state-of-the-art (SOTA) large vision- vision-language models (LVLMs) on R -Bnech, revealing substantial deficiencies in consistent multi-step social reasoning. We also fine-tune a 7B model R--FDT, achieving notable improvements across multiple relevant benchmarks. Our contributions are three-fold: (i) a novel benchmark with richly annotated, multi-step causal reasoning data; (ii) systematic evidence that SOTA LVLMs fall far short of human-level reasoning; (iii) scalable training dataset that significantly enhances social reasoning performance. The datasets and codes are available htttp:://qitnn com/ httpg//www.....oooooooooa//aaeea wtt.://eeaatttt-eeeeeooo Rociee--ooo.iii....
INTRODUCTION
"Read the room" requires employing Theory of Mind (ToM) (Premack & Woodruff 1978) read oth- ers' minds and perform social reasoning with subtle cues; it represents higher-level social intelligence, and plays key role in helping people navigate social scenarios smoothly. Humans are innate with the ability to perceive huge hidden information from very simple cues (Heider & Simmell 1944 Shu
cet al. 2018 Zhu al. 2020 Fan al. 2022); however, remains great challenge for current AT. As shown in Figure [] the visible physical world is only the tip of the iceberg; beneath it lies a vast and often invisible mental world. just a few seconds, people perceive layers of causally linked mental states: who is aware of what, who is hiding what, and how others respond. These interpretations rely not only on observable actions but also on unspoken norms and contextual reasoning. Effective social reasoning thus involves (i) detecting subtle behavioral cues, (ii) estimating diverse evolving mental states, and (iii) identifying causal chains connecting physical and mental worlds over time.
Large language models (LLMs) have recently demonstrated strong reasoning abilities across various domains (Brown 2020 Wei et al., 2022 Kojima et al.) 2022] Bubeck et al. 2023 Wang et al. 2024ch. However, they still struggle with complex reasoning tasks such long- planning and scientific problem solving (Srivastava et al. 2022; Wang et al.) 2024d] Mirzadeh et al.
2024 Glazer
al. 2024). Social reasoning- crucial subset of complex reasoning- also remains challenging for

---

Published as a conference paper at ICLR 2026
Visible Physical World
Grayil
A and C shake hands,
extends a hand, but doesn't nolke.
places the extended hand on his own head.
exchange giance. spreads
Invisible Mental World
Ohh he didn'tsee me and went to shake hands with someone eise."
h, I've been neliced after
hope
for my
next didn't didn't tried to shake hands."
B Belief
To Intend Shake Hands
To Conceal the Original Intent Shake Hands
Ease the Awkwardness
social cue
Pretand the Hand Raised to the Head
Intent
social event
Eager
belief
Happy
Disappointed
Resigned
mental state
Self Deprecating
Excited
Embarrassed
Emotion
chain
The visible physical world live in is just the tip of the iceberg compared to the vast, invisible mental world behind (Zhu et al., 2020). this example, a brief social interaction reveals complex and dynamic mental activities: extends his hand shake with A, but A fails notice. B then pretends his outstretched hand was meant to touch his head, concealing his embarrassment C, however, sees through B's mental state and pats him on the shoulder offer comfort. In response, B shrugs and gestures self-deprecatingly to ease the awkwardness. Social reasoning is a critical aspect of social intelligence. Yet in long-term, dynamic interactions, capturing subtle cues, recognizing social events, estimating mental states, and identifying causal chains becomes increasingly difficult, making social reasoning even more intricate.
Figure 1:
LLMs (Shapira et al. 2023 He et al. 2023 Gu et al. 2024 Wang et al. 2024a). Critically, modeling social cognition requires visual signals infer mental states that language alone cannot capture.
To address this, large vision- language models (LVLMs) Liu et al. 2024 Zhang et al. 2023 Lin et al.
2023 Team al. 2023 2024 Hurst et al. 2024) have emerged, enabling multimodal understanding. Yet, current benchmarks provide limited evaluation of (i) diverse mental state estimation and (ii) the consistency and depth of social reasoning complex interactions. Moreover, there remains no large- scale video dataset covering multiple mental states and their causal relationships, hindering further development in this domain. As illustrated in Table I current video datasets suffer from significant limitations. MVBench (Li et al., 2024), MMBench- Video (Fang et al. 2024) and Video-MME (Fu al.) 2024) primarily focus on factual or visual comprehension, offering limited support for deep mental state reasoning and causal inference. MMToM-QA (Jin et al.) 2024), MELD (Poria et al. 2018) and IntentQA (Li al. 2023) target a few types of mental states and lack multi-step reasoning. Social-IQ (Zadch et al.) 2019) and Social-IQ 2.0 (Wilf al. 2023) incorporate social contexts but do not model fine- aonegrained causal chains or assess reasoning across linked events. Also, the training sets of Social IQ and Social IQ 2.0, which offer the broadest (though still limited) coverage, contain merely 1k videos and 6k+ questions, respectively. It is insufficient to support the training a foundation model capable of grasping the core, diversity, and complexity of social reasoning.
Therefore, we introduce a fine- aineggrined evaluation benchmark named R <e-Benc (Read the Room Reasoning Benchmark) and large-scale training set named R--FDT (Read the Room Reasoning
2

---

Published as a conference paper at ICLR 2026
Datassets
Real-World
Belief
Intent
Mental State
Destre
Emotion
Causality
CC
* Video
Training Set
QQA
# Video
Test Set
+04
NEæT-QA Xiao al. 2021
8.5k
MVRench Gra
4024
K
X
X
X
X
3.6k
4k
MMBenche Yoleo
2024
X
x
x
0.6k
2k
Video MMB Fu
2.7k
MMTaMMQA Fin et 2024
x
V
X
x
X
X
0.1k
0.6k
MELD (Poria
K
7
x
x
10k
2.6k
IntemIQ di (1/4)
x
2.1k
CausalChaes Paamnr et al. 2024
x
x
+
4
V
3.4k
3.5k
0.7k
0.7k
Social-1Q Zatch
1.5k
Social-1Q 20 Will e A723
x
1.1k
6.2k
0.3k
1.7k
Our Work
R'-Bench
7
7
5.1K
R1-FDT
2.8k 41k
Table I: Dataset comparison. "CC" means mental-physical causal chain. v represents the dataset does not focus on content. "# Video" and "# QA" represent the number of videos and question- answer pairs, respectively. The set sizes of Social-IQ are estimated using an 80%/20% ratio.
Foundation Dataset for Training). They capture rich social interactions and include: (i) detailed social events, (ii) mental states and their transitions, and (iii) multi-step mental-physical causal chains. We evaluate state-of-the-an (SOTA) LVLMs on R²-Bench, and further fine-tune a 7B model on R²-FDT. The results demonstrate that: (i) current models still fall short of human-level social reasoning; (ii) our training data provides notable improvements across several benchmarks.
In summary, our contributions are three-fold: (i) we introduce R²-Bench, novel benchmark with complete, fine-grained annotations for social reasoning: (ii) we show that SOTA LVLMs remain far from human-level performance this benchmark; (iii) construet R FDD using scalable automated pipeline, offering valuable training data to improve LVLMs' social reasoning capabilities.
2 R -BENCH: A HIGH-QUALITY TESTBED
We design R3-Bench in a natural, intuitive, and principled way, grounded in the foundational Theory
of Mind (Premack & Woodruff 1978), the belief-desire--nteenion (BDI) framework (Bratman
1987
Bandura's social cognitive partry--priiculaarly triadic reciprocal determinism (Bandura et al.
1986
Bandura 1989 and other modern studies in social cognition (Tomasello 2010; Pearl 2014
| Reisenzein 2006: 2009 Puica & Florea, 2013: Schlafike et al. [015]] Fan et al. 2022). ).22).111 It
systematically integrates both physical and mental dimensions of social interaction. We include key mental state variables (belief, intention, desire, and emotion) alongside observable physical variables (actions, expressions, dialogue, and other social cues). We further capture comprehensive and dynamic causal interactions among these variables, reflecting the intricate interplay between internal states and external behaviors and environments. This framework positions it as a valuable testbed for developing and evaluating computational models of social reasoning.
2.1 OVERALL DESIGN
R--Bench is further divided into R Ri-Bench-Hard which consists of sufficiently challenging questions originally designed by humans, and R² -eBeech--DX a diagnostic set derived from causal chains. differs from traditional video question answering (VideoQA) benchmarks in the following aspects: (i) It includes fine-grained annotations of mental-physical causal chains; (ii) Question-answer (QA) pairs are derived based on these causal chains; (iii) It enables comprehensive evaluation of various social reasoning capabilities, including reasoning consistency via causal chains.
We illustrate the design and structure of R -BnnhhDDX with example in Figure 2a Each video has one or more mental-physical causal chains. We denote a causal chain as E G, and a subchain as gd E gaub, Here, & and gsub respectively represent all the causal chains and subchains. Note that a chain consists multiple subchains. A subchain zu comprises one result node n and one or several reason nodes [n]]. All reason nodes point the result node via causal edges {e;). Every reason node n is necessary to deduce the result node n².
For each causal chain g, we generate of related QA pairs according the following rules:
3

---

Published as a conference paper at ICLR 2026
MentallPPyysical Causal
Beldft
Th Qhiee!
Maked
Connet
Chain-based GAa
Event
Balief GA
un cottig
WWat C4
autioo 4Anda aurins beciame mastee
(a) General illustration of R²- -e-ncc--DDD design. As il- lustrated in the middle section, the man's 's mental states, collectively form
(b) Examples of each question type in R -Bench-DX. The correct option is marked in green. In MSE exam- ple, the woman thinks that the man manly while bandaging him. EU example, the driver throws leftover chicken bones. In CW example, Louise feels nervous due to potential trouble. In CH/W example, the woman realizes Leonard misunderstood her affec- tionate farewell, she says goodhye to clarify.
Penny's mental states, and the a mental- physical causal chain. The bottom section displays all QA pairs derived from it. The red circle highlights subchain indicating that "Event-3 causes Penny to form Belief-2." The green trapezoid repre- sents the corresponding derived QA pairs.
Figure 2: Illustration of dataset design and question types of R R2-Bench-DXX
(i) For each node y. we generate Event Understanding or Mental State Estimation QA pair, depending on its node content. The content of a node is either about an event or a mental state.
(ii) For each subchain gdlb, we generate Causal- Whh QA pair and Causal-How/WWin QA pair, depending on whether the reasoning is abductive or deductive.
Therefore, there are four types of questions in Re-Bench-DXX We provide four examples selected from our dataset for the four question types respectively in Figure
Event Understanding (EU). Understanding events is the premise of social reasoning. We generate factual QA pair for each event node. For example, "What happens at the end of the clip?" and "What does Person B do when Person A reaches out her hands?".
Mental State Estimation (MSE). Estimating mental states is a crucial aspect of social intelligence. We consider typical mental states including belief, intent, desire, and emotion. We generate an inferential QA pair for a mental state node, such as "How does someone feel at the end of the clip?"
Causal- Why (CW)& & Howsal-How/Whatt (CH/W). CW questions focus on abductive reasoning and inquire about causes of a given result-for example, the reasons why Louise feels nervous in the third case of Figure 2bb In contrast, CH/W questions emphasize deductive reasoning and ask about the effects of cause, such as the outcome of the woman's 's belief in the fourth case of Figure (2 To increase question diversity, we include variations such as "How" and "What" in phrasing. An example Causal-HHow QA pair is: Question: "How did Sheldon's 's revelation affect Penny's 's emotions?" Answer: "Sheldon's 's comment about the ring's 's true value led to Penny's 's disappointment."
2.2 DATASET CONSTRUCTION
R--Bench is comprehensive evaluation benchmark featuring high-quality and sufficiently challeng- ing data. This ensured through three aspects: (i) recruiting participants to independently submit diverse video sources such as ads, short films, and real-life clips-that that meet our criteria; (ii) using a powerful model to filter for difficult and unpolluted samples, meaning data unlikely to appear in standard pretraining sets; and (iii) involving domain experts to verify and annotate the content. To support this, we design a rigorous data construction pipeline illustrated in Figure
-emmanddesigned QA pairs, organize a challenge to collect data from crowdworkers and volunteers. To guide this process, we establish clear principles and standards: (i) Data composition: Each data sample includes video clip, question, four
Human Data Collection. To support high-qualis
4

---

Published as a conference paper at ICLR 2026
I.Humnn Data Collection
IlI. Causal Chain Annotation& Verification
Prompt
IV. QA Generation
an Mocaos
Redeet
Viniiba
Sullc1
Emaaksn11
morls
Burmtonai day a11d0--0000
Event Underssanding
Intert-)
Event-3
Even16
"Thaar" The iarts
Mental State Estimation GA
Gemini
The mant's
Exer14
You druudd markir the ener
Penty's
Causei-Why QA
Cacca nkw, dit womman Ruaes
Video
Causai-llw /hatt GA
Consult MModtyyy
TT erover about the sub chaltt muad
R--Bench-Dr DX
Queetion: Why siwee dan want tn kleep Perry's rosm?
deecaus Shaelinn concerned about henney' heslth coents mumber oineright. c Diecsuse use Shaidon reecc quart place work on tos nesearch and Panny's noom is the cuaeleet
Beeenchhtte FP
R Nr--Bench DX
Correct Discard
Beccuse is feeliin surprise Leonard
comtitit by being Perny's room
Sheidon
returns.
Experts
Explanation
R'-Bench
A though Sholdon said ho was doing it Pennyy dicorty don't calnn down from hai nagtirmrr.. His csstiiuss lone indicated that the only wanted to sheep Plarmyy house becassee was atrart
II. Human Data Verification
V. QA Verification
Figure 3: The R²- ee-Bench construction pipeline divided into five stages. i) Human Data Collection. Subjects select videos from YouTuub and designed QA pairs with explanations. ii) Human Data Verification. Experts review the collected data and remove those that didn't 't meet standards. iii) Causal Chain Annotation& Verification. Experts annotate, verify, and refine the causal chains. iv) QA Generation. We utilize GPT-4o to generate QA pairs based on the causal chains. v) QA Verification. The experts from the previous stage verify and modify the generated QA pairs.
distractors, a correct answer, and a reasoning explanation. (ii) Strict standard: Questions must involve at least one mental state and emphasize causal relations. (iii) No specialized external knowledge required: Answers must be grounded primarily in observable cues and general common sense, without relying on specialized external knowledge. (iv) Explanation requirement: Each sample must include an explanation describing the reasoning process and underlying causal chain. (v) Data pollution
prevention: We exclude any sample that Gemini 1.5 Pro (Team al., 2024) answers correctly, ensuring the data remains unseen and challenging. This design ensures that the collected data reflects nuanced, causal, and human-level reasoning that current models are not yet capable of replicating.
Human Data Verification. While many submitted QA pairs successfully challenge Gemini 1.5 Pro, not all meet our quality standards. To ensure consistency, we conduct expert review involving annotators with backgrounds in cognitive science, linguistics, and AI. Each sample is assessed against our criteria-such as causal depth, mental state relevance, and clarity of explanation- two independent reviewers. Only samples approved by both are retained in our benchmark, forming R3-Bench-Hand Hard. Given the nuanced reasoning and social subtleties involved, such validation cannot be reliably automated, highlighting the necessity of human judgment for constructing a benchmark.
Causal Chain Annotation & Verification. To capture the reasoning behind each question, we conduct expert annotation of mental-physical causal chains. Annotators receive training based on strict principles and follow the structured format defined Section 2.1] Each verified sample annotated by one of its original reviewers. The annotation process consists of three steps: (i) reviewing the QA pair and its explanation, (ii) identifying key events and mental states as nodes, and (iii) linking these nodes into subchains according to causal relations. After annotation, the second expert conducts an independent review. If revisions are needed, both experts iterate to reach agreement on the final chain. This multi-stage, fine-grained process ensures that each sample reflects a coherent and interpretable causal reasoning flow grounded in rich multimodal context.
QA Generation & Verification. We use GPT-4o (Hurst et al. 2024 to generate QA pairs for each annotated causal chain, following the rules in Section 2. We invite the same experts from the causal chain annotation stage to verify the QA pairs generated from their own annotated chains. Each QA pair is checked against the following standards: (i) adherence to generation rules, (ii) accurate and unambiguous time references, (iii) full coverage of the corresponding node or subchain content, and (iv) only one correct answer among five options. If QA pair fails to meet these criteria, experts revise it or discard it if correction is infeasible. This process yields 4,840 QA pairs, which, together with causal chains, constitute R². Bench--X.. Detailed statistical analysis is provided in Section A.4.2
5

---

Published as a conference paper at ICLR 2026
Alignment
Bacar
decp rill Eakeeeerbbeee.
Aligned Information
Causal Chains
Oue Enllect
Chomerande Corp
II
Raloant Anmoaaiies
Plot
Searnes
DUTIE belllinn
B1
E1
Hallucination Detection
Events
Mental States
aralaar sul moda (ier))) wnakk
E.D.
Matching
Script Clip Loaar
Dialogues
Higr
Dialgpac
Dataa
Self-Correction
QAROpttion
Generation
Figure 4: An automated pipeline for generating large-scale video social reasoning data using purely textual inputs. By aligning video clips with rich textual sources such movie scripts and annotations, the system generates mental-physica causal chains and QA pairs via GPT-4o, and applies self- correction and hallucination detection for quality control.
R2-FDT: AUTOMATICALIA GENERATED DATASET WITH CAUSAL CHAINS
The development of VideoQA datasets for social reasoning is constrained by two significant chal- lenges. First, there a scarcity of high-quality videos that capture the complexity social interactions. Second, the annotation process is prohibitively expensive, requiring intensive labor to infer nuanced mental states, followed by extensive verification to reduce ambiguity. These bottlenecks have impeded the development of foundational models in this domain. To overcome these limitations, introduce an automated pipeline using human annotations and large models. As illustrated in Figure 4 is designed to generate large-scale training datasets at reduced cost while maintaining high data quality. Our methodology is centered on the following principles:
(i) Mitigation of Cross-Modal Reasoning Deficiencies: Current models often exhibit limitations such as inconsistencies and hallucinations when processing videos. Our pipeline circumvents these issues by utilizing textual descriptions of videos to enhance the credibility of generated content.
(ii) Assurance of High-Fidelity Annotations: The pipeline exclusively uses -uman--nnottted descrip- tions that are aligned with videos. This ensures that the foundational data maintains high degree of accuracy and reliability, consistent with expert annotation standards.
The proposed pipeline offers a cost-effective and scalable solution for generating high-quality training data, addressing a critical need for advancing foundation models in social reasoning.
3.1 INFORMATION ALIGNMENT
To enable large-scale generation social reasoning data with mental-physical causal chains, we leverage movie data rich and structured source. Compared to open-domain videos, movies offer diverse content along with detailed scripts and annotations that describe both visual context and mental states. Therefore, they naturally suitable for constructing training data that aligns structurally with our human- curated benchmark. We extract information from publicly available movie datasets such as MovieNet (Huang al. 2020), MovieQA (Tapawwi et al. (016), and MondensedMovves (Bain et al.) 2020), as well as corresponding movie scripts. As shown in Figure ( we extract key elements from each clip, including scene context, annotated events, mental states, and aligned dialogue. To ensure all generated QA pairs remain within the temporal boundaries of each
clip, we use Whisper (Radford al. 2023) to align scripts with detected dialogue. The alignment algorithm is detailed in Section A.5.
3.2 DATA GENERATION
Causal Chains Generation. Based on the aligned information, we prompt GPT-40 to infer causal links between events and mental states using contextual cues. In addition to leveraging existing annotations, the model can also identify latent mental states implied by character behavior and dialogue, thereby enriching the reasoning structure with nodes that
explicitly labeled but contextually grounded. As illustrated in Figure 4 each causal chain is represented symbolically and paired with natural language explanation. For each video clip, the model generates multiple chains

---

Published as a conference paper at ICLR 2026
that reflect the most salient social interactions. This automated procedure retains the structural depth of camann-curttd reasoning while making it feasible to build diverse and scalable training data.
Self-Correction To further improve data quality, require GPT-4o to self-correct the causal chains generated in the previous stage. It needs to check the consistency between the symbolic and textual representations of chains, and remove redundant chains. In this stage, 6% of chains are removed. Although this accounts for only a small proportion, it is necessary to prevent the model from learning from erroneous noises.
QA & Options Generation. In accordance with the guidelines outlined in Section 2.1 QA pair is generated for each node and subchain. To enhance the quality of the data, the inherent causal structure is utilized to create distractors that are both challenging and plausible. Specifically, employing the corrected causal chains and aligned information, GPT-4o generates four distractor options for each question, adhering to two primary principles: (i) the option is designed to be plausible from a common-senss perspective but lacks grounding in the video context; or (ii) the option is grounded in the video context but contradicts the established causal or mentalistic logic. These strategies necessitate a nuanced comprehension of both physical and social cues to arrive at the correct answer.
Hallucination Detection. We establish a hallucination detection stage to filter for high-quality QA pairs. Specifically, we inpur the original video and QA pairs into Gemini 2.5 Flash (Comanici et al., 2025), requiring it to perform a detailed ballucination analysis. During this process. the model is required to complete three key tasks: (i) determine if a QA pair aligns with the video, (ii) provide detailed explanation, and (iii) output confidence level. Only when QA pair is determined to be completely free of hallucination do we retain it. Furthermore, we employ rule-based filtering to remove QA pairs with ambiguous speaker references. This strict quality control mechanism ensures the reliability of the final dataset, providing high-quality data for training a foundation model.
The resulting training set, R²-FDT, consists of 41k QA pairs distributed across 2.8k videos. This constitutes large-scale, causally structured dataset. Statistical analysis is provided in Section A.5.2
EXPERIMENTS
Our experiments consist of two primary components. First, we evaluate broad set of popular LVL.s on our benchmark 7B (Wang et al. 2024b) on our training set RDFFDT. We then evaluate the trained model on R²-Bench as well as on external social reasoning datasets to assess its generalization capabilities.
R² -e-Bench. Second, we train one of the best-performing models. Qwen2-VI. VL-
4.1 EVALUATION SETTINGS
We report the evaluation results under two experimental settings: video-only ("-")) setting and video-plus-subtitle ("+Sub") setting. For models that cannot process video data directly, we uniformly sample 16 frames from each video to serve as input. For Gemini 1.5 which accept video imputs, raw videos and frames respectively. We found that Gemini 1.5 Pro's ' performance when processing raw video was inferior to its performance when processing individual frames, we exclusively report on the performance of Gemini 2.5 Pro (Comanici et al. 2023) using frame-based inputs. We provide video subtitles generated using Whisper and incorporate them into the text prompts.
For Be-nenhhDDX, our evaluation approach consists of two dimensions: conventional accuracy metrics and our proposed consistency-based metries. (i) Accuracy Metrics. We report the overall accuracy for each model, as well as accuracies across different question types. For the MSE category, we further break it down into four ine-graiiedd subtypes: Emotion, Belief, Intent, and Desire, These distinctions help us identify which dimensions of social reasoning are more challenging. (ii) Consistency Metrics. Despite achieving relatively high overall accuracy, many models exhibit severe inconsistencies when answering logically related questions. For instance, a model might correctly answer the question "Why did A happen?" with "Because of B," yet fail when asked "Is B present in the video?" shortly after. Humans, in contrast, demonstrate coherent reasoning processes across multi-step chains. maintaining internal consistency throughout. To better capture this gap, we introduce two evaluation metrics: Chain Consistency and Subchain Consistency. They assess whether the model can consistently answer a set of questions derived from a single causal chain or its subcomponents.
7

---

Published as a conference paper at ICLR 2026
A chain (or subchain) is marked "conssstent" only if the model answers all associated questions correctly- partial correctness not rewarded. Let D(g) denote the set of all data samples corre- sponding causal chain 9, and (g)" for subchain 9Tth Denote [9] as the number of causal chains, and | for subchains. A data sample is denoted as (v, 9, a ,A), where v is the video, q is the question, ast is the ground truth answer and A is the corresponding answer set. A model parameterized by 0 selects an answer a' according to:
arg max Po(alu,q.A) atA
We define Chain Consistency as:
Cons EgEg Ille M)ED()) I(a = a
(2)
9
and Subchain Consistency as:
Cons Egmb [o (gg I(a = ast)
Caub
These metrics go beyond surface-leve accuracy by evaluating whether model can sustain coherent reasoning across entire social interaction chains. They are particularly effective at revealing hidden weaknesses in models, where high average accuracy may mask fragmented or inconsistent behavior.
4.2 EVALUATION RESULTS AND KBY INSIGHTS
As shown in Table current models still lag signif- icantly behind humans on challenging social reason- ing tasks. Gemini 2.5 Pro achieves the best perfor- mance among the models with 59. 18%, yet remains far below the human level of 80.06%. Many mod- els have accuracies below or around 30,, indicating that most models perform close random guessing on R--Bench-Hard and fail to demonstrate effective, high-level social reasoning capabilities.
Table 2: Ri-Bench-Hard evaluation results (%). All models are evaluated under video- plus-subtitle setting. "+ Ours-FT (SFT)" and "+ Ours-FT (RLFT)" denote SFT and GRPO training R FDD respectively.
Model
Overall
Random
20
Idefics2 8B [Laurengon et al.
2024h
15.19
As illustrated in Table the evaluation results on R Bench-DX yield two primary insights. First, models find reasoning about mental states substantially more challenging than understanding factual events. For example, in the "+Sub" setting, GPT-4ooss accuracy drops from 89.77% on EU to 72.44% on MSE, high- lighting the difficulty of inferring abstract internal states. Second, a critical limitation is the models' failure to maintain coherent reasoning, despite high accuracy on individual questions. This disconnect is evident across all models. For instance, in the "+Sub" setting, GPT-4o scores 82. 24.4% in overall accuracy but only 25.36% in chain consistency (Cons"). Gem- ini 2.5 Pro shows a similar gap, with 86.34% accuracy versus 36. 60% consistency. This "high accuracy, low consistency" paradox reveals that models lack holis- tic, structured understanding of social interactions, a weakness that single-question accuracy metrics fail to capture.
PLLaVA-7B
Xu
2024
17.09
Video LLaV
Lin el
21133
18.35
PLLaVA-13B
2024
20.25
Phi-3.5. Vision
Ahiin al A024
23.73
Idefics--SB SB-LLama
(Laurençin
2024a
24.37
InternV1.2 8B
OnenCyVLab
A174
24.68
IntemVLL-22655 (Opent
24.68
Gemini 1.5 Fladi Leain al
21192
(Vides)
28.48
mPLUG-OWIB (Yoel 2U2
29.11
PLLaVA 34B
el 2004)
30.06
Gemini 1.5 Flast
Trean al 2024
30.38
GPT-4o Mini
(Hurst
2029
30.70
IntemnVL2-7615 (Upeat
2024
31.96
Gemini Pro
TeeMl at
2024
Viico)
34.81
Gemini Pro
(Frames)
39.24
GPT-4o (Hurst cral. 202-
48.73
Gemini 2.5 Pio
(Comamcc
2025
(Frames)
59.18
Qwen2-V1.-PP Wang et 2024h
31.96
Qwen2-VL-7B -Uls-FF (KLFTT
39.87
Owen2-VVL-7R. Ours-FT (SFT)
42.09
Human
80.06
4.3 FINE-GRAINED ANALYSIS ON COGNITIVE DIMENSIONS
To delve deeper into the specific failures current models, we move beyond a monolithic performance metric on R R'-Bench-Har presents the performance breakdown of various models across these dimensions, benchmarked against human performance. The results reveal clear and consistent pattern of deficiencies.
fine-grade mee-grinedd analysis based on six cognitive dimensions. Figure 5a
8

---

Published as a conference paper at ICLR 2026
Table 3: R Ri-Bench-DX evaluation results (%). MSE: Mental State Estimation. EU: Event Understand- ing. CW: Srusal-Why, CH/W: Causal-How/Whne How/What. Cons": Chain Consistency. Cons Subchain Consistency. "_" _ denotes the video-only setting, and "+Sub" denotes the video-plus-subting setting. "+ Ours-FT (SFT)" and "+ Ours-FT (RLFT)" denote SFT and GRPO training on R RDDT respectively.
Model
Setting
MSE
CH/W
Overall
Cons"
Coss
Emsttion
Belief
Intent
Overall
Random
20
20
20
20
20
20
20
20
Video-LLoVA
18.03
18.69
30.22
16.07
18.82
20.26
20.57
21.38
20.33
0:00
0.14
Lin al. 2023
+Sub
19.08
19.63
24.38
21,43
20.90
28.25
21.64
22.88
23.14
0:00
0.36
PLLaVA.7B
28.93
27.98
14:29
27.48
31.10
32.74
20.25
0.29
1.42
Xt 2021
+ Sub
22.64
22.43
26.59
23.81
23.81
36.21
29.25
29.02
29.28
0.00
1,21
M=ffc22-88
10.48
8.41
10.80
2.38
9.74
8.93
8.30
11.48
9.77
0.00
0.07
L.aurenpom al. 2024b
Sub
21,80
21.50
22.99
19.05
21.98
20.56
22.99
22.64
22.15
0.29
1.14
PLLaVA-13B
25.48
22.45
22.71
16.67
22.73
23.77
10
90:72
28.00
0.00
0.64
Xu al. 2024
Sub
27.04
30.84
26.04
26.19
27.73
37.61
58.58
35.57
34.92
0.58
2.56
GPT-4o Mini
43.19
57.63
55.
52.38
51.04
62.39
73.52
69.36
64.59
6.05
18.92
Huust al... 2024
Sub
43.19
57.63
55.40
52.38
51.04
62.39
73.52
69.52
64.63
6.05
18.99
LLlemVL2.88
47.17
42.11
46.71
61.75
58.69
54.19
3,17
11.17
VPpnnGVLab 2024
Sub
49.90
68.54
54.57
52.38
$6.37
73.20
73.38
68.31
67.83
8.06
25.01
dedis33-88 8B Juma3
40.88
$7.94
18.75
35.71
47.63
51.15
63:70
62:49
56.82
3.17
12,02
[1 (_aurenpom
2024g
Sub
45.49
71.65
55.12
54,76
55.70
74.33
74.45
69.44
68.49
8.07
23.83
Phisi.5-Vision
49.86
70.74
62.87
AAxlin ot ml. 2024
Sub
46.75
62.62
53.46
40.48
52.79
68.71
76.16 16
77.99
68.00
8.65
23:47
mPLUG-Owl3
48.01
59.50
42.94
45.24
29.46
52.46
67.90
63.22
58.95
2.88
13:05
Ya ml 9024
Suh
51.15
71.03
50.97
50.00
56.37
75.03
74.66
70.82
69.21
9.22
25.39
Gemini Flaxb Frames)
60.94
56.95
70.57
67.31
6.05
Teams et
Sub
49.90
72.90
65.37
59.52
80.54
74.78
73.22
Gemini 1.5 Flach (Vdee))
37.95
60:44
53.74
$7.14
49.38
69.21
70.39
65.48
63.68
8:65
22.62
Team al
2024
Sub
47.38
72.59
64.37
64,29
59.78
80.34
78.58
73.24
72.91
11.
30.73
InlemVL12-268B
42.14
45.43
59.36
57.56
3.17
10:60
OpenCVVaay am 2024
Suh
71.96
58.73
57.14
57.30
73.62
74.45
71.22
69.17
27.03
PLI.VA-34B
49
59.81
55.40
54,76
54.37
52.46
69.54
70.57
62.52
5.19
16.22
Xu al. 2014
Suh
53.67
70.40
68.14
69.05
63.03
78.44
77.30
78.42
74.28
14. 12
X3.50
InlemVL.2-766
42.35
63.86
4%
50.04
5.75
15:58
OpentiVLas
Sub
53.46
75.70
65.10
76.39
75.19
17.29
Geminii 1.5 Pro (Vide))
50.31
75.08
74.79
69.05
64.95
84.65
80.13
80.52
77.39
15:85
38.69
Team et
2024
Sub
56.39
74.77
74.79
73.81
57.44
84.55
80.50
77.45
27.31
19.60
41.61
Camini 1.5 Pro (Fermes)
44.44
K.97
Team al
2024
$3.67
74.45
77.56
71.43
67.03
78.66
78.04
20.75
41.10
GPT-4n
60.17
80.37
70.50
76.19
71:94
89.17
85.84
N2.54
82.23
25,07
47.94
Huust et 2024
Sub
61.01
80.69
79.78
76.19
72.44
89.77
86.05
$2.94
82.64
25.36
48.91
Gemini 2.5 Pro (Prnees)
56.81
83.49
50.06
77.12
83.85
81.57
80.79
24. 30
43.53
Coomanic al. 2025
Sub
65,83
$5.60
96.18
$6.34
36,40
58.82
58.07
60.75
55.
$4,76
58.51
59.25
72.31
71.87
65.93
3.49
19.63
Owen2-V1-77
Sub
58.91
70.40
59.52
70.02
78.66
74.90
Wang 2024b
Ours-PT (RLFT)
77.36
83.80
50.33
76.19
83.67
88.16
91.25
87.63
36.35
29 11
55.83
Ours-FT (SFT)
80.08
87.85
83.66
88.10
83.51
89.57
92.81
88.52
N8.74
35.3 73
60.05
Iluman
93.55
92.50
92.31
100.00
92.98
93.91
90.21
92.37
92.24
60.47
75.32
Our analysis yields several key insights. First, the most profound weaknesses are exposed in tasks requiring detection of Contradiction in Words vs Behavior and pragmatic inference for Implication Reading Between Lines. Even the best-performing model, Gemini 2.5 Pro, lags significantly behind human accuracy (c.g., 48.2% vs. 78.8% in contradiction detection). This reveals fundamental deficit in handling modality conflicts and performing pragmatic inference, suggesting models favor literal interpretations over grasping nuanced social contexts involving sarcasm or deceit.
Second, the results point to architectural limitations. On one hand, models lack systematic modeling of temporal event structures and evolving mental states. This explains why even Gemini 2.5 Pro and GPT-4o fall short of human robustness (e.g.. 77.4% vs. 87.1%) on Plot Twist - Mental State Change tasks. On the other hand, predominant architectures often align multimodal features only superficially. Consequently, when tasks demand deep integration of distinct modalities to uncover underlying meaning, particularly within the Must Combine Visual and Verbal Cues dimension, their performance drops sharply (to 58.6% and 50.5%, respectively).
Finally, the Imagination Beyond Video category serves powerful diagnostic. Here, the perfor- mance of top-tier tier models like Gemini 2.5 Pro and GPT-4o (both at 68.8%) closely approaches the human baseline (75.0%). This highlights the inherent advantage of powerful linguistic world model
9

---

Published as a conference paper at ICLR 2026
Socal-10
Intern0A
9-2011
R--BenchOx
Bl-Benchiltard
(a) Radar chart of model accuracy (%) across six cognitive dimensions. Results highlight significant gap between models and numan-level reasoning, particularly in tasks requiring detection of multi- modal contradictions and pragmatic inference.
(b) Accuracy (%) on video social reasoning bench- marks before and after fine-tuning Qwen2-VL-7B. "-"= video-only setting. "+ Sub" = video-pie plus-pusssss subtite setting, "+ Ours-FT (SFT)" SFT FDT, and "+ Ours-FT (RLFT)" GRPO training on R²-FDT.
Figure 5: Comparison of cognitive and social reasoning capabilities. (a) Model accuracy across six cognitive dimensions on Re-BBenchHHar (b) Improvements on relevant benchmarks after tie-tuning.
for tasks that require reasoning beyond direct perceptual evidence, affirming that strong language priors are crucial for imaginative inference.
In summary, this analysis indicates that future advancements in multimodal social reasoning must prioritize: (i) robustly modeling mental states and motivations; (ii) enhancing pragmatic inference for non-literal language understanding; and (īi)) improving resolution of contradictions across modalities.
4.4 TRAINING ON R3-FDT
To address previously identified challenges in consistency and multi-step reasoning, we apply reinforcement learning fine-tuning (RLFT) using the GRPO algorithm (Shao et al. (2024) and supervised fine-tuning (SFT) on a sampled subset of 13k QA pairs from RTF with subtitles incorporated into prompts. For RLFT, the reward signal is formulated as multiple-choice task. For SFT, we convert 10% of the training samples into an open-ended format.
Although the training and test sets differ substantially video domain our training set consists of curated movie clips, while the test benchmark includes diverse YouTube-sty content- they share unified structure grounded in causal chains. This structural alignment allows the model benefit directly from training signals, leading to performance improvements closely aligned with evaluation objectives. As shown in Figure training on our training subset yields substantial gains. Specifically, RLFT achieves improvements of +1.998% SFT shows increases of +13.84% and +7.91 % respectively. Further, as reported in Tables and 3 the fine-tuned models outperform Gemini 2.5 Pro on R -eBechh DDX and surpasses Gemin 1.5 Pro R T--Bench-Hare demonstrating enhanced social reasoning across both question styles.
Ri-Bench-DX and +5.69% on R R--ench-Harr, while
To assess generalization, also evaluate the model two external video social reasoning datasets: Social-IQ 2.0 and IntentQA. As shown in Figure 5b) RLFT yields improvements of 6.89% and 7.59%, while SFT results in gains of 3.87% and 4. 4.7,, respectively, confirming the transferability of reasoning capabilities acquired from our training subset.
5 CONCLUSION
In this work, we model the causal flow between mental states and physical events in complex social interactions through mental-physical causal chains. Based on this structure, we construct R R--Bench for evaluation and FDD for model development. R²-Bench highlights that current SOTA LVLMs still exhibit deficiencies in social reasoning capabilities. Fine-tuning Qwen2-VL-7B to notable gains across multiple relevant benchmarks, demonstrating its value. We hope they can contribute the advancement of multimodal social intelligence and encourage further exploration.
R FDD leads
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our study involves human participants. Informed consent was obtained from all participants prior data collection. The released datasets have been carefully anonymized to remove any personally identifiable information, and participants were informed that their data may be shared for research purposes. To mitigate potential misuse, the datasets are distributed under research-only license, and documentation describing appropriate usage scenarios is provided. We believe that the potential benefits of these datasets for advancing research outweigh possible risks, and we have taken steps to minimize privacy, security, and fairness concerns in accordance with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have taken multiple measures to ensure the reproducibility of our results. We provide detailed description of our data collection method Section and Section B] and report implementation details in Section A To further support reproducibility, we release datasets, code, and models.
ACKNOWLEDGEMENT
This work is supported by National Natural Science Foundation of China (62406031) I), and National Science and Technology Major Project (2022ZD01 14900). Yisen Wang is supported by National Natural Science Foundation of China (62376010, 92370129). Beijing Major Science and Tech- nology Project under Contract No. 10000842220 Beijing Nova Program (2023048434, 20240484642), and State Key Laboratory of General Artificial Intelligence. Thanks to Ms. Zhen Chen for creating the beautiful figures.

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally your phone, arXiv preprint arXiv:2404. 142199 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-237355 2022.
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. In Proceedings of the Asian Conference Computer Vision, 2020.
Albert Bandura. Human agency in social cognitive theory. American psychologist, 44(9): 117,, 1989.
Albert Bandura et al. Social foundations of thought and action. Englewood Cliffs, NJ, 1986(23-28)2. 1986.
Michael Bratman. Intention, plans, and practical reason. 1987.
Tom B Brown, Language models are few-shot learners. arXiv preprint urXiv:2005.14160 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2r33v:2303.11277 2023.
Zhihao Cao, Zidong Wang, Siwen Xie, Anji Liu, and Lifeng Fan. Smart help: Strategic oppo- nent modeling for proactive and adaptive robot assistance in households. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18091-18101, 2024.
Howard Chen, Long Ouyang, Tatsunori B. Hashimoto, Eli Zelikman, and Percy Liang Vila: Learning human utility-aware agents using language feedback. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Peng Chen, Xiao- Yu Guo, Yuan-Fang Li. Xiaowang Zhang, and Zhiyong Feng. Mitigating language bias of Imms in social intelligence understanding with virtual counterfactual calibration. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 1300-1310, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06266 2025.
Haodong Duan, Junming Yang. Yuxuan Qiao. Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, al. VImevalkit: An open-source toolkit for evaluating large models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198 11211, 2024.
Lifeng Fan, Yixin Chen, Ping Wei, Wenguan Wang, and Song-Chur Zhu. Inferring shared attention in social scene videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6460-6468, 2018
Lifeng Fan, Wenguan Wang, Siyuan Huang, Xinyu Tang, and Song-Chun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5724-5733,200 2019.
Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, and Yixin Zhu. Learning triadic belief dynamics in nonverbal communication from videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7312-7321, 2021.
12

---

Published as a conference paper at ICLR 2026
Lifeng Fan, Manjie Xu, Zhihao Cao, Yixin Zhu, and Song-Chun Zhu. Artificial social intelligence: A comparative and holistic view. CAAI Artificial Intelligence Research, 1(2), 2022.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-vides A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. 14515, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprini arXiv:2405.21075. 2024.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding
social reasoning in language models with language models. Advances in Neural Information 2023.
Processing Systems,
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falk- man Olsson, Jean-Stnee -taanntnssslas Denain, Anson Ho, Emily de Oliveira Santos, et Frontiermath: benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint 1..442241..0487,, 2024.
Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, and Yejin Choi. Simpletom: Exposing the gap between explicit tom inference and implicit tom application in llms. arXiv preprint arXiv:2410. Att.:240..3648,, 13648, 2024.
Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: A benchmark for evaluating higher-order theory mind reasoning large language models. arXiv 2023.
preprint arXiv:2310.166
Fritz Heider and Marianne Simmel. An experimental study of apparent behavior. The American journal of psychology, 57(2):243-255 1944.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 709-727. Springer, 2020.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, and Yejin Choi. Social chemistry 101: Learning reason about social and moral norms. In EMNLP, 2021.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprins 24X1:24011.08444 2024.
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions. arXiv preprint arXiv:2310.1522 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions. arXiv preprint 12227 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.00 02246, 2024b.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. arXiv preprint arXiv:2010.07999, 2020.
13

---

Published as a conference paper at ICLR 2026
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11963-11974, 2023.
Kunchang LI, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 22195-22200 2024.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXtv:2311.0112., 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yi-Long Lu, Chunhui Zhang, Jiajun Song, Lifeng Fan, and Wei Wang. Do theory of mind benchmarks need explicit human- reasoning in language models? arXiv preprint mrXiv:2504.01698 2025.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning large language models. arXiv preprint arXiv:2410.05229 2024.
Team OpenGVLab. Internv12: Better than the best expandinn performance bound- URL
aries of open-source multimodal models with the progressive scaling strategy. 2024.
https://internul.gibbee www.:/oo/2002000-----e---------
Joon Sung Park, Joseph O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023.
Paritosh Parmar, Eric Peh, Ruirui Chen, Ting En Lam, Yuhan Chen, Elston Tan, and Basura Fernando. Causalchaos! dataset for comprehensive causal action question answering over longer causal chains grounded in dynamic visual scenes. Advances Neural Information Processing Systems, 37:92769-92802, 2024.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.025000 2018.
David Premack and Guy Woodruff. Does the chimpanzee have theory mind? Behavioral and brain sciences, 144):515-52, 1978.
Mihacla-Alexandra Puica and Adina-Magdi Florea. Emotional Previous work and proposed architecture. International Journal of Advanced Research in Artificial Intelligence, 1-2) 2013.
agent model:
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help- A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.099: 09890, 2020.
Xavier Puig, Tianmin Shu, Joshua B Tenenbaum, and Antonio Torralba. Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7628-7634, IEEE, 2023.
Alec Radford, Jong Wook Kim, Tao Xu. Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492-28511 28518. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Rainer Reisenzein. Emotions metarepreesentationnll states of mind. na, 2006.
Rainer Reisenzein. Emotional experience in the computational belief-desire theory of emotion. Emotion Review, 1(3):114-22 222, 2009.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense 2019.
reasoning about social interactions. arXiv preprint
Lara Schlaffke, Silke Lissek, Melanie Lenz, Georg Juckel, Thomas Schultz, Martin Tegenthoff, Tobias Schmidt-Wilen and Martin Brüne. Shared and nonshared neural networks of cognitive and affective theory-of-mind A neuroimaging study using cartoon picture stories. Human brain mapping, 36(1):29-39, 2015.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03.8 2024.
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory mind? stress testing social reasoning in large language models. arXiv preprini arXiv:2305. 14763, 2023.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 19256, 2024.
Tianmin Shu, Yujia Peng, Lifeng Fan, Hongjing Lu, and Song-Chun Zhu. Perception of human interaction based on motion trajectories: From aerial videos to decontextualized animations. Topics in cognitive science, 10(1):225-241, 2018.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4631 2016.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11803 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Michael Tomasello. Origins of human communication. MIT press, 2010.
Junqi Wang, Chunhui Zhang, Jiapeng Li. Yuxi Ma, Lixing Niu, Jiaheng Han, Yujia Peng, Yixin Zhu, and Lifeng Fan. Evaluating and modeling social intelligence: A comparative study of human and ai capabilities. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song- Chun Zhu, and Hangxin Liu. Llm" 3: Large language model-base task and motion planning with motion failure reasoning. arXiv preprint arXiv:2403. /552, 2024c.
Shu Wang. Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of lims in the chess testbed. arXiv preprint arXiv:2411.00 2024d.
15

---

Published as a conference paper at ICLR 2026
Ziqiang Wang. Yujia Zhou, Ning Ding. Zhiyuan Liu, and Maosong Sun. Soda: Million-scale dialogue distillation with social commonsense context. In ACL, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768) 2022.
Jianan Wei, Tianfei Zhou, Yi Yang, and Wenguan Wang. Nonverbal interaction detection. In European Conference on Computer Vision, pp. 277-295, Springer, 2024.
Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Youssouf Kebe, Paul Pu Liang, and Louis- Philippe Morency. Social-iq 2.0 challenge: Benchmarking multimodal social understanding. 2023.
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow13: Towards long image-sequence understanding multi-modal large language models. arXiv preprint arXiv:2408.044: 2024.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq:
A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8807-8817, 2019.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv: 02858, 2023.
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning, 2024. URL httLhhtt////// http://axxxvvv.oo httrgg/bb//244.... 05517
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 603):310-345, 2020.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMS in the following aspects: (i) We utilized LLMs the dataset construction process; (ii) We evaluated task performance several LLMs; (iii) We fine-tuned LLM; (iv) We used LLMs to assist us with paper writing slightly.
A.2 RELATED WORK
A.2.1 SOCIAL REASONING IN LARGE LANGUAGE MODELS
Recent studies have explored the extent to which large language models (LLMs) can perform social reasoning, such attributing beliefs (Lu et al. 2025), recognizing emotions, inferring inten- tions (Shapira et al. 2023 Kim et al. 2023). ToM (He et al. 2023), SimpleToM (Gu et al. 2024). and BigToM (Gandhi al. 2023) attempt to evaluate Theory-of of-i (ToM) reasoning in LLMs. They establish crucial frameworks and standards for evaluating high-order ToM reasoning capabilities of LLMs, thereby providing valuable insights and directions for improvement. Nevertheless, their scope limited, as they concentrate exclusively on two kinds of mental states (beliefs and intentions) and are restricted textual contexts, neglecting the assessment of multimodal social cues and reason- ing. Benchmarks like SocialIQa (Sap et al. 2019) and Social Chemistry 101 (Hwang et al., 202T provide scenarios where models must understand social norms or moral judgments. However, these benchmarks rely solely on static text, lacking the multimodal and dynamic context typical of real-life social interactions. These limitations motivate the need for evaluation environments grounded in realistic, multimodal social interactions. Our datasets address this gap by offering richly annotated video scenarios designed to test multi-step reasoning over belief, desire, intent, and emotion in context.
A.2.2 MULTIMODAL SOCIAL INTELLIGENCE AND VISION-LANGUAGE MODELS
Large vision-language models (LVLMs) such as Flamingo (Alayrac et al., 2022), VILA CChnn et al. 2023), and Video-LLaVA (Lin et al. 2023) have shown strong performance in general- video- language tasks. However, their ability to perform fine-grained social reasoning (Fan et al. 2018 2019) remains limited. Studies have found that these models often over-rely on language priors and fail to leverage visual evidence when answering socially grounded questions (Chen al. 2024). Datasets
like Social-IQ 2.0 (Wilf et al. 2023). VLEP (Lei et al. 2020), and SODA (Wang et al.
2023 introduce
social content into video question answering (VideoQA), but few offer systematic annotations of mental-state transitions or causal relations. Additionally, models rarely capture nuanced cues like interpersonal gaze, tone, or posture, which are essential for deeper social inference (Wei al. 2024). Our datasets complement these efforts by providing scalable training set and fine-grained benchmark with explicit labels for mental-state categories and their causal links in socially rich scenarios.
A.2.3 THEORY OF MIND IN VIDEO UNDERSTANDING
ToM - the ability to attribute mental states to others - has been long-standing challenge in AI.
Recent work like Match-and-Hell (Puig et al. (2020), Smart Help (Cao al.) (2024). NoPa (Puig et al.)
2023), and Generative Agents (Park et al. 2023) explores ToM in simulation-basee or scripted agent settings. MMToM-QA (Jin et al. (024) represents step toward wollw ToM inference, offering multimodal video- based questions about beliefs and desires. Fan et al. (202T) model belief dynamics directly from nonverbal video signals. Yet, most of these benchmarks focus on short clips, isolated mental states, or handcrafted settings. IntentQA (Li et al.) 2023) explores intention inference in VideoQA, but it does not capture full mental-state causal chains. Our proposed benchmark Re-Bench explicitly models belief, intent, desire, and emotion along multi-step causal paths, enabling more complete and diagnostic evaluation of ToM-like reasoning in LVLMs under naturalistic, temporally extended scenarios.
17

---

Published as a conference paper at ICLR 2026
Case
Rato
Case
0.4
Case
0.2
0.0
GPT-to
GPT-40 Mri
Gomii Flesh (Fiemes)
Gerin 1.5 Pro (Frames)
Gemiii 1.5 Pro (Vidao)
Idefiex3-8B 10---8844 1aa3-8--Liemmms Ll-Limmma
InternVL 2--288
InterVL2--B
PLLaVA
PLLaVA-34B
PLLaVA-78
Video-LLa LaVA
Guniii 1.5 Flesh (Frames)
(Frames)
Malks22-B 8B
MPLUG-OwlS Owls
Qween-M ML 7B
Intern
Gamini
Figure The ratio of the three cases. Case 1: The model answers a high-level causal question correctly without correctly answering the preceding, simpler event/mental state questions in the sub-chain. Case The model answers basic event/mental state questions correctly but fails at the more complex, high-level causal questions. Case 3: The model fails to correctly answer both the basic event/mental state questions and the high-level causal questions.
A.3 ADDITIONAL. EXPERIMENTS AND ANALYSIS
A.3.1 ANALYSIS OF THE CAUSES OF LOW CONSISTENCY
Inconsistent sub-chains can be categorized into the following scenarios:
(i) Case 1: The model answers high-level causal questions correctly without correctly answering the preceding, simpler event/mental state questions the subchain;
(ii) Case 2: The model answers basic event/mental state questions correctly but fails the more complex, high-level causal questions;
(iii) Case 3: The model fails to correctly answer both the basic event/mental state questions and the high-level causal questions.
Regarding Case 1, high-level causal questions inherently hackable (note that this is inherent challenge in evaluating high-level reasoning, rather than a specific defect in our dataset's design). Models can employ heuristics shorteuts-Sude as relying on common sense or eliminating options that did not occur in the video-to answer these questions correctly. This "hijacking" phenomenon typically goes undetected in other datasets. However, by leveraging causal chains and consistency metrics, we can easily identify it, thereby providing more reliable evaluation.
We analyze the proportions these three cases to determine the primary causes low consistency. As shown in Figure for power-perforance models (e.g., PLLaVA, Video-LLaVA), Case accounts for the largest proportion, indicating severe deficiencies both fundamental capabilities and high-level reasoning. Conversely, for perforerformnce models (e.g.. Gemini, GPT), the combined proportion of Case and Case 2 increases significantly, with Case being the most dominant. This highlights the prevalence and severity of the aforementioned "hijacking" phenomenon. While other datasets might overlook this issue and report overestimated causal reasoning capabilities, our consistency distinguish between flawed/hackable questions (Case
metric accurately detects it allowing 1) and genuine model reasoning deficits (Case and demonstrating the significant value of Re-Bench.
and offers a more reliable assessment, fully
18

---

Published as a conference paper at ICLR 2026
A.3.2 DETAILED ANALYSIS USING FURTHER CLASSIFICATION OF SUBCHAINS
Further Classification of Subchains We further divide subchains (stand for social causality) into six categories. Category I is MS - E, which means that an event is the result of one or more mental states. Category
E, which means that an event is the result of the combination of one or more mental states and one or more events. Category III is E + E, which means that an event the result of one or more events. Category IV is E MS, which means that mental state the result of one or more events. Category V is E & MS MS, which means that mental state is the result of the combination of one or more events and one or more mental states. Category is MS -> MS, which means that mental state is the result of one more mental states. Through division, can delve deeper into discussing large vision-language models' (LVLMs') performance on various types of social causality and analyze their strengths and weaknesses.
MS
We quantify the number of CW QA pairs and CH/W QA pairs according to each of the six categories of subchains, presented in Table 4 Among these, the questions that involve mutual reasoning between MS and E. (i.e., Categories and IV) are the most prevalent. In contrast, questions that employ both E and MS reason about MS (i.e., Category V) are the least numerous and constitute the most challenging subset of QA pairs.
Table 4: Total quantities per category. The table categorizes social causality into six categories: I.
MS - E. II. MS & E E. III. E IV. MS. V. E & MS + MS. VI. MS + MS.
Category
Total Subchains
CW
CH/W
Total
434
434
392
826
II
199
199
170
369
III
96
96
91
187
IV
464
463
403
863
79
79
65
144
VI
134
134
116
250
Total
1406
1405
1237
2642
Results and Analysis We conduct further analysis of the R RR-Bench-DX by categorizing them according to the six subchain classifications outlined in Section [A.3.2] As shown Table we use the same models and settings reported in the main paper to evaluate our dataset. These settings included scenarios without subtitles ("-") and with subtitles ("+ Sub"). The metrics we report are similar to those in the main paper, encompassing both the accuracy of the QA pairs and the subchain consistency (Cons 50) for each subchain. However, we further disaggregated the performance statistics based on the six subchain categories. This detailed analysis revealed several intriguing insights:
Subchain Consistency for Mental State Reasoning:
Reasoning about events generally exhibits higher subchain consistency compared to reasoning about mental states. Specifically, Categories I, II, and III, which involve inferring events from events or mental states, demonstrate higher performance than Categories IV, V. and VI, which involve inferring mental states from events or mental states. Notably, event-to-event reasoning (Category III) achieves the highest accuracy and subchain consistency, significantly outperforming the other categories that involve mental state reasoning (Categories I, II. IV, V, and VI). This indicates that current LVLMs are more adept at factual causal reasoning than inferring mental states, which remains more challenging.
Difficulty in Domman Reasoning: Inferring mental states from events or mental states is typically more challenging than inferring events from events or mental states. In Categories and II, the core of the reasoning process is based on mental states (MS), with events (E) serving as auxiliary information to aid in accurately inferring the events. This auxiliary role of events facilitates correct reasoning, resulting in Categories I (MS inferring E) and II (MS & E inferring E) not exhibiting pronounced increase in difficulty. Conversely, Category V. where both events (E) and mental states (MS) are used to infer mental states (MS), is the most difficult, significantly IV, which involves events (E) inferring mental states (MS). In these categories, the mental states are the primary focus of inference, and the inclusion of additional mental state information introduces
so than Category
19

---

Published as a conference paper at ICLR 2026
Table 5: Additional results on R R The table categorizes social causality into six categories: I. MS - E, II. MS & E - E, III. E - E, IV. E - MS, V. E & MS MS, VI. MS - MS. All values are reported as percentages (without % symbols), and the Cons columns are shown in bold. *+ Ours-FT (SFT)" denotes supervised fine-tuning on R² -DDT, while "+ Ours-FT (RLFT)" denotes
GRPO training on R--FDT
Mathed
III
TV
VI
[Ciee
Vale-LLoA
30:33
13.00
8.00
22.01
22.35
EUR1
12:30
3,20
0.89
21.11
23.13
9:55
22-76
25-25
12.35
23,00
1.75
10.30
0.80
14:13
0031
11:45
0.88
11.05
8.00
8.66
15.77
10:1
1.46
16.5
06.07
13.34
71.36
60.20
9:55
77.28
18:75
13.00
00.09
66.76
66:15
3.90
58.08
$7.26
11:19
13.81
73.45
72.35
3018
12.06
17.00
91.21
66.54
37.08
63.14
28.66
00.23
13.39
76.12
08.77
14.63
Halke livest
67344
15.3%
22.35
11.00
19.41
23.02
83.33
09.13
15.00
02.53
34.78
51.01
04.02
12.66
75.37
02.07
13.50
P.latAmm
18.00
30
0.69
15.64
19:00
0.80
15.00
70.80
3.84
31:10
93.00
3.16
51.65
15.35
1.27
35.30
76.05
0.15
PLLaVA-DR
50b
3880
32.42
2.81
51,10
$7.00
0.50
+1.32
20.67
5.21
33.56
31.12
331
51377
41.36
1.27
15.82
33.45
4.45
12.30
10.00
1,54
91.31
14:07
12.70
73.07
72.15
66.15
5.96
31.44
13:50
85.17
33.97
10.41
$3.55
22.15
92.39
56.52
58.10
10.10
33.59
35.19
51.36
12.51
20.35
13.55
76.45
11.54
96.35
60.52
60.15
7:04
73.95
99.34
15.03
00.00
50.25
66.15
1,59
04.33
53.45
14.55
64.0
71
25.13
16.96
19.41
35.11
91.11
06.15
19
26.75
75.03
63.00
15.19
14.93
60.07
15.13
56.35
9.05
10.30
60,50
1.56
01:33
40.15
0.31
63.01
93.35
13.36
10,99
38.40
2.83
33.97
56.02
14:95
76.35
12.00
11.60
63.31
74.75
19.55
04.32
15:35
04.00
73.85
06.77
18.36
60.35
09.00
15:97
74.39
13.53
01.85
75.33
62.04
14.35
13:00
03.33
15.00
72.15
03.0
1:86
55.63
02.07
14.00
13:00
10.43
$8.34
04.31
96.91
91.01
10.99
79.15
13.55
16.84
91:15
17.86
7511
12.11
6.53
99.13
18:15
50b
66:55
72.35
16.02
1000
12.30
16.85
99.45
64.54
27.16
13.46
05.34
23.41
12.18
12.51
6.30
55.42
02.15
18.15
Cieiin 15 S/bdicttorno
73.03
74
31.97
73.39
76.99
19.30
$3.51
34.77
24113
87.50
$1.13
72.53
58.10
14.11
07.35
35.41
75.05
75.05
13.04
10.77
13.05
13.03
76.36
33.90
07:31
72.41
15.07
36,07
Oceon: 1.3 proclinzed
13:55
79.26
75.1
$2.09
44.70
$3.66
14.00
70.00
11:15
14.24
18.04
74:13
65.42
17.91
75.00
$8.33
11:05
16.61
13.72
11:31
70.97
10.86
51.01
$1.01
10.99
70.77
27.89
11:36
74.36
16.35
09.33
75.01
16:13
45:53
Oeein 1.5 Bodicidon
6106
14.2
63.35
77..
06.51
66:25
79.00
$2.30
73.51
31.09
77.54
95.45
00.13
76/32
14.36
43.35
11:10
00.15
55.33
75.95
73.42
04.00
10.77
16.16
$2.09
70.15
75.94
50-13
15.30
18:46
Cenisi proditio
13.58
$3.35
34.79
$1.01
$3.00
35.65
$7.53
$1.33
$8.38
05:31
37.43
43.34
$6.81
75.50
25.33
76.13
76.14
36.57
CPG-4
81:14
8549
07.31
57766
50.00
$2.76
31.51
20111
64.06
07.41
75.03
06:90
19.01
13.38
11.90
31.65
$7.31
75.05
$1.79
Oesin prodieeee
56-45
K5.70
34.17
91.16
30.41
68:13
74.15
05:33
16.01
80:00
24.11
01.61
41:90
18:75
6350
16.33
17.11
75.30
15.29
05.85
82.35
00.65
15.79
16:46
67.14
31.77
14,88
12.31
15:19
73.39
72:41
16:27
Q1e82-V1-18
Dan (BLM
01.31
54.64
12.41
10.50
31.46
05.63
19011
19:31
31.13
34.11
77-22
90.77
43.84
23.30
58.00
Dan- (5P1)
90.06
9032
58.29
19:6
92.55
64.92
96.33
96.51
11.88
91.14
85.61
06.99
16.75
06.24
$1.50
54.18
96.21
63.00
complexity that leads to substantial decline in model performance. This further underscores the current limitations of LVLMs in inferring mental states.
A.4 MORE DETAILS OF R²-BENCH
A.4.1 VALIDITY OF ANNOTATED CAUSAL CHAINS
In our dataset, each causal chain is approved by three people: the participant who designed the QA pair and two experts for verification (one of whom annotated the causal chain). The participant submits the QA pair and the textual description of the reasoning process behind it, establishing the main content of the causal chain. After being verified by two experts, the filtered data is approved by all three people. The expert in charge of annotation only appropriately expanded and completed the reasoning process without affecting the essence of the causal chain. Also, the annotated causal chains are verified by the other expert. If they fail to pass the verification, the two experts will consult and revise them together. Therefore, we can consider the annotated causal chains to be widely acceptable.
A.4.2 STATISTICAL ANALYSIS
Video Statistics. R3-Bench contains 312 videos. Figure 7a shows the distribution of video durations. All videos are no longer than 180 seconds, with an average duration of 65.8 seconds. The increased video length amplifies the challenge of social reasoning, more difficult, and causal chains may become longer with more steps, involving a greater variety of events and mental states and requiring tracking of all dynamic changes of all states throughout the video.
identifying relevant social cues becomes
Causal Chain Statistics. Table 6 presents the statistics of causal chains in our R Ri-Bench-DX It includes 347 causal chains composed of 2198 nodes and 1406 single-step subchains. These nodes categorized into 997 Event nodes, 321 Belief nodes, 361 Intent nodes, 42 Desire nodes, and 477
20

---

Published as a conference paper at ICLR 2026
R BeBchh-DX
R Sr-BBech-Hard
Videos
EU
MSE
CW
CH/W
Overall
312
997
1201
1405
1237
4840
316
Nodes
Chains
Subchains
Event
Belief
Intent
Destre
Emotion
Overall
347
1406
997
321
361
42
477
2198
Table Statistics R3-Bench
(a) Video Length
(b) Causal Chain Length
(c) Question Length
(d) A&O Lengths
Figure 7: Statistics of
R R--Bench. A&O means Answer and Incorrect Options
Emotion nodes. Aside from Desire, each mental state category has over 300 nodes, providing ample cases to assess specific mental states. In total, 1201 mental state nodes indicate a rich presence of mental state dynamics. The lengths causal chains range from of 3.3 steps, as shown in Figure 7b Over 60% have less than three steps, suggesting that the videos require complex and in-depth reasoning.
steps, with an average length
QA Statistics. As detailed in Table 6 there are 4,840 QA pairs in R²-Bench-DXX, and there are 316 QA pairs in FF-BBech--Hrc Figure 7c and Figure 7 show the distributions of question, answer, and incorrect option lengths among R Re-Benc-DXX. The average question length is 13,2 words-longe than many popular VideoQA datasets. Answers and incorrect options average 14.4 and 12.8 words respectively, making the distractors similar in length to the correct answers and thus more challenging.
A.5
MORE DETAILS OF R3-FDT
A.5.1 MORE DETAILS OF INFORMATION ALIGNMENT
The algorithm extract annotations from existing movie datasets shown in Algorithm I We align the dialogues recognized by Whisper with the time-stamped annotations provided in MovieNet to identify the temporal boundaries of each clip within the full movie. Using these identified temporal boundaries, then extract the relevant data from both MovieNet and MovieQA, including question-answer pairs, action tags, place tags, and scene descriptions.
Algorithm 1 ExtractAmootation
Require: Video Recognized Dialogue D, MovieNet Annotation File F, MovieQA Set Q Ensure: Clip time (ta, t,), MovieNet Annotation M, MovieQA Annotation Y
1: S +- ExtracttSubtitle[[f..
2: Sa +- ExtractDDialoouue((..i)
3: E, GetEmbbeddings((5)
4: Ed < Get
5: (is, ie) MatchIndex Ea)
6: if i, None or i, = None then
7:
return None
8: end if'
(tt==Gettmmetttps((s... 10: M + Mat t..
9:
Y MatchMoviecAA(
12: return (4,,4)) M,
21

---

Published as a conference paper at ICLR 2026
Videos
EU
Belief
Desire
Intent
Emotion
CW
Overall
2812
14749
5645
220
1552
4532
14831
41529
Table Statisties of R²-FDT
1300
Odies
4000
6000
6000
2030
100
250
250
(a) Video Length.
(b) Question Length.
(c) A&zO Lengths.
Figure 8: Statistics of R²-FDT. A&O means Answer and Incorrect Options
Movie scripts include content beyond the target clips. To avoid using such unrelated information, we extract script segments based on aligned dialogues. As shown in Algorithm we first identify character names as anchors to locate dialogue in the script. We then parse the script S into parts: scene descriptions and dialogues, yielding structured script S', Finally, we match the dialogues recognized by Whisper with those in S' and extract the corresponding script segments aligned with each movic clip.
Algorithm 2 ExtractSeript
Require: Script S, Movie ID I, Recognized Dialogue D
Ensure: Matched Script S.
C GetCharacters())
2: S + ParseScript($ (S, C))
3: S. + Grounding(S" D)
4: if Len(S.)
then return S.
5:
6: end if
7: return None
A.5.2 STATISTICAL ANALYSIS
Video Statisties. Our RD-FDT includes 2812 videos. The average video duration is 128.76 seconds and the median duration 131 seconds. Figure Sa shows the distribution video durations.
QA Statistics. Figure 8b and Figure &c show the distributions of question, answer, and incorrect option lengths of RRFF... The average question length is 9.55, the average answer length is 11.32 and the average length of incorrect options is 9.88.
A.6 MORE DETAILS OF EXPERIMENTS
A. EVALUATION CONFIGURATION DETAILS
Our evaluation is based VLMEval (Duan et al. 2024) repository, open-source toolkit for assessing LVLMs across multiple benchmarks without extensive data preparation. By extending VLMEval, incorporates
dataset new benchmark.
Our task is formulated as a multiple-choice VideoQA problem. Models receive the raw video or selected frames, along with question and five options, and must select the correct option. We enforce output formatting constraints and determine selected option through exact matching, ensuring reproducible evaluation results. In Section 4.1
present specific evaluation metrics.
22

---

Published as a conference paper at ICLR 2026
A.6.2 GRPO TRAINING DETAILS
We implement GRPO training based on verl (Sheng al. 2024). During training, input 8 frames to Qwen2-VL-7B, each with a resolution of 280 x 280, and input the dialogue recognized by Whisper the model. We use a rule-based reward function as following:
the response meets the format requirements otherwise
(4)
2
if the answer is correct
Toce
-2 otherwise
(5)
r r= + Tace
(6)
We denote the reference model Tref, the old policy model as Teo. and the policy model me-
For each question q, we sample a group of responses o = for o2,... ON) and compute a group of rewards r {TI, (2 ... J. For each Ti E the corresponding advantage A; is computed as:
mean(r)
A
std(r)
Then the policy model is updated according to the following optimization objective:
(o: (GRPO(0) = (o))
Toma (0 [0)
(o: 14)
(o; 9)
min
Ar, clip
- Ai)
BBD KL (TellTres))
Tar (os 4)
Teaa (0: ())
Tref (0; 14)
Tre (0; 1.
DEL (Te Tre])
llog
Thg (0: q)
To (o, 9)
(8)
Both the video context and the dialogue are required to answer Be-BecchDDX and R Re-Bench-Hard and questions in Social-IQ 2.0 correctly. Therefore, when evaluating on these datasets, we also provide the dialogue to Qwen-2-VL-7B. Previous studies mostly input only the video context to models. When evaluating on IntentQA, we retain the marker for the dialogue section within the prompt but the dialogue content to "not provided", which is consistent with its "+Sub" setting. We provide the model with 16 x 360 x 640 video context when evaluating. We report the training parameters and visualizations in the supplementary material.
A.6.3 SFT DETAILS
We utilize SWIFT (Zhao et al. 2024) to implement SFT and train Qwen2-VL-7B using the same 13k subset. We convert 10% of them into an open-ended format (i.e., removing the options and requiring direct answers). We provide video frames with the model, with each frame having total pixel resolution of 280 x 280. The evaluation settings for this part are identical to those used for the RL fine-tuned model.
A.6.4 DETAILS OF HUMAN STUDY
For R He--BBnc-Haaee Hard, we invite human subjects to answer all questions. For Rd- -e-neehh-D we sample QA pairs corresponding 43 causal chains for human study. QA pairs derived from the same causal chain are interrelated. To avoid potential shortcuts caused by human memory, we divide all QA pairs corresponding to a single chain into multiple groups based on the following principles: (i) All node QA pairs constitute one group. (ii) For causal QA pairs: Within a group, the node corresponding to the "correct option" of each QA pair must be distinct from the nodes corresponding to the "question" and "correct option" of any other QA pairs in the same group. These principles ensure that no shortcuts exist within the same group. For a single chain, a human subject only allowed to answer group of QA pairs.
23

---

Published as a conference paper at ICLR 2026
A.6.5 COMPUTE RESOURCES
All evaluations were conducted on machines equipped with NVIDIA A100 GPUs. For models with fewer than 10B parameters (e.g., Idefics2-8B), we used a single A100 GPU. For mid-scale models in the 20-40B range, such PLLaVA-34B and VL2-66B we used two A100 GPUs. Large-scale models with over 70B parameters, including Intern VL2-76B, were evaluated using four A A100 GPUs. During the training stage, all models were trained using four NVIDIA A100 GPUs.
24


Reviews:
[{"review_id": "QwVvb0X60h", "reviewer": "Reviewer_curi", "summary": "The paper introduces the video social reasoning task with fine-grained mental–physical causal chains. The paper releases an evaluation benchmark with verified Multiple choice-QA pairs built from annotated chains and a training set generated via a movie-script alignment and LLM pipeline. The paper introduces chain and subchain consistency metrics, evaluate many LVLMs and humans, and show GRPO fine-tuning of Qwen2-VL-7B improves accuracy and consistency.", "questions": "- how does the performance correlate with the chain length?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Task novelty: Causal-chain–grounded evaluation of belief, intent, desire, and emotion in video; chain and subchain consistency move beyond single-QA accuracy.\n- Dataset annotation quality: Human-verified chains and QA, with explicit rules and hallucination filtering\n- Significance: Reveals large accuracy-consistency gaps in SOTA LVLMs and provides training data that measurably improves a strong open model.\n- Strong experiment results with ablation studies and generalization tests.", "weaknesses": "- Human study mismatch. Humans answer only one QA per chain, so their consistency is “severely underestimated,” complicating human–model comparisons. Remedy: redesign human protocol to mirror model setting or normalize metrics accordingly. \n- LLM-in-the-loop biases/leakage. GPT-4o is used to generate and self-correct chains and QAs; Gemini is used for hallucination detection. This can imprint model priors and evaluation artifacts. It'll be good to do ablation studies to assess its sensitivity to the design choice.\n- Lack of uncertainty estimation. Many tables lack confidence intervals and seed variance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhnBfSkhfK", "reviewer": "Reviewer_DYRn", "summary": "The paper introduces R3-VQA, a new video social-reasoning suite comprising (i) R3-Bench—an evaluation benchmark with fine-grained mental–physical causal chains and (ii) R3-FDT—a large-scale training set generated by an automated pipeline. The benchmark defines four QA types—Event Understanding, Mental State Estimation, Causal-Why, and Causal-How/What—generated from expert-annotated chains and verified by the domain experts. Reported scale: around 0.3k videos / 5k MC-QAs for R3-Bench and, for R3-FDT, about 3k videos / 68k MC-QAs (Table 1); elsewhere, the text states 2,812 videos / 41k QAs after quality filtering as well. Evaluations of many LVLMs demonstrate a stark gap between single-question accuracy and chain/subchain consistency (all QAs from a chain must be not wrong), highlighting fragmented reasoning. Fine-tuning Qwen2-VL-7B with GRPO on R3-FDT yields sizable gains on R3-Bench and transfers to Social-IQ 2.0 dataset / IntentQA dataset.", "questions": "1. How do OE and consistency scores change if the judge-LLM or prompt is varied? Any agreement with human raters on an OE slice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Explicit belief/intent/desire/emotion nodes and chain/subchain QA generation tie items to causal structure rather than one-off facts.\n2. Two-stage expert checks for QAs and chains; a clear human-in-the-loop pipeline to keep the ambiguity low.\n3. Chain and Subchain Consistency expose fragmented reasoning hidden by the average accuracy; formal definitions are offered.\n4. Large consistency-accuracy gaps ( for example, GPT-4o 82.64% overall vs 25.36% chain consistency; Gemini 2.5 Pro 86.34% vs 36.60%) and dimension-wise weaknesses.\n5. Automated R3-FDT pipeline (alignment, movie scripts, Gemini filtering, GPT-4o) plus GRPO improves Qwen2-VL-7B model ( for instance, plus 32.00% performance on R3-Bench-DX; and plus 9.81% on R3-Bench-Hard).", "weaknesses": "1. The OE scoring depends on an LLM judge. What is more, the sensitivity to prompts and judgment choice is not reported comprehensively. Frame or frame per second budgets differ comparatively over models. This confounds the absolute rankings to a considerable extent.\n2. 3k/68k for R3-FDT is stated in Table 1; however, the text reports 2,812/41k later. It is better to clarify the final counts and which subset is utilized for the GRPO algorithm to make the paper clearer.\n3. R3-FDT depends on movies and LLM-generated chains/QAs. The risk of stylistic bias or leakage still remains; in addition, the human IAA and leakage audits are not detailed.\n4. Even though consistency is measured, wall-clock latency and user-utility vs delay are not evaluated directly.\n5. The human chain or the subchain consistency is left out (with justification deferred to the Appendix), which limits context for the new metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFmHMfw9m2", "reviewer": "Reviewer_aZqY", "summary": "This paper introduces R3-VQA, a benchmark (R3-Bench about 5K QAs on diverse videos) and training dataset (R3-FDT about 40k QAs on movie videos) for evaluating video-based social reasoning. The dataset annotates causal chains linking beliefs, intents, desires, and emotions to observable actions.  Evaluation of 10+ SOTA models shows that they achieve high accuracy on individual questions but fail to maintain consistency across related questions in the same causal chain. The authors propose chain consistency and subchain consistency metrics to measure this. Training Qwen2-VL-7B with GRPO on R3-FDT improves performance by +13% on R3-Bench-DX and +6% on external social reasoning datasets, suggesting that causal structure provides useful training signal for social reasoning.", "questions": "* MMToM-QA (2024) also annotates mental states in video. How does causal chain depth compare quantitatively? \n*  What is performance using standard SFT (not GRPO) on the same 13k training examples? \n* Can you provide error analysis showing whether model failures correlate with System 1 (reactive/emotion-driven) vs. System 2 (deliberative/belief-driven) scenarios? \n* Why are Cohen's kappa or Fleiss' kappa not reported for causal chain annotations? What was the disagreement rate between the two expert reviewers before consensus?\n*  What % of R3-Bench videos are \"real-life\" vs. curated (ads/short films)? Does performance differ by video type? \n* Can you provide examples of Causal-Why questions where multiple answers seem valid? How were conflicts resolved?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Systematic annotation of mental-physical causal chains in video with multiple mental state types; consistency metrics effectively expose gap  in SOTA models\n* Chain consistency improves dramatically (5.48% → 19.63%) after training, demonstrating models learn coherent multi-step \"social\" reasoning \n* Rigorous construction: expert-verified annotations across multiple stages; comprehensive evaluation on 10+ models with accuracy and consistency metrics\n* Strong competitive results: 7B model beats Gemini 1.5 Pro on hard set and approaches GPT-4o on standard benchmark; +6% generalization to external datasets (Social-IQ 2.0, IntentQA) also convincing\n* Scalable automated pipeline (ARGUS) creatively leverages movie metadata to generate 41k training QAs; a valuable dataset for the community", "weaknesses": "* Related work somewhat underdeveloped: Dismisses MMToM-QA (2024, same year), Hi-ToM, SimpleTOM, and CausalChaos in single sentences without evidence or direct comparison—makes contribution difficult to situate relative to this work.\n* Causal theory incomplete: Treats emotions as generic mental states rather than motivational drivers; ignores System 1/2 distinction between reactive (emotion-driven) and deliberative (belief-driven) reasoning. Also missing some ToM citations. Overall, the causal chains are reasonable but simplistic compared to cognitive processing.\n* Human validation weak: No inter-rater agreement metrics. Consensus among annotators is not clearly discussed. It would be great if all the annotator information is provided in the dataset because social/affective labels are naturally ambiguous. Consistency in annotation is somewhat misguided.\n* Domain characterization insufficient: Cross-domain gap acknowledged (movies → YouTube) but not quantified. No breakdown of R3-Bench by video type and no evaluation on genuinely real-world social scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwEbrPL6UE", "reviewer": "Reviewer_G5Ui", "summary": "The authors introduce a new dataset, $R^{3}$-VQA, to address the challenge of teaching AI to infer complex social and mental states from video. The dataset contains two parts: $R^{3}$-Bench, a high-quality video QA benchmark with fine-grained manual annotations for mental states (belief, intent, desire, emotion) and the multi-step causal chains that connect them to physical events; $R^{3}$-FDT, an training dataset augmented from existing datasets using a similar automatic framework. The authors showed that state-of-the-art multimodal LLMs fall short of human performance on $R^{3}$-Bench, and finetuning a model on $R^{3}$-FDT with GRPO yielded improvements across datasets.", "questions": "- Could the authors please elaborate on the construction of $R^{3}$-Bench-Hard? The paper states it was \"sourced from the winning submissions of a social reasoning challenge\", but what were the specific selection criteria? What attributes (e.g., causal chain length, type of mental state, presence of deception) make this subset quantifiably \"harder\" than $R^{3}$-Bench-DX? Given its small size, providing a few qualitative examples that directly compare a \"Hard\" question to a \"DX\" question would be beneficial.\n- The novel consistency metric is appreciated, but its interpretation is difficult. The criterion -- where a chain is \"consistent\" only if all associated questions are answered correctly -- seems overly strict and may be the reason for the low consistency scores. The authors omit a human baseline for this metric, citing concerns about human memory. However, a baseline, even if imperfect, would be a good reference point to contextualize the models' low consistency scores. Could the authors provide this consistency score human baseline, or discuss why a modified human study was not feasible?\n- The \"high accuracy, low consistency\" paradox  is ambiguous. The consistency metric doesn't distinguish between two very different issues: 1) A model answers a high-level causal question correctly without correctly answering the preceding, simpler questions in the chain. This would suggest the high-level question is flawed or hackable with a heuristic, not by causal reasoning. 2) A model answers basic event/mental state questions correctly but fails at the more complex, high-level reasoning questions that come later in the chain. This indicates a genuine model failure. Can the authors clarify whether the current metric distinguishes these two cases? A more granular analysis or metric seems necessary to determine if the low consistency is due to flawed questions (Case 1) or genuine model reasoning deficits (Case 2).\n- Nit: there's an overlap of Table 1 on page 3\n\nOverall, while I lean towards acceptance due to the dataset's demonstrated utility for fine-tuning, I have the above reservations about the saturation of the evaluation set and whether the benchmark robustly measure genuine social understanding.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- $R^{3}$-Bench contains rich, manual annotation of social reasoning chains. It also has decent scale (312 videos, 4,840 QAs) for an evaluation set \n- While $R^3$-FDT has no human-in-the-loop verification, using it for RL finetuning yielded significant improvements across datasets (including Social-IQ and IntentQA). The pipeline also seems scalable and adaptable.\n- The difficult subset $R^{3}$-Bench-Hard showed significant gap between AI and human accuracy, demonstrating room for improvement.", "weaknesses": "- $R^3$-FDT is automatically generated, and the paper lacks crucial details on its verification. A \"Hallucination Detection\" step is mentioned, but the exact criteria for filtering, the specific prompts used, and the rejection rate (i.e., how many QA pairs were filtered out) are not reported. This makes it difficult to assess the final quality and potential noise level of the training data.\n- The main evaluation set, $R^{3}$-Bench-DX, appears to be approaching saturation. The top-performing models, such as the finetuned Qwen2-VL-7B and Gemini 2.5 Pro, are close to human accuracy performance. This limits its long-term utility as a challenging benchmark.\n- The \"Hard\" subset, which shows a much larger performance gap, is very small, containing only 316 questions. This limited scale and diversity make it difficult to rely on as a robust and comprehensive evaluation set for such a complex task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)