HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
Authors: Keliang Li
Hongze Shen
Hao Shi
RuiBing Hou
Hong Chang
Jie Huang
Chenghao Jia
Wen Wang
Yiling Wu
Dongmei Jiang
Shiguang Shan
Xilin Chen
Venue: ICLR 2026 Poster (2026)
Keywords: human-centric vision
multi-modal large language models
evaluation
video reasoning
Primary area: datasets and benchmarks
TL;DR: We introduce HumanPCR, a new benchmark to evaluate the perception, comprehension, and reasoning abilities of Multimodal Large Language Models (MLLMs) across diverse human-centric scenes.

Abstract: The aspiration for artificial general intelligence, fueled by the rapid progress of multimodal understanding, demands models to understand humans in diverse and complex scenarios, as humans manifests intelligence and embody the world. We propose HumanPCR, an evaluation suite for probing MLLMs’ capacity in human-centric visual contexts across three hierarchical levels: Perception, Comprehension, and Reasoning (denoted by Human-P, Human-C, and Human-R, respectively). Human-P and Human-C consist of over 6,000 multiple-choice questions evaluating 34 fine-grained tasks covering 9 essential dimensions. Human-R presents a manually curated challenging video reasoning test that requires integrating multiple visual evidence, proactively extracting implicit context beyond question cues, and applying human-like expertise. Each question includes human-annotated Chain-of-Thought (CoT) rationales with key visual evidence to support further research. Extensive evaluations on over 30 state-of-the-art models exhibit significant challenges in human-centric visual understanding, particularly in tasks involving detailed space perception, temporal understanding, and mind modeling. The analysis of Human-R further exposes a critical failure in reasoning: models struggle to proactively gather necessary visual evidence, instead showing a faulty reliance on query-prompted cues, with advanced techniques offering only marginal gains. We hope HumanPCR and our findings will advance the development, evaluation, and human-centric applications of multimodal models.

Published as a conference paper at ICLR 2026
HUMANPCR: PROBING MLLM CAPABILITIES IN DIVERSE HUMAN-CENTRIC SCENES
Keliang Li Hongze Shen' 12,3+ Hao Shi!. Ruibing Hou', Hong Chang Jie Huang 1.3 Wen Wang-.s, Chenghao Jial.3, Yiling Wu?, Dongmei Jiang", Shiguang Shan 1.3, Xilin Chen Institute of Computing Technology, Chinese Academy of Sciences "Peng Cheng Laboratory
University Chinese Academy of Sciences [kel aaliaag..1 1i, hongze. shen Sceene0v @vipl. ict. ac. chonghonge at..accc.m
ABSTRACT
The aspiration for artificial general intelligence, fueled by the rapid progress of multimodal understanding, demands models to understand humans in diverse and complex scenarios, as humans manifests intelligence and embody the world. We propose HumanPCR, an evaluation suite for probing MLLMs' capacity in human- centric visual contexts across three hierarchical levels: Perception, Comprchension, and Reasoning (denoted by Human-P, Human-C, and Human-R, respectively), Human-P and Human-C consist of over 6,000 multiple-choice questions evaluat- ing 34 fine-grained tasks covering 9 essential dimensions. Human-R presents a manually curated challenging video reasoning test that requires integrating multi- ple visual evidence, proactively extracting implicit context beyond question cues, and applying human-like expertise. Each question includes numan-annotate Chain-of-Thougee (CoT) rationales with key visual evidence to support further re- search. Extensive evaluations on over 30 state-of-the-ar models exhibit significant challenges in human-centris visual understanding, particularly in tasks involving detailed space perception, temporal understanding, and mind modeling. The analy- sis of Human-R further exposes critical failure in reasoning: models struggle proactively gather necessary visual evidence, instead showing a faulty reliance on query prrempe cues, with advanced techniques offering only marginal gains. We hope HumanPCR and our findings will advance the development, evaluation, and human-centric applications of multimodal models.
INTRODUCTION
The rapid advancement of Multimodal Large Language Models (MLLMs) has shown remarkable potential in understanding diverse contexts (Zhang al. 2024c Team et al.) 2023 Hurst al. 2024; Bai et al. 2025 Wang et al. 2024p. This progress fuels the aspiration toward artificial general intelligence, where key prerequisite lies in the ability to understand humans in diverse, complex, and dynamic contexts, as human behavior inherently reflects intelligence as well as the complexities of the world (Grauman et al., 2022; Jahangard systematically investigate how well MLLMs understand humans across critical aspects of perception, visual understanding scenarios.
2024 Caba Heilbron
2015). In this work, we
comprehension, and reasoning in diverse
Guman-centrin visual understanding (Tang et al.) 2023 Ci et al. 2023 Li et al. 2025b) remains a fundamental challenge artificial intelligence. However, current MLLM benchmarks provide limited assessment of such capabilities. They either isolate narrow tasks, such as action or facial recognition (Qin et al.) 2025) Mangalam et al.) 2023) (Salehi et al., (2024), or, when adopting broader scopes, overlook intricate yet crucial aspects such as gaze and contact, while reporting only coarse- grained scores (Yue et al., 2024 Fu et al.) 2024 Xu et al. 20233 Zhou et al. (024a)). Consequently, they lack the necessary probing power to rigorously evaluate MLLMs nuanced capabilities in
"Equal contribution.
1 "ooresponniig Author.

---

Published as a conference paper at ICLR 2026
Percaption
Zal to
Proondate
Comprehension
Greellgge Roard
For
Margle
Ktpp Miik
Reasening
Lace Lortle
Topfee
Aniastin
Recossorry fic Recognition
fie chroing
Highy SEAT/
Pedty Drientation
Emotiea
Bady Posture
cul andtn agga thx
Kncarledge Eased Betton le
bant buttoa losar
ci thai dinggaad
Interaction
FALmt hond hubl potille
di de budding buttoda
Procedure Rependesce E onlic Tarsaing oue and Ro bay
Muiti Euidence Re as sonin
St sor the vonan placis the price the with plo
Tawce CINC www.eoaeerrooooooooooooooooooo vacu and th or baather new Parn the phane
Figure I: Illustration of HumanPCR, consisting of 34 tasks spanning 9 dimensions across Perception, Comprehension, and Reasoning Levels. It features comprehensive human-centric visual understand- ing abilities coverage, and proactive visual reasoning based on multiple evidence.
human-centric scenarios, thus providing limited guidance for future research and applications. A more critical gap lies in the assessment of reasoning. Unlike humans, who naturally synthesize multiple visual cues in reasoning, current task-specific or fragmented benchmarks rarely challenge models to perform multi-evidence reasoning (Zhao et al. 2025 Hu al. 2025 Chen et al.) 2024a]
Fu et al.) 2024 Lu et al. (202)). Although some recent video benchmarks have featured intricate reasoning questions(Cheng et al.) 2025 [huu et al.) [0255] Cai al., (025), they often cannot necessitate sophisticated visual evidence demand. As discovered by our analysis in Figure 2 two critical reasoning faculties remain overlooked: 1) the ability of integrating multiple, disparate visual evidence to achieve coherent understanding, and 2) the ability of proactively seeking implicit visual cues. As a result, reasoning within complex and dynamic human-centric contexts remains a significant, open challenge.
To bridge these critical gaps in evaluation, we introduce HumanPCR, a comprehensive evaluation suite designed to meticulously probe the numan-centii visual understanding of MLLMs. HumanPCR is structured along a hicrarchical taxonomy, perception (Human-P), comprehension (Human-C) and reasoning (Human-R), as illustrated in Figure T To enable fine-grained probing, Human-P and Human-C feature a large-scale dataset of over 6,000 image- and video-based QA pairs, assessing 34 tasks that span 9 dimensions from individual attributes to spatio-temporal dynamies. Moreover, Human-R introduces a unique challenge through a manually curated, open-ended video reasoning benchmark. Sourced from 11 diverse human-related domains, it compels models to integrate multiple, disparate visual evidence and proactively seek implicit visual cues beyond what is explicitly prompted. To support further research, each question in Human- R is augmented with expert- annotaedd Chain-of- Thought (CoT) rationales (Wei et al. 2022) that detail all key visual evidence.
large suite of open-source and proprietary models is benchmarked on HumanPCR. Our analysis reveals several key findings. First, existing MILMs face significant challenges in visual understanding and expose inherent limitations in detailed space perception (Yang et al., 2024b).
temporal understanding (Fu et al. 20244, and mind modeling (Rezaei et al.) 2025 Jin et al. 2024). Second, Human-RR R highlights models' struggles with multi-evidence reasoning across diverse human scenes. A substantial proportion of errors arise from their reliance explicit query query-guided cues, failing to proactively seek implicit visual evidence. Third, merely scaling visual contexts offers little
gains for Human-R (Team et al. 2023 Hurst et al. 2024
Bai al. 2025
Wang al., 2025b Shen
2

---

Published as a conference paper at ICLR 2026
al. 2024). emphasizing the need for more precise visual context perception. Conversely, reasoning- enhanced models like o3 (OpenAl (025) reduce missed proactive evidence and achieve consistent improvements. We hope that HumanPCR will serve as crucial tool to accelerate the development of more capable ML.MMs and facilitate their adaptation to diverse human-related applications.
2 RELATED WORK
Multimodal Large Language Models (MLLMs). Evolving from LLMs, MLLMS now process
diverse modalities including image sequences, video, and audio (Liu al. 2023 Li et al. 2024a
Lin et al. 2023 Borsos et al. (023). Recent models can handle dynamic resolutions and long contexts Bai et al.
2025 Zhu et 2025a). To manage long videos within limited context windows,
common strategies include efficient content extraction via frame selection (Tang et al., 2025 Huang
al.
2025; Ye et al. (025) or token pruning (Wang et al. 2025b ZZhng et al. 2025 Shen et al.
2024). Concurrently, the rise of models with powerful intrinsic reasoning capabilities (Guo et al.
2025 Jaech et al. 2024| Comanici et al. 2025 OpenAl 2025) has inspired efforts to leverage these
cognitive strengths for more advanced visual understanding (Feng et al. 2025
Xu et al.
2024 Wang
ct al.
2025a
et al. 2025a).
Benchmarks for MLLM Evaluation. Evaluating MLLMs, particularly in comann-centrc contexts, is significant research focus. Many benchmarks target specific human-relaan related tasks like motion
analysis (Hong et al. 2025 Feng et al. 2023). face recognition (Qin et al. [025 Pham et al. 2024).
domain-specific actions (Salehi al. 2024
Cui et al. 2023 Plizzari et al. 2025p. while they have
concluded that MLLMs still have substantial limitations in nuanced human understanding (Rezaci
ctt al. 2025) Mangalam al. 2023 Zhou et al.
2024b et al. 2024c). Broader, general benchmarks also include human-centrr tasks but often lack the structured taxonomies needed for systematic, fine-grained ability diagnosis (Xu et al. 2023) Fu et al.] [2024] Li et al., 2023). On the reasoning front, benchmarks have grown in complexity, moving from image-based exams (uue et al. 2024
Chen et al. 2024a) to challenging video-based scenarios (Song et al. 2025 Cai et al. 2025 Cheng
et al.) 2025 Zhu et al.] 2025bb. However, they often focus on knowle nowledge-bassed reasoning from
instructional videos (Hu et al. 2025) or repurpose existing QA pairs (Han et al. 2024 Qi et al. 2025 which may not adequately test the synthesis of multiple. disparate visual cues. Our HumanPCR. features a hierarchical taxonomy for detailed ability diagnosis and assessing the critical skills of integrating multiple pieces of visual evidence and proactively seeking implicit information in complex, human-centric videos.
3 THE HUMANPCR BENCHMARK
3.1 TASK ARCHITECTURE AND DATA CONSTRUCTION
We introduce the HumanPCR benchmark to evaluate how MLLMs understand humans in real-world scenarios. The design of HumanPCR is motivated by the need for probing fine-grained model capability essential to downstream applications in buman--entric visual understanding. To this end, critical principle to construct sufficient number of tine-grained tasks with comprehensive coverage and low redundancy. And each task should be supported by data sources as rich and varied as possible. To do so, we survey a wide range of human-centric perception and understanding works to first define tasks, then match them with rich and varied datasets. This iterative approach, grounded in diverse data from daily to professional scenes, mitigates single-source bias and ensures broad capability coverage (Figure 3. Task definitions and sources are provided in Supplementary Materials. The taxonomy is briefly introduced as follows:
Level 1: Perception evaluates visual recognition across 5 dimensions and 17 tasks: (1) Spatiality: perceiving existence of people, objects, and their spatial relations; (2) Posture:
recognizing physical position and orientation of body parts, hands, and gaze; (3) Appearance: identifying human appear- ances, including inherent attributes and attirement; (4) Contact: recognizing detailed interaction
regions between people and objects, or themselves; (5) Identity: recognizing people's identity.
Level 2: Comprehension assesses visual concepts comprehension, from 4 dimensions and 17 tasks, based commonsense domain-specific cues: (1) Behavior: understanding human actions and
3

---

Published as a conference paper at ICLR 2026
Shortcut by Refered Single Evidence
Rejected Anno. Human-R
Duestion: How mony End codks are aoove the Freplico the and this video?
Visall Evidanca
Question Why dees the woman
Bosuwer:
Reulew::
Question 21 What does perfor??
Reulew::
Mubi evidence recogrttion
aeasoring
Question St Can the sled dkgg be replaced th Shite
Beulew: indeperdence
Reasoning Proactive and Multiple Evidence
Duestion: Dons
3including
would the
15-
Aeg Preuctuv Vidual Eridanue
Drisucc
Koho
while (Dihee was feeding the doge?
Chain-of- Thought: Alle feeding dage, the women dlerrr the vied urides oul ancemmolll horset coting that
Beswwer: Yee she ieree they come was feeding
propared arriess the yerd, sistaches the doge to the slds, and usos for toavll Th indicatos that the woman prepared the ded adxtroo for thom, she hust known that they wourd lour came while was leed ng dogs.
25-
20
15-
0.5
00
Lybecch Viace stap
Maso
t00
Figure 2: Illustration of multiple and proactive evidence reasoning in Human-R. (Left) Examples from Video-MME and Human-R. Clips of proactive and referred visual evidence are highlighted. Context matching by heuristics could resolve sparse referred evidence and bypass comprehensive reasoning, so it is crucial to assess reasoning over multiple and proactive evidence. (Right) By rigorously filtering precisely diagnose model's capabilities in video reasoning with multiple and proactive evidence, an area where existing benchmarks fall short.
annotations that fall below the required reasoning complexity, Human-R can
body movements, such as gestures and emotions; (2) Procedure: thoroughly understanding long-term activities, including underlying intentions and dependence among action sequences; (3) Relation: analyzing relations, roles and differences among individuals; (4) Scene: interpreting group dynamics or human activities within broader contexts.
Level 3: Reasoning examines whether models can integrate continuous, tightly coupled human dynamics within complex scenes for reasoning. We contend that the evaluation should satisfy three criteria: (1) Visual Complexity: questions should require sufficient visual evidence and exclude redundant content, going beyond simple concept retrieval; (2) Reasoning Necessity and Diversity: questions should engage diverse reasoning chains rather than be limited to few reasoning patterns; (3) Proactivity: questions should demand proactive extraction of visual evidence over the abundant c rather than relying solely on the referred evidence in the question.
For the Reasoning level (Human-R), we found that videos from public academic datasets are not diverse enough and usually short. Therefore, we defined 11 domains ranging from daily life to professional scenarios, as detailed in Figure [ Web videos were collected as supplementary data source to populate these domains. They were pre-filtered via domain-relevan tags and then manually reviewed before annotation to guarantee both content richness and safety.
3.2 QA ANNOTATION PROTOCOL
The prompt and other details of annotation process are provided in Appendix B
Human-P and Human-C. Benefiting from the data collection, we leverage annotations from existing datasets to efficiently scale up QA pairs without compromising data quality or diversity. Templates- and LLM- based generation used to create Culti-Choice questions and options based on dataset annotations, as illustrated in the lower-left panel of Figure 3 Moreover, for under-explored tasks in
In this work, visual evidence refers information conveyed in images or videos, such as instance attributes or visual relationship, which serve as information units or propositions in reasoning.
Proactive visual evidence" refers to visual information that is not, only partially, cued by the question, in contrast "refereed visual evidence" which explicitly indicated by the question.
4

---

Published as a conference paper at ICLR 2026
Taxonomy and Source
Fosture Contact
Sc enis
Complatty
Perception
Emotion
666- Comprehe nsten
Proceduro
Crowe22
VIfi
LEWIA
4 Behavior
Actorkii
knowledge- based Action
0000
CoptenGookdD
Holo
OHiAMMO
Beasoning
2. Annotation
3. Quality Control
6,000
@
Human-P
LUM Annotator
Answor
Multi-choice
Video
Distractor:
Fitteing Insald
Diffcclit Fileeing
Humaa Verifcattion
Human-C
Open-ended
Rejcet
Rejoot
Question
Chocklist
Alts
Anower
Human-R
Vi800
Rowlower
Scoring
Mela Besieser
Trial Labeling
Rojoct
Figure 3: A comprehensive pipeline for HumanPCR Construction. The process includes: (1) building a hierarchical task taxonomy through surveys and lask-driven data collection to ensure diversity, (2) recruiting annotators and conducting task-specific annotations, and (3) hybrid automated-manual verification with iterative refinement.
existing datasets, we manually generate QA pairs and complementary annotations with the assistance of domain-specifie expert annotators.
Human-R. Domain experts were recruited to annotate open-ended questions that encompass 5 distinct types of reasoning: Causal Reasoning, Prediction, Counter-Factual Reasoning. Assessment, and Planning. To satisfy our core reasoning criteria above, answering the questions requires integrating multiple pieces of visual evidence, engaging diverse reasoning chains, and extracting at least one proactive visual evidence from the video. In this process, the detailed reasoning steps and the necessary visual evidence were also annotated. As illustrated in Figure this design reliably tests model's s ability for holistic multimodal reasoning based on accurate evidence perception. This stands contrast to tests relying on sparse evidence fully specified refernces, which often lead to shortcut solutions instead of genuine vidco understanding and sufficient reasoning.
3.3 QUALITY CONTROL AND VERIFICATION
For Human-P and Human-C. C, QA pairs are first filtered by LLMs to climinate those solvable without visual input, followed by human verification conducted by trained annotators. Each annotation is carefully reviewed for linguistic quality, answer accuracy, distractor plausibility, and, most impor- tantly, its reliance on visual context. For Human-R, reviewers firstly fill out a detailed checklist that validates annotations' objectivity, factual accuracy, non-redundancy, and complexity; they then assign a quantitative score and deliver targeted feedback to the annotator for a chance to modify. Meta- reviewers further assess question complexity, ensuring that every question (1) requires integrating multiple visual evidence, which cannot be fully determined from the question alone, and (2) relies on least one essential proactive visual evidence. The interaction flow among the annotators, reviewers, and meta-reviewers is illustrated in the lower-right panel of Figure 3
This pipeline yields over 6,000 Human-P&e multiple-choice questions, and 442 Human-R open- ended questions with final acceptance rate 20%. Table 2 and Figure summarize HumanPCR scale and modality diversity, while Figures Tigures[land]3 55 that Human-R demands strong visual evidence across all video lengths, well above prior datasets, reflecting rigorous quality control.
3.4 COMPARISON WITH EXISTING BENCHMARKS
As summarized in Table I HumanPCR fills critical gaps across two key domains:
(I) Human-centric Benchmarks: Existing works are often either too narrow, limited to scopes like action (Hong al. 2025 Salehi et al. 2024), broad, targeting general visual understanding
5

---

Published as a conference paper at ICLR 2026
Table Comparison of HumanPCR and existing benchmarks with respect to: the number of ability dimensions (#Dim.) and tasks (#Tasks), covered modalities (Mod.), human-centric orientation (HC), taxonomy for probing and diagnostic analysis (Probing), average video duration (V.Len.). method of annotation (Anno., M/A means human-annotated/ automatically generated), open-ended questions (OE), sourced from a broad range of open domains (OD), availability rationales (CoT), requirement for multiple visual evidence (MVE) and proactive visual reasoning (Proactive).
Beachmarks
#Dim. #Tasks Mod. HC Probing OD
Benchmarks
V.Len. Anno, OE OD HC COT MVE Proactive
MotionBench
v
X
EguSchema
180
M
X
X
X
Face-Humant
VideoMME
ActionAtlas
V
V
X
X
CG-Bench
1624
M
/
/
X
X
MME
1
X
X
LVbench
4101
MVBench
V
/
/
MMVU
Human beach 16
V
V
X
X
Vidco-Holmes 60-300
A
X X
X
V
X
Human-P&C
9
34
I+V
V
/
V
Human-R
Table 2:
Key statistics of Hu-
Pcy Road Readi
manPCR.
Statisties
Value(Avg./Mar,
# Multi-Choice
6176
Options
4.9/5
100
Human-P&C # Images
1.1
Video Duration
584.1
# Open-Fnded Fineed
442
Question Length
19.2/79
Human
Col Length
86.2 183
Figure 5: The distribution of the visual evidence number.
Answer Length
18.6/64
Figure 4: Modality distri-
Video Duration
422.0
bution in HumanPCR.
without alignment to human-centrie tasks (Xu et al. 2023 Li at al. 2024d). Even dedicated human- centric benchmarks (Qin et al. 2025; Zhou et al. 2024b) like Human VBench are restricted by a few dimensions and single modality. HumanPCR provides comprehensive and fine-grained taxonomy for probing human-centnn cenannc capabilities across a diverse range of tasks and modalities.
(2) Video Reasoning Benchmarks: General video benchmarks typically assess shallow comprehen-
sion with limited reasoning diversity and depth (Fu et al. [024] (Zhou et al. [202a Wu et al., (024).
Early works either struggle with question quality due to semi automatic annotation (Qi al. 2025
Han et al. 2024) or are confined to narrow source or task (Jin et al. (2224) Song et al. 2025). While concurrent VideoHolmes (Cheng et al. 2025) and MMRV (Zhu et al. 2025b) require multi-frame reasoning, their task-specific designs limit the diversity of evidence and reasoning paths. Most importantly, their multiple-choice or certain question references often implicitly reveal the required visual evidence, failing to evaluate the crucial ability of proactive evidence secking. Human-R is thus distinguished by its multi-domain, open-ended design that uniquely demands both the integration of multiple visual cues and the proactive search for implicit evidence.
EXPERIMENTS
4.1 EVALUATION SETTINGS
(1)Models. We benchmark a diverse set of models, including 9 proprietary (e.g., Gemini-2.55Flaas o4-mini) and 30 open-source MLLMs (c.g.. Qwen-VL, InternVL) on HumanPCR. On Human-R, we add models specialized for video understanding and thinking. Human performance is also provided for comparison (Appendix C.4). (2)Configuration For evaluation, we employ Direct Answer prompts for multi-choice questions and CoT for open-ended ones. Video inputs are processed by sampling 32 frames for multi-choice tasks and the maximum allowable frames for open-ended tasks. Further details model configurations and prompts are in Appendix C. and C.2 M3)Metrics. Accuracy for multi-choice questions is determined by matching responses to the correct option. The average accuracy for each task, and the macro- average accuracy of tasks for each dimension or level is reported. For open-ended questions, we use a proprictary model, 03-mini, as a judge, which demonstrates high agreement with human evaluations (Sec 4.4

---

Published as a conference paper at ICLR 2026
Table 3: Results of Proprictary and Open-Source Models on HumanPCR. Macro-average accuracy of tasks within each level and dimension is reported. "I" indicates that the reported performance is based only on a subset of the HumanPCR dataset. Full results is in Appendix
Multi-
Open
Models
Human-P
Human-C:
Human-R
Spa.
App.
Ide.
avg.
Beh.
Rel.
Sce.
ang.
Acc.
Random Human"
20.00
20.00
20.00
20.00
35,00
23:00
21.00
20.00
20.00
20.00
20.25
21.78
0.00
89.98
$1.44
87.02
87.28
96.43
88.43
69.21
81.42
78,95
65.83
73,86
81.95
73.17
Open- Models
Aria
Liet
2024b
79.12
39.72
61.32
36.58
76.53
555.53
48.85
41.31
50.08
55,30
48.44
51.98
28.96
LongVi
2024b
75.96
63.28
88.87
41.63
44,97
31.91
51.12
53.20
44.51
21.49
NVILA-SB [Lu et al. (222-0)
76.39
35.40
64.53
42.53
47.95
52.22
44,10
28.59
50.69
53.18
43.23
47.72
22.17
MiniCPM W22.6 Yan et al., 2024
74.07
36,90
57.77
35.56
70.47
52.08
47.39
28.60
50.46
54.67
44.31
48:20
16.74
MiniCPM-n-2.6 Mpenit 21
80.08
43,80
63.58
36.95
71,26
56.88
47.88
31.43
53.98
55.38
46.23
51.56
21.04
LLaVA-Video-78 Zhane el al. 20244
74.73
37.13
58.03
32.62
63,50
50.99
46.
36.26
49.54
51.77
45.37
48.18
27.60
Videoo Zhhung al.
20245
76.64
42.97
21.68
56.54
63.63
60:49
51.74
43.73
55,41
60.92
52.41
56.45
28.28
[. LaVWACCeeVisiee-77 7B Li 2024ay
77.95
36.33
60.03
34.13
54,63
551002
47.67
35. 35.56
52.40
51.06
46.02
48.52
22.85
LLaVA-OteViiioe
Gretal 201260
82:57
70.40
71.95
43.19
52.99
Oryx-1.5.7B (Liu et 2024b
74.62
36.48
62.36
42.49
39.18
50.68
44.21
34.37
51.98
49.21
44.32
47.50
22.17
Oryx-1.5-328
202445
82.08
40.86
88
47.90
45.00
55.52
47.61
57.51
50.68
28.51
Owen2.5-VL-778
(Bai
2025
32.
55.
46.65
48.94
26.24
Qwen2.5-VL. 72B
13. et
2125
82.11
50.00
68.70
43.76
41.32
57.94
55.77
46.49
53.23
64.43
54.48
56.21
34.39
Chen et
60.37
75.45
55.887
39.11
48.51
52.17
23.53
IntemnVV2.55336
20746
84.34
45.14
68.70
50.02
84.29
63.07
$3.89
55.79
54.44
63.17
56.76
59.92
35.97
Ln15m\V2..5778
64.80
44.68
69.31
50.87
81.24
62.95
57.51
53.93
57.24
65.20
58.21
60.58
33.94
IntemVLL3-88 3-88
Zhu et al.
50255
81.64
40.54
67.20
38 22
72.84
57 .46
$1.39
40 40.86
55,00
59.11
50.97
54.21
31.45
MitmmLL33386
84.73
69, 16
66.02
59.32
35.75
LL3778
ZRu ot a1. 20225
86.54
16.45
73.39
50.82
86.42
65.34
57.96
54.31
59.78
70.21
60.20
62.77
37.56
Proprietary Models
Doubao 1.5-vision ByteDance, 1024
67.19
55.33
45.20
47.56
32.81
Grok-2-Vision 2024 20124
57.83
30:97
57.51
41.14
50.21
46.01
46.51
42:42
42.27
56.52
46.67
46, 34
36.20
1ande=15-Somnery2 Amiirppic 3024
67.99
39.84
59.35
44.68
66.26
53.36
50.39
46.88
49.08
59.3
51.12
52.24
39.59
Ge.ini-1..-Faahhh
54.99
53.78
32.82
54.45
47.63
41.92
51.81
46.23
46.03
35.97
Gumini-1.5-Po Teai c Fil 201e1
66.80
45.62
$6.01
39.34
69.03
53.45
51.67
44.84
50.16
61.81
$1.69
52.57
40.05
Gemini-2.0 Flass Luogge Decphilnd 2024
76.42
47
47.45
63.75
52.32
73.08
60.28
53.09
42.27
54.37
60.76
51.01
56.14
38.01
Gemini-2.5.Flach Commanee aral 2122
82.01
48.10
69.41
37
93.50
64.66
54.05
49.19
57.27
62 62.56
$5.38
60.02
45.44
GPT-4o Hursi el al. 20121
70.14
40.56
53.46
33.60
35.11
47.41
52.01
38.93
48.85
60.14
49.33
48.37
41,40
04-mini (penAr
53.13
54,43
65.97
60.42
53.39
4.2
MAIN RESULTS
Table 3 presents the main results of HumanPCR. Our principal findings are summarized as follows: Current MLLMs are far from reliable human-centric understanding. A significant gap exists across all levels, with humans outperforming the best MLLMs by over 15%. Leading models like InternVL33-78B achieve average accuracies of only 63.66% on Human-P&C. lagging behind the human baseline of 81.95% While models show promise in basic tasks like Spatiality, the deficits in fine-grained perception (Posture, Contact) and high-level understanding (Procedure, Relation) underscore the need for the detailed, probing evaluation that HumanPCR provides.
Open-source models rival in perception and comprehension, lag in reasoning. On Human-P and Human-C, open-source models match proprietary ones. Notably, VL3-78LL3-78 surpasses the top proprietary model, 04-mini. However, they underperform in reasoning- most remain below 30% accuracy on Human-R, R, whereas all proprietary models exceed this value. Models like o4-mini and 2e5ini-2.5-Flash illustrate the advantages of proprietary designs for reasoning. While open-source models match proprietary ones at perception and comprehension, they struggle on reasoning tasks involving complex visual evidence.
Understanding human-centric scenes reflects general capabilities. The consistent poor perfor- mance dimension Posture, Contact, Behavior, and Procedure points to fundamental limitations that transcend numan-centria scenarios. These results expose core deficiencies in fine- ame-grained spatial perception (especially with occlusion) and in the temporal understanding necessary to complex, long-term activities. Therefore, the challenges presented in HumanPCR do not merely identify gaps in human-centric understanding but also reflect general, critical shortcomings in current ML.MMs.
7

---

Published as a conference paper at ICLR 2026
140,205
23
18
Humbber m7
N2tr2re Teamses
Humber of Fremnss
Mumber Tramnss
(a) Overall Acc. Figure 6: Effect of frame sampling on Human-R. (a) Overall accuracy. (b-d) Accuracy grouped by the number of visual evidence required in the question: (b) 2-3 evidence, (c) 4-5 evidence, (d) 6 or more evidence. Increasing frames shows varying impact depending visual complexity.
(b) Visual Evidence: 2-3 (c) Visual Evidence: 4-5 (d) Visual Evidence: 6+
Frame Sel.
Token Sel.
Down Pooling
Mom6-Retri
Thinking
viow 27..66222
deatss
Owend Suuune 26,2522
uireme thi
MOT
Vodeo saary
Dispedert
Lompeny
Asstr Hendabenc
Figure 7: Accurary gain over video understanding baseline methods on Human-R and Video-MME. See the full configuration of them in Appendix
4.3 DELVING INTO VISUAL REASONING ON MULTIPLE EVIDENCE
On Human-R, we analyze the characteristics and limitations of MILMs' reasoning ability.
Effects of frame number. As shown in Figure 6 merely increasing the input frame count yields negligible accuracy gains for most models. Notably, only the reasoning-orienn oriented 04-mini improves consistently, implying that stronger reasoning is needed to make full use of broader visual context. This trend is amplified as the reasoning challenge intensifies: when a problem requiring integrating more pieces of visual evidence, overall accuracy drops. In these more complex scenarios, the marginal benefit of adding frames diminishes, even becoming detrimental in cases requiring six or more evidence items. This suggests larger visual context can introduce distractors, complicating the evidence extraction and integration process when the core reasoning task becomes more demanding.
Takeaway: More frames may miligate percepiual gaps, but cannot substitute for the core reasoning ability required effectively utilize brouder visual contexts.
Does advanced video understanding configuration help? We investigate advanced configurations to determine if they could address the challenges Human-R. (1) Visual context extraction. As shown in Figure 7 techniques that effective standard benchmarks like Video-MME (Fu et al.) 2024), such as token selection and memory-based retrieval, see a significant performance drop on Human-R. This performance gap verifies that integrating multiple visual required by Human-R presents a unique challenge that cannot be solved by simple query-guided or heuristic matching methods. (2) Test-time computation. Best-of-N (BoN) sampling consistently boosts performance by over 5%, with gains scaling with better reward models. In contrast, Self-Refing offers only marginal benefits. The strong performance of specialized reasoning models like Video-RI and the proprietary o3 (Figure 7 4 underscores that enhancing the core reasoning process itself is critical.
Takeaway: Complex reasoning demands finer context management than general understanding: simple, heuristic extraction fail to generalize problems requiring multi-faceted evidence.
Error analysis. Manual analysis of errors within subset of 200 questions for top models on Human- (shown in Figure Sa) reveals the predominant failure is risual-evidence extraction, specifically in identifying proactive visual evidence (in Figure (8b.. This suggests models often treat the question as a retrieval shortcut, bypassing necessary complex reasoning. Our findings support that while scaling the frames significantly reduces errors of 2emini-2.0-fllah in missing referred evidence, it provides
8

---

Published as a conference paper at ICLR 2026
Table Results for test-time scaling strategies on Human- Table 5: Benchmark evaluation across R. BoN uses 2M candidates and Self-Refne performs M input modalities on GPT-4o. 'Image' iterations. Results of more models are T2]
uses the central frame; 'Video' samples
Model
Method
Reward Model
Direct Mall (CoT)
M=1
M=2
32 uniform frames.
BaN
GPT-4o
Benchmark
Text (ratio.)
Image (radio
Video
GPT-4
BaN SeftRRrfire
BaN
04-mini
Geminn-25-FFssh
32.81
41,40
44.11
44.57
38
45.02
46.38
LongWikeBeech
Vide--MME
44 44.00 196)
54.14 (75.003))))
398)
71,85
2.55Pr--2.-PPo
Thinking
Human-R
2.94
5.7-somnet
Thinking
Thinking
59.28
fottaal lacrapias Aures
Whong
Gesti2 Pro
Exibakk
Referrel
Gred 00
Diidhl LA Pork (oS
159 100
Misell
(a)
(b)
Figure 8: Distribution of error types on Human-R R across top models ("f": the number of sampled
frames). (a) Counts of major error types. Most errors are visual-related. (b) Fine-grained breakdown
of visual-related errors. Proactive evidence is more frequently missed than referred evidence.
minimal benefit for finding proactive evidence. Consequently, models with superior reasoning, such as Gemini-2.5-Pro (Comanici al. 2025) and Claude-3.7-sonee
(Anthropic (022).. outperform
high-frame -oghfrrame-count models like Gemini-2.0- 2.0-Flas.. even with fewer frames. Furthermore, models exhibit distinct error tendencies; For example, while Gemini-2.5- reduces proactive evidence omissions compared to o3, it tends to introduce more irrelevant information.
Takeaway: Proactive evidence extraction is major practical challenge on Human-R. and difficulties in selecting implicit visual cues can limit performance, suggesting that evaluation setups should discourage purely query-driven shortcuts.
Interventions on evidence extraction difficulty. To further probe the role of proactive evidence, conduct intervention study Human-R, progressively lowering the difficulty of visual evidence extraction while keeping the question and answer fixed. We enrich prompts with three levels of guidance: (1) Relation awareness (Level 1), giving generic relation-type hints (c.g.. "check surrounding context"); (2) Logic awareness (Level 2), additionally highlighting which referred cues are logically linked to potential proactive evidence; and (3) Proactive guidance (Level 3), adding vague descriptions that loosely point to the proactive evidence without revealing the reasoning steps or answer. As shown 6 Level 1/2 hints yield only modest gains, whereas Level 3 consistently improves accuracy by about 10-13 points across models. This indicates that directly easing proactive evidence extraction has much larger impact than generic relational or logical hints, providing intervention- basdd support for our insight that proactive evidence extraction is a prominent practical weakness of current models on complex video reasoning.
Takeaway: On Human-R, vague guidance that directly targets proactive evidence yields the largest gains, reinforcing proactive evidence extraction as prominent practical weakness in complex video reasoning.
4.4 FURTHER ANALYSIS
Mixed CoT effects and diagnostic value of Human-P/C. Figure 9a shows that CoT has mixed effects on Human-Pir PPC:: proprietary models typically improve, whereas many open-source models see limited negative gains, and the trend already varies across high-level dimensions, with visually complex and understanding-relate questions benefiting more than simple attribute- -erbbutttprreptionn tasks. Figure 9h further reveals substantial spread in both accuracy and A (cot-direct) across neighboring

---

Published as a conference paper at ICLR 2026
Table Intervention on evidence extraction for Human-R with various evidence hints. Level 1/2/3 provide generic relational, logical hints, and vague descriptions of proactive evidence, respectively.
Model
Orig.
Level
Level
Level 3
04-mini
53.39
52.26 (.113)
54.07 (+0.68)
63.35 (+9.96)
GPT-4o
41.40
41.62 (40.22)
44.11 (+2.71)
52.35 (+10.55)
Gemini-2.5-Fllas
43.44
41,40
(.204)
45.93 (+2.49)
53.40 (+9.6)
Qwen-2.5-VL-722
34.39
34.61
(40.22)
38.46 (+4.07)
47.74 (+13.35)
Coida
Penter
Apporeee Oeeart
Meddy
Dieasstees
Cone
PBC
tis
E51
1.0
140
215
214
14
810
13
671
173
3.4
128
1:31
11:11
0.1
3.0
185
312
10:0
6.38
0144
0.3
3.3
609
0.30
Hossi Caees
Houe Orees
LAARR00-11 19-
1991
0.1
11
8.31
3.0
(b)
Figure 9: Ablation study examining the impact of CoT. (a) Relative improvements of CoT of models on all dimensions and levels. (b) Detailed performance across tasks in the Contact and Procedure dimensions. "A (cot-direct)" represents the accuracy difference between direct answers and CoT- prompted responses.
subtasks within the same dimension, with some tasks consistently benefiting from CoT while others degrade and model rankings change accordingly. These heterogeneous patterns indicate that Human- P/C subtasks are non-redundant and provide diagnostic resolution that would be lost under a single coarse "action" or "relation" score.
Human-R Quality Check To validate the quality of Human-R, we examine potential single-frame and textual biases and compare them with other impactful benchmarks. As shown Table 5 Human-R exhibits minimal bias, unlike datasets such as Video-MME where strong results can be achieved using only text or a single frame, revealing redundancy and bias in evaluating full video reasoning. This confirms that Human-R tasks require genuine temporal and multi-evidence reasoning. demonstrating the effectiveness of our rigorous curation and expert review.
Reliability of LLM-based Judges. To verify evaluation robustness, we compared multiple LLM judges against 4000 human annotations, in Figure TO All judges produced highly consistent model rankings, with strong Pearson correlations to human accuracy (higher than 0.64) and strong alignment on pairwise win-fail pref- erences (around 0.85). Additional analyses( (aalyses(Appendix Appendix C.3 show that annotated CoT could improve agreement with humans while no self selfferefeeene of LLM judge is observed.
Marh
Figure 10: Model performance under mul- tiple judge models and their agreement to Human Humanjudge.
CONCLUSION
We present HumanPCR, comprehensive benchmark for evaluating how well MLLMs understand humans across diverse scenarios. First, probes MILMSs nuanced visual understanding in human- centric scenarios through systematic, fine-grained taxonomy. Second, it introduces paradigm for video reasoning that integrates disparate visual evidence and proactively seeks implicit visual cues. Consequently, HumanPCR reveals persistent shortcomings and yields diagnostic insights. Specifically, MLLMs not only face challenges detailed space perception, temporal understanding. and mind modeling, but also often fail to proactively extract visual evidence in reasoning. Limitations of HumanPCR include reliance on academic datasets; future work will extend to professional domains and efficient annotation methods.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially supported by the National Natural Science Foundation of China (Nos. 02461160331, 62306301, and 62576334). We also thank Dingeun Chen, Yansong Li, Cunku Liang, Ailin Zhao, Weiming Liu, Jun Zhang, Yutong Shen, Yuhan Wang and Han Liu for their valuable suggestions and feedback on the annotation protocol of our dataset.
ETHICS STATEMENT
We have prioritized ethical considerations throughout the creation and planned release of HumanPCR. 1. Data Sourcing and Privacy: Our benchmark is built upon previously published, public datasets and supplemented with publicly available internet videos. To respect copyright and the privacy of content creators, will only release metadata (e.g.. public video IDs and timestamps) for internet- sourced videos, not the raw video files themselves. Our full data hosting and usage policy is detailed in our Data Use Agreement (DUA) in Figure 25 2. Annotation and Distribution Safeguards: During the annotation process, all annotators and reviewers were instructed to filter out questions that require time-sensitive, private, or personally identifiable information for their resolution. The HumanPCR benchmark is intended strictly for non-commercial, academic research purposes. The DUA explicitly prohibits any use of the data for biometric identification, tracking, surveillance, or the development of related applications. 3. Compliance and Responsibility: For any source data that we are permitted to host, we require end-users to agree to the original dataset's license terms. We are committed to being responsible stewards of this data and will promptly respond to any inquiries or takedown requests from copyright holders.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the full reproducibility of our work. Code and Prompts: All prompts used for model evaluation are detailed in Appendix C.2 The evaluation code has been included in the supplementary materials. Configurations for all evaluated models are detailed in Appendix[[]
C 2. Data and Annotations: All annotations created for HumanPCR will be released to public accordance with our DUA upon acceptance. For the small subset of source data we cannot host directly (due original license copyright), full metadata and data processing scripts would be provided to reconstruct the dataset. Our data release policy has been detailed in Appendix [ A portion of the data is available in the supplementary material demonstrate its structure.
REFERENCES
Anthropic.
Introducing Claude 3.5 Sonnet.
3- 5- bondelss-ndd-ooputeerre
June 2024.
Anthropic. Claude 3.7 sonnet and claude code. February 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.1392. 2025.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. JEEE/ACM transactions audio, speech, and language processing, 25333,,,,33,,,,33 2023.
ByteDance. Doubao. https: www. http://wwwwolleeggeeeee//co/rr//
2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
11

---

Published as a conference paper at ICLR 2026
Zikui Cai, Andrew Wang, Anirudh Satheesh, Ankit Nakhawa. Hyunwoo Jae, Keenan Powell, Minghui Liu, Neel Jay, Sungbin Oh, Xiyao Wang, et al. Morse-500: A programmatically controllable video benchmark to stress-test multimodal reasoning, arXiv preprint wwXiv:2556005522. 2025.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. cot: A novel bench- mark for multi-domain step multi-moda chain-of-thougee arXiv preprint arXiv:2405.16473, 2024a.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10115 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic risual-linguisti tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185-24198, 2024d.
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can milm think like holmes for complex video reasoning? arXiv preprint M.Xiv:2505.21374 2025.
Yuanzheng Ci. Yizhou Wang. Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihep: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
17840-175 17852, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Jieming Cui, Ziren Gong, Baoxiong Jia, Siyuan Huang, Zilong Zheng, Jianzhu Ma, and Yixin Zhu Probio: A cootocol-guider multimodal dataset for molecular biology lab. Advances in Neural Information Processing Systems, 36:41543 41571, 2023.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint 27X1v:2503.21776. 2025.
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Posegpt: Chatting about 3d human pose. CoRR, 2023.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21077 2024.
Google tic
DeepMind. Introducing gemini 2.0: Our new ai model for the agen- era.
https://blog 2024/ 2024.
Gnoglle-emingg ai www.ppt-aaeceeeeee
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 18995-19012 2022.
the
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllow Afouras, Kumar Ashutosh. Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and the IEEF/CV Conference on Computer Vision and Patern Recognition, pp. 19383-1944 2024.
perspectives. In Proceedings of
12

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thouge dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:2411.144 147794, 2024.
Wenyi Hong, Yean Cheng, Zhuoyi Yang. Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models. arXiv preprint arXiv:2501.02990 02955,, 2025.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13822 2025.
De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, and Jan Kautz. Frag: Frame selec- tion augmented generation for long video and long document understanding. arXiv preprint arXiv:2504. 17447, 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.1127, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720., 2024.
Simindokht Jahangard, Zhixi Cai. Shiki Wen, and Hamid Rezatofighi. Jrdb-social: A multifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22087-22097, 2024.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprint arXiv:2401.08743, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 033326, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. Aria: An open multimodal native mixture-of-exper model. arXiv preprint arXiv:2410.0599 2024b.
Keliang Li, Zaifei Yang, Jiahe Zhao, Hongze Shen, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen. Herm: Benchmarking and enhancing multimodal llms for human-centric understanding. arXiv preprint arXiv:2410.06777, 2024c.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 22195-22206, 2024d.
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning, arXiv preprint 200402504400658. 2025a.
13

---

Published as a conference paper at ICLR 2026
Yiheng Li, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen. Unipose: A unified multimodal framework for human pose comprehension, generation and editing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27805 27815, 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012) 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024a.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand teatial-temporal understanding at arbitrary resolution. arXiv preprint N.VV:22009.19611, 2024b.
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04322 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually- aligned sttrivall-uggmenn long video comprehension. 2024.
arXiv preprint arXiv:2411.
Zelun Luo, Wanze Xie, Siddharth Kapoor, Yiyun Liang. Michael Cooper, Juan Carlos Niebles, Ehsan Adeli, and Fei-Fei Li. Moma: Multi-object multi-actor activity parsing. Advances in neural information processing systems, 34:17939-17955, 2021.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very form video language understanding, Advances in Neural Information Processing Systems, 36:46212 2023.
OpenAl. Introducing OpenAI o3 and 04-mini. introducing 03 and- mini/ 2025.
http:://openai..com/itti
OpenBMB. Minicpm-v Accessed: 2024-05-21.
lat http://///ttttaaa..o///t/tttt//t
2024.
Chau Pham, Hoang Phan, David Doermann, and Yunjie Tian. Personalized large vision-language models. arXiv preprint arXiv:2412.1 /2. 17610, 2024.
Chiara Plizzari, Alessio Tonioni. Yongqin Xian, Achin Kulshrestha. and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms egocentric videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24129-24138, 2025.
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. Ver-bench: A comprehensive evaluation framework for video chain-of-thought reasoning. arXiv preprint arXiv:2504.07956. 2025.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception. decision, and reaction. arXiv preprint 250.02250,.,3218, 2025.
Lixiong Qin, Shilong Ou. Miaoxuan Zhang, Jiangning Wei, Yuhang Zhang. Xiaoshuai Song, Yuchen Liu, Mei Wang, and Weiran Xu. Face-human hen----mmmmnn: A comprehensive benchmark of face and human understanding for multi-modal assistants. arXiv preprint 2rXiv:2501.01243 2025.
14

---

Published as a conference paper at ICLR 2026
MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin, Caleb Ziems, Yanzhe Zhang, Hao Zhu, and Diyi Yang. Egonormia: Benchmarking physical social norm understanding. arXiv preprint arXiv:2502.22440, 2025.
Mohammadreza Reza Salehi, Jae Sung Park, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hanna Hajishirzi, and Ali Farhadi. Actionatlas: A videoqa benchmark for domain-speciilized action recognition. Advances in Neural Information Processing Systems, 37:137372-1374000 2024.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.1745/ 2024.
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi-discipline lecture understanding benchmark. arXiv preprint arXiv:2504. 14693, 2025.
Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang. Li Yi, et al. Humanbench: Towards general human-centrie perception with projector assisted pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11970-21982, 2023.
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. arXiv preprint arXiv:2502.21277 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's 's perception of the world at any resolution. arXiv preprint arXivv2409) 12191, 2024.
Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. Videorft: Incentivizing video reasoning capability in milms via reinforced fine-tuning, arXiv preprint arXiv:2505. 124344 2025a.
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nic. Adaretake: Adap- tive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503. 12559, 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 55:24424-248877 2022.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved languaae understanding. Advances Neural Information Processing Systems, 01128828-288555 2024.
xAI. Grok-2 beta release. https://x. ai/news//wwee htt///////ooo//22///// 2
May 2024.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al. Mmbench: Benchmarking end-to-end multi-modal dans and understanding their hardware-software implications. In 2023 IEEE International Symposium Workload Characterization (IISCC), pp. 154-166. IEEE, 2023.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:2411. 104440, 2024.
Chenyu Yang. Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang. Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. Pve: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint 22.X0::24/2.061133 2024a.
15

---

Published as a conference paper at ICLR 2026
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint
arXfv:2412.14177, 202444
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone, arXiv preprint arXiv:2408. 01800, 2024.
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyza- guirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. Re-thinking temporal search for long-form video understanding. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 8579 8591, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:24077 12772, 2024a.
Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internlm-xcomn pxcemml-xcoompse2... 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.09596, 2024b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024c.
Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang. and Linchao Zhu. Flexselect: Flexible token selection for efficient long video understanding. arXiv preprint arXiv:2506.0099) 2025.
Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501.12388 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04220 2024a.
Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, and Ying Shen. Humanvbench: Exploring bumann-centic video understanding capabilities of milms with synthetic benchmark data. arXiv preprint arXiv:2412. 77544, 2024b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su. Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 2.Xx:22504..4770 2025a.
Kejian Zhu Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Mmr-v: What's left unsaid? a benchmark for multimodal deep reasoning in videos. arXiv preprint 2506004000 2025b.
16

---

Published as a conference paper at ICLR 2026
SUMMARY OF APPENDIX
The appendix is organized as follows:
Details of the Taxonomy (Appendix(A)::
Annotation Setup (Appendix B):
Evaluation Setup (Appendix C
Additional Data Statistics (Appendix D):
Public Release and Data Usage Terms (Appendix E};
Additional Results (Appendix FF::
Additional Analysis (Appendix
G
Use of Large Language Models (LLMs) (Appendix H)
17

---

Published as a conference paper at ICLR 2026
A DETAILS OF TAXONOMY
The comprehensive presentation of the full taxonomy of HumanPCR is shown in Figure 11
Spatial Relation
Object Edistence
Spatiality
Human Presence
Body Posture
Hand Shate
Posture
Hand-Otject Interaction
Body Orientation
Estimation
Human-P
Appearance
Clothing Attribute
Accessory Rlecognttion
Bodypart Visbility
Physical Attribute
Contact
Human Object Contact
Human Human Contact
Human Self-Cortact
Identity
Face Recogrittin
Identity Gustering
Behavior
Baokt Action
Knswledge Resel Adbion
Sequential Action
Goal Planning
Procedure
Procedure Dependence
Inrelevant Action
Multiple Human Sequence Action
Human-C
Human Comparisan
Relation
Group Activity
Struggle Dotection
Crowd
Cultural Event
Scene
Crime
Understanding
Cscaal Ressaring
Prediction
Type
Assessment
Manning
Crunter-factual Beasaning
Human-R
Infoor tekure self Care
Outdoor Adventure
Nososhald
Ousdeer Laioze
Recreation
Domain
Elctric Crelting
Trursport
Performance Eahlaition
Dally Spanding
Education
Figure 11: A comprehensive presentation of the full taxonomy.
B ANNOTATION SETUP
B. ANNOTATION AND FILTERING PROMPTS
Question Generation Prompts.
To automatically generate high-quality questions, we employ two complementary strategies: M-se- generation and template-based generation. In the M.Le-
18

---

Published as a conference paper at ICLR 2026
pipeline, feed GPT-4o with carefully crafted prompts augmented by comprehensive auxiliary information (see example prompts in Figure 12). the template-ber based pipeline, we start from a set of seed templates and task the LLM with expanding them into more diverse collection of question templates (examples appear in Figure] 13)
#** I will provide caption about the spatial relationship of two object, the label the spatial relationship, For 3 inputs, you will generate one diverse question for each input that can examine the following aspects:
Correctly answer the multiple-choice exan about recognizing the spatial relationsip of two objects in the image, which is described by the caption.
A44 As for your questions, you should ensure:
1. Output only question text.
Anyone can not obtain the answer just by reading the question text. Avoid implying answers in the questions.
3. Do not include specific visual details of the scene in the question.
4. Use plain text. Do not use Markdown format.
5 Make the questions concise, simple and straight Do NOT add imaginary details to the questions.
6. Make the answer to the question point towards the spatial relation labe' of the two objects.
De NOT disclose the spatial relation label in the questions. not say "caption"
questions.
#* When announcing the question please label each question as Question 1,2,3; [full question]'. Don't repeat the input.
A4 The input form for an image will be: 1.
Spatial Relation Caption: (caption)
Spatial Relation Label: (label)
2.
Spatial Relation Captien: (caption) Spatial Relation Label: (label)
3.
Spatial Relation Caption: (caption) Spatial Relation Label: (label)
Figure 12: Example prompt of LLM-based question generation used in the task Spatial Relation
Option Generation Prompts. To generate high-quality options, we decouple option generation from question generation. During option generation, we provide the LLM (GPT-4o) with the question, its correct answer, and supplementary context. The model is then instructed to produce options that all pertinent the question, with incorrect options crafted to be plausible yet wrong, thereby increasing the overall difficulty. An example prompt employed for option generation is illustrated in Figure [4
Blind Filtering Prompt. After acquiring the automatically generated annotations, we eliminate any question that be answered correctly without visual input. Specifically, prompt GPT-4o to provide its optimal answer to each question in the absence of the corresponding image or video. Any question that is answered correctly across multiple with the answer choices randomly
19

---

Published as a conference paper at ICLR 2026
"Wnat is the position of the person's hand in the picture?" "How the person's s hand positioned in the image?" "what is the state of the person's s hand in the photo?"
"How is the hand of the person in the picture?"
"What condition is the person's hand in within the image?" "In the picture, what is the status "How dees the person's hand appear in the photograph?"
the person's hand?"
"What is the arrangement of the person's s hand in the imago?"
"What is the specific condition of the person' a hand in the picture?"
"How exactly is the person's s hand positioned in the image?"
"What precise state is the person's s hand in within the photo?" "Can you describe the exact state of the person's hand the picture?"
"What is the detailed posture of the person's hand in the image?" "How is the person's s nand specifically arranged in the photograph?" "What particular position is the person's hand within the image?" "How would you describe the exact condition of the person's s hand in the picture?"
"what is the exact gesture of the person's hand in the picture?" "How the person's s hand specifically posed in the image?" "What is the detailed form of the person's s hand the photo?" "Can you describe the precise hand gesture of the person in the picture?"
"What is the specific shape of the person's hand in the image?"
"How exactly
the person's s hand configured in the photograph?"
"What distinct position is the person's s hand assuming in the image?"
"How would you characterize the speci fic hand posture of the person in the picture?"
Figure 13: Example templates of template-based question generation used in the task Hand State. The ambiguous references in questions were subsequently identified and corrected by annotators.
permuted each time-is subsequently discarded. The prompt employed for blind filtering is illustrated in Figure 15
B.2 ANNOTATION AND REVIEW INTERFACE
We designed personalized annotation and review interfaces to maximize annotation quality and streamline the entire workflow. The annotation interface guides each annotators to record the question, detailed reasoning steps, final answer, question category, and the relevant time interval. The review interface then requires reviewers to verify every annotated field and to assign both objective evaluations and subjective scores according to comprehensive checklist. Finally, both interfaces include integrated messaging channels, enabling itcrative feedback loops between annotators and reviewers and thus ensuring consistently high-quality annotations. A few snapshots of annotation and review interfaces is shown in Figure 16
20

---

Published as a conference paper at ICLR 2026
**# I will provide a captien about the spatial relationship of two object, the label the spatial relationship and the questions, and diverse questions that can examine the following aspects:
Correctly answer the multiple-choice exan about recognizing the spatial relationsip of two objects in the image, which is described by the caption.
## Based on the provided questions, I want you to create difficult and diverse miltiple-choice exam that tests the above aspects.
Each question should have 5 short answers, including 1 correnct
answer and wrong answers. Rach answer option should reflect reasonable understanding of broadly similar but detail-different image.
The wreng answers should diverge from the correct ones only by fine-grained and subtle details that easily mistaken.
#*# As for your answers, you should ensure:
1. Only one answer will be correct.
2. Answers are short and concise. Answers should not include irrelevant details that weren't queried.
Use plain text. Do not use Markdown format.
4. Make the answers concise, simple and straightforward. Do NOT add imaginary details to the answers.
5. De NOT change the ven questions.
#*# Print a question and then print each correct answer on new line exactly as "Correct answer: [full answer]" Please print each wrong answer new 1ine and print each wrong answer "Wrong answer 1,2,3,4: [full answer]. Parse question-answer pairs by AnAn'.
The input form will be:
1.
Spatial Relation Caption: CAPTION Spatial Relation Label: LABEL Questions: QUESTIONS
2.
Spatial Relation Caption: CAPTION Spatial Relation Label: LABEL Questions: QUESTIONS
3.
Spatial Relation Caption: CAPTICN Spatial Relation Label: LABEL Questicns: QUESTIONS
Figure 14: Example prompt of option generation used in the task Spatial Relation
21

---

Published as a conference paper at ICLR 2026
You will be presented with a miltiple-choice question, each with a fixed set of Labeled options (e.g. A, B, C, D). Your task:
1. You **mUst** choose exactly one of the provided --options,, B, c, etc.
2. De *ADot** say "heers not enough information, "no answer", refuse.
If you genuinely cannot determine a "correct" answer, pick the option that seems **most plausiblers.
4. Only output a option label, e.g.; "A"
Figure 15: The prompt employed for blind filtering.
al BEC
(b)
(c)
(d)
Figure 16: Snapshots of annotation and review interfaces.
22

---

Published as a conference paper at ICLR 2026
B.3 DETAILED ANNOTATION AND REVIEW PROTOCOL FOR HUMAN-R.
This subsection extensively describes the annotation and review processes employed in constructing Human-R, elaborating details beyond the main text.
Table 7: Example video tags of 11 human-related domains for -lu
Domain
Tags
Electric & Crafting
Smart home devices, Kitchen appliances, Mobile device sctup, Audio- visual equipment, Power tools usage, 3D printers, Smart thermostats, Electric scooters, Solar panel installation, Portable power stations
Outdoor Leisure & Recreation
Trail running, Paddleboarding fitness, Aqua aerohics, Slacklining, Out- door yoga, Hot yoga, Balance and stability training, Kickboxing sessions, Animal flow workouts, Recreational cycling
Outdoor Adventure
Obstacle course training, Mountain biking, Marathon training plans, Ultra-trail running, Kayaking expeditions, Rock-climbing practice, Snowboarding sessions, Zip-line experiences, Caving basics, White- water rafting
Houschold
Wound cleaning and bandaging, Correct thermometer use, Home blood- pressure checks, Blood-glucose meter steps, Oral hygiene care, PPE donning methods, Nail disinfection, Skin antisepsis steps. Baby-bathing technique, Umbilical-cort care
Sports
HITT workouts, Powerlifting programs, CrossFit sessions, Spinning classes, tneength-training routines, Cardio circuits, Kettlebell workouts, Rowing- machine drills, Plyometric exercises, Speed-and andd--nd-agee and-agility drills
Transport
People in the subway, Passenger behavior on buses, Rush-hour metro crowds, Live bus footage, Commuters trains, Night-bus activitics, Airport-terminal scenes, Cycling commuters, Ferry passenger journeys. Fublic-transpee safety tips
Indoor Leisure & Self Care
Yoga sessions, Pilates classes, Stretching routines, Home Mlood-pressus check, Meditation practice, Resistance-baat workouts, Prenatal yoga, Postnatal fitness, Power yoga, Handstand training
Daily Spending
Supermarket shopping vlog. Online-shopping process, Restaurant dining vlog, Ordering food delivery, Thrift-store shopping, Paying bills per- son, Buying electronics in store, Unboxing subscription boxes, Booking movie tickets online, Coffee-shop visits
Performance & Exhibition
Attending concert vlog, Street artists on trains, Metro musical perfor- mances, Flash mobs on huses, Visiting a muscum, Public art stations, Art-workshop vlog. Open-mic night, Theatre-performance vlog, Dance- recital backstage
Education
Health-education workshop, Diabetes-educatte session, First-aid train- ing demo, Speech-therapy practice, Clinical-trial walkthrough, Nutrition- counseling session, Community- -halth education, Occupational--ect -halth lesson, hralth- briefing, CPR operation video
Others
Community health-center routine, Cancer-screening demo, Sleep-study procedure, Travel-heale -ealth consultation, Language- therapy session, Geri- atric patient care. Dietitian counseling. mainemaaagemmnt therapy, Vac- cination outreach, Sports-injury treatment
B.3.1
ANNOTATION PROCESS
Step 1: Video Collection. To ensure domain coverage and contextual diversity, videos were sourced from 11 human-related domains (as illustrated in Figure by two main channels. (a) We leveraged curated egocentric and third-person activity datasets- Ego4D Fgo4D (Graamaan et al. 2022).
23

---

Published as a conference paper at ICLR 2026
EgoExo4D (Grauman et al. 2024), ActivityNet (Caba Heilbron et al. 2013), and MOMA (Luo et al. 2021) and relied exclusively on their test or validation splits to avoid data leakage. (b) We collected internet videos from YouTube using domain-specifit auto-generated tags. Clips under five minutes or containing low-quality or off-topic content were excluded. Only videos published within the past two years were considered.
Step 2: Annotation Protocol. Visual reasoning involves extracting visual evidence across gran- ularities and integrating them with world knowledge to perform inference. To ensure this, each question must involve at least two distinct visual evidences- orientation, action) relationships (e.g.. events, interactions)- and must invoke knowledge triggered by visual content. Trivial tasks like unconditioned counting or basic captioning are excluded.
as either specific attributes (e.g.,
To ensure diversity and evaluability, questions created within of five designated reasoning types. Causal Reasoning involves identifying cause-effect relations in the video, such as "Why did the person fall after stepping on the mat?"; Prediction involves anticipating what might happen next, e.g.. "What will the person likely do after picking up the toolbox?"; Counter-Factude Reasoning asks about hypothetical alternatives, such as "What would have happened if the person had not pulled the lever?"; Assessment asks for judgment based on visual evidence and criteria, such as "Which athlete demonstrated better form during the lift?"; and Planning requires proposing a viable plan grounded in context, such as "What should the worker do next to safely continue the repair?"
Step Annotator Qualification and Tasks. Annotators were selected based on domain expertise. For specialized domains such as Education, Sports, Electric & Crafting, and Transport, annotators held relevant undergraduate degrees or had least three years of experience. For general topies, annotators were computer science graduate students unaffiliated with the author team. All annotators underwent standardized training and trial tasks.
Each annotator was responsible for formulating open-ended questions that met the complexity criteria and fell under one of the designated reasoning types. They provided concise, direct answers and explicitly detailed the reasoning process, identifying essential visual evidence and associated knowl- edge. Annotators were allowed to flexibly choose relevant video segments to preserve contextual richness while avoiding oversimplification To maintain fidelity in meaning, all annotations were first created in the anmotators' native language and later translated professionally. Based on reviewer feedback, annotators were allowed to revise their submissions once.
B.3.2 REVIEW AND QUALITY CONTROL PROCESS
Step 4: Initial Review. Trained reviewers manually evaluated each submission for objectivity, factual correctness, conciseness, and reasoning complexity. To encourage proactively seeking the visual reasoning evidence, a focal point in this process was eliminating reference redundancy, where questions explicitly mention imply the essential visual evidence, thus undermining the need for true reasoning. Instead, reviewers encouraged general references and annotated potential revision points with detailed feedback. Annotations that passed this stage without major issues advanced to the next round, while others were returned once for revision or rejected outright if irreparable.
Step 5: Meta-Review Meta-reviewers, typically senior researchers, conducted second round of checks with higher threshold for complexity and reasoning depth. They ensured each annotation incorporated at least two distinct pieces of visual evidence and required the integration of external knowledge triggered by visual content. Annotations relying solely on dominant cues or surface-leve understanding were filtered out. The number of visual and proactive evidence per question was also counted and finalized after cross-validation.
Outcome. This layered quality control pipeline, with stringent emphasis on complexity and reason- ing integrity, resulted in a final acceptance rate below 20%. Due to the meticulous verification process, the average cost per question, including annotation and review, reached approximately $12. The outcome was a set of 442 high-quality reasoning questions, each crafted to challenge and benchmark advanced visual understanding capabilities.
24

---

Published as a conference paper at ICLR 2026
B.4 ANNOTATOR RECRUITMENT HUMAN-R CONSTRUCTION
As detailed in Table 7 Human-R covers 11 diverse domains. In constructing this dataset, we prioritized domain relevance and expertise in annotator recruitment over a rigid "one-size-fits-al standard, so that the questions and reasoning chains reflect realistic domain-specific characteristics rather than generic templates.
For specialized domains (c.g., Sports, Electric & Crafting, Transport), we recruited annotators with relevant practitioner experience educational backgrounds. For instance, sports-related questions were assigned to fitness coaches or student athletes, while Electric & Crafting items were handled by annotators with engineering-relan training or degrees. This ensured that both the question design and the associated CoT--style reasoning remained faithful to how domain experts would actually reason in these settings.
For general domains (e.g., Household, Daily Spending), we instead engaged high-quality annotators with strong performance in our qualification trials (i.c., high pass rates), without requiring formal professional credentials.
Overall, the dataset was constructed by a dedicated team of annotators recruited specifically for this project. We ensured that each domain had at least two annotators. Each annotator assigned to domains aligned with their background and was required to pass qualification trials based on our annotation guidelines before contributing to the final dataset. This process helped ensure that domain expertise and annotation quality were systematically controlled across all domains.
C EVALUATION SETUP
C.I CONFIGURATION OF EVALUATED MODELS
Table detail the configuration of each evaluated models on HumanPCR. Across all experiments, we use the default settings from the official implementation of each model to process vision input while the temperature is set to 1.0 and the maximum output length to 1,024 tokens, except for proprietary reasoning models (e.g.. o3 and Gemini-2.5-prm 2.t.iii-2--ro---eeiiiiiii proo-prevee).. for which the maximum output length is extended to 4,096 tokens to accommodate their extended CoT reasoning mechanisms. Additionally. for the Human-R evaluation, we tested proprietary models and open-source models larger than 8B with frame numbers of 32, 64, and 128, and selected the configuration that yielded the best performance. All experiments are conducted using LMMs-Eval (Zhang et al.) 2024a) on 8 Nvidia A100 GPUs.
For experiments context extraction and RL-based thinking methods, we implemented their results by their official repository. Table Tineeishhnn shows their configuration.
C.2 EVALUATION PROMPT
We present the prompts in the evaluation in the following figures, for answering multiple-choice (Direct Answer in Figure I7 and CoT in Figure [8] and open-ended questions (Direct Answer in Figure 19 and CoT in Figure 20). respectively. The prompt for accuracy evaluation is presented in Figure 21
C.3 RELIABILITY LLM-BASED JUDGES
To ensure the robustness of our evaluation methodology. we tested multiple LLM judges and bench- marked their performance against human evaluation. For the human judgement, four annotators scored 4,000 model responses for per-instance accuracy and pairwise win-loss preference. The win-loss preference for I.LM judges was determined by comparing their assigned scores across instances. Figure 10 in the main paper illustrates the evaluation results from four distinct judge models: 03-mini, gemini-2.5 2.5-laah, gpt-4o, and o3 and our findings are listed as below:
High consistency in relative model rankings across all LLM judges Performance orders of the evaluated models remain stable regardless of the judge employed, indicating that our evaluation outcomes robust and not dependent any single judge.
25

---

Published as a conference paper at ICLR 2026
Table Configurations of evaluated MLLMs on HumanPCR.
Input Frames (P&C/R)
Organisation
Model
Releuse
Version
Open-source Models
Rhymes
Aria
2024-11
Aria-Char
32/64
IntemVL.2.88
2024-6
InternVi2-8b
32/32
V1.em
2024-11
InterrV12.5-15
32
IntemVL2.5-2P
2024-11
InLer=VL2.
32
Shanghai AT Lab
IntemV1.2.5-48
Interry12.554R
IntemVLL..58B
2024-11
Inter:VL2.5-11
32
IntemVL2.5-388
2024-11
InternVL2.
32/64
1n2.m8L2..5788
2024-11
S-
32/64
IntemV1.3311
2025.4
InternVi:3118
32
20254
InternVi3-2B
32
1otemVL3-8B
2025-1
InternVL3 6B
32/64
IntemVLL3-14B
Tntern811-118
32/64
1ntemVL3338B
2025-4
InteraV13 3B0
32/64
2025-
InternVL3-7BB
32
Owen2-VL-7B
2024-8
Qwen2 VL Instruct
Allbaba
Qwen2.5- VV-
2025-2
Qwen2. Swen2.5-vu-78-1nee 5-v--7e-Innttrce
32/64
Qwen2.5.VL.
LLaVA, ViXee 34B
2024-5
LLoVA--9eXT-V:dee--32
32
Imms-lab
Visionnn
2024-9
32/128
Viaio--ne
2024-9
1lava onevision qwen2
32
LLaVA-Video-78
2024-10
VLaVA-Video--B--0e
32/128
LLAVA-Vido-722B
2024-10
LLaVA
MiniCPM-V.2.6
2024-8
MiniCPM-V-2_6
32
Op
MiniCPM-o-2.6
2025-1
MiniCPM 2_6
32
Oryx-1.5-7B
2024-10
Oryx-1.5-77
32/64
Thu
Oryx-1.5-32B
2024-10
Oryx 1.5-52B
32
Nvidia
Long LongVILA-256
2024-12
qwen2 7a loagvila 256E
128
2024-12
Proprictary Models
GPT-4o mini o3
2024-8
gpz 06-min1-2025-04-11
1p=-4o-2024-0B-00
32/64
OpenAl
32/64
2025.4
03-2025-04-16
64
Gemini- 1.5. Flash
2024.9
gemini-1.S-flash
32/64
Semini-1.5-Pro
2024-9
gemini-1.3-pro
32
Google
Gemini 2.0. Flash
2024-12
gemini-2.0-flach
32/64
Gemini- 2.5.
32/64
Gemini-2.58 Comnini-2.5-Pa.
2025-3
gemial 2.5 preview 03 25
64
Claudo-3 S-Sonett-22
2024-10
32/64
Anthropic
Slaudd-3.7-Somme
2025-2
claude 3-7---3---00000 7-s-nnnet 20250213
64
ByteDance
Doubaoo1.5-visioee
2025-1
doubao 1.5 vision 32k 250115
32/00
xAI
Grok-2-Vision
5024-12
brok-2-vialor- 1212
32/80
Question: {question]
A: {option a} (option_It
C: (option_e) c)
D: (option .)
B: (option_e)
Answer with the option's letter from the given choices directly.
Figure 17: Direct Answer prompt for multiple-choice questions.
Strong correlation between LLM--based evaluations and human judgment The Pearson correla- tion coefficients for accuracy are consistently high (0.640 to 0.689), and the alignment for pairwise win-loss preference even stronger (0.832 to 0.872). We also found that providing judges with
26

---

Published as a conference paper at ICLR 2026
Table 9: Overview of evaluated context-extraction strategies and RL-based thinking models. The extraction granularity of the context of these strategies and all official code resources is also presented.
Model/MMthhdds
Baseline
Strategy
Granularity
Cut. Repusitory URL
Video La 2024
LLaVA Video 7B Rettrry&Retiivee
Frame
httpa: / //gthhh.
com/1eon 1297
Video RAG mastec
AKSrTing 2025
LLaVA-Vidoo-78 Frame Selection
Frame
128
Sttps::/ htttp::/esthub..
com/ com/ncrimiang ARS
FRAG FRang et al.
2025
Qwen2.5-VI.7B
Frame Selection
Frame
32
nttpa://githip
com/ NV abs/ PRAG
T"
Yo ot 2025
Qwen2.5-VL-7B
Frame Selection
Frame
32
attps: / //ghhbb.
com/m lab nu/
1Sta:
PVC Yang al. 2024a
LL2em
Down Pooling
Token
512
sttps://sithub
com/ Open/VLab/GG
Livii--e Zhang
2024b
VL2er8VL2-8B
Meemory& Rettrivve
Frame
Sttps://eithub
com/ Innnerii
Tntern Xcomposer
LongVU Shen et al.
2024
LLaVA-Vidoo-7B
Token Selection
Frame & Token
fps
Sttps://gitue
com/Vialon-CATT LoneV]]
OlafTán et al. 2025)
Qwen2.5-Vt.7B
Down Pooling
Token
64
sttpa: /gith/hh com/ 01a 0mn 0mn1/00 1/01a
Dian al.
2025
LLaVA-Vidoo-78 Memory &etrie&Retrievee
Frame
512
attps: / /githu.. com/ 221ng/ Diapider
AdaReTaKe Wang al.
2025h
Qwen2.5-VI-7B
Token Selection
Taken
4
sttps:/ /gbhh
com/ 8c2wangxiea
video- 2 dlexRedus
FlexSelect Zhang al.
Qwen2.5.VL. VL.7B
Token Selection
Token
2 Eps
Sttps://
gith.a.com
yunzhushang0318/
Elexselect
Video RI Feng et al.
2025
Qwen2.5-VLL VL-77
Thinking
Sttps:/ //gtthub.
com/tu erfeng/
Video 31
Vidco-RFTRR Feng al.
2025
Qwen2,5-VL-7B
Thinking
32
attps: : / /githb..
com/ Liuzi iuuzlyu77
Vigua
VideoChat-RR R 2025a
Qwen2.5-V1.778
Thinking
pat httpa://eithup //gthhu
com/ LoenGVLab/
VideoChat R1
Question: {question]
A: {option_ee }
{option_i) b)
C: (option_cr c}
D: (option_d]
B: {option_e)
Based on the given (modality), select the best answer to the multiple-choice question by thinking step by step. Begin by explaining your reasoning process clearly. Conclude by stating the final answer using the following format: "FINAL ANSWER: $LETTER" (without quotes), where SLETTER is one of the option's letter from the given choices. Think step by step before answering.
Figure 18: CoT prompt for multiple-choice questions, partially adopted from previous works (Zhao
et al. 2025 Yue et al. 2024). "modality" ranges in "image", "multiple images", and "video".
27

---

Published as a conference paper at ICLR 2026
Question: {question]
Answer the question based on the given video. Do not generate any intermediate reasoning process. Directly output the final answer.
Figure 19: Direct Answer prompt for open-ended questions.
Question: (question)
Answer the question based the given video step by step. Begin by explaining your reasoning process clearly. Conclude by stating the final answer using the following format: 'Therefore, the final answer is: "Answer: $ANSWER" (without quotes), where SANSWER is the final answer of the question. Think step by step before answering. Below are frames uniformly sampled from the video."
Figure CoT prompt for open-ended questions, partially adopted from previous works ZZhao et Yue et al. 2024).
2025
annotated CoT as a reference significantly improved their alignment with human annotators (e.g.. 0.642 0.689 for 03-mini W or W/o CoT accuracy).
Self-Bias Test. We further conducted a Welch's t-test on whether judge might favor models from its own family. The null hypothesis assumes that the score deviation over averages other models of a judge for models from its own family is not significantly different from that for non- non-family models. With a 0.05, the test yields high p-values (e.g., p=0.864 for 03-mini judging o3; p=0.774 for gemini-2.5-flass on Gemini-family models), leading us to fail to reject the null hypothesis. This indicates very little statistical evidence of self-bias in our evaluation setting.
Collectively, these findings validate that in setting, LLM judges serve reliable and robust alternative to human evaluation. We attribute this to the use of precise reference answers and detailed reasoning traces, combined with modern LLMs' strong instruction-following ability, which helps eliminate subjective noise and ambiguity in evaluation.
C.4 EVALUATION SETTING OF HUMAN PERFORMANCE
We have benchmarked human performance using subset of HumanPCR. For Human-P and Human- C, over 30 questions are sampled for each task while the evaluations were conducted by annotators who were involved the construction of the benchmark. For Human-RR R, accuracy was computed based on approximately 10% of Human-R questions completed by two graduate students in computer science who were not involved in the construction of the benchmark; during the annotation process, the annotators were allowed to consult search engine for factual lookup translation, but was instructed to refrain from locating the original source video or any accompanying ancillary materials by any means. The annotators received performance-contingeen incentives: correct answer paid twice the amount awarded for an incorrect one.
28

---

Published as a conference paper at ICLR 2026
Evaluate the accuracy score of the model's answer and whether its final answer is correct by comparing the ground truth answer provided for the given question.
You should first extract the final answer from the model's response, and then compare the extracted answer with the ground-truth answer to determine its accuracy. The final answer of the model does not need to match the ground-truth answer word-for-wort It should only be considered correct
1. if the final answer of the model demonstrates the consistent meaning or concept equivalent to the ground-truth answer.
2. if the final answer of the model meaningfully includes the ground-truth answer and does introduce fundamentally different or unrelated meanings.
3. if the final answer of the model is meaningfully included within the ground-truth answer and only differs by omitting explanatory details instead of lacking key concepts.
Then, you should provide the accuracy score of the answer, which ranges from to 4, based on both the conrectness of the final answer and the accuracy of the reasoning process.
If the final answer is incorrect, the score must be 0, or 2:
0: The final answer is incorrect, and the provided reasoning/evidence is either missing or entirely dissimilar to the groundtruth.
1: The final answer is incorrect, but some visual details, reasoning steps, or evidence partially overlap with the groundtruth; however, most of the reasoming incorrect. 2: The final answer is incorrect, but the majority of the reasoning process, including key visual evidence and logical steps, aligns with the groundtruth, with only minor deviations causing an incorrect conclusion.
If the final answer matches the groundtruth, the score must be 4:
3: The final answer is correct, but the reasoning process supporting evidence significantly differs from the groundtruth.
4: The final answer is correct, and the reasoning process, including supporting evidence, closely aligns with the groundtruth without major inconsistencies.
Output your response in the following structured json object format:
"extracted answer' // str value, the short final answer extracted from the 'oddes response, do not hallucinate one that not present in the response, correctness's // boolean value, True if the answer is correct, False otherwise,
sscoee; // int value, overall assessment of accuracy of the model's
Input Format:
Question: {question]
Ground Truth Answer: (ground _truth]
Model Response the Question: (model_response,
Figure 21: Evaluation prompt used for assessing the accuracy of open-ended questions.
29

---

Published as a conference paper at ICLR 2026
D MORE DATA STATISTICS
Detailed video duration statistics across tasks Human-P&C. domains and question types in Human- respectively. The average numbers of visual evidence across different domains and question types in Human-R are illustrated in Figure 23 The modality
R are presented in Figure 22a
22b
22c
distribution of tasks is presented in Figure 24
166
500
x
103
11.
Beoule Proetien
Pariii Romen bopuentt Achee
Exesiotyy Breeer Avine
cring Anverni
decer Besevin
Erees
Hen rocil Copersiicing
despeesta Aetie
Cance Rabonso
(b)
(c)
Figure 22: Statistics of video durations in HumanPCR. (a) The average durations for tasks in Human-P and Human-C. (b) The average durations(bar) and number of samples(line) for domains in Human-R. (c) The average durations(bar) and number of samples(line) for question types in Human-R.
5.50
Retorme
Prasciva
4.51
4.38
4.00
3.50
3.75
3.58
Anunge
Spansing
Bremii Charting
Infor Leiere Sief Care
Housthold
Others
Outaiior Adverbrre
Duddiaar Lesaae deermatton
Perfomance Chribston
Sports
Panssport
Asmestment
Caure Rataoning
Counterr Fareoal Beaaniing
Planning
Redidtin
Figure 23: Average number of visual evidence in Human-R across (a) domains and (b) question types. The referred and proactive evidence are denoted by different colors.
E PUBLIC RELEASE AND DATA USAGE TERMS
To ensure minimal privacy and copyright issues while enabling open access for the research com- munity, we have established the following Public Release and Data Usage Agreement (DUA). We publicly release all annotations created by (and any data for which are the rights holder) free of charge for academic research under our DUA, with the additional requirement that any use involving the original visual data must strictly comply with applicable privacy laws, institutional review policies, and obtain all necessary permissions. The DUA we formulated for the dataset open-soure dataset is also presented in Figure 25
Scope and Husting. We do not redistribute raw third-party source media without explicit permission. For videos and images from public platforms (e.g., YouTube),, release only factual metadata (e.g..
30

---

Published as a conference paper at ICLR 2026
600
Image Muttipe Images
Video
500
Number of samples
400
300
200
100
Spatial Relation
Object Exstence
Htman Prodence
Eudy Porture
Henm ripeet Interartiin eeitiinn
Hard State
Busty Drrentation
Gape Ertonation
cletting Aitrause
Acceatory Recoynaton
Bodypprt Veibiity
Plyykcal Attribua
Humery Kemer--bbeet Contect
Huran Hurtas Contact
Homan Self Contact
Pace Reet eygetitonn
Ideenity Chaitering
Gesturg
Knowifegn Basel Avaton
Emoston
Basic Artian
Sequrrtal Action
Muttlee Human Savuanic Action
Precertom Dependence
Goal Planting
trminsent Anton
Human Compar
Social Reattn
droup Actevty
Stroppio Derection
Crosed Event
Cultoral Evrnt
Disscter Indnstarding
Crima Abrormal
Figure 24: Distribution of samples of each modality across tasks.
video IDs, start/end timestamps for clips), and our task annotations. Source datasets that explicitly permit redistribution are hosted directly; those that prohibit it are provided via metadata, processing scripts, and links to original sources so users assemble content themselves.
Access Control and Backup Channel. controlled (backup) access channel requires explicit agreement (i) all upstream dataset licenses, (ii) non-redistribution commitment, and (iii) Data Use Agreement (DUA). Users must request access and may have it revoked upon breach.
Data Use Agreement. The DUA presented in Figure 25 forbids: re-identification, biometric template construction, surveillance or tracking applications, discriminatory or harmful usage, circum- vention of platform Terms of Service, and repackaging or further redistribution of third third-party content. Data are for evaluation/research; we do not train models on the benchmark annotations.
Privacy Minimization. Our annotation guidelines have excluded personal names, private locations, and sensitive attributes; quality control removed residual high-risk items. No biometric embeddings or identifiers are released. If source video is deleted or privatized, its entry becomes unusable.
Copyright Requests. Rights holders may request modification or removal of hosted data at any time; validated requests are processed promptly and reflected in the next version.
Disclaimer. The benchmark is provided "as without warranty: users bear responsibility for legal and ethical compliance. Continued use constitutes acceptance of updated terms.
31

---

Published as a conference paper at ICLR 2026
HumanPCR Data Use Agreement (DUA)
Centent Hosting and Download
The dataset only centains metadata (e.g., public video IDs and timestamps) for content sourced from public platforms.
We do not host distribute Users are responsible for using the original videos themselves, following the Terms of Service of the source platforms (e.g., YouTube).
video files.
License Compliance for Hosted Data For any datasets or files directly hosted by us, by accessing or using HumanPCR you automatically agree to comply with the original Licenses and usage restrictions of those catasets.
Usage Restrictions
HumanPCR is provided for **non-commercial, academic research
purposes** only.
You are Astriccly prchibited from using the data biometric identification, tracking, or surveillance technologies.
You must not attempt to re-identify, de-anonymize, or contact any individuals depicted in the images or videos.
The data may **no** be used to discriminate against, harase, negatively profile individuals or groups.
Without prior written approval, you may not redistribute, publish, or disseminate the 'datasets metadata, including video and image data, in whole or in part; task annotation inns created may be stribttb for non-commercial you 1ink to this DUA and indicate any changes.
with attribution, provided
Attribution requirement: When using HumanPCR metadata or our task annotations, you must give appropriate credit, provide a link to this DUA, and indicate if changes were made,
No additional restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the DUA permits.
4. Copyright and Removal Requests
All copyrights remain with the original content owners. If you are a rights holder and believe there is an issue with any content referenced in EumanPCR, please contact us. we will promptly review and remove update the relevant content as requested.
By accessing or using HumanPCR, you acknoxledge and agree to comply with the above terms.
Figure 25: HumanPCR Data Use Agreement (DUA). This document ensures that the dataset is distributed responsibly for non-commercial academic research purposes only.
F MORE RESULTS
F.1 FULL RESULTS ON HUMANPCR
The per-task accuracies of the evaluated model are presented in Table TO
32

---

Published as a conference paper at ICLR 2026
Table 10: Full resulls of evaluntedd models Sandllson on HumunPRC for each tusk. The healer of each dimension (e-g. g.. Spu. for Spatiality and each task (e.g. S.R. for Spucial Relution) are abbreviated for brevity.
Open
Roa.
B.A K.B.A.A. S.A. G... PD MIMM LA H.C.R.L G.A. S.D. C.E.C.I C.M. Rea.
45 25,005225200 LAW5 N6 07 3077 56.36 35 22.222222222 54 N6 2006
77 72 13 20 77 68 70 540556, 3021 19
25 7OL 22 61 30000 15.5 NO
2104
150000 1600 48 19 IN
36 65.67597266 7661 31 72 25 1346 1078 5-22 1656 1636610000000
38 31-91287711 1W80 10 TI TAE MONI MAN SAAA MO
35 4EE 701516
8S 1765
PA 1922:99
1 SS
19 - 3394
W.N-2 22 1199
39 H8 SS 65 20.76
SE51AS
ROW ROY 262375
65 1957.56
50,006 55 1110661-1155..555 1145515556665 20 2611 335.71 76.96 5949.4 32X1
40,0055 373770665 3601 20 1R STAID De 73 AA O2 70A9 O 36-30
3000000 7 OF AND OO T AO W I 3A 400
1:40
S9LG 75 25.223 51 1.555
36 O7 S O7 OF OF LOS 7 764
28 38.35 1962 150 AA D AN DO S 7 TA
Sox.
74.35
00002
40,0063 10 LAN XINK TTM
30 0006 1777 5550 SE AR P
Mat.
41 107T000 INROOUU UN
06557 06ST
53 03.07- W2 62
30 10000200100
41 68750 TOS
Comprebensionn
555
30.005 080000885777
35 3689
45 (0ST
67.5
26,00 04.055442
25 00--
25 00-5
41 41.6759
STOTE
21
28
28
Pmn.
72.00 NI 10.08 41 15.8 2022266333 34 43 23,0044 1222 2942.5
IK 1527.59
17
19.1111122
1264
41.03 39 222222222111 01612122144
73.1762.00 00 05 25.26- 5351 22335 34 45 35-3- SL 1966962.007
42.12 12--12-00-000 3O--0 16
45 A2 55 190999022000
79-89 18.82 40.15 26.37 12,0599 00000 30.51 32 416777 31.71 21 383860061111 1 12922000
51-2531000 100 27 PLEEESS-T 42360W ON 39230 29228-1--8611 10 12
A01R0ONN 6P- AS 10000 229 1X2171120
37.38 37 1855532 1/
32-0EP 2E-0EPU--L -2-.-------- WO 20 46. 16-10--0--0 2013832200
TE 9991 25T01-5 SS. 23055 A1 214159977
58 1341 E853 17757700 2017--00017773333 1763333
22002000 34.442 1000
IOLROSSI 36 26-00-0099 99112203
SALEEI 39 24 20000000 10 12222A4
116666166
36.76 1517
25 45
42-4 5125 13
5131 1655
00 13 22057771
60 66 56 22007333
41 50 11..00444111191111111111 1931150577
47 17.39-1 6711117791111119
1000.491155 530220
45 30300 35 1 L 1T
30 1600 1012-1122333
463444 6343 00 63 16-88 18 33 31 ST 52,5242 1623 122177255
11 53 11S51352244 1222777771
53 $3.92200 201203210177300000
83 R 57 S90055 ss SU190003
37 37.2231 7233311331
76
44 44.9936676
45 42500
2656 36 40
45 10-4
40.00
51
Boh..
50.550000 00 25 25.06440 35 4850
37 89:38555 193393566
00 79 78.054 2458
70.53 303333011 1155 20002000000
22--3-2-----22--- 4344--44422 300000
77. N9.99 2162556
29-47-11 24
20000
20 20
32.9372 00 68 07 22
52-7666
47.06 59.76766 00 95 7952 19 19 2273
00 96 81 MS 50.7472.8
13.1 N5.00 71.5 71.58455 N033547763300
19.51 72.00 28-42 40 15 N0.957.76
50.53 00 72452223558 05..16333550
35 29 77 00000 37 89730 30 -7.45 3.600
W.T1 200227 222200000 00 to 05 16 35 83.60
48 53 22.197.00 00 00 0051 46 AR 2864
60 1671 174474222 20 30 32 33 SWG 4N% 3248 42.26 46.97 36.266 17500060000 24 51 10 28
J.C. G.U.
29 23.1766 00 45 55.20433
6212 36 24590000 us 75.7966
Models
Progristory Medhel
25 DO 30
S.R. H.P. B.P. I.S. ILO. GE C.A. A.R. WWW.PA.....C............. C.S.C. FR
Open-searre
50.0058
52.9862.00
31.0072800
26.8367.0
766736600
31.7150
21-2520000
24.39000000
35.777000
195166 DO
62.0009
43 3339076..0
40.2482.000
20.00
39.02.50
Con.
29:41
20.85
85
32.04
36
30.88
31.35
39.24
41 WN
27.04
23
44.20
4697 20041
26-77
24.35
2658
47/06
22.06
23.55
27.91
42.65
44.12
45.59
29.41
50.00
52.27
47.73
47.73
40.91
46.70
40.15
62.12
4300
64.29
4242
5935
4773
1621
0212
20 41.8 5000
2340
42.55
42300
70445
62588
4091
4167
B1 60.11 12 53.70
59 0845 67.4
65.9
55,30
58.33
66.39
42.42
57.58
71 102 WHTT 60.99 6061
5994 2637 50 63.10 6867
55.05
67 52 1477 OR 25 S4 46 25 23 66 70 4990 4990.96 35, 16 56:47
55.88
DOE 38 1635 38 61.75
70.00
7AM8 52 22 54.70
71134
20 10:00
22 54.76
162 55.95
23 00059
38 55.95
II 44.12
3 0254
1N 50.41
05.29
84 06:01
25 47.00
57.74
67.36
70.21
00771
45.83
20N1 90 2225 61.90
5 77.35.0 50 50.50
57.14
DR 1E 66007
69 26 oR 29 T0 20021 26 55.55 79. ENN 53, 33,00
50
77 2ON7 M0 NI
BI
67.25 2/ 28 1952 3T 53 33.51 83 1900 16
BESA NO00000 20
03
26 19
21 31 08 5K WILL3 3100 98 36.72
O ON 20 12
App-
39
37 37 A067 21 7M 37.
Percepatio
32 59
28
Pos.
5%
3L
64.1679. 31 PALISAS 222
36-25
72.50 09.99.99 31 76 96 5151
60. 22 NO. 51 4A440T
N0 96 800S
74.14 1444422222 29 DE MEE
41 04 70, 69 763-
kZ 3543
TP. 77 7OX1 36000
F2 22.00 pe 6
61 1616688 78
LEAVA LL Wws.ooN------- 728 75577 72 NAN 21 NSS
LLAVA -NNNNNETTENNNN Viloo-- 3-1---16--651 55.9 1162
61 22 MANOL
Quent VL--B------- ----7B---nn En--oatttott 65 500 16
70.11 N4 CONET
20.35572 72 99060
Spa.
30 00-66655 53
51-15% WA
63.22 NA
60 80
73 99
78 61
81 50
50 35
67.00 OS
78
Doubun- 1.5-4isinn-po. pro
LLLVA- www.Oiieiioooooo.
Claude- Someeee
Lome VILA ATILA/28 256
MICCPM-V V.2.V-2..
CPM-0-20 2.66
LL WWWA-Nl00-700 78
LLVA LLVVA- Vileo 72B
Orye 1-5-7M
O5yx- 1.5 $23
20002-5-V2.. 718
Owen2.5 2.2.5..V... VL.
ImmnVi22 28
ImeenV 2...3-5..34 4B
imsnV245-88 389
M2 7888
Inten 1.
3
InterN 1B
Grok-2-V5slnn Visiom
Gemini-1 1.55iish
Gemini-L5- 558hro
Gemint-2.00 0-Fiish
Gemins- 3.5Fis-----Flls 3-5ihishh
XVILA
Medels
04-mini
Aria
M
GP
33

---

Published as a conference paper at ICLR 2026
Table 11:
Uncertainty Reporting. The average accuracy, standard deviation, and 95% confidence
intervals for Qwen2.5-VL--72 (Bai et al. 2025) and GPT-4o (Hurst et al. 2024).
Multi-Choice
Open
Human-P
Human-C
Human R
Spa.
Pos.
App.
Con.
Ide.
avg.
Beh.
Pro.
Rel.
See.
avg.
Acc.
Acc.
M-en2.5-VL-726
Mean
81.72
49.96
67.13
42.69
41.21
$7.90
55.21
47.96
51.50
61.93
54.75
56.29
34.31
Variance
0.15
0.01
1.90
0.99
0.06
0.01
0.24
1.62
2.26
4.73
0.06
0.01
0.32
95% CT Lower
80.75
49.79
63.71
40.22
40.58
57.63
54.00
44.80
47.76
56.52
54,
56.09
32,90
Upper
$2.69
GPT-4o
Meun
70.63
42.25
56.09
32.72
40.51
48.62
52,05
41.28
50.05
60.71
50.45
49.54
41.02
Variance
0.34
2.29
0.32
2.21
23.53
1.11
0.02
4.17
1.12
0.42
0.94
1.02
0.12
38.50
98% CI Upper
72.07
46.01
57.50
36.41
52.56
51.24
52.40
46.35
52.68
62.31
52.86
52.05
41.88
F.2
RESULTS ON HUMAN-R ACROSS QUESTION TYPES AND DOMAINS
The results of Human-R broken down by question types and video source domains are presented in Figure 26
Couster Rastaal Reoieesing
Indeer Lebere
Mdcattin
Dutdbor Adecntore
Daily Spcesbeu
Woutebald
Performacce
Ouddoo Leiour Recrestin
Cratting
Assessannt
sports
wransport
(a) Accuracy by Question Type
(b) Accuracy by Domain
Figure 26: Accuracy comparison Human-R across (a) different question types and (b) video source domains. Most models struggle with Planning- -tppe reasoning, while Education is the most challenging video domain.
F.3 UNCERTAINTY REPORTING VIA REPEATED RUNS
For representative models, we run each experiment three times and report, for every level and dimension, the average accuracy, standard deviation, and 95% confidence intervals. This provides explicit uncertainty estimates to complement single-point metrics as shown in Tbale II
F.4 COMPUTATIONAL RESOURCES AND RUNTIME DETAILS
To improve reproducibility and transparency, we report wall-clock runtimes and memory configura- tions across representative models and hardware setups. Our primary infrastructure consists of 8x NVIDIA A100 (80GB) GPUs. For models with around 7B paprameters (e.g., Intern VL3-88) under data parallelism on 8 Ax0000, Human-P and Human-C require approximately 30 minutes in total, corresponding to 4 GPUh, while Human-R requires approximately 15 minutes, corresponding to GPUh. For larger Intern VL3 variants, we run the full evaluation suite with the following configura- tions and costs: Intern Vieem 14B 1 X A100 takes 6 hours; Intern VL3-38B using model parallelism on A100 takes approximately 13 hours; and VnternVL3-788 on 3 A 100 takes approximately 16 hours.
34

---

Published as a conference paper at ICLR 2026
Table 12: Results of test-time scaling strategies on Human-R. Open-source and proprictary thinking models are both included. M represents the magnitude of the strategies. Specifically, the BoN strategy uses 2M candidates, while the
strategy performs M iterations.
Method
Reward Model
Direet
(CoT)
M=2
M=3
Test Test-Time Time Compute: Prompt Procedure
BoN
Gemini-2 Bemiini-.0--laahh 2.0-Flash (Self)
38.01
38.69
BoN
Gemini-2.5 2:5-ii-
37.55
37.10
41.86
Gemini-2.0-Fan
32.13
36.43
BoN
o4-mini
42.53
40.05
40.05
Seif-Retine
41.40
36.65
36.65
BoN
GPT-4o (Self)
43.21
45.70
45.93
GPT-4o
BoN
BoN
04-mini
Gemini-2.5-Plast
32.81
41.40
44.11
44.57
45.02
46.38
46.38
45.02
Self-Refine
39.59
39.82
40.95
BoN
VL (Self)
32.35
35.06
34.16
Gemini 2.5.
InternVL3-7888
BoN
o4-mini
35.52
37,56
38.68
41.40
42.30
BoN
GPT-4o
35.29
Self-Refine
33.71
35.07
34.62
Proprictary Thinking Model
Gemini preview Comanicii et 2025
Claude-3. 3.7 s-sunee Anthropic 2023
40.50
OpenAA 2025
G ADDITIONAL ANALYSIS
IMPACT OF MODEL SIZE.
70
Pearson Correlation 0.76
o4-mini
60
Gemini-2.5-flaal
Gemini-22.5ppo
Kerr 2.
Gemini-2.00 baa
Oward
Accuracy
Cemini-1.5-prr
Accuracy
10
50
Dras15
Garn 2-i-2-0-flast
Ger 6/20fo2
30
40-
20
35
90
10
88
32B
808
48
168 Parameters)
328
808
Model (Billion Paramebers)
(a) Results on Human-P and Human-C
(b) Results on Human-R
Figure 27: The relationship between model size and macro- avcro-average accuracy.
In Table scaling model size consistently improves accuracy across MLLMs series. As Figure 27 further reveals, performance Intern and In3ernVL3 suggests that complex reasoning tasks more sensitive to model size scaling, which brings steady improvements, while performance on Human-P and Human-C plateaus beyond approximately 38B parameters. We hypothesize that complex tasks more effectively evaluate the fundamental capabilities of the models, such world knowledge and logical inference.
G.2 FULL RESULTS OF TEST-TIME SCALING STRATEGIES ON HUMAN-R.
Table 12 show the effect of test-time scaling strategies beyond vanilla CoT on Human-R. Best-of- N (BoN) delivers over 5% gains all 3 models with incremental improvements from stronger reward models or more candidates. Self-Refine merely offers marginal gains and degrades with more iterations. Overall, test-time scaling strategies are generally effective.
35

---

Published as a conference paper at ICLR 2026
G.3 REASONING ERROR TYPE ANALYSIS
This section presents a structured analysis of error types based on comprehensive categorization of model failures from top-performing models. The error type distribution is derived from the average error breakdown of the results from various settings, including o3, o3 (8f), GPT-4o, Intern VL3-78B, 1irii-22.---r-PPeeieve Gemini-2.0-Flaat and 1ommini-2.0-Faast (1fps/1200f), computed over a representative set of 200 randomly sampled questions mentioned in the main text.
Visual Element Extraction Error (37.42%) This emerges as the most frequent failure type, highlighting fundamental difficulties in accurately identifying the necessary visual information from video contexts. This dominant category signifies that models often struggle to "see" the crucial components required for sound reasoning, either by failing detect key elements or by mistakenly incorporating extraneous ones. This category is further broken down:
Missing Proactive Evidences (21.87% of total errors): This is the largest sub-category, where models overlook essential visual cues not explicitly mentioned but to be extracted in the broader context. These include understanding the temporal order(e.g.. misidentification of repeated events and incorrect verification of completed steps), concurrency (e.g., inferring a person's 's state and actions from others' reactions and changes in scenes), and general causes and consequences. Such errors reveal fundamental weakness in holistic visual interpretation, indicating that models are not adept at spontaneously modeling context inner connections and structures.
Missing Referred Evidences (9.22% % of total errors): In these instances, models fail to identify visual elements that are explicitly mentioned or pointed to in the question. This challenge appears particularly in fine-grained actions (e.g., very brief actions background events) and ambiguous references (e.g., multiple similar candidates to references with only one correct match), even when clear guidance is given. This highlights the need for MLLMs to expand long context and more efficient context encoding.
This subtype occurs when models incorporate visual details that are extraneous or unimportant to the query at hand. This inclusion of irrelevant information can derail the reasoning process by introducing noise, creating confusion, or leading to flawed inferences.
Involving Irrelevant Evidences (6.32% of total errors):
Knowledge / Common-sense Error (36.63%) This represents the second most significant hurdle, occurring when the model correctly perceives relevant visual elements but lacks the factual knowledge (e.g., confusing distance and time in speed calculation). domain understanding (e.g., unaware that draw leads to penalty shootout and thus unable provide the correct score), or common sense (e.g., failing to explain that water cannot be poured because it has frozen due to low temperatures) needed to infer accurately and reach correct conclusions. This substantial percentage highlights that models' visual understanding is often weakly grounded in real-world elements or the logical principles required to derive broader implications or factual knowledge. This underscores the need for MILMMs to better adapt to expert human-domain data and uphold faithfulness during decoding.
Perception Error (13.04%). This occurs when the model explicitly misinterprets or incorrectly recognizes visual evidence that it has managed to extract, including misperception of visual attributes (e.g., incorrect classification of swimming styles). visual hallucinations, and confusion about attribute ownership(e.g. confusing subtitles from different video chapters). This type of error points to deficiencies in the model's 's core visual foundational capabilities.
Refusal / Incomplete Answer %)) This captures instances where the model either explicitly refuses to answer or provides partial, uncertain responses. Beyond built-in safety policies, we observe that models also exhibit selective answering behavior-ofter refusing to respond due to lack of evidence, especially when few frames are provided. Another cause of partial answers is the model falling into repetition or prematurely ending its output during long reasoning processes.
Duestion-understanddng Error (5.4%). captures failures caused by misinterpreting the query's 's intent or its constraints. For instance, models may provide generic video description instead of answering the question, or ignore required answer or reference formats- e.g., failing to use ordinal
36

---

Published as a conference paper at ICLR 2026
terms to identify actions, giving only an answer when rationale is explicitly requested. These errors highlight the model's fragility in instruction following and constrained decoding, indicating a need for tighter alignment between visual understanding, reasoning, and structured output formatting.
G.4 QUALITATIVE EXAMPLES OF REASONING ERRORS
We investigate error cases of Human-R, illustrated in Figure reasoning process and error type analysis of each model.
28
29
including detailed
Domain: Outdoor Leisure & Recreation
Annotation
Question: Why didn't the go
CoT:
In the video, the staff member dressed red seen shaking their head and gesluring towards two red different heights,
height. Ohserving that queun compased childran, becomas clear that the enforcss height lmmitaion. The combination these cusa the contood of the scana laacs to the conciusion that the woman did not moot the haight requiroment needed to participate the ride.
likely being
video?
Answer: She wesn't allowed ride because she too high meet the height requirement.
Answers
03: Step-by-stap reasoning
The woman walks up la the enrance of the final ride (trmes 13 13-19).
There is safoty sign and a heght-check post
Alter reading the sign she covers her mouth and looka disappointed (Famre 18-20).
2. Sine eaplains to the camera front the ride fframs 22-32)
Her gestures and facial expression indicate she telling the viewers why she cannot ride.
A ride operator then measures her with the height bar (frames 33-41).
Thee ber placed for below the of head, showiig she lalker than allowee height. (thmmss 40-41).
The operator shakes his head and she alepa out
Sine retums the only
that resigned, continues
from outside
(frames 42-45), contiring she did not ride. From
these
the
lifi.
Therefore, final
fall the teight
GPT-4a:
Interm VL3-78B
To determine why the woman did not on the last ride, Iwi" analyna the video frames step by step.
The women explains didn't go the last ride becsuse she too inches
night requirement inches, was only
Frames 10:
The woman she may be expariencing fear exclement.
green ride and repeatedly says "oh my god." Indicating
Therefore, the final answer "She was too short to ride It."
2.
Frames 11-16:
After ride, starde ds outside
entrance called The Dragon, She smiles
(Error Category: Perception Error
releved the talks, suggesting overwheimed.
after ride.
The model hellucinated, as the red height marker visibly lower then the women no height indicalor appeers video.;
17-22.
expression changes suggesta she might feeling
ciscomfort,
approaches another ride. appears to har, possibly explaining
something. ig. interaction seems about proaseding.
Fremes 34-13:
She continues nee to talk outside the rida, appeering desappointed Her gectures suggest she was not allowed chose not to rido.
Condusion:
From these obser diseevattons, is likely that the reeson the women did not on the
last ride related to regulation decision made by the park or herseff, possiblyy due to fooling unwell or ovenvheimed.
Answar:
The
lhely unable go the last ride due to feeling unwal park
regulation..
Category:
Extraction Error
Missing the proactive evidence of "the staff show the woman two red sticks of different heights")
Figure 28: An annotation example of Human-R - in the domain "Outdoor Leisure Recreation" and its error analysis.
37

---

Published as a conference paper at ICLR 2026
Domain: Electric & Crafting
DEWALT $09
X2
out $85.50
Annotation
How many of the tools introduced the usad for turning nuts them in the order in which they appear the video, such as "the first
Question:
on through
compassiive
these desigred surning nutt -pacthially the tourth toots shown The seguerce unaps futther contimm this the finst a Defftall gynacople elmee acrewthher, used divving screws the adcand is Temmpest pl-tierr conzas wrench a reichetingg
winhc designed tuen comporenta
a pair Brap outiing rg plens, cuiting salory intendedd
camsaped some heads saterr than stanccrd i.kkmiing.
Answer: Two, second and the fourth tooks are used for turning nuts.
Answers
Gemini-2 2. 2.5-Pro-Preview:
"Step Identity tools Inroduued video."
see which are presented indiviuuilly by the speaker. lools ap discuasn several tools
Obsarve
frames and speaker picks
one.
The first tool discuesed is the yellow DeWait power screwdriver (f (Frames 6-10)
The sccond tool discussed the socket wrondh/rathhet, kcetifodd as Tempest Oil Fitter Tarque Wrench (Frames 11-15),
The third tool discussed the small red handled too, which appoars wire cutters or strippers (Frames 16-19).
The fourth tool discussed the Knipae Pilars Wranch (Framos 20-24).
The fifh tool discussed is the pair of combination plers (f (Fremee 25-29).
"Step Determine the function of each introduced tool regarding tumingg nuts." **
The first tool (power screwdriver primarlly for drivnn screws, although with adapters, It could potentially tumn nuts, it's not main function presented.
The second lool (sochet wwwnchratd wtllqqe wnench) specifically designed for turning nuls, balls, and specific filling of filere.
The tourth tool (Kripox Plere Wirnnhh designed parallel lumnnng
The third sool (cutteraldrippera) for cutling prrppig
bott eftectivlly without rounding functioning
acjustable
The fifh (combination plans) grip but ideal damaga primary functions gripping. bending, cutting. generslly
primsey lool turring when proper wrenchee availeble.
"Step Count tools spectfically
tuming
design common function." tuming nuts/tttings.
second tool (sockot weendhratehhe)
Wiench) used turning The first third, and fith tools are not primanly designed or presented tuming nuts.
fourth tool (Kripex
"Stap Summarize the frndngs.
Two of the tozls Introducad in the video are used for tuming nuts: the second toal (socket wnonchitatchat) and the fourth tooi (Kripax Pilars Wranch).
"Answer: 2 (the second one and the fourhh ons)**
Gemini-2.6 2rmmint-2.0-Feehhyy O-ttto/((1fppaa/ (at////fpp///0000 2009. let' s analyze the video to determine how many of the tools shown are used for turning nuts.
The tools are shown on the table the following order:
pliora serreted jawa. pair cutting pliers.
Knipex pliors
DeWall gyrcecopic scrowdriver.
Tempest oll filter tonque wrench.
Now. let's determine which of these tools can be used for turning nuts:
jawa 2. outting pliers are dasigned for cutting and not sultable turning nuts.
first pliera
aspacially
damaged rounced.
3.Tha Knipex plaer wrench is spacificlly dasigned grip and turn nuts and balts.
4. DeWait gyroscopic inliee screwdriver designed for driiig screws and is not suttable for tumiing nuta. torque
5.The Tempest
dasigned for tightening filtes, which
shape.
Therafore, three of the tools shown can be used for turning nuts: the first pair of pllors, the Knipax plars wrench, and the l'empess ail filter torque wrench.
Answer:
(Error Category: Knowledge/ / Common- sense Error Due knowledge model mistakenly idemtified the first pliers (Snap cutting piirrs) suitable for handling
Figure 29: An anmotation example of Human-R in the domain "Electric & Crafting" and its error analysis.
38

---

Published as a conference paper at ICLR 2026
Domain: Education
Foot
Expanding
Drieding
misinformation to spread,
you would linagine the
out to your question
federcl government
bout it were real
innotation
What speculate about the man's profession in the video based on The image?
Question:
CoT: In the vidco, the man is seon explaining nearly ton sets of words and phrases, and he uses variety of complement descriptions, His continuous explanation and integration visual elements topics engaged
ilustrafive
merely describing online English teaching The presentation, combining detailed verbal explanations with visual support, characteristic of an instrucior delivering lesson, which leeds logically to the deduction about his profession.
sugpest
educational activity, typical
Answer: likeyy an online English teacher. indicated by Nis detalled word exp anations and use lustratvve images..
Answers
IL3077VL33-88-
The man in the video is likely an English teacher language instructor. This suggested by the educational content of the video, which focuses on slory about mystericus dranes. The man's presentation style, defintions, incorporation visual aids leaching vocabulary related profeesion. Addiionnaly, preserce microphome point
support
he conducting an instructional session.
Therefore, the final answer is: 'Answer: English teacher language instructor."
Gomini-2.0 Dimiini-220--Aaae 2.Fashhh:
speaking micrphone presenting
government
agencies,
appwars explaining analycing topics.
Thorafore,
commentator analyst"
(Error Category: Vizual Element Extraction Error
Involving irrelevant Evidences of 'the topic of illustratdd images ranges in drones, government agencies, and news events."!
Figure 30: An annotation example of Human-R in the domain "Education" and its error analysis.
39

---

Published as a conference paper at ICLR 2026
Domain: Household
000
1292
My
Annotation
Question: What caused shaking
CoT: The video shows man using muttimeter to lest battery that part of collection of mechanical components, which, based on the blue cap marking for windshield washer fluid, is ideniifed as being car's engine compartment, Initialy, engine had started and was generating power; process starting engine typically produces vibrations, and these vibrations likey transmilted to the camera during the measurement process.
battery vollage
12. 92V. which lator increased 14, 12V, indicating that the
Answer: camera because engine
vibrations were transmiteed to camera.
Answers
"Slsp-by-stee reasoning"
1. Look the frames:
person's hands working with multineters first workbench and later battery.
moving object running engine, machinery 3. The camera position clearly changes belween the workbench scene and the car-bot car-ballery scene, and sight shifls also occur while person's hands reach toward the mullimeter.
4. These emall jolls match moments when the person's hends are busy, imply the camera not foxed tripod but being held or sesisling).
reposilioned by
pereon (or 5. Thus the shaking best explained by normal hand movement the person holding the camera.
"Answer: The person holding moving the camera caused to shake." T*
(Error Category: Visual Element Extraction Error
Missing Referred Evidences- - The model missed the fine-grained referred evidence: "the abrupt camera shake after the engine starts", and Instead misideniifed carlior, semanticall similar molion as the evidence mentioned the query.)
GPT-4a:
Gemini-2.0- 2.0Fllhh
To determine what caused the shaking of the camera in the video, let's analyze the provided frames step slep:
shaliin Iialy
the while to the
the vitewfions
would slght
1. "Frames 1-2";
The frames st stable
workbench with
Therefore, final answer is: "Answer: The person holding camera battory."
batteries muttmattoss There's noticeable shaldng
opersting the multimeter connecting
suggesta S-4*t
(Error Category: Percaption
selling rgee engine bey with mutimeter pleced
The model hallucinanes, incorrectly identifeed that the operator was holding the camera while starting the ongine."
battery: enwionment remains steady.
"FFrames G-8"::
The scene continuss remain aleady a hand checking multimeter.
cemera acpwors
4. "FFrmes 24*h.
The person adjusts and attaches the mu muttmmatr leads the battery terminals. During this pariod, theie may ba subdle movements, but the camera does not mccitil notiosable shaking.
"*Concusion":
Since there are frames indicating shaking or instabiliy the video camera the question might based a mizundrrsanniing or Incomplesa data.
Therefore, the (inal
"Anawer: shaking observed."
Category: Element Extraction Error
Milaing Referred Evidences missing the referred evidence: "the shaking of camera"),
Figure 31: An annotation example of Human-R R in the domain "Household" and its error analysis.
40

---

Published as a conference paper at ICLR 2026
H THE USE OF LARGE LANGUAGE MODELS (LLMS)
In accordance with the ICLR guidelines, we disclose the use of Large Language Models (LLMs) as general-purpose assistive tools the preparation of this manuscript. Our use of LLMs was confined to two primary areas. First, during the initial literature review phase, we employed LLMs to assist in identifying and collecting relevant prior work and to generate concise summaries of several papers to expedite our understanding of the current research landscape. All referenced literature was subsequently read and critically analyzed by the authors. Second, the manuscript writing process, L.MSs served as advanced writing aid for sentence polishing to improve clarity and flow, as well as for conducting grammar and spelling checks. We want to emphasize that the core intellectual contributions of this work, including the formulation of research questions, the development of our proposed methodology, the design experiments, and the analysis and interpretation of results, carried out exclusively by the human authors. The LLM not used for generating key ideas, hypotheses, or conclusions, and therefore does not meet the criteria for authorship. The authors take full and final responsibility for all content presented in this paper.
41


Reviews:
[{"review_id": "PcP5ES8Pa8", "reviewer": "Reviewer_dhYK", "summary": "The paper proposes a fairly reasonable pipeline to curate complex VLM evaluation benchmarks across different difficulty levels (Figure 3).\nThe paper takes extra care to ensure that the quality and difficulty of the dataset is high. For instance, the questions solvable by blind LLM are filtered. In addition, the reasoning questions that one evidence, or lack of reasoning were not kept.  The paper tests a wide range of models in the open and closed world across several model scales. The fine-grained analysis reveals that only reasoning model o4-mini improves consistently with more visual context while the other models fail to do so.", "questions": "Mentioned above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a fairly reasonable pipeline to curate complex VLM evaluation benchmarks across different difficulty levels (Figure 3).\n\n2. The paper takes extra care to ensure that the quality and difficulty of the dataset is high. For instance, the questions solvable by blind LLM are filtered. In addition, the reasoning questions that one evidence, or lack of reasoning were not kept. \n\n3. The paper tests a wide range of models in the open and closed world across several model scales. The fine-grained analysis reveals that only reasoning model o4-mini improves consistently with more visual context while the other models fail to do so.", "weaknesses": "1. The process of human participation is not very clear at different stages of the work. (a) who are domain experts and for what domains? (b) Where are these domain experts sourced from and how many of them were there? (c) Did the same folks also solve the task for human evaluation number in Table 3? (d) Is human eval taken from the same kind of domain experts that made the dataset or are they average human workers on some platform?\n\n2. The writing-style and presentation could be improved. For instance, (a) it is pretty hard to read some figures like 7 and 8. (b) the spacing seems to be altered at many places – hardly any gap between paragraphs on page 3 and 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IedLytmwoy", "reviewer": "Reviewer_4oKa", "summary": "This paper presents HumanPCR, a hierarchical benchmark for human-centric visual understanding in MLLMs, consisting of Human-P/C (over 6k multiple-choice items across 34 tasks and 9 dimensions) and Human-R (442 open-ended video reasoning questions spanning 11 domains). A central design of Human-R is to require multi-evidence integration and at least one proactive (non-referred) visual evidence per question, with expert-annotated CoT rationales and rigorous filtering to avoid shortcutting via query cues. Evaluating 30+ SOTA models, the authors show that current MLLMs substantially underperform humans, with pronounced weaknesses in fine-grained spatial/temporal understanding and mind modeling, and that mere frame/context scaling yields diminishing or negative returns when evidence requirements grow. In contrast, reasoning-oriented models and test-time Best-of-N sampling provide consistent gains. The benchmark includes quality control, prompts, model configs, LLM-judge validation against human judgments, and ethical data terms.", "questions": "Can the authors provide controlled variants of Human-R items to quantify the marginal difficulty induced by proactive evidence, for example by (i) turning proactive evidence into referred cues, and (ii) adding minimal hints? Reporting Δ accuracy across models would causally validate the role of proactivity and identify which architectures benefit most. To disentangle perception/extraction vs reasoning/knowledge, have the authors run oracle-evidence experiments that supply the required clip timestamps or bounding boxes? Such upper bounds would localize the dominant bottleneck and guide method development. For judge robustness, would the authors consider releasing a small human-adjudicated “gold set” (balanced across domains and reasoning types) with reference CoTs to calibrate future LLM judges and track metric stability over time? Also, did judge agreement vary systematically by reasoning type (e.g., planning vs causal) or domain (e.g., education vs sports)? Given that adding frames can hurt when evidence count is high, have the authors tried structured evidence-gathering policies—iterative timeline search, subgoal decomposition, or tool-augmented retrieval—to mitigate attention dilution? Even preliminary results would clarify whether the bottleneck is context management or search policy. Finally, since several tasks touch on emotions and social relations, have the authors conducted bias or robustness audits across demographics and cultural contexts? A brief analysis would strengthen ethical readiness for real-world, human-facing applications.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The benchmark’s most distinctive contribution is Human-R’s explicit requirement for multi-evidence reasoning with at least one proactive, non-referred cue—an evaluation target largely missing from existing video QA benchmarks. This forces models beyond query-matching shortcuts and surfaces a realistic capability gap in long, complex, human-centric videos. The taxonomy across Human-P/C delivers both breadth and diagnostic depth; macro-averaging at task/dimension level makes failure modes actionable (notably posture, contact, procedure). The annotation and verification design is strong: expert-authored open-ended questions, evidence-grounded CoT rationales, blind filtering to remove textual shortcuts, and meta-review to ensure non-redundancy and proactive evidence. The empirical study is comprehensive: 30+ models, analyses of frame scaling, multiple context-extraction strategies, test-time scaling (Best-of-N vs self-refine), and a human-grounded error taxonomy highlighting proactive evidence omissions. LLM-judge reliability is quantified against 4k human ratings with high correlations and no detectable self-bias, supporting practical reproducibility. Overall, the work addresses an important and under-evaluated capability—human-centric multi-evidence reasoning—of clear significance to assistive systems, egocentric AI, and HRI.", "weaknesses": "While Human-P/C are valuable and thorough, their task space overlaps with prior perception/comprehension benchmarks; the strongest novelty lies in Human-R. The paper would benefit from ablations demonstrating added diagnostic value of Human-P/C design (beyond coverage), or clearer evidence that specific sub-tasks reduce confounds seen in earlier datasets. Human-R’s reliance on an LLM judge, albeit validated, still risks metric drift as judges evolve; a small, fully human-adjudicated gold subset would strengthen longitudinal stability and cross-paper comparability. The proactive-evidence constraint is compelling but mainly supported by qualitative/error analyses; controlled ablations—e.g., converting proactive evidence into referred cues, or masking question hints—could quantify its causal contribution to difficulty and discriminative power. Finally, given aggregation from public datasets and web videos, a more explicit analysis of potential pretraining overlap with popular MLLM corpora would help dispel contamination concerns and clarify generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KknyaM8toz", "reviewer": "Reviewer_c88T", "summary": "The HumanPCR dataset was constructed by first defining a three-tiered human-centric skill framework (Perception, Comprehension, Reasoning), and then sourcing video and image data from both public datasets (e.g., Ego4D, ActivityNet, MOMA) and recent web videos (e.g., YouTube clips from the last two years) spanning 11 real-world domains. For Human-P and Human-C, the team created over 6,000 multiple-choice questions using a mix of template generation, LLM drafting (e.g., GPT-4), and human review. For Human-R, they developed 442 open-ended video reasoning questions that require multiple pieces of visual evidence—at least one of which must be proactively retrieved, not directly cued—each paired with expert-authored Chain-of-Thought rationales. Rigorous quality control was applied, including blind filtering of text-answerable items, expert meta-review.", "questions": "1) How do you define “proactive” vs. “referred” evidence? Is this consistently labeled across annotators?\n2) Do models improve because they see more, or because they choose better?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) A reasoning paradigm that enforces multi-evidence integration and proactive evidence seeking, with rigorous filtering to avoid shortcutting.\n2) Most datasets just ask questions where the answer is directly visible in one frame or clip. This benchmark forces models to search multiple places in the video to figure things out—like humans would.", "weaknesses": "1) The paper says annotations will be CC BY 4.0, but the DUA also says “non-commercial academic only.” CC BY allows commercial use—this must be reconciled (e.g., CC BY-NC for annotations, or remove NC language from DUA).\n2) All answers are scored by a single LLM, even if correlations with humans are decent. To be rigorous, they should show confidence intervals, try multiple judges, and share prompts/seeds for transparency.\n3) Only ~10% of Human-R questions were answered by actual humans. To make a strong comparison, they should show how many annotators were used, what instructions they had, and how reliable those answers are.\n4) Many tables just give single accuracy numbers. They should report error bars, task-level variances, and statistical tests to show if differences are meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcHzy5VSkF", "reviewer": "Reviewer_9u8b", "summary": "The paper introduces HumanPCR, a human-centric benchmark for evaluating multimodal large language models at three levels: perception, comprehension, and reasoning. It contains over 6,000 multiple-choice questions across 34 fine-grained human-related skills. These reasoning questions require the model to combine evidence from multiple moments in a video and notice important details that are not directly pointed out in the question. By testing 30+ state-of-the-art models, the paper shows that even the strongest systems still fall well short of human performance on fine-grained spatial understanding, temporal reasoning, and intent inference.", "questions": "My questions have already been described in the Weaknesses section. I would like the authors to respond to these points in detail and clarify how they plan to address them.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and organized. \n2. The graph in the paper is clear and beautiful.\n3. The core idea of this paper is interesting and special. It fills the gaps in prior benchmarks, which were either too narrow or too broad without focusing on human-centric understanding.\n4. The dataset of this paper is carefully built with multi-stage human review and strict quality control. Each Human-R question must rely on multiple visual clues and include at least one piece of proactively gathered evidence, and only about 20% of questions are accepted.\n5. Extensive experiments have been conducted to demonstrate the effectiveness. The paper evaluates more than 30 state-of-the-art models and shows that humans still outperform the best model by over 15%, while many open-source models score below 30% accuracy on the most challenging Human-R reasoning tasks.", "weaknesses": "1. Evaluation setup is not perfectly fair. The paper evaluates many models, but they are run with different inference settings, so the comparison is not strictly fair. For comprehension, models only get 32 sampled frames and must answer directly, while for reasoning some models can use the maximum number of frames they can handle and are given stronger prompts plus tricks like Best-of-N sampling. \n2. Causal claims about failure modes are a bit strong. The paper concludes that proactive evidence extraction is the key bottleneck in complex video reasoning, based on manual analysis of 200 Human-R questions from top models; however, this is still correlational and not backed by controlled ablations.\n3. The paper does not report the actual compute cost (runtime, GPU memory usage, or GPU hours) required to run the benchmark, which makes reproducibility and practical adoption harder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)