HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

## Abstract

Abstract The aspiration for artifical general intelligence, fueled by the rapid progress of multimodal models, demands human-comparable performance across diverse environments.
We propose HumanPCR , an evaluation suite for probing MLLMs’ capacity about human-related visual contexts across three hierarchical levels: P erception, C omprehension, and R easoning (denoted by Human-P, Human-C, and Human-R, respectively).
Human-P and Human-C feature over 6,000 human-verified multiple-choice questions, assessing massive tasks of 9 dimensions, including but not limited to essential skills frequently overlooked by existing benchmarks.
Human-R offers a challenging manually curated video reasoning test that requires integrating multiple visual evidences, proactively extracting context beyond question cues, and applying human-like expertise. Each question includes human-annotated Chain-of-Thought (CoT) rationales with key visual evidence to support further research.
Extensive evaluations on over 30 state-of-the-art models exhibit significant challenges in human-centric visual understanding, particularly in tasks involving detailed space perception, temporal understanding, and mind modeling.
Moreover, analysis of Human-R reveals the struggle of models in extracting essential proactive visual evidence from diverse human scenes and their faulty reliance on query-guided retrieval.
Even with advanced techniques like scaling visual contexts and test-time thinking yield only limited benefits. We hope HumanPCR and our findings will advance the development, evaluation, and human-centric application of multimodal models.

## 1 Introduction

The rapid advancement of Multimodal Large Language Models (MLLMs) has shown remarkable potential in understanding diverse contexts llava_ov; llavavideo; gemini; gpt-4o; qwen2_5; qwenvl2. This progress fuels the aspiration toward artificial general intelligence and raises the expectation for MLLMs to operate effectively in the wild like humans ceval; mmlu; mmmu. A key prerequisite lies in the ability to understand humans in diverse, complex, and dynamic contexts, as human behavior inherently reflects the complexities of the world as well as real-world interaction
ego4d; egoexo4d; jrdb; social_jrdb; activitynet. In this work, we take this perspective and systematically investigate how well MLLMs understand humans across varied scenes, focusing on the critical aspects of perception, comprehension, and reasoning in human-centric visual understanding.

Human-centric visual understanding humanbench; unihcp remains a fundamental challenge in artificial intelligence, specially as recent benchmarks motionbench; face_human; zhou2024humanvbench; egoschema; actionatlas; feng2023posegpt; yollava tend to focus on specific tasks and scenarios (as in examples of Figure LABEL:fig:overview, top right). While these specialized datasets advance our knowledge in subdomains like ego-centric action or face understanding, they are inherently limited in scope and fail to capture the broad complexity of human activities. Meanwhile, general MLLM benchmarks mmmu; videomme; mmbench; mmtbench; zhou2024mlvu cover a wider range of topics but often lack the depth needed for fine-grained human-centric evaluation, overlooking intricate tasks such as gaze or body orientation estimation gaze360; posescript. With the rapid advancement of reasoning-specialized models GoogleDeepMind2025Gemini25; OpenAI2025o3o4mini; deepseek-r1, recent benchmarks mmmu; mmvu; mmrv; videommmu; m3cot; videomme; lu2023mathvista have sought to increase evaluation complexity to better test models’ ability to tackle real-world challenges, but they still fall short in visual-centric complexity. As shown in Figure LABEL:fig:overview (the lower left bar chart) and Figure [1](https://arxiv.org/html/2508.13692v1#S1.F1), current benchmarks lvbench; morse500; videoholmes often require less visual evidence and rarely assess models’ ability to proactively seek and reason with implicit visual cues beyond what is stated in the question. In contrast, the dense and dynamic human activities found in real-world settings—which are rich sources of intelligence worldqa; egonormia; mmtom—remain largely overlooked, as existing benchmarks seldom challenge models’ reasoning in these truly complex, human-centered scenarios mmvu; videommmu; qi2025vcr.

To address the aforementioned gaps, this work contributes an evaluation suite, HumanPCR, for benchmarking human-centric visual understanding. HumanPCR is structured into a hierarchical taxonomy of three levels: Human-P for perception, Human-C for comprehension, and Human-R for reasoning, as detailed in the upper left of Figure LABEL:fig:overview.
For Human-P and Human-C, a large-scale multiple-choice question-answering (QA) dataset is curated, featuring over 6,000 human-verified image- and video-based QA pairs, which assesses perception and comprehension capabilities along 9 dimensions, encompassing a total of 34 distinct tasks. Human-R presents a challenging, manually curated, open-ended video reasoning dataset. Sourced from 11 diverse human-related domains, Human-R requires models to solve complex problems by integrating multiple visual evidences, proactively extracting context beyond explicit cues in the questions, and applying human-like domain expertise.
To support further research, each question in Human-R is augmented with human-annotated Chain-of-Thought (CoT) cot rationales covering all key visual evidence.

We comprehensively evaluate a large suite of open-source and proprietary models on HumanPCR. Our findings reveal significant challenges for existing models in human-centric visual understanding and capability limitations in tasks involving detailed spatial perception thinking_in_space, temporal understanding videomme; zhou2024mlvu, and mind modeling ego4d; egonormia; mmtom.
Furthermore, analysis of Human-R highlights models’ struggles in reasoning on multiple visual evidence from diverse human scenes and their faulty reliance on query-guided cues longvideobench; video-rag; longvu. Consequently, merely scaling visual contexts gemini; gpt-4o; qwen2_5; adaretake; longvu or increasing test-time thinking OpenAI2025o3o4mini; internvl3; self-refine yields only limited benefits. Missing proactive evidence and recognition errors account for a substantial proportion of wrong cases in Human-R.
These results validate the necessity of our benchmark and imply critical shortcomings for MLLMs in solving real-world problems. We hope this benchmark will help to adapt MLLMs to diverse downstream applications and advance the development of more general and capable MLLMs.

Figure: Figure 1: Illustration of Human-R and Video-MME examples. The distribution of proactive and referred visual evidence is highlighted, while the score of recent video context extraction methods based on (1) Clip Score, (2) Object Detection Score, (3) MLLM Single-Frame Score, and (4) MLLM Attention Score, are also visualized. (Upper) The referred and sparse evidence required enables shortcuts without comprehensive video reasoning, so we exclude annotations solvable by single evidence or those not requiring reasoning. (Lower) on Human-R, high scores on referred/irrelevant frames, ignoring proactive evidence validate the shortcut, highlighting the necessity of multiple and proactive evidence for video reasoning evaluation.
Refer to caption: x1.png

## 2 Related Work

Multimodal Large Language Models (MLLMs). MLLMs have evolved from Large Language Models (LLMs) to process diverse modalities, including image sequences llava; llava_ov, video internvl; llava_ov; videollava, and audio borsos2023audiolm.
Recent models like Qwen2.5-VL qwen2_5 and InternVL3 internvl3 offers dynamic resolution processing and support extremely long contexts. For long video understanding, especially within limited context windows, context extraction strategiesliu2025ola; yang2024pvc such as frame sampling aks; huang2025frag; tstar and token pruning adaretake; zhang2025flexselect; longvu are employed to efficiently extract relevant content video-rag; yang2024pvc. Furthermore, the advent of models with powerful reasoning capabilities deepseek-r1; o1; GoogleDeepMind2025Gemini25; OpenAI2025o3o4mini has spurred pioneering efforts video-r1; llava-o1; videorft; li2025videochat to enhance visual understanding by leveraging and strengthening the model’s reasoning processes.

Evaluation on Human-Centric Visual Understanding.
Human-centric visual understanding has long stood as a cornerstone application in artificial intelligence humanbench; unihcp; activitynet. Riding on the recent advances of MLLMs, a growing body of work egonormia; egoschema; zhou2024humanvbench; li2024herm has begun to probe their competence in decoding human motion motionbench, recognizing faces face_human; mcllava; yollava, and interpreting domain-specific activities cui2023probio; actionatlas; plizzari2025omnia. Yet, across studies, a recurring conclusion remains: current MLLMs still exhibit pronounced limitations in human-centric visual understanding.
Another line of work has focused on applying MLLMs to 3D human modeling tasks feng2023posegpt; lin2025chathuman; li2025unipose, such as motion generation jiang2023motiongpt; chen2024motionllm and motion tracking hong2024egolm.
Given the prevalence of human activities in real-world scenarios, many comprehensive benchmarks mmbench; videomme; li2023seed also incorporate human-centric tasks, including celebrity identification and action recognition.
However, these benchmarks often lack structured taxonomies and unified evaluation protocols, which makes it difficult to derive systematic insights into human-centric visual understanding.

Multimodal Reasoning Benchmarks.
With the growing capabilities of MLLMs, plenty of benchmarks have involved increasingly complex tasks to assess their reasoning abilities m3cot; mmlu; thinking_in_space; mmcode; lu2023mathvista.
Early studies assessed reasoning capabilities mmmu using standardized examinations spanning multiple disciplines, and were subsequently extended to tasks involving natural images m3cot and multi-image inputs mmiu.
For video-based reasoning, recent progress chandrasegaran2024hourvideo; thinking_in_space; cao2025videosimple; videommlu; morse500; videoholmes; mmrv has been made. Comprehensive benchmarks longvideobench such as Video-MME videomme and MMBench-Video fang2024mmbenchvideo include predefined reasoning tasks, while MultiHop-EgoQA mh-grounded targets reasoning over object co-occurrence.
MMWorld he2024mmworld and MMVU mmvu introduce multi-disciplinary video reasoning questions, while VideoMMMU videommmu focuses on knowledge reasoning from subject explanation videos. More recent works, such as VideoEspresso han2024videoespresso and VCR-Bench qi2025vcr, construct video QA with reasoning processes by leveraging automatic annotations and repurposing existing QA pairs.
In contrast, our proposed Human-R distinguishes itself by providing human-annotated QA pairs along with human-annotated CoT rationales with key visual evidence, emphasizing multi-evidence integration and the challenges of information extraction under diverse human-centric contexts.

## 3 The HumanPCR Benchmark

### 3.1 Overview of HumanPCR

We introduce HumanPCR benchmark to evaluate how MLLMs understand humans in real-world scenarios. HumanPCR is structured within a three-level hierarchical taxonomy with holistic dimensions, which results in over 6,000 multiple-choice questions across 34 tasks and 442 open-ended questions includeing human-annotated CoT rationales.
More key statistics of HumanPCR are summarized in Table [1](https://arxiv.org/html/2508.13692v1#S3.T1). Definitions for
each task, detailed implementation, and metadata of annotations are provided in Appendix [References](https://arxiv.org/html/2508.13692v1#bib),  [References](https://arxiv.org/html/2508.13692v1#bib) and  [References](https://arxiv.org/html/2508.13692v1#bib).

Figure: Figure 2: Benchmark Construction. A hierarchical task taxonomy and a diverse human-centric vision-language corpus are first built on both an extensive survey of existing literature and the requirements observed in real-world scenarios. Next, task-specific strategies including query-template and LLM repurposing, are employed to suit each subtask, and finally, all annotations are sent to a rigorous, multi-stage quality control pipeline.
Refer to caption: x2.png

The taxonomy in HumanPCR are briefly introduced as follows:

- •
Level 1: Perception evaluates the recognition of specific elements. It covers five dimensions and encompasses 17 tasks: (1) Spatiality: perceiving existence of people, objects, and their spatial relations; (2) Posture: recognizing physical state and orientation of body parts, hands, and gaze; (3) Appearance: identifying human appearances, including inherent attributes and attirement; (4) Contact: recognizing detailed interaction regions between people and objects, or themselves; (5) Identity: recognizing a person’s identity.
- •
Level 2: Comprehension assesses the comprehension of basic visual concepts integrating commonsense or domain-specific cues. It covers four dimensions and encompasses 17 tasks: (1) Behavior: understanding human actions and bodily movements, such as gestures and emotions; (2) Procedure: thoroughly understanding long-term activities, including underlying intentions and dependence among action sequences; (3) Relation: analyzing relations, roles and differences among individuals; (4) Scene:
interpreting group dynamics or human activities within broader contexts.
- •
Level 3: Reasoning examines whether models can integrate continuous, tightly coupled human dynamics within complex scenes for reasoning. We contend that the evaluation should satisfy three criteria:
(1) Visual Complexity: questions should require suffient visual evidence(^1^11In this work, visual evidence is defined as a proposition in visual information containing single or multiple instances—such as a specific attribute of an instance or a group (e.g., orientation or action)—or as a visual relationship among multiple instances or group activities.), and exclude redundant visual content, going beyond simple concept retrieval;
(2) Reasoning Necessity and Diversity: questions should engage diverse reasoning chains rather than be limited to a few reasoning patterns;
(3) Proactivity: questions should demand proactive extraction of visual evidence over the continuous contexts(^2^22This refers to visual information that is not, or only partially, cued by the question, termed ”proactive visual evidence”, in contrast to ”referred visual evidence” which is explicitly indicated by the question.), rather than relying solely on evidence explicitly specified in the question.
Based on these criteria, we conduct the reasoning-level evaluation, requiring integrating multiple pieces of visual evidence from the video context and encouraging questions that can only be solved through proactive evidence extraction from the video, rather than simply disclosing all necessary information in the question. As illustrated in Figure [1](https://arxiv.org/html/2508.13692v1#S1.F1), this design systematically and reliably tests models’ abilities for faithful video understanding and complex multimodal reasoning, while tests relying on isolated evidence or fully specified questions often allow shortcut solutions based on single-frame matching or object priors, bypassing the need for genuine video understanding and sufficient reasoning.

### 3.2 Construction of HumanPCR

As illustrated in Figure [2](https://arxiv.org/html/2508.13692v1#S3.F2), the construction of HumanPCR follows a detailed three-stage pipeline involving task-driven data collection (Sec. [3.2.1](https://arxiv.org/html/2508.13692v1#S3.SS2.SSS1)), level-adaptive question-answer annotation (Sec. [3.2.2](https://arxiv.org/html/2508.13692v1#S3.SS2.SSS2)), and a rigorous multi-stage quality control process (Sec. [3.2.3](https://arxiv.org/html/2508.13692v1#S3.SS2.SSS3)). Details of the source datasets are provided in Appendix [References](https://arxiv.org/html/2508.13692v1#bib), and the annotation and review protocols are described in Appendix [References](https://arxiv.org/html/2508.13692v1#bib).

#### 3.2.1 Task Definition and Data Collection Protocol

We adopt a task-driven data collection approach, sourcing from diverse curated datasets and internet videos, as illustrated in the top panel of Figure [2](https://arxiv.org/html/2508.13692v1#S3.F2). For Human-P and Human-C, task design and data selection are guided by a systematic survey. Existing human-centric data sources spanning multiple domains and varying scene complexities are reorganized by aligning similar tasks and extracting distinctive aspects as independent tasks. This enables us to curate a diverse set of tasks, each supported by rich and varied datasets.
For Human-R, videos are sourced from academic datasets and YouTube videos. The YouTube subset is pre-filtered via domain-relevant tags and then manually reviewed before annotation to guarantee both content richness and safety.

#### 3.2.2 QA Annotation Protocol

Human-P and Human-C.
Benefiting from the data collection, we efficiently scale up QA pairs without compromising data quality or diversity by leveraging annotations from existing datasets.
Task-specific templates and LLM-guided generation are jointly used to create questions and options based on dataset annotations, as illustrated in the lower-left panel of Figure [2](https://arxiv.org/html/2508.13692v1#S3.F2).
Moreover, to address underexplored tasks in existing datasets, we manually generate additional QA pairs and complementary annotations by involving domain-specific expert annotators.

Human-R.
Experts from each source domain are recruited to annotate questions that encompass five distinct types of reasoning: Causal Reasoning, Prediction, Counter-Factual Reasoning, Assessment, and Planning, under detailed guidelines.
They are asked to create the questions with direct answers and explicitly outline the reasoning steps involved.
Annotators are also permitted to flexibly select relevant video segments for each question, thereby enhancing contextual diversity and ensuring that the challenges remain grounded in realistic video contexts.
They can also receive online feedback from the review stage and are given one opportunity to revise their submissions.
If the revised submission still fails to meet the requirements, the annotation will be discarded.

#### 3.2.3 Quality Control and Verification

For Human-P and Human-C, QA pairs are first filtered by LLMs to eliminate those solvable without visual input, followed by human verification conducted by trained annotators.
Each annotation is carefully reviewed for linguistic quality, answer accuracy, distractor plausibility, and, most importantly, its reliance on visual context.
This pipeline yields over 6,000 high-quality multiple-choice questions.
For Human-R, reviewers begin by filling out a detailed checklist that probes each annotation’s objectivity, factual accuracy, non-redundancy, and complexity; they then assign a quantitative score and deliver targeted feedback to the annotator.
Meta-reviewers further assess complexity, ensuring that (1) every question requires integrating multiple visual evidence; (2) the integration pattern cannot be fully determined from the question alone; and (3) each question relies on at least one essential proactive visual evidence.
The interaction flow among the annotator, reviewer, and meta-reviewer is illustrated in the lower-right panel of Figure [2](https://arxiv.org/html/2508.13692v1#S3.F2).
These criteria guarantee Human-R ’s necessary reasoning challenge and visual-centric novelty.
Common cases of valid but insufficiently complex questions are illustrated in Figure [1](https://arxiv.org/html/2508.13692v1#S1.F1) upper. This rigorous process yields an acceptance rate below one-fifth, resulting in 442 high-quality questions.

Figure: Table 1: Key statistics of HumanPCR.
Refer to caption: x3.png

### 3.3 Comparison with Existing Benchmarks

HumanPCR comprises over 6,000 multiple-choice and 442 open-ended questions across hierarchical levels, combining image and video modalities, as presented in Figure LABEL:fig:overview and [3](https://arxiv.org/html/2508.13692v1#S3.F3). Notably, Figure [4](https://arxiv.org/html/2508.13692v1#S3.F4) shows that Human-R consistently demands a high level of visual evidence, regardless of video length. This highlights our rigorous quality control procedures and the benchmark’s specialization in visual-centric reasoning, which requires the integration of multiple visual evidence as well as proactive inference.

In comparison to existing benchmarks, HumanPCR fills critical gaps across two domains:
(1) Human-centric understanding benchmarks: Recent works zhou2024humanvbench, e.g., MotionBench motionbench, ActionAtlas actionatlas, and Face-Human-Bench face_human, focus on specialized scopes and levles. HumanPCR provides a holistic framework covering a wide range of tasks across diverse dimensions, distinguished by its comprehensive evaluation and multi-faceted task design.
(2) Video reasoning benchmarks: Existing benchmarks fall short of Human-R’s focus on integrating multiple visual evidence, delving into intricate context, and performing diverse reasoning chains simultaneously. General video benchmarks videomme; zhou2024mlvu; longvideobench predefine task types with insufficient diversity in visual evidence and reasoning patterns, while automatically-curated works qi2025vcr; mh-grounded; han2024videoespresso often fall short in providing high-quality questions and reasoning processes that are characteristic of human-annotated data.
Recent benchmarks videoholmes; morse500; liu2025videoreasonbench that emphasize expert knowledge or multi-step reasoning often operate at a much lower level of visual-centric complexity than Human-R, as our manual analysis in Figure LABEL:fig:overview shows: they trigger fewer instances of visual evidence and demand far less proactive evidence seeking. To accommodate complex tasks, these benchmarks rely on video sources with limited scene diversity and real-world complexity—such as short, domain-specific demonstrations cao2025videosimple; mmvu, scripted or program-generated videos worldqa; videoholmes; morse500, or text-heavy presentations videommlu; videommmu. This results in reduced visual evidence diversity and minimal need for evidence integration. In contrast, Human-R is built on diverse, human-centric videos, where dense real-world activities and rich implicit rationales naturally support challenging multi-evidence reasoning, without sacrificing authenticity or content variety.

Table: Table 2: Results of Proprietary Models and Open-Source Models across three levels of HumanPCR. Macro-average accuracy of tasks within each level and dimension is reported. Full leaderboard is in the Appendix [References](https://arxiv.org/html/2508.13692v1#bib).

## 4 Experiments

### 4.1 Evaluation Settings

Models. On HumanPCR, we benchmark a diverse set of state-of-the-art MLLMs that support video and image inputs.
Specifically, we evaluate nine proprietary models, among which are Gemini-2.5-FlashGoogleDeepMind2025Gemini25 and o4-miniOpenAI2025o3o4mini.
We also evaluate 30 representative open-source MLLMs, notably including the Qwen-VL seriesqwen2_5; qwenvl2 and the InternVL seriesinternvl; internvl3.
On Human-R, we further include recent methods that advance video understanding, particularly in context extraction and enhanced reasoningadaretake; video-r1. Moreover, proprietary models specifically designed for reasoning under strict rate limits are also included, such as o3OpenAI2025o3o4mini and Gemini-2.5-ProGoogleDeepMind2025Gemini25.

Prompt strategies and frame sampling.
Unless otherwise specified, we employ Direct Answer prompts to evaluate models on multi-choice questions and Chain-of-Thought (CoT) prompts for open-ended questions.
For video-based questions,
we uniformly sample 32 frames for multi-choice questions, while for open-ended questions, we maximize frame utilization by inputting the largest number of frames allowed by each model’s context window and our computational resources.
Detailed configurations of the evaluated MLLMs are shown in Appendix [References](https://arxiv.org/html/2508.13692v1#bib).

Metrics. For multi-choice questions, accuracy is measured by matching the response to the correct option. We report the average accuracy for each task, along with the macro-average accuracy of tasks for each dimension or level. Following prior benchmarksfang2024mmbenchvideo; mmvu; cao2025videosimple, we use a quick proprietary model, o3-miniOpenAI2025o3mini, as the adjudicator for open-ended questions.
We evaluate the correctness of responses by comparing the extracted final answers with ground truth.
See Appendix [References](https://arxiv.org/html/2508.13692v1#bib) for more evaluation details.

### 4.2 Main Results

Table [2](https://arxiv.org/html/2508.13692v1#S3.T2) presents the main evaluation results of three levels on HumanPCR. Our principal findings are summarized as follows.

Current MLLMs still exhibit notable limitations in human-centric visual understanding.
As shown in Table [2](https://arxiv.org/html/2508.13692v1#S3.T2), tasks in HumanPCR remain largely unsolved by current models. Most models perform below 60% accuracy on Human-P and Human-C. Leading models like o4-mini and InternVL3-78B, with overall accuracies of 64.56% and 63.66% respectively, still show specific weaknesses, such as Posture for InternVL3-78B and Contact for o4-mini. On Human-R, o4-mini achieves the highest accuracy of 58.60%, with most other models below 40%. These results highlight a substantial challenge in real-world human-centric visual understanding and emphasize the contribution of HumanPCR.

Open-source models rival in perception and comprehension, lag in reasoning.
On Human-P and Human-C, open-source models match proprietary ones.
Notably, InternVL3-78B surpasses the top proprietary model, o4-mini.
However, they underperform in reasoning—most remain below 30% accuracy on Human-R, whereas all proprietary models exceed this value.
Models like o4-mini and Gemini-2.5-Flash illustrate the advantages of proprietary designs for reasoning.
While open-source models match proprietary ones at perception and comprehension, they struggle on reasoning tasks involving complex visual evidence.

Understanding human-centric scenes reflects general capabilities.
Considering the results across all dimensions, persistent weaknesses appear on Posture, Contact, Behavior, Procedure, and reasoning—all with accuracies around or below 50%.
An initial observation is that these results point to deficits in fine-grained perception and spatiality, particularly for recognizing body parts and contact regions under occlusion, consistent with prior findings.
Poor results on Behavior and Procedure further expose limitations in temporal understanding and mind modeling, both critical for interpreting human activities and long-term events.
Overall, the challenges in HumanPCR reveal not only human-centric gaps but also general shortcomings of current MLLMs.

Figure: (a) Results on Human-P and Human-C
Refer to caption: x5.png

Figure: (a)
Refer to caption: x7.png

Figure: (a) Overall Acc.
Refer to caption: Figures/Analysis/reasoning_frame_visual_element/Acc_accuracy.png

### 4.3 Further Analysis

To gain a deeper understanding of the capabilities and limitations of frontier models, we conduct a comprehensive analysis focusing on the key factors that influence performance on HumanPCR.

Impact of model size.
In Table [2](https://arxiv.org/html/2508.13692v1#S3.T2), scaling model size consistently improves accuracy across various model sizes in MLLMs series.
As shown in Figure [5](https://arxiv.org/html/2508.13692v1#S4.F5), performance of InternVL2.5 and InternVL3 suggests that complex reasoning tasks are more sensitive to model size scaling, which brings steady improvements, while performance on Human-P and Human-C plateaus beyond approximately 38B parameters.
We hypothesize that complex tasks more effectively evaluate the fundamental capabilities of the models, such as world knowledge and logical inference.

Mixed CoT effect regarding models and tasks.
As shown in Figure [6(a)](https://arxiv.org/html/2508.13692v1#S4.F6.sf1), the effectiveness of CoT prompting exhibits divergent patterns across models and evaluation dimensions. Specifically, CoT consistently enhances the performance of proprietary models, such as GPT-4o and Gemini-2.0-Flash, whereas it often degrades the performance of open-source models.
Additionally, the benefits of CoT are dimension-dependent, with improvements more concentrated on complex tasks that demand detailed perception or the integration of multiple visual evidence.
Figure [6(b)](https://arxiv.org/html/2508.13692v1#S4.F6.sf2) reveals significant disparities in task-specific accuracy and CoT improvements. This observation highlights the non-redundancy of our taxonomy and emphasizes the necessity of evaluating a diverse set of tasks beyond coarse-grained categories.

Figure: Figure 8: Comparisons of advanced video understanding methods on Human-R and Video-MMEvideomme, including RL-tuned Thinkingvideo-r1; videorft; li2025videochat model and context extraction strategies ranged among Frame Selectionaks; tstar; huang2025frag, Down Poolingyang2024pvc; liu2025ola, Memory&Retrievevideo-rag; internlm2.5ominilive; qian2025dispider, and Token Selectionadaretake; longvu; zhang2025flexselect.
Refer to caption: Figures/Analysis/ablation_compression/comparison_compression.png

**Table 3: Results of test-time scaling strategies on Human-R. Open-source and proprietary thinking models are both included. $M$ represents the magnitude of the strategies. Specifically, the BoN strategy uses $2^{M}$ candidates, while the Self-Refine strategy performs $M$ iterations.**
| Model | Method | Reward Model | Direct | M=0 (CoT) | M=1 | M=2 | M=3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Test-Time Compute: Prompt Procedure |  |  |  |  |  |  |  |
| Gemini-2.0-Flash | BoN | Gemini-2.0-Flash (Self) | 32.13 | 36.43 | 36.88 | 38.01 | 38.69 |
| BoN | Gemini-2.5-Flash | 37.55 | 37.10 | 41.86 |  |  |  |
| BoN | o4-mini | 42.53 | 40.05 | 40.05 |  |  |  |
| Self-Refine | - | 41.40 | 36.65 | 36.65 |  |  |  |
| GPT-4o | BoN | GPT-4o (Self) | 32.81 | 41.40 | 43.21 | 45.70 | 45.93 |
| BoN | Gemini-2.5-Flash | 44.57 | 45.02 | 46.38 |  |  |  |
| BoN | o4-mini | 44.11 | 46.38 | 45.02 |  |  |  |
| Self-Refine | - | 39.59 | 39.82 | 40.95 |  |  |  |
| InternVL3-78B | BoN | InternVL3-78B (Self) |  | 37.56 | 32.35 | 35.06 | 34.16 |
| BoN | Gemini-2.5-Flash |  | 36.65 | 40.95 | 38.91 |  |  |
| BoN | o4-mini | 35.52 | 38.68 | 41.40 | 42.30 |  |  |
| BoN | GPT-4o |  | 32.8 | 34.6 | 35.29 |  |  |
| Self-Refine | - |  | 33.71 | 35.07 | 34.62 |  |  |
| Proprietary Thinking Model |  |  |  |  |  |  |  |
| Gemini-2.5-Pro-previewGoogleDeepMind2025Gemini25 | - | - | - | 51.13 | - |  |  |
| Claude-3.7-sonnetAnthropic2025Claude37Sonnet | - | - | - | 40.50 | - |  |  |
| o3OpenAI2025o3o4mini | - | - | - | 59.28 | - |  |  |

### 4.4 Delving into Visual Reasoning on Multiple Evidence

Based on Human-R, we conduct a comprehensive analysis of leading open-source and proprietary MLLMs to gain in-depth insights into the characteristics and limitations of their reasoning capabilities.

The impact of frame scaling on multi-evidence reasoning.
In Figure [7](https://arxiv.org/html/2508.13692v1#S4.F7), we examine model performance with varying frame numbers and group results based on the number of visual evidence. Key observations are as follows:
(1) Frame number is not the bottleneck. Scaling the number of frames does not significantly improve reasoning accuracy, indicating that challenges in Human-R go beyond vanilla temporal retrieval of evidence.
(2) Increased visual evidence exacerbates challenges. Accuracy decreases as the need for visual evidence increases. For questions requiring 6+ evidence, performance saturates or even worsens with additional frames. We hypothesize that the requirement for multiple visual evidence renders additional context ineffective, while also raising complexity, hindering evidence extraction, and distracting models.

Does advanced video understanding configuration help?
We evaluate MLLMs under advanced video-understanding configurations, context extraction addressing visual redundancy longvu
, and test-time compute scaling cot; deepseek-r1. Key findings are as follows:
(1) Visual Context Extraction.
Figure [8](https://arxiv.org/html/2508.13692v1#S4.F8) compares context extraction methods with uniformly sampled baselines on Human-R and Video-MME.
Validated on Video-MME, most methods achieve only marginal or negative gains on Human-R. In particular, token selection and memory-based retrieval strategies show clear gaps in performance between the two datasets, while frame selection and pooling yield consistent improvements. We suggest that token and retrieval methods rely on text-guided token-level filtering, making them prone to missing proactive evidence beyond what the question refers to, unlike the more robust frame-level selection and pooling. This suggests that Human-R cannot be fully addressed by query-guided or heuristic methods, posing a new challenge for video reasoning.
(2) Test-Time Compute on Reasoning. Figure [8](https://arxiv.org/html/2508.13692v1#S4.F8) and Table [3](https://arxiv.org/html/2508.13692v1#S4.T3) show the effect of test-time scaling strategies beyond vanilla CoT. Best-of-N (BoN) delivers over 5% gains on all 3 models with incremental improvements from stronger reward models or more candidates. Self-Refine merely offers marginal gains and degrades with more iterations. Open-source thinking models Video-R1 improve the baseline by 3.07%, while proprietary model o3 achieves the best performance of 59.28%, leaving a large gap to all others. Overall, test-time scaling strategies are generally effective.

Figure: (a)
Refer to caption: x8.png

Error analysis.
As illustrated in Figure [9(a)](https://arxiv.org/html/2508.13692v1#S4.F9.sf1), we analyze 200 randomly sampled questions and classify the errors made by the top-performing models into five categories: Knowledge/Common-sense Error, Visual-evidence Extraction Error, Perception Error, Refusal/Incomplete Answer, and Question-understanding Error.
Moreover, Visual-evidence Extraction errors are further subdivided into three sub-types. Together with Perception errors, the four visual-related errors are shown in Figure [9(b)](https://arxiv.org/html/2508.13692v1#S4.F9.sf2).
First, Visual-evidence Extraction errors dominate, especially missing proactive evidence not mentioned in the question, which is difficult to resolve even by scaling context (e.g., o3 and Gemini-2.5-Pro with fewer frames outperform Gemini-2.0-Flash with 1200 frames). Second, models heavily rely on query-guided retrieval, as referred evidence in the question is much less likely to be missed, while proactive context understanding remains weak. Third, different models exhibit distinct extraction tendencies, e.g. Gemini-2.5-Pro reduces proactive evidence omissions but introduces more irrelevant evidence compared to o3.
These results suggest that the bottleneck of current MLLMs in video reasoning lies not in factual knowledge gaps but in the inability to holistically model and extract relevant information beyond question cues—a critical aspect that has been largely overlooked in previous evaluations.

## 5 Conclusion

We present HumanPCR, a comprehensive multi-level benchmark designed to rigorously evaluate the capacity of multimodal large language models (MLLMs) to understand humans across diverse real-world scenarios.
It features a systematic, fine-grained taxonomy from hybrid sources and a challenging video reasoning test centered on the integration of multiple visual evidence. HumanPCR reveals persistent challenges, especially in detailed perception, temporal understanding, and complex reasoning. MLLMs often fail to proactively extract key visual evidence in reasoning, relying on query-guided retrieval, with limited gains from scaling visual context or test-time configuration. HumanPCR thus lays a foundation for diagnosing gaps and advancing robust, human-level multimodal understanding. Current limitations of HumanPCR include the reliance on academic datasets for perception and comprehension level, and LLM-based metrics for reasoning. To address these, we plan to extend to more professional domains and develop objective evaluation protocols that retain reasoning difficulty in future work.

## References

- [1]
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al.
Llava-onevision: Easy visual task transfer.
arXiv preprint arXiv:2408.03326, 2024.
- [2]
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li.
Video instruction tuning with synthetic data.
arXiv preprint arXiv:2410.02713, 2024.
- [3]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.
- [4]
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
arXiv preprint arXiv:2410.21276, 2024.
- [5]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
arXiv preprint arXiv:2502.13923, 2025.
- [6]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
arXiv preprint arXiv:2409.12191, 2024.
- [7]
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
Advances in Neural Information Processing Systems, 36:62991–63010, 2023.
- [8]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
arXiv preprint arXiv:2009.03300, 2020.
- [9]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024.
- [10]
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.
Ego4d: Around the world in 3,000 hours of egocentric video.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022.
- [11]
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al.
Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024.
- [12]
Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese.
Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.
IEEE transactions on pattern analysis and machine intelligence, 45(6):6748–6765, 2021.
- [13]
Simindokht Jahangard, Zhixi Cai, Shiki Wen, and Hamid Rezatofighi.
Jrdb-social: A multifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22087–22097, 2024.
- [14]
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.
Activitynet: A large-scale video benchmark for human activity understanding.
In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
- [15]
Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, et al.
Humanbench: Towards general human-centric perception with projector assisted pretraining.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21970–21982, 2023.
- [16]
Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang.
Unihcp: A unified model for human-centric perceptions.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17840–17852, 2023.
- [17]
Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang.
Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models.
arXiv preprint arXiv:2501.02955, 2025.
- [18]
Lixiong Qin, Shilong Ou, Miaoxuan Zhang, Jiangning Wei, Yuhang Zhang, Xiaoshuai Song, Yuchen Liu, Mei Wang, and Weiran Xu.
Face-human-bench: A comprehensive benchmark of face and human understanding for multi-modal assistants.
arXiv preprint arXiv:2501.01243, 2025.
- [19]
Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, and Ying Shen.
Humanvbench: Exploring human-centric video understanding capabilities of mllms with synthetic benchmark data.
arXiv preprint arXiv:2412.17574, 2024.
- [20]
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.
Egoschema: A diagnostic benchmark for very long-form video language understanding.
Advances in Neural Information Processing Systems, 36:46212–46244, 2023.
- [21]
Mohammadreza Reza Salehi, Jae Sung Park, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hanna Hajishirzi, and Ali Farhadi.
Actionatlas: A videoqa benchmark for domain-specialized action recognition.
Advances in Neural Information Processing Systems, 37:137372–137402, 2024.
- [22]
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black.
Posegpt: Chatting about 3d human pose.
CoRR, 2023.
- [23]
Chau Pham, Hoang Phan, David Doermann, and Yunjie Tian.
Personalized large vision-language models.
arXiv preprint arXiv:2412.17610, 2024.
- [24]
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
arXiv preprint arXiv:2405.21075, 2024.
- [25]
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al.
Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-software implications.
In 2023 IEEE International Symposium on Workload Characterization (IISWC), pages 154–166. IEEE, 2023.
- [26]
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al.
Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi.
arXiv preprint arXiv:2404.16006, 2024.
- [27]
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu.
Mlvu: A comprehensive benchmark for multi-task long video understanding.
arXiv preprint arXiv:2406.04264, 2024.
- [28]
Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba.
Gaze360: Physically unconstrained gaze estimation in the wild.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 6912–6921, 2019.
- [29]
Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez.
Posescript: 3d human poses from natural language.
In European Conference on Computer Vision, pages 346–362. Springer, 2022.
- [30]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
arXiv preprint arXiv:2507.06261, 2025.
- [31]
OpenAI.
Introducing OpenAI o3 and o4-mini.
[https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/), 2025.
- [32]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948, 2025.
- [33]
Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al.
Mmvu: Measuring expert-level multi-discipline video understanding.
arXiv preprint arXiv:2501.12380, 2025.
- [34]
Kejian Zhu, Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao.
Mmr-v: What’s left unsaid? a benchmark for multimodal deep reasoning in videos.
arXiv preprint arXiv:2506.04141, 2025.
- [35]
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu.
Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.
arXiv preprint arXiv:2501.13826, 2025.
- [36]
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che.
M $3$ cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought.
arXiv preprint arXiv:2405.16473, 2024.
- [37]
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
arXiv preprint arXiv:2310.02255, 2023.
- [38]
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al.
Lvbench: An extreme long video understanding benchmark.
arXiv preprint arXiv:2406.08035, 2024.
- [39]
Zikui Cai, Andrew Wang, Anirudh Satheesh, Ankit Nakhawa, Hyunwoo Jae, Keenan Powell, Minghui Liu, Neel Jay, Sungbin Oh, Xiyao Wang, et al.
Morse-500: A programmatically controllable video benchmark to stress-test multimodal reasoning.
arXiv preprint arXiv:2506.05523, 2025.
- [40]
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan.
Video-holmes: Can mllm think like holmes for complex video reasoning?
arXiv preprint arXiv:2505.21374, 2025.
- [41]
Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu.
Worldqa: Multimodal world knowledge in videos through long-chain reasoning.
arXiv preprint arXiv:2405.03272, 2024.
- [42]
MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin, Caleb Ziems, Yanzhe Zhang, Hao Zhu, and Diyi Yang.
Egonormia: Benchmarking physical social norm understanding.
arXiv preprint arXiv:2502.20490, 2025.
- [43]
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu.
Mmtom-qa: Multimodal theory of mind question answering.
arXiv preprint arXiv:2401.08743, 2024.
- [44]
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao.
Vcr-bench: A comprehensive evaluation framework for video chain-of-thought reasoning.
arXiv preprint arXiv:2504.07956, 2025.
- [45]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
- [46]
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie.
Thinking in space: How multimodal large language models see, remember, and recall spaces.
arXiv preprint arXiv:2412.14171, 2024.
- [47]
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li.
Longvideobench: A benchmark for long-context interleaved video-language understanding.
Advances in Neural Information Processing Systems, 37:28828–28857, 2024.
- [48]
Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji.
Video-rag: Visually-aligned retrieval-augmented long video comprehension.
arXiv preprint arXiv:2411.13093, 2024.
- [49]
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al.
Longvu: Spatiotemporal adaptive compression for long video-language understanding.
arXiv preprint arXiv:2410.17434, 2024.
- [50]
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie.
Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding.
arXiv preprint arXiv:2503.12559, 2025.
- [51]
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
arXiv preprint arXiv:2504.10479, 2025.
- [52]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
Advances in Neural Information Processing Systems, 36:46534–46594, 2023.
- [53]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36:34892–34916, 2023.
- [54]
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024.
- [55]
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan.
Video-llava: Learning united visual representation by alignment before projection.
arXiv preprint arXiv:2311.10122, 2023.
- [56]
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al.
Audiolm: a language modeling approach to audio generation.
IEEE/ACM transactions on audio, speech, and language processing, 31:2523–2533, 2023.
- [57]
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao.
Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment.
arXiv preprint arXiv:2502.04328, 2025.
- [58]
Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai.
Pvc: Progressive visual token compression for unified image and video processing in large vision-language models.
arXiv preprint arXiv:2412.09613, 2024.
- [59]
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye.
Adaptive keyframe sampling for long video understanding.
arXiv preprint arXiv:2502.21271, 2025.
- [60]
De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, and Jan Kautz.
Frag: Frame selection augmented generation for long video and long document understanding.
arXiv preprint arXiv:2504.17447, 2025.
- [61]
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al.
Re-thinking temporal search for long-form video understanding.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8579–8591, 2025.
- [62]
Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang, and Linchao Zhu.
Flexselect: Flexible token selection for efficient long video understanding.
arXiv preprint arXiv:2506.00993, 2025.
- [63]
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
arXiv preprint arXiv:2412.16720, 2024.
- [64]
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
arXiv preprint arXiv:2503.21776, 2025.
- [65]
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan.
Llava-o1: Let vision language models reason step-by-step.
arXiv preprint arXiv:2411.10440, 2024.
- [66]
Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou.
Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning.
arXiv preprint arXiv:2505.12434, 2025.
- [67]
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang.
Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.
arXiv preprint arXiv:2504.06958, 2025.
- [68]
Keliang Li, Zaifei Yang, Jiahe Zhao, Hongze Shen, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen.
Herm: Benchmarking and enhancing multimodal llms for human-centric understanding.
arXiv preprint arXiv:2410.06777, 2024.
- [69]
Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al.
Mc-llava: Multi-concept personalized vision-language model.
arXiv preprint arXiv:2411.11706, 2024.
- [70]
Jieming Cui, Ziren Gong, Baoxiong Jia, Siyuan Huang, Zilong Zheng, Jianzhu Ma, and Yixin Zhu.
Probio: A protocol-guided multimodal dataset for molecular biology lab.
Advances in Neural Information Processing Systems, 36:41543–41571, 2023.
- [71]
Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari.
Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24129–24138, 2025.
- [72]
Jing Lin, Yao Feng, Weiyang Liu, and Michael J Black.
Chathuman: Chatting about 3d humans with tools.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8150–8161, 2025.
- [73]
Yiheng Li, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen.
Unipose: A unified multimodal framework for human pose comprehension, generation and editing.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27805–27815, 2025.
- [74]
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.
Motiongpt: Human motion as a foreign language.
Advances in Neural Information Processing Systems, 36:20067–20079, 2023.
- [75]
Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang.
Motionllm: Understanding human behaviors from human motions and videos.
arXiv preprint arXiv:2405.20340, 2024.
- [76]
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma.
Egolm: Multi-modal language model of egocentric motions.
arXiv preprint arXiv:2409.18127, 2024.
- [77]
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative comprehension.
arXiv preprint arXiv:2307.16125, 2023.
- [78]
Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma.
Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems.
arXiv preprint arXiv:2404.09486, 2024.
- [79]
Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al.
Mmiu: Multimodal multi-image understanding for evaluating large vision-language models.
arXiv preprint arXiv:2408.02718, 2024.
- [80]
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li.
Hourvideo: 1-hour video-language understanding.
Advances in Neural Information Processing Systems, 37:53168–53197, 2024.
- [81]
Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, et al.
Video simpleqa: Towards factuality evaluation in large video language models.
arXiv preprint arXiv:2503.18923, 2025.
- [82]
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang.
Video-mmlu: A massive multi-discipline lecture understanding benchmark.
arXiv preprint arXiv:2504.14693, 2025.
- [83]
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen.
Mmbench-video: A long-form multi-shot benchmark for holistic video understanding.
Advances in Neural Information Processing Systems, 37:89098–89124, 2024.
- [84]
Qirui Chen, Shangzhe Di, and Weidi Xie.
Grounded multi-hop videoqa in long-form egocentric videos.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2159–2167, 2025.
- [85]
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al.
Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos.
arXiv preprint arXiv:2406.08407, 2024.
- [86]
Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu.
Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection.
arXiv preprint arXiv:2411.14794, 2024.
- [87]
Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y Charles, Xinyu Zhou, and Xu Sun.
Videoreasonbench: Can mllms perform vision-centric complex video reasoning?
arXiv preprint arXiv:2505.23359, 2025.
- [88]
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al.
Aria: An open multimodal native mixture-of-experts model.
arXiv preprint arXiv:2410.05993, 2024.
- [89]
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al.
Longvila: Scaling long-context visual language models for long videos.
arXiv preprint arXiv:2408.10188, 2024.
- [90]
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al.
Nvila: Efficient frontier visual language models.
arXiv preprint arXiv:2412.04468, 2024.
- [91]
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al.
Minicpm-v: A gpt-4v level mllm on your phone.
arXiv preprint arXiv:2408.01800, 2024.
- [92]
OpenBMB.
Minicpm-o: An open-source tool for compressing llms with orthogonal projection.
[https://github.com/OpenBMB/MiniCPM-o](https://github.com/OpenBMB/MiniCPM-o), 2024.
Accessed: 2024-05-21.
- [93]
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao.
Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution.
arXiv preprint arXiv:2409.12961, 2024.
- [94]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
arXiv preprint arXiv:2412.05271, 2024.
- [95]
ByteDance.
Doubao.
[https://www.volcengine.com/product/doubao](https://www.volcengine.com/product/doubao), 2024.
- [96]
xAI.
Grok-2 beta release.
[https://x.ai/news/grok-2](https://x.ai/news/grok-2), May 2024.
- [97]
Anthropic.
Introducing Claude 3.5 Sonnet.
[https://www.anthropic.com/news/3-5-models-and-computer-use](https://www.anthropic.com/news/3-5-models-and-computer-use), June 2024.
- [98]
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530, 2024.
- [99]
Google DeepMind.
Introducing gemini 2.0: Our new ai model for the agentic era.
[https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/), 2024.
- [100]
QwenLM.
Qvq-max: Think with evidence.
[https://qwenlm.github.io/blog/qvq-max-preview/](https://qwenlm.github.io/blog/qvq-max-preview/), 2025.
- [101]
Anthropic.
Claude 3.7 sonnet and claude code.
[https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet), February 2025.
- [102]
OpenAI.
Introducing OpenAI o3-mini.
[https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/), 2025.
- [103]
Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al.
Internlm-xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions.
arXiv preprint arXiv:2412.09596, 2024.
- [104]
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang.
Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.
arXiv preprint arXiv:2501.03218, 2025.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Mentioned above
soundness: 3
strengths: 1. The paper proposes a fairly reasonable pipeline to curate complex VLM evaluation benchmarks across different difficulty levels (Figure 3).

2. The paper takes extra care to ensure that the quality and difficulty of the dataset is high. For instance, the questions solvable by blind LLM are filtered. In addition, the reasoning questions that one evidence, or lack of reasoning were not kept. 

3. The paper tests a wide range of models in the open and closed world across several model scales. The fine-grained analysis reveals that only reasoning model o4-mini improves consistently with more visual context while the other models fail to do so.
summary: The paper proposes a fairly reasonable pipeline to curate complex VLM evaluation benchmarks across different difficulty levels (Figure 3).
The paper takes extra care to ensure that the quality and difficulty of the dataset is high. For instance, the questions solvable by blind LLM are filtered. In addition, the reasoning questions that one evidence, or lack of reasoning were not kept.  The paper tests a wide range of models in the open and closed world across several model scales. The fine-grained analysis reveals that only reasoning model o4-mini improves consistently with more visual context while the other models fail to do so.
weaknesses: 1. The process of human participation is not very clear at different stages of the work. (a) who are domain experts and for what domains? (b) Where are these domain experts sourced from and how many of them were there? (c) Did the same folks also solve the task for human evaluation number in Table 3? (d) Is human eval taken from the same kind of domain experts that made the dataset or are they average human workers on some platform?

2. The writing-style and presentation could be improved. For instance, (a) it is pretty hard to read some figures like 7 and 8. (b) the spacing seems to be altered at many places – hardly any gap between paragraphs on page 3 and 8.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: Can the authors provide controlled variants of Human-R items to quantify the marginal difficulty induced by proactive evidence, for example by (i) turning proactive evidence into referred cues, and (ii) adding minimal hints? Reporting Δ accuracy across models would causally validate the role of proactivity and identify which architectures benefit most. To disentangle perception/extraction vs reasoning/knowledge, have the authors run oracle-evidence experiments that supply the required clip timestamps or bounding boxes? Such upper bounds would localize the dominant bottleneck and guide method development. For judge robustness, would the authors consider releasing a small human-adjudicated “gold set” (balanced across domains and reasoning types) with reference CoTs to calibrate future LLM judges and track metric stability over time? Also, did judge agreement vary systematically by reasoning type (e.g., planning vs causal) or domain (e.g., education vs sports)? Given that adding frames can hurt when evidence count is high, have the authors tried structured evidence-gathering policies—iterative timeline search, subgoal decomposition, or tool-augmented retrieval—to mitigate attention dilution? Even preliminary results would clarify whether the bottleneck is context management or search policy. Finally, since several tasks touch on emotions and social relations, have the authors conducted bias or robustness audits across demographics and cultural contexts? A brief analysis would strengthen ethical readiness for real-world, human-facing applications.
soundness: 3
strengths: The benchmark’s most distinctive contribution is Human-R’s explicit requirement for multi-evidence reasoning with at least one proactive, non-referred cue—an evaluation target largely missing from existing video QA benchmarks. This forces models beyond query-matching shortcuts and surfaces a realistic capability gap in long, complex, human-centric videos. The taxonomy across Human-P/C delivers both breadth and diagnostic depth; macro-averaging at task/dimension level makes failure modes actionable (notably posture, contact, procedure). The annotation and verification design is strong: expert-authored open-ended questions, evidence-grounded CoT rationales, blind filtering to remove textual shortcuts, and meta-review to ensure non-redundancy and proactive evidence. The empirical study is comprehensive: 30+ models, analyses of frame scaling, multiple context-extraction strategies, test-time scaling (Best-of-N vs self-refine), and a human-grounded error taxonomy highlighting proactive evidence omissions. LLM-judge reliability is quantified against 4k human ratings with high correlations and no detectable self-bias, supporting practical reproducibility. Overall, the work addresses an important and under-evaluated capability—human-centric multi-evidence reasoning—of clear significance to assistive systems, egocentric AI, and HRI.
summary: This paper presents HumanPCR, a hierarchical benchmark for human-centric visual understanding in MLLMs, consisting of Human-P/C (over 6k multiple-choice items across 34 tasks and 9 dimensions) and Human-R (442 open-ended video reasoning questions spanning 11 domains). A central design of Human-R is to require multi-evidence integration and at least one proactive (non-referred) visual evidence per question, with expert-annotated CoT rationales and rigorous filtering to avoid shortcutting via query cues. Evaluating 30+ SOTA models, the authors show that current MLLMs substantially underperform humans, with pronounced weaknesses in fine-grained spatial/temporal understanding and mind modeling, and that mere frame/context scaling yields diminishing or negative returns when evidence requirements grow. In contrast, reasoning-oriented models and test-time Best-of-N sampling provide consistent gains. The benchmark includes quality control, prompts, model configs, LLM-judge validation against human judgments, and ethical data terms.
weaknesses: While Human-P/C are valuable and thorough, their task space overlaps with prior perception/comprehension benchmarks; the strongest novelty lies in Human-R. The paper would benefit from ablations demonstrating added diagnostic value of Human-P/C design (beyond coverage), or clearer evidence that specific sub-tasks reduce confounds seen in earlier datasets. Human-R’s reliance on an LLM judge, albeit validated, still risks metric drift as judges evolve; a small, fully human-adjudicated gold subset would strengthen longitudinal stability and cross-paper comparability. The proactive-evidence constraint is compelling but mainly supported by qualitative/error analyses; controlled ablations—e.g., converting proactive evidence into referred cues, or masking question hints—could quantify its causal contribution to difficulty and discriminative power. Finally, given aggregation from public datasets and web videos, a more explicit analysis of potential pretraining overlap with popular MLLM corpora would help dispel contamination concerns and clarify generalization.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1) How do you define “proactive” vs. “referred” evidence? Is this consistently labeled across annotators?
2) Do models improve because they see more, or because they choose better?
soundness: 2
strengths: 1) A reasoning paradigm that enforces multi-evidence integration and proactive evidence seeking, with rigorous filtering to avoid shortcutting.
2) Most datasets just ask questions where the answer is directly visible in one frame or clip. This benchmark forces models to search multiple places in the video to figure things out—like humans would.
summary: The HumanPCR dataset was constructed by first defining a three-tiered human-centric skill framework (Perception, Comprehension, Reasoning), and then sourcing video and image data from both public datasets (e.g., Ego4D, ActivityNet, MOMA) and recent web videos (e.g., YouTube clips from the last two years) spanning 11 real-world domains. For Human-P and Human-C, the team created over 6,000 multiple-choice questions using a mix of template generation, LLM drafting (e.g., GPT-4), and human review. For Human-R, they developed 442 open-ended video reasoning questions that require multiple pieces of visual evidence—at least one of which must be proactively retrieved, not directly cued—each paired with expert-authored Chain-of-Thought rationales. Rigorous quality control was applied, including blind filtering of text-answerable items, expert meta-review.
weaknesses: 1) The paper says annotations will be CC BY 4.0, but the DUA also says “non-commercial academic only.” CC BY allows commercial use—this must be reconciled (e.g., CC BY-NC for annotations, or remove NC language from DUA).
2) All answers are scored by a single LLM, even if correlations with humans are decent. To be rigorous, they should show confidence intervals, try multiple judges, and share prompts/seeds for transparency.
3) Only ~10% of Human-R questions were answered by actual humans. To make a strong comparison, they should show how many annotators were used, what instructions they had, and how reliable those answers are.
4) Many tables just give single accuracy numbers. They should report error bars, task-level variances, and statistical tests to show if differences are meaningful.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: My questions have already been described in the Weaknesses section. I would like the authors to respond to these points in detail and clarify how they plan to address them.
soundness: 3
strengths: 1. The paper is well written and organized. 
2. The graph in the paper is clear and beautiful.
3. The core idea of this paper is interesting and special. It fills the gaps in prior benchmarks, which were either too narrow or too broad without focusing on human-centric understanding.
4. The dataset of this paper is carefully built with multi-stage human review and strict quality control. Each Human-R question must rely on multiple visual clues and include at least one piece of proactively gathered evidence, and only about 20% of questions are accepted.
5. Extensive experiments have been conducted to demonstrate the effectiveness. The paper evaluates more than 30 state-of-the-art models and shows that humans still outperform the best model by over 15%, while many open-source models score below 30% accuracy on the most challenging Human-R reasoning tasks.
summary: The paper introduces HumanPCR, a human-centric benchmark for evaluating multimodal large language models at three levels: perception, comprehension, and reasoning. It contains over 6,000 multiple-choice questions across 34 fine-grained human-related skills. These reasoning questions require the model to combine evidence from multiple moments in a video and notice important details that are not directly pointed out in the question. By testing 30+ state-of-the-art models, the paper shows that even the strongest systems still fall well short of human performance on fine-grained spatial understanding, temporal reasoning, and intent inference.
weaknesses: 1. Evaluation setup is not perfectly fair. The paper evaluates many models, but they are run with different inference settings, so the comparison is not strictly fair. For comprehension, models only get 32 sampled frames and must answer directly, while for reasoning some models can use the maximum number of frames they can handle and are given stronger prompts plus tricks like Best-of-N sampling. 
2. Causal claims about failure modes are a bit strong. The paper concludes that proactive evidence extraction is the key bottleneck in complex video reasoning, based on manual analysis of 200 Human-R questions from top models; however, this is still correlational and not backed by controlled ablations.
3. The paper does not report the actual compute cost (runtime, GPU memory usage, or GPU hours) required to run the benchmark, which makes reproducibility and practical adoption harder.


Decision: Accept (Poster)