EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
Authors: Baoqi Pei
Yifei Huang
Jilan Xu
Yuping He
Guo Chen
Fei Wu
Jiangmiao Pang
Yu Qiao
Venue: NeurIPS 2025 poster ()
Keywords: egocentric video
video reasoning
multimodal large language models
Primary area: applications

Abstract: Egocentric video reasoning centers on an unobservable agent behind the camera who dynamically shapes the environment, requiring inference of hidden intentions and recognition of fine-grained interactions. This core challenge limits current multimodal large language models (MLLMs), which excel at visible event reasoning but lack embodied, first-person understanding. To bridge this gap, we introduce EgoThinker, a novel framework that endows MLLMs with robust egocentric reasoning capabilities through spatio-temporal chain-of-thought supervision and a two-stage learning curriculum. First, we introduce EgoRe-5M, a large-scale egocentric QA dataset constructed from 13M diverse egocentric video clips. This dataset features multi-minute segments annotated with detailed CoT rationales and dense hand–object grounding. Second, we employ SFT on EgoRe-5M to instill reasoning skills, followed by reinforcement fine-tuning (RFT) to further enhance spatio-temporal localization. Experimental results show that EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained spatio-temporal localization tasks.

Ehonhin:e:: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
Baoqi Peil. Yifei Huang!.3; Jilan Xu Yuping Heb, Guo Chen', Fei Wu², Yu Qiao', Jiangmiao Pang'
"Shanghai Artificial Intelligence Laboratory, "Zhejiang University, 'The University of Tokyo, 'Fudan University. 'Nanjing University peibaoqi @gmai com; hyfßiis. u-tokyo.ac.
Abstract
Egocentric video reasoning centers unobservable agent behind the cam- era who dynamically shapes the environment, requiring inference hidden in- tentions and recognition of fine-grained interactions. This core challenge lim- its current multimodal large language models (MLLMs), which excel at visi- ble event reasoning but lack embodied, first-person understanding. To bridge this gap, we introduce EgoThhnner, a novel framework that endows MLLMs with robust egocentric reasoning capabilities through apatio-tempora chain-of- thought supervision and a two-stage learning curriculum. First, we introduce EgoRe-5M, large-scale egocentric QA dataset constructed from 13M diverse egocentric video clips. This dataset features multi-minute segments annotated with detailed CoT rationales and dense hand-object grounding. Second, we em- ploy SFT on EgoRe-SM to instill reasoning skills, followed by reinforcement fine-tuning (RFT) to further enhance spatio-temporal localization. Experimen- tal results show that EgoThinker outperforms existing methods across multi- ple egocentric benchmarks, while achieving substantial improvements in fine- grained spatio-temporal localization tasks. Full code and data are released at /tt.i//ooooott////////oo/oooooon http:p///tttuu.ccccmmmmmmmotttta
Introduction
Humans possess a remarkable ability to reason, plan, and execute complex, goal-oriented behaviors within dynamic real-world environments. Recent works in Multimodal Large Language Models (MLLMs) have advanced the field visual understanding (55) [I 77 54 12]. Techniques such as chain-of-thoughe prompting [75 [9] and reinforcement fine-tuning 39 27 further underscore the potential of MLLMs in high-level reasoning. However, existing approaches mainly address visual reasoning from an observer-centric, third hird-ppesoon viewpoint. This perspective fails capture the embodied cognitive processes central to human reasoning, which naturally occur from the egocentric perspective.
Egocentric reasoning differs fundamentally from conventional visual reasoning due to the presence of the observer as an active participant in the scene. Thus, models must infer not only visible events but also the internal cognitive states, intentions, and future behaviors of the individual behind the camera. This reasoning process poses several unique challenges: (1) Reasoning for complex tasks: Understanding the rationale behind an action and predicting what comes next demands explicit cause-effect chains rather than isolated event recognition. (2) Human-objes interaction recognition: Successful reasoning hinges on accurately localizing hands and manipulated objects.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Egocentric Video Reasoning
General Video Reasoning
MLLMs
Reasoning Steps:
Reasoning Steps:
poured glass.
1. Cyan cube is not present the initial state. It clearly visible the final state.
2. The camera bobbed up and down.
put empty plass down.
Tginhinker
Answer: I just drank water.
Answer: cube entered..
EgoRe-5M Dataset
Training Paradigm
Web data
Dota mining
Short-term
SFT
Ego relttedatta
EMORe-SM-FG
& Filtering
EgoRe-SM Genera data
Long-term
Ego Thinkee
Egocentric
CoTT
Egonkinker
Completions
GRPO
Specialized prompting
milili
FG grounding
Output
Applications of EgoThinker
Egocentric Reasoning Chat
Hand-object grounding
Temporal grounding
ssuld the rulexand bakery process?
Localica the
with scoop vegetables plate"
"spoon" Uorputyour thought process
</thnll> tags. Thes provide bounding hus following camserr [(s (1)) (x2y2]]
your thought process
<think tags Provtde
"stari,nnf within the canswers s/anowerss lags-
& The man the professtonal baker and am the assistant. This because the man as seen uxcnng the oven and the dos gi whii la wose uvnn cleaning the table and preparing the d dough for baling.
</answer> format.
: -thi the event eccurs after the person siirs the find in the pot and before the person pounthhe food into the plate </thinks. san.wer- (15.23, 29.66) */amnous>>
-think- The right hand is holdie yt spoon. which in the mıddle the picture e/think- <answer>> 20.38,099.00.00 (t.o:....55)/////ooeee....o
Figure 1: Overview of ThiThineer. Unlike general video reasoning, egocentric video reasoning poses unique challenges because it must infer an unobservable camera wearer's s interactions and intentions. Ego Thinker addresses this by curating EgoRe-5M, large-scale egocentric reasoning dataset, and applying two-stage supervised and reinforcement fine-tuning paradigm. This design empowers robust egocentric reasoning chat, hand-object grounding, and temporal grounding, making EgoThinker promising foundation for wearable assistants and embodied AI.
(3) Multi-horizon temporal integration: Egocentric video streams span minutes to hours, requiring models to track evolving context and retain ime-grained details across thousands of frames.
Existing egocentric datasets 997 102 are mainly derived from Ego4D (25 and EgoExo4D (26)
They provide extensive collections of egocentric videos but lack explicit reasoning chains, temporally spanned annotations, and detailed fine-grained grounding data. Consequently, existing MI.LMs [85]
101 3. although successful in general visual understanding tasks, often struggle when reasoning about complex tasks in long-term egocentric videos.
In this work, we propose Tgonhine novel framework designed to enable robust egocentric reasoning MLLMs. pipeline to extract egocentric videos from large-scale web data to capture diverse real-world scenarios. Leveraging this, we construct EgoRe-5M, a large-scale egocentric QA dataset, featuring diverse questions spanning from seconds to several minutes. In the dataset, we incorporate detailed Chain-of- Thought (CoT) annotations to explicitly model the causal relationships underlying complex human activities, enabling models to emulate human-like causal inference and planning. Furthermore, recognizing the critical role of hand-object interaction in egocentric reasoning, we additionally introduce specialized data focusing fine-grad fine-grained spatio-temporal hand-object interactions.
illustrated Figure overcome dataset limitations, we develop
Inspired by recent advances reinforcement fine-tuning (RFT) 39 27]. EgoThinker employs two-stage training strategy. Initially, the model undergoes supervised fine-tuning (SFT) 88 [01].
utilizing Eg to establish foundational understanding and reasoning. Subsequently, we employ RFT using spatio-temporal grounding data via GRPO [27] method. This paradigm significantly enhances the model's capability fine-grained localization, temporal reasoning, and causal inference. Experiments demonstrate that EgoThinker outperforms existing state-of-the-art methods across multiple egocentric benchmarks, showcasing strong performance gains egocentric QA
[42
long-term video reasoning [15 64), and spatio-temporal hand-object interaction localization
36

---

In summary, our contributions are: (1) EgoRe-5M, large-scale egocentric QA dataset with chain-of-thoughh and hand-object annotations curated from diverse video sources. (2) A two-stage training regime combining supervised fine-tuning and reinforcement fine-tuning via GRPO to ef- fectively couple high-level reasoning with low-level grounding. (3) EgoThinner, an MILLM setting new state-of ttate-of-the-art multiple egocentric video benchmarks, demonstrating coherent egocentric reasoning and precise spatial and temporal grounding.
Related work
Egocentric Video Understanding. Egocentric video understanding has recently garnered increas- ing research attention, with the introduction of large-scale egocentric datasets such as Ego4D. Previous works mainly focus on action understanding [72] 33 24 32, 34, 699 22 TO vision-language pre-
70 and hand-object interaction understanding 100 9 22 93 Recently, some methods [5] 97 022gg
training [56 35
108
30 37 have attempted to use MLLMs for egocentric video understanding, and
some targeting long-form form egocentric videos (94 877. Different from these works, EgoThinker the first model that enables reasoning and precise hand-object understanding in first-person videos.
Multimodal Large Language Models. Recent advancements in MLLMS 55
1
77
31
54 12
have shown robust comprehension and perception abilities. Video-LLaVA (55) and Videochat2 [51
enable MILMs with general video understanding and temporal localization capability. Recent works extend to diverse domains including long-form video understanding [T04
83 robot
learning [90] [45], and spatio-temporal perception [67 62). Chain-of-Thousee (CoT) prompting 89 has proven effective in eliciting multi-step reasoning LLMs. Recently, some works B
use CoT techniques to enhance MLLM's visual reasoning capabilities. In our work, we construct large-scale QA samples with causal CoT captions equip MILMs with egocentric reasoning ability.
Recently, OpenAl-ol 33 and DeepSeek-R RI [2 have shown that reward-driven fine-tuning substantially enhance LLM reasoning, For MLLMs, many 61 focused on leveraging reinforcement learning (RL) techniques with verifiable rewards to enhance visual reasoning capabilities with only small amount of data. Videochat-R (52] enhances MLLM's temporal perception ability via RFT for general video understanding. Building on these advances,
Reinforcement Learning for MLLMs.
works [10 59 D
20,
71
60
95
[03
construct fine grained apatio-temport grounding datasets and apply GRPO approach to endow MLLMs with precise hand-object localization and honizoorizon causal inference in egocentric videos.
EgoThinker: Enhancing MLLMs Egocentric Reasoning Ability
In this section, we introduce EgoThinker framework. designed to equip MLLMs with egocentric reasoning capabilities. We begin with the curation of EgoRe-5M, a large-scale egocentric instruction tuning dataset. We then describe how Eg fuels our two-stage learning curriculum: initial supervised fine-tuning to establish foundational reasoning skills, followed by a reinforcement fine- tuning paradigm sharpen hand-object grounding and temporal reasoning ability.
3.1 EgoRe-5M
have constructed egocentric QA datasets but lack data for long-term causal reasoning and fine- mme-grained spatio-temporal localization. This deficiency hinders the develop- ment of models capable of egocentric reasoning. To address this limitation, we propose Eg,Re-5M, large-scale egocentric QA dataset designed with rich causal reasoning and grounding data.
Recent works 97 102
3.1.1 Egocentric Video Collection
Accurate egocentric reasoning demands a vast and diverse collection of egocentric data. While existing datasets like Ego4D [25] and EgoExo4D [26] are valuable, their scale is notably smaller compared to web-sourced data. To overcome this scale bottleneck, we develop a multi multi-stage filtering pipeline to mine high-quality egocentric clips from web-sourced videos.
(1) Web-scale Mining. We choose the large instructional video dataset How 1000 (65 as
data source in which numerous instructional video are recorded by head-mounted or handheld
3

---

Data Filtering Pipeline
EgoRa-5M
Benz-dbjer
30M Web Video Clips
Thhnggh Ansossing
Responal
Egowts-ao Filtering
Groasding
Gra-Gra (POK]
12M Eige-Style Clips
Objerr Gran!
Oyhemi Filtering
3008
+1IX
Jasporal Graandiy
8.7M Ciiso With
Atrilatt
S28K
Q&A
QA
+
Interantion
Olyect
200K
Short-tor -oort-ttrrm [2.4M]
Long-term [2.4M]
418X
Attos
Duni
Altr Altrbaat
Russoniny
Actlon Descripoivs
290K
Action Sunmurry
Figure 2: Data Filtering Pipeline and EgoRe-5M Statistics. With our multi-stage filtering pipeline, construct EgoRe-5M, large-scale QA dataset to facilitate egocentrie reasoning in MLLMs.
camera featuring fine-grained hand-object interactions. Moreover, the task diversity and naturalistic narration within HowTo 100M make especially rich in egocentric perspectives. In particular, select HTM-AA 28 dataset containing a large number temporally aligned video-narration pairs and Powto--nterlinnk7M. [81), which additionally provides high-quality video caption annotations as our primary data sources, resulting in a total of 30M initial video clips. The duration of the video clips spans from a few seconds to several minutes.
(2) Ego-vs-Exo Filtering. distinguish true egocentric footage from exocentric content, we train a classification model on balanced sets of manually labeled ego- and exo exo-centric clips. The classification model utilizes Intern Video backbone (55 followed by a two layer MLP. The model achieves 92% accuracy and 89% AUC on held-out validation data. Applying this model reduces pool to 12M clips that exhibit clear first-person camera motion and range from seconds to minutes.
(3) Dynamic Interaction Filtering. Even after egocentric filtering, many clips remain static or depict group activities, offering limited value for reasoning about egocentric activities. Since most egocentric activities are centered on hand-object interactions, we run a pre-trained hand-object detector [74) to identify frames containing both visible hand and an active object. We design several rules filter the clips that exhibit hand-object interaction and dynamic changes, with each clip having minimum duration of seconds. This refinement yields 8. 8.7M bigh-quality egocentric clips, each containing rich, dynamic interactions suitable for downstream QA annotation.
As result, we combine our filtered 8.7M web data with existing egocentric datasets (Ego4D 25 EPIC-Kitchens [18. EgoExoLean (36] and EgoExo4D (26)) to form a collection of 13M egocentric video clips in total. For details regarding the pipeline, please refer the Supplementary A.
3.1.2 Egocentric Reasoning Data Construction
Prior egocentric QA datasets focus primarily short video clips and narrations, falling short on long-term causal chains and spatial-temporal grounding, which are key components of egocentric reasoning. To address these gaps, build EgoRe-5M, automatically generated QA corpus containing four complementary task dimensions: short-term perception, long-term causal reasoning, tfuin-of-thoughh rationales, and fine- ane-grained grounding. Figure shows an overview of the data source of our EgoRe-5M. Since some video clips have no corresponding text annotations, or the text annotations come from low-quality automatic speech recognition, we employ Videochat2-HHe [46). an efficient and robust video caption model, to annotate these videos with a sampling rate of fps. Questions are formulated by specialized LLMSs per split, detailed as follows.
To instill foundational egocentric perception skills, we generate lcrge-scale short-term QA split covering clips of 1-10 seconds. We design seven perceptual question types to capture immediate scene understanding: object existence, object attribute, object count, object interaction, action description, action reasoning and background attribute. For each clip, we combine the original text annotation with VideoChat2-HD captions and apply DeepSeek- V3 to instantiate and
Short-term Data.

---

answer these question templates. This process yields 2.4M QA pairs, ensuring broad coverage of objects, interactions, and immediate causal cues essential for downstream model pretraining.
Long-term Data. Human activities often progress through multiple steps, which makes long- form understanding essential for egocentric reasoning. To capture such extended causal chains, we aggregate consecutive clips into segments of 15-120 seconds and integrate their narrations into a single, coherent caption. We then design six question types assess temporal and causal understanding within each segment: action sequence, temporal grounding, object count, action prediction, action summary and action reasoning. Utilizing DeepSeek- V3 on these concatenated captions, we automatically generate 2.5 million QA pairs. This aims to enhance models' abilities to connect events over long durations and deduce causal relationships.
Chain-of- -f-thhughthhhttttt Data. Recent advances in chain-of-though (CoT) prompting 50 shown that explicitly conditioning L.LMs on intermediate reasoning steps can significantly boost performance on complex inference tasks such mathematics and coding. Visual CoT 775 extends this idea by supplying models with region-level hints to guide step-by-ster reasoning. Thanks to the success of the open-source DeepSeek R1 [27] model, it is possible to utilize LLMs to generate reasoning processes. Similar to long-term data, we select video clips with dense captions and concatenate them form captions. Each description is then fed to DeepSeek RI, producing a question and an accompanying step-by-step rationale. We design a prompt to allow the model to decide whether given segment warrants CoT question, ensuring that only clips amenable to multi-step inference are annotated. The resulting split comprises 50K each pairing complex egocentric scenario with an explicit chain-of-thought process.
107 have
CoT QA pairs,
Fine-Grained Grounding Data. Hand object interactions lie at the heart of egocentric reasoning. Existing specialized methods [100] perform well these tasks, but existing MLLMs exhibit poor performance. To remedy this, we construct QA data for two complementary grounding tasks: Hand-object Grounding and Temporal Grounding.
First, leveraging EK- EK-Visor's 's pixel-level masks for hands and active objects 19], asking about the spatial positions of hands/objects. The input can take the form of an image. The model must first articulate its intermediate reasoning and then output a normalized bounding box. This split trains models to map hand and object visual cues to precise coordinates.
we generate questions
For temporal grounding, using EgoExoLearn's 's fine-grained temporal annotations 36 pose questions that require selecting the exact time interval in clip that contains the evidence needed to answer. Models need to provide step-by-step reasoning and output start-end times seconds. This formulation emphasizes the ability to pinpoint moments of interest for downstream inference.
3.1.3 Data Statistics
As shown in Figure 2] our EgoRe-5M dataset comprises 5M question-answer annotations across four complementary splits. To verify the annotation quality, we randomly sample 500 QA pairs and check the correctness of answers and logical coherence of the intermediate rationales. Over 95% of reviewed samples meet our standards for accuracy and annotation quality. We provide qualitative examples in the supplementary demonstrate the dataset's quality and range in object categories, action types, causal chains, and precise grounding scenarios.
3.2 Training Thinhineer
We employ two-stage curriculum to imbue MLLMs with robust egocentric reasoning capabilities. First, we perform supervised fine-tuning on carefully balanced mixture of general visual, egocentric, and QA datasets, including EgoRe-5M's short, long, and CoT splits, to establish core capabilities object perception, causal inference, and multi-step planning (see details the supplementary). Second, we refine spatio-temporal grounding via Reinforcement Fine-Tuning (RFT) using GRPO approach on E5oRe-5M's 's fine-grained grounding data. Together, these stages transform an off-the- shelf MLLM into our Thinkin capable of egocentric reasoning.
5

---

Table I: Overview of fine-tuning datasets used during training.
Stage
Domain
Data Type
Amount
Training Dataset
Caption
100K
Llavor LlavorTOOS] Sharggstio 17],
Shorgprdrideo [3] VC2 [ebvid][3] Videochhiggr 63
General
VQA
70K
Nest-QA (T) TVQA (8 ClevrerQA (03) TGIF-QA 40 StorQ44
SFT
RellRed
390K
SSV2 (19). Ego-QA @ EgotimeQA 20
Eggo
M
860K
EgoRe-5M- SM-She,, Egolte-5M Soo EgoRe-5M-CoT
Ego
Grounding
EgoRe-SM-FC
3.2.1 Supervised Fine-tuning
Supervised fine-tuning establishes the foundational reasoning, perception, and language skills of EgoThinker. To also preserve general visual understanding and conversational fluency, we assemble a 1.5M sample fine-tuning corpus drawn from a diverse mix of datasets of 4 categories as Table
The details of the SFT training data are as follows: (1) High quality caption datasets. To maintain the general visual understanding and conversation ability, we select five high-quality datasets (Ilavar- gpt4-20k
[055] sharegpt4o [17], sharegpt4video [13]. webvid-2M
84 YouCook2 (YC2) [[I]] and
videochatgpt (63] ) containing image/video captions and conversational data. (2) VQA datasets: We
use five commonly used QA datasets (Next-QA 99], TVQA (48]. Clevrer-QA 98 TGIF-QA (0),
and STAR-QA (0)) to sharpen QA skills. (3) Ego-related datasets. To emphasize the egocentric perspective understanding, we include existing ego-related datasets. Sometthng-SSomethng V2 (SSV2) [109] is relevant to action recognition and contain numerous ego-style clips. EgoTimeeA [21) and Ego-QA [51] are constructed from egocentric datasets, covering action recognition, and long-term video comprehension tasks. (4) EgoRe-5M: We sample from our EgoRe-5M short-term, long-term, and COT splits to instill egocentric reasoning patterns. For details about SFT training, please refer to Supplementary B.
3.2.2 Reinforcement Fine-Tuning via GRPO
To enhance EgoThinker's spatio-temporal reasoning ability, we employ reinforcement fine-tuning (RFT) on the M-0R split utilizing Group Relative Policy Optimization (GRPO) approach. Unlike traditional reward-model approaches [66), we adopt the rule-based scoring functions outlined
in 47 39 76 to score the outputs of MLLMs directly. GRPO (27) then optimizes the policy by comparing groups of candidate responses without a separate critic network in.
Preliminary. Reinforcement Learning with Verifiable Reward (RLVR) replaces learned reward models with rule-based scorers that classify each output as correct or incorrect. DeepSeek R1-Zero's GRPO algorithm builds on RIVR by generating N distinct candidate responses = {o1..... ON } for an input question & through policy sampling. Each candidate evaluated by a reward function to produce corresponding scores {ri...., J. To compare answers fairly, GRPO computes normalized advantage for each response:
F3 mean((r;)))
A
std (r:))
where A; represents the relative quality of the i-th answer. The policy update maximizes expected advantage- wdvintage-weighted likelihood, with a Ki-divereence penalty to the reference model Trof
To(o:) To (o:) A:
DKI. (To (eto)) )
max E BowTTe (p) X9
where Ty is the policy model, Tret is the reference model before optimization and (3 is regularization coefficient control the KI-ddivergence. This objective encourages the model to allocate higher probability to top-ranked candidates while maintaining stability through KI. regularzattoo....
Reward function design. The design of the reward model is critical step. We design two complementary rule-based rewards tailored to assist MLLM in egocentric reasoning.
Formal reward. Following Deepseek-R1, we employ format reward function to enforce the model to output the prediction value and thinking process in the specified format. Specifically, we expect the model to output its thinking process within <think>. </th/ink>>>>>>> and the answer with

---

Table 2: Results egocentric video benchmarks. For EgoTaskQA, convert the dataset into multiple-choice question format following [122]
EgoTasQA
QAEgoAD
ERQA
EgoPlan-Val
EguSchema
EgoMCQ
VLN-QA
RES
Method
Acc.
Acc.
Acc.
Acc.
Acc.
Inter-Ace.
Intra-Ace.
Acc.
Acc.
LLaVA-Vidao-78 TBX
55.0
68.6
39.8
57.3
86.7
37.3
32.0
31.1
LLaVA-DneViioon-78
37.3
36.0
25.3
LaM3 2
56.6
62.4
36.4
61.1
77.3
29.8
34.5
21.3
VideoChan2-HD
45.5
35.7
87.1
36.4
43.0
24.5
InterVL2-88
61.0
664
34.2
64.2
79.0
31.0
46.0
30.0
Exo2Fgo-7H 78 102
42.7
61.3
88.4
41.2
44.5
Qwen2-VL-7B
57.9
60.3
37.0
38.3
63.3
86.4
34.1
42.0
26.3
EgoThinker
39.5
<answer>.... ..tsserr....aaaseeee... and design a format reward Rformaa- We utilize regular expression matching to determine whether the output of the MLLM matches our specified format. matches, assign Rtormat = 1; otherwise, assign Rformat O.
Grounding. The mloU (mean intersection over union) metric offers clear guidance for the grounding task, making it appropriate choice for the reward function. Therefore, using the fine- fine-grained annotations from EgoRe-5M-FG, we calculate the spatial/temporal IoU between the predicted boxes/intervals and the ground-truth boxes/intervals as reward function Ro /Ri Thus, for hand-object grounding task, we get Rog Riormat Rojou and temporal grounding task, we get Rig Rformat + Rejou'
loU Reward in
During RFT, use the combined rewards to strengthen EgoThinker's ability generate well- structured reasoning traces and accurate spatial and temporal groundings.
4
Experiments
The main experiments are all conducted based on Qwen2-VL VL-7B 82]. To train EgoThinker, we first perform SFT our curated datasets, then apply RFT via GRPO.
Benchmarks. To thoroughly assess performance, we organize our evaluation benchmarks as
follows: (1) Egocentric Benchmarks. We evaluate first-person reasoning capabilities on six es-
tablished datasets: Egotask [42]. QAEgo4D
(11, EgoPlan [15]. EgoSchema (64), EgoMCQ (56),
ERQA 77 and VIN-QA [5]. (2) Cross-View Skill Transfer (RES). To gauge the model's ability
generalize learned skills across perspectives, we introduce the Referenced Egocentric Skill (RES)
benchmark 4-way MCQs using paired clips from EgoExoLearn (36] and EgoExo4D 226 (3)
Fine treeggra Opatio-Tempont Grounding. We assess spatial and temporal grounding on two newly constructed specialized HOI benchmarks derived from Ri-Visor [99 and EgoExoLearn [36] (4)
General Video Understanding. Finally, to confirm that EgoThinker maintains broad applicability. report results on three general video benchmarks including MVBench [5]], Perception Test (2] and VideoMME 23 Please refer to the Supplementary for details about these benchmarks.
4.1 Quantitative Evaluation of Egothinker
Egocentric Benchmarks. Table shows comprehensive comparison of EgoThinke. against our baseline Qwen2-VL-7B and other leading MILMs on the egocentric benchmarks. EgoTThinker establishes new state-of-the-an performance across all tasks, achieving a 4.4% absolute gain on EgoPlan, 3.4% on EgoSchema, and 8.0% on VLN-QA. This clearly demonstrates TgoThinker's strong capacity long-horizon planning, semantic inference, and goal-oriented question answering in egocentric video. In contrast, baseline models exhibit inconsistent strengths: L.L leads on QAEgo4D while Intern VL2 excels on EgoSchema and EgotaskQA, revealing an apparent lack of generalization in existing MLLMs to egocentric perception and reasoning.
On the Referenced Egocentric Skill (RES) cross-view benchmark. where prior MILMs perform at near-random levels, EgoThinker outperforms the second best model by 8.4%, underscoring its unique ability to transfer learned skills between perspectives. These results confirm that our EgoThinker effectively unleashes the egocentric reasoning capabilities of MLLMs.
7

---

Table 3: EgoThinker results hand-object and temporal grounding tasks.
Table 4: EgoTThinker results general video benchmarks.
EK- Visor
EgoExoLearn
Method
MVBench
Perception
VideoMMEE
Method
mloU
Loc Acc.
mloU
R 6:0055
MiniCPM-V2.6
96
67.1
58.1
60.9
LLaVA- Video
106
46.7
67.7
InternV1.2
[6
66.4
60.1
54.0
Qwen2VL-7B
82
56.7
64.5
1,53
5.4
LLaVA- Video
106
58.6
67.9
63.3
Qwen2.5VLL72B 79
64.1
71.7
21.1
49.9
1
N6
74,0
76.2
65.1
EgoThinker
53.7
80.3
25.2
63.9
Qwen2VL 8
68.2
70.3
62.9
EguThinker
(+1.8) (+2.4)
62.9 (+0.0)
-patto-Temmpoal Grounding Benchmarks. For spatial grounding of hand-object, we measure mean Intersection over Union (mloU) and Localization Accuracy (Loc-Acc). For Loe-Acc, determine the correctness by checking whether the predicted box's center is within the ground truth box. As shown Table 3, off-the-shelf 7B 18-parameter MLLMs achieve only modest mloU and Loc-Acc on Visor. Scaling up to Qwen2.5-VL. VL-72B improves these scores; however, EgoThinker, after our two-stage training paradigm, surpasses this baseline by wide margin.
For temporal grounding, we employ the temporal window's 's mloU and RI @0.05 as metrics. In Table results in EgoFxoLeearn show that 7B- 7B-parameter MILMSs perform near-zero levels, indicating almost no temporal localization capability. Qwen2.5-VLL-722 improves the performance, but still underperforms our EgoThinker. These improvements confirm that our training paradigm effectively equips MLLMs to reason about "when" and "where" in dynamic egocentric video.
General Benchmarks. We further evaluated EgoThinker on three standard video understanding benchmarks. As shown in Table E EgoTThinker exhibits no degradation across any of these tasks, matching or exceeding the Qwen2-VL VL.-7B baseline. Notably, it achieves clear gains on MVBench and Perception Test, underscoring that our two-stage paradigm not only unlocks robust egocentric reasoning but also enhances broad video understanding capabilities.
4.2 Ablation Studies and Discussions
Table 5: Ablations on EgoRe-5M. We use different splits of the data for training to validate the effectiveness of our dataset.
Training Data
EgoTaakkQA
QAEgo4D
Egoschema(val)
EK Visor
Method
Short
Long
CoT
FG
Acc.
Acc.
Acc.
mIoU
Loc-Ace.
Baseline
57.7
60.3
68.2
28.6
64.5
+SFT
61.6
63.1
69.1
29.1
64.9
64.2
63.7
71.1
28.9
64.5
V
d
64.3
67.2
71.9
28.5
64.4
+RFT
V
v
64.4
66.1
71.8
53.7
80.3
Training data. To systematically evaluate the contribution of each component in EgoRe-5M dataset, perform ablations by applying SFT the short-term, long-term, and CoT splits, while employing RFT on the fine- fine-grained grained split. As shown in Table[5] the short-term split boosts performance on three QA benchmarks especially EgotaskQA, highlighting the value of dense, scenario-divend long-term split improves EgoSchema as expected, while incorporating the CoT split leads to marked enhancements in QAEgo4D, dataset focusing ppisodic-memory-bale -baed question answering. This shows that explicit causal reasoning traces can benefit memory-driven tasks. Finally, the FG split significantly enhances spatial and temporal grounding. Overall, these results verify that each split in 5g provides essential, complementary information for different aspects of egocentric reasoning, and that jointly leveraging them enables more holistic understanding of first-person activities.
clips. The
Comparison of SFT and RFT. 6 a direct comparison of SFT and RFT on the EgoRe- 5M-FG split. We can observe that compared to SFT, utilizing RFT significantly enhances the model's

---

performance across all tasks. Additionally, utilizzig SFT affects performance on Egoschema and QAEgo4D, whereas RFT maintains robust performance across these benchmarks. This demonstrates that our method not only enhances patio-ttemoral perception but also preserves the model's ability to generalize across diverse egocentric tasks.
Table 6: Ablations on different training paradigms.
Visov
Egoexolearn
EgoSchema(val)
QAEgo4D
Stage
Method
Data
mloU
Loc-Ace.
mloU
RI@0,05
Acc.
Acc.
Baseline
28.6
64.5
1.53
68.2
60.3
Stagel
SFT
SFT Data
28.5
64.4
1.81
7.0
71.9
67.2
Stagel SFT
EgoRe-5M-FG
38.9
74.1
9.84
24.9
71.4
62.1
Stage2
Stagel RFT
EM0Re-5M-PG
53.7
80.3
25.2
66.1
Input Frame Number. We investigate the im- pact of input sequence length by adjusting the number of sampled frames from 64 during inference, with the results presented in Figure Across all benchmarks, it evident that as the number of frames increases, the performance improves progressively. Notably, a marked de-
Egedeterm Vil
tjerosiie OAEgaIO
30.4
Accunry
cline in performance is observed when only single frame is used, especially for EgoSchema. This indicates that long-term reasoning is intrinsic need in egocentric reasoning.
57.6
54.0
We investigate the impact of input sequence length by adjusting the number of sampled frames from to 64 during inference, with the resufts resufisyreeeemmbbbiiiiii.g. www.gagwwwwmeeeeeeaaaaaaaaaaaaae
Prames
marks, it evident that as the number frames increases, the performance improves progressively, Notably, a marked decline in performance is observed when only single frame is used, especially for EgoSchema. This indicates that long-term reasoning is an intrinsic need egocentric reasoning.
Ablations on ballucination detection. To validate whether our model exhibits hallucinations, we conduct evaluations on two widely-adopted benchmarks: (1) VideoHallucer [8]] for assess- ing MILMs, which contains object-relation, temporal, semantic detail and extrinsic hallucination detection. (2) POPE (Polling-basee -baeed Object Probing)
[53 for object hallucination detection.
The experimental results in demoonsrate
Table 7: Results on hallucination detection bench- marks.
marginal performance decrease (0.6%) on VideoHallucer, while achieving a significant im- provement of 3.2% on the POPE benchmark. We attribute this differential performance to our model's enhanced hand-object grounding capa- bility, which particularly enhances its robustness against object hallucination.
Method
VidcoHallucer
POPE
Qwen2VL X2
47.6
83.6
EgoThinker
47,0
86.8
GPT4o
53.3
Grounding Data and Egocentric Video Understanding. To find the relationship between the spatio-temporal grounding data and egocentric video understanding, we conduct further analysis these benchmarks. We found that benchmarks such as Tgooask QAEgo4D, EgoSchema, and EgoPlan focus on high-level understanding, planning, and reasoning. They only require the identification of simple objects and do not involve fine- fine-grained interaction details. Our baseline model, Qwen2-VL, already possesses certain level of capability, which explains why our grounding data did not yield significant improvements.
Previous works 668 43 demonstrated that using hand and object as supervisory signals enhances egocentric video understanding abilities. Therefore, select three more fine- grained benchmarks: ERQA, EgoMCQ and EGTEA for evaluation.
As shown in Table 8 the inclusion of grounding data significantly improved model performance across all benchmarks. These datasets involve tasks directly related to our hand-object grounding data, such hand/robotic arm motion, object identification, and object classification.

---

Table 8: Ablations the grounding data.
Method
ERQA
EgoMCQ
EGTEA
Qwen2-VL
37.0
86.4/34.1
32.4
Ego SgoThinker(withouw grounding data)
40.1
87,6/38.3
33.1
TgoThinker
41.8
89.3/41.4
35.4
4.3 Qualitative Results
In Figure 4 we show the hand object grounding and reasoning traces for methods. Our baseline Qwen2- VL performs poorly both hand and object grounding. GPT-4o [38] generates coherent chain-of thoougt but misidentifies the target object (mislabeling the knife) and inaccurately localizes the hand. Grounding-DINo (58) is expert model specialized in object grounding, however, it cannot distinguish left from right hand. In contrast, EgoTTiinee first approximates the object location and then outputs accurate bounding box after thinking, demonstrating its superior fine-grained spatial reasoning in egocentric contexts.
Qwee2-VL(BBsstee (sseell xthinksthe object chopping board//hinkk- <answer=[[0661, (0.88, 0.75)]=/answwn
GPT-4o: The chopping board appears to be restangular and located
towards image, primarily some onions. The answer (0.4, 0.3), (0.6, 0.5)].
EpoThinler: <thinks chopping of the image, with portion of visible. it placed on the countertop and used for outting vepetables. </think> <answer> ((0.59, 0,39), (0.9, 1a/ss//
the right side
Grounding: DINO: chopping board knifff/ right hand
Figure 4: Hand-object grounding visualization on Ei-Visor dataset. We compare our method to bascline Qwen2-VL, GPT-4o and expert model Grounding-DINg DINO. We utilize different prompts tailored each model and for each image, we use "chopping board", "knife", "right hand" as query for grounding.
Conclusion
We introduce EgoThiiner, framework toenhance egocentric reasoning in MLLMs. By constructing Eg, a large-scale egocentric instruction-tuing dataset, we provide the rich, structured data required for human-like spatio-temporal understanding and reasoning. To efficiently leverage our data, we employ the SFT-RFT combined paradigm further equip EgoThinker with robust causal planning, long-horizon context integration, and precise localization capabilities. Experimental results demonstrate that EgoThinker achieves state-of-the-arn performance across multiple egucentric benchmarks and significantly improves performance apatio-tempord perception tasks, while maintaining general video understanding abilities. Despite these advantages, Ego possesses certain limitations, such reliance extensive annotations and offline fine-tuning, and cannot perform real-time inference in resource-constraingg time adaptation, richer multimodal integration, and self- supervuee learning to further enhance its robustness and efficiency and we hope to extend EgoThinker into the field Embodied AI, enabling interactive agent behaviors in real-world environments.
environments. Future work will focus on real-
Acknowledgement This work funded in part by the National Key R&D Program of China (202),)0)60201), JSPS KAKENHI JP25K24384, and Shanghai Artificial Intelligence Laboratory.
10

---

References
[1] Jean-Baptiste Alayrac, Jefi Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson. Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 55:23716-23723 2022.
[2] Susan Ashdown and Marilyn DeL.ong. Perception testing of apparel ease variation. Applied Ergonomics, 26(1 4:77-4, 1995.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint #xXiv:2502.13925 2025.
[4] Godas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? ICML, volume 2, page 4, 2021.
[5] Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, and Chenliang Xu. Eagle: Egocentric aggregated language-video engine. arXiv preprint M0Niv::249.172222. 2024.
[6] Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi. Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar. et al. Nemotron-h: A family of accurate and efficient hybrid mamba-transfommer models. arXiv preprint arXiv: 1xXiv:2504.036244 2025.
[7] Zesen Cheng Zhiqiang Hu Yuqian Yuan Guanzheng Chen Sicong Leng Yuming Jiang Hang Zhang Xin Li Peng Jin Wengi Zhang Fan Wang Lidong Bing Deli Zhao Boqiang Zhang, Kehan Li. Videollama 3: Fron- tier multimodal foundation models for image and video understanding. arXiv preprint ar Xivvv:2501.1 13166
2025. URL httpshhts:://// /arpiv////rxvv..ea / abs 25011 13106
[8] Joao Carreira, Eric Noland, Andras about kinetics-600. arXiv preprint arXiv: 1808.0 2018.
Chloe Hillier, and Andrew Zisserman. A short note
[9] Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. Opening the vocabulary of egocentric actions. Advances in Neural Information Processing Systems, 36:33174-33118 2023.
[10] Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al. Interwvicco-eood:: A pack of champion solutions to ego4d challenges. arXiv preprint (tti::22111092222 2022.
[11] Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong La, and Limin Wang. Cg-hench: Clue-grounded question answering benehmark for long video understanding. arXiv preprint 2rXiv:2412.1200 2024.
[12] Guo Chen, Zhiqi Li. Shihao Wang. Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le. Tuomas Rinamaki, et al. Eagle 2.5: Boosting long-contes post-training for frontier vision language models. arXiv preprint arXiv:2504.1522 15271, 2025.
[13] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpe4vidoo: Improving vidco understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472- 19495, 2024.
[14] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3cot: A novel benchmark for multi- multi-step multi-modal chain-of-thought. arXiv preprint 2405. 16473, 2024.
[15) Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models. CORR, 2023.
[16] Zhe Chen, Weiyun Wang. Yuc Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and lest-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[17] The Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1682. 2024.
[18] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitcher kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence. 43(11):4125-414) 2020.
11

---

[19] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar. Richard Higgins. Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Dataxets and Benchmarks, 2022.
[20] Huilin Deng, Ding Zou, Rui Ma. Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generaliza- tion and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint ar 200002500300065, 2025.
[21] Shangzhe Di and Weidi Xie. Grounded question-answwringg in long egocentric videns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934-12943, 2024.
[22] Victor Escorcia, Ricardo Guerrero, Xiatian Zhu, and Brais Martinez. Sost self-supervised learning over sets of handled objects in egocentric action recognition. In European Conference on Computer Vision, pages 604-604-00 620. Springer, 2022.
[23] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi- Ilms in video analysis. arXiv preprint 24ti::2000.2207555 2024.
[24] Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13505-1 1051-,355,,,111 2021.
[25] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[26] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understand- ing skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19383-19400, 2024.
[27] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
[28] Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment network for long-term vidco. In CVPR, 2022.
[29] Yuping He, Yifei Huang, Guo Chen, Lidong Lu, Baoqi Pei, Jilan Xu, Tong La,, and Yoichi Sato. Bridging perspectives: A survey on cross-view collaborative intelligence with egncentric-exocentri vision. arxiv preprint 2r11v:2506.06255 2025.
[30] Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang. Egoexobench: benchmark for first-and third-person view video understanding milms, arXiv preprint arXiv:2507. 2025.
[31] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Miniepm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint XxXiv:2404.06339 2024.
[32] Yifei Huang, Minjie Cai, Zhengiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning dask-dependent attention transition. European Conference on Computer Vision, 2018
[33] Yifci Huang, Minjic Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato. Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions on Image Processing, 29:7795-7806, 2020.
[34] Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph- based temporal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14024-14034 2020.
[35] Yifei Huang. Lijin Yang, and Yoichi Sato. Weakly supervised temporal sentence grounding with umcertainty-guided self-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18908-18918 18918, 2023.
[36] Yifei Huang. Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo- exo-centric view of procedural activities in real world. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 2-0066 2024.
12

---

[37] Yifei Huang, Jilan Xu. Baoqi Pei, Lijin Yang, Mingfang Zhang, Yuping He, Guo Chen, Xinyuan Chen, Yaohui Wang, Zheng Nie, et al. Vinci: A real-time smart assistant based on egocentric lsionnlanguage model for portable devices. Proceedings of the ACM on Inteructive, Mobile, Wearable and Ubiquitous Technologies, 9(3) 2025.
[38] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[39] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
[40] Yunseok Jang. Yale Song. Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1758-2766,22 2017.
Toward spatio-
[41] Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-lask clivities. European Conference on Computer Vixion, pages 767-786. Springer, 2020.
[42] Baoxiong Jia, Ting Lei, Song-Chun Chun Zhu, and Siyuan Huang, Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 1113343-33600 2022.
[43] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11 11107-11116, 2021.
[44] Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al. Token-efficient long video understanding for multimodal llss. arXiv preprint MrXiiv2503.04130 2025.
[45] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-languaag- action model. arXiv preprint 094.04 2024.
[46] Yi Wang Yizhuo Li Wenhai Wang Ping Luo Yali Wang Limin Wang KunChang Li, Yinan He and Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023.
[47] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TV"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 1.15124, 2024.
[48] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara I. Berg. Tvqa: Localized, compositional video question answering. arXiv preprint 1 2018.
[49] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024.
[50] Jia Li, Ge Li,, Yongmin Li, and 7hi Jin. Structured ohain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1-23, 2025.
[51] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-2231 2222022 2024.
[52] Xinhao Li. Ziang Yan. Desen Meng, Lu Dong, Xiangyu Zeng. Yinan He, Yali Wang, Yu Qiao, Yi Wang. and Limin Wang. Videnchat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.066958 2025.
[53] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ro-g Wen. Evaluating object hallucination in large vision- langugge models. arXiv preprint 2rXiv:2355.103: 2023.
[54] Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang. Vibashan Vs, Yishen Ji. Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post- trainiraining data strategies from scratch for frontier rision-langual models. arXiv preprint arXiv:2501.148. 2025.
13

---

[55] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
[56] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan. Eric Z. Xu, Difei Gao. Rong Boog-ChengTTee Tu, Wenzhe Zhao, Weijie Kong. et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7555 2022.
[57] Tsung- Yi Lin. Michael Maire. Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 6-12, 2014, proceedings, part v 13, pages 740-755. Springer, 2014.
[58] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[59] Yugi Liu, Bohao Peng. Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning chaain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 2025.
[60] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0178; 2025.
[61] Lidong Lu, Guo Chen, Zhiqi Li, Yicheng Liu, and Tong Lu. Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for milms. arXiv preprint 2rXiv:2556.03322 2025.
[62] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgnt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv22306.05422 2023.
[63] Muhammad Maaz. Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
[64] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36:46212 46244,, 2023.
[65] Antoine Micch, Dimitri Zhukov, Jean- bran-Baptiste Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto Learning a text-vidco embedding by watching hundred million narrated vidco clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630-2640, 2019.
[66] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models follow instructions with human feedback. Advances in neural information processing syxtems, 35:27730-27744, 2022.
[67] Viorica Patraucean, Lucas Smaira. Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36: 42748-427611 2023.
[68] Baoqi Pei, Guo Chen, Jilan Xu, Yuping Hc, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang. Tong Lu, Limin Wang, et al. Egovideo; Exploring egocentric foundation model and downstream adaptation. arXiv preprint 2rXiv:2406.1880 2024.
[69] Baoqi Pei, Yifei Huang, Guo Chen, Jilan Xu, Yali Wang, Limin Wang. Tong Lu, Yu Qiao, and Fei Wu. Guiding audio-visual question answering with collective question reasoning. International Journal of Computer Vision, pages 1-18, 2025.
[70] Baoqi Pei, Yifei Huang, Jilan Xu, Guo Chen, Yuping He, Lijin Yang, Yali Wang, Weidi Xie, Yu Qiao, Fei Wu, et al. Modeling fine-grained hand-object dynamics for egocentric video representation learning. arXiv preprint arXiv:2503.009m 2025.
[71] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu. Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint 250300500.. 2025.
14

---

[72] Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Matteo Matteucci. and Barbara Caputo. E2 (go) motion: Motion augmented event stream for egocentric action recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19935- 19947, 2022.
[73] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovipv2: Egocentric video- Iadoo-anguaage pre-training with fusion in the hackbone. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5285-5297, 2023.
[74] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 9869-9878, 2020.
[75] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang. Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of offin-of-hhughh reasoning in molti-modal language models. arXiv e-prints, pages arXiv- 2400, 2024.
[76] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu. Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning open language models, arXiv preprint arXiv:2402.0 030003000 2024.
[77] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550 2024.
[78] Gemini Robotics Team, Saminda Abeynuwan, Joshua Ainslie, Bran-Baptiste Alayrac Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl. et al. Gemini robotics: Bringing ai into the physical world, arXiv preprint arXiv:2503 2025.
[79] Qwen Team. Qwen2.5-vl, January 2025. URL btttps://peen.m.giiiub.....
v/
[80]
Daniel Theron, Sentence transforrmer for audit retrieval question-answertt (star-qa), Feb 2024, URL at t ps: wttp://uuuggggaaccco.oooo
STAR-SARRRRO QA.
[81] Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin. Zhengyuan Yang. Lijuan Wang, and Mike Zheng Shou. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training. arXiv preprint arXiv:2401.0084 2024.
[82] Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[83] Shihao Wang, Guo Chen, De-an Huang, Zhiqi Li, Minghan Li, Guilin Li, Jose M Alvarez, Lei Zhang, and Zhiding Yu. Videoitg: Multimodal video understanding with instructed temporal grounding. arXiv preprint arXiv:2507.1 13353,230000 2025.
[84] Yi Wang, Yinan He. Yizhuo Li. Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Intervvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint (xXix:2307.06944 2023.
[85] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internviden2: Scaling video foundation models for multimodal video understanding. arXiv preprint 3ixx:2403.153777 2024.
[86] Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video milmss with long and rich context modeling, arXiv preprini arXiv:2501.12356 2025.
[87] Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging Ilms for answering queries in long- form egocentric videos. arXiv preprint arXiv:2312.05526, 2023.
[88] Yuxuan Wang. Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint 1X0:22406.16338 2024.
15

---

[89] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[90] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang. Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data- lssionlanguaaee robotic manipulation. IEEE Robotics and Automation Letters, 2025.
models for
[91] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9777-9786, 2021.
[92] Junbin Xiao, Angela Yao. Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13204-13214, 2024.
[93] Jilan Xu, Yifci Huang, Junlin Hou, Guo Chen, Yuejic Zhang, Rui Feng, and Weidi Xie. Retrieval- augmented egocentric video captioning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13525-13536, 2024.
[94] Jingkang Yang, Shuai Liu, Hongming Guo. Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zátang Zhou, Binzhu Xie, Ziyue Wang. Bei Ouyang. Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang. Bo Li, and Ziwei Liu. Egolife: Towards egocentric life assistant, 2025. WtRphhtss://arx. /rg/
abs/ 2203.503..o0 03803
[95] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI-onevision; Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[96] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junho Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.00 018000 2024.
[97] Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang. Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal llms. arXiv preprint 24100:241 2024.
[98] Kexin Yi. Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenen- baum. Clevrer: Collision events for video representation and reasoning, arXiv preprint arXiv: 1910.0044 2019.
[99] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang- Vision-rl: Evolving human-free free alignment in large vision-languaa models via vision-guided reinforce- ment learning. arXiv preprint 2xXi:25503.18133 2025.
[100] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13901-139111 2023.
[101] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunee tuned audio-visud language model for video understanding, arXiv preprint arXiv:2306.0246 028558 2023.
[102] Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided milm for egocentric video understanding, arXiv preprint arXiv:2503.091 09143, 2025.
[103] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning reason with multimodal large language models via step- group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
[104] Peiyuan Zhang, Kaichen Zhang. Bo Li, Guangtao Zeng, Jingkang Yang. Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwci Liu. Long context transfer from language to vision. arxiv preprint arXiv:2406. 16852, 2024.
[105] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023.
16

---

[106] Yuanhan Zhang. Jinming Wu. Wei Li. Bo Li. Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. URL https: /arxiv.ivvv.. org/ abg//bbs/2400.... 02713
[107] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 2022.
[108] Yue Zhao and Philipp Krähenbühl. Training large video model on single machine in day. arXiv preprint arXiv:2309. 20w.22999.666, 2023.
[109] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedingy of the European conference compuler vision (ECCV), pages 803-818. 2018.
[110] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero's" aha moment" in visual reasoning 2b non-sft model. arXiv preprint arXiv:2503.05133 2025.
[111] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[112] Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotexivqa: Towards egocentric scene-text video question answering. arXiv preprint DXXiv25022.07111 2025.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have written contributions and scope in the abstact.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and our future works in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the statistic of the data and hyperparameters in training.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification: We provide the statistic of the data and training setting in the supplementary, but we will not provide data and code in the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipss..
public/ htte:/guu.dess/oeSuuummm CodeSuummeieieeiiicooeo sssonooliccyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
c/ des CodeSubmn Sum ss onPol icyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide these information in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the training cost and time for SFT and RFT training in the experiment section and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpppp///euuiipp.cc///a/beaaaaa
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
This work a foundational research, and will not have societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetatett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not involve LLMs any important, original, non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: www.eeeripp..c/oofeeeeeeeeeeeeee 2025/ LLM) for what should should not described.
24

---

A Details About EgoRe-5M
Video Source. After obtaining our filtered 8.7M video clips from web data, we combine them with existing egocentric datasets including Ego4D [25]. EPIC-Kitchens [88], EgoExoLearn [36). and EgoExo4D [26] to form a comprehensive dataset totaling 13 million clips. Specifically, we utilize 4M clips from [73] within Ego4D. For EPIC-Kitchens, we select 56,000 frames from the EK-Visor dataset. From EgoExoleearn, we carefully curated 10,000 L1-level video clips for temporal grounding training and 400 samples for the RES benchmark. Regarding EgoExo4D, we incorporated 500 samples for the RES benchmark evaluation.
Dynamic Interaction Filtering. Since many video clips remain static or depict group activities, we need to filter out irrelevant clips for egocentric reasoning. After obtaining object bounding boxes for each frame using a hand-object detector, we design the following filtering rules to select video clips with dynamic interaction:
Step 1: For all video clips, we first examine the hand bounding boxes. If the number of hands detected exceeds two (N hands > 2) indicating multi-person activities, we discard such clips:
Filter Clip+>Nhands >
(3)
Step 2: For the remaining clips, given clip C' with N frames, we set a threshold = 0.7. The clip is discarded if the total number of object bounding boxes is less than N:
N
Filter Clip
N objects <axN
Step 3: After the first two steps, we obtain clips containing single-person hand-object interactions. To further select dynamic clips, we calculate the maximum inter-frame hand displacement. For clip C. we compute the hand center (zt, ye) in each frame The clip kept if the maximum center displacement exceeds 10% of the image size (min H, W)):
Filter Clip EmE 11 VEr, 22,,) + (y,, Vr,) > 0.1 x mán (H, W)
(5)
Through these three steps, we filter out high-quality dynamic video clips, which can be used to train egocentric reasoning MILLM.
Data Statistics. Table presents the details of EgoRe-5M, including 16 question types and cor- responding QA examples for each type. Notably, in the Chain-of- -hhoughht (CoT) split, we provide detailed and meaningful reasoning processes to the question. For the Fine-Grained split, we introduce two special tokens, <think> and <answer>, to structurally format the reasoning process and final answers, which facilitates the execution of our reward function during model training.
B
Training
SFT Data. To balance training efficiency and model performance, we carefully curated our training dataset as shown in Table 1. While using the complete dataset would lead to prohibitive computational costs and performance degradation due to data imbalance, We filter each dataset for video caption dataset, we select 170K samples on total; for ego-related dataset, we select 390k QA samples in total; for our 55, we select 810K samples, including 410K from short-term splits, 400K from long-term split and 50K from CoT split.
Training Details. We use QwenVL2-78 as our baseline for training. For SFT, we adpot max pizels = 200704, min pizels 3136, Ir le 6, epoch = for training. We utilize 32 A100 GPUs and train for 30 hours. For RFT, we adpot Ir le 5. epoch = 1 for training. We utilize 8 A100 GPUs and train for 12 hours. Notably, during RFT training phase, we first train hand-object grounding task and then train temporal grounding task.
25

---

Data Split
Question Type
Number
Example
Objoct
302K
What object is the person interacting with in the vidso?
Existance
The person inteneting with grass-trimmin
Objoct
326K
What the stute the garlic during the sliclin process?"
Atribuue
The garlic heing slioed und
place by the left
Object
How mony people ore the video?
Coun:
There are three people in the video.
Short-term
Object
the oleff doing with whiie snup bost??
Interactiio
The thef is rinsing the w hitte soup bowl under running water ensure it thuroughll cleam.
Action
What actions being performedd the hands the rideo?
280K
Deseription
The left hand moves dowsward to an object, while the right hand pulls a handle to the side.
Action
440K
Why does the left hand move to the center of the frome?
Reasoning
The left hand moves the center the frame pick book.
Background
What the setting or buckgroond of the video?
Attribuee
293K
The back ground appeors be outdooe srea graes, likely lown or garden.
What the seguence of actions performed by the lift and right hunds whin catting and placing nungo pievess
Action
412K
The left hand moves downward to place the mango the cutter while the right hand holds cutter steady. Then, the left hand stabilizes the mango while the right hand cuts after cutting, the left hand moves downward to place the mango piece while the right hand remsins
Sequence
When does person demonstrate a clear proference for uring
lift hond?
Grounding
Temporal
411K
Penson a demonetratos clear preference for using their left hand when they remove the eap
and place on table, which occurs between 7.33 to 10.25,
How mony objerte being interarted will actore the video clijs?
Object Coust
411K
The objects being interacted
include an brush, point, an board.
Long term (15-1206)
piece of paper, tissue, and paint poletic, toraling six objocts.
Action
410K
After person e places a piece of closk into the bog, whhr in the fikelly next action?
Prediction
The likely nest action person suning clothes in bug.
What are the fey actions involingg the left hand across the video clips?
Action
411K
The key octions involvinng the left hand include person y picking up the phone, person 2 removing and placing the cup on the tiblle person picking up camen from parson k. and the left moving in various discetions while interacting with objects.
Summory
Why does persoe book hunns apply plester the relling?
Action Reasaning
The person uses both hands to ensure precision and control while applying plaster. One hand scoops
412K
guides the plaster, while the other
the scrapes,
allowing for in even and smooth application on the exiling.
How did the person udjost their handd postiooing white operating the losommnes? As deserihed in 'Peson uses both hands to lift shovel"), then dug into the soil with it
('Perana digs into the soil with shovel, nsing hands"). ") After digging, he spead the will using his left hand while stealying container with his right ('Man& A spreads soil with his left hand. while his right hand steadies the container's ). Finally, after vompleting task, he carefilly placed the shovel on grass guiding down with his left hand and steadying the handic with his right ( ('Man A carefully places the shovel on the gmass hand guiding down while his right hard steadies the handle'), ) This sequence shows a logical workllow litlng the tool, executing the primary action (digging) distributing the material, safely storing
CoT (15-200ls)
Chain-of-thought Reasoning
SOK
afterward.
identify intervai from which the answer to the question can be deduced. Output your thought process within the cchinks c.hiink-taggg. Thos, provide placess in the farmat "(ssort endy" within the commerrs aonnerr-cuussseee counswerrssugs
To accurately pinpoint the "QUESTION]"
video, you
Temporal Grounding
10K
end times seconds, precise decimal
Fine-grained Grounding
This an image containing un object. "[OBJECT"" and ouput bounding of this abject in the image. Output your thought procers within the <think 2/hhink stnge, This provide your bounding hox within the <answer>> commoerrstng tngs following <ommer>>(( miny min) maxy mar) <Jomner>-forott The bounding hox coondinates are nommllccd the rauge (0, 1j, relative to the width and hright the image.
Hand-Giiject Grounding
56K
Table 9: Statistics of the proposed EgoRe-5M. The table shows 4 data splits and 16 question types, along with the corresponding example question-answer pairs. More details can be found in Section 3.1.2
C Benchmark Details
EgoTaskQA. Eg 42 is large-scale egocentric video questionanswering dataset designed to evaluate models' understanding of goal-oriented human tasks. It is derived from LEMMA dataset [41), focusing on aspects such as action effects, intent, multi-agent collaboration, and object interactions. The dataset emphasizes reasoning types including spatio-temporal understanding, causal dependencies, and task planning, supported by 30K annotated state transitions. It includes a variety of question formats, such as binary and open-ended queries. to ensure balanced and unbiased evaluation. To evaluate this dataset, we reformulate the original open-ended QA samples into a multiple-choice question through systematic conversion process. Specifically, we first aggregate all potential answers into a list. For each question, BERT 44 is used to compute semantic similarity scores between the ground-truth answer and all candidate answers in the pool. The four most semantically similar answers were then selected to construct the new multiple-choice question, with the fround-truth answer serving as the correct option.
26

---

QAEgo4D. QAEgo4D represents a specialized benchmark for assessing episodic memory through video-based question answering. This dataset, derived from the Ego4D, measures the ability of vision-language models to comprehend and reason about dynamic visual sequences. Each entry consists of four key components: (1) an egocentric video clip, (2) a manually crafted question, (3) its corresponding answer, and (4) precise temporal localization of the relevant visual evidence. To ensure annotation quality, the dataset employs redundant textual descriptions that undergo sross-verification QAEgo4D provides researchers with robust framework for investigating memory-relatee video understanding tasks. To evaluate the dataset, we select the closed-set QA split parsed by [92].
EgoPlan. EgoPlan serves as multimodal benchmark for evaluating human-lite like planning abilities in AI systems through egocentric video understanding. Derived from large-scale egocentric datasets including Epic-Kitchens and Ego4D, the benchmark comprises 4,939 rigorously validated multiple- choice questions, spanning 3,296 distinct task objectives and 3,185 executable action sequences across 419 diverse real-world environments. By simulating real-world decision decision-making scenarios, the benchmark facilitates progress in multimodal reasoning for practical planning applications. In our experiments, we adopt the dataset's 's predefined validation split for evaluating planning performance, as ground-truth annotations for the test set remain undisclosed.
EgoSchema. EgoSchema represents a novel benchmark framework for assessing long-form video comprehension in multimodal AI systems. Derived from the Ego4D video corpus, this evaluation suite comprises 5, 5,000+ meticulously annotated multiple-choice hours of unscripted daily human activities captured in real-world settings. The benchmark presents unique challenge where AI models must analyze three-minute video clips and select the most accurate response from five plausible alternatives, testing their capacity for sustained visual-temporal reasoning and contextual understanding.
peesstion-nnswer pairs, sourced from 250+
EgoMCQ. EgoMCQ is multiple-choice question-answerring dataset designed to assess video-text alignment in egocentric vision systems. Derived from Ego4D, it includes 39,000 questions based on 468 hours of egocentric video covering wide range of human activities. Each question involves selecting the correct video clip from five options based on a narration, with two settings: "inter-video", for distinguishing between different videos, and "intra-video", for fine-grained context within the same video.
VLN-QA. VLN-QA represents a specialized evaluation benchmark for assessing multimodal navigation understanding in indoor environments through question-answering tasks. Derived from the VLN-CE framework, this dataset comprises thousands of carefully annotated multiple-choice items paired with egocentric video sequences that replicate authentic navigation scenarios. The benchmark specifically examines a system's 's ability to interpret visual-spatial information and correlate it with textual queries. For our implementation, we utilize the preprocessed dataset version established in VideoChat2's s experimental setup.
To validate our model's 's cross-view reasoning capability, we developed a Cross- Skill Transfer Benchmark named RES (Referenced Egocentric Skill). RES leverages paired exocen- tric- eoocengcct clips from the EgoExoLearn and EgoExo4D datasets. Each example presents one exocentric video as a reference and four candidate egocentric clips, and the model must identify which egocentric view corresponds to the reference. This multi-choice protocol rigorously tests the ability to transfer observed skills across perspectives. The final benchmark comprises 936 curated samples. Although RES is crafted to validate EgoThhinkrr 's cross- -ross-view reasoning, we anticipate it will become a valuable resource for the broader embodied AI community.
RES.
Grounding Benchmark. For the grounding benchmark construction, we select existing annotations to derive our evaluation dataset. Specifically, for the hand-object grounding benchmark, we curated our dataset from Ei-Visor, which provides bounding box annotations. Our methodology involved extracting bounding boxes from segmentation masks in the validation set, serving as ground-truie references. This process yielded comprehensive collection of 13,000 object queries for evaluation purposes. For the temporal grounding task, we strategically selected the Egofixol Learn dataset due to its unique dual-level annotation structure, which makes it suitable for temporal localization tasks. We select LI -leell (coarse-grained grained) video clips as our primary video sources and L2-level (fine- (ine-grained)
27

---

temporal windows as precise ground truth annotations. To this end, we curate an evaluation set of 3,000 test instances.
D Additional Experiments
Effects Of Extra Video Sources. Table I0lleTO0iaa provides a comparative analysis of model performance with and without the inclusion QA samples sourced from the How To100M dataset. The results indicate consistent performance improvements across all evaluated benchmarks when leveraging the HowTo100M-deree data, with particularly notable gains on long-term understanding tasks such as QAEgo4D and EgoPlan. We attribute these improvements to the long-term split in EgoRe-5M, which primarily derived from How TowTooooe 100M, significantly enhancing the model's capacity for extended temporal reasoning. These findings underscore the effectiveness of our data curation strategy in enabling robust egocentric reasoning ability.
Table 10: Ablations on our EgoRe-5M. We evaluate the impact of incorporating data filtered from the HowTo100M dataset on performance.
EgoTaskOO Acc.
QAEgo4D Acc.
EgoPlan-Vas Val Acc.
VLN-QA Acc.
Data
Baseline
57.9
60.3
38.3
42.0
w/o Howtol 10M
62.2
61.6
41.3
50.0
Howto100M
64.4
66.2
47.1
54.0
Results On General Grounding Task. To further validate EgoTThink's grounding capabilities, we conduct additional experiments on the COCO dataset [57]. As evidenced by Table [1] our model demonstrates significant performance improvements despite never being trained on COCO data, which substantiates its strong generalization ability for object grounding tasks.
Table 11: Results on COCO detection dataset.
testA
testB
Method
mloU
mIoU
Qwen2VL-7B
34.1
33.6
EgoTThinker 55.2(+21.1)
57.8(+24.2)
Additional Visualization Results. Figure compares temporal grounding outputs for the baseline, sample query. We observe that Qwen2-VL fails to identify the query and lacks reasoning process in both examples. Gemini2.5-Pro provides dense video captions and accurately align the captions with the corresponding events. However, the experimental results reveal that the model exhibits inferior localization performance when processing scenes with hand-object interactions. Egonhinker combines an explicit reasoning trace with precise temporal window prediction, accurately identifying the start and end times of the queried event. These visualizations underscore EgoThinker's unique ability deliver both interpretable reasoning and robust spatio-tempore temporal grounding in first-person video.
Gemini 2.5-Pro [77], and EgoThinker
28

---

Video
With his right hand holding the pot. he walked towards the sink. With his left hand, he turned on the faucet, filling the pot with water. Then, be used his left hand turn off the faucet.
Query:
GT:
3.0s
Qwenn--Ve (i.aallll
times (448,.
34%
canswer=
70)- aanswers verify0::00 Person stove, puts
Person picks
the wok (pot) with their right 0a0d:::03: Persen is walking towards the sink with the wok in their right hand.0:44: Person places the flowing water, 0:11: Person uses their left hand to turn off the faucet.0:12 Faucet completely eftSo the start time when the person picks turned (0:12).The description
sink
Wok filling
pot to pot, be walked towards the sink' starts at 0:02.
(0:02),
right holding
Egothinker: sthinkethh event "With his right hand holding the pot, he walked towards the sink. With his left hand, turned on the fauret. flling up the pot with water: Then, he used his left hand turn off the faucet." occurs in the first the videos videos/thinle> prcanseer=(344 (344, 10.44)</anwwer>
3.46
10:4ss
Videex
Query: Twist gas stove with your right hand. Turn on flame.
GT:
12.4s
17.0
Qwen2- (i.asllll
(1.1,23.1).
Is
23
canswers (0:13 1300:155 5 /onstern Start/Ean TimesOkay, understand timestamps in the video where the action begins and concludes. I'll focus on the visual cues: the right hand approaching the knob, the twisting motion, uppearance your right hand. Turn on the flame" occurs in the video between 0:13 and 0:16.
to pinpoint precise
flame. straightforward analysis really: action knob ofe gas stove
Egothinker: ethinkethhe event "Twist knob of the gas stove with your right hand. Turn flame." occurs after the person puts the pot the gas and before the opens the tridges//hinee (hhine/////////// waw.le////ssssseee111.. 16.0)=/answer>
10s
16s
Figure 5: Temporal grounding visualization on the EgoExoLeat dataset. We compare our method to baseline Qwen2- VL and one of the strongest MILM Gemini2.5.
29


Reviews:
[{"review_id": "RmY6UDZujB", "reviewer": "Reviewer_vznn", "summary": "EgoThinker introduces a two‑part contribution to egocentric video reasoning.  \n1. **EgoRe‑5M dataset** – 5 million question‑answer pairs source from HowTo100M and existing EGO dataset spanning short‑term/long‑term/chain‑of‑thought/fine‑grained spatial/temporal grounding four main tasks.\n2. **Two‑stage training pipeline** – SFT on EgoRe‑5M followed by GRPO fine‑tuning on the grounding split.  \nThe resulting 7 B‑parameter model outperforms prior multimodal LLMs on egocentric benchmarks while matching general‑video performance.", "questions": "As the point to differentiate the paper is egocentric reasoning, the following questions are most related to the reasoning aspect of the paper.\n\n1. Justification of reasoning\nWhile it is acknowledged that certain egocentric tasks require varying degrees of reasoning, the paper does not discuss which types of queries benefit from what types of reasoning paths, nor does it justify whether such reasoning is necessary or accurate for each case. Please provide several fully worked examples spanning different query types, and clarify whether all long-horizon questions require a CoT rationale, or if the model can skip CoT when a direct answer suffices. Additionally, please comment on whether the DeepSeek V3-generated reasoning paths are effective for solving these queries.\n\n2. Your current experiments stop at a 7B-parameter backbone, yet recent CoT literature (When More is Less: Understanding Chain-of-Thought Length in LLMs, Small Models Struggle to Learn from Strong Reasoners) shows that stronger reasoning benefits often surface only at larger scales. Could you report results for at least another series model—say, InternVideo 2.5-8 B to verify that the CoT-induced gains persist or even strengthen? \n\n3. Hallucination and factual alignment audit\nBecause the CoTs are generated by a language-only DeepSeek LLM that never “sees” the video, there is a risk that rationales introduce objects or events absent from the footage. Could you (a) evaluate on an existing hallucination benchmark such as VQ2A-Hallucination or Image-CoT-Faithfulness. Presenting these numbers would let readers judge whether CoT length correlates with fidelity.\n\n4. Grounding relevance\nEgocentric frames are often cluttered with irrelevant objects. Please include an ablation study where the grounding reward is either removed or down-weighted:\n    • Does rationale faithfulness (as measured above) improve or degrade?\n    • Are certain question categories (e.g., action anticipation vs. spatial localization) more sensitive to grounding?\nThese results would help clarify whether grounding genuinely supports reasoning or simply provides an additional supervision signal.\n\nAddressing these questions would substantially strengthen the paper, as many of the practices employed are standard in the field and offer limited novelty.", "limitations": "Same in the Questions section.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Extensive experiments demonstrate consistent state-of-the-art gains.\n- Paper is well-structured with clear figures for pipeline, data statistics, and qualitative results.\n- EgoRe-5M dataset will likely become a community resource.\n\n**Weaknesses:**\n- Chain-of-Thought justification: The paper does not clarify to what extent perception queries require CoT reasoning or what formal structure is assumed, only a few illustrative examples are provided.\n- Hallucination / misalignment risk: CoTs are produced by a language-only DeepSeek model that never “sees” the video, so hallucinations are possible. No benchmark or audit quantifies factual alignment.\n- CoT reliability: Longer rationales often amplify hallucinations, the paper adds more/longer CoTs but provides no hallucination benchmark (even an image-based one).\n- Grounding relevance: Reward encourages pointing to any object, yet egocentric frames are cluttered. The authors do not show that grounding improves CoT quality or downstream accuracy, nor do they justify which objects matter.\n- All annotations are synthetic, human validation is limited to a 500-sample spot-check.", "was_revised": false, "final_justification": "", "rebuttal": "Due to the word limit, we will simplify the description in the rebuttal.\n\n### **[Q1,W1] CoT data justification and egocentric reasoning cases.**\nWe thank the reviewer for this insightful question. \nFor cot data justification, We analyze from two perspectives:\n\n1.Additional Benchmarks:\nWe test EgoThinker on 2 additional benchmarks designed to measure complex reasoning, not just perception: ERQA[1] and VidEgoThink[2].\n| Model    | ERQA | VidEgoThink(Reward Model) |\n|---------|---------|---------|\n| Qwen2-VL | 37.0 | 48.8 |\n| EgoThinker | 41.8 | 51.7 |\n| Gemini1.5 | 41.8 | - | \n\nERQA, proposed by Gemini Robotics, has various reasoning types such as spatial, trajectory and action reasoning. As shown in the table, our model's performance is comparable to Gemini 1.5, surpassing the baseline by 4.8%. For tasks requiring the model to infer whether an action was completed or understand long-term information (VidEgoThinker and QAEgo4D), EgoThinker's performance on both benchmarks shows our model's superior performance in reasoning tasks.\n\n2.Examples of Egocentric Reasoning:\nThe following examples, derived from our dataset, illustrate why a direct-answer approach is insufficient for these query types. As we are unable to upload images, so we only present QA examples.\n\n(1) Task Planning:\nNaturally, solving this problem requires breaking down tasks and providing a reasoning path based on logical relationships.\n\n【Q】: How to add mortar mix and cement into the concrete mixer?\n\n【A】: Pick up a sack of mortar mix with left hand while using right hand to hold tools. Then open the sack by tearing the seal with left hand while the right hand steadies the bag. Afterward, pour both the mortar mix and cement into the concrete mixer.\n\n【Why CoT is necessary】: Without reasoning, the outputs is \"You should pick up mortar mix and cement, and pour them into the concrete mixer.\" This is clearly insufficient since it lacks the actionable, sequential details.\n\n(2) Complex Event & State Inference:\nThis task requires combining multiple implicit cues for accurate judgment; relying solely on direct observation without reasoning can lead to incorrect answers.\n\n【Q】: Did the person complete the vegetable washing process?\n\n【A】: The man put the vegetables into the basin and turned on the faucet, but then the camera moved elsewhere. However, when the camera returned to the basin, the man had turned off the faucet and removed his hands from the basin. Therefore, it can be inferred that he completed washing the vegetables.\n\n【Why CoT is necessary】:In the scenario, we can't directly determine if the task is complete. However, by reasoning through the man's actions, camera movements, and the logical sequence, we can arrive at the answer.\n\n3.Embodied reasoning: This requires combining spatial understanding with the context of an action.\n\n【Q】:How do you need to rotate the dumbbell for it to fit back in the weight holder? \n\n【A】:'To determine how to rotate the dumbbell for it to fit back in the weight holder, we first need to understand the orientation of the dumbbell and the weight holder.\\n\\n1. **Grounding of the Key Object**: The dumbbell is the key object here. It is shown lying on its side with the handle pointing upwards. The weight holder is positioned above the dumbbell, with its base facing downwards.. **Current Position**: The dumbbell is currently lying horizontally on the floor, with its handle pointing upwards. The weight holder is positioned above it, ready to receive the dumbbell.\nThus you need to rotate dumbbell counter-clockwise 90 degrees and move it above to fit back in the weight holder.\n\n【Why CoT is necessary】The answer depends on a multi-step mental process: (1) Identify the spatial information of the dumbbell and the target holder. (2) Calculate the necessary transformation. A direct perceptual guess is likely to fail.\n\nFurthermore, in scenarios involving camera transitions, spatial understanding, and motion understanding, integrating these pieces of information for reasoning is vital. However, most existing egocentric benchmarks focus on understanding and simple reasoning. Developing more complex egocentric reasoning benchmarks is a promising direction for future exploration.\n\nRegarding whether CoT is always required, the answer is no. As described in the paper, we first assess if a video segment was complex enough to warrant a CoT. For simpler queries, it is not generated or enforced.\n\n### **[Q3,Q5,W2] Experiments on hallucination detection.**\nWe thank the reviewer for raising this critical point regarding the risk of hallucination. Since the CoT rationales are generated from text captions without direct video access, ensuring factual alignment is paramount. We acknowledge this challenge.\n\nWe've researched Hallucination detection and found that most benchmarks to test models for hallucinations rather than verifying if captions contain hallucinations. Methods like VQ2A-Hallucination or Image-CoT-Faithfulness aren't open-source. We also tried using the latest FClip, and ALOHa for verification, but these methods primarily evaluate image hallucination, and weren't suitable for our complex semantic and temporal information. Therefore, we conduct a 4-part audit of our EgoRe-5M CoT data to quantify its factual fidelity.\n\n1.Annotation Process.\nAs detailed in the paper, the CoT data was generated by DeepSeek-R1, a model with strong instruction-following capabilities. Our prompt explicitly constrained the model to generate rationales only from the information present in the source video caption, strictly forbidding the introduction of external or imagined details. This largely guarantees that the generated CoT won't contain hallucination.\n\n2.CLIP-based Alignment Quality Analysis. We analyze the alignment quality between the caption and the video via the CLIP score. Specifically, we chose ViCLIP[3] pre-trained on a large amount of video data, and compared it against the original concatenated captions as a baseline. We found that the score of our CoTs is 62.3, while the original caption score is 70.6. \n\nWhile the CoT score is lower than the descriptive caption baseline, this is an expected outcome. The source captions describe concrete, visible objects and actions, which aligns perfectly with CLIP's training objective. In contrast, our CoT rationales articulate abstract reasoning steps, causal links, and inferred goals (e.g., “to align the dumbbell, a rotation is needed”), which do not always have a direct visual correlate. A score of 62.3 still indicates strong semantic alignment, confirming that the rationales are highly relevant to the video's content.\n\n\n3.Manual Review.\nBased on the length of the generated CoT, we selected 5k samples from our CoT data split for manual review. The goal of this scoring was to assess whether the generated caption contained any non-existent objects or action descriptions when compared to the video. We set up a 4-point scale to categorize hallucination severity:\nSevere Hallucination, Moderate Hallucination, Partial Hallucination and No Hallucination.\nWe get 4068, 577, 279, 76 for score = 3,2,1,0. We observe that hallucinations primarily manifest as the model outputting non-existent goals or actions based on the provided captions, with hardly any instances of non-existent objects. Furthermore, the majority of captions exhibit no hallucinations or only minor ones, indicating good alignment quality in our CoT generation.\n\n4.To validate whether EgoThinker exhibits hallucinations, we conducted evaluations on two widely-adopted benchmarks: (1) VideoHallucer[4] for assessing large video-language models, which contains object-relation, temporal, semantic detail and extrinsic hallucination detection. (2) POPE (Polling-based Object Probing)[5] for object hallucination detection. \n\n| Model    | VideoHallucer | POPE | \n|---------|---------|---------|\n| Qwen2-VL | 47.6 | 83.6 | \n| EgoThinker | 47.0 | 86.8 |   \n| GPT4o | 53.3| - |\n\nThe experimental results demonstrate a marginal performance decrease (0.6%) on VideoHallucer, while achieving a significant improvement of 3.2% on the POPE benchmark. We attribute this differential performance to our model's enhanced object grounding capability, which particularly enhances its robustness against object hallucination.\n\n### **[Q2] Experiments on another series model**\nWe finetune the more powerful video MLLM InternVideo2.5-8B. The table below shows our fine-tuned results across various datasets. From the table, we can see that our dataset shows minor improvements on EgoTaskQA and QAEgo4D. We believe this is because InternVideo2.5-8B inherently possesses strong video understanding capabilities. However, on EgoSchema and ERQA, which are more dependent on reasoning and are more complex, InternVideo2.5 achieved more significant improvements.\n\n| Model    | EgotaskQA | QAEgo4D | EgoSchema | ERQA |\n|---------|---------|---------|---------|---------|\n| QwenVL2-VL-7B | 57.9 | 60.3 | 63.3 |37.0 |\n| QwenVL2-VL-7B+ours | 64.4 | 66.2 | 67.6 |41.8|\n| Internvideo2.5 | 70.3 | 67.0 | 63.9 | 35.6 |\n| Internvideo2.5+ours | 72.4 | 68.3 | 66.4 | 40.0 |\n\nDue to resource and time limitations, we haven't yet completed experiments with larger models, such as the 13B-level model. We will supplement these experiments in the future and further analyze the role of reasoning data.\n\n### **[Q4] Grounding Data with Egocentric Video Understanding** \nDue to word limit, please refer to the **Relationship Between Grounding Data and Egocentric Video Understanding** section in our rebuttal for Reviewer k3DC(last).\n\n[1] Gemini robotics: Bringing ai into the physical world.\n\n[2] Videgothink: Assessing egocentric video understanding capabilities for embodied ai.\n\n[3] Internvid: A large-scale video-text dataset for multimodal understanding and generation.\n\n[4] Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.\n\n[5] Evaluating object hallucination in large vision-language models."}, {"review_id": "UmWKaVcdGp", "reviewer": "Reviewer_HXga", "summary": "This paper introduces EgoThinker, a new framework designed to equip multimodal large language models (MLLMs) with strong egocentric video reasoning capabilities. Unlike traditional models that struggle with first-person understanding, EgoThinker combines spatio-temporal chain-of-thought (CoT) supervision and a two-stage learning curriculum. The authors first create EgoRe-5M, a large-scale dataset containing 13 million egocentric video clips with detailed QA annotations, CoT rationales, and hand–object grounding. EgoThinker is trained through supervised fine-tuning on EgoRe-5M and further refined with reinforcement learning to improve spatio-temporal localization. Experiments demonstrate that EgoThinker significantly outperforms prior approaches on egocentric benchmarks, especially in fine-grained reasoning and localization tasks.", "questions": "My major concern is that ST-Think tackles a very similar problem with a comparable training recipe; could you explicitly delineate how EgoRe-5M, your curriculum, or your evaluation protocol differs from ST-Think, and add the appropriate citations so that readers can clearly see the unique contributions of EgoThinker?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. Rich, purpose-built dataset. EgoRe-5M offers 5 million QA pairs drawn from 13 million egocentric clips, with chain-of-thought rationales plus dense hand–object and temporal grounding, giving the community the first truly large-scale resource that couples reasoning and fine-grained localization. \n\n2. Effective two-stage learning pipeline. The combination of supervised fine-tuning on EgoRe-5M and GRPO-based reinforcement fine-tuning tightly links high-level causal reasoning with low-level spatial–temporal grounding, as confirmed by ablation studies. \n\n3. State-of-the-art egocentric performance. EgoThinker sets new highs on six egocentric QA benchmarks and more than doubles mIoU on EK-Visor relative to strong baselines, proving its practical impact on both reasoning and localization tasks. \n\n4. Broad generalization. Despite its specialization, the model maintains or even slightly improves scores on standard video-understanding suites such as MVBench and Perception Test, indicating the method scales beyond egocentric data\n\nCons:\n\n1. Novelty overlap. The core idea of pairing an egocentric QA dataset with chain-of-thought supervision and GRPO fine-tuning closely parallels the contemporaneous “ST-Think” work [1], which also introduces an ego-centric benchmark and an RL-enhanced MLLM. \n\n[1]. Wu, P., Liu, Y., Liu, M. and Shen, J., 2025. St-think: How multimodal large language models reason about 4d worlds from ego-centric videos. arXiv preprint arXiv:2503.12542.", "was_revised": true, "final_justification": "This paper makes a compelling contribution with EgoRe-5M, a large-scale and richly annotated dataset that fills a critical gap in egocentric video reasoning. The proposed two-stage training pipeline effectively combines supervised and reinforcement learning to align causal reasoning with spatial-temporal grounding. The model achieves state-of-the-art performance on multiple egocentric QA benchmarks while maintaining generalization across standard video tasks. Although there is some overlap with concurrent work, ST-Think, the authors highlighted the unique contributions of their dataset and compared the methodological differences between the two approaches in the rebuttal. \n\nGiven the significance of the dataset and the demonstrated effectiveness of the approach, I recommend this paper for acceptance.", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for your positive feedback to our work.\n\nOur work is differentiated from ST-Think by two fundamental aspects: our data and our methodology.\n\n1.Data field. Our work introduces EgoRe-5M, a large-scale, richly annotated dataset created specifically to train models for egocentric reasoning. In contrast, ST-Think's principal contribution is a benchmark designed to evaluate the reasoning capabilities of existing models.\nOur efforts to build this training dataset include two novel contributions. First, we developed a novel data sourcing pipeline to automatically filter and curate egocentric video clips from large-scale web data. This directly addresses the critical scarcity of egocentric video sources for training. Second, we have several QA annotation types with a comprehensive annotation scheme to specifically enhance understanding, grounding, and reasoning abilities within egocentric contexts.\n\n2.Method.\nWhile both studies use a two-stage training framework, our implementations and objectives at each stage are substantially different. SFT Stage: We strategically integrate our specialized EgoRe-5M data with general-purpose data. This approach is explicitly designed to enhance the model's expert egocentric reasoning skills while simultaneously preserving its broad, general-purpose capabilities. RFT Stage: Our reward functions target different objectives. ST-Think insightfully uses AI-generated scores to refine the model's general reasoning process. In contrast, our RFT stage is tailored to a core challenge in egocentric vision: understanding hand-object interactions. We formulate a reward signal that leverages limited grounding data to specifically sharpen the model's comprehension of these critical interactions, a focus essential for egocentric scenarios.\n\nIn summary, while ST-Think provides a valuable benchmark for evaluation, our work introduces the data, resources, and a tailored methodology for training robust egocentric models. We agree that ST-Think offers several insightful ideas, and we will add a detailed discussion and citation in our revised manuscript to properly contextualize our contributions. Thank you again for your constructive feedback. We trust this clarifies the unique aspects of our work and addresses your concerns.\n\nWe appreciate the reviewer pointing us to ST-Think, and we would like to respectfully note that this is a concurrent work with our submission, as it was released publicly as pre-prints very recently. Therefore, while it tackles a very similar problem in the field, our work was developed independently and made independent contributions in both data and method for egocentric reasoning.\nIf you have any additional questions or suggestions, please feel free to tell us."}, {"review_id": "RGvItlYuAm", "reviewer": "Reviewer_Fem7", "summary": "This paper introduces EgoThinker, a framework designed to enhance egocentric reasoning in MLLMs. Its core contributions include:\n- EgoRe-5M：A large-scale egocentric QA dataset;\n- Two-stage training：SFT on EgoRe-5M for foundational understanding and reasoning, followed by RFT using GRPO to improve spatio-temporal grounding.\n\nThe empirical results demonstrate that EgoThinker achieves state-of-the-art performance across multiple egocentric video benchmarks", "questions": "Is it possible to leverage RFT to improve performance on tasks other than Spatio-Temporal Grounding? Specifically, could this method be adapted or extended to enhance long-horizon planning, semantic inference, or goal-oriented question answering?", "limitations": "Despite mentioning manual screening of the data, there remains a lack of rigorous quality assurance measures during the dataset construction phase. This could potentially impact the reliability and generalizability of the results obtained from EgoThinker. Further details on quality control mechanisms would strengthen the robustness of the study.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\nThe authors built a large dataset (EgoRe-5M) for egocentric reasoning in MLLMs and validated a two-stage training method (SFT+RFT), showing significant improvement in Hand-object Grounding. Extensive comparative experiments against leading models demonstrate EgoThinker's effectiveness.\n\n### Weakness\nWhile EgoRe-5M and the two-stage training approach are notable, they suffer from a lack of novelty, as they represent incremental rather than groundbreaking innovations in dataset creation and fine-tuning strategies. Similar efforts in data creation:\n\nJi, Yuheng, et al. \"Robobrain: A unified brain model for robotic manipulation from abstract to concrete.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nCheng, Sijie, et al. \"Videgothink: Assessing egocentric video understanding capabilities for embodied ai.\" arXiv preprint arXiv:2410.11623 (2024).", "was_revised": true, "final_justification": "After considering the authors’ rebuttal and clarifications, I find that several concerns, especially around data quality assurance, were satisfactorily addressed. The responses on dataset novelty provided a reasonable distinction from prior work, though the incremental nature of the contributions remains. The discussion on adapting RFT to other tasks was informative but remains speculative without empirical evidence. Overall, the paper is technically solid, presents a substantial dataset and training framework, and demonstrates strong empirical results, but the originality is moderate. Given the balance of strengths over remaining limitations, I maintain the borderline accept rating of 4.", "rebuttal": "We thank the reviewer for insightful questions and feedback. We believe that we can address your concerns in the rebuttal.\n\n### **[Weaknesses] Contributions of Our Dataset.**\n\nWhile our dataset, EgoRe-5M, shares some similarities with existing datasets, our core contributions are distinct and address key limitations in the field.\n\nFirst, we address the challenge of limited video sources in the egocentric domain. For instance, RoboBrain's data is exclusively sourced from the Open-X-Embodiment[1] dataset, while VidEgoThink's video content is derived entirely from the GoalStep[2] subset of Ego4D. This limited video sources limit the model's generalization capabilities. In contrast, our work introduces a pipeline to curate data from large-scale web data, significantly broadening the scope of training data.\n\nSecond, our dataset encompasses a broader spectrum of question types and achieves data scaling. While RoboBrain focuses on action planning and VidEgoThink includes four dimensions of questions, EgoRe-5M further incorporates complex reasoning and temporal grounding tasks. Furthermore, we have achieved a significant scale-up in data volume (5k in VidEgoThink vs 5M in our case), which is crucial for training more powerful egocentric understanding models.\n\n### **[Questions] Adapting RFT for Other Tasks.**\nWe agree that adapting RFT to other tasks is a feasible direction. The critical element lies in the design of the reward function, which must be tailored to the specific task objectives. For example, for tasks such as long-horizon planning or semantic inference, we can employ a large language model (e.g., GPT-4o) to score the generated outputs with the ground truth, using these scores as the reward signal. For goal-oriented QA, the task could be framed as a classification problem to supervise the model's selection of the correct target. These are relatively intuitive approaches. We believe that the design of more sophisticated reward functions, as well as the exploration of joint multi-task RFT, are valuable and promising avenues for future research.\n\n### **[Limitations] Data Quality Assurance.**\nTo ensure the quality and reliability of our dataset, we conducted several steps to quantify:\n\n1.**CLIP-based Alignment Quality Analysis.** We analyze the alignment quality between the caption and the video via the CLIP score. Due to the large scale of the dataset, we sampled 100K instances to compute the score. Specifically, we chose ViCLIP[3] pre-trained on a large amount of video data, and compared it against the original concatenated captions as a baseline. We found that the score of our CoTs is 71.4, while the original caption score is 74.7. \n\nWhile the score of our data is lower than the descriptive caption baseline, this is an expected outcome. The source captions describe concrete, visible objects and actions, which aligns perfectly with CLIP's training objective. In contrast, our data contains reasoning steps, causal links, and inferred goals (e.g., “to align the dumbbell, a rotation is needed”), which do not always have a direct visual correlate.  A score of 71.4 still indicates strong semantic alignment, confirming that our annotations are highly relevant to the video's content.\n\n2.**Hallucination detection.** To validate whether EgoThinker exhibits hallucinations, we conducted evaluations on two widely-adopted benchmarks: (1) VideoHallucer[4] for assessing large video-language models, which contains object-relation, temporal, semantic detail and extrinsic hallucination detection. (2) POPE (Polling-based Object Probing)[5] for object hallucination detection. \n\n| Model    | VideoHallucer | POPE | \n|---------|---------|---------|\n| Qwen2-VL | 47.6 | 83.6 | \n| EgoThinker | 47.0 | 86.8 |   \n| GPT4o | 53.3| - |\n\nThe experimental results demonstrate a marginal performance decrease (0.6%) on VideoHallucer, while achieving a significant improvement of 3.2% on the POPE benchmark. We attribute this differential performance to our model's enhanced object grounding capability, which particularly enhances its robustness against object hallucination.\n\n3.**Manual Review.** We conducted extensive manual verification on the most complex Chain-of-Thought (CoT) data split to validate our data quality. Based on the length of the generated CoT, we selected 5k samples from our CoT data split for manual review. The goal of this scoring was to assess whether the generated caption contained any non-existent objects or action descriptions when compared to the video. We set up a 4-point scale to categorize hallucination severity: Severe Hallucination, Moderate Hallucination, Partial Hallucination and No Hallucination. We get 4068, 577, 279, 76 for score = 3,2,1,0. We observe that hallucinations primarily manifest as the model outputting non-existent goals or actions based on the provided captions, with hardly any instances of non-existent objects. Furthermore, the majority of captions exhibit no hallucinations or only minor ones, indicating good alignment quality in our CoT generation.\n\nIf you have any additional questions or suggestions, please feel free to tell us.\n\n[1] Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration.\n\n[2] Toward hierarchical understanding of procedural activities.\n\n[3] Internvid: A large-scale video-text dataset for multimodal understanding and generation.\n\n[4] Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.\n\n[5] Evaluating object hallucination in large vision-language models."}, {"review_id": "NDay3JjpeV", "reviewer": "Reviewer_k3DC", "summary": "Current multimodal large language models (MLLMs) excel at reasoning about observable events but struggle with embodied egocentric understanding. To bridge this gap, this paper introduces EgoRe-5M, a dataset encompassing four complementary task dimensions. Leveraging this dataset, they first equip the model with fundamental egocentric video understanding and reasoning abilities via supervised fine-tuning (SFT) on short-term, long-term, and chain-of-thought (CoT) data. Subsequently, they apply reinforcement learning on fine-grained grounding tasks to further enhance the model’s spatiotemporal localization capabilities. Experimental results across multiple egocentric benchmarks demonstrate that the proposed approach significantly improves both basic egocentric video understanding and fine-grained spatiotemporal localization.", "questions": "1. The process by which 13 million egocentric video clips were converted into 5 million QA pairs is insufficiently detailed. A more transparent and systematic description is needed to clarify this transformation pipeline.\n2. The introduction of the proposed benchmarks, such as EK-Visor and EgoExoLearn, lacks sufficient detail. The paper should provide a clearer rationale for their necessity and include evidence that these benchmarks are unbiased, representative, and appropriate for evaluating spatiotemporal grounding.\n3. Table 5 does not adequately demonstrate the contribution of fine-grained grounding tasks to egocentric video understanding. This weakens the central claim made early in the paper, that successful reasoning depends on accurately localizing hands and manipulated objects. A more thorough analysis is needed to substantiate this connection.", "limitations": "For certain datasets, such as QaEgo4D and EgoExoLearn, the use of EgoTimeQA and the inclusion of fine-grained grounding data effectively place the evaluation in a fine-tuned setting. However, zero-shot performance would be more convincing for demonstrating egocentric video understanding capabilities. Therefore, I recommend conducting additional zero-shot experiments and providing a clearer explanation of the current evaluation protocol.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. The paper includes comprehensive ablation studies that effectively demonstrate the contributions of different data components and the effectiveness of the two-stage training strategy.\n2. This work involves extensive data collection and processing, which requires substantial effort and resources.\n3. The first claimed contribution of this paper is the construction of the EgoRe-5M dataset. Its distinction from traditional egocentric video datasets such as MM-Ego [1] primarily lies in the inclusion of CoT data and grounding data. However, the CoT annotations are mainly generated by DeepSeek R1 without a specific design tailored to the characteristics of egocentric videos. On the other hand, Table 5 shows that the introduction of grounding data has no significant impact on egocentric video understanding, yet the paper does not clearly explain how these results relate to the connection between grounding data and egocentric video understanding. These suggest a lack of deep understanding of the fundamental differences between egocentric and exocentric perspectives.\n4. The second contribution is the application of a two-stage training regime combining SFT and reinforcement learning(RL) for model training. Similarly, this approach does not incorporate considerations specific to egocentric videos. Compared to UniVG-R1 [2], the main difference lies only in the task setting. Although the work is built upon egocentric video datasets and tasks, the methods are equally applicable to exocentric videos and thus do not make a distinctive contribution to advancing the field of egocentric video understanding.\n5. The two main contributions of this paper can essentially be summarized as the application of the CoT-SFT+RL paradigm to egocentric video tasks. Recent studies have introduced task-specific enhancements to RL methods, such as the Difficulty-Aware Weight Adjustment Strategy proposed in UniVG-R1 [2] and the AVA-GRPO approach presented in Vad-R1 [3]. In contrast, this paper only modifies the data and task settings, without providing deeper insights into how the RL paradigm could be adapted or improved to better address the unique challenges of egocentric video understanding.\n\n[1] Ye H, Zhang H, Daxberger E, et al. MMEgo: Towards building egocentric multimodal LLMs for video QA.\n[2] Bai S, Li M, Liu Y, et al. Univg-r1: Reasoning-guided universal visual grounding with reinforcement learning. \n[3] Huang C, Wang B, Wen J, et al. Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought.", "was_revised": true, "final_justification": "The rebuttal has satisfactorily addressed most of my concerns. I would like to revise my score to BA.", "rebuttal": "We thank the reviewer for insightful questions and feedback. Due to the word limit, we will simplify the description in the rebuttal.\n\n### **Details of the QA Generation Process**\n\n1.We first use a Dynamic Interaction Filtering Process. Due to the word limit, please refer to Supplementary A for details.\n\n2.After filtering, we construct the QA data. \nIn Section 3.1.2, we provide a detailed description of the formatting and procedures for different data splits. We believe your primary concern is that the selection of clips is not clear, particularly for long-term questions. We add additional details below.\n\nOur source long videos contain hundreds of clips. For generating long-term and CoT QA pairs, we employ the following protocol:\n\n(1) We sequentially scan each video's clips, randomly selecting 30-120s segments (30% for 30-60s, 70% for 60-120s).\n\n(2) We concatenate all clips within each selected window, discarding segments with <5 clips (due to insufficient captions).\n\nAfter compiling long-term QA clips, we remove duplicates from the short-term subset to create our final clip collection. We will add the complete detail to the revision.\n\n### **Details of Our Benchmarks**\n\n1. EK-Visor: \nThis dataset provides hand-object segmentation annotations, ensuring the quality of bounding boxes. It covers over 200 representative scenes with diverse hand-object interactions, and it is the most widely used dataset for egocentric spatial grounding. It addresses a critical gap in standard benchmarks (e.g., RefCOCO for images, NExT-GQA[1] for videos), which lack egocentric interaction data.\n\n2. EgoExoLearn: There are very few benchmarks for egocentric temporal grounding. Current benchmarks, such as Ego4D-NLQ[2], require locating a temporal window from a natural language query. However, these queries often suffer from poor quality and inconsistent lengths. EgoExoLearn contains the largest number of high-quality temporal boundary annotations. Its annotations are generated by trained human annotators, ensuring the data is high-quality, unbiased, and representative. Furthermore, the dataset provides timestamped annotations at coarse and fine-grained levels, making it ideally suited for building a temporal grounding benchmark to evaluate a model's ability to localize egocentric events.\n\n### **CoT annotations has no specific design tailored to the characteristics of egocentric videos.**\nFor the selection of clips for our CoTs, we utilized captions from the Ego-HOD[3] dataset. These captions augment high-level semantic information with the hand and object interactions and movements. This ensures that our source information is inherently rich with unique egocentric cues. \n\nBuilding on this, our CoT generation process is context-aware. It is guided by temporally localized questions grounded in the egocentric clips, ensuring that the reasoning traces are linked to egocentric cues such as hand-object interaction, temporal progression, and egocentric intent. As a result, our CoT annotations contain information for both fine-grained and multi-horizon reasoning, which is crucial for egocentric videos.\n\n### **Relationship Between Grounding Data and Egocentric Video Understanding**\nWe thank the reviewer for the suggestion and conducted further analysis on these benchmarks. We found that benchmarks such as EgoTaskQA, QAEgo4D, EgoSchema, and EgoPlan focus on high-level understanding, planning, and reasoning. They only require the identification of simple objects and do not involve fine-grained interaction details. Our baseline model, Qwen2-VL, already possesses a certain level of capability, which explains why our grounding data did not yield significant improvements.\n\nPrevious works[3,4,5] demonstrated that using hand and object as supervisory signals enhances egocentric video understanding abilities. Therefore, we selected three more fine-grained benchmarks, ERQA, EgoMCQ and EGTEA for evaluation.\n\n| Model    | ERQA | EgoMCQ | EGTEA | \n|---------|---------|---------|---------|\n| Qwen2-VL | 37.0 | 86.4/34.1 | 32.4 |\n| EgoThinker(without grounding data) | 40.1 | 87.6/38.3 | 33.1 |\n| EgoThinker | 41.8 | 89.3/41.4 | 35.4 |  \n| Gemini1.5 | 41.8 | - | - | \n\nAs shown in the table, the inclusion of grounding data significantly improved model performance across all three benchmarks. These datasets involve tasks directly related to our hand-object grounding data, such as understanding hand/robotic arm motion, object identification, and object classification.\n\nTo provide a more intuitive demonstration of the grounding data's effectiveness, we present inference examples below:\n\n(1) Q:How do you need to rotate the dumbbell for it to fit back in the weight holder? \n\nA:'To determine how to rotate the dumbbell for it to fit back in the weight holder, we first need to understand the orientation of the dumbbell and the weight holder. **Grounding of the Key Object**: The dumbbell is the key object here. It is shown lying on its side with the handle pointing upwards. The weight holder is positioned above the dumbbell, with its base facing downwards.. **Current Position**: The dumbbell is currently lying horizontally on the floor, with its handle pointing upwards. \nThus, you need to rotate dumbbell counter-clockwise 90 degrees and move it above to fit back in the weight holder.\n\n(2) Q:The right hand holding a blue object is in motion. What is it nearest to? \n\nA:The gripper is nearest to the stove top burner. The key objects: left hand, right hand,the stove top burner, and the blue object. The right hand is described as holding the blue object, which is located near the stove top burner. Since **the right hand is moving towards the stove top burner**, it must be closest to that area. During the process, the left hand does not move. Therefore, the right hand is nearest to the stove top burner.\n\nThese examples demonstrate that accurate grounding of objects and hands and reasoning about their relationships is essential for deriving correct answers. As discussed in our paper, current MLLMs exhibit limited capability in distinguishing hands, underscoring the critical role of precise visual grounding in such tasks.\n\nIn summary, grounding data provides a more pronounced benefit in egocentric scenarios that require fine-grained understanding, such as object classification, action recognition, and embodied reasoning in robot manipulation.\n\n### **The Contribution of Our Training Paradigm**\nWe thank the reviewer for highlighting these recent works in RFT and prompting us to clarify our contributions to the training paradigm.\n\nOur primary goal with RFT was to enhance two crucial skills for egocentric reasoning without degrading the general reasoning abilities.\nUnlike previous works which mainly optimize for a single task, our framework simultaneously optimizes for spatial and temporal grounding. Our investigation into this multi-task challenge revealed a critical challenge: mixed training caused unstable updates and slow convergence, requiring over 3k steps.\n\nWe address this through proposing a simple yet effective **single-task batching** strategy, where each training batch contains data from only one grounding task, alternated throughout training. This approach stabilized the learning process, accelerating convergence to 1k steps and leading to improved final performance on both grounding benchmarks.\n\nThis insight offers practical value for multi-task RFT applications, and we will ensure it is included in the final version to provide a more complete description of our methodology. We agree that further exploration into multi-task RL optimization is a promising direction for future work.\n\nWe appreciate the reviewer pointing us to UniVG-R1 and Vad-R1. We would also like to respectfully note that these papers are concurrent work with our submission, as both were released publicly as pre-prints very recently. Therefore, while they represent exciting and parallel advancements in the field, our work was developed independently and addresses the distinct challenge of multi-task egocentric grounding.\n\n### [Limitations] Zero-shot evaluation\nWe thank the reviewer for this important feedback.\nFor EgoTimeQA, we identify that it may share identical information with QAEgo4D, indicating potential data contamination issues. To ensure evaluation fairness, we excluded EgoTimeQA and conducted retraining.\n\n| Model    | QAEgo4D | EgoTaskQA | EgoSchema | EgoMCQ |\n|---------|---------|---------|---------|---------|\n| Qwen2-VL | 60.3 | 57.9 | 63.3 | 86.4/34.1 |\n| EgoThinker | 66.2| 64.4 | 67.6 | 89.3/41.4 |  \n| EgoThinker w/o EgoTimeQA | 64.5 | 64.6 | 67.4 | 89.0/41.5\n\nAs the table shows, removing EgoTimeQA resulted in a 1.7% drop on QAEgo4D, confirming shared QA patterns. In zero-shot setting, our model still outperforms the baseline by a significant margin of 4.2%, demonstrating its robust reasoning capabilities. \n\nFor the evaluation protocol, we conduct zero-shot experiments in MCQ format for QA benchmarks. Regarding spatial-temporal grounding tasks, evaluation is performed after RFT on a small training set. \nDue to the lack of suitable grounding benchmarks in the egocentric domain, we validate our approach through grounding-relevant benchmarks including ERQA, QAEgo4D, and EgoPlan(Table 2 in the paper). Additionally, we provide results between SFT and RFT in temporal grounding to demonstrate the effectiveness of our RFT paradigm. \n\n| Model    | mIoU | R1@0.05 | \n|---------|---------|---------|\n| SFT | 14.3 | 42.8 | \n| RFT | 25.2 | 63.9 | \n\n[1] Xiao et al, Can i trust your answer? visually grounded video question answering.\n\n[2] Ramakrishnan et al, Naq: Leveraging narrations as queries to supervise episodic memory.\n\n[3] Pei et al, Modeling fine-grained hand-object dynamics for egocentric video representation learning.\n\n[4] Zhang et al, Helping hands: An object-aware ego-centric video recognition model.\n\n[5] Xu et al, Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions."}]

Author rebuttal:

Meta-review: The submission initially received mixed ratings, with two reviewers leaning toward rejection (vznn, k3DC) and two toward acceptance (HXga, Fem7). Reviewers appreciated the introduction of a useful dataset (EgoRe-5M), the two-stage training pipeline with comprehensive ablations, and the demonstrated state-of-the-art performance. Concerns were raised, however, about the impact of hallucinations in the CoT process and the degree of novelty given overlaps with existing work (e.g., ST-Think, Robobrain, UniVG-R1).

The authors provided an effective rebuttal. Reviewer k3DC subsequently raised their score to borderline accept, while the reviewers already leaning toward acceptance maintained their positions. Although vznn did not update their score or provide post-rebuttal feedback, the AC sees that the rebuttal addressed their concerns regarding hallucinations and the justification of CoT.

Overall, the AC agrees with the majority of reviewers in recommending acceptance. While the contribution is moderate, the paper introduces a valuable dataset and an effective two-stage approach that achieves strong performance. The authors are encouraged to incorporate the rebuttal into the final version, particularly the discussion on hallucinations and comparisons with related work.
Decision: Accept (poster)