OpenMMEgo: Enhancing Egocentric Understanding for LMMs with Open Weights and Data
Authors: Hao Luo
Zihao Yue
Wanpeng Zhang
Yicheng Feng
Sipeng Zheng
Deheng Ye
Zongqing Lu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal LLMs
Egocentric Video Understanding
Primary area: deep_learning

Abstract: Recent advances in large multimodal models have significantly advanced video comprehension, yet their performance remains limited in first-person scenarios. The interactive nature of egocentric videos is critical for applications like embodied intelligence, but introduces complex visual contexts that conventional models struggle to capture. To bridge this gap, we introduce OpenMMEgo with innovations across three dimensions: data, model, and training strategy. To provide rich spatiotemporal visual knowledge, we curate a large-scale, high-quality dataset named OME10M, comprising over 8.2M egocentric video QA pairs synthesized from Ego4D series. We also establish OMEBench, a comprehensive benchmark for rigorous egocentric understanding assessment. To alleviate the frequent viewpoint shifts inherent in egocentric videos, we implement semantic-aware visual token compression. Further, a curriculum learning strategy is complemented to foster stable learning across various data complexities. OpenMMEgo consistently improves the performance of LMMs on egocentric benchmarks without sacrificing general video understanding performance. Notably, Qwen2.5-VL tuned with OpenMMEgo substantially outperforms other models of the same size in egocentric video understanding. The data, weights and training code will be put at https://github.com/BeingBeyond/OpenMMEgo.

OpenMMEgo: Enhancing Egocentric Understanding for LMMs with Open Weights and Data
Hao Luo
Zihao Yue
Wanpeng Zhang!:
Yicheng Feng
Sipeng Zheng²
Deheng Ye*
Zongqing Lu 1.2
Peking University BeingBeyond
3 Remin University of China Tencent
Abstract
Recent advances in large multimodal models have significantly advanced video comprehension, yet their performance remains limited in first-person scenarios. The interactive nature egocentric videos is critical for applications like embod- ied intelligence, but introduces complex visual contexts that conventional models struggle to capture. To bridge this gap, we introduce OpenMMEgo with inno- vations across three dimensions: data, model, and training strategy. To provide rich spatiotemporal visual knowledge, we curate a large-scale, high-quality dataset named OME10M, comprising over 8.2M egocentric video QA pairs synthesized from Ego4D series. We also establish OMEBench, a comprehensive benchmark for rigorous egocentric understanding assessment. To alleviate the frequent view- point shifts inherent in egocentric videos, we implement semantic-aware visual token compression. Further, curriculum learning strategy is complemented foster stable learning across various data complexities. OpenMMHgo consistently improves the performance of LMMs on egocentric benchmarks without sacrific- ing general video understanding performance. Notably, Qwen2.5-VL. tuned with MPen substantially outperforms other models of the same size in ego- centric video understanding. The data, weights and training code will be put at http:/ttww....oooooooea/////ee/a
Introduction
Despite advance in Large Multimodal Models, showing promise in interpreting well-captured images and videos (Liu et al., 2023, 2024a; Bai al., 2023), their ability to understand first-person scenarios remains limited (Majumdar et al., 2024; Ye et al., 2025). Unlike third-person recordings, egocentric videos involve the camera wearer as an active participant, with head movements causing frequent camera rotations and viewpoint shifts, posing unique visuospatial challenges. Given their applications in areas like robotics and AR/VR (Yi et al., 2024), empowering LMMs to comprehend first-person videos is urgently needed.
Due to this reason, egocentric video understanding (Grauman al., 2024; Xu et al., 2025) has gained increasing attention in recent years. Nevertheless, existing efforts remain in their early stages, primarily focusing on specialized tasks like retrieval or grounding (Ye et al., 2025). Some recent works (Ye et al., 2025) convert data from large-scale egocentric datasets (Song et al., 2023) into grestion-answering formats for instruction tuning. Yet, these datasets are typically either inaccessible rely text-only labels of coarse- grained event descriptions. For instance, Lin et al. (2022a); Pramanick et al. (2023) exploit short-term video segments (merely one second), where follow-ups
*corresponding author <zongqing lu @ pku.edu.cns
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are also constrained by high-level visual-text pairwise supervision. This overlooks the essence of egocentric scenarios, which demands fine-grained observation to model human activities effectively.
To foster this area, we introduce OpenMMEgo a family of state-of-thh-arr (SOTA) ego-centric large multimodal models (EgoLMMs)), releasing model weights, training data, and learning details. We start from devising meticulous data curation framework, which advances prior counterparts in two key ways: multi-level visual supervision and ii) diverse task types. First, unlike existing action-centric datasets, we address the scarcity of fine-grained visual cues (c.g.. object attributes, scene context, interactions), generate instructional data with both text and RGB RGB-pixel prompts. Specifically, we use textual annotations to synthesize events or action-related data for high-level video semanties. In addition, we extract subtle details (e.g.. object motions) from raw frames offer dense spatial-tempol
details for low-level supervision. Second, We incorporate wide range of egocentric tasks - spanning spatiotemporal, descriptive, and deductive reasoning -to maximize model generalization. Based on this framework, we introduce OME1 a fine-tuning dataset comprising: 8.2M egocentric video QA pairs covering diverse scenarios and tasks, and over IM general instruction samples sourced from web videos. This in-box dataset is tailored to equip LMMs with robust egocentric video understanding. Additionally, we include OMEBench, a new challenging benchmark for evaluating egocentric models on complex perception tasks. OMEBench features 4K multiple-choice questions derived from 372 hold-out videos, enabling assessment of EgoLMMs.
We train OpenMMEgo using a standard pretraining paradigm, enhanced by an innovative design mo- tivated by key observation: first-person videos exhibit rich spatiotemporal dynamics due to frequent camera shifts. Ffficiently capturing these dynamics from dense visual cues is thus crucial. Prior work shows that visual compression not only reduces computational costs (Kim et al., 2022: Choudhury et al., 2024), but also improves the efficiency of dynamic feature encoding (Shen et al., 2024; Chen et al., 2024b; Cheng et al., 2024; Jin et al., 2024). Inspired by this, we propose Dual Semantic-aware Token Compression to derive compact egocentric representation, DuaSTC comprises two modules: i) ceutiall-eddudant Token Merging (STM), which aggregates frame-level tokens into higher-level semantic entities, enabling processing in a condensed semantic space; ii) Temporat-irrelevanl Token Pruning (TTP). which preserves only motion- saliont and semantically critical tokens in the fast frames for local motion understanding, drawing from SlowFast principles (Feichtenhhoer et al., 2019; Xu et al., 2024; Huang et al., 2024). TTP mitigates noises from egocentric camera shifts, sharpening focus on dynamic content.
While token compression enhances our model architecture, a key challenge remains: ensuring training efficacy given the diverse video durations, task types and difficulty levels in our dataset. To address this, we further devise a Dual Curriculum Learning Strategy (Bengio et al., 2009; Wang et al., 2021) with two complementary approaches: i) Offline Data Curriculum. We leverage pre-trained LMM as the reference to assess sample difficulty and categorize them into three tiers. Training progresses from easier to harder examples, allowing gradual adaption egocentric video complexities. úi) Online Data Dropout. During training, we dynamically filter samples based on forward loss, excluding the most challenging examples when they exceed the model's current learning capacity. This dual strategy prevents overwhelming the model in early training stages while maximizing learning from appropriate-difficlty samples, improving both efficiency and final performance.
Following this systematic exploration, we conduct extensive evaluations on SOTA video LMMs (Li et al., 2024a; Bai et al., 2025). Our results show that OpenMMIEgo consistently enhances egocentric comprehension while maintaining general video understanding capabilities. Notably, it elevates QWen2.5-VL to achieve new SOTA performance across multiple egocentric benchmarks among models of comparable scale.
2 Related Works
Large Multimodal Models. Built on Large Language Models (Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023) and advnaced vision encoders (Radford et al., 2021: Zhai al., 2023), LMMs (Fu et al., 2024c; Cheng et al., 2024; Wang et al.. 2025) have demonstrated remarkable capabilities in vasion-language tasks, driving extensive research. Recent progress focuses on model architecture design (Alayrac et al., 2022; Li et al., 2023, 2024b; Sun et al., 2023), vision-language alignment (Zhu et al., 2023; Liu et al., 2023), high-quality instruction data curation (Gu et al., 2024; Chen et al., 2024a; Wang et al., 2023), and evaluation benchmarks (Fu et al., 2024b; Liu et al., 2024b;

---

crotgooentric Datasets
Bebaviour Based Egocentric QAs
General Video QAs
Anno. N
q What can be inter red sbout the personts ourentt activay?
Sub Clip L, Sub Sub-Clip
the is the mike food
prepration, spenihaaly feaned creating haachut speead by meblin
Egucentic Visall Prompt
Multimodal LLM
sugar Aux Bariyy the pesson does some clearing actins to mammann bay worlspace
QA Pronpee
Vision Centric Egucentric QAt
Via, Anioo
Anno. M
Q what otjects in the envremment
does vicco
Finttro
mane, attricatss positio
are uned the curnent procesc?
lake sloc?
Tetaporal
dhungss perppctiis
the person meiting granu a make haee spe aad. sugar from the container
sugar
The place
moverent
the
right penoon
cooy Michon light
interation dite, actin Letals
Figure I: Illustration of OMEIOM data curation framework (Left) and examples (Right. The action- relevant knowledge and visual-relevant information are systematically processed with (Multimodal) LLMs synthesize Behavior-Bas Egocentric QAs and Vision-Centro Egocentric QAs. OMEIOM contains both synthesis QAs and web-sourced general video QAs.
Yue et al., b).4a,b). contrast to pird-persoon view videos, egocentric recordings face unique challenges like scarcity of open-source video data long visual contexts that exceed model limits. To avoid data scarcity, prior research used synthetic data to train vidco models (Li et al., 2024a), or adapted image models for video (Xu et al., 2024). Recently, some studies (Chen et al., 2024b; Jin et al., 2024) have explored to manage the extended contexts from overwhelming frame counts by frame sampling techniques (Shen et al., 2024; Ye et al., 2025; Song al., 2024), efficient frame representation (Li et al., 2024e), and visual token selection or merging algorithms (Shang et al., 2024; Shen et al., 2024; Zhang et al., 2025). In this paper, combine large-scale, diverse video dataset with novel strategies for efficient visual token reduction, enabling effective context compression without sacrificing performance on egocentric scenarios.
Egocentric Video Understanding. This has emerged as a vital research area due to its real-life application. Early studies like Epic-Kitchens (Damen et al., 2018), Ego4D (Grauman al., 2022), and Ego-Exo4D (Grauman et al., 2024) focus on proposing new datasets with an increasing scale. These datasets consist of laboratory or daily videos recorded by head-mounted cameras, which provide rich first-person collections through crowd-sourced recordings. For evaluation, question- answering benchmarks such EgoSchema (Mangalam al., 2023), EgoPlan (Chen al., 2023), EgoTaskVQQ (Jia et al., 2022), QaEgo4D (Bärmann and Waibel, 2022), and OpenEQA (Majumdar et al., 2024) have been developed alongside specialized models for retrieval and recognition, including EgoCLIP (Lin et al., 2022a), R-VLM (Xu et al., 2023), and Helping Hands (Zhang et al., 2023). While MM-Ego (Ye et al., 2025) represents initial work egocentric LMMs (Fg)).MMs), it remains inaccessible to the public. Our work advances the field through novel data curation, model design, and training recipes, all of which will be open-sourced support community research efforts.
3 OME10M OM Data Curation
To establish robust training foundation, we develop a data synthesis framework that delivers multi-level, multi-faceted knowledge specifically designed for egocentric scenarios. Our framework leverages two primary sources: the Ego4D (Grauman et al., 2022) and Ego-Exo4D (Grauman et al., 2024) datasets, which collectively provide 4,900 hours of egocentric videos capturing diverse daily activitics (c.g., cooking, cleaning, and repairing), along with brief human annotations describing the events. In addition, we incorporate Ego4D-GoalStep (Song al., 2023), which offers goal- oriented dense annotations for over 400 hours of video. By combining human annotations with visual information from these videos, we synthesize comprehensive dataset of 8.2M egocentric video QA pairs, categorized into two major dimensions: Behavior-based QAs and Vision-centric QAs.
Multi-Level Dataset Design. Our design addresses a critical gap: while most egocentric instruction tuning datasets rely heavily on high-level human annotations (e.g,, actions, goals, events), they overlook low-level visual grounding- contrasting with training samples for LMMs that focus on basic perceptual signals like object appearances and spatial configurations. This mismatch in supervision granularity may contribute to LMM's inconsistent performance across general and
3

---

egocentric video tasks. We posit that that robust egocentric understanding, especially of high- level behavioral semantics (e.g., human intent, procedural goals), requires anchoring in rich low- level visual cues. To bridge this gap, we construct a multi-level dataset combining high-level rehavior-based supervision with dense, low-level visual facts, which includes behavior- based QA pairs (human actions, procedural reasoning) and vision-centric QA pairs (granular visual facts: objects, surroundings, and motion patterns). As shown in Figure 1, this joint alternative creates a comprehensive knowledge base. We further augment diversity by incorporating open-source general video QAs to maintain the performance on general video understanding capabilities.
Visual Details Annotation. Generating detailed visual annotations for egocentric videos presents unique challenges due to current LMM limitation. While direct annotation using Gemini-1.5. Pro (Team et al., 2024) is possible, we implement a structured pipeline to enhance reliability and coherence. (1) Pre-annotation: We first extract core visual facts (objects, environments, fine- fine-grained motions, perspective shifts) from each clip instead of directly generating QA pairs. (2) Segmented Processing: To reduce cognitive difficulty, videos are divided into 10-second subclips, each of which is independently annotated with frames sampled at IFPS, (3) Multi-step Prompting: For each subclip, we design egocentric-specifie prompts that guide the LMM to output in a structured format: entities, temporal differences, scene setting, and interaction description. Notably, the prompt not only specifies the expected output format but also serves as a form of egocentric task decomposition, explicitly informs the LMM how each component should be reasoned about and linked: i) tracking identified entities to perceive temporal changes; ii) distinguishing static background from interactive elements to infer perspective shifts and camera motion; and iii) aggregating temporal differences to describe scene layout, body movement, and manipulation details. The reasoning path is directly embedded in the prompt, making annotation form of structured visual thought, where each step builds upon the previous one. This design improves the reliability and coherence of the annotations. Annotation Examples and the prompt are provided in Appendix B.I.
Egocentric QAs Synthesis. With both human and visual annotations in place, we synthesize large- scale QA pairs that support multi-level supervision. Videos are segmented into clips of various lengths (10s, 30s, Imin, 3min, 5min) to cover a spectrum of short- and long-term temporal reasoning. For behavior- based QA, use human annotations within each video segment LLM's input, along with prompts covering multiple questioning dimensions, including atomic actions, complex behaviors, and goal-step inference. A total of 3.5M behavior-basee QA pairs synthesized, including 0.9M multiple-choice questions, 0.4M detailed video descriptions, and 2.2M open-ended QA pairs. Similarly, for vision-centric QA, we utilize both human and visual annotations within each segment to guide the synthesis process. Prompts are designed to target diverse visual reasoning perspectives, including spatial-temporal reasoning. self-motion analysis, object interaction, and vision- informed behavioral inference. A total of 4.7M vision-centric QA pairs are synthesized, comprising 1.IM multiple-che choice questions, 0.5M detailed video descriptions, and 3.1M open- QA pairs. In addition to these synthesized data, we incorporate 1M general video QA pairs from existing large-scale sources such as LLaVA. Video-I 178k (Li et al., 2024a) and ShareGPT4Vides (Chen et al.. 2024a), resulting in final dataset of 9. M QA pairs, which we name OME1OM. To ensure quality, we perform post-hoe filtering based on the predictive loss from an open-source LMM, removing QA pairs with excessively high loss values. Further details and QA examples are provided in Appendix B.
OMEBench. In addition training data, we also construct new benchmark with 372 hold-out videos. Following similar way of the training data construction, we synthesize 4,000 multiple- choice QA data, comprising a behavior-based subset and vision-centric subset, for OMEBench. This benchmark, concentrating on both event and visual details, serves as a supplement to existing egocentric benchmarks and can offer useful feedback for model development.
4 OpenMMEgo
To enhance EgoLMMs' egocentric comprehension using the rich knowledge provided in OME1OM, we target two critical dimensions: model architecture and training strategy. First, we introduce dual semantic-aware token compression to handle complex egocentric contents, reducing computational costs while preserving essential visual information (Section 4.1). Second, we employ a dual curricu- lum learning strategy to optimize training stability and effectiveness (Section 4.2). Together, these innovations pave the way for more robust performance in first-person scenarios.

---

Spatial-redundant Token Merging (STM)
@ Beuzzibe the event the vidzo dip previded?
Vision Tower
Youaa tikees
Stup Calculaic simileilis adjarent pairs
Retain
Step
Step
top 4% of roizn paiss
STM
Tokenizer
TTP
X Temporal- irrelevant Token Pruning (TTP)
compressse visual
LLM
holds handled repeatudly dried
Step 1: Split the frames into Slaw Frames and
Step 2: Retuin the tokens with siniilattics minking betwoas and 56 Find Frame, Prans the tokens the
Fast Promees, Find closee tokerz the Slow Frames for each token in the Led. Fint Framin..
ground renemblingg
the surrounding
Figure 2: Illustration of dual semantic-aware token compression. The token compression modules are plugged between the vision tower and the LLM. Details of and cemporal-irrelevnn Token Pruning (TTP) are shown in the right part.
Token Merging (STM)
4.1 Dual Semantic-aware Token Compression
Egocentric videos exhibit shift viewpoint changes that complicate visual perception - even stationary objects may appear differently across frames. While visual representations vary, the underlying semantics of objects remain consistent throughout the video. This observation reveals two key properties. The intra-frame semantic consistency indicates spatial redundancy which can be reduced to alleviate the comprehension burden. The inter-frame temporal consistency provides valuable hints against viewpoint changes. In this paper, leverage these properties through: i) Spatial-redundant Token Merging (STM) which aggregates tokens representing the identical visual elements into high- level semantic ones; ii) -ereeeatl Token Pruning (TTP) which eliminates semantically redundant tokens across frames for efficient temporal modeling. Figure 2 shows the illustration.
Spatial-redundant Token Merging (STM). To reduce redundancy in visual tokens, STM aggregates tokens representing identical visual elements. This not only reduces computational cost by decreasing token count, but also enhances temporal reasoning by grouping individual patch tokens into higher- level semantics to correlate these elements across frames. Unlike conventional token merging methods that aggregate the most similar pairs between the two after bi-partition (Bolya al., 2023; Lee et al., 2024; Jin et al., 2024), we exploit the localized nature of visual representations where a specific element is typically represented by its adjacent tokens. Following this, we design STM, a simple yet effective approach to merge adjacent visual tokens into flexible patch islands with varying sizes and shapes. Specifically, STM comprises four steps. Edge Construction: Build set E containing edges between all pairwise visual tokens that are spatially adjacent. Each edge is weighted by cosine similarity ros( 1;)). where U; and "j denotes token features. ii) Edge Pruning: Retain only top-k% weighted edges to form E* each of which indicates high semantical similarities between adjacent tokens. iii) Component Detection: Identify connected components in E* via a union-fing find algorithm; iv) Token Fusion: Merge tokens within each component into combined semantic tokens.
Temporal--rreeevant Token Pruning (TTP). Viewpoint shifts in egocentric videos challenges MMs to maintain consistent visual understanding across frames. While recognizing cross-fram frame concept co-occurrence is crucial for viewpoint movement awareness, rapid viewpoint shifts introduce distinct visual changes even in adjacent frames, leading to noises that obscure temporal dynamics. To address this, we propose TTP simplify the visual context using spatially compressed semantic tokens. Inspired by SlowFast architectures (Feichtenhofer et al., 2019; Xu et al., 2024; Huang et al., 2024), we process frames in two pathways: i) Slow pathway which retains all tokens for spatial detail preservation; ii) Fast pathway which preserves only visual elements that co-occur in slow frames, demonstrating cross-frame invariance to capture temporal dynamics. As Figure 2 illustrates, we calculate cosine similarities between semantic tokens from each slow frame and subsequent fast frames. Tokens exhibiting low semantic similarity are considered view-shift noise, while those with excessive semantic similarity indicate temporal redundancy. TTP prunes both types of tokens, and
5

---

retains only semantic tokens within the [r,%, %6 similarity percentile range. These remaining tokens maintains crucial visual dynamics cues while eliminating redundant or noisy information.
The synergistic combination of STM and TTP effectively reduces both redundancy and noise in video representations. This dual approach yields two benefits: i) significantly lowering computational demands for long video processing, and ii) enabling the model to better focus on learning meaningful spatiotemporal relationships between semantic entities.
4.2 Dual Curriculum Learning Strategy
Unlike previous counterparts, training dataset OMEION features unique challenges due to its diverse video durations, viewpoint changes, task types and multiple difficulty levels. To address the complexity, we implement this training strategy based on Bengio et al. (2009) combining both offline and online approaches to ensure stable and effective model training.
Offline Data Curriculum. We first pre-assess the difficulty of each sample in OMEIOM, using pretrained LMM like Qwen2-VL. (Wang et al., 2024) as the reference model. Each sample's difficulty is quantized by its forward loss and categorized into three levels easy, medium, and hard, based on loss percentiles. We train Mgo through three stages with increasing difficulty, each of which involves different data recipes. Due to space limitation, more details can be seen in our Appendix A.2.
Online Data Dropout. To further enhance learning efficiency, we introduce this online strategy that dynamically filters challenging in-batch samples during training. For each forward pass, we compute per-sample losses and selectively exclude the hardest samples - those with the highest loss values- from the subsequent backward pass. Formally, given the average batch loss l and the loss 4 for the i-th sample, the dropout probability defined as p. = Clip (4 1), 0, 1]. By iteratively pruning overly difficult samples, this online strategy ensures the model prioritizes samples with higher learnability its current training state, fostering more efficient optimization.
5
Experiments
5.1 Experimental Setup
Implementated Details. To evaluate the effectiveness of OpenMMEgo, we apply it two state-of- the-art 7B video MILMs,, LLa Video (Zhang et al., 2024b) and Qwen2.5- VL (Bai et al., 2025), to train two variants of OpenMMEgo. For visual token compression, we set k = 35 for STM and ry = 35 and г = 95 for TTP. To enhance offline difficulty estimation dual curriculum learning, we leverage two additional models, LLaVA-OV-7B (Li al., 2024a) and Qwen2-VL-7B (Wang et al., 2024), and compute a combined loss from both to derive the final difficulty score for each training sample. The hyperparameter a for online in-batch data dropout is each video is processed as up to 192 frames (resized 384 x 384), with a maximum visual token context length of N 13, 440. Following the training framework of LLa LLaVA--ext 2, we train both variants for epoch with a global batch size of 128 across 128 NVIDIA A800 GPUs.
0.3. In our implementation,
Baselines. To provide comprehensive evaluation,
compare our models with series of state-of-
the-art open-sourced video MLLMs of similar seale. Most relevant is MM-EGO (Ye et al., 2025), an EgoLMM built upon LLa OVMA-Oneissii Vision (Li et al., 2024a). Although MM-EGO is not yet publicly available, we include its reported performance for comparison.
5.2
Main Results
We assess OpenMMEgo from two aspects: i) its effectiveness in enhancing egocentric video under- standing, and ii) its potential trade-off's in general video understanding capabilities after egocentric adaptation. To systematically evaluate these aspects, we conduct comprehensive benchmarking against SoTA and open-sourced video LMMs across representative question-answering tasks.
Egocentric Video Understanding. To thoroughly evaluate improvements on this aspect. we assess OpenMMEgo on five multiple-choice QA benchmarks focusing on egocentric scenarios. We use EgoSchema (Mangalam et al., 2023) for comprehensive evaluation, while the validation set

---

Table 1: Comparisons with state-of-the-art LMMs egocentric video understanding tasks. We evaluate accuracy (%) across multiple-choice QA benchmark, with the best results highlighted in bold. For egocentric-oriented LMMs, report the performance gap relative their initialized LMMs in parentheses, where positive improvement are underlined.
EgoSchema
EgoPlan
QAEgo4D EgoTask VQA OMEBench (bch) OMEBench (vis)
General Video MLLMs
Chat-UniVi (Jin et 2024)
45.1
22.1
36.2
35.4
25.1
23.7
VideoLLaMA2 (Cheng et ul., 2024)
51.7
28.9
47.4
44.0
31.2
34.3
VidcoChat2
2024c)
54.4
26.2
43.6
45.2
32.5
35.8
LLavA-OneVision (Li al., 2024a)
60.1
32.4
55.2
53.6
46.7
40.5
Qwwen-VVL. (Wang et al., 2024)
66.7
39.2
$5.4
54.1
48.1
45.3
LLaVA-Video (Zhang et al., 2024b)
57.3
37.5
52.2
5U.3
47.5
44,3
IntemV122.5 (Chen et al., 2024c)
51.5
32.1
51.4
44.3
40.2
Qwen2.5-VL. (Bai et al., 2025)
65.0
45.2
59.4
53.7
55.3
49.4
VideoChat-Flash (Li al., 2024d)
64.3
40.8
62.4
54.8
57.1
54.5
Egecentric-Oriented MLLMs MM-EGO (Ye et al., 2025)
89.0(+8.9)
OpenMMEEo (LLaVA-Video)
65.8 (+8.5)
46.7 (+9.2)
62.0 (+9.8)
55.4 (+5.1)
64.4 (+15.0)
59.3 (+15.7)
OpenMMEgo (Qwen2.5-VL)
69.3 (+4.3)
50.2 (+5.0)
65.6 (+6.2)
56.2 (+2.5)
65.7 (+10.4)
63.2 (+13.8)
of EgoPlan (Chen et al., 2023) and the closed test set of QAEgo4D (Di and Xie, 2024) specialize in assessing goal-step reasoning and episodic memory, respectively. To evaluate generalization across video sources beyond Ego4D, we evaluate on the indirect test set of EgoTask VQA (Jia al., 2022) using LEMMA (Jia et al., 2020) videos. Additionally, domain evaluations with two splits: behavior-based QAs (OMEBench(bah.)) and vision-centric QAs (OMEBench(yis.)) aligned with OMEIOM's data synthesis pipeline. We conduct zero-shot evaluations on all benchmarks except OMEBench. As shown in Table I, OpenMMEgo significantly outperforms its base models across all benchmarks, demonstrating the ability capture egocentric video knowledge from OME10M and generalize effectively. Notably, The Qwen2.5- VL variant of our model surpasses all other LMMs of comparable size.
OMEBench provides in-
General Video Understanding. We fur- ther investigate how introducing egocen- tric video knowledge affects general video understanding. To this end, evalu- ate MMen on the following bench- marks: Video-MME (Fu al., 2024a) (w/o subscripts). MVBench (Li et al., 2024c), and PerceptionTest (Paträucean et al., 2023), which measure models' ca- pabilities video perception, reason- ing, and related aspects. The results on these benchmarks are presented in Ta- ble 2. When compared with the base model before adaptation, OpenMMEgo maintains stable performance general benchmarks, with only minor fluctuations slight drops on MV HVBench for LLaVA- Video and on Video-MME for Qwen2.5-
Table 2: Comparisons with state-of-the-art LMMs on general video understanding tasks. We provide accuracy (%) and best results highlighted bold as well.
Video-MME MVBench Perception Test
General Video MLLMs
Chat-UniVii
40.6
Lad22
54.6
51.4
VideoChat2
39.5
60.4
47.3
LLaVA-One Vision Qwen2-VL
58.3
56.7
57.1
63.3
67.0
62.3
LLaVA- Viden
63.3
70.8
67.9
In2em
64.2
72.0
68.2
Qwen2.5-Vi VI-
65.1
69.6
70.5
VideoChat-Flas Flash
65.3
74,0
76.2
Egocentric MLLMs
MM-EGO
OpenMMEgo (Ldeeo-Video)
63.4 (+0.1)
70.6 (-0.2)
68.1 (40.2)
OpenMMMgg (Qwen2.5.VI.
65.0 (-0.1) 70.8 (+1.2)
71.2 (40.7)
VL, alongside positive gains elsewhere. We attribute this to well-designed training strategies based on dual curriculum learning. Notably, MM-EGO, initialized from Va-aVAA-Oeeisiii exhibits more pronounced performance decline on Video-MME compared to its base model. This contrast underscores the advantage of OpenMMEgo's plug-in adaptation, which minimizes interference with general video understanding. Moreover, SaanaMMMEoo-Ownn22.. on par with understanding without compromising - sometimes even improving general video comprehension.
achieves performance LMMs. In summary, our work effectively enhances egocentric video
Additional Baselines and Benchmarks. complement the main results, we provide additional experiments with new baselines and benchmarks in Appendix D.I. The new baselines include

---

Table 3: The results of ablation experiments on data components and curriculum learning of Open- MMEgo with SannuMMMgo-QWWn.55555 on 5 egocentric video benchmarks, and the average aceuracy (%) on 3 general video benchmarks (Video-MME, MVBench, and Perception Test) is reported under 'Gen. Ben.'. The performance gap of each ablation compared OpenMMFgo-QWwen2255
We report the accuracy (%) of multiple-choics QAs
is shown in parentheses.
EgoSchema
EgoPlan
QAEgoiD
EgoTisVQQ
OMEBenchea))
OMEBench
Gen. Ben.
Ablations on Data Components
w/o Rchavior-Based QAs
66.1 (-3.2)
48.1 (-2.1)
64,4 (-1.2)
54.8 (c144
(-7.()
59.0 (-4.2)
68.5 (-0.5)
Centric
66.5 (-2.8)
45.1 (:5.1)
(c.2)
(-0.)
59.4
52.1
68.3 (-07)
w/o General Video QAs
(-1.7
49.4 (-0.8)
65.0 (-0.6)
55.9 (.0.3)
63.6 (-2.1)
60.2 (-3.0)
67.9 (-1.1)
Ablations on Carricltun Learning
w/o Online Curiculuu
67.9
50.3 (+0.3)
63.6 (-2.0)
54.9 (-1.3)
63.7 (-2.0) (0.6
60.1 (-3.1) 59.0 (-4.2)
68.2 (-0.9)
Offline Curricllum
(-2.)
48.7 (-1.5)
(-08)
53.5 (-2.7)
(40.3)
Petformance Data Scale
RgaSeknee
(4)
01.0 11.4
path
ONEEcec Dskk
EpoSchenca
QAEgedD
EyolPtnr
EirrbakVQA
OMEikahhvvi))
Ope STU--TI
Ope 77P
Figure 3: The performance of Open Qwen2.5-Vi on benchmarks across training dataset scales (left) and token compression variants (right). the left chart, each colored line corresponds to the trend of one benchmark. In the right radar chart, report the performance of Open variants trained with 1.6M QA pairs sampled from the dataset due to the resource limit.
closed-source and synthetic-data models for fairer comparison egocentric tasks, while the new benchmarks cover datasets beyond Ego4D to assess broader generalization.
5.3
Ablation Study
We conduct comprehensive ablation experiments to investigate the impact of cach part in OpenM- MEgo on egocentric video understanding and general video understanding. The ablation experiments performed with OpenMMEgo- -ar.igoo-wwen.5--1 and the results are demonstrated in Table 3 and Figure 3. We evaluate the performance of each ablation on the same benchmarks as in Section 5.2.
Q1: Whether all of the components in OME10M are essential and effective? The results in Table 3 show that excluding any sub-dataset leads to a noticeable drop performance across the benchmarks, underscoring the significance of all three components in OME1OM. Notably, the removal of vision-centric egocentric QAs results in the most pronounced decline, nearly halving the improvement observed on the in-domain OMEBench and even yielding performance inferior to the base model Qwen2.5-Ve 5-VL on EgoPlan. Featured with more vision-centric facts, vision-centti egocentric QAs may act pivotal bridge, assisting models to aggregate fundamental visual elements into higher-order, behavior-oriented knowledge within the egocentric perspective.
Q2: Whether the LMMs benefit consistently from scaling up the training dataset ? Given that all the synthesized data originates from the Ego4D dataset family and constitutes a major portion of OME10M, we evaluate the performance of the models trained with different data scales. As shown in Figure 3, the model exhibits steady performance gains across all benchmarks as the training data scale increases. Notably, the model does not suffer from overfitting as the data scale grows, which reflects the effectiveness the data synthesis. Through multi-level synthesis framework, OpenMMFgo provides rich and diverse visual supervision to continuously empower the LMMs.

---

Frame
Frame 2
Frame 3
Frame
Prame 5
Prame
Frame 2
Frame
Frame
Prame 5
Raw
STM
TTP
Figure 4: Visualization of the token compression process two video clips with OpenMMEEg- LL-Ve-Vide.
Q3: Whether both of the curriculum learning strategies facilitate better data utilization? We exclude one curriculum strategy time and report the performance in Table 3. The results show that relying on a singular approach leads to a degradation overall performance, which implies that the two curriculum learning strategies function complementarily. Thus, the improvements achieved by OpenMMEgo in egocentric video understanding stem not merely from increased data volume but from the effective utilization of data diversity and richness.
Q4: Whether the visual token compression modules contribute to performance? We conduct ablation experiments the two visual token compression modules in OpenMMEgo. Since different compression strategies support varying maximum numbers of video frames, we evaluate each configuration under its maximum allowable frame length, meaning that the comparison is made under approximately equal total visual token budgets. As shown in Figure 3, introducing the STM module into the uncompressed model leads consistent performance gains across benchmarks. Further adding TTP on top of STM brings additional improvements. However, directly applying TTP without STM yields mixed results, with 3 of the benchmarks improving and the others declining. As TTP is designed based on semantic entity grouping, which may not hold in the absence of STM and thus introduces noise. These findings suggest that blindly increasing the number of input video frames does not necessarily improve performance. Instead, performance gains arise from selective attention critical visual information. In this regard, the combination of STM and TTP proves effective.
We also report additional analyses on model design in Appendix D.2. These include comparisons with alternative strategies for token compression and curriculum learning, well hyperparameter ablations of key components.
5.4 Token Compression Cost vs. Benefit
We further investigate the computational trade-off of our compression design. To ensure fairness, we normalize the inference time of Qwen2.5-VL without any compression of OpenMMEgo 1.0 and report relative cost for other variants. As shown in Table 4, applying STM or TTP alone moderately increases the maximum supported frames (from 64 to 106 or 91), with inference time rising to 1.31 and 1.19x respectively. When combining STM+TTP, OpenMMEgo processes up to 192 frames, which is nearly three times the base model, while the inference cost is only 1.47x. This demonstrates that our joint compression strategy yields a highly favorable cost- tradeoff, enabling longer egocentric sequences be handled within practical budgets.
Table 4: Computation overhead and maximum supported frames under different compression settings. Inference time is normalized to the base model without any compression (w/o STM+TTP), set as 1.0.
Variant
Max Frames
Normalized Inference Cost
Qwen2.5-VL
64
1.00
Qwen2.5-VI. W. TTP
106
1.31
Qwen2.5-VL. VL STM
91
1.19
Mpen (STM+TTP)
192
1.47

---

Qn: Where are the bugs and piece of clothing hanging?
Q2: What does the away empty plastic bag?"
first do after throwing
OpenMMEEo: The bags and clothing are hanging on the wall above the trash carss.
OpenMMEgo: The camera-wearcr first puts on the counter back into the drawer below.
Qs: Why does camera-wearer open the under counter?
to discard the OpenMMEgo: The camera wearer discards the spring onion with the left hand.
Qt Which hand does small piece of spring anion?
OpenMMEgo: To retrieve liiife for cutting the spring
onions
Figure 5:
QA examples of OpenIMMEgo--wenn22552 on two-minute egocentric video clip. The model can answer these questions regarding environmental understanding, temporal reasoning, goal- step reasoning, and action detail perception.
5.5
Case Study
In this subsection, pose some specific cases for analysis.
Token Compression Visualization. We visualize the token compression process in OpenMMEgo on two video clips with LLLLLLLLM---LLLL------- patches with similar semantics are merged together. the TTP, tokens related to local motion on fast frames are retained, such as the tokens related to the hand in the top case. Even in the bottom case, where there a significant perspective shift and even noise, TTP still retains some tokens that can be used for motion understanding, such as the items on the shelf and the blurred checkout counter by the door. This reflects that the semantic-aware visual token compression module OpenMMEgo effectively compresses the visual context across different scenes.
in Figure 4. In the STM, tokens corresponding
QA Examples of OpenMMEgo. For a better understanding of our model capabilities, we present As shown in Figure 5, we propose questions over two-minute egocentric video clip, which is sourced from Ego4D and is not included in the training data. The questions are designed to examine the model's abilities in environmental understanding, temporal reasoning, goal-step reasoning, and action detail perception. Answering these questions requires capturing basic visual information, perceiving actions from the video, and performing some reasoning. OpenMMFFgo-QWenn 1esoo--wen..5-VV VI. can answer these questions well, demonstrating its capabilities in egocentric video understanding.
examples of generated results from OpenMMEge-Q
6 Conclusion
In this paper, we present OpenMMEgo to enhance LMMs in egocentric video understanding from data, model, and training aspects. We construct large-scale dataset, OME1OM, containing 8.2M egocentric video QA pairs synthesized from Ego4D series. We also introduce benchmark, OMEBench, for comprehensive evaluation of EgoLMMs. To alleviate the frequent perspective shifts in egocentric videos, we implement semantic-aware visual token compression and employ a curriculum learning approach foster stable learning across various data complexities. Our experiments on 7B size LMMs demonstrate that Mpen significantly improves the performance LMMs on egocentric benchmarks without sacrificing general video understanding performance. Notably, Qwen-2.5- VL tuned with OpenMMEgo outperforms other models of the similar size in egocentric video understanding.
10

---

Acknowledgments
This work was supported by NSFC in part under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
References
Jean-Baption Alayrac, Jeff Donahue. Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Proc. of Neural Information Processing Systems, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint 2rXiv:2309. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Leonard Bärmann and Alex Waibel. Where did leave my eeeepisodd--mmmory. based question answering on egocentric videos. In Proc. of Computer Vision and Puttern Recognition, 2022.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proc. of International Conference on Machine Learning 2009.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmman. Token merging: Your vit but faster. In Proc. of International Conference on Learning Representations, 2023.
Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. Transactions Machine Learning Research, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners, In Proc. of Neural Information Processing Systems, 2020.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang, Fong Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Proc. of European Conference Computer Vision, 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang, An image worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Proc. of Eurppaa Conference on Computer Vision, 2024b.
Yi Chen, Yuying Ge. Yixiao Ge. Mingyu Ding, Bohao Li, Rui Wang. Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- bench: Benchmarking multimodal large language models for human-level planning. arXiv preprint arXiv:2312.06722, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.055: 052771, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.074:4 2024.
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niiuuma, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. In Proc. of Neural Information Processing Systems, pages 281 28117728149) 28149, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic- kitchens datasct. In Proc. of European Conference on Computer Vision, 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for spic-kitchens. 100. International Journal of Computer Vision, 130(1):33-$5 55,, 2022.
11

---

Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In Proc. of Computer Vision and Pattern Recognition, 2024.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for vidco recognition. In Proc. of International Conference on Computer Vision, 2019.
Zhanzhou Feng and Shiliang Zhang. Efficient vision transformer via token merger. IEEE Transactions on Image Processing, 32:4156-222 1264166,, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprchensive evaluation benchmark of modal Ilms in vidco analysis. arXiv preprint arXiv:2405.21075 2024a.
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin. Bo Li, Xinyu Fang. Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411 1.l.1.155 15296, 2024b.
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context I1m inference. arXiv preprint arXiv:2407.14057 2024c.
Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours egocentric video. In Proc. of Computer Vision and Pattern Recognition, 2022.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proc. of Computer Vision and Pattern Recognition, 2024.
Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang. Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mme mm: Scaling multimodal performance with large-scale and high-quality instruction data, arXiv preprint arXiv:2410. 185558 2024.
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed comporal-localizuin assistant. In Proc. of European Conference on Computer Vision, 2024.
Hsiang-Wee Wei Huang, Wenhao Chai, Kuang-Ming Chen, Cheng- Yen Yang. and Jenq-Neng Hwang. Tosa: Token merging with spatial awareness. arXiv preprint arXi::2506.20666 2025.
Baoxiong Jia, Yixin Chen, Siyuan Huang. Yixin Zhu, and Song-Chun Zhu. Lemma: A multiview dataset for learning multi-agent agent multi-view activities. In Proc. of European Conference on Computer Vision, 2020.
Baoxiong Jia, Ting Lei, Song-Chun Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in
egocentric videos. In Proc. of Neural Information Processing Systems, 2022.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representa- tion empowers large language models with image and vidco understanding. Proc. of Computer Vision and Pattern Recognition, pages -3710, 2024.
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Leammed token pruning for transformers. In Proc. of ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Dong Hoon Lee and Seunghoon Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. Proc. of Neural Information Processing Systems, 2024.
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long-form video understanding. In Proc. of Neural Information Processing Systems, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwci Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXt::2408.0333 2024a.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang. Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang. Bei Chen, and Junnan I.i. Aria: An open multimodal native mixture-of-experts model. arXiv preprint 24X0022410.05993, 2024b.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2. 2: Bootstrapping iaguuage-mmag pre-training with frozen image encoders and large language models. In Proc. of International Conference on Machine Learning, 2023.
12

---

Kunchang Li, Yali Wang, Yinan He. Yizhuo Li, Yi Wang, Yi Liu. Zun Wang, Jilan Xu. Guo Chen, Ping Luo, al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proc. Computer Vision and Puttern Recognition, 2024c.
Xinhao Li, Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat :adsochat-flas:: Hierarchical compression for long context video modeling. arXiv preprint arXiv:2501.000000 00574, 2024d.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In Proc. of European Conference on Computer Vision, 2024c.
Kevin Qinghong Lin, Jinpeng Wang. Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong- Cheng Tu. Wenzhe Zhao, Weijie Kong. et al. Egocentric video-language pretraining. Proc. of Neural Information Processing Systems, 2022a.
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Chene Tu, Wenzhe Zhao, Woijie Kong, et al. Egocentric video-language pretraining. In Proc. of Neural Information Processing Systems, 2022b.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proc. of Neural Information Processing Systems, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proc. of Computer Vision and Pattern Recognition, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang. Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Proc. of European Conference on Computer Vision, 2024b.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff. Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Mcier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In Proc. of Computer Vision and Pattern Recognition, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Proc. of Neural Information Processing Systems, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. of Neura Information Processing Systems, 2022.
Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Fmara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detaleed egocentric video dataset. In Proc. of Computer Vision and Pattern Recognition, 2025.
Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion the backbone. In Proc. of International Conference on Computer Vision, 2023.
Viorica Päträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Proc. of Neural Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of International Conference on Machine Learning, 2021.
Yuzhang Shang. Mu Cai. Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint ar XrXiv:2403. 153888 2024.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding, arXiv preprint 24X14:2410.17444, 2024.
13

---

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token sparse memory for long video understanding. In Proc. of Computer Vision and Pattern Recognition, 2024.
Yale Song,
Fugene Byrne Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. EgoAd goal- step: Toward hierarchical understanding of procedural activities, Proc. of Neural Information Processing Systems, 2023.
Alessandro Suglia, Claudio Greco, Katie Baker, Jose L Part, loannis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon. Alanavlm: A multimodal embodied ai foundation model for egocentric vidco understanding. arXiv preprint arXiv:2406. 13807, 2024.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang. Yueze Wang, Hongcheng Gao. Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint vrXiv:2307.05222 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302 13971, 2023.
Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zaxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. In Proc. of European Conference Computer Vision, 2022.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4y for better visual instruction tuning. arXiv preprint arX1:22311.07574, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception the world at any resolution. arXiv preprint 24ti::2409.11991 2024.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEFE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555-45) 4576, 2021.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfci Gao, et al. Internvido2..6 5: Empowering video milms with long and rich context modeling. arXiv preprint arXiv:2501. 12386, 2025.
Boshen Xu, Ziheng Wang, Yang Du, Zhinan Song, Sipeng Zheng, and Qin Jin. Do egocentric video-language models truly understand hand-object interactions? In Proc. of International Conference on Learning Representations, 2025.
Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-hased video language model for efficient long video question answering. arXiv preprint arXiv:2312.04931, 2023.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llawe A strong training- free baseline for vidco large language models. arXiv preprint arXiv:2407. 15841, 2024.
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Haoxuan You, Dan Xu, Zhe Gan. Jiasen Lu, Yinfei Yang. and Bowen Zhang. MMEgo: Towards building egocentric multimodal LLMs. In Proc. of International Conference on Learning Representations, 2025.
Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in ego-sensed world. arXiv preprint arXiv:2410.03665. 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proc. of Computer Vision and Pattern Recognition, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang. Kai Zhang, Shengbang Tong. Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pre pro: A more robust multi dasti-dissipline multimodal understanding benchmark. arXiv preprint arXiv:2409.0 02813,,..1...... 2024b.
14

---

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proc. of International Conference Computer Vision, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang. Hang Zhang. Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2301.33100 2025.
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recognition model. In Proc. of International Conference on Computer Vision, 2023.
Guozhen Zhang, Jingyu Liu,, Shengming Cao, Xiaotong Zhao, Kevin Zhao, Kai Ma, and Limin Wang. Dynamic and compressive adaptation transformers from images videos. arXiv preprint arXiv:2408, 06840, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 2410:2241..,7113, 2024b.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. Proc. of Computer Vision and Pattern Recognition, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprini arXiv:2304.1055 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize the paper's 's key contribu- tions across data, model, and training strategy, without overstating the scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
No theoretical results are involved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details are clearly included in the paper, and the work will be fully open- sourced after publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: In the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
onnolicy)))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccceeeeececceccccccce httpa//gggiiiee/eeeuuuuiiiiii
/CooeSuboiieeeeee cetommisssion wwwsssssiiiiioiieeeee seionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the experiments section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Too many resources are needed to run experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The main resources consumption included in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: /ttp///eurppp.e/eee//ee cccpppss......ccccccoo.oo..cccce httpp//lccccchhccccseeeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Not included in the work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wneerswwthhodeee thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [No]
Justification: Document will be provided after publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
21

---

Justification:
Used for article polishing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.M policy fattps://nnt http::///eeuuuuueeeeeeeeeee http:////////ooeeeeeeea///////// for what should or should not be described.
22

---

A Implementation Details
A.1 Training Details
We follow the training framework of LLaVA-Next 3, using learning rate of le-5 and a warmup ratio of 0.03 throughout the entire instruction tuning process. As for the based models, we use the -B----iddo------ee
version of
Instrucct respectively. The training process is conducted on 128 A100 GPUs with the hyperparameters listed in Table 5. With at most 5 QA pairs the same video clip integrated in one single data point, the whole training process takes 31 hours for epoch.
and Qwen2.5
Table 5: Training hyperparameters for instruction tuning.
Hyperparameter
Value
Global Batch Size
128
Frame Number
192
Input Resolution
384
Learning Rate
1e 1e-
Weight Decay
0
Warmup Ratio
0.03
Learning Rate Scheduler
cosine
Numerical Precision
6
Epochs
1
Max Sequence Length
32768
Max Visual Context Length
13440
A.2 Curriculum Details
In our offline curriculum learning, we quantify the difficulty of each data via the loss value on capable LMMs. To refine our dataset, we first exclude data points exhibiting excessively high loss values, with threshold set at 3.0 in our implementation. Similarly, for benchmark construction, we select the 4,000 QA pairs with the highest loss values, constrained to those below 3.0. According to the loss values, the filtered dataset is divided into three subsets in equal size, easy, medium, and hard. We then organise the datasets for three-stage training, with the overall difficulty increasing as the stages progress. The specific recipe for each stage is presented in Table 6.
Additionally, the distribution of difficulty scores across the three splits of OMEIOM is depicted in Figure 6. According to our difficulty metric, all three splits exhibit a long-tai distribution. Notably, the general video QAs generally exhibit lower difficulty compared to egocentric QAs. Within the egocentric data, the vision-centie egocentric QAs exhibit relatively lower difficulty, with a more even distribution across various difficulty levels, in contrast to the behavior- behaeeor-basee egocentric QAs. This distribution pattern suggests that Vision-Centrie Egocentric QAs are more amenable to learning, as they incorporate fundamental visual facts, thereby partially bridging the gap between Egocentric QA and General Video QA.
Table 6: Data partition for each training stage.
Easy
Medium
Hard
Stage-1
60%
30%
10%
Stage-2
35%
50%
15%
Stage-3
5%
20%
75%
http://hhuggiiiiirrrccoovvvveeev Instructtt
23

---

Eanettenti EpoestarQts
1.0
1.
20
Figure 6: Distribution of the difficulty scores for offline curriculum learning over the 3 splits of OME10M. Dashed lines indicate the tri-quantile points of the difficulty scores used to split the dataset into easy, medium, and hard subsets.
A.3 Egocentric Benchmarks
In addition to our OMEBench, we also conduct experiments on several open-source egocentric benchmarks for thorough evaluation in egocentric video understanding.
long-form egocentric video QA benchmark sourced from Ego4D, designed assess temporal reasoning over 3-minute clips, with focus on high-level activity understanding through multiple-choice questions.
EgoSchema.
EgoPlan. A goal-oriented benchmark that evaluates planning ability in egocentric contexts by predicting the next plausible action task sequence. We adopt the validation for evaluation.
QAEgo4D. A egocentric benchmark derived from Ego4D, targeting episodic memory in first-person videos. Models required to select concise, factual answers from fixed vocabulary. We use the closed-set version of QAFgo4D for evaluation.
Aa. A diagnostic benchmark based on the LEMMA dataset, designed to assess goal-directed reasoning through four types of questions: descriptive, predictive, explanatory, and counterfactual. We use the indirect test set to increase evaluation difficulty. To enable consistent evaluation, we convert the open-ended questions into a multiple-choice format by prompting Gemini-11 5-pro to generate 2-4 distractor options based on the correct answer.
B Dataset Details
B.1 Data Synthesis Framework
In this section, we describe the details of the data synthesis framework. First, we segment the videos into clips spanning the range of 0-5 minutes, specifically 10s, 30s, 1min, 3min, and 5min. These clips serve as the QA targets of OME1OM. We utilize the Gemini- 1....ppmm 5-pro to generate QA pairs for the clips in two manners, respectively producing behavior-based QAs and vision-centric QAs.
For behavior-based data synthesis, we provide all the human annotations associated with each video clip to the Gemini-1- 5-rrr for QA generation. To enhance the quality of the generated questions, we predefine several question types, which can be categorized into three main levels: (I) atomic actions, including action recognition, interactive object grounding, action temporal reasoning, and action counting: (2) chunked behaviors, where atomic actions are aggregated into descriptive behavior paragraphs; and (3) goal-step reasoning, encompassing goal analysis, next-step prediction, and precondition identification. The human annotations relevant to each video clip are iteratively provided to Gemini-i 5.5ppz 5-pro multiple times, prompting separate data synthesis from these distinct levels. Besides the predefined aspects, we also prompt Gemini-3 (attnii1.5-pppp. 1.5-pro to generate free-form QAs based on the information. The prompt of behavior-based data synthesis provided in Figure 7 and Figure 10.
24

---

For vision-centric data generation. we first segment the original videos with 10-second intervals, which are then densely annotated with ppatial-tmpoora information relevant to vision-centrio analysis. Specifically, each short clip is frame-extracted at fps, and both the extracted frames and correspond- ing human annotations are input into Gemini-1- -1.5--rro for vision-centric annotation. the system prompt, we include necessary step-by-step reasoning guidance, which covers the following:
1. Analyze the spatial location, shape, color, and size of all objects each frame.
2. Compare adjacent frames, identifying the changes in the background objects to annotate the perspective shift between frames. Infer motion information, such whether the camera wearer is moving or turning in a specific direction, based on perspective shifts.
3. Describe the scene in detail according to the spatial relationships among the objects.
4. Annotate the action descriptions to specify the interactive objects and their interaction details.
The prompt used for visual annotation synthesis is provided in Figure 8.
Based the four-dimension vision annotations at the granularity of 10-second segments, we generate viston-centric QA data in a similar manner. We also specify several question directions for generating vision-centric QAs: (1) spatial-temporal reasoning, focusing on object features, object relationships, and scene descriptions; (2) self-motion analysis, mainly questioning about the motion of the camera wearer reflected by perspective changes; (3) object interaction, querying the features and details of the interacting objects; and (4) vision-injected behavior, which incorporates the question types in behavior-based QAs but requires the model integrate annotated visual information in the synthesis QA. The vision-centric annotations, together with corresponding human annotations, are fed to Gemini-1 5-pro in the same manner as the behavior-based data synthesis. The prompt of vision-centric data synthesis is provided in Figure 9 and Figure 10.
B.2 OME10M and OMEBench Examples
We provide more examples of the synthesized QA pairs in OMEIOM. As shown in Figure 11 (Left), the generated QAs cover a wide range of topics in egocentric videos, including movement analysis, causal reasoning, environmental understanding, next-step prediction, detailed action description. Additionally, we also provide examples from OMEBenchtoggtteer with the corresponding answers from OpenMMEgo SaArMMEoo-QQe2.5--.. Qwen2.5-VL and Qwen2.5- VL. As shown in Figure 11 (Right), OpenMMEgo- Qwen2.5-VL VL is able to answer the questions correctly while Qwen2.5-VL. fails to do so. These exam- ples involve the capability of environmental understanding, action recognition, temporal grounding, and object motion perception. The outperformance of OpenMMFEgo- Seegg--Qee2.5-VV VL. over Qwen2.5- VL in these examples indicates the effectiveness of our method in enhancing the egocentric video understanding capability of MILMs.
C Additional Related Works
Visual Token Merging. Visual token merging has demonstrated significant effectiveness in improving representation efficiency while reducing computational costs, initially in caagecclassiiication tasks. Key techniques include similarity-based merging (Bolya et al., 2023), learned merge-ratio (Feng and Zhang, 2023), learned threshold-based merging and pruning (Bonnaerens and Dambre, 2023), decou- pled embedding modules (Lee and Hong, 2024), and depth-aware spatial token aggregation (Huang et al., 2025). These approaches have since been extended to video inputs, with innovations such as learnable spatiotemporal merging (Lee et al., 2024), spatial-temporal token selection (Wang et al., 2022), and temporal interpolation (Zhang et al., 2024a). Similarly, in large multimodal models (LMMs), token compression has been adopted to improve inference efficiency and handle long contexts, including methods based on static pixel token reduction (Zhang et al., 2025), cross-modal querying reduction (Shen et al., 2024), and hierarchical compression (Li al., 2024d). Building on these advances, we propose method specifically designed for egocentric videas, which present unique challenges due to dense visual cues and frequent camera motion. We introduce a dual-branch compression strategy that restricts spatial token merging to adjacent tokens for preserving semantic entity boundaries, and performs temporal pruning over entity-aligned tokens in SlowFast-inspired view to retain motion salient information.
25

---

Input & Task
You are tusked with generating high-quality. Question-Answer (QA) pairs Multiple-Choiie Questions (MCQs) to evaluate participants" understanding egocentric videos. Instead raw video input, you are provided human annotations atomic actions are also provided to you. Some marks used the narrations: "#C" for the camera wearer. '#not sure' for the uncertain information, '#summary' for the summary of the narration sequence, '#O' for other action conductons.
Your task is to generate question-ansse answer pairs that probe the participant's comprehension of visual, temporal, spatial, causal, motor, and interactional aspects of egocentric video, based on the human annotations.
Each question must have one correct answer and three plausible distractors. The distractors should be confusing but incorrect, grounded in elements that appear the video but do satisty the question, artificial elements that could reasonably be inferred but are wrong.
Guidelines:
1. Question Scope: You should generate questions about Equestion scope]. You should generate appropriate questions in
these angles or their composition.
2. QuQuuuuue-Asseeee Quality:
The generated questions should vury complexity and scope. Across the generated QA pairs, ensure coverage of diverse question angles within the assigned dimension. Avaid produciig repetitive patterns or overly similar question formulations. Some questions should target fine-grained recognition result short, require reasoning
The generated questions should vary in complexity and scope. Across the generated MCQs, ensure coverage diverse question angles within the assigned dimension. Avoid producing repetitive patterns or overly similar question formulations. Some questions should target fine-grained recognition and others should require reasoning across multiple behaviors. For distractors:
answers. Others should multiple behaviors, resulting longer, integrative answers. Both short-form and loag-form questions must be present in each generation batch. For the divensity of questions, only part of the information should be involved each QA pair. Though the answers may be long. the questions should short and concise hit the nail on the head.
event. Distractors must be plausible the
b) They should differ subtly from the correct answer.
egg, similar object, wrong time, incorrect goal. c) Avoid obviously false or ocopp distractors.
3. No Answer Leakage: The answer must not be directly inferable from the question text alone. Avoid phrasing that embeds hints, reuses descriplive terms from anmolations, or averly specifies entities or outcomes. The video must be watched to answer.
4. Answer Certainty and Grounding: Each question must have an unambiguous correct answer that fully inferable from the structured annotations, Do not include speculative or uncertain phrasing in the answers. Avoid temms like "maybe," "appears to." "possibly."
5. No Internal Identiflers or Serial Labels: Do not use labels such as "Person A," "Rope 1," or "Frame 3" in questions or answers. Instead, refer to entities using their deseribed attributes (e.g. "the thick red-and-blue rope," "the hand with pale skin and slender fingers"), " use the scene or the activity ground the time instend the specific frames or second numbers, that viewer can ground them visually. Also, the segment information not provided the participants, so do mention the segment index time in the question answer.
Output Format
Provide the output as JSON string, structured as Python List and each entry a Python Dictionary. Do not include a comma at the end of the last entry.
Example Ourput:
["Question" : "squestion 1>*, "Anmerr" "canswer I>", "Distracton"::
(" ('Question": "cquestionn 1>", 'Answer": *cansmer-->") I>*),
["sdistractor-1-i "odistracter- 1-2=*, "Cdisttactor 1-3="]]
("Question": "cquestion-N"" "Anwwer" "Ganswer-NN>")
("Question:: "squestion-N=1 "Answer": "canswer-No", "Distractors": ["ddistractooN.I>>> "adistretornNN "diittcaarn..33.)
Attention:
1. Output must be pure JSON without any markdown code blocks or other formatting.
2. The output must be valid JSON without any comments. No ellipsis (...) comments (6) about similar patterns.
3. no relevant QA pairs can be generated, simply retur the word 'None' in string format.
Figure 7: The prompt used for behavior-based data synthesis with Gemini-1- 1. 5-pro. The texts in the yellow block are used for open-ended QA synthesis, while the texts in blue blocks are used for Multiple-Choices Question synthesis. The question_sceee scope is replaced with conerete descriptions in Figure 10 in the implementation.
D Additional Experiments and Ablations
In this section, we complement the main results with additional experiments: (i) expanded baselines on egocentric tasks, (ii) new benchmarks beyond Ego4D, (iii) ablations on token compression and curriculum strategies, (iv) cost-benefit of token compression.
26

---

Table 7: Performance of expanded baselines egocentric benchmarks. Numbers the accuracy (%). The performance gains over Qwen2.5-VL VL are reported in parentheses.
Method
EgoSchema
EgoPlan
QAEgodID
EgoTaskVQA
OMEBench (beh.)
OMEBench (vis.)
Gemini-2.0-Fash
71.1
51.5
62.4
59,2
60.2
62.1
AlanVLM
50.7
27.6
41.8
43.2
27.3
33.2
Qwen2.5 VI.
65.0
45.2
59.4
53.7
55.3
49.4
VideoChat-Flash
40.8
54.8
57.1
54.5
OpenMMEgo (Qwen2.5-VL)
69.3 (44.3)
50.2 (-5.0)
65.6 (46.2)
56.2 (42.5)
65.7 (+104)
63.2 (413.8)
Table 8: Results on new benchmarks. Numbers are accuracy (%) except that EP-100 MIR is mAP
(%).
Method
HD-EPIC
OpenEQQ
EP-100 MIR
EgoVLP
26.0
LaViLa-L
40.0
Gemini-2.0-Fiaar
38.8
61.5
49.3
AlanVI.M
25.7
46.7
23.1
Qwen2.5-VT.
34.2
55.7
42.1
VideoChat-Flash
39.4
58.1
45.3
OpenMMEEg (Qwen2.5-VL)
42.4 (+8.2)
60.2 (+4.5)
49.7 (47.6)
D.1 Expanded Baselines and Benchmarks
Additional Baselines. We introduce two additional baselines to better situate OpenMMEgo among open and closed models. First, since the Gemini-1.5-Pro API was no longer available at evaluation time and its scale is much larger than our 7B backbone, we instead report results of Gemini-2.0-Fi F.aFaahh.. backbone, and it is reasonable to assume that its training data and
This model is closer in size process are highly similar to Gemini- 1.5-i.r. making it a suitable closed-source reference. Second, include ALM (Suglia et al., 2024), which is trained on synthetic dataset, supplementary baseline for using synthetic egocentric data. Results are shown in Table 7, where Mpen consistently improves upon all open baselines, narrowing the gap with Gemini despite being trained with fewer resources.
Additional Benchmarks. Beyond Ego4D-based suites, we further evaluate OpenMMEgo on several newly added benchmarks to examine its generalization ability. Specifically, we include results on the HD-EPIC VQA (Perrett et al., 2025) benchmark, which emphasizes fine-grained egocentric video question answering, and the open-ended OpenEQA (Majumdar et al., 2024), which tests broader video-language reasoning in unconstrained settings. In addition,
adapt the EPIC-Kitchens- 100 (Damen et al., 2022) multi-instane retrieval (V - T) task into a multiple-choice format, enabling a fairer comparison with video-language models; performance is reported in terms of mAP. For this retrieval setting, we also report results from Ego VIP (Lin et al., 2022b) and La (Zhao al., 2023) strong references. Together, these benchmarks complement Ego4D tasks by providing both fine-grained and open-domain evaluation scenarios, thereby offering a more comprehensive assessment of egocentric video understanding. Table shows that Open achieves consistent improvements over base LMMs, e.g. +8.2% on HD-EPIC and +7.6% mAP on EP-100, indicating robustness beyond Ego4D.
D.2 Strategy Ablations
We conduct the additional ablation experiments the hyperparameter a, k.r1, and Th to investigate their influences on OpenMMEgo. We also include the comparison with alternatives to our design in token compression and curriculum learning. Given the limited computation resources, the Open- MMEgo variants in this section are all trained with 1.6M pieces of data sampled from the overall OME10M.
Online Dropout Ablation. The hyperparameter a governs the proportion of units dropped out in each batch during online learning. This parameter is intended to encourage the model to focus on the more readily learnable aspects of the current stage during fine-tuning. As the results in the Table indicate, performance suffers when is either too small or too large. However, values around 0.3
27

---

Table 9: The results of additional ablation experiments the hyperparameters a, k,y, and yд of Open with OpenMMMggo-Qwen2... VL. We report the accuracy (%) of multiple-choice QAs on egocentric video benchmarks, and the average accuracy (%) 3 general video benchmarks (Video-MME, MVBench, and Perception is reported under 'Gen. Ben.'.
EgoSchema
BgoPlan
QAFgo4I)
EgoTaskVQA
OMFBenchhesh)
OMHBench (ve.)
Gen. Ben.
Ablations on Online Dropout
0.05
65.1
47.1
60.4
53.8
57.6
56.9
63.7
0.3
66.5
47.4
62.6
53.9
62.1
58.0
68.5
0.5
66.7
47.7
61.4
52.8
61.6
68.1
64.2
45.8
58.6
50.6
51.5
53.2
62.2
Abblations on STM
10
65.1
45.7
61.4
52.8
58.6
55.3
66.3
66.5
47.4
62.6
53.9
62.1
68.5
= 50
64.7
46.1
60.4
52.6
55.7
57.3
67.8
Ablations on TTP
ry = 0,ra 60
59.3
45.1
58.4
47.8
55.3
54.1
62.3
75
65.1
46.1
60.4
51.8
60.7
57.0
67.9
35, 95
66.5
47.4
62.6
53.9
62.1
58.0
68.5
T1 Th 100
63.2
45.1
60.4
50.8
57.3
56.1
63.7
Table 10: Ablation on alternative compression and curriculum strategies. We compare Mpen with two alternatives: (i) VTM, learnable spatiotemporal merging bascline for token compression, and (ii) standard short->long curriculum. Numbers are accuracy (%) except that EP-100 MIR is mAP (%).
Benchmark
OpenMMEgo
VTM
w. short->long
EgoSchema
66.5
65.8
65.3
EgoPlan
47.7
46.3
45.9
QAEgo4D
62.6
61.8
62.4
EgoTask
53.9
52.1
53.7
OMEBench (beh.)
62.1
60.4
58.7
OMEBench (vis.)
58,0
55.7
56.2
HD-EPIC
36.1
34.2
35.4
OpenEQA
58.4
54.3
53.1
EP-100 MIR (mAP)
46.1
45.2
45.7
(such 0.5) have limited impact performance. This suggests that online dropout effectively improve performance as long as the dropout ratio remains within reasonable range.
STM Ablation. The hyperparameter k governs the compression ratio in spatial token compression. With experiments conducted using the upper limit of input frame count, the results the Table indicate that both excessively high and low compression ratios lead to performance degradation. This suggests that aggregating semantic entity information is effective, and granularity of 35% is suitable level for aggregating semantic entities.
TTP Ablation. The hyperparameters ri and r', are related to the compression ratio temporal token compression. The experiments in Section 5.3 have already demonstrated the effectiveness of temporal compression. Here, we keep the token pruning ratio constant to investigate the impact of different compositions of high and low pruning similarity model's performance relatively sensitive to this parameter. possible reason is that excessive pruning of tokens with high similarity will cause semantic loss, while excessive pruning of tokens with low similarity may lead to loss temporal information.
the experiment. As shown in the Table 9, the
Strategy Alternatives. We also test alternatives to our design, including VTM (Lee et al., 2024) (learnable spatiotemporal merging) for compression and short-Hlong curriculum. As shown in Table 10, both underperform our dual strategy: VTM drops entity-awareness and hurts OMEBench, while short~>long yields only minor gains, confirming the effectiveness of our difficulty-awee curriculum.
28

---

E
Additional Visualization on Visual Token Compression
For better understanding of the visual token compression process in different scenarios, we here provide more visualization examples of the visual token compression process in OpenMMEgo- LLaVA-Video, As shown in Figure 12, the videos vary in the distance of the activity area and the degree of perspective change. Our method effectively compresses the visual tokens in the video while retaining important semantic information across these different scenarios.
F
Limitation Discussion
While OpenMMEgo enhances large multimodal models' (LMMs) ability understand egocentric videos by constructing a large-scale synthetic dataset, this data-driven process implicitly injects certain reasoning priors into the models. In the context of the recent surge in interest toward reasoning capabilities, a natural question arises: can these reasoning abilities be further improved through reinforcement learning (RL), particularly tailored for egocentric scenarios? At present, this direction remains unexplored in our experiments. Our observation is that the effectiveness of RL eliciting reasoning hinges the model's ability produce basic reasoning-orienter outputs point. However, current LMMs still exhibit notable limitations in egocentric video reasoning, which may hinder RL-based enhancement. From this perspective, dataset serves to equip LMMs with foundational capacity for egocentric video understanding, potentially paving the way for future work that leverages reinforcement learning induce more advanced, task-specific reasoning abilities.
starting
29

---

Input & Task
You are provided with sequential frames extracted from ego ggo-centric video, with consistent time interval of seconds between adjacent frames. As an aid, some narrations are also provided to you. Some marks are used in the narrations: "#C" för camera wearer, 'Nnot sure' the uncertain information, 'summary' for summary numratonn sequence, '#O' for other action conductors. Your task to analyze the scene provide response JSON format which depicts the scene sets and motion analysis of the scene in the ego-centric video.
Guidelines:
identify objects in seene and describe them, including their names, attributes (such size, color, feature. condition, and any distinguishing characteristics), and positions. For each object that appears, provide:
Object Identification and Description: Analyze all
<ohject-entryy:: "Name" <strings, "Attributes': <strings, "Position": <string=)
2. Tracking Adjective Changes: For every pair of adjacent frames, document changes in each object's attributes and position. If an object is partially visible or temporarily disappears, infer and describe its motion, including the reasoning, Additionally, classify each object into one
following categories:
Environmental Background: Stationary elements the scene whose apparent movement to camera motion. Interactive Object: Objects manipulated used by the
often moving closer into view.
Moving Passerby: Objects people moving independently of the camera wearer.
For each entity, provide:
Sentity-changee chaneee-hnnnn-rr "Name" <strings, "Change": <sttrnng>. "Visibility" <string?, "Type": <string?
3. Camer Movement Analysis: Infer the camern wearer's 's movement from changes perspective across frames, considering shifts in various object types. Describe the trajectory, including direction, angle, speed variations. Deduce possible body part movements (e.g. head, arms, legs) based these perspective changes and available pair, provide:
each adjacent frame
comeer--novvment--otrryy- ("Perspective': estring>, "Camera Movement" scsttringg "Body Part Movement": <strings)
4. Scene Summary: After processing all frames, synthesize a comprehensive scene summary. Include:
a) A structured spatial description of the environment and object layouf, with reference to logical groupings
b) summary of the camera wearer's 's motion across the scene, detailing trajectory and changes in orientation or speed.
<) A summary of inferred body movements throughout the scene.
For each video clip. provide:
Kacene-summar yentryy:: Soantryytt"SSettee "Seene" <strings, "Camera Movement": <string?, "Body Part Movement "sestring?"
and objects in the envirenment. Include every plase of interaction (e.g.. approaching the objeci, manipulating it, relocating it). Clearly describe: object characteristies,
Interaction Analysis: Describe all interactions between the camera
affordance, specific body parts and tools involved, purpose of interaction, detailed motion trajectory (including precise angles, distances, speed. path curvature, rotation, depth changes, and sputial shifts) For each interaction, provide: cinteraction-entry- ""Interaction" <string?, "Start Time". <inc>, "End Time". "Description" <string>]
Output Format
A JSON formatted oulput:
"Entities": [=object-entty-D>, 2object-ontty-23....
"Adjective Changes":
*0,1":
"Entity": |samtty-changge-mtn-tt cenity-chnngeeeettyy.>.
"Camera Movemmentt=ccmmmeemmeeeeee
"1,2%
"Entity": [seniit-change-uuty-1,, catily-change-entt-2....
"Camera Movement": scamera-mmovemetteetyy
"Seene Summary": -scom--uummar---nnny-> "Interactions": (<interaction.entry... <interactinneenrr.22
Key Requirements:
1. Output must be pure JSON without any markdown code blocks other formatting.
2. All information must be explicitly stated. Take each adjacent frame pair into consideration for motion and spatial analysis.
3. All the descriptions must be detailed and specifie, avoiding generic terms.
4. The output must be valid ISON without any comments. Every object's motion must be fully described for each time interval, without any omissions or shortcuts. No ellipsis (...) or comments about similar patterns.
Figure 8: The prompt used for vision annotation with Gemini -1. 1.5 S-pro.
30

---

not Input & Task
You are tasked with generating high-quality. Qnestion-Answer (QA) pairs Multiple- Questions (MCQs) to evaluate participants' understanding of egocentric videos. Instead raw video input, you are provided structured descriptions each video segment, spanning durations from seconds several minutes. Each segment divided 10-second intervals and annotated with:
* detailed list of Entities, including their Names, Attributes (e.g. color, material, size, status), and Positions (relative
the ogocentric view).
Per-second Adjective Changes för each entity: tracking movement, interaction, rotation, occlusion, and visibility changes. Detailed Camera Movement and Body Part Movement per second: covering perspective shifts, self-motion, and viewpoint adjusmmetts.
A high-level Scene Summary and Interaction Summary: describing ongoing actions, spatial layout, object interactivity, and bodily involvement.
As an assistance, some concise human annotations of the atomic actions are also provided to you. Some marks are used in the narrations: #C' for the camern wearer, '#not sure' for the uncertain information, #summary' for the summary of the narration sequence, '#O' for other action conductors.
Your task is generate question-answer poirs that probe the participant's comprehension of visual, temporal. sputial, causel, motor. and interactional aspects egocentric video, based mainly on the annotated visual information and slightly the human unnotations.
The concrete visual attributes of the objects and the spatial-temporal relationships between them should be integrated in the answers.
Each question must have one correct answer and three plausible distractors. The distractors should be confusing but incorrect. grounded in visual elements that appear the video but do satisfy the question, or artificial elements that could reasonably be inferred but are wrong.
Guidelines:
1. Question Scope: You should generate questions about Equestion_scone You should generate appropriate questions in composition.
these angles
2. Question-Answer Quality:
The generated questions should vary in complexity and scope. Across the generated QA pairs, ensure coverage of diverse question angles within the assigned dimension. Avoid producing repetitive patterns or overly similar question formulations. Some questions should target fine-gruined visual details and result in short, factual unswers. Others should require reasoning across multiple temporal segments entities, resulting in longer, integrative answers. Both short-form and forg-form questions must be present in each generation batch. For the diversity of questions, only part of the information should involved each QA pair. Though the answers may be long. the questions should be short and concise to hit the nail on the head.
The generated questions should vary in complexity and scope. Across the generated MCQs, ensure coverage of diverse question ungles within the assigned dimension. Avoid producing repetitive patterns or overfy similar question formulations, Some questions should target fine-grained visual details and others should require reasoning across multiple temporal segments entities. For the distractors: a) Distractors must be plausible in the scene or event. b) They should differ subtly from the correct answer in visual facts, egg. a wrong object, wrong timing, wrong attribute, slightly incorrect reasoning. c) Avoid obviously false out-of-scope distractors.
3. No Answer Leakage: The answer must not be directly inferable from the question text alone. Avoid phrasing that embods hints, reuses descriptive terms from annotations, or overly specifies entities or outcomes. The video must be watched
4. Answer Certainty and Grounding: Fach question must have unambiguous correct answer that is fully inferable from the structured annotations, Do not include speculative or uncertain phrasing in the answers. Avoid terms like "maybe," "appears to." or "possibly."
5. No Internal Identifiers or Serial Labels: Do not use labels such as "Person A," "Rope 1," or "Frame 3" in questions or answers. Instead, refer to entities using their described attributes (e.g. "the thick red-and-blue rope." "the with pule or the activity to ground the instead of specific frames or second numbers,
skin and slender fingers"), ",, use the that viewer ground them visually. Also, the segment information not provided the participants, mention the segment index time in the question answer.
Output Format
Provide the output as JSON string, structured as Python List and each entry is Python Dictionary. Do not include comma at the end of the last entry.
Example Output:
("Question" "squestion 1=*, "Answer": "Sanswer: 1>*, "Distractoes": ["<ddistactte- 1-1o". "odistractor 1-20", "odistractor 1-sesttoo---3331] I13>
["Question": 1 "Squestion-122 ",", "Answer" "Canswor-1 [=*",
["Question::
"squestion-Vss" "Answer": "canswer.Ns"))
""Questinn:: "squestion-N>" "Answer": "<answer-N"" "Dismmeton"": ["<distractooN- "edistracter-N.22 "cdistnntornN-33-
Attention:
1. Output must be pure JSON without any markdown code blocks or other formatting.
2. The output must be valid JSON without any comments. No ellipsis (...) comments (?) about similar patterns.
3. no relevant QA pairs be generated, simply return the word 'None' in string format.
Figure 9: The prompt used for vision-centria data synthesis with Gemini-3 1iitini--.--pprr the yellow block are used for open-ended QA synthesis, while the texts in blue blocks are used for Multiple- Cholcess Question synthesis. The question_scom scope is replaced with concrete descriptions in Figure 10 in the implementation.
5-pro. The texts in
31

---

Atomic Actions, including:
Action recognition; Identifying specific movements or manipulations mentioned in the narration.
Interactive object grounding: Determining which objects are involved in a specific action.
Temporal reasoning: Understanding action durations, sequences, changes time.
Action counting. Inferring how many times an action occurred, or comparing frequencics.
Chunked Behaviors, where atamic actions grouped into higher-level descriptions. These questions should:
Aggregate sequenees of atomic actions into meaningful behavioral units.
Ask for interpretations overall behavior (e-g.. preparing tea, cleaning up).
Compure contrust different behavior phases repetitions.
Goal-Step reasoning, including:
Goal analysis: Inferring the likely objective based behavior.
Next-step prediction: Predicting what should logically happen next to complete the lask. Procondition identification: Deternining what must have already occurred for the next stop to procedd.
Open Form. Generate unique and creative questions specific to the given egucentric video segment. These questions should beyond generic templates and instead target the most distinctive, non-obvious, or surprising aspocts observed the structured annotations.
context-sonsitive questions that specific the unique content the narrated video:
Identifying un unexpected behaviur or anomaly uccurring in the scene.
Spotting subtle or unusual interaction sequences not typical in everyday contexts.
Reasoning about cause-effect relationships that unique to this video (e.g., why certain reaction happened after minor motion).
Test story like comprehension or complex causal chains.
understanding spatial relationships of objects and their evolution time. They should probe the participant's 's ability to truck object positions, relative displacements, and visual transitions, considering both
Temperal Reasoning. These questions
local and global temporal contexts. Allowed question angles are as follow:
Tracking the spatial movement of single entity
time
Comparing the relative positions of multiple entitiss across different time points
Detecting consistent motion treads, such as convergence, divergence, or rotational drift
caused by camera motion object motion Identifying spatial formationss or layout shifts of objects and surfaces
Inferring whether observed movements
Reasoning about positional symmetry or alignment between objects at different moments
Any visual reasoning that involves the coupling of temporal change and spatial layout
Self-Motion Understanding. Questions should facus on interpreting self-induced motion putierns, inc including buth intentional actions and subtic balance adjustments. Allowed question angles include:
Detecting changes in body pose (c.g., sitting, standing, leaning, turnig)
Inferring head or torso rolation and the resulting shift in visual perspective
Analyzing hand movement trajectories relation to torsolcamera motion
Identifying transitions between static and dynamic bodily states
Detecting compensatary movements for bulance (c shifting weight before or after an action)
Evaluating how visual stability maintained or disturbed across
Any reasoning involving bodily posture, physical coordination, or propriocoption from first-person view
Interactions. These interactions may be tool-based, hand-driven, involve multi-obiect manipulation. Allowed question angles include: Describing fine grningr manipulation of specific object (e gripping, rotating, pulling)
Analyzing two-hand two-handed coordination on a single object or across multiple items Inferring mechanical functional changes caused by interaction (c.g.. rope tightening, opening)
Identifying when and how objects are combined, derached, or used in sequence
Detecting interaction phases (initiation, mid-action, completion) and associated leedlback
Recognizing when tool affordansess are being utilized repurposed
Any inference involving dynamic physical contact or control over objects
Vision-Inlien lsiien-nfferred Behavior. High-level reasoning about intentions, planning, affordances, and goals. should be grounded visible cues and contextual transitions, avoiding speculative purely textual logic. Allowed question angles include:
Inferring the goal or purpose an action based on current visual and physical context.
Determining whether an object's properties (e.g. position. material, accessibility) afford a specific action.
Assessing the sequencing of actions to evaluate goal-directed planning proparatory behavior.
Reasoning about behavioral changes in response to visual scene dynamics (c g. new object entering view).
Judging whether a particular behavior has achieved intended result, using visual evidence.
Predicting plausible next actions based ongoing activity and objoct affordances.
Connecting current actions to previously observed steps in longer behavioral chain.
Any reasoning that connects perceptual cues with implied agent intention, planning, or task logic.
Figure 10: The detailed description of question scopes used in behavior- based data synthesis (top)
and vision-centric data synthesis (bottom).
32

---

O: After rumoving id from the how does the camera after the from
Where biee located relotvve to the kitchen counter?
Choicee: behind the
wiible
from
kihhen, vislen-crntrte
visbll
Q: What artion precedes moving the vacuum deaner around living room?
Plugging the into precodes
from: behaviour based
What docs the camera wearer do ofter driling the first screw?
Wipss the drilled
Moves to differest location.
inspecs deil bit.
0. Starts d' d'illng another scrow
Q: Describe the overall sotting in the video clp
place warkshuo, There's wooden materia's Eareerch, spray deth, Many tonh banging on woel behind workinench partisly light
datteredd
Sirpcle entrance and
bicycle roposiee warbbench.
from: vidios centric spill
Where pump?
Back in the bact
On the ground nest to the breycle.
the cat's passenger footwell.
D. On the pasaenge sed.
Qr Based the obsenved actione what the camera wea most Holly
doing neet?
At Afber taking out ingred ants from the dramar, the person operatiss the leptop,
ihayy to look up recipe or sae timer.
from:
gray appear to move
Cholces: Dirctly overtaknn from bahind the camera weare's car From left to right zcress the frame.
right left across the frems.
action A: The menerr--wsttrrr holds the wdlge of beigs, conduroyela be febric with and Solds the fabris over bados.
Starts triling another scwe immedatell.
From: vidonugeetric
Figure 11: Examples from OME10M (Left) and OMEBench (Right). The QA pairs from dual splits of OME10Mare the left. Examples from OMEBench and the corresponding answers from OpenMMFFg-QQwen2.555 and Qwen2.5-VI are provided on the right.
Frumcz2
Frunce
RAW
STM
actiityy recronee perppactreathh
[b] alttiiiyy zta maderao dittans, obtees penpattrestiit
actitty z014. perppcciov
Figure 12: Visualization of the visual token compression process of OpenMMFigo-LLaV--Wieee RAW refers the original frames from videos. STM and TTP represent the visual tokens after Spatial-redundant Token Merging and Trreeval Token Pruning, respectively.
33


Reviews:
[{"review_id": "mDmV0xWxoW", "reviewer": "Reviewer_iYek", "summary": "This paper presents OpenMMEgo, an egocentric-focused VLM that is trained with novel data, model architecture and training strategies. In terms of data, the paper introduces OME10M, a large-scale (8.2M) synthetically-generated video QA dataset (for training) based on the Ego4D videos, and a corresponding evaluation split called OMEBench. On the modeling side, the paper performs two levels of semantic token compression to reduce the number of video tokens used in standard LLaVA-like architectures. Spatial-redundant Token Merging merges tokens within an image based on visual similarity-based connected components. Temporal-irrelevant Token Pruning follows a slow-fast strategy where slow frames retain all tokens and fast frames remove tokens that are too similar (i.e., redundant) or too different (i.e., irrelevant) to the previous slow frame. Finally, during training, the paper proposes a dual curriculum learning strategy that divides training data into different difficulty levels (offline) and trains on them one after the other. When training on a particular difficulty level, a hard-negative rejection strategy is used to downweight the probability of sampling hard samples. Experiments performed across different egocentric benchmarks demonstrate the benefits of the proposed method over several open-source baselines.", "questions": "I liked the paper in general because the ideas are interesting and the results are good. However, I have serious concerns that I would like the authors to address:\n* Why is there a lack of comparison with related work and baselines suggested in the weakness section? These are basic and necessary to understand the contributions of the paper. \n* Can we have more results on non-Ego4D video benchmarks?\n* Could the authors clarify the missing details in the paper?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n* The paper proposes an interesting token merging approach that reduces number of video tokens spatially and temporally with benefits for overall performance.\n* OME10M is curated sensibly and looks promising for the community to make progress on egocentric video understanding with VLMs\n* The authors have promised to open-source the code and data (these are not included in the initial submission)\n* Experiment design covers some essential egocentric video baselines and benchmarks (Table 1), and critical ablation studies indicating that each component of the model and data helps to varying degrees (Table 3, Figure 3) The results in Table 1 are promising since OpenMMEgo goes beyond MMEgo and provides large gains on the base VLM used for finetuning.\n\n# Weaknesses\n**Related work section is incomplete:** There are several related works on token merging and synthetic video QA generation. These have not been discussed sufficiently in the paper, and the benefits of the proposed method over these have not been explained nor studied.\n* Token merging\n\t* ToME: https://arxiv.org/pdf/2210.09461\n\t* Token Merger: https://pubmed.ncbi.nlm.nih.gov/37440394/\n\t* LTMP: https://openreview.net/pdf?id=WYKTCKpImz\n\t* DTEM: https://arxiv.org/pdf/2412.10569\n\t* ToSA: https://arxiv.org/abs/2506.20066\n\t* STTS: https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136950068.pdf\n\t* VTM: https://arxiv.org/pdf/2410.23782\n\t* InTI: https://arxiv.org/html/2408.06840v\n* Synthetic annotations for egocentric videos\n\t* LaViLa: https://openaccess.thecvf.com/content/CVPR2023/papers/Zhao_Learning_Video_Representations_From_Large_Language_Models_CVPR_2023_paper.pdf\n\t* LLava video: https://llava-vl.github.io/blog/2024-09-30-llava-video/\n\t* EVUD: https://huggingface.co/datasets/AlanaAI/EVUD\n\t* Eagle 2.5: https://arxiv.org/pdf/2504.15271\n\n**Missing baselines:** The experiment section neglects benchmarking alternate strategies for token merging, curriculum learning, and synthetic data annotation.\n* **Token merging:** There are several works with varying motivations for reducing video tokens (listed above). The proposed STM and TTP strategies need to be benchmarked against these to verify that it is indeed useful.\n* **Synthetic annotations:** The simplest synthetic annotations to generate are to ask a VLM like Gemini 1.5 Pro to generate questions given a video. There are more sophisticated strategies like Llava video and Eagle 2.5 that perform hierarchical captioning + QA generation, and LaViLa that generates dense narrations by training a narrator (which can then be used for QA generation). The proposed synthetic pipeline needs to be compared against these alternatives to show its benefits.\n* **Curriculum learning:** A common strategy for training video models is to train on images first, short videos and then long videos (e.g., see https://arxiv.org/pdf/2501.00574v1 and https://www.arxiv.org/pdf/2408.10188). The proposed strategy based on sample difficulty is interesting, but simpler alternatives have not been compared against. Similarly, hard-negative filtering (L234) is interesting, but it is not clear how this compares to more traditional strategies like hard negative mining (or easy positive rejection) --- see page 2 here: https://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf.\n* Gemini 1.5 Pro is used for generating annotations, but this model is not used as a baseline. Does the proposed approach go beyond Gemini 1.5 Pro? Or is it limited by the performance of the model (in which case, this would basically be a distillation from a closed-source model to the proposed model)?\n\n**Benchmarks primarily focused on Ego4D:** The benchmarks in Section 5.2 are primarily focused on Ego4D. EgoPlan also includes a good part of Ego4D, and EgoTaskVQA is the only benchmark that goes outside of Ego4D. The results in Table 1 indicate that non-Ego4D benchmarks have the smallest gains (likely because the videos are out of domain from the training set). I would encourage the authors to include results on more non-Ego4D benchmarks to get a better sense of the broad applicability of the model. Here is a list (not comprehensive):\n* EPIC Kitchens  \n* OpenEQA  (https://open-eqa.github.io/)\n* EgoLifeQA (https://arxiv.org/abs/2503.03803)  \n* HD-EPIC (https://hd-epic.github.io/index#starthere)\n\n**Missing details / clarifications needed:**\n* **L171 - 172:** Is OMEBench human verified?\n* **Section 4.1:** How expensive are the token merging operations? Can they be performed efficiently? How much time and compute does it take relative to the rest of the model operations?\n* **L203:** How are tokens merged within a component?\n* **L215:** \"excessive semantic similarity\" --- isn't this necessary to understand temporal movements of the object? Will removing redundant tokens like these cause a model to not understand basic flow of time?\n* **L245 - 247:** Why not use the base model that is being finetuned for the difficulty estimation?\n* **Figure 3:** The charts are a misleading here. \n\t* The left chart shows somewhat linear scaling of results, but the X axis is not linear. I would recommend a scatter plot with proper X scaling.\n\t* The right chart is very confusing because the scaling is different for each benchmark. For example, on EgoPlan, OpenMMEgo w/o STM performs exactly the same as EgoPlan, but a big margin is shown in the chart. I recommend having uniform scaling across all benchmarks to have a better understanding of the results.\n* **Figure 5:** What is EgoCatch?\n\n# Post-rebuttal update\nMajority of my concerns were addressed, so I'm raising my rating to border-line accept. However, some core concerns on token compression, synthetic annotations and baseline comparisons were not sufficiently addressed. I'm not inclined to raise it further without these being addressed sufficiently.", "was_revised": true, "final_justification": "The authors addressed several of my concerns, but not some of my core concerns on token compression, synthetic annotations and baseline comparisons (summarized below). I've raised my rating to borderline accept (but not accept) because of these reasons.\n\n# Remaining concerns\n\n* **Token compression:** This is one of the core contributions (L56 - 62). The paper lacks a discussion of what prior token compression methods do and how this paper innovates over it. The authors have included one sample method in the rebuttal response, but it is unclear how this was selected and why other methods are not included. This needs a thorough study.\n\n* **Synthetic annotations:** *LlavaVideo dataset* is used as a baseline, but the *approach for generating the LlavaVideo* dataset is not compared against. I would like to emphasize this difference. Simply using prior datasets in comparison is not what I'm suggesting. My key concern is that the data curation approach proposed here is not vetted against prior data curation approaches in an apples-to-apples fashion (same videos, same VLMs for annotations, but different curation process).\n\n* **Baseline comparison is not apples-to-apples:** For context on Gemini 2.0 Flash vs. Gemini 1.5 Pro: Gemini 1.5 Pro is quite a bit better on video benchmarks as shown in Table 6 here: https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf. So the numbers reported here are likely worse than what 1.5 Pro would have scored.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript.\n  \n**Potential Related Works**\n1. **Token Compression techniques:** The reviewer listed many token merging methods developed for efficient visual representation learning, including ToMe, which we have cited in Line 194,  and ToSA, a paper posted after the NIPS submission Deadline. Our focus is on adapting token‐compression techniques (merging and pruning) in video understanding LMMs. Accordingly, we cited the classic origins (ToME for merging, LTP for pruning) in single-image and language-model contexts, and we discussed their extensions in video LMMs (e.g., VideoChat-Flash, LongUV, VideoLLaMA 3, etc.). VideoChat-Flash is also used as our baseline.    \n2. **Synthetic Annotations:** LLaVA-Video was indeed cited and is one of our baselines, based on which our SFT variant yields substantial gains. LaViLa focuses on visual-representation pretraining and thus differs in scope from LMMs. EVUD, a four-page unpublished manuscript, was inadvertently omitted, and Eagle 2.5 (a concurrent work) also went unnoticed. \n\nWe appreciate the reviewer’s suggestions and will include discussions of all these works in the next revision as an additional Related Works section.\n\n---\n**Additional Baselines & Benchmarks**\nWe thank the reviewer for pointing out potential baselines and suggesting the non-Ego4d benchmark, which will indeed help us better present the performance of OpenMMEgo.  \n- **Gemini Model Comparison**: Considering the scale difference with Gemini-1.5-Pro and the fact that the Gemini-1.5-Pro API is no longer available, we report the performance of the Gemini-2.0-Flash API on our egocentric benchmark as a comparison. The model size of Gemini-2.0-Flash is closer to our 7B model, and it is reasonable to assume that its training data and process are very similar to Gemini-1.5-Pro.  \n- **AlanVLM**: We have included AlanVLM as an extra baseline (the LMM trained on EVUD), serving as a supplementary baseline for synthetic data.  \n- **New benchmarks:** We have added results from the HD-EPIC VQA benchmark and the open-ended OpenEQA.  We also convert the multi-instance retrieval (V → T) task in Epic Kitchens 100 into a multiple-choice task and compared the mAP performance with EgoVLP and LaViLa-L.  \n\n|                        | EgoSchema   | EgoPlan     | QAEgo4D         | EgoTaskVQA  | OMEBench (beh.)  | OMEBench (vis.)  | HD-Epic        | OpenEQA    | EP-100 MIR(V →T) |\n| ---------------------- | ----------- | ----------- | --------------- | ----------- | ---------------- | ---------------- | -------------- | ---------- | ---------------- |\n| EgoVLP                 | -           | -           | -               | -           | -                | -                | -              | -          | 26.0             |\n| LaViLa-L               | -           | -           | -               | -           | -                | -                | -              | -          | 40.0             |\n| Gemini-2.0-Flash       | **71.1**    | **51.5**    | 62.4            | **59.2**    | 60.2             | 62.1             | 38.8           | **61.5**   | 49.3             |\n| AlanVLM                | 50.7        | 27.6        | 41.8            | 43.2        | 27.3             | 33.2             | 25.7           | 46.7       | 23.1             |\n| Qwen2.5-VL             | 65.0        | 45.2        | 59.4            | 53.7        | 55.3             | 49.4             | 34.2           | 55.7       | 42.1             |\n| VideoChat-Flash        | 64.3        | 40.8      | 62.4            | 54.8        | 57.1             | 54.5             | 39.4           | 58.1       | 45.3             |\n| OpenMMEgo（Qwen2.5-VL）  | 69.3 (+4.3) | 50.2 (+5.0) | **65.6** (+6.2) | 56.2 (+2.5) | **65.7** (+10.4) | **63.2** (+13.8) | **42.4**(+8.2) | 60.2(+4.5) | **49.7** (+7.6)  |\n\nFurthermore, the reviewer notes that certain prior works in token compression, synthetic data, and curriculum learning could serve as alternatives to some of our components, thus forming potential baselines. Among the potential baselines pointed out by the reviewer, due to time and resource constraints, we adapt VTM  to OpenMMEgo as the baseline for video token compression and the strategy from short to long videos as the baseline for curriculum learning. In terms of synthetic annotations, LLaVA-Video is already our baseline, and we have additionally included a comparison with AlanVLM (fine-tuned on synthetic data EVDU), so no further comparison is made. We perform SFT on Qwen2.5-VL with 1.6M data.\n|                        | EgoSchema   | EgoPlan     | QAEgo4D         | EgoTaskVQA  | OMEBench (beh.)  | OMEBench (vis.)  | HD-Epic        | OpenEQA    | EP-100 MIR(V →T) |\n| ---------------------- | ----------- | ----------- | --------------- | ----------- | ---------------- | ---------------- | -------------- | ---------- | ---------------- |\n| OpenMMEgo              | 66.5        | 47.7        | 62.6            | 53.9        | 62.1             | 58.0             | 36.1           | 58.4       | 46.1             |\n| OpenMMEgo w. VTM       | 65.8        | 46.3        | 61.8            | 52.1        | 60.4             | 55.7             | 34.2           | 54.3       | 45.2             |\n| OpenMMEgo w. shot→long | 65.3        | 45.9        | 62.4            | 53.7        | 58.7             | 56.2             | 35.4           | 53.1       | 45.7             |\n\nAside from being comparable to Gemini 2.0 on certain metrics, OpenMMEgo consistently outperforms all other baselines, further demonstrating the effectiveness of our method.\n\n---\n\n**Token Compression Cost vs. Benefit**\nToken compression does incur additional computation. Normalizing the inference time of “w/o STM+TTP” to 1.0, we compare supported maximum frames and relative cost:\n\n\n|                | w/o STM+TTP | w/o STM | w/o TTP | OpenMMEgo |\n| -------------- | ----------- | ------- | ------- | --------- |\n| frames         | 64          | 106     | 91      | 192       |\n| inference cost | 1.0         | 1.31    | 1.19    | 1.47      |\n\nOur token compression strategy enables processing **two times additional frames** with only a 47 % increase in inference time.\n\n**Missing Details / Clarifications**\nWe appreciate the reviewer’s corrections and provide the following clarifications:\n1. **Figure 5:** All answers are produced by OpenMMEgo; “EgoCatch” was a mislabel.\n2. **Figure 3 (right):** Our radar chart is plotted with the scale set to 0 and 1, where the minimum value is a multiple of 5 below the lower limit and the maximum value is used as the upper limit. However, for aesthetic purposes, we manually labeled the values in the plot with PowerPoint. Unfortunately, due to a saving error, there are two typos (OpenMMEgo w/o STM+TTP's Gen.vis. is 67.5, and OpenMMEgo's Egoplan is 47.7). The proportions of the plot itself are correct. We sincerely apologize for any confusion caused.\n3. **L171–172 (OMEBench Verification):** No, limited by our human resources.\n4. **L203 (Merging):** We merge tokens within each component by summing, following ToME.\n5. **L215 (“Excessive Semantic Similarity”):** Tokens with excessive semantic similarity are pruned only in fast frames. On one hand, in fast frames, we focus on the dynamic changes of these semantic tokens. In our visualization (Fig. 4), the tokens with the highest similarity are often static ones, such as background elements, while the dynamic parts are not pruned. On the other hand, in slow frames, the static temporal information of these semantic tokens is preserved.\n6. **L245–247 (Curriculum Difficulty):** In the Offline Data Curriculum, we are more concerned with the inherent difficulty of the data itself rather than its ease of learning. Using a fine-tuned model to evaluate might be influenced by the extent to which the data can be optimized. In the Online Data Curriculum, our dropout mechanism compensates for this.\n7. **Figure 3 (left):** We will update the left plot per the suggestion; the right-plot typos have been corrected as above.\n\nWe hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions."}, {"review_id": "P7EQHv88nK", "reviewer": "Reviewer_pEBA", "summary": "In the field of egocentric video understanding, this paper presents OpenMMEgo, a large multimodal model that addresses challenges across three dimensions: data curation, model architecture, and training strategy. To mitigate frequent viewpoint shifts, the study introduces two key strategies: semantic-aware visual token compression and a curriculum learning framework. Additionally, the authors curate a new dataset and benchmark, named OME10M and OMEBench, respectively. Experimental results show that OpenMMEgo achieves significant performance gains on multiple egocentric benchmarks.", "questions": "1. Previous egocentric LMM studies often rely on Ego4D-based data augmentation. The paper could benefit from an in-depth comparison of different data construction strategies and their impact on model performance.\n\n2. While the Spatial-Temporal Merging (STM) and Temporal Token Pruning (TTP) methods aim to reduce redundancy, the paper does not explicitly report computational costs (e.g., inference time, memory usage) compared to baseline models.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. OpenMMEgo proposes a multi-faceted approach, including: (i) Hierarchical data curation (both high-level and low-level) to enhance semantic richness. (ii) A model architecture designed to reduce computational overhead. (iii) Training strategies that optimize both stability and effectiveness.\n\n2. The model demonstrates significant improvements on standard egocentric understanding benchmarks and shows comparable performance on general video understanding tasks, highlighting its generalizability.\n\n3. Extensive ablation studies systematically validate the contribution of each core component, providing clear insights into the model’s design.\n\nWeaknesses:\n\n1. The writing in Section 3 (OME10M curation) lacks clarity, particularly regarding: (i) The specific methods used to construct behavior-based QAs and vision-centric QAs. (ii) The relationship between visual details annotations and the two QA types, making it difficult to replicate the curation process.\n\n2. The paper omits recent advancements in egocentric research, including: (i) Benchmarks: EgoVQA, EgoThink, VidEgoThink, EgoPlan-2; (ii) LMMs: EgoVLP, EgoVLPv2, AlanaVLM.\n\n3. A more comprehensive comparison with existing egocentric LMMs would strengthen the model’s performance validation.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviews, there are no remaining concerns. I raised my rating accordingly.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript. \n\n**Data Curation Details**: Due to the limited space in the main text, we have only outlined the core design concepts. In Appendix B, we provide a detailed description of our data generation process, including the workflow, the specific types of data generated, and the prompts used. The visual details annotations are exclusively utilized in the visual details split of OME10M, and questions and answers are generated based on these annotations.\n\n**Additional Baselines & Benchmarks**\nWe thank the reviewer for pointing out potential baselines and suggesting the non-Ego4d benchmark, which will indeed help us better present the performance of OpenMMEgo.  \n- **AlanVLM**: We have included AlanVLM as an extra baseline, serving as a supplementary baseline for synthetic data.  \n- **Video Language Pretrained Models**: We convert the multi-instance retrieval (V → T) task in Epic Kitchens 100 into a multiple-choice task to enable a comparison with  Video Language Pretrained Models (EgoVLP and LaViLa-L).\n- **New benchmarks:** We have added results from the HD-EPIC VQA benchmark and the open-ended OpenEQA.   Due to time constraints, EgoVQA and VidEgoThink have not yet completed testing, but they are in our plan and will be supplemented once finished.\n\n|                         | EgoSchema   | EgoPlan     | QAEgo4D         | EgoTaskVQA  | OMEBench (beh.)  | OMEBench (vis.)  | HD-Epic        | OpenEQA    | EP-100 MIR(V →T) |\n| ----------------------- | ----------- | ----------- | --------------- | ----------- | ---------------- | ---------------- | -------------- | ---------- | ---------------- |\n| EgoVLP                  | -           | -           | -               | -           | -                | -                | -              | -          | 26.0             |\n| LaViLa-L                | -           | -           | -               | -           | -                | -                | -              | -          | 40.0             ||\n| AlanVLM                 | 50.7        | 27.6        | 41.8            | 43.2        | 27.3             | 33.2             | 25.7           | 46.7       | 23.1             |\n| Qwen2.5-VL              | 65.0        | 45.2        | 59.4            | 53.7        | 55.3             | 49.4             | 34.2           | 55.7       | 42.1             |\n| VideoChat-Flash         | 64.3        | 40.8     | 62.4            | 54.8        | 57.1             | 54.5             | 39.4           | 58.1       | 45.3             |\n| OpenMMEgo（Qwen2.5-VL） | **69.3** (+4.3) | **50.2** (+5.0) | **65.6** (+6.2) | **56.2** (+2.5) | **65.7** (+10.4) | **63.2** (+13.8) | **42.4**(+8.2) | **60.2**(+4.5) | **49.7** (+7.6)  |\n\nOpenMMEgo consistently outperforms all other baselines, further demonstrating the effectiveness of our method.\n\n**Cost of STM and TTP**\nIndeed, during the process of token compression, additional operations introduce new costs. To better illustrate the extra time cost of token compression, we use the inference time of OpenMMEgo without STM+TTP as the unit, comparing the costs of different operations with the maximum number of frames under the same memory.\n\n|                | w/o STM+TTP | w/o STM | w/o TTP | OpenMMEgo |\n| -------------- | ----------- | ------- | ------- | --------- |\n| frames         | 64          | 106     | 91      | 192       |\n| inference cost | 1.0         | 1.31    | 1.19    | 1.47      |\n\nOur token compression strategy enables processing **two times additional frames** with only a 47 % increase in inference time.\n\n\nWe hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions."}, {"review_id": "idlRMap5Kc", "reviewer": "Reviewer_FPzo", "summary": "This paper introduces OpenMMEgo, a family of egocentric large multimodal models (EgoLMMs). It also presents OME10M, a large-scale fine-tuning egocentric dataset with 8.2M egocentric video QA pairs from Ego4D, and OMEBench, a benchmark for egocentric understanding. The authors propose Dual Semantic-aware Token Compression, an egocentric representation compression method comprising two modules: Spatial-redundant Token Merging (STM) and Temporal-irrelevant Token Pruning (TTP). Additionally, the work introduces two enhancements to Dual Curriculum Learning Strategy: Offline Data Curriculum and Online Data Dropout. Experiments demonstrate that OpenMMEgo elevates Qwen2.5-VL to state-of-the-art performance in egocentric understanding.", "questions": "OpenMMEgo achieves similar performance to other methods on video understanding tasks, but outperforms other methods on egocentric video understanding tasks. The paper discusses some theories behind it, such as egocentric vision requires rich low-level visual cues. Are there any analysis and experiments that helps explain this?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This paper offers good technical novelty by introducing improvements in model architecture and training strategy.\n\nIt proposes Spatial-redundant Token Merging (STM) and Temporal-irrelevant Token Pruning (TTP), which together reduce redundancy and noise in video representation learning.\n\nFor training strategy, it introduces Offline Data Curriculum and Online Data Dropout, quantifying sample difficulty using loss and dynamically filtering challenging in-batch samples during training.\n\n2. OpenMMEgo achieved comparable performance with existing methods in general video understanding tasks, and achieves state-of-the-art in egocentric video understanding tasks.\n\n3. The paper includes ablation studies that validate the effectiveness of the proposed methods.\n\n4. All components of this work will be open sourced, promoting transparency and collaboration in this field.\n\nWeakness:\nNo major weakness was identified.\n\nMinor weaknesses:\nLine 35: Missing comma after OpenMMEgo?\n\nSome designs are under-explained in the main paper, such as the curation of OME10M, likely due to page limitations.", "was_revised": true, "final_justification": "The rebuttal addressed my questions and I will maintain my original rating.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We reply to the concerns below and will incorporate the mentioned revisions in the next version of our manuscript.\n\nLine 35: Indeed, a comma is missing at this position and will be inserted in the revised manuscript.\n\n**Data Curation Details**: The main text outlines only the core design concepts due to space constraints. A comprehensive description of the OME10M generation pipeline appears in Appendix B, covering the end-to-end workflow, all categories of synthetic QA pairs, and the exact prompt templates employed for behavior-based and vision-centric question creation.\n\n**The Role of Visual Cues**: Intuitively, for general understanding, third-person training data inherently contains descriptions of visual details, which are primarily lacking in egocentric video data. Therefore, introducing annotations of visual details from egocentric video makes the data distribution more uniform and thus easier to learn. On the other hand, the inference of events and actions in egocentric video often relies on these visual details. When egocentric data lacks visual details, general understanding and egocentric video understanding become significantly different tasks and are hard to foster each other. Experimentally, we removed the visual details data in ablation studies, which resulted in a comprehensive performance decline.\n\nThese clarifications will be incorporated into the revised version to enhance reproducibility and to shed light on why visual-detail supervision is essential for egocentric video understanding.\n\nWe hope these clarifications resolve the reviewer's concerns. Please let us know if any further questions."}, {"review_id": "svkzihXSz4", "reviewer": "Reviewer_3kaY", "summary": "The paper introduces OpenMMEgo, a recipe for improving large multimodal models (LMMs) on first-person (egocentric) video. The authors curate OME10M and OMEBench for model training and evaluation. The authors propose a Dual Semantic-aware Token Compression module that compresses long videos by (i) Spatial-redundant Token Merging (STM) and (ii) Temporal-irrelevant Token Pruning (TTP), aiming to keep informative tokens while discarding redundancy. Moreover, a Dual Curriculum Learning Strategy is introduced to combine an offline difficulty curriculum and an online loss-based dropout to stabilize learning on diverse data.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n\n1. Comprehensive egocentric dataset & benchmark\n\nOME10M and OMEBench fill a gap of the study of egocentric video understanding: public, large-scale resources for first-person video QA, with multi-level visual supervision.\n\n2. Thorough empirical evaluation\n\nResults cover five external egocentric benchmarks plus three general-video suites, and ablations dissect data components, curricula, and compression variants.\n\n3. Marginal degradation on generic tasks\n\nThe proposed egocentric tuning slightly harms general video understanding.\n\n### Weaknesses:\n\n1. Incremental technical novelty\n\nSTM builds on existing token-merging work, and TTP mirrors SlowFast-style sampling; both ideas are adaptations rather than fundamentally new. Similarly, offline/online curricula echo established schedules. The conceptual advance may be viewed as a limited aggregation of prior techniques.\n\n2. Unclear computation cost of STM and TTP\nAlthough STM and TTP can remove redundant tokens, their computational cost is underexplored. It remains unclear how the additional operations introduced by STM and TTP affect inference speed.\n\n3. Limited analysis of failure modes\n\nThe paper provides no qualitative error analysis or robustness checks.\n\n4. Limited comparison between different egocentric datasets \n\nThe reviewer appreciates the efforts of curating the OME10M dataset. It is recommended to compare OME10M with existing general video or egocentric video datasets to emphasize the contribution or specialty of the OME10M dataset.", "was_revised": true, "final_justification": "I have read the rebuttal and would like to keep my original rating.", "rebuttal": "Thank the reviewer for the thoughtful and balanced assessment of our work. We are grateful for the recognition of OME10M and OMEBench as valuable large-scale resources for egocentric video QA, as well as for the thorough empirical evaluation across multiple benchmarks and ablation studies. Below, we address each of these points in detail.\n\n **Technical novelty**: Indeed, some of the ideas in strategy we employed have appeared in previous work, but our design is tailored to the OME10M dataset. Determining which training strategies are more effective for learning on multi-level visual supervision data is a topic worth exploring. On one hand, our outperformance over the baselines to some extent demonstrates the effectiveness of our designs (for instance, VideoChat-Flash also employs various video token compression strategies). Additionally, we introduced further experiments, using VTM as an alternative for video token merging and adopting \"from short videos to long videos\" as a straightforward curriculum learning baseline. The experimental results on the 1.6M dataset are shown in the table below. These results highlight the value of investigating appropriate training strategies for LMMs and datasets.\n\n|                        | EgoSchema | EgoPlan | QAEgo4D | EgoTaskVQA | OMEBench (beh.) | OMEBench (vis.) | HD-Epic | OpenEQA | EP-100 MIR(V →T) |\n| ---------------------- | --------- | ------- | ------- | ---------- | --------------- | --------------- | ------- | ------- | ---------------- |\n| OpenMMEgo              | 66.5      | 47.7    | 62.6    | 53.9       | 62.1            | 58.0            | 36.1    | 58.4    | 46.1             |\n| OpenMMEgo w. VTM       | 65.8      | 46.3    | 61.8    | 52.1       | 60.4            | 55.7            | 34.2    | 54.3    | 45.2             |\n| OpenMMEgo w. shot→long | 65.3      | 45.9    | 62.4    | 53.7       | 58.7            | 56.2            | 35.4    | 53.1    | 45.7             |\n\n**Computation Cost of STM and TTP**\nIndeed, during the process of token compression, additional operations introduce new time costs. To better illustrate the extra time cost of token compression, we use the inference time of OpenMMEgo without STM+TTP as the unit, comparing the costs of different operations with the maximum number of frames under the same memory.\n\n\n|                | w/o STM+TTP | w/o STM | w/o TTP | OpenMMEgo |\n| -------------- | ----------- | ------- | ------- | --------- |\n| frames         | 64          | 106     | 91      | 192       |\n| inference cost | 1.0         | 1.31    | 1.19    | 1.47      |\n\nOur token compression strategy enables processing **two times additional frames** with only a 47 % increase in inference time.\n\n**Analysis of Failure Modes**: We thank the reviewer for the suggestion. Due to the limitations of not being able to include images in the response, we shall address this point in the next version of the paper by incorporating erroneous examples and the difference in results from multiple inferences."}]

Author rebuttal:

Meta-review: The paper receives consistent positive ratings from the reviewers. They appreciate the novelty of the proposed token reduction strategy and the curated large-scale benchmark OEM-10k for ego-centric video understanding with VLM. The reviewers had questions about some details of the method design and experiments. The authors' rebuttal addressed the problems well. AC also agrees with the comments of the reviewers and recommends acceptance as a poster for this submission.
Decision: Accept (poster)