MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA
Authors: Hanrong Ye
Haotian Zhang
Erik Daxberger
Lin Chen
Zongyu Lin
Yanghao Li
Bowen Zhang
Haoxuan You
Dan Xu
Zhe Gan
Jiasen Lu
Yinfei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal models
Primary area: foundation or frontier models, including LLMs
TL;DR: MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding with the contribution on: (i) Egocentric QA Data Engine (ii) Memory Pointer Prompting Mechanism (iii)  EgoMemoria benchmark.

Abstract: This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding.
To achieve this goal, we work on three fronts. 
First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data.
This is one of the largest egocentric QA datasets.
Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated.
Third, we propose a specialized multimodal architecture featuring a novel ``Memory Pointer Prompting" mechanism. This design includes a global glimpse step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content.
With the data, benchmark, and model, we build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.

Published as a conference paper at ICLR 2025
MM-EGO: TOWARDS BUILDING EGOCENTRIC MULTIMODAL LLMS FOR VIDEO QA
Hanrong Yelt, Haotian Zhang*t, Erik Daxberger', Lin Chen², Zongyu Lin', Yanghao LP,, Bowen Zhang,, Haoxuan You², Dan Xu', Zhe Gan?; Jiasen Lu Yinfei Yang": CSE, HKUST AAplee 3UCLA
saarenng.eeconnecee .us.uutt .nk, {haotian. zhang2, yinzeiyy
Human Narmaion
Shep I. Global
Glimpse
Entoklings
Enboddigs
spountes
Open-Reeporec
QA
Langrage
Model
Mulipio-Chiie Chiie
Q4
Language Model
Useetias: W auttiiiiityy dal resuuccly with cat?
Risd Soloc
Optinss:
benttd the with tetttrab.
Egocentric QA
S-Wei se on do the bicottiratn
1lgh
Enboakinge
Exboödings
LIM
C. wasked cat' ther with wabliets
Di carbee the ors dar with sowh.
(4) Egooennrie QA Samples
(b) Model: Memory Pointer Prompting
(c) Benchmark: EgoMemoria
Figure 1: We introduce foundation model for egocentric video understanding, contributing from three key perspectives: (a) 7 million egocentric QA samples generated from human narrations via a data engine, (b) multimodal language model designed for egocentric video comprehension, and (c) the curation of a challenging egocentric video understanding benchmark.
ABSTRACT
This research aims to comprehensively explore building multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D (Grauman et al.) 2022) based on human-annotated data. This is one of the largest egocentric QA datasets. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel "Memory Pointer Prompting" mechanism. This design includes global glimpse step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.
INTRODUCTION
Study on egocentric videos explores how machines can see and understand the world from a first- person, self-centered perspective. Egocentric videos differ significantly from static-camera videos, such as movies or animations, both in terms of content and viewpoint. The content of egocentric videos primarily revolves around human daily activities. These videos typically share perspective similar to human vision, where the camera and viewpoint frequently move. As a result of these char- acteristics, egocentric videos exhibit distinct data distribution compared to mrttc-cament videos,
Work done during internship at Apple. First Authors. Senior Authors.

---

Published as a conference paper at ICLR 2025
which has motivated a new area of research. In recent years, research interest in egocentric intel- ligence has been on the rise (Sigurdsson et al. 2018 Damen et al,, 2018 Grauman et al. 2022 Mangalam et al. 2023 Phizzri et al. (024). This growing interest driven by the rapid vance- ments in AR/VR headsets and robotics, where cameras capture long-form egocentric videos in a manner akin to human vision. Research on egocentric videos will allow these devices under- stand their surroundings and human intentions, fostering more advanced machine intelligence and improving the human-machine interaction experience, with immeasurable research and application potential.
However, research on understanding egocentric videos remains in its early stages, with previous re- search primarily centered on specialized tasks such as story summarization (Lee et al., (2012). hand- object relationship understanding (Cai et 20.6, action classification (Cartas et al. 2017 Li et al, 2021), and temporal or spatial grounding (Grauman et al. 2022). In contrast, works focusing on de- veloping more general egocentric video understanding model capable of complex understanding remain rare. Despite that video multimodal large language models (MLLMs) demonstrate strong video understanding and reasoning ability (Zhang et al. 2023a] Wang et al. 2024b Lin et al. 2024 Zhang et al., 2024b), most of these works unsuitable for egocentric video understanding from data, benchmark, and model design perspectives.
(a) From data standpoint, although many MLLMs some egocentric videos from Activi- tyNet (Yu et al. 2019). Ego4D) (Grauman et al. 2022). and Charades (Sigurdsson et al.) 2018) in their training, they have not been trained on large-scale egocentric video datasets, which inherently restricts their ability to comprehend lengthy first-person videos and accurately extract visual details. While Ego4D (Grauman et al. 2022) offers valuable human-annotated videos and labels for certain egocentric video understanding tasks, particularly episodic memory (which assesses a model's 's abil- ity to retain visual details in such videos), its annotations are not structured for generating language responses, making them unsuitable for training MLLMs. Therefore, large-scale egocentric video QA corpus is still needed. (b) In terms of benchmarking, exisiting video QA benchmarks either focus on shorter videos- such as EgoSchema (Mangalam et al.) (023) and QaFgo41), which eval- uate using around 3-minute and 8-minute vidcos, respectively - or concentrate on Internet video content (e.g.. Video-MME (Fu et al., 2024 This creates a notable gap in egocentric video under- standing benchmarks that encompass videos ranging from seconds to an hour in length. (c) From model design perspective, previous video MLLMs have primarily addressed long videos in two ways. The first approach involves uniformly sampling a limited number of video frames as visual input, as seen in Li et al. (2024a); Lin al. (2024). Despite its simplicity, this approach achieves better performance among open-source models on public video benchmarks (Fu et al., 2002)), largely because its design ensures high training efficiency and good scaling properties. The second approach involves feeding a large volume of visual tokens into the transformer backbone and employing en- gineering techniques, such as tensor parallelism and sequence parallelism (Xue et al. 2024 Zhang et al.) 2024a), to facilitate training with millions of visual tokens in context. However, these long- context transformers suffer from slow training speeds and small overall batch sizes, which hinder performance improvements given the constraints of computational resources and training time. In- tuitively, even humans cannot remember every detail of an hour-long video. We believe a more effective approach is to understand the video progressively: first get overview of the entire video, then focus on specific details with particular questions in mind.
Building the observations mentioned above, we introduce MM-Ego, egocentric MLLM de- signed to process and understand long egocentric videos. Our contributions are threefold:
(i) Data. To scale training data for MLLMs with egocentric understanding ability, develop an efficient data engine, using a "narration to egocentric QA" strategy, to automatically synthesize a large-scale egocentric QA dataset based on video narration data. Notably, rather than relying on existing vision-language models (VLMs) as labelers, we generate egocentric QAs based on the human -aman-annotated fine-graid ane-ggrineed video clip narrations. This approach, conceptually related to (Di & Xic, 2024 Li et al. 22024b), ensures that our data quality is not constrained by the limitations of existing labeling VLMs. In this way, we create one of the first large-scale egocentric QA datasets, consisting of over 7 million egocentric QA samples that span video lengths from seconds to over an hour. This dataset enables the training of models to recognize and retain visual details from egocentric videos.
2

---

Published as a conference paper at ICLR 2025
Key Frames
Key Frames
What activity did repeatedly
Which hand did the place on his chest?
do with the cal?
A: I combed the cat's for with comb..
A: The man placed both hands on chest.
Video Narrations
Video Clip "I opens the rice cooker."
What did do with pear aft slicing it!
How man adjust camern his head?
Clip
A: I moved the pear on the tray the knifé my right
Video Clip 3: "I closes the rics cooker.
A: The man adjusted camera hands.
LLM
Prompe
What did
the
What object did rinse lap'
Egucentric QA
wooden plank?
Video: [Clip 1, Clip 2, Clip
A: I used the tape rule to uncasure wooden plank.
A: rinsed spoon the tap.
Question:
leave the rioe
open aller using itt"
"No, closed afler stiming the rice."
cooker
Which hand did furm tup initially! A: used my left hand.
Which hand did use to pick nother fabric from A: used my left hand.
Key Frame: 3
Figure 2: "Narration to Egocentric QA" data engine. Given a sequence of narrations, instruct language model (GPT-4o) to generate egocentric understanding-rche related questions and answers, along with identifying the key frames necessary to answer those questions.
video
(ii Benchmark. To evaluate the MLLMs' performance in understanding and memorizing visual details from egocentric videos, propose the EgoMemoria benchmark. This challenging bench- mark includes 7,026 multiple-choice questions for 629 egocentric videos ranging from 30 seconds to 1 hour. In the experiments EgoMemoria, we further investigate the impact of inevitable language biases across different models during evaluation and introduce a debiased metric to more accurately assess the models'
egocentric understanding capabilities.
(iid) Model. For our MM-Ego model, we develop progressive approach to handle egocentric videos by introducing Memory Pointer Prompting method. It consists of two steps: "global glimpse" and "fallback". In the global glimpse step. we extract compressed frame-level visual em- beddings from the entire video get global understanding. Then, we employ a memory pointer embedding, designed to examine all compressed frame-level visual embeddings along with the ques- tion embeddings, to aid in identifying key visual embeddings in question-aware manner. In the following fallback step, the selected key visual embeddings, in a used as final input to the LLM for processing and generation. This approach allows us to achieve global understanding of the entire video while also identifying and utilizing key visual information to answer questions related to visual details.
form, are then
2
METHOD
2.1 "NARRATION TO EGOCENTRIC QA" DATA ENGINE
As outlined in Section I high-quality egocentric QA pairs are lacking for training MLLM with egocentric video understanding ability. To address this gap, we develop an innovative "narration to egocentric QA" data engine that automatically generates episodic memory-related QA samples based on human-annotated video clip narrations from the Ego4D dataset (Grauman et al. 2022) without the need for additional manual annotations.
Our approach leverages over 3,000 hours of privacy-protected, de-identified egocentric videos accompanied by more than million high-quality, human-created narrations. These fine-grained language descriptions provide rich resource for generating QA pairs.
0.5-1 1-2 min
0.5-1 min
1-2 min
2-4
min 10-20 20-40 min
10-20
40-60 min
90-60
The workflow of the data engine is illustrated in Figure 2 By organizing sequential video clips (Clip Clip 2, .... Clip N) and their corre- sponding narrations [Narration I, Narration 2, ... Narration N in proper chronological order, we create comprehensive narration paragraphs that describe entire video sequences. We then employ a powerful text-only language model, i.e., GPT-4o, to generate diverse and confident QA pairs related episodic memory based these
MIL
(a) Before balancing
[a] After balancing
Figure 3: Video length distribution in our egocen- tric QA dataset.
3

---

Published as a conference paper at ICLR 2025
narration paragraphs. The language model is instructed attach the index of the narration sentence upon which each QA pair is based. This indexing allows us to map each QA pair back to the corre- sponding time frames in the original videos, enabling the extraction of key frame information crucial for subsequent model training.
Applying this data engine to the extensive Ego4D dataset allows to efficiently scale the creation of egocentric QA data. We partition the dataset into training and testing sets according to the official Ego4D episodic memory task. The egocentric QA dataset provides more than 7 million QA samples in 938K multi-turn conversations. The data encompasses videos of varying durations, ranging from 30 seconds hour, as illustrated Figure [ To ensure comprehensive coverage and prevent bias towards shorter videos, we balance the number of conversations across different video lengths in training. This one of the first large-scale egocentric QA datasets featuring videos of such extended ranges of duration.
Through these steps, our "narration to egocentric QA" data engine addresses the scarcity of large- scale, high-quality egocentric QA data for egocentric scenes, and sets solid foundation for building MM-Ego, a sophisticated egocentric MLLM, which we introduce in the following section.
2.2 MM-EGO MODEL
Our modeling goal is to develop an MLLM for handling egocentric videos, which are lengthy and rich in visual details. On the one hand, frame-level information is necessary to capture the full content of the video, as skipping frames during sampling could result significant loss of visual details. On the other hand, processing all visual tokens generated by the visual encoder is compu- tationally challenging for the transformer model. For instance, if each image is encoded into 729 visual embeddings (tokens). the total number of visual embeddings for a 300-frame video would be 218,700. However, most MLLMs are trained with a context length of less than 10,000 tokens (Li et al. (024a). Taking these factors into account, introduce the MM-Ego model, which is built for handling a large volume egocentric video frames while maintaining manageable computational costs within the transformer backhone. MM-Ego introduces innovative Memory Pointer Prompt- ing mechanism, which operates in two main steps: global glimpse and fallback. We will introduce the details of MM-Ego in the following sections.
2.2.1 VISUAL AND TEXTUAL EMBEDDING
Given input video and the question, the first step is to embed them into visual and textual em- beddings separately for later processing. We begin by uniformly sampling the video into up to N frames, where N can be in the range of hundreds. Then, we extract per-frame visual feature maps from these frames using a robust vision encoder, SigLIP-so400m (Zhai el al. (023). Following the method outlined by [i et a (204a), we apply 2-layer MLP project the visual feature maps to the LLM embedding space and use average pooling to reduce the height and width of the visual feature maps by factor of two and flatten the height and width dimension, resulting in N relatively high-resolution visual embeddings {V' E '2''C, i E [1] where T is the embedding length and C' is the embedding dimension. For the textual embedding, since we use Qwen2 Yang et al. 2024) as the LLM, we use its tokenizer and embedding layer to transform the input text into textual embed- dings. For question q, we denote the corresponding textual question embedding as { EEq IR TT, TC
q E [1, @]} where Q is the total number of questions and T, is the embedding length of question q.
2.2.2 MEMORY POINTER PROMPTING
As processing all N high-resolution visual embeddings with the LLM is computationally diffi- cult, we propose to identify key visual embeddings in question-aware manner and only send those selected embeddings to the subsequent LLM. Inspired by previous works on Pointer Net- works (Vinyals et al.) 2015; Merity et al. 2016), we propose a Memory Pointer Prompting mech- anism, which is illustrated in Figure Memory Pointer Prompting consists of two steps during inference: global glimpse and fallback. In the global glimpse step, key visual embeddings are iden- tified from all frame-level embeddings, guided by the context of the question. During the subsequent fallback step. the important visual embeddings are selected, and their higher-resolution versions are provided to the L.L transformer backbone for further processing and language response generation.
4

---

Published as a conference paper at ICLR 2025
LLM Iaput Training
Questionn Did
Compressd
EmbeSdings
Ei
Question limbedddng Bypa
Encoder
Encoder
y
Encoder
Takenizer
Memory Pointer Finhedding
Manory Emnbalding
Inpur
LLM
Compress
Compress
Compress
Embed
Ground- Key Highh Res Visuil Enthedkings
E
B
P
Question Finheckding Reue
1.1M
Embedding Rane
(b)
E
Softmax
High Embhokkings
Inpui
Tup-k
1
Correlation
Selected Indices
Binory Cross Entrayy Loss
(8,),
Question Finhekkingg
Eque
Ground-Traid Key Frame Indices
Step Global Glimpse
Step 2. Fallback
(u)
Figure 4: (a) Overview of the proposed Memory Pointer Prompting mechanism. Its inference con- sists of two steps: (1) Global Glimpse: We concatenate the compressed visual embeddings from all frames, denoted as Evs for i [1, with the question embeddings Eque and the memory pointer embedding P1 This combined embedding sequence is then input into the LLM. From the last layer, we extract embeddings and compute the dot product between the memory pointer embedding and compressed visual embeddings generate the correlation scores. The indices of the frames with the top l scores are selected. During training, the correlation scores are supervised by ground truth key frame indices via binary cross-entropy loss. (2) Fallback: The high-resolution visual embeddings corresponding to the selected indices are fed into the L.LM along with the question embeddings for final processing and response generation. Illustration of LLM input sequence during training.
Global Glimpse Step. We begin by compressing the visual embeddings through average pooling along the embedding length dimension, resulting in set of compressed visual embeddings {E E RIXE; i E [1]. Next, we introduce learnable memory pointer prompt embedding P IR 1
duplicate it Q times, yielding (P' 1R1 i [1, Q)), and concatenate the embeddings follows:
E
E Vist E.Y Eque P').
Here Q 1 as MLLMs generate answers for only one question at time. In this way, the question embedding is followed by a pointer embedding, which will used to identify key visual embed- dings with knowledge of the question embedding. The entire embedding sequence is then fed into the LLM, from which we obtain the output embedding sequence the final layer:
E viss
vis: ..... E E'1 P'1I.
We extract and stack the processed visual embeddings (F E R [1, to obtain the matrix Evis E Nxx We conduct softmax dot product operation between Evis and p't.
s = Softmax (Ex P' E R
(1)
Here is correlation score vector indicating the correlation between the question and each frame.
Balancing Exploration and Exploitation. Our approach to selecting key visual embeddings par- allels the principles of Bayesian Optimization (Frazier, 2018). where the objective function is ex- pensive to evaluate. In such cases, it's important to balance exploration (sampling in areas where the uncertainty high) and exploitation (sampling areas where the surrogate model prediets high performance). However, relying solely the aforementioned Memory Pointer Prompting may lead to overemphasizing certain areas of interest, potentially undermining the exploration process. To mitigate this issue, we introduce perturbations into the score distribution by incorporating uniform sampling distribution. The probability vector uniform sampling can be written as:
if E linspace N, k))
(2)
otherwise.
Here is an explore-explee balancing parameter to adjust the probability distribution. We overlap the probability vector of uniform sampling and score matrix
S f- u. We then identify the t indices as the set {S é E [1, [1] In this way, we find the key visual embeddings in question-aware manner.
5

---

Published as a conference paper at ICLR 2025
Table 1: Distribution of videos and QA samples with different lengths.
Table 2: Distribution of correct options in MCQs.
Class
Short
Medium
Long
Sum
Minutes
10.20
20-40
Option
A
B
D
Videos
100
100
100
100
100
100
629
Counts
1776
1751
1770
1729
QAs
500
498
987
997
1715
1792
537
7026
Fallback Step. During inference, as shown in Figure with the set of indices {S;, [1, k]} for the selected visual embeddings, we now assemble the LLM imput sequence as follows:
V
V
V
quel
Sclected Top-k Visual Embesdings
As previously introduced, V.S v S. denote the selected top-k high-resolution visual em- beddings, which provide more visual details than the compressed visual embeddings. This new embedding sequence is fed into the LLM to generate the final language response. In summary, the proposed Memory Pointer Prompting approach allows us to consider the full scope of video information while filtering out redundant data in the LLM transformer, ensuring computational effi- ciency. The new input serves as the final input of the LLM to generate the language response given the visual and textual information.
Training Procedure. Given the novel design of MM-Ego, its training procedure is different from popular MLLMs (Liu al., 2023). Specifically, let the answer embedding for question q E [1, Q] be denoted as Eans, then the imput embedding sequence during the training process is represented as:
E que E
que E
que
ans!
Compressed Visual Emberkiings
Selected High-Res Visual Emneedinng
We also provide a simplified illustration (where Q 1) of the input embedding sequence structure during training in Figure 4 Here, we begin by inputting the compressed visual embeddings for all N frames, followed by the question embedding and memory pointer embedding. Next, integrate the h selected high-resolution visual embeddings (based on the ground-trute key frame labels), and finally, incorporate both the question and answer embeddings. Once the input sequence is prepared as outlined above, we can train MM-Ego similarly to traditional large language models. The compressed visual embeddings, question embedding, and memory pointer embeddings used as prefixes do not contribute to the language cross-entropy loss.
When training on samples from our curated egocentric QA dataset where there are ground-trutt truth key frame labels for each question, compute the correlation score vector in the global glimpse step, and supervise it using binary cross-entropy loss. For training samples that lack ground-truth key frame labels, omit the prefixes, which results in the traditional MLLM training process.
EXPERIMENTS
In the experiment section, we will first present new egocentric video understanding benchmark, specifically designed to assess episodic memory capabilities. Following this, we will perform com- prehensive experiments evaluate MM-Ego, utilizing both the newly introduced benchmark and existing public benchmarks.
3.1 EGOMEMORIA BENCHMARK
times hand
put
place
man
man
walk
do
move
To evaluate the performance of egocentric MLLMs, especially in terms of episodic mem- ory ability, we propose a new benchmark called EgoMemoria. Specifically, we generate memory-related questions and answers from human-annotated narrations in the validation set the Ego4D dataset. To ensure diversity, for each video only generate limited number of questions. We divide the videos into seven different length ranges: 0.5 min, I to 2 min, to min, 4 to 10 min, 10 to 20 min, 20 to 40 min, and 40 60 min. We aim balance the number of samples in different video lengths. The
ac retionperson table
use
drop
p
bic
ricke
Kheld
held
video floor Wood Noun
phone
interact Verb
Figure 5: The most frequently occurring verbs and nouns in EgoMemoria.

---

Published as a conference paper at ICLR 2025
Question I: Which hand did touch the handle of the drilling machine?
Choices: A: louched the handle of the drilling machine with my right hand. B: touched the handle of the drilling machine with both hands, C: touched the handle the drilling machine with my foot. touched the handle of the drilling machine with my left hand.
Prediction: D
Selected Frames
Label: D
Scores
Question Did use a ladder during process?
Choices: A: No, used step stool instead. No, stood a chair. Yes, climbed down the ladder. D: No, reached up without assistance.
Prediction: C
Label: C
Frames
Scores
Question 3: What did connect to the socket?
Choices: A: connected electric cleuner to the socket. B: connected digital clock to the socket. C: connected table lamp to the socket. D: connected coffee maker to the socket.
Prediction:A
Selected
Label:
Frames
Scores
Question 4: What color was the car that drove past last?
Choices: A: 1. The car that drove past last was red. B: 2. The car that drove past last was blue. C: The car that drove past last
was black. D: 3. The car that drove past last was white.
Prediction: A Label: C
Selected Frames
Scores
Question 5: Where did place the knife after peeling the zucchini?
Choices: A: I left the knife in the sink. B: put the knité back in the drawer. C: placed the knife the counter. D: dropped the knife on the cutting board.
Prediction:
Selected
Label: D
Frames
Figure 6: EgoMemoria QAs visualization and prediction analysis the global glimpse step. We find high consistency between the identified key frames and the questions, demonstrating the effective- ness of the proposed Memory Pointer Prompting method. The visualized correlation scores show distinct distributions for different questions given the same video, indicating its question-specifi nature. The indicates that the selected frames are relevant to the questions.
distribution of videos and corresponding question-anse answer pairs (QAs) for each category is shown in Table Furthermore, we group these video lengths into three broader categories: short (0.5 to 2 min), medium (2 to 20 min), and long (20 to 60 min). In total, collect 629 videos with 7,026 questions. The most frequently occurring verbs and nouns in the questions are visualized in Figure(5]
Since free-form answers typically evaluated using closed-source LLM judge, the evalu- ation can be inconsistent and subject to significant variance, especially due to model version up- dates. To ensure free-form answers into multiple-choice questions (MCQs), which helps reduce score instability. In practice, based on the free-form answer, we instruct ChatGPT generate three additional choices that are plausible but incorrect, considering the original question and answer. We then randomize the order of these choices achieve a uniform distribution of correct options, as shown in Table to minimize bias in option placement. We visualize some randomly sampled examples in Figure
reliable, standardized, and consistent performance evaluation,
convert the
3.2 EXPERIMENTAL SETUP
Training Data. We employ a joint image-video supervised fine-tuning (SFT) strategy. To enhance the model's capability in understanding broader range of visual data, combine our egocentric QA dataset with a variety of multimodal datasets. We curate an SFT dataset mixture consisting of our egocentric QA dataset, Ego4D narration dataset (Grauman et al. 2022 LLaVA-NeXT SFT

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison and language bias analysis of different models on the EgoMemo- ria benchmark. Our MM-Ego model demonstrates the best performance both before and after ex- cluding the language bias of different models.
LLaVA-OV eLl
2024a
Ego SFT
MM-Ego
Method
Short
Medium
Long
Avg
Short
Medium
Long
Avg
Short
Medium
Long
Avg
Original
70.24
64.94
61.19
65.45
79.06
76.34
73.51
76.30
79,96
79,64
79.09
79.56
Exclude LLaVA-OV OV Bias
56.44
49.64
44.83
50.30
66.37
64.15
60.03
63.52
71.97
70.68
68.15
70.26
Exclude Ego SPT Bias
56.75
49.27
45.21
50.08
61.73
59.59
54.50
58.61
67.70
66.33
63.89
65.97
Exclude MM MM-E
47.41
35.22
41.58
50.60
46.39
40.38
49.80
49.11
43.81
47.58
Mcan Debinsed Accuracy (MDA)
53.20
47.01
41.76
47.32
59.56
56.71
51.64
55.97
63.16
62.04
58.62
61.27
collection (including ChartQA (Masry et al.
2022), A12D (Hippala et al.
2021), Voc (Mathew
ict al. 2021). DVQA (Kafle et al. 2018), COCO (Lin et al. 201 10. ShareGPT4V (Chen et al.)
2023a), synthdog-en (Kim et 2021 SPareGPP-440 (Chen al. 2023b), ALLaVA instruct (Chen et al.
2024a). ShareGPT4Videoo CChen et al. 2024b), sherlock (Hessel et al. 20022) ScienceQA (Lu
al.
2022), NEXT-QA (iiao et al. 2021), and ActivityNet-QAA (Yu et al. 2019).
Implementation Details. The model is trained for epoch with base learning rate of x 10-5, uu-- ing a cosine scheduler. The batch size is set to 128. We sample a maximum of 300 frames (N 300) and select 32 visual embeddings in the proposed memory pointer prompting mechanism. By default, we set the explore-exploit balancing parameter a 0.1. Greedy decoding used in generation.
Pretrained Models. Our MM-Ego model is initialized from LLaVA-OV 7B (Li et al. 2024a). a ttate-of-theeart MLLM known for its good performance on general multimodal understanding tasks. Following the same architecture, we use the SigLip-so4000. ViT (Zhai et al.) 2023) as the visual encoder for embedding video frames and Qwen2-7B (Yang et 2024)) as the LLM architecture
3.3
MAIN RESULTS
We first conduct experiments on our EgoMemoria benchmark, primarily comparing three models: LLVVA-OV (Li al. (0024)), its fine-tuned version using our MM-Fgo SFT data mixture (referred to as "Ego SFT"), and our MM-Ego model, which incorporates the proposed Memory Pointer Prompt- ing mentioned in Section 2.2.2 We show the EgoMemoria accuracy in the first row of Table 3 We observe a significant improvement in the model's performance on egocentric QAs after training on our MM-Ego data mixture, attributed to the rich egocentric knowledge provided by our curated ego- centric QA training data. Moreover, leveraging the MM-Ego model architecture further enhances performance, thanks to the effective Memory Pointer Prompting mechanism.
However, we notice that the original overall performance metrics are higher than anticipated, rais- ing curiosity about the extent to which language bias contributes to the models" accuracy. To an- swer this question, we conduct additional experiments aimed at eliminating these language biases. Specifically, we test the three model variants on the EgoMemoria benchmark without any visual inputs, identifying questions that could be correctly answered without videos as "language-biar questions". Then, we evaluate the models' performance language-biases biased questions. For fairness, we apply this debiasing process across all three models so that they are evaluated the same sets of data. We calculate the mean accuracy of the debiased variants, referred to as the "Mean Debiased Accuracy (MDA)". The results are presented in Table 3
the subset of the benchmark without
As expected, after removing the language-biased questions, the accuracy of all three models drops significantly to a more reasonable level. The performance decline is notably more pronounced in the "Medium" and "Long" classes compared the "Short" class. For example, the average accuracy of LLaVA-OV across the three classes (short, medium, and long) drops from 65,45 to 47.32. The decrease in the "Short" class is 17.04, in the "Medium" class is 17.93, and in the "Long" class is 19.43. Despite this, we still observe improvements in MDA after training with SFT data generated by our MM-Fgo data engine (+8.65) and applying our Memory Pointer Prompting method (+13.95). These results demonstrate the effectiveness of our approach even after considering language bias.
To better understand the capability of MM-Ego, we compare its performance with state-of-the-art video MLLMs on EgoMemoria and prevalent large-scale video QA benchmarks, including the long egocentric video understanding benchmark EgoSchema (Mangalam et al., 2023) and the Internet- video-basee long-video understanding benchmark Video- MME (Fu et al.) 2024). The results shown in Table 4 On EgoMemoria, GPT-4o is evaluated using 32 uniformly sampled frames from the videos, while other models follow their respective official inference settings. The MDA on
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparison with state-of-the-an video MLLMs. MM-Ego shows strong performance on egocentric understanding and competitive performance on Internet video understanding.
EgoMemoria (MDA)
EgoSchema
Video-MME (w/o subs)
Method
Short
Medium
Long
Avg
Shoet
Medium
Long
GPTT4o
64.31
59.47
$7.65
60.48
72.2
80.00
70.30
65.30
71.90
MANNAANeXT--AA 7B-DPO
Zhung et al.
2024b
30.38
25.95
21.49
25.94
LLaVA-NeXT- owwn (Zhang et
2024b
43,78
33.76
31.04
36.
60.85
60,20
LLaVA-OV 7B ct 2024g)
53.20
47.01
41.76
47.32
60.10
69.30
56.00
49.40
58.30
MM-Ego Ego (ours)
63.16
62.04
58.62
61.27
69.03
67.60
55.70
47.80
57.00
Table 5: MDA on EgoMemoria when inferring with different numbers of frames. Our MM-Ego
model shows a smaller relative drop on average when decreasing the number of sampled frames.
Short
Medium
Long
Arg
Framus
LLaVA-OV
Ego SFT
MM-Egu
LLaVA-OV
Ego SFT
MMM-Ego
LLaVA-OV
Ego SFT
MM-Egu
LLaVA-OV
Egu SFT
MM-Ego
53,20
59.56
63.16
17.01
56.71
62.01
41.76
51.64
58.62
17.32
55.97
61.27
46.37
60.81
60.93
50.76
59.59
62.22
44.82
54.55
58.23
39.41
49.11
55.19
44.99
54.42
58.55
50.43
55.36
62.30
12.51
52.08
58.14
38.88
48.40
51.65
43.95
$1.95
58.16
DIIT
5.20%
1.36%
9.49%
8.16%
581%
6.80%
6.36%
6.77%
7.12%
7.19%
EgoMemoria is computed using the debiased subsets used in Table 3 Notably, MM-Ego exhibits the highest performance on EgoMemoria, particularly in the 'Medium' and 'Long' classes. On the EgoSchema benchmark, our model achieves substantial performance gain of +8.18 over the previous state-of-the-art open-source model effectiveness of both our data and model design for egocentric understanding. Additionally, on the challenging Internet video understanding Video-MME benchmark, our model is on par with the leading model of similar parameter size although our data mixture is less diverse compared with (Li) et al. (024a). These results showcase MM-Ego's 's capability in egocentric video understanding while preserving its general video comprehension abilities.
underscoring the
3.4 MODEL ANALYSIS
60.5
80.0
Quantitative Analysis of Different Numbers of Frames. To evaluate the influence of sampling different numbers of frames for different models, we calculate the mean debiased accu- racy (MDA) in Table The relative performance drop from sam- pling 32 frames to sampling 4 frames is also calculated. As ex- pected, all models exhibit decrease in performance with fewer sampled frames. Notably, MM-Ego exhibits a smaller average performance drop when the number of frames is reduced due to its ability to identify key frames given lower computational bud- get. The relative performance drop in the short category con- siderably smaller compared to the medium and long categories, likely because shorter videos require fewer frames to comprehend.
$0.0
55.5
38.0
0.00
0.35
5.30
0.15
0.20
Apha
Figure 7: MDA scores with dif- ferent a values for explore-and- exploit balancing.
Qualitative Analysis of Memory Pointer Prompting. In Figure we present qualitative analysis of the accuracy of Memory Pointer Prompting on EgoMemoria. We randomly select samples and visualize the key frames identified by the global glimpse step in Memory Pointer Prompting. The results show a strong alignment between the questions and the selected frames. In failure cases, we observe that the issues are often due to the ambiguity of the questions, causing the model to struggle with accurately localizing the key visual embeddings. Furthermore, the visualized correlation scores during the global glimpse step show distinct patterns across various videos and questions, confirming its effectiveness in selecting key visual embeddings tailored to the specific questions.
Quantitative Analysis of Expllore-Exploit Balancing Parameter As discussed in Section 2.2.2 we design an explore-exploit balancing parameter a to fuse the uniform distribution and the sam- pling probability computed by Memory Pointer Prompting. We illustrate MM-Ego's performance with varying values of a in Figure 7 The results show that a = 0.1 achieves the best performance, while larger or smaller values of a tend either over-explore over-exploit.
Conversation Examples by MM-Ego. In Figure 3 we show real-world demo of MM-Ego, where the input video is 2-minute long egocentric video captured by camera an off-the-shelf wearable

---

Published as a conference paper at ICLR 2025
Question I: Where did I leave waller?
minutes video captured by wearable device
MM-Ego: I left the wallet on the table. (Correct!) Question Did see anything can play with?
MM-Ego: Yes I have guitar at comer. (Correct!)
Question How many times Interact with remoter?
MM-Ego: interacted with the remoter three times. (Correct))
Figure 8: Real-world conversation examples generated by MM-Ego. The input 2-minute long egocentric video recorded using camera on an off-the-shelf wearable device. MM-Ego can accu- rately identify key visual details and provide correct answers the user's memory-related questions.
device (this video is not used in our dataset). MM-Ego is able to correctly answer the episodic memory-related questions given the egocentric video, despite the difference in data domain.
4 RELATED WORK
Multimodal Large Language Models. Recent advancements in Large Language Models (Ope-
nAl 2023: Touvron et al., 2023) have sparked significant interest in developing Multimodal Large Language Models (MLLMs) that combine the language understanding capabilities of LLMs with
multi-modal perception abilities (Alayrac et al. 2022 Dai et al. 2023 Zhu et al. 2023 McKinzie
ctt al. 2024). For video-based MLLMs, most works follow a structure akin to image-based MLLMs.
To handle the large volume of video frames, some methods reduce the number of frames (Zhang ctt al. 2023a; Wang et al.] 2024b; Maaz et al.] 20244] Xu et al.] 2024). which results the loss of many visual details. Others extend the LLMS' context length by employing parallel techniques (Xue et al. 2024), but this often leads to low training efficiency. Unlike these approaches, our method preserves global awareness of the entire video, allows for attention to visual details, and is efficiently trainable.
Egocentric Video Understanding. While the growing field of egocentric video understanding is still in its infancy, there have been many influential works. For comprehensive overview of ego- centric vision please refer to Plizzari et al. (2024). On the data/benchmmrkk side, representative works include Ego4D (Grauman al.) 2022), Ego-Exo4D (Grauman et al., 2024), and EPIC-KITCHENS, 100 (Damen et al. 2018). When also considering language, prior work on egocentric video- languae benchmarks include QaEgo4D (Bärmann & Waibel 2022) and EgoSchema (Mangalam et al. 2023). ) For understanding long egocentric videos, prior modeling efforts include GroundVQA (Di & Xie 2024] Eicode-Store-Retne
(Shen et al.] (023). and R-VLM (Xu al., 2023) However, most previous works focus on classic video understanding tasks such as activity recognition and temporal grounding, and hence they do not involve a large language model for complex understanding. In contrast, we propose to develop an MLLM to tackle comprehensive egocentric video understanding.
CONCLUSION
In this paper, we make three key contributions towards the development of egocentric foundation models: the creation of large-scale egocentric QA training dataset, the introduction of novel model designed for effective long egocentric video comprehension, and the establishment of the EgoMemoria benchmark for assessing models' ability to capture visual details from egocentric videos. We hope that these efforts will benefit further research on egocentric MI.MMs.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our proposed method does not involve the creation or introduction of any new video content. All generated data is derived from publicly available, crivacy-protecter datasets (Grauman al. 2022). The data is intended exclusively for academic research purposes and will not used for any com- mercial applications. We have adhered to ethical standards by ensuring that private or sensitive data has been used or compromised.
REPRODUCIBILITY STATEMENT
We provide detailed explanation of the data synthesis process our data engine in Section We also elaborate on our model design in Section 2.2.2 Additionally, we outline the implementation details, including the training hyperparameters in Section 3.2
ACKNOWLEDGMENT
This research is supported in part by the Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321 and SAIL Research Project.
REFERENCES
Cean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Leonard Bärmann and Alex Waibel. Where did leave my keys?-ppisaiic--mmory--bbee question answering egocentric videos. In CVPR- W, 2022.
Minjie Cai, Kris M Kitani, and Yoichi Sato. Understanding hand-object manipulation with grasp types and object attributes. Robotics: Science and Systems, volume 3, 2016.
Alejandro Cartas, Juan Marín, Petia Radeva, and Mariella Dimiccoli. Recognizing activities of daily living from egocentric images. In Pattern Recognition and Image Analysis, 2017.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. arXiv, 2024.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang. Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gptav-synthesized data for a lite vision-language model. Xrviv, 2024a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv, 2023a.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang- Sharegpi4video: Improving video understanding and generation with better captions. arXiv, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv, 2023b.
Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first-person perspective thinking capability of vasion-language models. In CVPR, 2024.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose lingun-language models with instruction tuning. arXiv, 2023.
11

---

Published as a conference paper at ICLR 2025
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Fumari,, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic- spic-kitchens dataset. In ECCV, 2018.
Shangzhe Di and Weidi Xie. Grounded question-answan in long egocentric videos. In CVPR, 2024.
Peter Frazier. A tutorial on bayesian optimization. arXiv, 2018.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv, 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari. Kiran Somasundaram. Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Cran- dall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
Kristen Grauman, Andrew Westhury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.
Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The Abduction of Sherlock Holmes: Dataset for Visual Abduc- tive Reasoning. In ECCV, 2022.
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Sera- fina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661- 688, 2021.
Shaul Hochstein and Merav Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36(5):791-804, 2002.
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang. Lu Dong, Yali Wang, Limin Wang, et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In CVPR, 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In CVPR, 2018.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without ocr. arXiv, 2021.
Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In CVPR, 2012.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018.
12

---

Published as a conference paper at ICLR 2025
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv, 2024a.
Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In CVPR, 2021.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024b.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-We Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ACI., 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. In NeurlPS Datasets and Benchmarks Track, 2023.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa document images. In Proc. WACV, 2021.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv, 2016.
OpenAl. ChatGPT: Optimizing language models for dialogue. httts::/opeeai.ccm hwm.p///oi..c./m/ooo..o..o chatgpt] 2023. Accessed: 2023.
Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Gio- vanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocen- tric vision. IJCV, pp. 2024.
Junxiao Shen, John Dudley, and Per Ola Kristensson. Encode-store-retriive: Enhancing memory augmentation through language-encoded egocentric perception. arXiv, 2023.
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. In arXiv, 2018.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv, 2023.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. NewrIPS, 28, 2015.
13

---

Published as a conference paper at ICLR 2025
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang. Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv, 2024a.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv, 2024b.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv, 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, 2021.
Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-based video language model for efficient long video question answering. arXiv, 2023.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang. and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv, 2024.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-contex, visual language models for long videos. arXiv, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv, 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Semir Zeki. Area v5 a microcosm of the visual brain. Frontiers in integrative neuroscience, 9:21, 2015.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunne audio-visual language model for video understanding. arXiv, 2023a. URL http:://arrxivvorr/bb////333
02858
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen- Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile ineestion-nswerrin for long-form movie understand- ing. arXiv, 2023b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv, 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b, URL https:// /ttpp/////aaaaai....m
video/
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv, 2023.
14

---

Published as a conference paper at ICLR 2025
A MORE ANALYSIS OF MEMORY POINTER PROMPTING
To further assess the effectiveness MM-Ego the proposed Memory Pointer Prompting meeh- anism, we present additional visual results of key frame identification during the global glimpse step in Figure MM-Ego demonstrates the capability to extract relevant visual information from large set of frames based on the given questions.
Question Where did walk towards with hose my hands?
Choices: walked backward fowards a walked sideways towards wooden walked towards brick pathway. D: walked forward towards iron fènce.
Prediction: Selested I.bell:
Frames
Question 2: What the color of the I tried remove from the wood?
Choices: A: The color of the tape was bluc. B: The color of the tape was red. C: The color of the my was green. D: The color
tape was yellow.
Prediction: Label:
Selested Framess
Question 3: Which hand did use to pick the frying pan from the boot of the pickup truck?
Choices: picked a frying pan from the boot of the pickup truck with my right hand. B: 1. picked a frying pan from the the pickup picked fiying pan from
my hand. picked fryingpan
pickup truck with hands. of the pickup truck using cloth my hand.
Prediction: A Label:
Selected Framess
Question What did pass to my left hand?
Choices: passed cup my left B: passed plate my left hand. passed book to my hand. passed the zemote my left hand.
Prediction: B Label:
Selected Frames
Question S: What action did take with the frying at the end?
Choices: A: moved the frying pan on cooker with my left and then stirred the content with chopsticks. B: placed the frying pan sink washed using sponge dish soap. transferred the frying pan the dining table and served food plates. hung liying pan on the rack and wiped the clean.
Prediction: A Label:
Selected
Framas
Question 6: How did I add spice the frying pan the first time?
Choices: A: grabbed handful of spice and sprinkled it over the frying B: shook the spice container directly ahove the frying pan. C: measured the spice with a teaspoon and added it to the frying pan. D: scooped out some spice with the spoon and poured it in the frying pan.
Prediction:
Selected
Label: D
Figure 9: More key frame identification results of the global glimpse step on EgoMemoria. We find high relevance between the identified key frames and the questions, demonstrating the effectiveness of the proposed Memory Pointer Prompting method. The / indicates that selected frames are relevant to the questions.
15

---

Published as a conference paper at ICLR 2025
B MORE DISCUSSION ON RELATED WORKS
Egocentric QA Data Generation After finishing the project, we find that the generation pro- cess in MM-Ego data engine shares some similar processes with the recently published LLaMA- VID Li al. 2024b) and GroundVQA (Di & Xie 2024). LLaMA-VID utilizes movie synopses to prompt LLMs to produce movie summaries and plot-related QA pairs. GroundVQA generates short-term (around 8 minutes) episodic memory QA from video narrations, but the goal and im- plementation details are different. MM-Fgo collects and processes videos with significantly more diverse video lengths from 30 seconds hour. The scale of our produced egocentric QA dataset is also significantly larger (7M vs. 303K).
Long Video Understanding and Egocentric Understanding Evaluation Benchmarks In recent years, there have been some pioneering benchmarks for assessing the performance of multimodal models in understanding long videos (Lei et al.. 2018) [Song et al.] [023] Zhang et al.] 2023b Fu
et al., 2024; Wang et al. 2024a Wu et al.) 2024). Since the content of egocentric scenes differs
from that of YouTube videos or movies, researchers have proposed specialized datasets for ego-
centric scene understanding. There are benchmarks for egocentric images (Cheng et al., (0224) and
videos (Mangalam et al. 20233 Di & Xie 2024; Bärmann & Waibel (022). QAEgo4D (Bärmann & Waibel 2022) benchmark assesses shorter-term video understanding (around 8 minutes) with considerably smaller dataset (1,850 questions across 166 videos). EgoSchema (Mangalam et al, 2023) has more video clips, yet the video lengths are still relatively short. Concurrent with work, VourVideo (Chandrasegaran et al. 2024) introduces an important egocentric QA benchmark consisting of 121,976 QA samples across 500 videos, ranging in length from 20 to 120 minutes. Our EgoMemoria benchmark provides 7,026 QA samples for 629 videos and encompasses a wide range of video lengths, spanning from 30 seconds benchmark significantly more challenging and closer to real-world egocentric video use cases. Fur- thermore, we contribute a large-scale egocentric QA training dataset with more than 7 million QA samples for 8,933 egocentric videos, which enables further research on training more powerful ego- centric video understanding models. We show the statistics of these datasets Table 6 Even when compared with other general/movie long video understanding benchmarks, the total numbers of QA samples and video counts in our Egomemoria benchmark are still significant.
hour. The diverse video lengths make the
Benchmark
Videos
QAs
Video Length Distribution
Data Type
TVQA-test (Lei al
2018
1,889
7,623
0 min
TV shows
MovieChat- I-ttest Song
2023
100
1,950
6 min
Movie
MoVQA Zhang al.
2023b
20
21,953
7,5 - 120 min
Movic
Video-MME (uu ct
2024
900
2,700
0 60 min
Intemet
L.Veench Wang et al.
2024
103
1,549
30 140 min
Internet
Long VideoBench (Wu
2024
3,763
6,678
0 60 min
Mixed
EgnSchema Mangalam al.
2023
5,063
5,063
0.5 min
Egocentric
QAEgo-ID-test (Bärmann Waibel
166
1,850
0
Egocentric
GrounVQAA-ies (QAEgoo&D close)
Di
2024
148
500
Egucentric
HourVidoo (Concurrent Work) Chandrascgaran ot al.
2024
500
12,976
120 min
Egocentric
MM Ego Training
8,933
7M
0.5 60 min
Egocentric
MM Ego Evaluation (EgoMemoria Benchmark)
629
7,026
0.5 60 min
Egocentric
Table 6: Comparison with exiting long video understanding datasets.
Limitation and Future Work While MM-Ego demonstrates strong ability in egocentric under- standing, there is still room for further improvement. On the data and benchmark side, we can introduce more diverse egocentric understanding corpus (Grauman et al. [2024] Huang al. 2024)_ For the model itself, we plan to enhance its capacity to process a larger number of frames, such as at the order of thousands, to better handle longer or even always-on egocentric videos.
C FINE-TUNING EXCLUSIVELY ON EGOCENTRIC QA DATA
To evaluate model performance when fine-tuning exclusively egocentric QA data, conduct an ablation study, with the results presented in Table For both LLaVA-OV (Li et al. (204a)) and MM-Fgo, consider two variants: one fine-tuned on our comprehensive data mixture and the
16

---

Published as a conference paper at ICLR 2025
other trained solely egocentric QA data. The results demonstrate further performance improve- ments on the EgoMemoria benchmark. However, it is important to note that such domain -peeifi fine-tuning largely restricts the models' capacity for general video understanding. On the other hand, we observe that MM-Fgo still achieves significantly better performance egocentric video understanding, attributed the learning of the memory pointer prompting mechanism.
Name
Short
Medium
Long
Avg
LLaVA-OV (Data Mixture)
56.71
LLaVA-OV (Egucentric Only)
58.13
Ego (Dala Mixture)
63.16
62.04
58.62
61.27
MM Ego (Egocentric Data Only)
65.41
64.89
61.05
63.78
Table 7: Performance comparison of different models on EgoMemoria (MDA) with fine-tuning
exclusively on egocentric QA data.
D ANALYSIS OF USING LANGUAGE MODEL IN DATA ENGINE
The motivation for using language model to convert egocentric video captions into egocentric QA conversations is to address the "chicken-or-egg dilemma". If we rely on a vision-language model (VLM) to generate egocentric QA pairs, the quality of the data is inherently limited by the egocentric understanding capabilities of the labeling VLM. Consequently, downstream VLMs trained on this synthetic data cannot outperform the labeling VLM. This creates the chicken-or-egg problem: do we have a strong egocentric VLM first, or do we have good egocentric QA data first?
To address this dilemma, model to generate QA samples. This approach circumvents the hallucination and inaccuracy issues often associated with long video understanding, which is a very challenging task by itself. Since we have access to high-quality, fine-grained, and densely annotated video narrations created by human labelers in Ego4D, the essential visual information has already been effectively translated into the narrations.
poiraation-t--goccentie QA" data engine leverages language-only
We conduct a preliminary experiment of using image frames augmented GPT-4o to produce egocen- tric QA. We uniformly sample 32 frames with resolution of 336x448 from a 30 seconds egocentric video.
Input Narrations: VID VID_NARRATION = ["l close the tap.". "I press the buttons on the rice cooker.", "I open the rice cooker.". "I stir the rice.", ", close the rice cooker.", "I press the buttons on the rice cooker.)]
Text Prompts: Please design at most questions and answers about sel irqqustiooing my memory based on video. For example, ask questions about what I have done, what objects did I interact, where did I go, what did other people do, the locations, colors, and shapes of the objects in the scenes if mentioned, based solely the video narrations NVID_NARRATION The answers should be in perspective or objective description of the objects. Remember this rule: special word 'I', 'c', or 'c' represent me (first-person perspective), special word 'x', 'y', 'Y', 'z', 'z', 'o', 'O', 'x', etc. represent somebody else, this is very important. Ask very diverse questions that you are sure about the answers, and the questions cannot be answered without watching the video, which means no common sense questions. Don't generate anything if you do not have enough information. Attach index of the sentence that answer is based upon. Directly reply by strictly following this example: 'QUESTION: Where did I leave my key? ANSWER: put it the table..0,2 DUE I close the door when left? ANSwER:Yes I did. QUESTION: What is the color of the cup? ANSWER:White. )'r
17

---

Published as a conference paper at ICLR 2025
We present the generated QAs by GPT-4o both with and without video input in Table and Table [ In this example, we observe that the generated QAs have similar quality whether videos included not. However, incorporating video into GPT-4o's input significantly increases the time and computational costs. Moreover, when applying VL.M to large volume of videos, it becomes challenging to verify if any visual hallucinations arise from the video understanding process. Therefore, we choose to use a mature pure language model in our data engine.
Generated Egocentric QAs:
Question
Answer
What did do to the tap in the video?
I closed the tap.
How did interact with the rice cooker buttons?
I pressed the buttons the rice cooker twice.
What did use to stir the rice?
I used spoon to stir the rice.
Did leave the rice cooker open after stirring the rice?
No, closed the rice cooker.
At what points in the video was the rice cooker interacted with?
The rice cooker was opened, stirred, and closed.
Table 8: GPT-4o generated egocentric QAs with video input.
Question
Answer
What action did I perform the tap?
I closed it.
What did do after pressing the buttons on the rice cooker the first time?
opened the rice cooker.
What did use to stir the rice?
stirred the rice.
How many times did press the buttons the rice cooker?
Twice.
What sequence of actions involved the rice cooker?
pressed the buttons, opened it, stireed the rice, closed it, and pressed the buttons again.
Table 9: Generated egocentric QAs without video input.
E MM-EGO'S CONNECTION WITH COGNITIVE NEUROSCIENCE
Our proposed "Memory Pointer Prompting" two-step mechanism inspired by the way human naturally process videos. When answering specific question about long video, typically start by quickly scanning the entire video to identify frames relevant to the question, which is similar "global glimpse" step. We then closely examine those frames find the answers, which resembles our "fallback" step.
In the study of human visual perception system, researchers identify two distinct pathways for pro- cessing visual information in the brain: the magnocellular and parvocellular pathways Zeki 2015 Hochstein Ahissar 2002). Our "global glimpse" step mirrors the functionality of the magnocellu- lar pathway which is responsible for handling information about large, fast-moving objects. On the other hand, our "fallback" step aligns with the role of the parvocellular pathway which specializes in processing details of small, slow-moving objects.
F FUTURE DIRECTION ON PROCESSING LONGER VIDEOS
To further enhance MM-Ego's capability to handle even longer videos, we can adopt two strategics. First, leveraging aggressive parallelism techniques, such as sequential and tensor parallelism (Xue)
18

---

Published as a conference paper at ICLR 2025
et al. 2024), significantly extend the context length of the transformer model. This will extend the model's 's ability to do reasoning in more frames. Second, we can introduce a hierarchical structure to the compressed visual embeddings by further consolidating embeddings from multiple frames into a single representation. Then we can design multiple global glimpse steps, enabling the model to identify relevant frames in coarse-to-fine manner.
19


Reviews:
[{"review_id": "gJBrgEPr1U", "reviewer": "Reviewer_hocP", "summary": "The paper introduces MM-ego to process and understand long egocentric videos. It includes \"narrations to QA\" strategies for creating scalable training data. The paper also introduces a new benchmark called EgoMemoria to assess the ability of reasoning and memorizing visual details and evaluate the impact of language biases. The final contribution is the MM-Ego model which is based on a progressive memory pointing prompting consisting of global compressed features and fallback aka learnable memory pointers.", "questions": "What are your thoughts on enabling more number of frames into the reasoning pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\nMM-Ego data engine: augmenting Ego4D dataset with scalable QA is valuable\nEgo-memoria benchmark is a good contribution, so is the MM-Ego model\nthe assessment of the impact of language bias is useful, and also shows the value of the data engine", "weaknesses": "The the model struggles with long videos. Whereas, egovideos are known for always ON camera meaning the ability to process long / unlimited length video is utmost important. It's a general research question for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zM00ZS0Urn", "reviewer": "Reviewer_bd41", "summary": "This work takes a step towards building egocentric MLLMs by contributing a large-scale dataset, a challenging benchmark, and an innovative model architecture. The model, named MM-Ego, demonstrates strong performance on egocentric video understanding. It is designed to process and understand long egocentric videos by progressively understanding the video: first getting an overview, then focusing on specific details with particular questions in mind.  The paper introduces also an egocentric QA benchmark called EgoMemoria, which contains 7,026 questions for 629 videos of different lengths. It also introduces a de-biasing evaluation method to mitigate language bias in model evaluations.", "questions": "It would be great if the authors could answer my two points in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors curate a large-scale dataset with over 7 million QA samples for egocentric videos of varying lengths and introduce a QA benchmark called EgoMemoria, which contains 7,026 questions for 629 videos of different lengths. \n\n2. A specialized multimodal architecture is proposed, using a \"Memory Pointer Prompting\" mechanism. This includes a global glimpse step for overarching video understanding and a fallback step that uses key visual information to generate responses, enabling the model to comprehend extended video content more effectively.", "weaknesses": "1. The first claim of contribution, the “narration to egocentric QA” data pipeline, I believe, should not be emphasized as a major contribution. This type of generating QA from dense captions have been used in multiple previous works, from the non-LLM era (like TVQA) to the LLM era (like LLama-VID). I believe it is better to tone down this statement.\n2. The generated EgoMemoria Benchmark does not stands itself out of many long video understanding benchmarks. Even we narrow down to only egocentric videos, the GroundVQA dataset is also good to be compared and especially be used to test the MMEgo model. I would also recommend the authors to compare a long of long video datasets, providing more proofs that this benchmark is not so incremental.\n\nOverall, I think this paper is proposing a good model, while the benchmark side is relatively weak. I would recommend the authors to either tone down the benchmark in the paper to put more emphasize on this model, or improve the benchmark. I recommend the authors to consider using the datasets of EgoExo4D and EgoExoLearn, both of which also contain dense narrations of egocentric videos and should be very suitable for enriching your benchmark in terms of both size and diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGlu6zzV3h", "reviewer": "Reviewer_JQHv", "summary": "This paper introduces three aspects of a multimodal foundation model for egocentric video understanding. First, it proposes a data engine designed to generate egocentric QAs based on human-annotated narrations automatically. Specifically, it relies on a text-only language model (GPT4-o) to generate the QAs. Second, it presents a Memory Pointer Prompting method, which can help generate question-related responses by identifying key visual details. Third, it introduces a new benchmark called EgoMemoria and a new metric that effectively de-biases language data in the evaluation process.", "questions": "What is the performance of LLaVa-OV and MM-Ego when using only the EgoMemoria for training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The new set of annotations introduced in the proposed dataset does look practical and timely, considering that it is based on narrations annotated by humans instead of language models. The dataset should be of higher quality than other synthetic datasets using large language models as pseudo-labelers.\n- The proposed benchmark, EgoMemoria, with the suggested debiased metric, also looks interesting and well-defined. The QAs are distributed well for different video lengths, and the answers are uniformly distributed between multiple choices.\n- The proposed MM-Ego model with the “Memory Pointer Prompting” mechanism is straightforward and intuitive. It understands a video in a progressive way, first getting an overview and then focusing on details.\n- The paper is well-written and is easy to follow. It looks well-organized, with the required figures and tables placed throughout.", "weaknesses": "- [W1] The introduced data engine works only with human-annotated narrations. In other words, it is not trivial to scale up since it relies on human labor. The data engine looks like it is designed explicitly for the Ego4D dataset, which has human-annotated video clip narrations. I am not too sure if the data engine is a practical and valuable contribution.\n- [W2] The authors have not compared the introduced EgoMemoria with other datasets. Even if it has high-quality data thanks to the use of human annotations, it is hard to say 7k MCQs large-scale. The number of annotations in the test split would be even smaller. I believe the authors should make a table comparing the EgoMemoria with other egocentric datasets regarding the number of clips, annotations (QA pairs), and clip lengths.\n- [W3] The authors have not shown the results when using only the EgoMemoria for training since they use an SFT dataset mixture. It would also be better to include the results using only the EgoMemoria for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JODElQbfMD", "reviewer": "Reviewer_g472", "summary": "The contributions of this paper include three aspects. 1. It implemented an automatic annotation process, providing a high-quality, large-scale QA dataset for egocentric videos. 2. It constructed a challenging egocentric QA benchmark, consisting of 629 videos and 7026 questions and introduced a new metric to mitigate inevitable language biases in evaluated models. 3. It proposed a novel model structure, including the global glimpse step and fallback step. By fine-tuning, MM Ego was built and demonstrated excellent performance in egocentric video understanding.", "questions": "Please refer to \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- A new model structure is proposed, with the motivation of incrementally understanding videos: first, providing an overview of the entire video, then focusing on specific details, and keeping in mind particular questions.\n\n- This paper provides dialogue examples of MM Ego in real-world scenarios, offering a paradigm for understanding human instructions in real-world settings.\n\n- This work provides a large amount of Ego video data to the community through an automated annotation process and it is valuable to assess language biases in the LLM evaluation process.", "weaknesses": "- The authors suggest that in terms of benchmarking, existing egocentric video benchmarks either focus on shorter videos, such as EgoSchema and QaEgo4D or on internet video content, such as video MME. This results in a significant gap in egocentric video understanding benchmarks, but the data for the proposed benchmark EgoMemoria still comes from Ego4D. I do not feel that EgoMemoria is significantly different from the benchmarks mentioned earlier, and the authors need to clarify this point further.\n\n- Is it reasonable to only use GPT-4o in the automated annotation process? When generating QA pairs, I think that videos are equally important as dense captions, which will further ensure the quality of the QA pairs. The authors need to provide ablation experiments to validate that the existing method is more reasonable and ensures higher annotation quality.\n\n- To ensure reproducibility, the authors need to provide the prompt templates for GPT-4o and ChatGPT used in the automated annotation and evaluation processes.\n\n- The benchmark proposed in this work includes 629 videos. The authors classified them by length but did not provide the distribution of videos for each length category. This is crucial for the comprehensiveness and robustness of model evaluation. The authors should further elaborate on the distribution of video lengths.\n\n- In Table 4, MM Ego's performance on Video-MME is not as impressive as LLaVA OV. The authors should provide a more comprehensive analysis to explain the reasons for this discrepancy. Otherwise, I might conclude that the construction method of MM Ego sacrifices its ability to understand general videos in favor of enhancing ego understanding.\n\n- Language bias is inevitable in LLM, but whether it will be influenced by random responses in ego videos also needs to be verified through an ablation study.\n\n- The Global Glimpse Step and Fallback Step described by the author would be more complete and convincing if they could be correlated with mechanisms related to cognitive neuroscience and brain science.\n\n- Figure 5 may be better represented using a word cloud.\n\n- Please note that it is not GPT4-o, but GPT-4o.\n\nIf the author can address the questions above, I will improve my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)