MM-Ego: Towards Building Egocentric Multimodal LLMs
Authors: name: Hanrong Ye; name: Haotian Zhang; name: Erik Daxberger; name: Lin Chen; name: Zongyu Lin; name: Yanghao Li; name: Bowen Zhang; name: Haoxuan You; name: Dan Xu; name: Zhe Gan; name: Jiasen Lu; name: Yinfei Yang
Published: 2024-10-09 17:59:59

Abstract: This research aims to comprehensively explore building a multimodal
foundation model for egocentric video understanding. To achieve this goal, we
work on three fronts. First, as there is a lack of QA data for egocentric video
understanding, we develop a data engine that efficiently generates 7M
high-quality QA samples for egocentric videos ranging from 30 seconds to one
hour long, based on human-annotated data. This is currently the largest
egocentric QA dataset. Second, we contribute a challenging egocentric QA
benchmark with 629 videos and 7,026 questions to evaluate the models' ability
in recognizing and memorizing visual details across videos of varying lengths.
We introduce a new de-biasing evaluation method to help mitigate the
unavoidable language bias present in the models being evaluated. Third, we
propose a specialized multimodal architecture featuring a novel "Memory Pointer
Prompting" mechanism. This design includes a global glimpse step to gain an
overarching understanding of the entire video and identify key visual
information, followed by a fallback step that utilizes the key visual
information to generate responses. This enables the model to more effectively
comprehend extended video content. With the data, benchmark, and model, we
successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
performance on egocentric video understanding.