VidEgoThink: Assessing Egocentric Video Understanding Capabilities for
  Embodied AI
Authors: name: Sijie Cheng; name: Kechen Fang; name: Yangyang Yu; name: Sicheng Zhou; name: Bohao Li; name: Ye Tian; name: Tingguang Li; name: Lei Han; name: Yang Liu
Published: 2024-10-15 14:08:53

Abstract: Recent advancements in Multi-modal Large Language Models (MLLMs) have opened
new avenues for applications in Embodied AI. Building on previous work,
EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating
egocentric video understanding capabilities. To bridge the gap between MLLMs
and low-level control in Embodied AI, we design four key interrelated tasks:
video question-answering, hierarchy planning, visual grounding and reward
modeling. To minimize manual annotation costs, we develop an automatic data
generation pipeline based on the Ego4D dataset, leveraging the prior knowledge
and multimodal capabilities of GPT-4o. Three human annotators then filter the
generated data to ensure diversity and quality, resulting in the VidEgoThink
benchmark. We conduct extensive experiments with three types of models:
API-based MLLMs, open-source image-based MLLMs, and open-source video-based
MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform
poorly across all tasks related to egocentric video understanding. These
findings suggest that foundation models still require significant advancements
to be effectively applied to first-person scenarios in Embodied AI. In
conclusion, VidEgoThink reflects a research trend towards employing MLLMs for
egocentric vision, akin to human capabilities, enabling active observation and
interaction in the complex real-world environments.