Title: Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Language Models

Egocentric

Exocentric

Visual Question Answering
Abstract: Large vision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality, where a first-person (egocentric) view captured by head-mounted cameras serves as key input.
While this view offers fine-grained cues about user attention and hand-object interactions, its narrow field of view and lack of global context often lead to failures on spatially or contextually demanding queries.
To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views, providing complementary information such as global scene layout and object visibility to LVLMs.
We present E3VQA, the first benchmark for multi-view question answering with 4K high-quality question-answer pairs grounded in synchronized ego-exo image pairs.
Additionally, we propose M3CoT, a training-free prompting technique that constructs a unified scene representation by integrating scene graphs from three complementary perspectives.
M3CoT enables LVLMs to reason more effectively across views, yielding consistent performance gains (4.84\% for GPT-4o and 5.94\% for Gemini 2.0 Flash) over a recent CoT baseline.
Our extensive evaluation reveals key strengths and limitations of LVLMs in multi-view reasoning and highlights the value of leveraging both egocentric and exocentric inputs.
The dataset and source code are available at [https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding](https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis a benchmark and analysis paper, focusing on evaluating egocentric (Ego) and exocentric (Exo) reasoning in frontier VLMs, and introduces a dataset derived from the prior EgoExo4D work.\n\nThe paper introduces E3VQA, a dataset containing 4K pairs of questions and answers related to ego-exo images. Each question is crafted so that it is difficult to answer using only text or a single image; instead, integrating information from both views is essential to select the correct answer.\n\nThe authors also introduce a prompting technique called M3CoT, which uses three agent VLMs that focus on both the ego and exo views, respectively, to extract information from the input views, form scene graphs, and use majority voting to select the best answer.\n\nThe E3VQA benchmark demonstrates that frontier VLMs still struggle with cross-view reasoning between ego and exo perspectives. The M3CoT prompting technique improves these models' performance by about 5% on the benchmark. Overall, the paper highlights the limitations of current VLMs in multi-view ego-exo reasoning, analyzes the cases where these models struggle most, and proposes potential improvements.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is clearly written and effectively conveys that frontier VLMs have limitations in ego-exo multiview reasoning. The proposed prompting technique is simple yet effective, requiring no additional training.\n2. The E3VQA QA pairs are mostly auto-generated and filtered using LLMs and VLMs. The pipeline is clearly described and should be easy to scale as more GT Ego-Exo frames and annotations become available.\n3. The benchmark is filtered based on the consistency of model responses when given different access to the views (both, ego, exo, or only text). This process ensures that the questions included truly require reasoning over both Ego and Exo views, rather than allowing models to rely on shortcuts or general knowledge to answer.\n4. The simple yet effective M3CoT prompting technique improves Ego and Exo reasoning during inference, and its effectiveness is demonstrated across different VLM models.\n\n### Weaknesses\n\n1. As stated in the paper, the benchmark data is curated solely from EgoExo4D video frames and annotations, which limits the diversity of data sources and may restrict the generalizability of the results and findings.\n2. The paper does not sufficiently discuss the unique challenges posed by Ego-Exo view integration compared to Exo or Exo multiview scenarios.\n3. It is unclear whether the main challenge for frontier VLMs stems from the inherently unique egocentric viewpoint, or from the specific characteristics of the EgoExo4D ego view format (such as its FOV and the dark areas outside the viewpoint), which may differ significantly from the images typically seen during VLM training. If the egocentric images were calibrated and rectified to resemble more conventional photographs, would the observed difficulty for VLMs persist?\n4. The analysis section primarily focuses on the fairness of the benchmark but lacks a analysis of the typical failure patterns of current VLMs and a deeper investigation into why these models struggle with ego and exo settings. For instance, visualizations such as attention maps between ego and exo views could help illustrate how models interpret the relationship between perspectives and whether their focus aligns with the question. If not, identifying and analyzing these failure patterns would provide valuable insights into the limitations of VLMs in this context.\n\nOverall, the quality and clarity of the paper are good. However, due to the aforementioned limitations, I rate the significance of the paper as fair.\n\nQUESTIONS:\nThe paper is well written and convey the message clearly, I don't have questions regards the paper.\nThe questions I have is more about the focus to improve the significance of the paper, but may out reach of the scope of this paper:\n\n1. This is unclear that the VLMs performance drop in Ego views compared to Exo views, is not because the uniquce image format or charateristics of the EgoExo4D data. But is the narrowed view sight or context imformation in the ego view.\n2. Is the Ego-Exo has its unique challenges compared to Ego-Ego, Exo-Exo settigs? As the paper lacks the comparison between how Ego-Exo compared to Exo-Exo in the same timestamp views in EgoExo4D dataset.\n3. The analysis of VLM failures in Ego-Exo settings lacks significance. It remains unclear whether these failures stem from substantial visual differences in the appearance of the same object across views, or if powerful visual encoders still produce similar visual embeddings. Furthermore, it is not clear whether VLMs are establishing visual correspondences between ego and exo views, or if they are primarily relying on extracting language descriptions (as suggested by M3CoT) and matching correspondences in the textual space.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the importance of integrating first and third-person views for Vision-and-Language models. This is an incredibly relevant setup particularly for assisted living scenario where the personal assistant might have access to multiple cameras that are installed in a smart home. To tackle this problem, the authors automatically generate a VQA dataset from EgoExo4D. The output of the automatic data collection is then verified by four expert annotators, who were able to produce 4,000 high-quality examples. \n\nGiven this dataset, the authors complete an evaluation of both closed-source and open-weights models. Showcasing interesting results which highlight that GPT4o is the top-performing model overall with InternVL3-14B leading the way for open-weights models. The authors also present M3CoT, a prompting technique that uses scene graphs to prompt the model with different information that are used to help the model predict the right answer. This involves multiple steps of scene graph extraction and refinement. The authors evaluate this technique for Gemini and OpenAI models, showcasing how it helps improve performance over other prompting techniques.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is well-written and easy to follow. The narrative is clear, and the main methodology as well as the results are well described. \n2. High-quality data collection defined to derive the final dataset. This represents an important resource for the community and I hope that the authors will release both the dataset as well as their data generation procedure.\n3. Interesting prompting technique proposed however I am a bit skeptical of its ability to perform at scale (see weaknesses)\n4. Really comprehensive evaluation of several closed and open models. Additionally, the authors provide a detailed error analysis that I find really informative.\n\n### Weaknesses\n\n1. The paper does not feature a specific related work section. I understand that this paper represents a very novel contribution however I wonder whether it would be useful to include a description of the different egocentric question answering datasets such as OpenEQA, and describe how this dataset is different from others. This could be not only in terms of the different perspectives available but also in terms of the different linguistic properties and capabilities required by each question\n\n2. I think it would have been useful to also include some text-only baselines as well to understand whether there are inherent biases in the data. \n\n3. I really liked this paper because I think that the automated data collection pipeline is very interesting and is designed to ensure high-quality annotations. However, I don't find the scene graph prompting technique as a strong contribution of this paper. Specifically, I wonder how scalable this solution is, especially if we imagine a scenario where the model has to react in real-time because it has to process the video stream coming from some smart glasses (e.g., Rayban Glasses). Running multiple copies of the model for each query sounds extremely expensive. It would be good if the authors could provide more details regarding this.\n\nQUESTIONS:\nOverall, I really enjoyed the paper and I hope that the authors find my weaknesses informative to help them shape the next iteration of this paper. I'll report below the question I had which I hope the authors will help me improve my understanding of this paper:\n\n1. Who are the expert annotators? What was the procedure they followed to annotate the data?\n2. What is the computational costs of running your M3CoT compared to other state-of-the-art methods as well as compared to the baseline? I think it's important to be transparent about this, considering that you are running multiple copies of the same model. I understand that you applied this only to API models therefore you didn't consider the scenario where the user has to deploy this.\n3. Related to the previous point, why did you apply the M3CoT only to frontier models? Was there anything that prevented you from applying it to other models like QwenVL?\n4. I think it would be great if your table of results could always report the \"both\" setting because this will highlight a more realistic scenario which is where the model has to truly use both perspectives to answer the question\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces a new framework for comprehensive scene understanding in large vision-language models (LVLMs) by integrating both first-person (egocentric) and third-person (exocentric) visual perspectives. The authors develop E3VQA, a benchmark consisting of 4,000 high-quality question–answer pairs grounded in synchronized ego–exo image pairs, covering tasks such as action understanding, object recognition, spatial reasoning, and numerical estimation.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The paper tackles an under-explored but highly relevant problem—comprehensive scene understanding by integrating first-person and third-person views in LVLMs. This dual-view approach goes beyond typical single-view VQA and aligns well with real-world use cases in AR/VR and robotics.
2. The E3VQA dataset is carefully constructed and covers a diverse set of tasks.

# Weaknesses
1. The E3VQA benchmark is derived from the EgoExo4D dataset, which, while large and varied, may still be limited in terms of scene diversity, activities, and environmental conditions.
2. The evaluation is limited to multiple-choice VQA with synchronized image pairs, and the number of samples is small (~4k).
3. The dataset is generated using GPT4o on images, but only the val set, no training set and finetuing existing VLMs. I think the contribution is not enough.

QUESTIONS:
1. Why not use real images in Figure 1?
2. What is the difference between using the multi-view images compared with video data?
3. How to make sure the VLM can generate correct and scene graphs? Especially the presented image is quite complicated.

LIMITATIONS:
No.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nhis paper tackles the challenge of enabling large vision-language models (LVLMs) to jointly reason over first-person (egocentric) and third-person (exocentric) views. The authors introduce E3VQA, a new benchmark comprising 4K high-quality question–answer pairs grounded in synchronized ego–exo image pairs, covering tasks such as action understanding, attribute recognition, spatial reasoning, and counting. They also propose M3CoT, a training-free prompting method that constructs multi-perspective scene graphs (Ego&Exo, Ego→Exo, Exo→Ego), which are progressively unified to form a complete scene representation. Experiments on leading closed-source (GPT-4o, Gemini) and open-source LVLMs show that M3CoT consistently outperforms strong chain-of-thought baselines (e.g., CCoT) with up to 5.9% absolute accuracy improvement, especially in numerical reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1 the paper the well-writen and easy to follow.\n2 E3VQA fills an evaluation gap in video understanding by targeting egocentric–exocentric joint understanding with carefully curated multiple-choice QA pairs. Its design rigorously tests multi-view integration rather than single-view shortcuts.\n3: M3CoT enhances reasoning across perspectives without additional model training, and achieves consistent performance gains over state-of-the-art CoT prompting baselines, especially in spatial and numerical questions.\n\nWeaknesses\nThe approach relies on synchronized ego–exo image pairs, limiting its applicability to real-world settings where such pairing may be noisy, missing, or unavailable.\nM3CoT involves multiple scene graph generations and iterative refinement. The cost and stability of this prompting procedure across different LVLMs are not fully analyzed, and it may pose scalability concerns.\n\nQUESTIONS:\nSince M3CoT applies a full multi-perspective prompting pipeline to all questions, would it be possible to selectively activate only certain views or strategies based on question type or view relevance (e.g., skip full graph fusion for single-view solvable questions) to reduce inference overhead?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces E3VQA, a novel benchmark designed to evaluate large vision-language models (LVLMs) on multi-view question answering that combines egocentric and exocentric perspectives. Recognizing the complementary nature of both views, fine-grained hand-object interactions from ego and spatial context from exo. The authors argue that current LVLMs fail to reason effectively across both.\n\nThey additionally propose M3CoT, a training-free prompting technique that uses multi-perspective scene graphs and iteratively fuses them for improved reasoning. Extensive experiments on E3VQA show that M3CoT yields consistent and significant performance gains over recent prompting baselines across multiple LVLMs including GPT-4o and Gemini 2.0 Flash.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n1. Timely and Novel Benchmark: It is a first-of-its-kind dataset focusing on ego-exo reasoning, addressing a growing need in augmented reality, embodied AI, and multimodal understanding.\n2. The dataset includes 4K QA pairs evenly distributed across four categories (pose, object, numerical, spatial), with fine-grained filtering and expert human verification.\n3. The multi-agent, multi-perspective scene graph construction and refinement mechanism is original, cleverly designed, and provides demonstrable performance improvements.\n4. M3CoT significantly outperforms multiple prior CoT prompting methods across a range of LVLMs and question types. The performance analysis is thorough and includes ablations.\n5. The paper includes implementation details, error bars, dataset access, compute info, and a discussion of limitations and ethical concerns.\n\n### Weaknesses:\n1. While the E3VQA benchmark is impactful, its reliance on a single source dataset (EgoExo4D) may limit its diversity and generalization across real-world applications.\n2. Although M3CoT is \"training-free,\" its reliance on multiple inference passes with different prompt agents may incur significant inference cost, which isn’t discussed in depth.\n3. While the benchmark generation pipeline is verified, there’s no qualitative user study or human preference test to assess the final outputs or user experience improvements from M3CoT.\n4. While many open-source models are evaluated, M3CoT’s effectiveness is demonstrated only on two closed-source models. It would strengthen the paper to evaluate M3CoT on open-source LLMs (e.g., InternVL3).\n\nQUESTIONS:\n1. Can M3CoT be applied in real-time systems (e.g., AR devices), or is its inference-time cost too high? Consider discussing latency and compute implications of multi-agent prompting.\n2. Is there a risk of conflicting information between scene graphs (e.g., ego says “holding a cup,” exo says “not holding anything”)? How is this resolved during refinement?\n3. Do you leverage any visual feature alignment or object detection models (e.g., pose estimation) to help match entities across views, or is everything handled purely via language prompting?\n4. Do you expect E3VQA-trained models or the M3CoT prompting approach to generalize to other ego-exo datasets like Charades-Ego or OpenEQA?\n5. Some qualitative examples of M3CoT failure or ambiguous scene interpretations would help readers understand the boundary of your method's capability.\n6. How does your model handle ambiguous or inherently subjective questions (e.g., “What am I likely to do next?”) where multiple answers might be plausible?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}