Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
Authors: Insu Lee
Wooje Park
Jaeyun Jang
Minyoung Noh
Kyuhong Shim
Byonghyo Shim
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Vision-Language Models
Egocentric
Exocentric
Visual Question Answering
Primary area: applications
TL;DR: We introduce E3VQA, a multi-view QA benchmark, and M3CoT, a prompting method that improves LVLM reasoning by combining egocentric and exocentric views.

Abstract: Large vision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality, where a first-person (egocentric) view captured by head-mounted cameras serves as key input.
While this view offers fine-grained cues about user attention and hand-object interactions, its narrow field of view and lack of global context often lead to failures on spatially or contextually demanding queries.
To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views, providing complementary information such as global scene layout and object visibility to LVLMs.
We present E3VQA, the first benchmark for multi-view question answering with 4K high-quality question-answer pairs grounded in synchronized ego-exo image pairs.
Additionally, we propose M3CoT, a training-free prompting technique that constructs a unified scene representation by integrating scene graphs from three complementary perspectives.
M3CoT enables LVLMs to reason more effectively across views, yielding consistent performance gains (4.84\% for GPT-4o and 5.94\% for Gemini 2.0 Flash) over a recent CoT baseline.
Our extensive evaluation reveals key strengths and limitations of LVLMs in multi-view reasoning and highlights the value of leveraging both egocentric and exocentric inputs.
The dataset and source code are available at [https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding](https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding).

Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
Insu Leel
Wooje Park Jaeyun Jang', Minyoung Noh
Kyuhong Shim?t, Byonghyo Shim
SSouul National University, "Sungkyunkwan University (islee, sjpark, jyjang, mynoh, bshim)@islab snu. ac.kr, khshim@skku.
Abstract
Large vision- lision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality, where first-person (egocentric) view captured by head-men mounted cameras serves key input. While this view offers fine-grained cues about user attention and hand-object interactions, its nar- row field of view and lack of global context often lead to failures on spatially or contextually demanding queries. To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views, providing complementary information such as global scene layout and object visibility LVLMs. present E3VQA, the first benchmark for multi-view question an- swering with 4K high-quality question-answar pairs grounded in synchronized ego-exo image pairs. Additionally, we propose M3CoT, a training free prompting technique that constructs unified scene representation by integrating scene graphs from three complementary perspectives. M3CoT enables LVLMs to reason more effectively across views, yielding consistent performance gains (4.84% for GPT- 4o and 5.94% for Gemini 2.0 Flash) over a recent CoT baseline. Our extensive evaluation reveals key strengths and limitations of LVLMs in multi-view reason- ing and highlights the value of leveraging both egocentric and exocentric inputs. The dataset and source code available at htttp://gitthub com//eeen/nel Towards Sowards-Compretae wwwprreeeeiee-eeeeeeeeeee Saaeannaveee-Seeeeeeeenertaaaaaa
Introduction
In recent years, large vision-language models (LVLMs) have received significant attention for their unprecedented performance diverse tasks, including information retrieval, content generation, and multi-modal interaction 47
[3, 255 [6]. Their applications are now extended to inter- active and immersive systems like virtual and augmented reality and embodied robotics [5] 24 28
18 17
Key characteristic of these applications is that images associated with the first-person view (a.k.a. egocentric view), captured by
cameras or smart glasses, are used input. Although the first-person view is crucial in revealing the user's 's intention, LVLMs might not interpret it well since most of the training datasets, often obtained from generic web images, are not acquired from first-person perspective.
Recently, several approaches overcoming the scarcity of this so called egocentric data have been proposed. Notable ones include synthetic generation of egocentric data, incorporating large-scale egocentric datasets during the pre-training stage, and leveraging parameter-efficient fine-tuning on small-scale egocentric data 23 33 36). While these approaches are effective to some extent, due inherent constraints of the egocentric view, such as a limited field of view and lack of global
Co-first authors. Co-corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

What should do next?
What the player's back number in front of me?
Cut the vogetables.
LNLM
How many players are in our hald?
Where is the cup?
OON
On the right.
players,
LYLM
How many carrots are left?
there striker my right?
Player
Figeaantni view
Eaccentric view
Egnanttrcc vizw
Exceantric view
LYLM
(a) Cooking Scenario
(b) Sports Scenario
Figure 1: Conceptual illustration of example scenarios that require joint understanding of egocentric (first-person) and exocentric (third-person) views. In each scenario, the first question can be answered using only the egocentric view, while the subsequent two questions require integrating information from both views. Yellow and gray overlays indicate egocentric and exocentric views, respectively.
context, LVLMs might not properly handle user queries requiring broader contextual understanding. Consider the scenario in Figure [a), where the user interacts with a visual assistant to make a food. input, LVLMs can readily answer the question "What
When egocentric image is provided should do next?", but may not provide the proper answer to questions such as "Where is the cup?" "How many carrots are left?", since the egocentric view has limited spatial coverage and thus fails to capture the comprehensive picture of the user environment. The moral of the story is that the egocentric view alone is not sufficient to handle the full range of user queries.
An aim of this paper is to propose a framework integrating multi-view information to enhance LVLMs' capacity to understand the comprehensive context of user's surrounding environment. The core of our approach is to integrate images from the egocentric view and third-person view (a.k.a. exocentric view) extract the merits of both. The exocentric view provides broader contextual cues, such the user's posture and the visibility of objects in blind spots, while the egocentric view offers information about user focus, including hand-object interactions and fine- grained object details. While integrating multiple viewpoints can help answer diverse user queries, it remains unclear whether conventional LVLMs can fully leverage them and really enhance the output quality. To answer the question systematically, we design 1) an ego-exc" multi-view question answering benchmark, referred to as the Ego-Exo Expanded Visual Question Answering (E3VQA) benchmark, and 2) a novel prompting technique called Multi-view, Multi-perspective, Multi-turn Chain-of- Thoughh prompting (M3CoT).
The purpose of E3 is to evaluate the ability of LVLMs to jointly perceive and reason when egocentric and exocentric views are provided. To this end, we meticulously curate dataset comprising 4K question-answe -nnwwer pairs, each coupled with synchronized ego-exo images collected from the Ego-Exo4D dataset [22 covering four distinct tasks, including 1) action understanding, 2) object recognition, 3) spatial reasoning, and 4) numerical estimation. To fully leverage the complementary cues of ego-exo views, we further propose a novel prompting technique called M3CoT. Specifically, construct unified description of the scene (i.e., scene graphs) that guides the LVLM understand the entire scene environment. This process consists of two main stages. First, we build three distinct scene graphs, each capturing the scene from a different perspective. The first graph is generated by simultaneously processing both ego and exo images, providing a holistic view of the scene while potentially overlooking fine-graines details. The second and third graphs constructed by sequentially processing, either from ego to exo or from exo to ego, allowing the model to capture minute details. Second, we iteratively unify the three scene graphs, enabling each to progressively converge into more robust and comprehensive representation. These unified scene graphs are used by L.L.Ms to generate the final responses.
We evaluate M3CoT on the E3VQA benchmark using state-of-the-ar LVLMs, including GPT-4o 13
and Gemini 2.0 Flash (37]. and observe a considerable gain in accuracy of 4.84% and 5.94% over the recent CoT method CCoT 32 We also observe the noticeable gain (6.88% and 8.13%) on numerical reasoning questions, demonstrating our method's 's effectiveness at integrating dual-view information.
In summary, our contributions are as follows:
Egoo and denote the egocentric and exocentric, respectively.

---

Pose & Action Perception
Ego Vien
Exo View
Spatial Reasoning
&s plysical staie and movimnent such their posns, actions
Queniton about
Quesiions about the relative possiioning and
arrongomen: of objects people
Q: What am I doing with right hand?
Q: Which object is to my left?
A) Chopping onions Graabbing
Mising
()) Washing dishes
()) Holding fork
()) Peeler
D) Spice container
Q: is person with glasses pocitioned?
Q: object
person?
A) Sitting
B) Kneellin
A) Mixing bowl
B) Cutting board Plate
() Crouching
Standing
Spice
Object Attribute Perception
Ego
View
Numerical Reasning
Questions about idontiiiing objects and their foutures, like color type
Questions about counting elemouts, such az mumber people objects
Q: What type of shoes am wearing?
Q: Hew many people are there, including me? Three
A) Sucakers
Cleats
Two
C) Sandals
Flip-Plops
C) Four
Five
Q: What color is the goalposi?
01
Q: How many socver balls the field?
A) Blue
Green
A) One
Two
C) Yellow
D) Red
C) Three
D) Four
Figure 2: Categories in the E3VQA benchmark. Each question is paired with ego-exo images and multiple-choice answers. The answers are highlighted bold. The left part shows recognition categories, assessing the ability to focus question- relevant parts. The right part shows reasoning categories, evaluating the ability to integrate information across views.
We build the ego-exo multi-view VQA benchmark, E3VQA, consisting of 4K rigorously curated question-answe pairs with synchronized ego-exo image pairs. We construct E3VQA through a systemically designed pipeline to ensure that each instance evaluates the capabilities of LVLMs in integrating and reasoning across ego and exo views.
To address the challenges posed by E3VQA, we propose M3CoТ, a training-free prompting technique that combines scene graphs from three complementary perspectives into unified scene graph. M3CoТ improves the question answering accuracy on the E3VQA benchmark, achieving gains of 4.84% and 5.94% on two leading LVLMs, GPT-4o and Gemini 2.0 Flash.
We perform a detailed analysis of leading LVLMs on E3VQA, uncovering their specific failure modes in multi-view reasoning and quantifying how egocentric and exocentric inputs affect performance.
2
E3VQA Benchmark
2.1 Motivation and Objectives
When compared to single-image setting, LVLMs incorporating multi-view images face number of challenges. First, the model needs to identify which image and what regions in an image are relevant to the question. Second, the model needs to filter out redundant content appearing both views. Third, the model should deliberately extract 'complementary' cues from both views to generate a complete answer. Because single image is provided per question for the conventional visual question answering (VQA) benchmark, they cannot evaluate multi-image reasoning capabilities.
To address this gap, we introduce E3QQA, a multiple-choice benchmark specifically designed for paired ego-exo images. Each question accompanied by set of plausible but incorrect options (i.e., distractors) that target typical failure patterns. These patterns include relying solely on one image (ego or exo), ignoring visual input altogether, or failing to merge complementary information from both views. These carefully crafted distractors enable E3VQA precisely evaluate model's ' ability to reason across ego-exo image pairs.
2.2 E3VQA Composition
We organize E3VQA into four categories: pose and action perception, object and attribute perception, numerical reasoning, and spatial reasoning, to encompass a wide range of real-world scenarios (see Figure 2 Each category contains 1,000 question-answar (QA) pairs, evenly divided between egocentric and exocentric questions (e.g., "What am doing?" vs. "What is the person doing?"), which supports the evaluation of the model's generalization capability to diverse forms of user queries.
3

---

f. T LVLM
Q: How many people are in front of me?
Co
Q: How many people are in the room?
Aegsi Two people.
Auso people.
LVLM
Ego View
Exo View
Ego QA
Exo
(a) Single View QA Generation
Aego Two people.
Two.
Aasce Two.
Atuer' Only one.
Ego View
Both Views
Only
How many people front of me?
Viowpoint Options
Candidatc Answers
SingleVieww Questioas
(b) View-Specific Response Expansion
@: How many people are in front of me!
Q:
How many people are the room?
Aego Two people.
Aezo Two.
Aego Two.
Apxoi Three
Abota Two
Acsuti Only
LVLM
Asuta: Five.
Aczuri One.
Filter the questions based on the given critetia.
Filtered Questions and Answers
Filtering Prompt
Gencrated Questions and Answers
Response-Based Based Question Filtering
Figure 3: Overview of the E3VQA benchmark's 's three-step automated QA generation pipeline: (a) single-view QA generation step, (b) view-specific response expansion step, and (c) response-based question filtering step.
To solve the questions, the model must identify the relevant object or person both views and align their features across the two images. Variations in viewpoints and fields of view make the same element look distorted, partially occluded, or differently scaled, making it difficult to establish correspondences and integrate visual cues. Detailed explanations of the challenges for each category and dataset statistics are provided in Appendices(A. A.I A.2
2.3
Dataset Construction Pipeline
2.3.1 Source Data Pre-Processing
The E3VQA benchmark is constructed utilizing the large-scale synchronized ego-exo dataset, Ego- Exo4D [12] which composed of diverse user interactions (e.g., cooking, bike repair, soccer) filmed in various environments and countries. To ensure diversity within the dataset, video clips are uniformly sampled from the Ego-Exo4D dataset with respect both user activities and recording locations. Each selected video clip is downsampled to frame per second, and 8 frames are uniformly sampled from the downsampled clip. As result, 4,600 ego-exo image pairs are obtained from the 575 video clips. Note that all video clips are selected from the test split to prevent any potential dataset contamination.
2.3.2 Automated QA Generation
We introduce a three-step automated QA generation pipeline designed minimize human effort and improve scalability (illustrated in Figure 3 Throughout the entire process, we utilize GPT-4o [3],
powerful off-the-shelf LVLM.
Step Single-Vies View QA Generation We begin by generating QA pairs independently from either the ego or exo images, under the assumption that recent LVLMs show stronger understanding of single images than of multi-view inputs. Specifically, we instruct the model to generate Ego QA pairs (c.g., Q: What am I doing?, Aego Pouring water) from the ego images, and Exo QA pairs (c.g.. Q:

---

What is the person doing?, Acso Stirring eggs) from the exo images. This design ensures that the generated questions align with the visual characteristics of each view: the ego image enables clear identification of the 'I', while the exo image provides broader field of view and richer contextual information about 'the person'. For each image, we generate three QA pairs per category balance diversity and relevance; generating more than three often results in questions that are not visually grounded. As a result, this process yields a total of 110,400 single-view QA pairs.
Step 2: View-Specific Response Expansion In this step, generate diverse candidate answers by presenting the model with the same question under four distinct input conditions: 1) ego view only, 2) exo view only, both ego and exo views, and text only (no visual input). The resulting answers are denoted by Angon Arzo Aboth. and Atezt, respectively. These responses serve two key purposes in subsequent steps: 1) they are used as criteria for identifying low-quality QAs during the filtering stage, and 2) they function as hard candidate options for constructing multiple-choice questions in the human verification stage.
Step 3: Response-Based Question Filtering A large proportion of the questions generated in the previous step are either too easy or disqualified for multi-viww question answering. To filter out such questions at scale, we introduce response-based question filtering strategy that uses the initial answer from Step (denoted as Asnii either Aego or Aezo) as a reference. First, we discard questions where Atezt matches Asnit, since this indicates that the question can be answered without any visual input. Second, we remove questions for which Aboth matches Amnit which indicates that the answer remains unchanged even when both ego and exo views provided, and therefore implies that multi-view reasoning is unnecessary. Applying these two criteria retains only those questions that cannot be answered without integrating both views. As result, approximately 78.5% of the initial questions are filtered out, leaving a set of 23,694 challenging, multi-view QA samples.
2.3.3 Human Verification
Following automated QA generation, we perform human verification with four expert annotators. The experts thoroughly review all questions, discarding unclear or low-quality questions. They also carefully craft the answer options, leveraging the responses generated in the previous step: A Aego- A Aboth and Ateat This process results in the final E3VQA dataset, comprising 4,000 high-quality QA pairs, representing just 3.6% of the original 110,400 samples after filtering and refinement. Please refer to Appendix A.3 for additional details.
3 M3CoT: Multi-Perspective Scene Understanding
3.1 Multi-Perspective Scene Graph Generation
In our proposed ego-exo multi-image question answering scenario, we expect the LVLM to generate the most appropriate answer given a query Q and a pair of ego and exo images I = {Lego Jexo To help the model understand ego and exo images comprehensively, scene graph generation approach. Specifically, three instances an LVLM act as distinct agents, denoted as F1, F2, and F.,, each generating scene graph from different perspective as follows:
employ
multi-perspective
Ego&Exo: Agent Fi generates a joint scene graph S in single step by simultancously processing both Lego and Lexo input.
Ego2Exo: Agent F2 first generates scene graph using only Tego. which is then sequentially expanded by incorporating information from Leso to generate scene graph Sz.
Exo2Ego: Agent F3 follows the reverse approach, generating a scene graph based solely on Texo and subsequently supplementing it with Tego to generate scene graph S3-
Together, the three agents can capture both view-specific details and holistic scene context from complementary perspectives. For each perspective, prompts are carefully designed to capture the complementary information present in the ego-exo images. Please refer to Appendix E.2 for the complete prompts.
5

---

Inputs & Node Legend
Ego2Exo Perspective
Exo2Ego Perspective
Q: How many cggs are left?
eeoo [Q]
[exo [Q]
For the provided image and its associated question, generate scene graph
For the provided image and its associated question, generate scene graph
Uegol Identified node
[exo
Scate GraphrG11)
A Misidentified node
Unidentified node
Ego& Exo Perspective
Vegnn] [exnn] [Q]
[exo [Q]
[eeoo] [Q]
Using the provided two images and their ussociated question, generate unified scene graph
For the provided image from different view and the scene graph generated from the previous view, refine the scene graph
For the provided image from different view and the scene graph generated from the previous view, refine the scene graph
S1
S
Answer: 3 eggs
Answer: 2 eggs
Answer: epgs
<5²><< Sz>
S
Sz²
SSp
S2
Using scene graphs generated different perspectives as additional context, generate refined scene graph
Answer: 2 eggs
Answer: 2 eggs
Answer: 2
Figure 4: Overview of the M3CoТ method. Left: Scene graph generation process from the Ego&Exo perspectives. Center: Scene graph generation process from the Ego2Exo perspective. Right: Scene graph generation process from the Exo2Ego perspective. Scene graphs from each perspective are merged to complement missing objects and relations, enabling the model to perform coherent reasoning and answer generation.
3.2 Iterative Multi-Agent Scene Graph Refinement
To further refine each scene graph S; (for 1, 2,2,3) generated by agent F, we iteratively incorporate information from the other two agents. At iteration t, agent F; takes the other two scene graphs (c.g.. st-1 and S for Fi) to examine their objects, attributes, and relationships, and then adjusts S to S to better align with both lego and Texo- Here, S denotes the scene graph S; after the (-th update. By leveraging complementary information from multiple perspectives, this process improves both the accuracy and completeness of each agent's s scene graph. At each iteration step every agent F, generates an answer to the question, conditioned on S.. We then aggregate the agents' responses via majority voting. If consensus achieved, we accept the majority answer and terminate the process. If the agents' answers remain inconsistent after fixed number of iterations, the final answer is selected from the response of Fi. This iterative loop yields progressively richer scene representations and promotes convergence among the agents' answers.
4 Experimental Results
4.1 LVLMs' Performance on E3VQA
To assess ego-exo multi-image reasoning capabilities, we evaluate five closed-source and nine open-source LVLMSs on the E3VQA benchmark using their default configurations. Detailed model specifications, experimental settings, as well as system and user prompt templates are provided in Appendices B. and B.2

---

Table I: Performance comparison of recent closed and open-source models on the E3VQA benchmark.
Pose Action
Object & Attribute
Numerical
Spatial
INLMs
Avg.
Ego
Exo
Ego
Exo
Ego
Exo
Ego
Exo
Closed-Source
1.1
1.122
77.00 1.40
114
1.122
61.13 u.
g7.13g 1012
GPT-4o mini B
41.33 631
49.00
66.07
71,00
35.80 0us
44,00
44,00
41.00=
49.03
Gemini 2.0 Hish
37
53.27 6.12
60.33
74.00
76.47
46 46.33
56.20
58.27
53.80
39.80
Gemini
53.73 646
63.40
69.60
44.07 101
22.00
58:27 2.4
54.00= 1.00
Claude 3.5 Sonnet
40.33 642
0..6
59. 19.13 000
0.3
44 103
49.4 125
50.73
20 0.0
49.69
Open-Source
44.73 130
54.93
68. 18.13
73.73
35.60
53.00
45.67 035
48.33 199
53.02
Qwen2.5 V1.
50.87 023
69.60 +: 0:20
75.93 6.66
35.93 012
47.87 0.11
46.07 0.12
41.27=023
Owen2-VVL-77
99
53.67 6:0
56.07 = 1.72
67. 6.76
67.47 1.14
32.87 009
38.07 1.42
43.27
42.27 1.i
50.10
LLaVAoOneVisioo.77
200
39.87
50.73
67.60
68.80
34.87
40.87
49.20
42.93 070
49,36
IntemVLL288B 38
42.20 o.s
61.67 o.11
64.67 am
33.40 n.5i
38.53 now
43.67
41.13
46.18
LLaVA-NoXT-7B
33.87 0.6
57.33 e.12
62.27 62s
30.27
39.07 0.46
47.20
40.67
Mantis-BB-idcfies]
15
28.07
56.53 63.73 o3
35:67
37.73
=1.53=021
32.53
40.16
Deepseek: V1.-Chat 7B 27
32.60
34.27
51.80 0.3
52.47 22.47
32.80
29.80 o.5
41 21:00=00::
36.60
38.92
Qwen.
25.20 1n
26
33.60 1.11
36.80 a: 1.20
21.07 2.6
21.73 1m0
29.47 1.70
30.53 036
Table 1 reports model accuracy across categories, each consisting of 500 egocentric (Ego) and 500 exocentric (Exo) questions. Even the best-performing model, GPT-4o, achieves only 60.90% accuracy on E3VQA, underscoring the benchmark's s difficulty. Among open-source models, Intern 14B attains the highest accuracy, while Qwen2.5- - delivers competitive performance despite its smaller number of parameters. Overall, LVLMs struggle the most with numerical reasoning yet perform relatively well on object and attribute recognition. Notably, models consistently underperform on egocentric questions compared to exocentric questions, highlighting difficulties in resolving the first-person perspective.
4.2 Performance Evaluation M3CoT
To demonstrate the effectiveness of our M3CoT prompting scheme, we compare our technique with three recent multimodal CoT techniques (DDCoT, CoCoT, and CCoT) on the E3VQA benchmark. Experiments are conducted using two leading LVLMs, GPT-4o and Gemini 2.0 Flash, both of which achieve the best performance on E3VQA. Table 2 presents category-wise accuracy for egocentric and exocentric questions. M3CoT improves Gemini 2.0 Flash. addition, it surpasses DDCoT and CoCoT by 4.15% and 5.71% on GPT-4o, and by 5.03% and 5.81% % Gemini, respectively. The substantial gains in Numerical Reasoning (6.88% and 13% over CCoT) highlight M3CoT's ability to integrate multi-view information for a more complete and accurate understanding. In contrast existing methods, which exhibit limited or inconsistent improvements and occasionally even show performance drops, M3CoT achieves consistent and substantial gains across all categories. These results validate the effectiveness of our approach in addressing the limitations of current multimodal CoT techniques. For additional results on open-source LVLMs, please refer Appendix C.2
CCoT by 4.84 % GPT-4o and 5.94 %
5 Analysis
5.1 Analysis of Automated QA Generation Pipeline
To examine how the source of distractors affects the question difficulty, we sample 160 questions (40 per category) and construct four alternative option sets. In each set, all four answer choices are drawn from a single source: text-only, ego view, exo view, or both views. This setup contrasts with our standard configuration, where each distractor is drawn from different source. As shown in Figure the model's error rate increases in the following order: text-only, both-view, single-view, and our composite setting. This result highlights that constructing answer options from diverse
7

---

Table 2: Performance comparison of recent multimodal CoT methods top-performing models.
Pose & Action
Object & Attribute
Numerical
Spatial
Methods
Avg.
Ego
Exo
Ego
Exo
Ego
Exo
Ego
Exo
GPT-4o
Default
49.87 1.00
63.47
72.47 1 0.z2
77.00 1
48.47 1 1.4
57.67 1.112
61.13 1.0
57.13 1
60.90
DDCaT
24
=
69.53
73.80
78.80 0.40
48. 26
57.87 57.87 4
67.27
64.87 0.50
64.43
CoCol
50
50.93
66.80
72.20
76.33
49.93
60.93 1aw
65.07
60.80
62.87
CC.T
17
55.53
67.47
73,00
77.67
48.27
62.27
63.73
62.00
63.74
M3CoT (Ours)
58.40
69:40 1:5
78,90
82.80
56.40
67.90 1:11
71.90
62,90 2.14
68.58
Gemini Flash
Default
53.27 0.12
60.33
74.00
76.47 0.50
46.33
56,20
58.27
53.80
59.80
DDCoT
34
55.60_em
62.60
75.53
81.13
46
54.67
57,47
55.60
61.09
CoCoT
55.40
61.67
73.80
77.93
45,27
56.20 035
58.67 0:42
53.53
60.31
CCuT
55.93 0.46
61.53
71.47 =
76.93 0.58
46.67 pm
60.73
57.27 a
50.93
60.18
M (Ours)
$7.80 0.30
65.80 0.00
a: 0072
82.80 0.40
55.60 1.06
67.40 1.25
62.67 0:08
58.07
66.12
40
Ego-only
Lavor
30
Exo-only
Multi- view
20
Fig
Accuracy (5)
(a) Error rate across different option sets
40
19%
88%
68%
20
13%
Whong
Concect Fileed Out
Any
Ego
Exo
Both
(b) Proportion of correctly answered questions
(c) Model accuracy across different visual input
Figure 5: Analysis of the benchmark construction pipeline and model performance under varied input conditions: (a) error rate across option-generation strategies, (b) proportion of correctly answered questions between retained and excluded questions, and (c) performance across different visual input modalities.
input sources increases question difficulty, thereby providing a more rigorous evaluation of LVLMs' multi-view reasoning abilities.
To assess the effectiveness of question filtering process, compare model accuracy two subsets: filtered subset (68% of the data) removed during our filtering process and an unfiltered subset (the remaining 32%). As shown Fig 42% of questions the unfiltered subset answered incorrectly with ego-exo multi-view input, whereas only 12% of questions in the filtered subset are answered incorrectly. This substantial gap indicates that the filtering pipeline effectively removes questions solvable by superficial cues while retaining more challenging ones that better evaluate model's 's' reasoning ability. As result, the subsequent human verification process becomes more reliable and efficient.
5.2 Analysis LVLM Performance under Single and Multi-Viee Inputs
We partition the E3VQA benchmark into four subsets, Any, Ego, Exo, and Both, based on which view(s) are required to answer each question. Specifically, questions in the Any subset can be an- swered using either the egocentric or exocentric image alone, as each view individually contains sufficient information. The Ego and Exo subsets, in turn, require only the egocentric or exocentric image, respectively, since relevant information appears a single view. The Both subset requires integrating cues from both egocentric and exocentric views. We then investigate how the model's responses differ berween single-vies view and multi-view inputs within each subset. As shown in Fig- ure 5tc), in the Any subset, providing both images yields marginal accuracy gain, indicating that

---

Sub question What is doing the inages! Sub Answer I: The user cooking in pas.
DICGe
Final Answer:s ()
Sub questioa 2: What is the user holding in their ledt hand?
Sub Answer 2: The
holding
their left hand.
CC&T
Scene graph:
Final Auswer ()
('object:: ("hame": : "spandla".... "relation": "holding", "aange" "lefi hamd"))
Ego View
Eso View
Q: What am doing with my left hand?
('objets": [ (""aune": "pan", "relation" : "held by". "urget" "left hand").
graph:
A) Stirring
By) Hotding n Choppinn vegetables
Final Auswer B)
('name": "spatula". "relation" "held by", "tange"" "right") 1]
c) Holding sputula
Sub question I: How tripods are visible the first image??
Sub Answer
image shows tnpods.
Final Answer AA)
images show Sub Answer 2: Yes, both images show the same number of impods.
qucstion
sumber trpodi??
CCoT Final Answer A
["object": ("id"; "tppodi", "tame": "sipod"" ("dd": "sspod2", "name": "uipod"), ).
graph:
I'sd": "tripod"" "name": "Inpod" []]
Ego
graph:
How many trpods up me?
I'objects' [["name": "rrpod", "positions" ""lef"), J. "triood" "position"" "right", ("tame" "tripoc"; "positionn" "back left"), ("tame"
3
Final Answer D)
2)
4
"tripdd" "pocitions" "back right"]]
Figure 6: Qualitative examples of answers and reasoning processes generated by different prompting methods. Blue/Red words indicate the key cues that lead correct/wrong answers.
consistent cues across views help reinforce the model's prediction of the correct answer. In the Ego and Exo subsets, providing both images leads to noticeable performance drop, suggesting that redundant context may confuse the model. In the Both subset, multi-view inputs improve accuracy compared to single-view setups; however, performance remains low, staying below 40%.
5.3 Analysis of Multi-Perspective Scene Graph Generation Strategies
To analyze the advantages of our three scene graph generation perspectives (Ego&Exo, Ego2Exo, and Exo2Rgo), report in Table their respective performances across the Any, Ego, Exo, and Both subsets. The Ego&Exo strategy achieves the largest accuracy gain in the Both subset, demonstrating its strength in integrating complementary cues across viewpoints. In contrast, Bgo2Exo performs best on the Exo subset, while Exo2Bgo yields the highest improvement on the Ego subset, reflecting their specialization in inferring one view from the other. These results highlight that different scene graph generation strategies provide complementary advantages depending on where the key information required answer the question is located within the ego-exo view images. In addition, our scene graph refinement stage improves performance beyond any individual strategy by combining their strengths and compensating for their limitations. Overall, these findings confirm that fusing diverse scene graph perspectives produces more robust reasoning in ego-exo multi-view scenarios.
5.4 Qualitative Examples
Figure shows qualitative examples of answers and reasoning processes generated by DDCoT, CCoT, and M3CoT methods using Gemini 2.0 Flash. Although CCoT produces plausible scene graphs, it fails to integrate information across multiple views, resulting incorrect answers. Specifically, it often misidentifies the same ob- ject observed from different perspectives as sep- arate entities. In contrast, our method effectively extracts key information, aligns observations across views, and accurately identifies the same object to answer the question through a multi- perspective, multi-turn turn reasoning process.
Table 3: Performance comparison of M3CoT's three perspectives across subsets grouped by the image view(s) required to answer.
Required View(s)
Perspective
Avg.
Any
Ego
Exo
Both
Default
66.29
64.83
54.21
37.49
59.80
Ego&Exo
66.29
67.44
56.67
50.87
63.65
Ego2Exo
68.08
62.71
61.51
43.91
62.83
Exo2Ego
66.94
68.02
59.92
39.13
62.98
M3CoT (ours)
69.79
69.28
62.91
53.04
66.12

---

Related Work
6.1 Ego-Exo Datasets and Tasks
Egocentric and exocentric views offer complementary information for understanding users and their enviromments. Early datasets like Charades- [35] and LEMMA (14] introduced paired ego-exo data, while Ego-Exo4D [12] further scaled this paired ego-exo data with large, synchronized videos capturing diverse real-world scenarios. To generalize semantic understanding across multiple perspec- tives, a body of work has focused on learning view-invariant representations [4]] 45 Furthermore, efforts to align ego-exo content have emerged, including object-level mappings [0] and techniques for identifying and segmenting camera wearers in exocentric scenes [9 In parallel, cross-view knowledge transfer has been actively explored, with each perspective leveraged to improve the 43 34]. Several studies have addressed viewpoint selection across perspectives by proposing methods for dynamically selecting informative views over time [29 30]. Others have explored generating egocentric videos from exocentric video inputs using diffusion-based models [26] [44] or cropping exocentric image frames to distill egocentric-reles
understanding of the other [51] 22
cues [6]. Despite these advances, task that jointly reasons over synchronized egocentric and exocentric views within LVL.Ms remains underexplored, highlighting a promising direction for future research.
6.2 Visual Question Answering with IVLMs
Visual Question Answering (VQA) benchmarks test LVLMs' ability to interpret and reason over diverse visual content. Most existing VQA benchmarks are constructed from large-scale web-crawled data, typically consisting of images captured from fixed third-person cameras 442
To sup- port scenarios that require understanding from the user's ' perspective, egocentric VQA benchmarks capturing first-person views have been introduced. VgA [ evaluates first-person visual un- derstanding capabilities by offering both egocentric and exocentric queries on first-person visual inputs. EgoSchema 31 evaluates long-form egocentric video understanding by assessing model's ability to recall previously observed objects and events. EgoTThink 4 evaluates first-person reasoning capabilities across diverse categories that reflect practical real- woald scenarios. Another line of work includes embodied QA benchmarks such as EmbodiedQA (5] and OpenEQA (28). where agents are required to navigate or interact with their environments answer given queries. Although numerous VQA benchmarks aim to evaluate LVLMs across diverse aspects, no existing benchmark assesses model's ability to seamlessly combine complementary visual information from paired ego and exo views. For detailed comparison of VQA benchmarks, please refer to Table in the Appendix.
48 49
6.3 Chain-of- Thought Prompting in LVLMs
Building on its success in large language models (LLMs), Chain-of-Thought (CoT) prompting has been extended to LVLMs to enhance inference-time time reasoning. DDCoT breaks down question into a sequence of sub-questions and corresponding sub-answers, which are then used collectively derive the final answer to the original question [54]. CoCoTT, introduced for multi-image input scenarios, compares the similarities and differences between images, guiding the model to answer questions based on the identified visual contrasts [50). CCoT facilitates understanding the overall context of an image through scene graphs, which are first generated by the LVL.M and then incorporated into the prompt to enable compositional reasoning over objects, relations, and attributes (32). Despite their successes, their applicability to ego-exo multi-image contexts remains unexplored, raising an open challenge for extending CoT methods to these settings.
7 Conclusion
In this work, we introduced E3VQA, the first benchmark that systematically assesses whether LVLMs can reason jointly over egocentric and exocentric views. By curating 4K high-quality question-answer pairs grounded in synchronized ego-exo images, E3VQA serves as a rigorous testbed for multi- view understanding. In addition, we proposed M3CoT, novel prompting strategy that merges scene graphs from diverse perspectives into unified graph. Extensive experiments demonstrate that M3CoT consistently outperforms the strong CCoT baseline, underscoring the importance of multi-perspective integration for multi-view understanding. By establishing a benchmark for evaluating LVLMM'' ego- exo reasoning and enhancing their multi-view understanding ability, this work takes a step toward more context- visual assistants capable of operating in complex, real-world environments.
10

---

Acknowledgments
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (ITTP) (RS-2024-0038815 and IITP under the artificial intelligence semiconductor support program to nurture the best talents (ITTP-202-RS (023-RS-2023-0023 20330033-0026608 1), and by the National Research Foundation of Korea (NRF) (RS-2023-00208855) all funded by the Korea government (MSIT).
References
[1] Anthropic. Claude 3.5 sonnet model card addendum, 2024.
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vi vl: frontier large vision- lsiounlanguage model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[4] Sijie Cheng. Zhicheng Guo, Jingwen Wu, Kechen Fang. Peng Li. Huaping Liu, and Yang Liu. Egothink: Evaluating first- pest-ppesonn perspective thinking capability of vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14291-14302, 2024.
[5] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra, Embodied question answering. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 1-10, 2018.
[6] Zi-Yi Dou, Xitong Yang, Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Unlocking exocentric video-language data for egocentric video representation learning. arXiv preprint arwiv:2448.. 035677 2024.
[7] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36:70293-70332, 2023.
[8] Chenyou Fan. Egovqa-an an egocentric video question answering benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.
[9] Chenyou Fan, Jangwon Lee, Mingze Xu, Krishna Kumar Singh, Yong Jac Lee, David J Crandall, and Michael Ryoo. Identifying first- pust-person camera wearers third-person videos. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 5125-5133, 2017.
[10] Yuqian Fu, Runze Wang, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, and Luc Van Gool. Objectrelator: Enabling cross-view object relation understanding ego-centric and exo-centric videos. arXiv preprint 1rXiv:2411.19083 2024.
[11] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens context. arXiv preprint arXiv:2403.055: 2024.
[12] Kristen Grauman, Andrew Westhury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 19383-19400, 2024.
[13] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- system card. arXiv preprint 2rXiv:2410..126. 2024.
[14] Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent ulti-task ctivities. In European Conference Computer Vision, pages 767-786. Springer, 2020.
[15] Dongfu Jiang, Xuan He, Huaye Zeng. Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research, 2024.
[16] Donghoon Kim, Minji Bae, Kyuhong Shim, and Byonghyo Shim. Visually guided decoding: Gtadienn-frree hard prompt inversion with language models. arXiv preprint arXiv:2505.08622. 2025.
11

---

[17] Donghoon Kim, Gusang Lee. Kyuhong Shim, and Byonghyo Shim. Preserving pre-trained representation space: On effectiveness of prefix-tuning for large multi-modal models. arXiv preprint arXiv:2411.0002 2024.
[18] Jiseob Kim, Kyuhong Shim, Junhan Kim, and Byonghyo Shim. Vision transformer based feature extraction for generalized zero-shot learning. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 5, IEEE, 2023.
[19] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James I. McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.0224
2022.
[20] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 033266 2024.
[21] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang. Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interlee Tackling multi-image. video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. 2024.
[22] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representa- tions from third- perr-p to first-person vidcos. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943-6953, 2021.
[23] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray. Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egacentric video-language pretraining. Advances Neural Information Processing Systems, 35:7575-7586, 2022.
[24] Chao Liu, Chi San Cheung, Mingqing Xu. Zhongyue Zhang, Mingyang Su, and Mingming Fan. Toward facilitating search in vi with the assistance of vision large language models. In Proceedings of the 30th ACM Sympoxium on Virtual Reality Software and Technology, pages 1-14, 2024.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[26] Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to to-poreectttteeeenttiieeeeee video generation. Advances in Neural Information Processing Systems, 37:136149- 136172, 2024.
[27] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real-world laiiuale understanding. arXiv preprint arXiv:2403.05525 2024.
[28] Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14488-16498, 2024.
[29] Sagnik Majumder, Tushar Nagarajan, Ziad Al-Halah, and Kristen Grauman. Switch-a-view: Few-shot view selection learned from edited videos. arXiv preprint 1rXiv:2412.18386, 2024.
[30] Sagnik Majumder, Tushar Nagarajan, Ziad Al- Al-Halah, Reina Pradhan, and Kristen Grauman. Which viewpoint shows it best? language for weakly supervising view selection in multi-view vidcos. arXiv preprint arXiv:241 1. 08755, 2024.
[31] Karttikeya Mangalam, Raiymbek Akshulakow, and Jitendra Malik. Egoschema: A diagnostic henchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-46244, 2023.
[32] Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420-14431, 2024.
[33] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5285-5297, 2023.
12

---

[34] Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. From my view to yours: Ego-augmented learning in large vision language models for understanding exocentric daily living activities. arXiv preprint arXiv:2501.055 200.225000.0711,, 2025.
[35] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint 10X.::88844092222 2018.
[36] Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, loannis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon, ALa A multimodal embodied AI foundation model for egucentric video understanding. In Findings of the Assaciation for Computational Linguistics: EMNLP 2024, pages
11101-11122. Association for Computational Linguistics, November 2024.
[37] Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
[38] OpenGVLab Team. Internv12: Better than the best expandinn performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024.
[39] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world any resolution. arXiv preprint arXiv:2409. 12/91, 2024.
[40] Qitong Wang, Long Zhao, Liangzhe Yuan, Ting Liu, and Xi Peng. Learning from semantic aligmment be- tween unpaired multiviews for egocentric vidco recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3307-3317, 2023.
[41] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[42] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[43] Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval- augmented egacentric video captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13525-13536, 2024.
[44] Jilan Xu, Yifci Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejic Zhang, Rui Feng, and Weidi Xic. Egoexo-gen: Ego-centric vidco prediction by watching exo-centric videos. arXiv preprint arXiv:2504.1173: 11732, 2025.
[45] Zihui Sherry Xue and Kristen Grauman. Learning fine- tineggrained view-invariant representations from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36:53688-53710. 2023.
[46] Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European conference on computer vision, pages 178-196. Springer, 2022.
[47] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer. and Wen-tau Yih. Retrieval-augmeentr multimodal language modeling. arXiv preprint Xrv: I. 12561, 2022.
[48] Heessengg Yun. Youngjae Yu, Wonsuk Yang, Kangil Lee. and Gunhee Kim. Pano-avqa: Grounded audio- visual question answering on 360deg videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2031 1-241, 2021.
[49] Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis Phillppi Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807-8817, 2019.
[50] Daoan Zhang, Junming Yang. Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thouugh prompting for large multimodal models with multiple image inputs. arXiv preprint or Xiv:24011.0222 02582, 2024.
[51] Haoyu Zhang. Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided milm for egocentric video understanding, arXiv preprint 4rX1v:2503.01143, 2025.
13

---

[52] Juexiao Zhang, Gao Zhu, Sihang Li. Xinhao Liu, Haorui Song, Xinran Tang, and Chen Feng. Multiview scene graph. Advances in Neural Information Processing Systems, 37:17761-17788, 2024.
[53] Ziwei Zhao, Yuchen Wang, and Chuhua Wang. Fusing personal and environmental cues for identification and segmentation of fiss-peeroon camera wearers in third-person views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16477-16487, 2024.
[54] Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yis Yu Zhou, and Sibei Yang. Ddcot: Duty-distinet chain-of-thought prompting multimodal reasoning in language models. Advances Neural Information Processing Systems, 36:5168 5191, 2023.
[55] Jinguo Zhu. Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Tian, Yuchen Duan. Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang. Yue Cao. Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang. Weiye Xu, Hao Li, Jiahao Wang. Nianchen Deng, Songze Li, Yinan Hc, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui He, Botian Shi, Xingeheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong. Wenwen Qu, Peng Sun. Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction explicitly state that LVLMs struggle to integrate egocentric and exocentric views, introduce the E3VQA benchmark with 4K high-quality QA pairs, and propose the M3CoT prompting method which yields significant performance gains, fully matching the theoretical framework and empirical results presented later in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a "Limitations" section in Appendix that discusses the reliance of the E3VQA benchmark on the EgoExo4D dataset- highhighinn potential biases in recording environments and action diversity- and outlines future work to mitigate this dependency and broaden dataset coverage.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper is proposing the E3VQA benchmark and its dataset creation pipeline, well introducing the M3CoT prompting method and does not involve any formal theoretical results or proofs, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The paper provides full disclosure of the main experimental results with error bars, clearly identifies each LVLM variant evaluated, and includes all system and user prompt templates along with detailed specifications the Appendix, enabling faithful reproduction of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All data including E3VQA and code are publicly accessible, and all related materials materials-includding the prompt templates be found in Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
poolic/guides// CoweSSumisssooooooo.....o
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https wwwpii/cggie//eeieeieiiiiciieeet www.//ooieeiiioonno...o
/ //nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We evaluate off-the-sheiff LVLMs using their default settings (no additional training) and provide full details of our dataset construction and test setup: all system and user prompt templates are documented in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars (+) alongside all main accuracy results--cpturring variability across test samples or repeated runs-in Table and Table 2, enabling readers to assess statistical significance of the reported improvements
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide detailed specifications of our compute environment in Appendix, ensuring all reported results can be faithfully reproduced
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We fully adhere to the NeurIPS Code Ethies all aspects-including data collection and usage, privacy safeguards, and licensing- and provide discussion of these ethical considerations in Appendix.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
18

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We outline both potential benefits and risks associated with our E3VQA benchmark and M3CoT method in the Ethics Statement in Appendix, covering improved scene understanding applications alongside considerations of data privacy, bias, and potential misuse.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The proposed E3VQA dataset is based the existing dataset EgoExo4D dataset, and our paper does not introduce any new pretrained language models or image generators. Therefore, research is free from the risk of misuse.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
19

---

Answer: [Yes]
Justification: Reference 11 cites EgoExo4D, and Section explicitly states that the dataset is based on it.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwtpssppppasseeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Section provides details on the generation pipeline of the proposed E3VQA
dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: E3VQA datasets are created from existing EgoFxo44 dataset, so crowdsoure- ing was not used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The paper uses LLMs for dataset generation and multi-image VQA, de- scribed in Sections 3 and 5.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to our LLM policy //npur:////uuraie cc. for what should or should not be described.
21

---

Table 4: Comparison between E3VQA and existing VQA benchmarks. Unlike other benchmarks, E3VQA is designed to evaluate comprehensive scene understanding and reasoning across diverse question perspectives using paired ego-exo images.
Benchmark
Task Objective
Visual Perspective
Question Perspective
Answer Type
Evaluator
"Questions (test)
MSVD-QA
General Understanding
Exo
Exo
Predefined-Labl
Accuracy
13K
MSRVTT-QA
42
General Understanding
Exo
Exo
Predefined-Lane
Accuracy
72K
Social-IQ
Social Understanding
Exo
Exo
Accuracy
7.5K
Pano-AVQA 48
Spatial Audio-Visuu Reasoning
Exo
Exo
Predefined-Label
Accuracy
5.3K
EgoVQA
Egacentric Visual Understanding
Ego
Ego or Exo
Multi Choice
Accuracy
1
EgoSchema
Long-Term Reusoning
Ego
Egoo
Multi-Choiee
Accuracy
5K
EgoTThink 4
Pirst-Person Thinking
Ego
Eigo
Open-Ended
LLMs
700
EmbediedQA S
Goal-Driven Scene Understanding
Ego
Exo
Predefined-Label
Accuracy
529
OpenRQA
2N
Environment Understanding
Ego
Ego
Open-
Comprchensive Scene
E3VQA
Ego and Exo
Ego Exo
Multi-Chole
Accuracy
4K
Understanding Reasoning
A
Additional Details of the E3VQA Benchmark
A.1 Categories and Challenges
In addition to the challenges described in Section 2.2 each of the following four categories highlights a distinct challenge the ego-exo multi-image scenario:
Pose & Action Perception focuses on recognizing a person's s physical state and movement, such as how their body is positioned and what kinds of gestures or actions they are performing. The presence of multiple people, including the user and duplicated individuals across views, can confuse the model when identifying the question's 's target. The model must correctly identify the intended individuals and interpret their physical state and behavior.
Object & Attribute Perception involves identifying objects and their attributes, such as color, pattern, type. Objects may appear in only one view, be partially occluded, look different due to variations in viewpoint and field of view. To answer correctly, models must resolve such ambiguities and ground the object consistently across views.
Numerical Reasoning addresses tasks involving counting and comparing quantities, such as determining the number of people objects in scene. A single view may not include all instances necessary to answer the question, and the same object may appear redundantly across different views. To produce accurate counts, the model must integrate information from both views by handling overlapping objects and aggregating evidence across views.
Spatial Reasoning focuses understanding the spatial information of scene, including how objects and people are positioned relative to one another and how they are arranged within the environment. In multi-view spatial reasoning, differences in viewpoint angle and field of view can cause the same object to appear at varying positions in each image, become occluded in some views, or exhibit different spatial relationships with surrounding objects. To overcome these challenges, the model must align positional information from multiple views to construct coherent understanding of spatial relationships within the scene.
A.2 Dataset Composition and Statistics
Figure 7 iillustrates the statistics of the E3VQA benchmark. Figure ([) shows the distribution of correct answer positions among the four options (A-D) for each category. The uniform distribution of correct answers indicates that the dataset is balanced with respect to answer positions. Figure 7 shows the distribution of source video types used to construct E3VQA, demonstrating the benchmark's broad coverage of real-world user-interaction scenarios. Finally, Figure (c) illustrates the detailed composition of question types within each category, underscoring E3 E3VQA's broad scope of evaluation.
22

---

Slmmple
2555
24%
28%
CPK
Madic
Actine
D
D
167%
30%
Covid
Hosh
Coanteg
Action Pose
Object Altribute
Cooking
Bosket
Sports
Attritan
half
25%
20%
28%
24%
Highr
Secur
Spatia
D
Type
15%
Pat
Abort
Spatial
Numerical
Below
(a) Answer distributions
(b) Source video types
(c) Question composition
Figure 7: E3VQA statistics: (a) Distribution of the correct answers among the four options (A-D), (b) Distribution of source video types used to construct E3VQA, and (c) Composition of question types within each category.
What om doing with my left hand? (22/561)
EEgst Outions, (Hokiing o soutAd, (Halking contaiee) (roling bying (holdin buttee (Ex0) Oytors: (Hroding slrin) Pourin lepad) (lamiin on stowe, (Neatlig Arawer (nt holling Answer (Egos Hokking comaaterr
Arawer (5o03 Pouiing lighd Arowor (Botts Pouring
Arewer quedton
cofooe.
Arewer
Figure 8: User interface used by annotators during human verification.
A.3
Details of Human Verification
The human verification conducted by four members of our research team (the authors), all whom have domain expertise in vision-language reasoning and extensive experience with ego-exo data. Each annotator is assigned to specific question category and conducts verification using the user interface shown in Figure g Annotators first remove questions that are ambiguous or unanswerable when both ego and exo views are provided. Next, they discard overly simple questions that can be solved through shallow pattern matching, such as cases where Aboth correct regardless of reasoning, even if they pass the question filtering stage. Then, annotators utilize the view-specific responses generated Section 2.3.2 to construct answer options. To supplement potentially redundant or low-quality responses, they additionally use two option sets, each containing four candidate options derived from the ego and exo images, respectively. Using these option sets, annotators curate complete question sets that include the question, the correct answer, and the distractor options. Finally, each annotator
23

---

Table 5: Comparison of open-source LVLMs in terms of architecture (vision encoder and LLM) and use of egocentric data during training.
Model
Vision Encoder
L.M Backhone
Train w/ Ego Data
IntemVL3-14B
Qwen2.5-14B
Not Provided
Qwen2.5-Ve L-L77B
VIT (customized)
Qwen2.5-7B
Not Provided
Qwen2-VI.-78
ViT-I.
Qwen2-7B
LLaVA-NeXT-Onee Vision--B
Sig
Qwen2-7B
IntemVL2-8B
Intern ViT-300M
Qwen2.5-7B
ALaVA-NexTT-nnerteave--
SigLIP-SO
Qwenl 5-BB
MANTIS-Idefics2-BB
SigLIP
Mistral-BB-ve
Deqpscck-VI-chat-8
SigLIP-L, SAM-B VTT-higG
DeepScck-LLM-77
Qwen-VI.Chaa-778
Qwen-7R
Table 6: Performance comparison of recent closed and open-source models on E3VQA in the text-only setting (without visual input).
Action
Object
Numerical
Spatial
LVLMs
Avg.
Ego
Exo
Ego
Exo
Ego
Exo
Ego
Exo
Closed-Source
Gemini 2.0 Flash
20.20 25.00
24.80
20.60
22.80
27.80
30.20
16.20
15.20
22.23
GPT-4o
30.40
12.20
23.20
7.40
30.80
11,60
15.40
19.50
Open-Source
31.00
37.20
33,80
30,20
38.40
32.80
10.80
11.80
28.25
Qwen2.5-VL-22
32,20
34.60
32,80
31.00
35.20
36.20
8.00
5.40
26.93
Visioo-n
32,60
38.00
31.40
26.80
27.40
28.80
7.00
5.60
24.70
reviews subsets created by others to ensure consistency and clarity across the dataset, filtering out any ambiguous or low-quality instances.
A.4 Robustness to Temporal Misalignment
To evaluate the robustness of the E3VQA benchmark under temporal misalignment between egocen- tric and exocentric views, we analyze how synchronization gaps affect the consistency of ground-truth answers. Specifically, we randomly sample 80 examples (10 from each category) and evaluate how the ground truth changes when the ego and exo views are misaligned by to 3 seconds. When the synchronization gap is or seconds, 99% and 97% of the ground answers remain unchanged, respectively. Even with 3-second gap, 94% of the answers remain consistent. Although maintaining perfect temporal alignment can be challenging in real-world applications, our analysis shows that small temporal mismatches (e.g., few seconds) have minimal impact the consistency of the ground truth.
B Experimental Details
B.1 LVLMs and Evaluation Setup
Table[]]rooideer overview of the LVLMs used in experiments in terms of the model architecture and the use of egocentric data in training. These models are selected based on their capability of processing multi-image inputs. By evaluating models with diverse vision-language architectures. examine how recent LVLMs respond to and reason through the challenges posed by the E3VQA benchmark. For evaluation, the mean and standard deviation over three independent runs, using each model's ' default generation settings.
NVIDIA RTX A6000 GPUs. All evaluation results reported as
B.2 Text-Only Baseline Results
To investigate potential biases in the E3VQA benchmark, we evaluate recent LVLMs under text-only setting, where only the textual question provided without any visual input. This experiment allows us examine whether models can exploit linguistic patterns biases present in the data without

---

Table 7: Performance comparison of recent multimodal CoT methods E3VQA constructed from
LEMMA.
Pose Ego
Action Exo
Object & Attribute Ego
Numerical
Spatial
Methods
Avg.
Exo
Ego
Exo
Ego
Exo
GPT-4o
Default
39.68
43.55
63.49
72.58
56.45
46.03
39.68
45. 16
50.83
DDCoT
38.10
66.67
70.97
64.52
47.62
47.62
38.71
52.83
CoCoT 50
44.44
46.77
69.84
70.97
54.84
47.62
46.03
51.61
54.02
CCoT 32
36.51
48.39
68.25
69.35
53.23
47.62
47.62
40.32
51.41
M3CoT (Ours)
44.44
51.61
71.43
75.81
59.68
65.08
63,49
50.00
60.19
Gemini 2.0 Flash
Default
39.68
53.23
66.67
74.19
$1.61
46.03
46.03
46.77
$3.03
DDCoT 54
39.68
46.77
60.32
77.42
54.84
47.62
55.56
40.32
52.82
CoCoT 50
46.03
48.39
65.08
74.19
50.00
47.62
47.62
40.32
$2.41
CCoT 32
44.44
51.61
66.67
72.58
56.45
49.21
46.03
40.32
53.41
M3CoT (Ours)
44.44
59.68
66.67
72.58
56.45
42.86
55.56
50.00
56.03
Table 8: Performance comparison recent multimodal CoT methods on open-source models.
Pose & Action
Object & Attribute
Numerical
Spatial
Methods
Avg.
Exo
Ego
Exo
Exo
Egg
Exo
InternVL3 14B
Default
44.73
$4.93
63.13
73.73
35.60
53.00)
45.67
48.33
53.02
DDCoT S4
47.87
58.33
68.47 :
72.67 :: 1.45
35.33 255
2
50.67
45.93
53,26
CoCoT
49.53 081
57.27
68.27 +
72.53 1.14
34.87 1.5
47.93 9.4
20,0
46.27 0.9
53.23
CCuT
44.60 1991
58.40
2.09
h.4
73.80
37.80 3.00
20.00
46.27 +
48.80 1.78
53.12
M3CoT (Ours)
45.87 1211
60.00 0:3
70.60 ; 1400
75.73 y 008
35.07 0.50
50.87 1:000
50.80
49,40 0.72
54.79
Intern VL. 8B
Default
43.70
54.90
64,80
70.30 071
35.90
45.20
4
46.60
50.44
DDCoT 64
48.10 6.9
59.20 ::
67.20 n.s
68
1.15
34.
47,00 0:0
46.30 am
45.80 1.41
52.13
o31
58.20 1.1
65.10 n42
68.40 1.3
37.10 184
48.70
43.50 29p
43.40
CCoT 32
44.00 oas
55.00 1.11
63.60 nsr
68.30 1.
35.50
51.20 1:11
43.40 1 0.57
44.70 1.44
50.71
M3CaT (Ours)
45.50=
57.20 500
68.20 +
71.20 0.00
37.60
47.
53.80
49.20
53.78
relying on visual reasoning. The results in Thhl that all models exhibit substantially lower performance in the absence of visual information, confirming that the E3VQA benchmark requires ego-exo mmage-grounder reasoning rather than relying solely on textual cues.
C
Additional Experiments and Analysis of M3CoT
C.1
Generalization across Datasets
To validate the generalization of our proposed framework across different datasets, we further extend E3VQA using LEMMA [4], a multi-view dataset that captures goal-directed daily activities in home environments. Unlike Ego-Exo44, the ego images in LEMMA in standard rectangular format (i.c., rectified) without the dark peripheral regions. Following the same data generation pipeline in construct 500 additional samples evenly distributed across all categories.
Section 2
As shown in Table 7 the results demonstrate that our E3VQA benchmark construction pipeline generalizes well across datasets, indicating that the difficulty of ego-exo multi-view reasoning arises not from dataset-specifie visual properties but from inherent challenge of integrating complementary perspectives for visual understanding. Furthermore, the consistent improvements achieved by M3CoT across datasets highlight the robustness of its multi-perspective reasoning mechanism.
25

---

Table Performance comparison across question subsets grouped by required views.
LVLMs
Methods
Any
Ego
Exo
Both
Avg.
Default
65.20
63.96
58.59
39.28
56.76
DDCoT
69.71
69.37
60.83
37.68
59.40
GPT-4o
CoCoT
66.94
64.41
61.47
43.48
59.08
CCoT
67.94
67.50
61.05
42.03
59.63
M3CoT
73.45
74.30
64.23
52.61
66.15
Default
65.81
66.08
54.38
37.52
55.95
DDCoT
66.48
66.36
58.22
31.13
55.55
Gemini 2.0 Flash
CoCoT
67.23
65.43
56.44
32,43
55,38
CCoT
65.76
66.18
55.29
43.21
57.61
M3CoT
69.76
62.97
53.19
63.79
C.2 Evaluation on Open-Source Models
We present experimental results of our M3CoT prompting technique compared existing CoT methods on open-source LVLMs. Specifically, we apply M3CoT on Intern VL3-14B (55), the top- performing open-source model, and further evaluate the performance on Intern VL3-8B. As shown in Table g most CoT methods result in only marginal performance gains, with several failing to improve accuracy and even causing degradation in certain categories. This aligns with prior findings suggesting that the CoT method is often ineffective in smaller models with limited reasoning capability [41 19 7] Despite the limitations observed in smaller models, our M3CoT consistently achieves superior performance compared to other CoT methods, highlighting its robustness across model sizes.
C.3 Performance across Question Subsets Based on Required Views
We further compare model performance actoss the four subsets (Any, Ego, Exo, and Both) introduced in Section 5.2) to examine how our M3CoT approach provides advantages across different question types. Among these, performance the Both subset reflects model's ability to integrate multi-view information. Results on the Any subset reflect how well a model handles redundant or overlapping information, while those the Ego and Exo subsets indicate model's 's robustness when unneces- sary or irrelevant information is included in the input. As shown in Table 9 M3CoT consistently outperforms other CoT-bbaed methods across all subsets, demonstrating its overall effectiveness in multi-view reasoning.
C.4 Evaluation of Scene Graph Quality
The quality of scene graphs is often evaluated using the intersection-over-umie (IoU) with ground- truth scene graphs 46 52). However, ground- truth scene graphs often include overly exhaustive details that may not always be beneficial for answering questions accurately. better assess the practical utility of scene graphs generated by M3Co'T, we introduce two complementary metrics:
False Discovery Rate (FDR) is measured as the proportion of incorrect elements among the predicted scene graph components, including object classes, relations, and attributes. This metric measures how accurately the scene graphs represent the visual scene without including non-existent elements.
Answer Accuracy is measured by model accuracy when the generated scene graph serves as input to the model, indicating how effectively the graph supports reasoning when solving questions.
To evaluate the scene graphs with respect to these two metrics, we constructed subset of 120 samples from E3VQA, evenly distributed across all question categories. We then evaluated the scene graphs before and after the refinement step of M3CoT. After refinement, the FDR of the generated scene graphs decreased from 9.37% to 6.21 %, while the answer accuracy increased from 46.88% 60.42%. These results demonstrate that the refinement stage of M3CoT not only reduces errors in the scene graphs but also reorganizes their visual information to better support the model in generating accurate answers.
26

---

66.5
63.5
Accuricy
64.3
63.5
Exo&
Ego2Eso
ExoZigo
Voting
61,5
licratinn
Iieration
Iieration 2
Dicration
Figure 9: Performance across different perspectives and majority voting results over iteration steps.
Table 10: Comparison of computational overhead between M3C0T and existing CoT prompting methods on GPT-4o and Gemini 2.0 Flash.
L.LMss
Method
Token Usage
Time (s)
Acc. (%)
Default
2,120.25
2.62
60.90
DDCcT
6,807.18
9.57
64.43
GPT-4o
CoCoT
2,349.23
6.25
62.87
CCoT
9,225.30
22.59
63.74
M3CoT
47,250.12
37.56
68.58
Default
790.60
2.56
59.80
DDCoT
2,752.09
5.48
60.31
Gemini 2.0 Flash
CoCoT
950.71
3.84
61.09
CCcT 2,817.97
7.14
60.18
M3CoT
26,164.36
20.03
66.12
C,5 Analysis of Iteration Steps in Multi-Agent Scene Graph Refinement
To analyze the effect of iteration steps in M3CoT, we report the accuracy each individual perspective well as the majority-vote answer derived from them at each iteration. As shown in Figure g without any information exchange across perspectives, all individual perspectives and the majority- voted answer achieve relatively low accuracy (iteration 0). As agents begin exchange their scene graphs, we observe steady improvement in the accuracy of each individual perspective, suggesting that iterative refinement facilitates mutual enhancement through shared contextual understanding, This process also leads to corresponding increase in voting accuracy, reflecting not only the enhanced quality of individual predictions but also stronger consensus across perspectives. However, beyond the second iteration, we find that both individual accuracy and voting accuracy plateau. We attribute this saturation the convergence of information across agents: while initial iterations benefit from the diversity of complementary perspectives, excessive alignment diminishes the gains from their integration. This observation highlights a trade-off in our pestp--perspective refinement strategy between refinigg individual scene representations and preserving representational diversity. Note that all experiments and analyses in this paper are conducted with fixed iteration count of using Gemini 2.0 Flash unless otherwise specified.
C.6 Analysis of Computational Cost
To examine the trade-off between computational efficiency and performance, we measure inference time, token usage, and accuracy of recent CoT prompting methods, including M3CoT, on GPT-4o and Gemini 2.0 Flash (see Table 10). Please note that the reported inference time may be unreliable due to variability in API response, so it should be interpreted as general trend rather than an exact value. While multi-perspective reasoning requires additional computation, enables a more accurate understanding than other recent prompting methods.
27

---

D Qualitative Examples of M3CoT
D.1 Comparison with Other CoT Methods
We provide additional examples that illustrate how M3CoT improves reasoning compared other CoT approaches (see Figure T0
D.2 Scene Graphs from Three Perspectives
To further examine how different perspectives in M3CoT contribute to capturing complementary infor- mation, we present additional qualitative examples scene graphs derived from each perspective. As shown in Figures strengths depending on the question, particularly regarding which image should be referenced to answer it.
22 and [3 the scene graphs from the three perspectives exhibit complementary
D.3 Conflict Resolution during Refinement
We provide qualitative examples illustrating how M3CoT resolves conflicts among scene graphs from different perspectives during the refinement stage (see Figure 14).
D.4
Typical Failure Cases
We provide qualitative examples illustrating the failure cases of M3CoT (see Figure 15)
E Prompt Templates
E.1 Prompt Templates for E3VQA Construction
To guide LVLMs in understanding the question categories and tasks for generating meaningful question-answen pairs, we carefully design the prompts for each stage. To generate question-answer pairs from a single viewpoint, generation, we apply the prompts in F 26 For response-basee filtering, we use the prompts shown in Figure 34 and 335 Finally, generate four candidate options from either the ego exo image, we use the prompts illustrated Figure andand
use the prompts shown in Figure 25 For view-specific response
E.2 Prompt Templates for Baselines and CoT Methods
The system and user prompts used in the baseline experiments of E3VQA are shown in Figure 38 In addition, for M3CoT, the prompts for scene graph generation from each perspective are shown in
Figure 41 and the prompts for scene graph refinement across agents are presented in Figure 42 The prompts used other CoT baselines are shown in Figure 43
F
Limitations
Despite its contributions, this work has several limitations. First, the E3VQA benchmark is constructed solely from the Ego-Exo4D and LEMMA datasets, which may exhibit dataset bias and limited generalizability in diverse real-world scenarios. Second, although the queries and answer options in E3VQA are carefully crafted, they may not fully capture the diversity of natural language expressions and user intents encountered in real- world interactions with visual AI assistants. Third, while recent AI APIs offer a solution for scaling the benchmark, their use entails substantial financial costs. Fourth, M3CoT introduces increased computational overhead due to its multi-step reasoning across multiple perspectives, which may limit its applicability in since E3VQA is constructed from images rather than videos, the benchmark may not fully assess an LVLM's s ability to capture temporal cues and motion dynamics, aspect leave for future work.
scenarios. Finally,
28

---

G
Ethics Statement
This work has the potential positively impact society by enhancing the capabilities of visual assistants and embodied Al systems, particularly in scenarios that require comprehensive scene understanding from both egocentric and exocentric views. Such advancements may enhance human- Al interaction and improve support in assistive technologies. However, the use of egocentric visual data may raise important privacy concerns, especially sensitive environments. We acknowledge these risks and emphasize the importance of implementing safeguards and transparency mechanisms future deployments. As part of our commitment responsible data use, we have obtained the appropriate licenses from the contributing institutions for the of the Ego-Exo4D dataset in this research.
29

---

right
amsgee?
secend umage. Sub question Which the answer choices is the right of sink?
Final Answer A)
Sub Answer The curting board is on the counder the right of the sinik
Scone graph:
OCeT Final Answer A)
"pesso"; "relation" "to right "turget" "sank"), right "target" ]
["objeet" "cunting
Whet object of relative
graphe
Cunting board Blender
Stave Tounter
Finall Answer R)
[[ [["bjees" | (1"A") "passc"""s"c""" "relatio"" "near", "Trrett) "sitk" J. idt: "stove"... "telltion": "Lo right of", "uarget: "sink"), [..])
Sub question 1: Where the blue plate located? Sub Answer The buue located on kitchen icland.
DDCaT Finsl
plate?
Answer cooking ulanal
plate
CCoT Final Answer AA
[ ("name": plate
"Blue" "relation" "on" "same" Book" "relation" on "uugge" "; plato "J]
graph:
"uuget"
View
Eso View
Which couking stznnil the blze plate pootioned the kincen inkind?
Scene graph:
M3CoT Final
(""obects" [["mmme:: "isladd" "attribute" kiichen"), ("nnmm": " plate" stribute" "blue" "sdaaiin"
Fork Spatuts
Piller
()
island
"patuls"
Kail
"elltion" "arget plate "]]...]]
Figure 10: Qualitative examples of answers and reasoning processes generated by different prompting methods.
["abbecss" [["M", "person"" "dosciption" "panse", yonr "smmmarr"" ("loal ("honalppople" 1)
Anweer
Eun2Fgn Final Answers D)
Sccse graph:
[["abjcce" [["M"" "person "hame" "penson", "count: .....]
Ego View
Evo View
Scane graph:
How the scene?
Ego&Exo Auswer
["ubjests:: [["A": "penson"" "penspotive" "insgel "), "poran2" "penpection" "image
"persportvve" "imaget ",, "persond", "penspective" "image2"
Sccne graph:
FigilFso Final
(("ojjens' [C'M" "penson"", "mome" "penson" "snmilutes": (ssanding" "relatiesshpp"" [["sssss""""""""""" "personl".
graph:
Eun2Fgo Final Answer D)
[ dbjjesss" [["l", "yesen"", "attrinuie" ("wcuring glasace" "wazinng
white shirt T: "id":'peroon"", "utmibute": "sanding") ('id": "persona" "stributos"
"wininng '[], ........ "summary" "There see theee people the some
Ego View
View
graph:
many people
Fgn Final Answer B)
"ubjozs" [1"l" "peren"" "penpoctive" "image"" J. ('il": "prrsn?" "perpecctvv"" "image/") "), ("58:: "person!", "perspective:: "imagel " (itt: "macr" "perspctioo" "image?")). "ootl poople":4)
A)2
BB
C1
D
Figure 11: Qualitative examples of answers and reasoning processes generated by different perspec- tives. The scene graph from the Ego&Exo perspective demonstrates strong capability to capture the information necessary for answering questions grounded in both ego and exo views.
30

---

Sccae
Ego2Fse Auswer :A)
[""ajjast" [1"W" light. shirf' "sttribnne".
"light .... ("50:: "prrson, crossing arms' ....l
"hands resting by sides"))
EuaiEgg Finall Answer:
Sccse gruph:
[jrjaan [[ "rame"" "penan"" "sttriutes" "wearing light bls shin") ("hame"" "amms", "relation": "crossed", "arget" "person"]]
Ege View
Exo View
Scame graph:
the light blue shirt postiinnd??
EgodEsa
(joobeers"" [('dd" "personi" "description" light biue shint,
Crteiing sides
cale"": "Tighe blue", "position" "sonding"
leaning buller
"crossng
EgolFise Final Answer R)
[ ["abjeco": : [["d": "person", "description" "penson playing violin" ..... I"ld": "tiipdf" "relation:: "No the right of", "srget" "penort")
Sccne graphe
FxullFgg Final
Scane graph:
[d"wlyects' i [["id" "penon"; "mlaation" "playing", "target" "violin"), "person")
('Mt: "uipod". "relation" "ln
View
Exo View
graph:
object the person playing Music C) Light switch
Egn&EEo Final Answer : C
[""éjjuss"" [[",", "tipal", "relation: the right of", "angel" "peron") J.... (Mler "light monch" "relatio" rightighttof" "urget": "person" [B]
Tripod
D) Tinsh
Figure 12: Qualitative examples of answers and reasoning processes generated by different perspec- tives. The scene graph from the Ego2Exo perspective demonstrates strong capability to capture the information necessary for answering questions grounded in the exo view alone.
Scene ("mame" "manf, "attiuas" shin", "Sining"] "relation: "holding". "uappt":
Ego2Eso
"let"", ("tame":
"relatine" "bege" "uble"]]]
Evo2Ego Final Flswer Answer ()
Scene gruph:
[P"obiccts" : l"namc"; "mun" "attrbutee" (">itinn",, "light gramshin")) ("relorion" "holding", "torer" "nimer H:Bl.... ....
Ego View
Fae View
Seene graph:
Wher the man light holding his left hand?
EpodEis
[f"obyocte' [["let "min", "dosription" "Min in light shin!. "sarthalca" oske": "ligh goeee"), ("Wt, Néjjctt Junal ("tpe": "swab")
Box
B) Instruction manual
HgooExoo Final Auswer C
[objects: ( ("name": "dyying pan", "relation"; " "lac", "urget": "window" ["name" "ges cylinite 'velation", "raoe" "brge"" "widoww []]]]
Secne grapli:
Eso2Ego
Seene graph:
[fobjacic [ ["hume" "windo""
("sume": "fying "rdutionn
"aange") "window"
Ego View
Exe View
EgodEEs Anvwer C)
.... ("relatioshhig": (l'souce", "fiying pus", "urger" Type" Mar from"), ("some" "pee rylinder" "targis" "windmn" "ype": "near") [i]
Frying
By Catting
Gas cylinder
Dy Knife
Figure 13: Qualitative examples of answers and reasoning processes generated by different perspec- tives. The scene graph from the Exo2Ego perspective demonstrates strong capability to capture the information necessary for answering questions grounded in the ego view alone.
31

---

ExoREEo
Scene graph:
[[bbeett:] ("SP", "bieyele", "doscriptinn" "bicycla being worked on"" "suribusee" ("wheels prached"" 1111111
Answer
FaalFign
Scene graph:
[[objects: (1" l'obyeet": "wheel. "relation" "attnched to", "sampe"" "bicycle"),
Answer:
"'oject" "wheel "velation" "uttached "sarge"" "bicyele",
Scene
[iobject::
bicycle",, ", ("id": "deacriptoo"" "Bom of bicyck"" "]]1
"descrimtin"
Ego
Exo View
A
How many wheels are uttached the bicycle?
19
Exe2Uggo Anowert
[lobjests: ("'s";
graph:
"tarae"" "Nicycle")
I'ld": "witeel "relatio": "trachhd to", "urget": "bicycle") ).
Upn2Uixo Answer B)
Sccne graph:
[[objett: (1"a") "pensr") "doasippinn" "wearing pny shin". "rcleiondipe" ("pecaliaai"" "deing" "uljez"" "wasking disis"
Exa22igo Answer: C)
Scene graph:
[[objett:]
("ojjcct" "hands"; "velation": "dnisg", "largef" "stining with
squen")
EgodExo
Scene ("S)", "pense",
"wearing aray
Ego
"relatinddip"" ("perdicate" "deing" "olyen"" "siiing
spoon" []]]]]
Q Whet is the peraa the gray shirt doiing with their hande?
A) Cleaning the coutattop Stirring with
B] Washing distes
FoetFigo
Scene graph:
"wearing gny "relationhips: ("yedieau"" "dking". "ubject" "Mtiing with spont"]
[[objects: "'obect"" "person";
Scrubbing
Figure 14: Qualitative examples illustrating how conflicts among graphs from different perspec- tives are resolved by M3C0Т. Scene graphs with inaccurate elements are refined during M3CoT's refinement stage, yielding accurate and consistent representations.
graph:
[t (T'byeecs" P"oject" "boul". "yello"" "J.... ('abjec"" "paree"" "eddaiio" "uxing". "beng"" "kmifc"
Final Ammwr
ExutEgo Final Aniwer :C)
Scene graph:
[[ [Fuhhocs" [["ubjez" "peran"", "relation" "teing", "bega"" "Kmife") !'objctt': "small bowl", "atmbunes": ppeen")]l]]
Figo View
Ese View
What tiem using exok?
Egodtiso Final
[['objett" [[idd; "person". "description";
Scmne graph:
cooking". "relationship"::
Knife
Spanula
Spune
[F'epe" "USING", "arget" "knife"]]) []]
Gio2lixo Final Answer
[f'obects" ["objecc": "cuting board", "relasion": left
Serne graph:
"I") ("ubjed" "Insh "selation" "withe lall "ungel"
ExuZEgo Finall
[[ ("obyszi"" [["i") "cutting
"lacation" "right res""), ("W" "insh. can" "perspective" "image"" "location": the left from use"")))
graph:
Ego
Scene graph:
Q. What object is my left?
FgoRFFso Final Answer :A)
[| [1"sddiinssppp"" "[''ubbee"" "cuting boand", "relation" "le the right
A) Catting hoted Rocipe
Ry Trash can D) lonell mill
"object": "use"".... ""subject", "wasa, can", "selatio"" "to the left "abjoot" "Iae""
FgolFFin Answer:
Scene graph:
[f ("objects" : [["Sd" "hand", "attibutes" ["right]] "relationship"' [" p"redicate"" "reaching" "ubjec"" "Surtace"))
FgoREiso Answers A)
Serne graph:
[I [f'retiionssps' [... "iyp"" "holding"
("source") "right, hond", "tarer" "Sppys_pppiinnee
Egollixo Final
Seene graph: [P"elzzionshps's ("source": "right band", "targe" "Ipp""
Eigo
Q What and doing with my right hand?
A) Speaying Cleanin
10 Reaching
Egodtiio
Scene graph:
("sslzionshpps" ("souree" hand", "targe"
D) Pointing
"Ipp" "helding"
Figure 15: Qualitative examples illustrating M3CoT's failure cases. The top shows examples where all three perspectives produce incorrect predictions. The middle row illustrates cases where the scene graphs correctly generated but fail to yield accurate answers. The bottom shows cases where the refinement stage fails due to erroneous scene graphs.
32

---

Egocentric Single-View View QA Generation Prompt
(Ego Image)
You are given the visual input from the camera worn by the user (referred to '1').
Based on this visual input, generate three question- answer pairs. Ensure that the generated question answer pairs are directly based on the visual input.
[Category-wise Prompt]
Requirements:
Each question must explicitly include the pronoun 'I" or 'me' to ensure the focus remains on the user Each answer should be single word Ensure that all three question-answar pairs meet these criteria and are relevant the visual input. Strictly adhere to the format of the provided examples.
short phrase.
Figure 16: Egocentric single-view QA generation prompt.
33

---

Egocentric Single-View QA Generation Prompt: Action & Pose
Instructions:
Each question must focus on my actions, body posture, or gestures The answer must be a verb or verb phrase (e.g.. writing stretching, crossing arms).
Do not generate QA pairs with overly generic answers like 'standing" or 'reaching"
Question Categories & Templates:
Actions (What am doing?)
What am doing?
What am doing with my [body part]?
Body Posture (How am positioned?? ?)
How is my body positioned?
How am sitting standing lying?
What is my posture?
Gestures (What movement am I making
What am doing with my hands?
What gesture am making?
How am moving my arms/ arms/legs head?
Examples:
Q: How is my body positioned?
A: Sitting cross -legged
What am doing with my left hand?
A: Holding book
What gesture am making? A: Waving
Figure 17: Egocentric single-view view QA generation prompt: Pose & Action.
34

---

Egocentric Single-View QA Generation Prompt: Object & Attribute
Instructions:
Each question must focus identifying specific object (e.g., mug cup, laptop) or describing an attribute of an object (e.g.. navy blue striped pattern) associated with me.
The answer must be noun or noun phrase, avoiding overly generic responses such as something or object.
Question Categories & Templates:
Object Identification (What am I interacting with ?)
What am holding?
What object is on the table beside me?
Which item I picking up?
Object Attributes (What does it look like ?)
What color is the shirt
wearing?
What pattern is on jacket?
What type of shoes am wearing?
Examples:
Q: What color is the shirt I am wearing?
A: Navy blue
Q: Which object am holding in my right hand?
A: small notebook
Q: What pattern does my sweater have?
A: Checkered pattern
Figure 18: Egocentric single-view QA generation prompt: Object & Attribute.
35

---

Egocentric Single-View QA Generation Prompt: Spatial
Instructions:
Each question must focus on the spatial relationships between me and objects in my surroundings. The answer must be a specific object or location descriptor (e.g., coffee cup, bookshelf, under the table). Do not generate QA pairs with overly generic answers.
Question Categories & Templates:
Object Proximity (What is closest or farthest?):
What object is closest to me?
Which object is the farthest from me?
What is the nearest object my [body part]?
Relative Positioning (Where are objects located ?)
What object is to my left/right/fron fright//root/bbhind
Which object is above/below me?
Spatial Relations (How are objects arranged ?)
Which object is between me and [another object]?
Examples:
Q: What object is closest to my left hand?
Coffee cup
Q: Which object is the farthest from me?
A: Bookshelf
Q: What object is on right side?
A: Tissue
Figure 19: Egocentric single-view QA generation prompt: Spatial.
36

---

Egocentric Single-View QA Generation Prompt:
Numerical
Instructions:
Each question must focus on numerical reasoning by counting or quantifying specific elements directly related to me. This may include the number of people, objects, or other countable items present in my surroundings.
The answer must be a numerical value that accurately represents the count of the indicated elements.
Do not generate questions about overly generic objects (e.g., items, objects).
All numerical answers must be within the range of 0 to 5.
Question Categories & Templates:
Counting People (How many people
around me?)
How many people are in the image excluding me?
How many individuals are facing the same direction as am?
Counting Objects (How many things are near or with me?)
How many [objects] ] am I holding?
How many [items] are on the table beside me?
Quantitative Comparisons (How do the numbers compare to what have? ?)
How many more books are on my desk than on the shelf?
By how much does the number of items in my hands exceed the number on the table?
Examples:
Q: How many people are in the image excluding me?
A: 3
Q: How many more bowls are my table compared to the table behind me?
Q: How many apples am holding?
A: 3
Figure 20: Egocentric single-view QA generation prompt: Numerical.
37

---

Exocentric Single-Vier View QA Generation Prompt
[Exo Image)
You are given with the visual input from fixed ppsition camera capturing scene.
Based on this visual input, generate three question -nnswe pairs Ensure that the generated question -nnser pairs are directly based on the visual input.
[Category-wise Prompt]
Requirements:
Each answer should be a single word or a short phrase.
Ensure that all three aanswrr pairs meet these criteria and are relevant to the visual input. Strictly adhere to the format of the provided examples.
Figure 21: Exocentric single-view QA generation prompt.
Exocentric Single- View
View QA Generation Prompt: Action & Pose
Instructions:
Each question must focus on the actions, body posture, or gestures within the scene.
The answer must be verb verb phrase (e.g., writing stretching, crossing arms)
Do not generate QA pairs with overly generic answers like 'standing' or 'reaching'.
Question Categories & Templates:
Actions (What is the person doing?)
What is the [descriptive] person doing?
What is the [descriptive] person doing with their [body part]?
Body Posture (How is the person positioned?)
How is the descriptivee person positioned?
What is the posture of the [descriptive] person?
Gestures (What movements is the person making
What kind of gesture is the [descriptive] person making?
How is the [deccriptive] person moving their arms/ legs head?
Examples:
Q: What is the man sitting in the chair doing?
A: Watching phone
Q: What is the posture of the person wearing a green shirt?
A: Raising one arm
Q: What is the woman in the black jacket doing with their right hand?
Holding book
Figure 22: Exocentric single-view QA generation prompt: Action & Pose.
38

---

Exocentric Single-View QA Generation Prompt: Object & Attribute
Instructions:
Each question must focus on identifying specific object in the scene (e.g.. 'mug cup', 'laptop or describing an attribute of an object (e.g., 'navy blue', 'striped pattern
Questions should reference people or objects by descriptors (e.g., the woman in the white top 'tee man with the striped shirt '). The answer must be a noun or noun phrase, avoiding overly generic responses such as "something" or 'object'.
Question Categories & Templates:
Object Identification (What is present?)
What is the man with the striped shirt holding?
What object is placed on the table?
Which item the woman wearing blue top picking up?
Object Attributes (What does it look like ?)
What color is the shirt worn by the man wearing a cap?
What pattern is on the jacket worn by the woman carrying handbag?
What type of shoes is the man standing near the window wearing?
Examples:
Q:
What color is the top worn by the woman holding the towel?
A: White
Q: Which object is the man in the black shirt holding in his right hand?
A: Smartphone
Q: What pattern does the sweater worn by the person holding cup have?
A: Checkered pattern
Figure 23: Exocentric single-view QA generation prompt: Object & Attribute.
39

---

Exocentric Single- View View
QA Generation Prompt: Spatial
Instructions:
Each question must explicitly reference an object 's or a person ss spatial relationship within the scene.
The answer must be specific object location descriptor (e.g.,
scissors, frying pan, under the table). Do not generate QA pairs with overly generic answers.
Question Categories & Templates:
Object Proximity (What is closest or farthest?)
Which object is closest to the person wearing [specific item]?
Which object is the farthest from [reference point]?
What is the nearest object to [specific location or object]?
Relative Positioning (Where objects located
What object is to the
of the man with
specific item]?
What object is the left/ nniight/frontbbhiieee [reference object]?
Which object is positioned above /below [reference object]?
Spatial Relations (How are objects arranged?)
Which object is positioned between [object A] and [object B]?
What item is placed underneath/inse inside [object]?
Which object is located between the two people sitting on the \ bench?
Examples:
Q: What is the object the far right of the desk?
A: Scissors
Which cookware is closest the woman wearing a striped shirt? A: Frying pan
What object is placed directly in front of the man wearing a cap?
A: Backpack
Q: What object is placed underneath the table?
A: Storage box
Figure 24: Exocentric single-view QA generation prompt: Spatial.
40

---

Exocentric Single-View View QA Generation Prompt:
Numerical
Instructions:
Each question must focus numerical reasoning by counting quantifying specific elements within the scene This may include the number of people, objects, or other countable items present in the image.
The answer must be numerical value that accurately represents the count of the indicated elements.
Do not generate questions about overly generic objects (e.g.. items, objects).
All numerical answers must be within the range of 0 to 5.
Question Categories & Templates:
Counting People (How many are there?)
How many people are in the scene?
How many individuals are facing the camera?
Counting Objects (How many things are visible
How many objects is [person descriptor] holding? the table?
How many items
Quantitative Comparisons (How do the numbers compare?) ?)
How many more books are on the table than on the shelf?
By how much does the number of items in the man's s hands exceed the number on the table?
Examples:
Q: How many people are in the scene?
A:
Q: How many objects is the woman in the striped shirt holding?
A:
Q: How many oranges are placed on the table?
A: 5
Figure 25: Exocentric single-view QA generation prompt: Numerical.
41

---

View-Specific Response Expansion Prompt: Ego View
(Ego Image)
You are given visual input from a camera worn by the user (referred to as ')') along with a corresponding question.
Based the visual input, generate the best possible answer.
(Category-wise Prompt)
Requirements:
Each answer option should be single word or a short phrase. Follow the provided format strictly.
Q: [Question)
Figure 26: View-specific response expansion prompt: Ego view.
View-Specifie Response Expansion Prompt: Exo View
(Exo Image)
You are given visual input from a fixed camera capturing scene along with corresponding question.
Based the visual input, generate the best possible answer.
(Category-wise Prompt)
Requirements:
Each answer should be single word or a short phrase. Follow the provided format strictly.
Q: [Question]
Figure 27: View-specific response expansion prompt: Exo view.
42

---

View-Specific Response Expansion Prompt: Both Views
(Ego Image]
[Exo Image)
You are provided with two visual inputs in sequence, each captured from a different perspective:
1. The view from the camera worn by the user ('I').
2. The view captured by external camera observing the user ('I). These two images capture the same event at the same time. Based on the visual inputs, generate the best possible answer
[Category-wise Prompt)
Requirements:
Each answer should be single word short phrase. Follow the provided format strictly.
Q: [Question]
Figure 28: View-specific response expansion prompt: Both views.
View-Specific Response Expansion Prompt: Text Only
Based on the question, generate the best possible answer.
[Category-wise Prompt)
Requirements:
Each answer should be single word or a short phrase. Follow the provided format strictly.
Q: [Question]
Figure 29: View-specific response expansion prompt: text only.
View-Specific Response Expansion Prompt: Action & Pose
Instructions:
The answer must be verb or verb phrase (e g., writing stretching,
crossing arms)
Do not generate overly generic answers like "standing 'reaching
Output format:
How is my body positioned?
A: Sitting cross-legged
Q: What is the man sitting in the chair doing?
A: Watching a phone
Figure 30: View-specific response expansion prompt: Action & Pose.
43

---

View-Specifie Response Expansion Prompt: Object & Attribute
Instructions:
The answer must be a noun or noun phrase, avoiding overly generic responses such as 'something" or 'object".
Output format:
Q: What color is the shirt I am wearing?
A: Navy blue
Q: What color is the top worn by the woman holding the towel? White
Figure 31: View- View-specific response expansion prompt: Object & Attribute.
View-Specific Response Expansion Prompt: Spatial
Instructions:
The answer must be a specific object location descriptor (e.g., coffee cup, bookshelf, under the table).
Do not generate overly generic answers.
Output format:
Q: What object is closest to my left hand?
A: Coffee cup
Q: What is the object on the far right of the desk?
A: Scissors
Figure 32: View-specific response expansion prompt: Spatial.
View-Specific Response Expansion Prompt: Numerical
Instructions:
The answer must be a numerical value that accurately represents the count of the indicated elements.
All numerical answers must be within the range of 0 to
Output format:
How many people are in the image excluding me?
A: 3
How many people are in the scene?
A:
Figure 33: View-specific response expansion prompt: Numerical.
44

---

Response-Based Question Filtering Prompt
Here is the question: '(Question)
The provided answer is (answer both), and the given label is (answer _arr__nit_... Do they convey the same meaning based on the question? Respond with a single word phrase.
Figure 34: Response-basee based question filtering prompt (1).
Response-Based Question Filtering Prompt 2
Here is the question: "[Question) '.
The provided answer is (answer_text) and the given label is '(answer_ini) init]
Do they convey the same meaning based on the question? Respond with single word or phrase.
Figure 35: Response-basee based question filtering prompt (2).
Option Generation Prompt: Ego
(Ego Image)
You are given visual input from camera by the (referred to as 'I').
Based on the following question and answer, generate four multiple- choice options.
Question: (Question)
Answer: (answer_ego)
Ensure that each incorrect option is closely related to the visual content, making it challenging to easily identify the correct answer. Follow the format below exactly:
Options:
[Option1] ]
[Option2]
[Option3] ]
[Option4]
Figure 36: Option generation prompt: Ego.
45

---

Option Generation Prompt: Exo
(Exo Image) You are given visual input from a fixed-position camera capturing a scene.
Based the following question and answer, generate four multiple- choice options.
Question: [Question]
Answer: (answer_exx) exo]
Ensure that each incorrect option is closely related to the visual content, making it challenging to easily identify the correct answer. Follow the format below exactly:
Options:
[Option1] ]
[Option2]
[Option3]
[Option4]
Figure 37: Option generation prompt: Exo.
46

---

System Prompt Question (Instruction) Prompt
System Prompt
You are a helpful assistant. You are provided with two visual inputs in sequence, each captured from different perspective:
1. The view from the camera worn by the user (1)).
2. The view captured by an external camera observing the user ('I)).
The first image shows what the user ('I') sees from their perspective
The user's 's full body cannot be visible; you may only see parts their body, like their hand, foot, or arm, or in some cases, none of the user 's body at all.
The second image shows both the user and the environment from third -person perspective with broad view.
The user's 's full body is visible, but due to the fixed viewpoint, some parts may not be visible.
These two images capture the same event at the same time. Your task is analyze both images along with the question and provide the most accurate response based on the visual information from both perspectives.
Question (Instruction) Prompt
(Ego Image)
(Exo Image) (Question)
Only one option is correct.
Present the answer in the form X)
Figure 38: System Prompt and Question(Instruution)) Prompt.
47

---

M3CoT Prompts EgooExoo Perspective
Scene graph generation phase (Ego2Exo)
Task
For the provided image and its associated question, generate scene graph in JSON format that includes the following:
1. Objects that are relevant to answering the question.
2.
Object attributes that are relevant answering the question.
3. Object relationships that are relevant to answering the question.
Just generate the scene graph in JSON format. Do not say extra words.
(Ego Image)
[Question Prompt)
Scene graph refinement phase (Ego2Exo)
Task:
For the provided image from different view and the scene graph generated from the previous view, refine the scene graph in JSON format as follows:
1. Review and Update Existing Objects and Relationships:
Examine the objects and relationships in the initial scene graph. Update their attributes views. Remove only elements that are clearly erroneous (e.g.. annotation errors or duplicates).
positions based observations from both
2. Incorporate New Information: Identify and add any new objects relationships that appear in the new view.
3. Align and Reconcile Across Views:
For overlapping objects and relationships, align them using spatial proximity and semantic similarity If attribute discrepancies arise, select values that best reflect the combined observations.
Ensure that the updated scene graph is logically and physically consistent, avoiding contradictions or impossible configurations. Just generate the refined scene graph in JSON format. Do not say extra words.
(Exo Image) [Question Prompt)
(Assistant's response(Ego-ond only SG))
Initial question response phase (Ego2Exo)
Use the images and the refined scene graph context and answer the following question.
(Ego Image)
(Exo Image)
[Question Prompt)
(Assistant's response(Refined SG))
Figure 39: M3CoT prompt (1).
48

---

M3CoT Prompts Exo2EEgo Perspective
Scene graph generation phase (Exo2Ego)
Task
For the provided image and its associated question, generate scene graph in JSON format that includes the following:
1. Objects that are relevant to answering the question.
2.
Object attributes that are relevant answering the question.
3. Object relationships that are relevant to answering the question.
Just generate the scene graph in JSON format. Do not say extra words.
(Ego Image)
[Question Prompt)
Scene graph refinement phase (Exo2Ego)
Task:
For the provided image from different view and the scene graph generated from the previous view, refine the scene graph in JSON format as follows:
1. Review and Update Existing Objects and Relationships:
Examine the objects and relationships in the initial scene graph. Update their attributes views. Remove only elements that are clearly erroneous (e.g.. annotation errors or duplicates).
positions based observations from both
2. Incorporate New Information: Identify and add any new objects relationships that appear in the new view.
3. Align and Reconcile Across Views:
For overlapping objects and relationships, align them using spatial proximity and semantic similarity If attribute discrepancies arise, select values that best reflect the combined observations.
Ensure that the updated scene graph is logically and physically consistent, avoiding contradictions or impossible configurations. Just generate the refined scene graph in JSON format. Do not say extra words.
(Exo Image)
[Question Prompt)
(Assistant's response(Exo-only SG))
Initial question response phase (Exo2Ego)
Use the images and the refined scene graph context and answer the following question.
(Ego Image)
(Exo Image)
[Question Prompt)
(Assistant's response(Refined SG))
Figure 40: M3CoT prompt (2).
49

---

M3CoT Prompts Ego&Exo Perspective
Scene graph generation phase (Ego&Exo)
Task:
Using the provided two images and their associated question, generate a unified scene graph in JSON format that includes the following:
1. Objects that are relevant answering the question.
2. Object attributes that are relevant to answering the question.
3. Object relationships that are relevant to answering the question.
4. Ensure that objects and relationships from both perspectives are appropriately aligned, integrated, and refined to provide complete scene representation.
Just generate the unified scene graph in JSON format. Do not say extra words.
(Ego Image) (Exo Image)
[Question Prompt)
Initial question response phase (Ego&Exo)
Use the images and the unified scene graph as context and answer the following question.
(Ego Image) (Exo Image)
[Question Prompt]
(Assistant's Response(Ego8Exo SG))
Figure 41: M3CoT prompt (3).
50

---

M3COТ Prompts- SG Refinement between Agents
Scene graph cross-refinement phase (Ego&Exo /Ego2Exo /Exo2Ego)
Task:
Below are different scene graphs generated using different reasoning
methods:
One scene graph: (Ego2Exo SG) / (Exo2Ego SG) / (Exo&Ego SG]
One scene graph: (Exo2Ego SG) / (Exo&Ego SG) / (Ego2Exo SG)
Using the scene graphs generated from different methods as additional context, generate refined scene graph in JSON format for the provided images and their associated question as follows:
1. Review the objects and relationships from the scene graphs and make any necessary adjustments to better align with both views. 2. Ensure that overlapping objects or relationships between the two views are appropriately aligned and refined, enhancing the accuracy of the scene graph.
Just generate the refined scene graph in JSON format. Do not say extra words.
(Ego Image) (Exo Image)
[Question Prompt)
Question response phase (Ego&Exo /Ego2Exo /Eg02xxo//xx2390 /Exo2Ego)o
Use the images and the unified scene graph context and answer the following question:
(Ego Image) (Exo Image)
[Question Prompt)
(Assistant's response(Unified SG))
Figure 42: M3CoT prompt (4).
51

---

Other CoT Prompts- DDCoT
For the provided images and their associated question, think step-by- step about the preliminary knowledge required to answer the question. Deconstruct the problem as completely as possible into necessary sub- questions.
Then, with the aim of helping humans answer the original question, attempt to answer those sub-questions.
The expected answering format is as follows:
Sub-questions:
1. <sub-question 1>
2. <sub-question
Sub-answers:
1. answer 1>
2. -ssueanswwr 2>
[Question Prompt)
Context:
(Assistant's response)
Give your answer the question according the sub-questions and sub-answers.
[Question Prompt)
Figure 43: DDCoT Prompt.
Other CoT Prompts CoCoT
Please tell me the similarities and differences of these two images, and answer to the question.
[Question Prompt)
Figure 44: CoCoTT Prompt.
52

---

Other CoT Prompts CCoT
For the provided images and their associated question, generate scene graph in JSON format that includes the following:
1. Objects that are relevant to answering the question.
2. Object attributes that are relevant to answering the question.
3. Object relationships that are relevant to answering the question.
Just generate the scene graph in JSON format. Do not say extra words.
[Question Prompt]
Scene Graph
(Assistant's response)
Use the images and scene graph as context and answer the following question.
[Question Prompt)
Figure 45: CCoT Prompt.
53


Reviews:
[{"review_id": "u4dAVliRHx", "reviewer": "Reviewer_BgYM", "summary": "This paper introduces E3VQA, a novel benchmark designed to evaluate large vision-language models (LVLMs) on multi-view question answering that combines egocentric and exocentric perspectives. Recognizing the complementary nature of both views, fine-grained hand-object interactions from ego and spatial context from exo. The authors argue that current LVLMs fail to reason effectively across both.\n\nThey additionally propose M3CoT, a training-free prompting technique that uses multi-perspective scene graphs and iteratively fuses them for improved reasoning. Extensive experiments on E3VQA show that M3CoT yields consistent and significant performance gains over recent prompting baselines across multiple LVLMs including GPT-4o and Gemini 2.0 Flash.", "questions": "1. Can M3CoT be applied in real-time systems (e.g., AR devices), or is its inference-time cost too high? Consider discussing latency and compute implications of multi-agent prompting.\n2. Is there a risk of conflicting information between scene graphs (e.g., ego says “holding a cup,” exo says “not holding anything”)? How is this resolved during refinement?\n3. Do you leverage any visual feature alignment or object detection models (e.g., pose estimation) to help match entities across views, or is everything handled purely via language prompting?\n4. Do you expect E3VQA-trained models or the M3CoT prompting approach to generalize to other ego-exo datasets like Charades-Ego or OpenEQA?\n5. Some qualitative examples of M3CoT failure or ambiguous scene interpretations would help readers understand the boundary of your method's capability.\n6. How does your model handle ambiguous or inherently subjective questions (e.g., “What am I likely to do next?”) where multiple answers might be plausible?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n1. Timely and Novel Benchmark: It is a first-of-its-kind dataset focusing on ego-exo reasoning, addressing a growing need in augmented reality, embodied AI, and multimodal understanding.\n2. The dataset includes 4K QA pairs evenly distributed across four categories (pose, object, numerical, spatial), with fine-grained filtering and expert human verification.\n3. The multi-agent, multi-perspective scene graph construction and refinement mechanism is original, cleverly designed, and provides demonstrable performance improvements.\n4. M3CoT significantly outperforms multiple prior CoT prompting methods across a range of LVLMs and question types. The performance analysis is thorough and includes ablations.\n5. The paper includes implementation details, error bars, dataset access, compute info, and a discussion of limitations and ethical concerns.\n\n### Weaknesses:\n1. While the E3VQA benchmark is impactful, its reliance on a single source dataset (EgoExo4D) may limit its diversity and generalization across real-world applications.\n2. Although M3CoT is \"training-free,\" its reliance on multiple inference passes with different prompt agents may incur significant inference cost, which isn’t discussed in depth.\n3. While the benchmark generation pipeline is verified, there’s no qualitative user study or human preference test to assess the final outputs or user experience improvements from M3CoT.\n4. While many open-source models are evaluated, M3CoT’s effectiveness is demonstrated only on two closed-source models. It would strengthen the paper to evaluate M3CoT on open-source LLMs (e.g., InternVL3).", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s invaluable comments and feedback on our paper. We tried our best to address each of your questions.\n\n**Weakness** **1:**  E3VQA's reliance on a single dataset, EgoExo4D, may hinder its diversity and real-world applications.\n\n\n**Response:** We understand the reviewer’s concern that relying only on the EgoExo4D dataset may not be sufficient to capture diverse real-world scenarios. In fact, to improve diversity, we extended E3VQA using LEMMA, which is a multi-view dataset focused on goal-directed daily tasks in home environments. Following the same data generation pipeline in this work, we constructed 500 new samples, evenly distributed across all categories. From experimental results on this new subset, we show that our E3VQA benchmark construction pipeline works well across different datasets, and the proposed M3CoT method provides a consistent performance gain (see table below). \n\n\n\n|LVLMs|Method||Action|||Object|||Num.|||Spatial|||Avg.|\n|-|:-:|-|-|:-|-|-|:-|-|-|:-|-|-|:-|-|-:|\n||||Ego|Exo||Ego|Exo||Ego|Exo||Ego|Exo|||\n|GPT-4o|Default||39.68|43.55||63.49|72.58||56.45|46.03||39.68|45.16||50.83|\n||DDCoT||38.10|48.39||66.67|70.97||**64.52**|47.62||47.62|38.71||52.83|\n||CoCoT||44.44|46.77||69.84|70.97||54.84|47.62||46.03|**51.61**||54.02|\n||CCoT||36.51|48.39||68.25|69.35||53.23|47.62||47.62|40.32||51.41|\n||**M3CoT**||**44.44**|**51.61**||**71.43**|**75.81**||59.68|**65.08**||**63.49**|50.00||**60.19**|\n||||\n|Gemini 2.0|Default||39.68|53.23||66.67|74.19||51.61|46.03||46.03|46.77||53.03|\n||DDCoT||39.68|46.77||60.32|**77.42**||54.84|47.62||55.56|40.32||52.82|\n||CoCoT||**46.03**|48.39||65.08|74.19||50.00|47.62||47.62|40.32||52.41|\n||CCoT||44.44|51.61||66.67|72.58||56.45|**49.21**||46.03|40.32||53.41|\n||**M3CoT**||44.44|**59.68**||**66.67**|72.58||**56.45**|42.86||**55.56**|**50.00**||**56.03**|\n\n\n\n\n\n\n**Weakness** **2** **&** **Question** **1:**  M3CoT’s reliance on multiple inference passes with different prompt agents may incur significant inference cost.\n\n**Response:** We appreciate the reviewer’s thoughtful concern regarding the computational cost of M3CoT. To evaluate the tradeoff between efficiency and performance, we report inference time, token usage, and accuracy for recent CoT prompting methods, including M3CoT, evaluated on both GPT-4o and Gemini 2.0 Flash. Please note that the reported inference time may be unreliable due to variability in API response, so that it should be interpreted as a general trend rather than an exact value.\n\nAs shown in the table below, compared to the baseline, M3CoT incurs an increase of approximately 45K tokens and 34.49 seconds when run on GPT-4o, and 25K tokens and 17.47 seconds on Gemini 2.0 Flash. This increase in token usage is primarily due to the refinement process that leverages three different perspectives to construct richer scene graphs. While multi-perspective reasoning requires additional computation, it enables more accurate understanding. Although the current token cost limits its use in real-time applications, we believe our multi-perspective reasoning approach provides a strong foundation for future research on efficient and scalable solutions.\n\n\n|LVLMs|Method|Token Usage|Time (s)|Acc. (%)|\n|:-|:-:|-:|-:|-:|\n|GPT-4o|Default|2,120.25|2.62|60.90|\n||DDCoT|6,807.18|9.57|64.43|\n||CoCoT|2,349.23|6.25|62.87|\n||CCoT|9,225.30|22.59|63.74|\n||**M3CoT**|**47,250.12**|**37.56**|**68.58**|\n||||\n|Gemini 2.0|Default|790.60|2.56|59.80|\n||DDCoT|2,752.09|5.48|60.31|\n||CoCoT|950.71|3.84|61.09|\n||CCoT|2,817.97|7.14|60.18|\n||**M3CoT**|**26,164.36**|**20.03**|**66.12**|\n\n\n\n\n**Weakness** **3:** M3CoT lacks user studies or human evaluations to validate its output quality or user preference.\n\n**Response:** In our experiments, M3CoT generates an answer from the choices (e.g., A, B, C, or D), as we design E3VQA in a multiple-choice format to simplify the evaluation process. Nonetheless, extending E3VQA to support open-ended responses and evaluating M3CoT's outputs through a user study or human preference assessment is indeed a valuable future direction. We sincerely appreciate this suggestion and consider it a meaningful avenue for future research.\n\n**Weakness** **4:** M3CoT is only tested on closed-source models; evaluating it on open-source LLMs like InternVL3 would strengthen the work.\n\n**Response:** In Appendix D.1 and Table 6, we compare our proposed M3CoT prompting technique with existing CoT methods on open-source LVLMs, specifically InternVL3-8B and InternVL3-14B. When the CoT-based approach is applied to small-sized models, it is known to be less effective due to their limited reasoning capabilities. However, we observe that M3CoT performs well even with small-sized models, which is not the case for conventional CoT-based approaches. \n\n\n\n\n|LVLMs|Method||Action|||Object|||Num.|||Spatial|||Avg.|\n|-|:-:|-|-|:-|-|-|:-|-|-|:-|-|-|:-|-|-:|\n||||Ego|Exo||Ego|Exo||Ego|Exo||Ego|Exo|||\n|InternVL3-14B|Default||44.73|54.93||68.13|73.73||35.60|**53.00**||45.67|48.33||53.02|\n||CCoT||44.60|58.40||65.27|73.80||**37.80**|50.00||46.27|48.80||53.12|\n||**M3CoT**||**45.87**|**60.00**||**70.60**|**75.73**||35.07|50.87||**50.80**|**49.40**||**54.79**|\n||||\n|InternVL3-8B|Default||43.70|54.90||64.80|70.30||35.90|45.20||42.10|46.60||50.44|\n||CCoT||44.00|55.00||63.60|68.30||35.50|**51.20**||43.40|44.70||50.71|\n||**M3CoT**||**45.50**|**57.20**||**68.20**|**71.20**||**37.60**|47.50||**53.80**|**49.20**||**53.78**|\n\n\n\n\n**Question** **2:** Is there a risk caused by conflicting scene graphs between different perspectives, and how is it resolved during refinement?\n\n**Response:** You're absolutely right to highlight this potential risk, and we fully acknowledge its importance. In fact, addressing this issue is precisely the motivation behind our refinement stage. The primary goal of the refinement stage is to resolve inconsistencies between scene graphs generated from different perspectives. When the Ego2Exo and Exo2Ego perspectives produce conflicting scene graphs, the refinement stage encourages each perspective to “reconsider” its own scene graph by referencing the other’s scene graph and the corresponding ego-exo image pair. This process can be interpreted as encouraging the model to overcome its own bias. Details of the prompt used during this refinement stage can be found in Figure 37. We intend to include additional qualitative examples of such conflict resolution in the final manuscript. \nIn addition, the effectiveness of the refinement stage is indirectly demonstrated by the results shown in Appendix D.2 and Figure 9. Compared to iteration 0, iteration 1 (e.g., after the refinement stage) exhibits notable improvements in accuracy, both in the individual predictions from each perspective and in the final answer obtained via majority voting. These results support the effectiveness of the refinement stage in resolving conflicts between perspectives.\n\n**Question** **3:** Do you use visual alignment tools (e.g., pose estimation), or rely solely on language prompting?\n\n**Response:** We do not use any external guidance. Our primary goal is to explore how far the LVLM’s inherent capabilities can be utilized through prompting alone, without relying on external modules or tools. We expect that M3CoT might achieve further accuracy gains with the addition of external information.\n\n**Question** **4:** Can M3CoT generalize to other ego-exo datasets like Charades-Ego or OpenEQA?\n\n**Response:** Thank you for the insightful question. We expect that models trained on E3VQA or using the proposed M3CoT prompting approach can be applied to other datasets such as Charades-Ego and OpenEQA. While Charades-Ego involves asynchronous ego-exo pairs and OpenEQA focuses on embodied question answering, both datasets fundamentally require the model to extract and integrate information from multiple images to answer the questions. Given this shared requirement, we expect that our work would be effective and applicable in such datasets.\n\n\n**Question** **5:** Showing failure cases would help readers better understand M3CoT’s capabilities.\n\n**Response:** Thank you for the valuable feedback. Unfortunately, please note that the addition of figures is not allowed this year. We will incorporate qualitative examples analyzing M3CoT's failure cases into the revised manuscript. As you suggested, we believe these examples will help readers better understand the limitations and strengths of our proposed method.\n\n**Question** **6:** How does your model handle subjective questions with multiple plausible answers?\n\n\n**Response:** Regarding the E3VQA benchmark, we try our best to avoid ambiguous or subjective questions and answers. With respect to M3CoT, we design it to generate accurate scene graphs that not only capture multi-view information but are also relevant to the given question. These scene graphs help the model generate image-grounded answers, even for ambiguous queries like 'What am I likely to do next?'\". For instance, consider a scenario where both “slicing a tomato” and “washing a cucumber” are plausible next actions. If the scene graph generated by M3CoT includes elements such as <I, holding, tomato>, and  <cucumber, on, table>, the model is more likely to answer “slicing a tomato”, as this action is more directly supported by the visual evidence in the scene. In summary, M3CoT guides the model to focus on visual cues rather than prior knowledge, enabling more image-grounded responses even in ambiguous situations.\n\n\n\nWe will add these supplements to the revised manuscript. We hope our responses can address your concerns."}, {"review_id": "jXWIdhATcN", "reviewer": "Reviewer_2B1U", "summary": "his paper tackles the challenge of enabling large vision-language models (LVLMs) to jointly reason over first-person (egocentric) and third-person (exocentric) views. The authors introduce E3VQA, a new benchmark comprising 4K high-quality question–answer pairs grounded in synchronized ego–exo image pairs, covering tasks such as action understanding, attribute recognition, spatial reasoning, and counting. They also propose M3CoT, a training-free prompting method that constructs multi-perspective scene graphs (Ego&Exo, Ego→Exo, Exo→Ego), which are progressively unified to form a complete scene representation. Experiments on leading closed-source (GPT-4o, Gemini) and open-source LVLMs show that M3CoT consistently outperforms strong chain-of-thought baselines (e.g., CCoT) with up to 5.9% absolute accuracy improvement, especially in numerical reasoning tasks.", "questions": "Since M3CoT applies a full multi-perspective prompting pipeline to all questions, would it be possible to selectively activate only certain views or strategies based on question type or view relevance (e.g., skip full graph fusion for single-view solvable questions) to reduce inference overhead?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1 the paper the well-writen and easy to follow.\n2 E3VQA fills an evaluation gap in video understanding by targeting egocentric–exocentric joint understanding with carefully curated multiple-choice QA pairs. Its design rigorously tests multi-view integration rather than single-view shortcuts.\n3: M3CoT enhances reasoning across perspectives without additional model training, and achieves consistent performance gains over state-of-the-art CoT prompting baselines, especially in spatial and numerical questions.\n\nWeaknesses\nThe approach relies on synchronized ego–exo image pairs, limiting its applicability to real-world settings where such pairing may be noisy, missing, or unavailable.\nM3CoT involves multiple scene graph generations and iterative refinement. The cost and stability of this prompting procedure across different LVLMs are not fully analyzed, and it may pose scalability concerns.", "was_revised": true, "final_justification": "I appreciate the response from the author. I would keep my positive rating.", "rebuttal": "We appreciate the reviewer’s invaluable comments and feedback on our paper. We tried our best to address each of your questions.\n\n**Weakness** **1:** Dependence on synchronized ego-exo pairs could limit real-world use where such data is incomplete or noisy.\n\n**Response:** Thank you for the reviewer’s insightful question. While we acknowledge that maintaining synchronization between ego and exo views can be challenging in real-world settings, our analysis suggests that minor temporal mismatches (e.g., a few seconds) have minimal impact on the correctness of the ground truth. To examine the effect of temporal mismatches on the ground truth in E3VQA, we randomly sample 80 examples (10 from each category) and analyze how the ground truth changes when the ego and exo views are misaligned by 1 to 3 seconds. As shown in the table below, when the synchronization gap is 1 second, 99% of the ground truth answers remained unchanged. Even with a 3-second gap, 94% of the ground truth remained consistent. In addition, regarding the question of potential noise or missing information in one of the views, we expect that the use of complementary multi-view inputs can itself mitigate such issues. Even if part of the information is missing or corrupted in one view, the model can leverage the other view to fill in the gaps. Therefore, we expect that our framework can provide more accurate and robust answers than single-view baselines, even when there are small synchronization issues or partial data loss.\n\n\n\n**Weakness** **2:** M3CoT's multi-step prompting lacks analysis of cost, stability, and scalability across LVLMs.\n\n**Response:** We appreciate the reviewer’s thoughtful concern regarding the computational cost of M3CoT. To evaluate the tradeoff between efficiency and performance, we report inference time, token usage, and accuracy for recent CoT prompting methods, including M3CoT, evaluated on both GPT-4o and Gemini 2.0 Flash. Please note that the reported inference time may be unreliable due to variability in API response, so that it should be interpreted as a general trend rather than an exact value.\n\nAs shown in the table below, compared to the baseline, M3CoT incurs an increase of approximately 45K tokens and 34.49 seconds when run on GPT-4o, and 25K tokens and 17.47 seconds on Gemini 2.0 Flash. This increase in token usage is primarily due to the refinement process that leverages three different perspectives to construct richer scene graphs. While multi-perspective reasoning requires additional computation, it enables more accurate understanding. Although the current token cost limits its use in real-time applications, we believe our multi-perspective reasoning approach provides a strong foundation for future research on efficient and scalable solutions.\n\n\n|LVLMs|Method|Token Usage|Time (s)|Acc. (%)|\n|:-|:-:|-:|-:|-:|\n|GPT-4o|Default|2,120.25|2.62|60.90|\n||DDCoT|6,807.18|9.57|64.43|\n||CoCoT|2,349.23|6.25|62.87|\n||CCoT|9,225.30|22.59|63.74|\n||**M3CoT**|**47,250.12**|**37.56**|**68.58**|\n||||\n|Gemini 2.0|Default|790.60|2.56|59.80|\n||DDCoT|2,752.09|5.48|60.31|\n||CoCoT|950.71|3.84|61.09|\n||CCoT|2,817.97|7.14|60.18|\n||**M3CoT**|**26,164.36**|**20.03**|**66.12**|\n\n\n\n\n**Question** **1:** Can M3CoT optimize inference by dynamically selecting views or reasoning paths based on question demands?\n\n**Response:** Thank you for the insightful question. We truly agree that integrating a query-driven view selection strategy into M3CoT could make inference much more efficient, especially in cases where the relevant view can be inferred from the question. That said, we would like to mention that we intentionally design E3VQA to avoid such cases so that the model is encouraged to consider both views when answering. Therefore, selecting the appropriate view based solely on the query may be challenging in many E3VQA examples. Nonetheless, we fully support the idea that your proposed strategy would be highly effective in practical applications beyond the benchmark setting, where efficiency is critical and many real-world questions naturally provide cues that make view selection more predictable.\n\nWe will add these supplements to the revised manuscript. We hope our responses can address your concerns."}, {"review_id": "bYMXtMYN8O", "reviewer": "Reviewer_itMh", "summary": "This paper introduces a new framework for comprehensive scene understanding in large vision-language models (LVLMs) by integrating both first-person (egocentric) and third-person (exocentric) visual perspectives. The authors develop E3VQA, a benchmark consisting of 4,000 high-quality question–answer pairs grounded in synchronized ego–exo image pairs, covering tasks such as action understanding, object recognition, spatial reasoning, and numerical estimation.", "questions": "1. Why not use real images in Figure 1?\n2. What is the difference between using the multi-view images compared with video data?\n3. How to make sure the VLM can generate correct and scene graphs? Especially the presented image is quite complicated.", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper tackles an under-explored but highly relevant problem—comprehensive scene understanding by integrating first-person and third-person views in LVLMs. This dual-view approach goes beyond typical single-view VQA and aligns well with real-world use cases in AR/VR and robotics.\n2. The E3VQA dataset is carefully constructed and covers a diverse set of tasks.\n\n# Weaknesses\n1. The E3VQA benchmark is derived from the EgoExo4D dataset, which, while large and varied, may still be limited in terms of scene diversity, activities, and environmental conditions.\n2. The evaluation is limited to multiple-choice VQA with synchronized image pairs, and the number of samples is small (~4k).\n3. The dataset is generated using GPT4o on images, but only the val set, no training set and finetuing existing VLMs. I think the contribution is not enough.", "was_revised": true, "final_justification": "The authors have addressed my concerns by extending their benchmark to include a new dataset and introducing additional evaluation metrics to better assess scene graph quality. I appreciate these improvements, and encourage the authors to further enhance the generalization and task variety.", "rebuttal": "We appreciate the reviewer’s invaluable comments and feedback on our paper. We tried our best to address each of your questions.\n\n**Weakness** **1:** The E3VQA benchmark is derived from the EgoExo4D dataset, which, while large and varied, may still be limited in terms of scene diversity, activities, and environmental conditions.\n\n**Response:** We understand the reviewer’s concern that relying only on the EgoExo4D dataset may not be sufficient to capture diverse real-world scenarios. In fact, to improve diversity, we extended E3VQA using LEMMA, which is a multi-view dataset focused on goal-directed daily tasks in home environments. Following the same data generation pipeline in this work, we constructed 500 new samples, evenly distributed across all categories. From experimental results on this new subset, we show that our E3VQA benchmark construction pipeline works well across different datasets, and the proposed M3CoT method provides a consistent performance gain (see table below). \n\n\n\n| LVLMs  | Method   |   |  Action  |          |        |  Object |         |        | Num.  |         |        | Spatial |         |           | Avg.  |\n|-----------|:------:|-|------------|:-------|-------|---------|:-------|-------|--------|:-------|-------|--------|:-------|---------|-------:|\n|              |          |      | Ego      | Exo    | | Ego         | Exo  |   | Ego        | Exo   |  | Ego        | Exo   |  |    |\n| GPT-4o  | Default  || 39.68     | 43.55   | | 63.49        | 72.58 |   | 56.45       | 46.03|    | 39.68       | 45.16   | | 50.83 |\n| | DDCoT || 38.10     | 48.39   | | 66.67        | 70.97   | | **64.52**       | 47.62   | | 47.62       | 38.71   | | 52.83 |\n| | CoCoT || 44.44     | 46.77   | | 69.84        | 70.97   | | 54.84       | 47.62  |  | 46.03       | **51.61**   | | 54.02 |\n| | CCoT  || 36.51     | 48.39   | | 68.25        | 69.35   | | 53.23       | 47.62  |  | 47.62       | 40.32   | | 51.41 |\n| | **M3CoT**  || **44.44**    | **51.61**  |  | **71.43**        | **75.81**  |  | 59.68       | **65.08** |    | **63.49**       | 50.00  |  | **60.19** |\n||||\n| Gemini 2.0   | Default  ||  39.68  | 53.23   ||66.67    |74.19|   |51.61    |46.03  |  |46.03    |46.77   | |  53.03  |\n| | DDCoT  ||   39.68 |  46.77 | | 60.32   |**77.42**  | |54.84    |47.62   | |55.56    | 40.32 |  | 52.82   |\n| | CoCoT  ||  **46.03** |  48.39 | | 65.08   | 74.19 | | 50.00   |47.62  |  |47.62    |40.32    ||52.41    |\n| | CCoT   ||   44.44 |  51.61  | |66.67   |72.58  | |56.45    |**49.21**   | |46.03    |40.32  |  |53.41    |\n| | **M3CoT**   ||  44.44  |**59.68**   | |**66.67**    |72.58 |  |**56.45**    |42.86    ||**55.56**    |**50.00**  |  | **56.03**   |\n\n\n\n\n\n**Weakness** **2:** The evaluation is limited to multiple-choice VQA with synchronized image pairs, and the number of samples is small (~4k).\n\n**Response:** We appreciate the reviewer’s invaluable comment. While we understand that our multiple-choice questions can be easily converted to open-ended ones by removing answer choices, after careful thought, we decided to use the multiple-choice format to assess the model’s ability to choose the right answer from confusing alternatives. Nevertheless, we recognize the value of diverse question types (e.g., yes/no and open-ended) and will extend our benchmark.\nRegarding the concern about dataset size, we note that E3VQA is substantially larger than most existing egocentric VQA datasets. As shown in Appendix A.1 and Table 4 of the supplementary material, it is over 33 times larger than EgoVQA (120), and several times larger than EgoThink (700), EmbodiedQA (529), and OpenEQA (1.6K). While its size remains smaller than some large-scale exocentric datasets such as MSVD-QA (13K) or Social-IQ (7.5K), this gap primarily stems from the fact that third-person data is significantly more abundant than first-person data, and even more so for ego-exo paired data, making large-scale curation more challenging. \nIn addition, just 3.6% of 110K GPT-4o-generated question–answer pairs were preserved in E3VQA after rigorous multi-stage filtering, and all included examples were manually verified by human annotators to ensure high quality. This strict filtering process reflects our focus on quality over quantity and ensures that the final benchmark includes challenging, well-grounded multi-view reasoning examples. We believe E3VQA provides meaningful scale and strong evaluation capacity as a benchmark for multi-view visual understanding. \n\n\n**Weakness** **3:** The dataset is generated using GPT-4o on images, but only the val set, no training set, and fine-tuning existing VLMs. I think the contribution is not enough.\n\n**Response:** We believe that the reviewer’s comment is a bit excessive, given that several other reviewers (BgYM, 2B1U, Gcmm, and 9jna) have highlighted this aspect as a strength.\n\nPlease note that the primary contribution of E3VQA lies in its novel benchmark design targeting ego-exo multi-view reasoning, a setting not explored in previous VQA datasets. Please also note that our work is the first trial that systematically evaluates whether LVLMs can reason across egocentric and exocentric perspectives without task-specific fine-tuning. \nAdditionally, our data generation pipeline is uniquely designed to construct plausible answer candidates by intentionally leveraging the view misreferencing behaviors of LVLMs under different visual input configurations.\nMoreover, our proposed method, M3CoT, is a training-free prompting strategy that encourages LVLMs to integrate and reason over complementary visual perspectives. It consistently improves performance by promoting cross-view alignment and understanding, without requiring any model-specific tuning. This demonstrates the potential of prompting as a plug-and-play approach to multi-view visual reasoning, particularly in scenarios where fine-tuning is not feasible.\n\n\n**Question** **1:** Why does Figure 1 not use real images?\n\n**Response:** In Figure 1, we intentionally used a conceptual illustration rather than real images to provide an intuitive explanation of our framework. We believe this synthetic image effectively highlights the complementary nature of the ego and exo views by illustrating how their fields of view differ, something that is difficult to depict clearly with real images. To avoid confusion, we will clarify in the caption that Figure 1 is a conceptual illustration and is not part of the actual dataset. For real examples from the dataset, we will kindly refer the reader to subsequent figures throughout the paper.\n\n\n**Question** **2:** What is the difference between using the multi-view images compared with video data?\n\n**Response:** Although multi-view images and video data seem to be similar since they consist of multiple images, they differ in how the visual information is captured and ordered. Video captures a continuous sequence of frames from a single viewpoint over time, preserving temporal order. In contrast, multi-view images are captured simultaneously from different angles at a single time point, where the sequence order is not meaningful. While video can offer complementary views over time as the camera moves, potentially covering different angles of the scene, the context may change between frames, making it difficult to align information across time. As a result, video cannot fully replace multi-view images, which capture different viewpoints at the same moment without temporal drift.\n\n\n**Question** **3:**  How to make sure the VLM can generate correct scene graphs? The presented image is quite complicated.\n\n**Response:** We appreciate the reviewer’s insightful comment. While related analysis is included in Appendix D.2 of the supplementary material, we provide an extended explanation here to enhance clarity and better highlight its significance.\nScene graph quality is often assessed by computing the intersection-over-union (IoU) with ground-truth scene graphs. However, overly exhaustive graphs that capture all visual details may not always be beneficial for answering questions accurately. To better assess the practical utility of scene graphs generated by M3CoT, we introduce two complementary metrics:\n* False discovery rate (FDR) is measured as the proportion of incorrect elements among the model's predicted scene graph components, such as object classes, relations, and attributes. This reflects how accurately the model captures the visual content.\n* Answer accuracy is measured based on the VQA performance when the generated scene graph is used as a prompt. This reflects how well the scene graph supports downstream reasoning.\n\nTo evaluate the relationship between FDR and answer accuracy of the generated scene graph, we constructed a subset of 120 samples from E3VQA, equally distributed across all question categories. For this subset, we evaluated the scene graph before and after the refinement stage of M3CoT. The FDR of the scene graph decreased from 9.37% to 6.21%, with an increase in answer accuracy from 46.88% to 60.42%. This result demonstrates that our refinement stage not only corrects errors in the scene graph but also organizes the information in a way that better supports question answering.\n\nWe will add these supplements to the revised manuscript. We hope our responses can address your concerns."}, {"review_id": "UP1QjadRhu", "reviewer": "Reviewer_Gcmm", "summary": "This paper studies the importance of integrating first and third-person views for Vision-and-Language models. This is an incredibly relevant setup particularly for assisted living scenario where the personal assistant might have access to multiple cameras that are installed in a smart home. To tackle this problem, the authors automatically generate a VQA dataset from EgoExo4D. The output of the automatic data collection is then verified by four expert annotators, who were able to produce 4,000 high-quality examples. \n\nGiven this dataset, the authors complete an evaluation of both closed-source and open-weights models. Showcasing interesting results which highlight that GPT4o is the top-performing model overall with InternVL3-14B leading the way for open-weights models. The authors also present M3CoT, a prompting technique that uses scene graphs to prompt the model with different information that are used to help the model predict the right answer. This involves multiple steps of scene graph extraction and refinement. The authors evaluate this technique for Gemini and OpenAI models, showcasing how it helps improve performance over other prompting techniques.", "questions": "Overall, I really enjoyed the paper and I hope that the authors find my weaknesses informative to help them shape the next iteration of this paper. I'll report below the question I had which I hope the authors will help me improve my understanding of this paper:\n\n1. Who are the expert annotators? What was the procedure they followed to annotate the data?\n2. What is the computational costs of running your M3CoT compared to other state-of-the-art methods as well as compared to the baseline? I think it's important to be transparent about this, considering that you are running multiple copies of the same model. I understand that you applied this only to API models therefore you didn't consider the scenario where the user has to deploy this.\n3. Related to the previous point, why did you apply the M3CoT only to frontier models? Was there anything that prevented you from applying it to other models like QwenVL?\n4. I think it would be great if your table of results could always report the \"both\" setting because this will highlight a more realistic scenario which is where the model has to truly use both perspectives to answer the question", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is well-written and easy to follow. The narrative is clear, and the main methodology as well as the results are well described. \n2. High-quality data collection defined to derive the final dataset. This represents an important resource for the community and I hope that the authors will release both the dataset as well as their data generation procedure.\n3. Interesting prompting technique proposed however I am a bit skeptical of its ability to perform at scale (see weaknesses)\n4. Really comprehensive evaluation of several closed and open models. Additionally, the authors provide a detailed error analysis that I find really informative.\n\n### Weaknesses\n\n1. The paper does not feature a specific related work section. I understand that this paper represents a very novel contribution however I wonder whether it would be useful to include a description of the different egocentric question answering datasets such as OpenEQA, and describe how this dataset is different from others. This could be not only in terms of the different perspectives available but also in terms of the different linguistic properties and capabilities required by each question\n\n2. I think it would have been useful to also include some text-only baselines as well to understand whether there are inherent biases in the data. \n\n3. I really liked this paper because I think that the automated data collection pipeline is very interesting and is designed to ensure high-quality annotations. However, I don't find the scene graph prompting technique as a strong contribution of this paper. Specifically, I wonder how scalable this solution is, especially if we imagine a scenario where the model has to react in real-time because it has to process the video stream coming from some smart glasses (e.g., Rayban Glasses). Running multiple copies of the model for each query sounds extremely expensive. It would be good if the authors could provide more details regarding this.", "was_revised": true, "final_justification": "The authors provided some important details that were not reported in the original paper. I think these additions will make the paper stronger and I hope that the authors will include all the suggested revisions. My score remains the same and I would be happy to see this paper accepted.", "rebuttal": "We appreciate the reviewer’s invaluable comments and feedback on our paper. We tried our best to address each of your questions.\n\n\n**Weakness** **1:** The paper does not feature a specific related work section. \n\n**Response:** We would like to politely point out that Table 4 and Appendix A.2 in the supplementary material provide a detailed comparison between E3VQA and existing VQA datasets, including OpenEQA.\nTo summarize, EgoVQA evaluates the understanding of egocentric visual information. EgoSchema extends this line of work by focusing on long-term reasoning over egocentric experiences. EgoThink emphasizes first-person cognition through questions grounded in the user's perspective, involving the pronoun “I”. EmbodiedQA and OpenEQA also utilize egocentric visual input, but place more emphasis on environment-level understanding, rather than the user’s state or interactions.\nNotably, E3VQA not only leverages multiple visual perspectives (ego and exo) but also includes a broad range of questions targeting the user, the environment, and their interactions, using both first-person and third-person question forms. In particular, it uniquely evaluates a model’s ability to integrate complementary information across ego and exo views, which distinguishes it from prior benchmarks.\n\n\n**Weakness** **2:** I think it would have been useful to also include some text-only baselines as well to understand whether there are inherent biases in the data.\n\n**Response:** We appreciate the reviewer’s suggestion that text-only baselines can be a helpful way to examine potential biases in the data. To address the reviewer’s suggestion, we conducted experiments using text-only inputs (i.e., questions without images), and the results are shown in the table below. (These results will be included in the final manuscript.)\n\n| LVLMs|Action|||  Object ||| Num.  ||| Spatial ||| Avg.  |\n|-----------------|------------|:-------|-------|---------|:-------|-------|--------|:-------|-------|--------|:-------|---------|-------:|\n|| Ego| Exo|| Ego| Exo || Ego| Exo|| Ego  | Exo   |||\n| **Closed-Source**|\n| Gemini 2.0| 20.20| 24.80||20.60| 22.80  | |27.80  | 30.20  || 16.20    | 15.20  || 22.23 |\n| GPT-4o| 25.00   | 30.40  || 12.20| 23.20  || 7.40   | 30.80  || 11.60    | 15.40  || 19.50 |\n| **Open-Source**|\n| InternVL3-14B| 31.00| 37.20  ||33.80   | 30.20  || 38.40  | 32.80  || 10.80    | 11.80  || 28.25 |\n| Qwen2.5-VL-7B| 32.20   |34.60   || 32.80   | 31.00  || 35.20 | 36.20  || 8.00     | 5.400   || 26.93 |\n| LLaVA-OneVision-7B | 32.60 | 38.00  || 31.40   | 26.80  || 27.40  | 28.80  || 7.00     | 5.60   || 24.70 |\n\n\n\n**Weakness** **3** **&** **Question** **2:** What are the computational costs of running your M3CoT compared to other state-of-the-art methods, as well as compared to the baseline?\n\n**Response:** We appreciate the reviewer’s thoughtful concern regarding the computational cost of M3CoT. To evaluate the tradeoff between efficiency and performance, we report inference time, token usage, and accuracy for recent CoT prompting methods, including M3CoT, evaluated on both GPT-4o and Gemini 2.0 Flash. Please note that the reported inference time may be unreliable due to variability in API response, so that it should be interpreted as a general trend rather than an exact value.\n\nAs shown in the table below, compared to the baseline, M3CoT incurs an increase of approximately 45K tokens and 34.49 seconds when run on GPT-4o, and 25K tokens and 17.47 seconds on Gemini 2.0 Flash. This increase in token usage is primarily due to the refinement process that leverages three different perspectives to construct richer scene graphs. While multi-perspective reasoning requires additional computation, it enables more accurate understanding. Although the current token cost limits its use in real-time applications, we believe our multi-perspective reasoning approach provides a strong foundation for future research on efficient and scalable solutions.\n\n\n|LVLMs|Method|Token Usage|Time (s)|Acc. (%)|\n|:-|:-:|-:|-:|-:|\n|GPT-4o|Default|2,120.25|2.62|60.90|\n||DDCoT|6,807.18|9.57|64.43|\n||CoCoT|2,349.23|6.25|62.87|\n||CCoT|9,225.30|22.59|63.74|\n||**M3CoT**|**47,250.12**|**37.56**|**68.58**|\n||||\n|Gemini 2.0|Default|790.60|2.56|59.80|\n||DDCoT|2,752.09|5.48|60.31|\n||CoCoT|950.71|3.84|61.09|\n||CCoT|2,817.97|7.14|60.18|\n||**M3CoT**|**26,164.36**|**20.03**|**66.12**|\n\n\n\n**Question** **1:** Who are the expert annotators, and what procedure did they follow to annotate the data?\n\n**Response:** The human verification stage was performed by members of our research team, which consists of the authors, all with domain expertise in vision-language reasoning and extensive experience working with ego-exo data. \nFor details of the annotation procedure, including the user interface used during curation, please refer to Appendix B.3 and Figure 9 of the supplementary material. Even so, we acknowledge that additional clarification would be helpful and plan to revise the description to better convey the full curation process.\nIn short, our human verification stage involved several steps to ensure quality and consistency:\n\n* We first removed questions that were ambiguous or unanswerable when both ego and exo views were provided.\n\n\n* Next, we discarded overly simple questions that could be solved through shallow pattern matching, such as cases where $A_{\\text{both}}$ was correct regardless of reasoning, even if they had passed the automatic filtering stage.\n\n\n* To construct distractor options, we primarily selected responses generated under different visual input configurations. When the initial candidates were redundant or low-quality, we additionally generated options from the ego and exo views individually to supplement the distractor pool and ensure both diversity and relevance.\n\n\n* Finally, team members cross-reviewed subsets annotated by others to ensure consistency, clarity, and coverage. \n\n**Question** **3:** Why did you apply the M3CoT only to frontier models?\n\n**Response:** We appreciate the reviewer’s question on the choice of models used in our evaluation. While our main results focus on frontier models, we also conducted experiments on open-source models, including InternVL3-14B and InternVL3-8B, as shown in Appendix D.1 and Table 6 of the supplementary material. \nEach model consistently showed improved accuracy, from 53.02% to 54.79% for InternVL3-14B and from 50.44% to 53.78% for InternVL3-8B, outperforming all CoT baselines by up to 3.07%. As noted in prior works, prompting methods often struggle with smaller models due to limited reasoning capacity. Nevertheless, M3CoT consistently outperformed other approaches, demonstrating its robustness even in low-capacity settings.\n\n\n\n**Question** **4:** I think it would be great if your table of results could always report the \"both\" setting.\n\n**Response:** We agree with the reviewer that reporting results under the “both” setting, where questions require information from both views, provides valuable insight into a model’s ability to integrate multi-view information. At the same time, reporting results under the “any”, “ego”, and “exo” settings offers complementary information. The “any” setting assesses the model’s ability to handle redundant or overlapping information, whereas the “ego” and “exo” settings evaluate its robustness when faced with unnecessary or irrelevant views. Following this suggestion, we conducted the corresponding experiments, and the results are now presented in the table below. We are grateful for this insightful suggestion, and we will incorporate the corresponding results into the final manuscript.\n\n\n|LVLMs|Method||Any||Ego||Exo||Both||Overall|\n|-|:-:|-|-|-|-|-|-|-|-|-|-:|\n|GPT-4o|Default||65.20||63.96||58.59||39.28||56.76|\n||DDCoT||69.71||69.37||60.83||37.68||59.40|\n||CoCoT||66.94||64.41||61.47||43.48||59.08|\n||CCoT||67.94||67.50||61.05||42.03||59.63|\n||**M3CoT**||**73.45**||**74.30**||**64.23**||**52.61**||**66.15**|\n||||\n|Gemini 2.0|Default||65.81||66.08||54.38||37.52||55.95|\n||DDCoT||66.48||66.36||58.22||31.13||55.55|\n||CoCoT||67.23||65.43||56.44||32.43||55.38|\n||CCoT||65.76||66.18||55.29||43.21||57.61|\n||**M3CoT**||**69.76**||**69.24**||**62.97**||**53.19**||**63.79**|"}, {"review_id": "A5mzcm99Qm", "reviewer": "Reviewer_9jna", "summary": "This a benchmark and analysis paper, focusing on evaluating egocentric (Ego) and exocentric (Exo) reasoning in frontier VLMs, and introduces a dataset derived from the prior EgoExo4D work.\n\nThe paper introduces E3VQA, a dataset containing 4K pairs of questions and answers related to ego-exo images. Each question is crafted so that it is difficult to answer using only text or a single image; instead, integrating information from both views is essential to select the correct answer.\n\nThe authors also introduce a prompting technique called M3CoT, which uses three agent VLMs that focus on both the ego and exo views, respectively, to extract information from the input views, form scene graphs, and use majority voting to select the best answer.\n\nThe E3VQA benchmark demonstrates that frontier VLMs still struggle with cross-view reasoning between ego and exo perspectives. The M3CoT prompting technique improves these models' performance by about 5% on the benchmark. Overall, the paper highlights the limitations of current VLMs in multi-view ego-exo reasoning, analyzes the cases where these models struggle most, and proposes potential improvements.", "questions": "The paper is well written and convey the message clearly, I don't have questions regards the paper.\nThe questions I have is more about the focus to improve the significance of the paper, but may out reach of the scope of this paper:\n\n1. This is unclear that the VLMs performance drop in Ego views compared to Exo views, is not because the uniquce image format or charateristics of the EgoExo4D data. But is the narrowed view sight or context imformation in the ego view.\n2. Is the Ego-Exo has its unique challenges compared to Ego-Ego, Exo-Exo settigs? As the paper lacks the comparison between how Ego-Exo compared to Exo-Exo in the same timestamp views in EgoExo4D dataset.\n3. The analysis of VLM failures in Ego-Exo settings lacks significance. It remains unclear whether these failures stem from substantial visual differences in the appearance of the same object across views, or if powerful visual encoders still produce similar visual embeddings. Furthermore, it is not clear whether VLMs are establishing visual correspondences between ego and exo views, or if they are primarily relying on extracting language descriptions (as suggested by M3CoT) and matching correspondences in the textual space.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is clearly written and effectively conveys that frontier VLMs have limitations in ego-exo multiview reasoning. The proposed prompting technique is simple yet effective, requiring no additional training.\n2. The E3VQA QA pairs are mostly auto-generated and filtered using LLMs and VLMs. The pipeline is clearly described and should be easy to scale as more GT Ego-Exo frames and annotations become available.\n3. The benchmark is filtered based on the consistency of model responses when given different access to the views (both, ego, exo, or only text). This process ensures that the questions included truly require reasoning over both Ego and Exo views, rather than allowing models to rely on shortcuts or general knowledge to answer.\n4. The simple yet effective M3CoT prompting technique improves Ego and Exo reasoning during inference, and its effectiveness is demonstrated across different VLM models.\n\n### Weaknesses\n\n1. As stated in the paper, the benchmark data is curated solely from EgoExo4D video frames and annotations, which limits the diversity of data sources and may restrict the generalizability of the results and findings.\n2. The paper does not sufficiently discuss the unique challenges posed by Ego-Exo view integration compared to Exo or Exo multiview scenarios.\n3. It is unclear whether the main challenge for frontier VLMs stems from the inherently unique egocentric viewpoint, or from the specific characteristics of the EgoExo4D ego view format (such as its FOV and the dark areas outside the viewpoint), which may differ significantly from the images typically seen during VLM training. If the egocentric images were calibrated and rectified to resemble more conventional photographs, would the observed difficulty for VLMs persist?\n4. The analysis section primarily focuses on the fairness of the benchmark but lacks a analysis of the typical failure patterns of current VLMs and a deeper investigation into why these models struggle with ego and exo settings. For instance, visualizations such as attention maps between ego and exo views could help illustrate how models interpret the relationship between perspectives and whether their focus aligns with the question. If not, identifying and analyzing these failure patterns would provide valuable insights into the limitations of VLMs in this context.\n\nOverall, the quality and clarity of the paper are good. However, due to the aforementioned limitations, I rate the significance of the paper as fair.", "was_revised": true, "final_justification": "My initial major concern was the unique fisheye lens format of the egocentric view images. The rebuttal clarified that LEMMA uses standard rectangular images for the ego view, and the observed trends are consistent across both LEMMA and EgoExo4D datasets.\nThus I would like to improve my previous rating to acceptance.", "rebuttal": "We appreciate the reviewer’s invaluable comments and feedback on our paper. We tried our best to address each of your questions.\n\n**Weakness** **1:** E3VQA is curated solely from EgoExo4D video frames and annotations, which limits the diversity.\n\n\n**Response:** We understand the reviewer’s concern that relying only on the EgoExo4D dataset may not be sufficient to capture diverse real-world scenarios. In fact, to improve diversity, we extended E3VQA using LEMMA, which is a multi-view dataset focused on goal-directed daily tasks in home environments. Following the same data generation pipeline in this work, we constructed 500 new samples, evenly distributed across all categories. From experimental results on this new subset, we show that our E3VQA benchmark construction pipeline works well across different datasets, and the proposed M3CoT method provides a consistent performance gain (see table below). \n\n\n\n\n| LVLMs  | Method   |   |  Action  |          |        |  Object |         |        | Num.  |         |        | Spatial |         |           | Avg.  |\n|-----------|:------:|-|------------|:-------|-------|---------|:-------|-------|--------|:-------|-------|--------|:-------|---------|-------:|\n|              |          |      | Ego      | Exo    | | Ego         | Exo  |   | Ego        | Exo   |  | Ego        | Exo   |  |    |\n| GPT-4o  | Default  || 39.68     | 43.55   | | 63.49        | 72.58 |   | 56.45       | 46.03|    | 39.68       | 45.16   | | 50.83 |\n| | DDCoT || 38.10     | 48.39   | | 66.67        | 70.97   | | **64.52**       | 47.62   | | 47.62       | 38.71   | | 52.83 |\n| | CoCoT || 44.44     | 46.77   | | 69.84        | 70.97   | | 54.84       | 47.62  |  | 46.03       | **51.61**   | | 54.02 |\n| | CCoT  || 36.51     | 48.39   | | 68.25        | 69.35   | | 53.23       | 47.62  |  | 47.62       | 40.32   | | 51.41 |\n| | **M3CoT**  || **44.44**    | **51.61**  |  | **71.43**        | **75.81**  |  | 59.68       | **65.08** |    | **63.49**       | 50.00  |  | **60.19** |\n||||\n| Gemini 2.0   | Default  ||  39.68  | 53.23   ||66.67    |74.19|   |51.61    |46.03  |  |46.03    |46.77   | |  53.03  |\n| | DDCoT  ||   39.68 |  46.77 | | 60.32   |**77.42**  | |54.84    |47.62   | |55.56    | 40.32 |  | 52.82   |\n| | CoCoT  ||  **46.03** |  48.39 | | 65.08   | 74.19 | | 50.00   |47.62  |  |47.62    |40.32    ||52.41    |\n| | CCoT   ||   44.44 |  51.61  | |66.67   |72.58  | |56.45    |**49.21**   | |46.03    |40.32  |  |53.41    |\n| | **M3CoT**   ||  44.44  |**59.68**   | |**66.67**    |72.58 |  |**56.45**    |42.86    ||**55.56**    |**50.00**  |  | **56.03**   |\n\n\n**Weakness** **2** **&** **Question** **2:** The paper does not sufficiently discuss the unique challenges posed by Ego-Exo view integration compared to Ego or Exo multiview scenarios.\n\n**Response:** Thank you for raising the important point here. When compared to Ego-Ego or Exo-Exo multi-view scenarios, the Ego-Exo setting presents a unique challenge: answering egocentric questions (e.g., What am I doing?) without explicit information about the identity of “I”. Specifically, when presented with such questions, the model must first infer who “I” refers to, typically by identifying the camera wearer in the egocentric view, and then locate the same person in the exocentric view. This cross-view grounding requires additional reasoning that is not needed in the Ego-Ego or Exo-Exo settings, as “I” cannot be identified in those settings without extra context or annotations. We will incorporate this discussion into the revised manuscript.\n\n\n**Weakness** **3** **&** **Question** **1:** It is unclear whether the main challenge for frontier VLMs stems from the inherently unique egocentric viewpoint, or from the specific characteristics of the EgoExo4D ego view format.\n\n**Response:** As we mentioned, we additionally constructed a benchmark using the LEMMA dataset. Unlike EgoExo4D, the ego images in LEMMA are in a standard rectangular format (i.e., rectified) and do not exhibit dark peripheral regions along the edges. While the ego images exhibit a standard rectangular format, the experimental results for both the baseline and M3CoT show trends consistent with those observed on the E3VQA benchmark generated from EgoExo4D. This consistency implies that the challenge does not arise merely from the specific visual characteristics of EgoExo4D, but rather from the inherent difficulty of integrating egocentric and exocentric views for visual reasoning.\n\n**Weakness** **4** **&** **Question** **3:** The analysis section lacks an analysis of the typical failure patterns of current VLMs.\n\n\n**Response:** To analyze the failure patterns in Ego-Exo multi-view settings, we conducted an experiment in Section 4.1 and Figure 5(c), examining how model performance varies when provided with one view versus both views. We find that, in the ‘Ego’ and ‘Exo’ subsets where the answer-relevant information is present in only one view, providing both views to the model leads to worse performance compared to providing only the informative view. This reveals a failure pattern in which the additional, uninformative view distracts or confuses the model.\n\nWhile this analysis provides some insights, we acknowledge that it does not fully reveal the fundamental reasons why current LVLMs struggle in Ego-Exo multi-view scenarios. We have tried to interpret the model’s behavior by analyzing attention maps from the decoder layers of the LVLM. However, we were unable to identify consistent and meaningful patterns. Some examples produced correct answers despite weak attention to regions containing the relevant information, while others failed even with seemingly appropriate focus on those regions. We suspect that this may be due to multiple factors, including the lack of sufficient ego-exo paired training data with strong multi-view alignment signals.\n\nWe assume that guiding the model through step-by-step reasoning, as done in M3CoT, is beneficial because it is inherently difficult for the model to simultaneously establish correspondences across views and link them to the user’s query in a single step. We believe that M3CoT unlocks the LVLM’s inherent ability to build such correspondences to some extent. We will include all analyses and visualizations in the revised manuscript. We hope our responses can address your concerns."}]

Author rebuttal:

Meta-review: **1. Summary of Scientific Claims and Findings**

This paper introduces E3VQA, a novel benchmark designed to evaluate large vision-language models (LVLMs) on multi-view question answering (QA), with a major argument that integrating egocentric (first-person) and exocentric (third-person) perspectives  can provide complementary information such as global scene layout and object visibility to LVLMs. The authors found that current LVLMs struggle to reason effectively across these complementary views (e.g., fine-grained hand-object interactions from ego views and spatial context from exo views). Additionally, they proposed M3CoT, a training-free prompting technique that uses multi-perspective scene graphs (ego-exo, ego→exo, exo→ego) and iteratively fuses them to enhance cross-view reasoning. Experiments on E3VQA (4K expert-verified QA pairs across four categories: pose, object, numerical, spatial) showed that M3CoT consistently outperforms existing prompting baselines across multiple LVLMs, including GPT-4o and Gemini 2.0 Flash.

**2. Major Strengths**

2.1 Novel and rigorous benchmark. E3VQA fills a critical gap in evaluating multi-view (ego-exo) reasoning, with 4K high-quality, expert-verified QA pairs that ensure reliance on cross-view integration rather than single-view shortcuts.

2.2 Effective training-free technique. M3CoT yields significant performance gains over state-of-the-art prompting baselines across LVLMs, particularly in spatial and numerical reasoning tasks.

2.3 Thorough evaluation. The paper includes extensive experiments on both closed-source (e.g., GPT-4o) and open-source (e.g., InternVL3) LVLMs, with ablations, error analysis, and transparent details (implementation, dataset access, compute).

2.4 Clarity and rigor. The paper is well-written, with a clear narrative, rigorous data collection pipeline, and discussion of limitations/ethical concerns.

**3. Major Weaknesses**

3.1 Benchmark limitations. E3VQA relies solely on EgoExo4D, limiting diversity and generalization. Its size (~4K) and focus on multiple-choice QA may restrict broader applicability.

3,2 M3CoT scalability. While training-free, M3CoT’s multiple inference passes raise concerns about high inference cost and scalability, which are not deeply analyzed. Its effectiveness is under-tested on open-source LVLMs.

3.3 Missing analyses. No qualitative user studies, human preference tests, or text-only baselines to assess output quality or data biases. Insufficient discussion of ego-exo integration challenges compared to other multi-view scenarios, and limited analysis of VLM failure patterns.

3.4 Structural gaps: Lack of a dedicated related work section to contextualize E3VQA against existing egocentric QA datasets.

**4. The most important reasons to accept**

This paper received ratings of (4, 5, 4, 5, 5), whose avage rating is the highest among my batch of 14 submissions. The paper has good novelty and the developed benchmark is expected to have a high impact to the field of vision-language models.

**5. Summary of discussions**

There has been some discussions between the authors and reviewers. Most of the questions from the reviewers have been addressed by the authors. After the rebutal, Reviewer 9jna has raised his rating from 4 to 5.
Decision: Accept (spotlight)