EgoExo-Con: Exploring View-Invariant Video Temporal Understanding

## Abstract

Abstract Can Video-LLMs achieve consistent temporal understanding when videos capture the same event from different viewpoints? To study this, we introduce EgoExo-Con (Consistency), a benchmark of comprehensively synchronized egocentric and exocentric video pairs with human-refined queries in natural language. EgoExo-Con emphasizes two temporal understanding tasks: Temporal Verification and Temporal Grounding. It evaluates not only correctness but consistency across viewpoints. Our analysis reveals two critical limitations of existing Video-LLMs: (1) models often fail to maintain consistency, with results far worse than their single-view performances. (2) When naively finetuned with synchronized videos of both viewpoints, the models show improved consistency but often underperform those trained on a single view. For improvements, we propose View-GRPO, a novel reinforcement learning framework that effectively strengthens view-specific temporal reasoning while encouraging consistent comprehension across viewpoints. Our method demonstrates its superiority over naive SFT and GRPO, especially for improving cross-view consistency. All resources will be made publicly available. 1 1 1 https://github.com/minjoong507/EgoExo-Con

## 1 Introduction

Recent advances in video large language models (Video-LLMs) have shown impressive capabilities in question answering (Zhang et al., 2024b; Wang et al., 2024a; 2025a) and temporal grounding (Ren et al., 2024; Huang et al., 2024; Zeng et al., 2024; Wang et al., 2025b), and are stepping towards fine-grained and long-ranged reasoning (Zhang et al., 2024a; Shen et al., 2024; Wang et al., 2025c; Feng et al., 2025a). However, most benchmarks (Mangalam et al., 2023; Fu et al., 2024; Wu et al., 2024; Zhou et al., 2025) and methods assume a fixed or minimally varying viewpoint, e.g. third-person view (exo) videos, leaving open a critical question: *can Video-LLMs achieve consistent temporal understanding across different camera perspectives?*

Often, videos of the same event appear strikingly different when captured from different perspectives. A cooking demonstration filmed from a head-mounted camera (ego view) looks unlike a side-mounted tripod shot (exo view). Yet, the underlying temporal dynamics, such as cutting vegetables and stirring a pot, are identical. For humans, this view variation rarely impedes understanding; we easily track the sequence of actions and localize their temporal moments across viewpoints. This makes temporal reasoning particularly critical: while appearance cues can vary drastically with viewpoint, the temporal structure of events is invariant. Thus, evaluation of cross-view consistency is essential and can be effectively carried out through temporal understanding tasks; however, such capabilities remain largely underexplored in current Video-LLMs.

To study this, we introduce EgoExo-Con, a benchmark comprising 491 synchronized ego-exo video pairs and 3,178 temporal-bounded event queries, to evaluate whether models can provide consistent predictions across viewpoints - a key indicator of view-invariant video-language understanding. The benchmark focuses on two temporal understanding tasks: *temporal verification* (Jung et al., 2025b) and *temporal grounding* (Gao et al., 2017). Temporal verification is a binary QA task that asks whether a given event occurs within a specific video moment, while temporal grounding requires identifying the relevant video moment (start and end timestamps) corresponding to an event query. In both tasks, we ask the same event but with synchronized videos of different viewpoints, and check if the tested models can output correct and consistent responses.

We evaluate both the advanced closed-source models (Comanici et al., 2025; OpenAI, 2025) and open-source Video-LLMs comprising general-purpose (Li et al., 2024a; Zhang et al., 2025a; Bai et al., 2025; Cheng et al., 2024b) and time-aware variants (Ren et al., 2024; Huang et al., 2024; Zeng et al., 2024; Jung et al., 2025b). Our benchmark results reveal that all models, especially the open-source ones, struggle with cross-view consistency. They generally exhibit a modest performance gap between individual ego and exo videos, but achieve consistency scores barely over half their single-view performance in both tasks. This indicates that the relatively stable performances across viewpoints may be sourced from view-specific biases rather than robust cross-view temporal understanding. Our further investigations show that naive multi-view supervised fine-tuning (SFT) with synchronized video-language data is insufficient. In fact, it even underperforms the counterpart with single-view training, reflecting that naively merging viewpoints may introduce conflicting priors, undermining temporal signals and consistency. These collectively suggest that *viewpoint variation remains a significant challenge for current Video-LLMs in robust video temporal understanding*.

Finally, we propose View-GRPO, a reinforcement learning (RL) framework that strengthens temporal reasoning across viewpoints while aligning the final conclusions. Experiments demonstrate that this approach yields more robust and consistent video understanding than standard finetuning. In summary, EgoExo-Con establishes a new paradigm for evaluating and improving view-invariant temporal understanding in Video-LLMs. We hope it will foster future research on models that truly capture the essence of dynamic events independent of perspective.

Our primary contributions are summarized as follows:

- •
We study the robustness of Video-LLMs in cross-view video temporal understanding and introduce EgoExo-Con, a synchronized ego–exo benchmark constructed with manual annotation efforts.
- •
We reveal that current Video-LLMs achieve cross-view consistency barely better than half of their sing-view performance, and naively blending perspectives for training could introduce conflicting priors, undermining consistency rather than improving it.
- •
We propose View-GRPO and construct View30K, a reinforced approach and dataset to explicitly strengthen temporal reasoning while encouraging view-invariant comprehension, significantly outperforming naive SFT and GRPO.

## 2 Related Work

Video Large Language Models.
Video-LLMs (Li et al., 2024a; Zhang et al., 2024b; 2025a; Bai et al., 2025; Wang et al., 2025a; Comanici et al., 2025) integrate pretrained video representations into powerful LLMs (Grattafiori et al., 2024; Bai et al., 2025; Yang et al., 2025) to enable chatting about videos. While advances to date are mostly achieved in short and coarse-grained question answering (Xu et al., 2017; Yu et al., 2019), more recent Video-LLMs (Ren et al., 2024; Huang et al., 2024; Qian et al., 2024; Guo et al., 2024; Wang et al., 2024b; Zeng et al., 2024; Meinardus et al., 2024; Jung et al., 2025b; Li et al., 2025) have explored grasping fine-grained temporal moments (“when”) (Gao et al., 2017; Jung et al., 2025a). However, all of these models solve videos captured in a single camera viewpoint (either exo or ego) and do not evaluate whether temporal reasoning remains stable across views of the same event. This work thus fills such gap by conducting a comprehensive analysis.

Ego-Exo Benchmarks.
Most benchmarks target either exocentric (Gao et al., 2017; Yu et al., 2019; Xiao et al., 2021; 2024; Fu et al., 2024) or egocentric (Mangalam et al., 2023; Di & Xie, 2024; Cheng et al., 2024a; Ye et al., 2015; Xiao et al., 2025) video understanding, with only a few offering paired views: CharadesEgo (Sigurdsson et al., 2018), LEMMA (Jia et al., 2020), Ego-Exo4D (Grauman et al., 2024a), Assembly101 (Sener et al., 2022), EgoExo-Fitness (Li et al., 2024b), and EgoExOR (Özsoy et al., 2025). Yet, they are either domain-specific or do not evaluate cross-view temporal reasoning. A concurrent effort, EgoExoBench (He et al., 2025), explores cross-view temporal reasoning, but it primarily targets action ordering via multi-choice selection, and does not consider prediction consistency. Our EgoExo-Con, however, studies a more challenging setting that requires aligning queries with corresponding video moments across viewpoints in a consistent manner.

Ego-Exo Learning.
Research on egocentric–exocentric video understanding primarily studies representation alignment and cross-view adaptation. For instance, prior efforts (Sigurdsson et al., 2018; Wang et al., 2023; Xue & Grauman, 2023; Luo et al., 2025) in action recognition have proposed self-supervised methods based on contrastive objectives for view-invariant representation learning. Another line of studies (Li et al., 2021; Luo et al., 2024; Zhang et al., 2025b) has explored distilling knowledge from one view to another. However, these efforts rarely examine whether the temporal reasoning of Video-LLMs remains consistent when identical events are observed from different viewpoints. In this work, we investigate this and propose a RL-based approach to improve temporal reasoning consistency across viewpoints, inspired by the recent success of RL learning in video reasoning (Feng et al., 2025b; Wang et al., 2025b; Liao et al., 2025; Zhang et al., 2025c).

## 3 EgoExo-Con Dataset

### 3.1 Data Collection

We source data from three datasets: CharadesEgo (Sigurdsson et al., 2018), LEMMA (Jia et al., 2020), and Ego-Exo4D (Grauman et al., 2024b), as they cover diverse and general domains, whereas other benchmarks are often restricted to specific domains (e.g., fitness (Li et al., 2024b), toy assembly (Sener et al., 2022), and surgery (Özsoy et al., 2025)). CharadesEgo and LEMMA feature daily human-object interactions, while EgoExo-4D spans diverse skilled tasks, such as bike repairing and rock climbing. Among them, we collect synchronized video data annotated with query-timestamp pairs to support our focus on temporal understanding tasks. To ensure reliable evaluation, we carefully balance video diversity with model feasibility. For example, our preliminary experiments find that current models hardly perform effective temporal localization for long videos, making it infeasible to analyze consistency. Thus, we segment videos longer than five minutes into multiple clips surrounding the ground-truth moments, with each video clip lasting for at least two minutes, thus maximally preserving content diversity while keeping the task manageable for existing Video-LLMs.

Figure: Figure 1: Examples of queries and corresponding video moments from existing datasets. (a) and (b) highlight fundamental limitations, with the egocentric view (top) in (a) being insufficient due to differing focuses, and the exocentric view (bottom) in (b) being ambiguous due to occlusion and distance. Although the query in (c) is identifiable from both viewpoints, we enrich it with details.
Refer to caption: x1.png

### 3.2 Data Filtering and Refinement

Unfortunately, the original temporal queries in datasets often do not meet our requirements. In Fig. [1](https://arxiv.org/html/2510.26113v1#S3.F1), (a) queries in CharadesEgo are template-based, with categorical actions, and (b) queries in LEMMA rely on atomic actions and objects, both of which tend to miss details. More critically, viewpoint-induced ambiguities hinder reliable evaluation for cross-view consistency: key elements may be visible from one viewpoint but obscured from another due to varied focus and temporal alignment. For instance, the query “A person is smiling” in Fig. [1](https://arxiv.org/html/2510.26113v1#S3.F1)-(a) is not visible from the egocentric video. Overlooking these issues undermines reliable evaluation and analysis.

To address this, we reformulate queries in multiple stages. We convert the per-frame Human-Object Interaction (HOI) labels (e.g., put + cup, fridge) in LEMMA into natural-language queries. Since HOI labels are grounded to short 1–2 second intervals, we aggregate consecutive annotations into longer spans, extract salient verbs and nouns, and verbalize them into natural-language queries using simple rules for targets and prepositions. Next, we utilize a powerful large model (i.e., GPT-4o (Achiam et al., 2023)) to enrich the original queries across all datasets. Specifically, given sampled frames from the target moments, the model verifies whether a query contains elements that cannot be reliably inferred from one or both viewpoints and produces refined alternatives. Additionally, the model generates a *misaligned query* containing irrelevant content, which serves as a negative sample for temporal verification, thus balancing answers for “yes” and “No” in the verification task. The full prompt is shown in Appendix Fig. [11](https://arxiv.org/html/2510.26113v1#A4.F11).

Finally, we perform human validation for all samples. Four human evaluators review the generated queries alongside associated synchronized video pairs. They confirm whether refined queries are accurately grounded in both viewpoints, and misaligned queries intentionally conflict with the visual content. Queries passing this validation are retained, while ambiguous or low-quality samples are refined further or discarded (e.g., if the video itself is too noisy). Uncertain cases are cross-checked with the authors to ensure reliability. Fig. [3](https://arxiv.org/html/2510.26113v1#S3.F3) presents examples of refined and misaligned queries generated from the original query and its associated video.

Figure: Figure 2: Statistics of EgoExo-Con. The numbers below (a) show the video and moment counts per subset, and those in (b) and (c) show their average lengths, respectively. The statistics suggest the high diversity of EgoExo-Con in data sources, video and moment lengths.
Refer to caption: x2.png

### 3.3 Dataset Statistics

Eventually, we obtain 491 synchronized video pairs (213, 169, and 82 pairs are from CharadesEgo, EgoExo4D, and LEMMA, respectively) and 3,178 tightly aligned queries with timestamps. As shown in Fig. [2](https://arxiv.org/html/2510.26113v1#S3.F2), each subset introduces distinct challenges specific to its domain and contributes a diverse range of video and moment lengths. Collectively, they provide extensive coverage of content, making EgoExo-Con a highly diverse and comprehensive benchmark to challenge temporal capabilities.

Figure: Figure 3: Examples of test data and the corresponding model responses. We create refined and misaligned queries from each original query, use them for temporal verification (V) and grounding (G), and assess cross-view answer consistency.
Refer to caption: x3.png

## 4 Evaluation

### 4.1 Models

Baseline. We evaluate a series of open-sourced models and categorize them as general-purpose or time-aware models, depending on whether they are designed for generic question-answering or specifically tuned to provide answers with corresponding video timestamps. Four general-purpose models: VideoChat2 (Li et al., 2024a), Qwen2.5-VL (Bai et al., 2025), and Video-LLaMA2 (Cheng et al., 2024b), and Video-LLaMA3 (Zhang et al., 2025a), and four time-aware models: VTimeLLM (Huang et al., 2024), TimeChat (Ren et al., 2024), TimeSuite (Zeng et al., 2024), and TimeChat-VT (Jung et al., 2025b), are included. We provide details of each model in Appendix [A](https://arxiv.org/html/2510.26113v1#A1). Additionally, we include two powerful closed-source models: GPT-5 (OpenAI, 2025) and Gemini-2.5 Flash (Comanici et al., 2025). We also benchmark human performance as a reference. We invite four evaluators and present each viewpoint independently to avoid biased predictions, reporting the average of their scores. Note that close-sourced and human performance are reported on a randomly sampled subset (i.e., $\simeq$30% of the whole set) to control the expenses. Additionally, we benchmark a random method that always returns “yes” or the entire video span for temporal verification and grounding, respectively.

Evaluation Metric.
We use accuracy in percentage for temporal verification (V) and R@1, Intersection over Union (IoU)=0.5 for temporal grounding (G). For grounding, predictions are considered correct if their IoU with the ground-truth moment exceeds 0.5. A model is evaluated separately for each viewpoint: V-Ego and V-Exo measure binary accuracy for egocentric and exocentric videos, respectively. Similarly, G-Ego and G-Exo denote grounding performance. Consistency metrics, V-EgoExo and G-EgoExo, measure whether a model correctly verifies or grounds specific moments (i.e., IoU $<$ 0.5) for both synchronized videos; consistent but wrong answers are not considered.

**Table 1: Performance on EgoExo-Con. F: input frames. Ego: include ego data for training.**
| Methods | # F | Ego | EgoExo-Con |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo |  |  |  |
| Human | - | - | 92.1 | 91.3 | 89.4 | 72.4 | 73.0 | 67.3 |
| Closed-source |  |  |  |  |  |  |  |  |
| GPT-5 (OpenAI, 2025) | 32 | - | 60.5 | 61.3 | 52.5 | 34.5 | 32.8 | 20.1 |
| Gemini-2.5 Flash (Comanici et al., 2025) | 1 fps | - | 70.4 | 70.1 | 52.3 | 42.0 | 45.9 | 20.8 |
| Random | - | - | 50.0 | 50.0 | 50.0 | 12.5 | 12.5 | 12.5 |
| General-purpose |  |  |  |  |  |  |  |  |
| VideoChat2 (Li et al., 2024a) | 16 | ✓ | 46.0 | 45.1 | 23.4 | 5.6 | 5.3 | 4.0 |
| Qwen2.5-VL (Bai et al., 2025) | 1 fps | ✗ | 54.3 | 56.3 | 33.0 | 14.2 | 11.4 | 6.9 |
| Video-LLaMA2 (Cheng et al., 2024b) | 8 | ✓ | 53.3 | 52.1 | 27.9 | 12.0 | 11.5 | 7.5 |
| Video-LLaMA3 (Zhang et al., 2025a) | 1 fps | ✓ | 56.7 | 54.6 | 36.6 | 27.7 | 28.0 | 16.2 |
| Time-aware |  |  |  |  |  |  |  |  |
| VTimeLLM (Huang et al., 2024) | 100 | ✗ | 48.9 | 48.5 | 23.5 | 12.6 | 11.1 | 6.5 |
| TimeChat (Ren et al., 2024) | 96 | ✗ | 48.9 | 48.4 | 25.1 | 21.3 | 20.5 | 12.8 |
| TimeSuite (Zeng et al., 2024) | 128 | ✓ | 47.4 | 48.5 | 25.6 | 28.2 | 27.3 | 18.7 |
| TimeChat-VT (Jung et al., 2025b) | 96 | ✗ | 62.1 | 61.4 | 42.1 | 27.8 | 26.2 | 16.3 |

### 4.2 Performance Analysis on EgoExo-Con

Our analyses on the results of EgoExo-Con in Table [1](https://arxiv.org/html/2510.26113v1#S4.T1) are as follows:
(1) Single view vs. Cross view. While all models show a modest performance gap between individual ego and exo videos, they struggle with cross-view consistency in both tasks. Open-source models in particular achieve barely half of their single-view performance. This demonstrates that the relatively stable performances across viewpoints are largely due to view-specific bias cues but not robust cross-view reasoning.
(2) Time-aware vs. General-purpose. Time‑aware models generally lead on grounding. TimeSuite attains the strongest grounding consistency, and TimeChat‑VT is competitive while also giving the best verification consistency. This advantage likely comes from better instruction tuning. However, VTimeLLM underperforms most general-purpose models, showcasing that time-aware models are not necessarily superior to general-purpose ones in temporal reasoning.
(3) Closed-sourced vs Human. Closed-source models generally outperform open-source ones, reflecting their stronger capabilities. Yet a substantial gap (37% 47%) in cross-view consistency remains compared to humans, with scores approaching random, underscoring the challenge of EgoExo-Con and the room for improvement.
(4) Training with ego data is not sufficient. Models including egocentric videos for training do not consistently yield higher consistency than others trained on exocentric videos alone, showing that simply mixing ego and exo data does not benefit consistency.
(5) Temporal reasoning outweighs increasing frames for better results. Video‑LLaMA2 (8 frames) outperforms VideoChat2 (16 frames) across all metrics. TimeChat‑VT (96 frames) also outperforms several models that use more/less context, suggesting reasoning and temporal modeling matter more than sheer frame count.

Furthermore, we analyze model behaviors across subsets by plotting the ego–exo performance gap in Fig. [4](https://arxiv.org/html/2510.26113v1#S4.F4). Patterns are consistent across grounding and verification: in CharadesEgo, most models perform better on exocentric views (blue), whereas in LEMMA, they tend to favor egocentric views (red); EgoExo-4D shows mixed but generally smaller gaps. These trends likely reflect domain characteristics. As shown in Fig. [3](https://arxiv.org/html/2510.26113v1#S3.F3), when a person performs in a fixed position, egocentric videos often provide favorable cues such as clearer hand–object interactions. Conversely, when a person moves around or changes location, egocentric views can become more challenging due to rapid scene shifts, while exocentric views offer greater stability. Such trends are particularly pronounced in CharadesEgo. Detailed results across subsets are provided in Appendix Table [5](https://arxiv.org/html/2510.26113v1#A2.T5). Although the relative effectiveness of each viewpoint varies by domain, they do not significantly affect overall consistency. Overall, across models, consistency scores lag far behind single-view metrics, underscoring that achieving robust, view-invariant temporal understanding remains an open challenge.

Figure: Figure 4: Heatmaps of the performance gap. All values are reported in percentage points. Red and blue indicate higher performances on *ego* and *exo* perspectives, respectively. i.e., a blue cell indicates that the corresponding model performs better on exocentric videos than on egocentric ones.
Refer to caption: x4.png

### 4.3 Supervised Performance on EgoExo-Con

In this section, we study whether supervised fine-tuning (SFT) with synchronized ego-exo video data improves performance.
We first collect 3.6k from CharadesEgo and 2.3k videos from Ego-Exo4D training sets. We exclude LEMMA due to its limited size (i.e., 243 videos for training). Then we finetune two general-purpose models: VideoChat2 and Video-LLaMA for temporal verification, and two time-aware models: TimeChat and TimeSuite for temporal grounding. All models apply LoRA (Hu et al., 2022) fine-tuning with their official configurations. More implementation details are presented in Appendix [B.2](https://arxiv.org/html/2510.26113v1#A2.SS2).

**Table 2: Fine-tuned performance. The left and right tables report model performance for temporal verification (V) and temporal grounding (G). The first row in each model reports zero-shot performance, while the subsequent rows present results after fine-tuning on either (+Ego, +Exo) or both viewpoints (+ EgoExo). Notably, models trained on both viewpoints are not always the best-performing, and then they barely outperform those trained on a single viewpoint.**
| Methods | CharadesEgo | EgoExo-4D |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| V-Exo | V-Ego | V-ExoEgo | V-Exo | V-Ego | V-ExoEgo |  |
| VideoChat2 | 46.3 | 44.4 | 22.2 | 41.4 | 41.5 | 19.8 |
| - Ego | 56.2 | 59.2 | 36.4 | 46.5 | 49.2 | 29.7 |
| - Exo | 56.6 | 57.5 | 35.5 | 46.1 | 47.3 | 29.4 |
| - EgoExo | 56.4 | 57.1 | 34.7 | 48.3 | 50.1 | 30.1 |
| Video-LLaMA2 | 54.0 | 52.4 | 28.2 | 51.5 | 52.8 | 28.1 |
| + Ego | 57.0 | 58.2 | 31.7 | 60.2 | 60.3 | 38.8 |
| + Exo | 57.6 | 56.1 | 31.4 | 59.8 | 60.1 | 39.2 |
| + EgoExo | 58.5 | 57.3 | 31.0 | 57.5 | 59.6 | 39.1 |

Table [2](https://arxiv.org/html/2510.26113v1#S4.T2) reports results of three different settings: training on either viewpoint (Ego, Exo) or both viewpoints together (ExoEgo). All of them consistently improve over zero-shot baselines. Interestingly, despite utilizing twice the data, training on both viewpoints (ExoEgo) yields only marginal gains and often underperforms the models trained on a single view. Specifically, in CharadesEgo, TimeSuite shows a notable 8.1% gap in consistency between training on both viewpoints and training only on exocentric videos. Furthermore, the improvements are still limited when unfreezing the visual encoder (Please refer to Appendix Table [4](https://arxiv.org/html/2510.26113v1#A2.T4)).

The above findings are consistent with observations in cross-view learning (Li et al., 2024b; Sigurdsson et al., 2018), where naively blending perspectives does not always bring improvement. Without explicit alignment, conflicting priors across tasks or domains undermine temporal signals and consistency rather than improvement.

Figure: Figure 5: Overview of our approach. (a) In supervised fine-tuning, the model is trained to directly predict the same query answers (e.g., video moments) for synchronized video pairs. (b) View-GRPO trains a model to provide viewpoint-specific reasoning chains, which are generated by GPT-5 (top).
Refer to caption: x5.png

## 5 Method

Although synchronized videos depict the same content, the reasoning process often differs across viewpoints because of distinct focuses and perspectives. To address this, we propose a reinforcement learning (RL) framework that guides models toward developing viewpoint-specific reasoning while encouraging shared consistency. Rather than simply enforcing identical outputs, our approach explicitly promotes robust reasoning across viewpoints. We build on Group Relative Policy Optimization (GRPO), which is particularly well-suited as it leverages relative rewards instead of absolute scores.

### 5.1 Background: Group Relative Policy Optimization (GRPO)

GRPO (Shao et al., 2024) is a reinforcement learning algorithm designed to refine large language model outputs by leveraging relative ranking among multiple candidate responses. Instead of treating each response independently with absolute rewards, GRPO evaluates a set of responses produced for the same prompt, assigning rewards in relation to the group. This group-wise normalization helps reduce reward variance and makes optimization more stable compared to approaches relying solely on absolute scores or pairwise comparisons.

Given a prompt $p$, the model generates $G$ candidate responses $o=\{o_{1},\ldots,o_{G}\}$. Each response receives a reward value $r(o_{i})$. GRPO then standardizes these scores within the group and optimizes a weighted objective:

$$ $R(o)=\sum_{i=1}^{G}\frac{\pi_{\theta}(o_{i})}{\pi_{\theta_{\text{old}}}(o_{i})}\cdot\frac{r(o_{i})-\text{mean}(\{r(o_{i})\}_{i=1}^{G})}{\text{std}(\{r(o_{i})\}_{i=1}^{G})},$ (1) $$

where $\pi_{\theta}(o)$ denotes the current policy and $\pi_{\theta_{\text{old}}}(o)$ is the previous policy. To prevent divergence from the base model, KL-divergence regularization is added:

$$ $\max_{\pi_{\theta}}\;\mathbb{E}_{o\sim\pi_{\theta_{\text{old}}}(p)}\Big[R(o)-\beta D_{\text{KL}}\big(\pi_{\theta}\;\|\;\pi_{\text{ref}}\big)\Big],$ (2) $$

where $\pi_{ref}$ is the base model and $\beta$ controls the strength of the regularization. Please refer to the original paper for more details.

### 5.2 Learning Temporal Reasoning across Viewpoints

To adapt GRPO for cross-view reasoning, we first curate training data, including temporal reasoning chains for both egocentric and exocentric views. Fig. [5](https://arxiv.org/html/2510.26113v1#S4.F5) (Top) illustrates the generation of video reasoning data. We prompt GPT-5 to produce step-by-step reasoning chains for each video in order to solve the given task, ensuring viewpoint-specific reasoning while aligning the final answers. We discard samples lacking valid reasoning in either view, as these typically indicate ambiguous or low-quality data. Specifically, we exclude cases where the model explicitly states its failure in the answer, or where the predicted moment has a temporal IoU (tIoU) below 0.7 with the ground-truth. After filtering, we retain 3.3k videos with 30k reasoning instances, which we name as View30K.

With the dataset, we then design reward functions comprising three major components:

#### 1. Format Reward.

To encourage structured reasoning and easy answer extraction, responses must follow the template:
<think>...</think><answer>...</answer>. Formally:

$$ $r_{\text{form}}(o)=\begin{cases}1,&\text{if $o$ follows the required format},\\ 0,&\text{otherwise}.\end{cases}$ (3) $$

#### 2. Accuracy Reward.

We unify task-specific accuracy into $r_{\text{acc}}$. For temporal grounding, the reward is the temporal Intersection-over-Union (tIoU) between ground truth $[t_{s},t_{e}]$ and prediction $[t^{\prime}_{s},t^{\prime}_{e}]$. For verification, it is binary correctness:

$$ $r_{\text{acc}}(o)=\begin{cases}\tfrac{|[t_{s},t_{e}]\cap[t^{\prime}_{s},t^{\prime}_{e}]|}{|[t_{s},t_{e}]\cup[t^{\prime}_{s},t^{\prime}_{e}]|},&\text{if {grounding}},\[10.0pt] \mathbbm{1}[o=o^{*}],&\text{if {verification}}.\end{cases}$ (4) $$

#### 3. Reasoning Reward.

We design $r_{\text{sim}}$ to measure how closely the generated reasoning aligns with the target ones. We employ LLM (i.e., Qwen2.5-3B (Qwen et al., 2024)) as a judge and design a tailored prompt (see Appendix Fig. [12](https://arxiv.org/html/2510.26113v1#A4.F12)). Specifically, the judge model provides a similarity score on a scale of 0 to 1 between the candidate output $o$ and reference $o^{*}$.

Overall, the total reward integrates all components to balance correctness and reasoning quality:

$$ $r(o)=r_{\text{acc}}(o)+r_{\text{form}}(o)+r_{\text{sim}}(o),$ (5) $$

thus enabling models to learn correct and consistent cross-view temporal reasoning. We name the overall approach View-GRPO, as shown in Fig. [5](https://arxiv.org/html/2510.26113v1#S4.F5)-(b) (Bottom).

### 5.3 Implementations

**Table 3: Performance of View-GRPO on EgoExo-Con.**
| Methods | EgoExo-Con |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo |  |
| Qwen2.5-VL-3B | 51.0 | 52.5 | 28.1 | 10.1 | 9.9 | 7.9 |
| + SFT | 52.7 | 51.2 | 30.6 | 16.3 | 16.6 | 12.9 |
| + GRPO | 52.5 | 52.9 | 30.3 | 15.4 | 16.3 | 10.3 |
| + View-GRPO | 54.0 $\uparrow$3.0 | 54.1 $\uparrow$1.6 | 33.9 $\uparrow$5.8 | 18.6 $\uparrow$8.5 | 17.9 $\uparrow$8.0 | 14.8 $\uparrow$6.9 |
| Qwen2.5-VL-7B | 54.3 | 56.3 | 33.0 | 14.2 | 11.4 | 6.9 |
| + SFT | 57.6 | 58.0 | 41.4 | 18.3 | 17.8 | 14.9 |
| + GRPO | 55.2 | 57.6 | 39.8 | 18.6 | 16.1 | 14.3 |
| + View-GRPO | 58.3 $\uparrow$4.0 | 58.1 $\uparrow$1.8 | 44.7 $\uparrow$11.7 | 21.5 $\uparrow$7.3 | 21.0 $\uparrow$9.6 | 18.3 $\uparrow$11.4 |

We use Qwen2.5-VL (Bai et al., 2025) as a base model. During training, all experiments set the same frame sampling rate (i.e., 2 FPS) and the same number of input video pixels for fair comparison. The training involves 8 $\times$ A100 GPUs and requires over 1 day for the 3B model and 2 days for the 7B model. Further implementation details are in Appendix [B.3](https://arxiv.org/html/2510.26113v1#A2.SS3).

### 5.4 Analyses on View-GRPO

Table [3](https://arxiv.org/html/2510.26113v1#S5.T3) shows the performance of our method, View-GRPO, on EgoExo-Con compared to SFT and basic GRPO implementation with only format and accuracy rewards. View-GRPO, which highlights an additional reasoning reward and related data for GRPO training, consistently outperforms SFT and basic GRPO. The most significant improvements are often on cross-view consistency (V-ExoEgo and G-ExoEgo), although it also benefits individual views. We conjecture that the reasoning reward plays a central role, as it delivers noticeably higher consistency compared to naive GRPO. By encouraging models to produce faithful, step-by-step temporal explanations tailored to each viewpoint while converging toward consistent temporal conclusions, the model reduces reliance on view-specific biases and instead learns shared temporal abstractions. Fig. [7](https://arxiv.org/html/2510.26113v1#S5.F7) shows the step-by-step temporal reasoning contents generated by View-GRPO and how reliable reasoning gives rise to robust temporal grounding across viewpoints. Overall, View-GRPO demonstrates its effectiveness and potential for more consistent cross-view video temporal understanding.

Figure: Figure 6: Reward across different LLM judges. Qwen2.5-0.5B raises calibration concerns due to its overly high reasoning rewards from early steps.
Refer to caption: x6.png

While LLMs commonly serve as judges (Zheng et al., 2023; Xie et al., 2023), their role in optimizing models within View-GRPO remains unclear despite observed improvement. In fact, they often introduce potential bias and uncertainty in video evaluation (Cores et al., 2024; Liu & Zhang, 2025). To provide insights into this, we train Qwen2.5-VL-3B and employ judge models of different scales (i.e., Qwen2.5-0.5B and Qwen2.5-3B) for temporal grounding and analyze their impact on optimization. In Fig. [6](https://arxiv.org/html/2510.26113v1#S5.F6)-(a), format and accuracy rewards remain relatively stable across scales. However, Qwen2.5-0.5B produces overly high reasoning rewards from the very first training steps in Fig. [6](https://arxiv.org/html/2510.26113v1#S5.F6)-(b), raising concerns about calibration and reliability. We find that this behavior leads to a measurable consistency degradation (i.e., –3% in G-EgoExo), underscoring the critical influence of LLM-judge choice and utilization in model optimization. We leave further exploration of this direction for future work.

Figure: Figure 7: Visualization of View-GRPO. The model provides step-by-step temporal reasoning with accurate grounding prediction, achieving high reasoning reward scores from LLM-judges.
Refer to caption: x7.png

## 6 Conclusion

In this work, we introduced EgoExo-Con that comprises synchronized egocentric and exocentric videos paired with human-refined queries and evaluates whether models perform consistent temporal understanding across viewpoints. EgoExo-Con revealed that Video-LLMs struggle with cross-view consistency, often lagging behind single-viewpoint performance, and that naively training on both viewpoints does not reliably help. To address this, we proposed View-GRPO that encourages viewpoint-specific temporal reasoning while promoting cross-view alignment, demonstrating its effectiveness over alternative training strategies. We hope EgoExo-Con establishes a solid benchmark for faithful, consistent temporal understanding across viewpoints, and View-GRPO provides insights toward achieving robust, view-invariant video comprehension.

## Acknowledgements

This research was supported by the IITP (RS-2021-II212068-AIHub/10%, RS-2021-II211343-GSAI/15%, RS-2022-II220951-LBA/15%, RS-2022-II220953-PICA/20%), NRF (RS-2024-00353991-SPARC/20%, RS-2023-00274280-HEI/10%), and KEIT (RS-2024-00423940/10%) grant funded by the Korean government. And we thank Il-Jae Kwon, Dohyun Yu, and Shin Jane for their valuable feedback.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Cheng et al. (2024a)
Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu.
Egothink: Evaluating first-person perspective thinking capability of vision-language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14291–14302, 2024a.
- Cheng et al. (2024b)
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al.
Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.
*arXiv preprint arXiv:2406.07476*, 2024b.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Cores et al. (2024)
Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano.
Lost in time: A new temporal benchmark for videollms.
*arXiv preprint arXiv:2410.07752*, 2024.
- Di & Xie (2024)
Shangzhe Di and Weidi Xie.
Grounded question-answering in long egocentric videos.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12934–12943, 2024.
- Feng et al. (2025a)
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025a.
- Feng et al. (2025b)
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025b.
- Fu et al. (2024)
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
*arXiv preprint arXiv:2405.21075*, 2024.
- Gao et al. (2017)
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.
Tall: Temporal activity localization via language query.
In *Proceedings of the IEEE international conference on computer vision*, pp. 5267–5275, 2017.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Grauman et al. (2024a)
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al.
Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 19383–19400, 2024a.
- Grauman et al. (2024b)
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al.
Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 19383–19400, 2024b.
- Guo et al. (2024)
Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Xi Chen, and Bo Zhao.
Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding.
*arXiv preprint arXiv:2405.13382*, 2024.
- He et al. (2025)
Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang.
Egoexobench: A benchmark for first-and third-person view video understanding in mllms.
*arXiv preprint arXiv:2507.18342*, 2025.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 2022.
- Huang et al. (2024)
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu.
Vtimellm: Empower llm to grasp video moments.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14271–14280, 2024.
- Jia et al. (2020)
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu.
Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities.
In *European Conference on Computer Vision*, pp. 767–786. Springer, 2020.
- Jung et al. (2025a)
Minjoon Jung, Youwon Jang, Seongho Choi, Joochan Kim, Jin-Hwa Kim, and Byoung-Tak Zhang.
Background-aware moment detection for video moment retrieval.
In *Proceedings of the Winter Conference on Applications of Computer Vision*, pp. 8575–8585, February 2025a.
- Jung et al. (2025b)
Minjoon Jung, Junbin Xiao, Byoung-Tak Zhang, and Angela Yao.
On the consistency of video large language models in temporal comprehension.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 13713–13722, 2025b.
- Li et al. (2024a)
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al.
Mvbench: A comprehensive multi-modal video understanding benchmark.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 22195–22206, 2024a.
- Li et al. (2021)
Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman.
Ego-exo: Transferring visual representations from third-person to first-person videos.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 6943–6953, 2021.
- Li et al. (2024b)
Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng.
Egoexo-fitness: Towards egocentric and exocentric full-body action understanding.
In *European Conference on Computer Vision*, pp. 363–382. Springer, 2024b.
- Li et al. (2025)
Zeqian Li, Shangzhe Di, Zhonghua Zhai, Weilin Huang, Yanfeng Wang, and Weidi Xie.
Universal video temporal grounding with generative multi-modal large language models.
*arXiv preprint arXiv:2506.18883*, 2025.
- Liao et al. (2025)
Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng.
Improved visual-spatial reasoning via r1-zero-like training.
*arXiv preprint arXiv:2504.00883*, 2025.
- Liu & Zhang (2025)
Ming Liu and Wensheng Zhang.
Is your video language model a reliable judge?
*arXiv preprint arXiv:2503.05977*, 2025.
- Luo et al. (2024)
Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman.
Put myself in your shoes: Lifting the egocentric perspective from exocentric videos.
In *European Conference on Computer Vision*, pp. 407–425. Springer, 2024.
- Luo et al. (2025)
Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman.
Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 15802–15812, 2025.
- Mangalam et al. (2023)
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.
Egoschema: A diagnostic benchmark for very long-form video language understanding.
*Advances in Neural Information Processing Systems*, 36:46212–46244, 2023.
- Meinardus et al. (2024)
Boris Meinardus, Hector Rodriguez, Anil Batra, Anna Rohrbach, and Marcus Rohrbach.
Chrono: A simple blueprint for representing time in mllms.
*arXiv preprint arXiv:2406.18113*, 2024.
- OpenAI (2025)
OpenAI.
Introducing gpt-5.
2025.
- Özsoy et al. (2025)
Ege Özsoy, Arda Mamur, Felix Tristram, Chantal Pellegrini, Magdalena Wysocki, Benjamin Busam, and Nassir Navab.
Egoexor: An ego-exo-centric operating room dataset for surgical activity understanding.
*arXiv preprint arXiv:2505.24287*, 2025.
- Qian et al. (2024)
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang.
Momentor: Advancing video large language model with fine-grained temporal reasoning.
*arXiv preprint arXiv:2402.11435*, 2024.
- Qwen et al. (2024)
A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, et al.
Qwen2. 5 technical report.
*arXiv preprint*, 2024.
- Ren et al. (2024)
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou.
Timechat: A time-sensitive multimodal large language model for long video understanding.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14313–14323, 2024.
- Sener et al. (2022)
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao.
Assembly101: A large-scale multi-view video dataset for understanding procedural activities.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 21096–21106, 2022.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shen et al. (2024)
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al.
Longvu: Spatiotemporal adaptive compression for long video-language understanding.
*arXiv preprint arXiv:2410.17434*, 2024.
- Sigurdsson et al. (2018)
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari.
Actor and observer: Joint modeling of first and third-person videos.
In *proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 7396–7404, 2018.
- Wang et al. (2023)
Qitong Wang, Long Zhao, Liangzhe Yuan, Ting Liu, and Xi Peng.
Learning from semantic alignment between unpaired multiviews for egocentric video recognition.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 3307–3317, 2023.
- Wang et al. (2025a)
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al.
Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.
*arXiv preprint arXiv:2508.18265*, 2025a.
- Wang et al. (2024a)
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy.
Videoagent: Long-form video understanding with large language model as agent.
*arXiv preprint arXiv:2403.10517*, 2024a.
- Wang et al. (2025b)
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al.
Time-r1: Post-training large vision language model for temporal video grounding.
*arXiv preprint arXiv:2503.13377*, 2025b.
- Wang et al. (2025c)
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al.
Internvideo2. 5: Empowering video mllms with long and rich context modeling.
*arXiv preprint arXiv:2501.12386*, 2025c.
- Wang et al. (2024b)
Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao.
Hawkeye: Training video-text llms for grounding text in videos.
*arXiv preprint arXiv:2403.10228*, 2024b.
- Wu et al. (2024)
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li.
Longvideobench: A benchmark for long-context interleaved video-language understanding.
*Advances in Neural Information Processing Systems*, 37:28828–28857, 2024.
- Xiao et al. (2021)
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.
Next-qa: Next phase of question-answering to explaining temporal actions.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 9777–9786, 2021.
- Xiao et al. (2024)
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua.
Can i trust your answer? visually grounded video question answering.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 13204–13214, 2024.
- Xiao et al. (2025)
Junbin Xiao, Nanxin Huang, Hao Qiu, Zhulin Tao, Xun Yang, Richang Hong, Meng Wang, and Angela Yao.
Egoblind: Towards egocentric visual assistance for the blind people.
*arXiv preprint arXiv:2503.08221*, 2025.
- Xie et al. (2023)
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu.
Text2reward: Reward shaping with language models for reinforcement learning.
*arXiv preprint arXiv:2309.11489*, 2023.
- Xu et al. (2017)
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.
Video question answering via gradually refined attention over appearance and motion.
In *Proceedings of the 25th ACM international conference on Multimedia*, pp. 1645–1653, 2017.
- Xue & Grauman (2023)
Zihui Sherry Xue and Kristen Grauman.
Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment.
*Advances in Neural Information Processing Systems*, 36:53688–53710, 2023.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Ye et al. (2015)
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al.
Mmego: Towards building egocentric multimodal llms for video qa.
In *The Thirteenth International Conference on Learning Representations*, 2015.
- Yu et al. (2019)
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.
Activitynet-qa: A dataset for understanding complex web videos via question answering.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2019.
- Zeng et al. (2024)
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al.
Timesuite: Improving mllms for long video understanding via grounded tuning.
*arXiv preprint arXiv:2410.19702*, 2024.
- Zhang et al. (2025a)
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al.
Videollama 3: Frontier multimodal foundation models for image and video understanding.
*arXiv preprint arXiv:2501.13106*, 2025a.
- Zhang et al. (2025b)
Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie.
Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding.
*arXiv preprint arXiv:2503.09143*, 2025b.
- Zhang et al. (2024a)
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu.
Long context transfer from language to vision.
*arXiv preprint arXiv:2406.16852*, 2024a.
- Zhang et al. (2025c)
Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang.
Tinyllava-video-r1: Towards smaller lmms for video reasoning.
*arXiv preprint arXiv:2504.09641*, 2025c.
- Zhang et al. (2024b)
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li.
Video instruction tuning with synthetic data.
*arXiv preprint arXiv:2410.02713*, 2024b.
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*Advances in neural information processing systems*, 36:46595–46623, 2023.
- Zhou et al. (2025)
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al.
Mlvu: Benchmarking multi-task long video understanding.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 13691–13701, 2025.

## Appendix

We provide further details that are not included in the main paper due to the page limitation.

## Appendix A Details of Models

In this section, we describe eight open-source Video-LLMs: VideoChat2 (Li et al., 2024a), Qwen2.5-VL (Bai et al., 2025), Video-LLaMA2 (Cheng et al., 2024b), Video-LLaMA3 (Zhang et al., 2025a), VTimeLLM (Huang et al., 2024), TimeChat (Ren et al., 2024), TimeSuite (Zeng et al., 2024), and TimeChat-VT (Jung et al., 2025b), which are utilized in our evaluation. Note that the size of the models is 7B.

- 1.
VideoChat2 design three different video instruction tuning stages. Specifically, they align multimodal inputs in the first stage and then generate captions from various image-text pairs. Finally, they conduct instruction tuning to better align responses with human instructions. VideoChat2 demonstrates significant improvements in video question answering benchmarks in zero-shot settings.
- 2.
Qwen2.5-VL Qwen2.5-VL is one of the latest models of Qwen vision-language series. It achieves strong benchmark performances through enhanced visual recognition, precise object localization, robust document parsing, and long-video comprehension.
- 3.
Video-LLaMA2 is one of the state-of-the-art Video-LLMs, demonstrating superior performances on video question-answering tasks. They seamlessly integrate both visual and audio modalities in videos and propose STC connector to understand spatiotemporal video information.
- 4.
Video-LLaMA3 is a series of Video-LLaMA family. It emphasizes a vision-centric training paradigm and vision-centric framework design. With curated high-quality image and video data, Video-LLaMA3 achieves compelling performances across diverse image and video understanding benchmarks.
- 5.
VTimeLLM aims to include video timestamps along with the answer for human instruction. It designs a three-stage instruction tuning. Initially, it trains a visual projection layer with image-text pairs and then incorporates video datasets with temporal modeling. VTimeLLM devises two types of QA dialogue templates, including single-turn and multi-turn, to prompt questions requiring a comprehensive description of all events and their corresponding timestamps.
- 6.
TimeChat is developed to localize and identify specific video moments from a given human instruction. It utilizes a time-aware frame encoder that injects timestamp information into visual features, leveraging Q-Former, and designs a sliding video Q-Former to handle temporal information.
- 7.
TimeSuite tackles achieving high performance on both question-answering and grounding for long videos. It argues that previous Video-LLMs struggle to achieve both capabilities and develop a VideoChat-T and a temporal-centric instruction-tuning dataset, TimePro.
- 8.
TimeChat-VT is a model that specifically takes into account consistency modeling. Upon TimeChat, it develops a new instruction tuning method, VTune, that converts temporal grounding into a verification process. This requires not only precise temporal grounding, but also confirming the occurring events from specific video moments.

## Appendix B Details of Experiments

### B.1 Prompt Templates

For temporal verification, we give variations to prompt templates beyond utilizing misaligned queries. Specifically, following the previous work (Jung et al., 2025b), we also include templates like “Is the event missing in the video?” or “Is the event not present in the video?” to shift the correct answer “Yes.” to “No.”

For temporal grounding, some general-purpose models do not officially provide prompt templates for grounding. Therefore, we borrow the prompts from the previous work (Jung et al., 2025b) for the general-purpose models and closed-sourced models, and design the prompt “Give the query, when does the described content occur in the video? Please return its start and end time using ‘start - end seconds’.” to ensure the model includes timestamps in its answer.

### B.2 Details of Supervised Fine-tuning for Video-LLMs

For each task, we design question–answer templates. In temporal verification, we use two formats: “Does ‘event’ happen from ‘st’ to ‘ed’ in the video?” and “Does ‘event’ not happen from ‘st’ to ‘ed’ in the video?”. Here, ‘event’, ‘st’, and ‘ed’ are replaced with the annotated query and its start and end timestamps. Answers are restricted to “yes” or “no.” For temporal grounding, the template “Localize the ‘event’ in the video and return its start and end times.” is employed. Note that both tasks utilize the same number of videos and queries. Again, we follow the official code and configurations in each model. All experiments run 3 epochs with 4 $\times$ A100 GPUs.

### B.3 Details of View-GRPO training

To generate video reasoning data, we design a prompt as shown in Fig. [13](https://arxiv.org/html/2510.26113v1#A4.F13) and [14](https://arxiv.org/html/2510.26113v1#A4.F14). We sample frames every 1 second (i.e., 1 FPS) from videos and give them to GPT-5. We perform batch processing using GPT-API, and it involves less than 1 day for 3.6k videos. Additionally, the model fails to generate reasoning data for 0.3k videos, and a total of 3.3k with 61k reasoning for each task remains. For training, we set the max pixels for video processing as $14\times 16\times 16$. We use the AdamW optimizer with a learning rate of 1e-6 and set the batch size to 8, 1 batch for each GPU.

**Table 4: Performance comparison between different training settings for a visual encoder. In Vis. column, represents when we freeze the visual encoder, while denotes the visual encoder is trainable. All models are trained on both viewpoints. If there is an improvement, we use a blue color; otherwise, we use a red color.**
| Methods | Vis. | EgoExo-Charades-Con | EgoExo-4D-Con |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo | V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo |  |  |
| VideoChat2 |  | 56.35 | 57.07 | 34.69 | 28.77 | 27.61 | 22.74 | 59.06 | 60.38 | 37.25 | 4.52 | 4.97 | 2.33 |
|  | 55.78 | 57.01 | 32.78 | 29.76 | 30.13 | 23.86 | 58.32 | 54.55 | 31.25 | 2.13 | 2.63 | 1.13 |  |
| TimeChat |  | 52.49 | 48.59 | 27.91 | 60.30 | 61.79 | 46.51 | 51.64 | 50.31 | 28.33 | 10.57 | 11.48 | 4.33 |
|  | 50.76 | 45.11 | 21.64 | 55.87 | 59.76 | 42.57 | 51.97 | 51.34 | 29.09 | 10.32 | 11.76 | 4.23 |  |

**Table 5: Performance on EgoExo-Con across different subsets. While performance for temporal verification remains steady across subsets, there is a noticeable performance gap between subsets in temporal grounding, likely due to the different difficulty of tasks.**
| Methods | CharadesEgo | EgoExo-4D | LEMMA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo | V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo | V-Exo | V-Ego | V-ExoEgo | G-Exo | G-Ego | G-ExoEgo |  |
| General-purpose |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VideoChat2 | 48.6 | 53.6 | 27.2 | 13.6 | 12.5 | 10.6 | 41.4 | 41.5 | 19.8 | 0.9 | 1.0 | 0.4 | 40.5 | 42.5 | 21.2 | 0.6 | 1.1 | 0.6 |
| Qwen2.5-VL | 59.1 | 58.4 | 42.2 | 31.4 | 23.1 | 16.1 | 58.4 | 59.3 | 49.5 | 5.3 | 5.3 | 2.7 | 55.6 | 59.2 | 41.9 | 0.6 | 1.7 | 0.0 |
| Video-LLaMA2 | 54.0 | 52.4 | 28.2 | 27.8 | 27.8 | 17.6 | 51.5 | 52.8 | 28.1 | 2.5 | 1.6 | 1.5 | 50.6 | 50.8 | 25.4 | 2.2 | 2.8 | 1.7 |
| Video-LLaMA3 | 61.8 | 57.9 | 40.3 | 57.1 | 51.1 | 37.3 | 52.6 | 51.5 | 33.6 | 8.7 | 12.0 | 3.1 | 53.9 | 59.5 | 36.3 | 11.7 | 22.9 | 5.6 |
| Time-aware |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VTimeLLM | 49.5 | 47.0 | 23.8 | 19.5 | 12.5 | 7.5 | 49.0 | 49.0 | 23.6 | 8.4 | 10.1 | 6.3 | 46.4 | 51.7 | 23.5 | 10.1 | 11.7 | 6.1 |
| TimeChat | 48.0 | 48.1 | 25.2 | 44.9 | 46.1 | 30.1 | 49.5 | 49.3 | 25.4 | 4.9 | 6.3 | 3.3 | 49.2 | 46.4 | 23.5 | 5.0 | 5.0 | 3.4 |
| TimeSuite | 47.7 | 47.6 | 24.7 | 63.4 | 56.2 | 44.8 | 45.8 | 47.3 | 25.2 | 5.8 | 8.3 | 2.3 | 54.0 | 56.3 | 30.6 | 10.6 | 16.2 | 6.7 |
| TimeChat-VT | 64.6 | 63.7 | 44.8 | 56.6 | 54.2 | 35.8 | 60.4 | 59.4 | 38.8 | 9.3 | 8.4 | 3.3 | 60.9 | 65.1 | 45.5 | 10.1 | 12.8 | 3.9 |

## Appendix C Additional Experiments

### C.1 The impact of Unfreezing Video Encoder

In Table [4](https://arxiv.org/html/2510.26113v1#A2.T4), we conduct fine-tuning models: VideoChat2 (Li et al., 2024a) and TimeChat (Ren et al., 2024) when fine-tuned with their video encoders unfrozen. Despite updating the visual encoders during training, we observe no further improvements; in fact, the models often underperform compared to the settings with frozen encoders. We conjecture that this may be due to overfitting or the limited scale of training data, which may not sufficiently support end-to-end tuning of large visual backbones. Furthermore, this supports the previous findings that naively mixing both viewpoints does not easily lead to improved view-invariant understanding.

### C.2 Performance on EgoExo-Con across subsets

Table [5](https://arxiv.org/html/2510.26113v1#A2.T5) reports performance across models and subsets. The performance for temporal verification remains steady across subsets. In contrast, there is a noticeable performance gap between CharadesEgo and the others. Specifically, the models tend to struggle with accurate grounding for EgoExo-4D and LEMMA than CharadesEgo due to their lengthy videos and short moments, as shown in Fig. [2](https://arxiv.org/html/2510.26113v1#S3.F2). Despite domain differences, we find consistent findings in Table [1](https://arxiv.org/html/2510.26113v1#S4.T1), significantly lagging in consistency compared to single-viewpoint performance.

## Appendix D Additional Visualization

Figure: Figure 8: A video sample in EgoExo-Fitnesses. The given query is hard to identify in the egocentric video (top).
Refer to caption: x16.png

### D.1 Discarded Cases

As we discussed in the dataset section, some videos are naturally unsuitable to include in our benchmark. In Fig. [8](https://arxiv.org/html/2510.26113v1#A4.F8), both videos are annotated with the query “Kneeling pushing-ups.” However, the egocentric view is insufficient to identify the action due to a limited field of view that only shows the ground, while the exocentric view clearly shows the full body. Due to this kind of noise, we conducted a strict curation and verification during the construction of EgoExo-Con.

### D.2 Model Responses

We further provide model responses on EgoExo-Con across tasks. Fig. [9](https://arxiv.org/html/2510.26113v1#A4.F9) shows model responses for temporal verification from CharadesEgo, and Fig. [10](https://arxiv.org/html/2510.26113v1#A4.F10) illustrates grounding predictions from LEMMA and EgoExo-4D. Note that we do not utilize misaligned queries for temporal grounding.

Figure: Figure 9: Examples of test clips and model responses for temporal verification.. Each row is from CharadesEgo and shows an exocentric–egocentric pair, a Refined Query (positive) and a Misaligned Query (negative), and per-view answers from each model. We mark consistency (✓) and inconsistency (✗) cases.
Refer to caption: x17.png

Figure: Figure 10: Examples of test clips and model responses for temporal grounding. The videos in the top two rows are from LEMMA, and the bottom row shows EgoExo4D videos. Each row presents an exocentric–egocentric pair with a Refined Query; models output per-view time spans (with IoU to ground truth when available), and we indicate cross-view consistency (✓) vs. inconsistency (✗).
Refer to caption: x18.png

Figure: Figure 11: Prompt for refined and misaligned queries.
Refer to caption: x19.png

Figure: Figure 12: Prompt for the reasoning reward function in View-GRPO.
Refer to caption: x20.png

Figure: Figure 13: Prompt for generating temporal reasoning for temporal verification.
Refer to caption: x21.png

Figure: Figure 14: Prompt for generating temporal reasoning for temporal grounding.
Refer to caption: x22.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - There are substantial performance gaps between closed-source models and open-source models, as shown in Table 1. What makes closed-source models stronger? This is just a question to know the authors' opinion.
- Please see weaknesses
soundness: 3
strengths: **[S1] Writing and motivation**
- The paper is well-written and easy to follow. The motivation is clearly presented.

**[S2] Experiments**
- The paper covers extensive range of experiments, including evaluation both open- and closed-source models, reporting detailed per-subset results (CharadesEgo, LEMMA, EgoExo-4D), and fine-tuning analyses.
summary: This paper introduces EgoExo-Con, a benchmark to evaluate the capability of VideoLLMs in understanding temporal consistency across ego- and exocentric viewpoints of the same events. The benchmark focuses on two tasks, including temporal verification (binary QA) and temporal grounding. The authors found that the existing VideoLLMs achieve good single-view performance while showing poor cross-view consistency. To this end, they propose View-GRPO, an RL-based fine-tuning framework, extending Group Relative Policy Optimization with an additional reasoning reward and a curated dataset to enhance cross-view temporal reasoning. Empirical results on EgoExo-Con show that View-GRPO improves consistency over standard SFT and GRPO baselines.
weaknesses: **[W1] Dataset and task clarification**
- EgoExo-Con combines pre-existing datasets, so its domain diversity still depends on those sources. The paper claims “comprehensive” coverage, but 491 pairs is small compared to modern multimodal benchmarks.
- Evaluating temporal consistency across viewpoints is critical in view-invariant video understanding. Traditionally, cross-view temporal consistency has been evaluated through cross-view phase progression or Kendall's $\tau$. Additionally, cross-view frame or clip retrieval may also measure the capability of temporal consistency of the models. Compared to previous evaluation tasks, temporal verification may provide a limited insight, as it measures whether the model identifies an event as the same or not. The necessity of temporal verification should be clearly presented in the paper.
- In sum, the contribution of the introduced benchmark may not be significant.

**[W2] View-GRPO**
- View-GRPO itself seems not inherently specific to cross-view reasoning. In other words, rewards corresponding to each view can be optimized separately, and co-optimization across viewpoints is not guaranteed.

**Minor issues**
- Repeated entries in reference (e.g., Feng et al., 2025a/b and Grauman et al., 2024a/b)
- Typo: L77 their sing-view
- Figure 6 should be moved before Figure 7 (or swap the order of figures)

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Can you somehow apply your View-GRPO approach with existing model on tasks like Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment? It is also tackling the similar problem.
2. How many video frames do you take for training? The training time seems too long --"8 xA100 GPUs and requires over 1 day for the 3B model and 2 days for the 7B model".
soundness: 2
strengths: 1. The paper is clearly written. 
2. The proposed EgoExo-Con eval set can be useful for this area of research.
3. The proposed reinforced approach enhanced view-invariant comprehension in video-LLMs.
summary: The paper introduces evaluation benchmark EgoExo-Con to evaluate cross-view video temporal understanding. The authors propose View-GRPO and construct View30K, to explicitly enhance temporal reasoning.
weaknesses: 1. The proposed evaluation set EgoExo-Con only contains 491 items, which is pretty small and is hard to say weather it is simply finding a hard set for the current Video LLMs.
2. The close-sourced and human performance are reported on a randomly sampled subset, which may be sensitive to sample selection and cannot be fairly compared with open-sourced models and proposed model.
3. In Figure 1 (b), I don't think it is appropriate to expect model understanding "put a knife" from the provided exo video. Because the object of interest is too small. Even for human, it is hard to identify that action. The top example in Figure 3 also makes me confused about how could human even be able to understand the person is opening a cabinet door. 
4. The evaluation is pretty limited, only on the proposed set. And compared Table 1 and Table 2, the proposed method is not better than previous models, e.g. TimeChat-VT.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Audio is view-invariant. Have you considered including audio in the mix and using Qwen-Omni like models?

2. Is there a way to automatically quantify if a given query for a video is exo-friendly, ego-friendly or both-friendly? For example, "a person laughing" is likely only exo-friendly and "a person holding something tight" is ego-friendly (clearly visible in ego view but perhaps not visible in exo view).

3. Another type of consistency could be to pass both ego and exo views together and ask if they depict the same action or ask to match coresspondences. Have you considered something like that?
soundness: 3
strengths: 1. **Useful problem and benchmark.** The proposed problem on cross-view consistency is interesting and has several practical applications like learning from human demonstrations. The ExoEgo-Con benchmark, although not as big in size, is thoroughly constructed and is useful in measuring this consistency.

2. **Strong baseline.** The proposed View-GRPO is simple, well-formulated and shows strong performance on the proposed benchmark.

3. **Presentation.** The paper is very well written and was easy to understand.
summary: This paper studies the cross-view consistency in predictions of video models.

* First, the authors propose a benchmark (EgoExo-Con) of 491 synchronised videos and 3,178 temporal-bounded queries. While video models perform well in individual views, their predictions are not consistent across views.
* It is shown that naive SFT on a mix of ego-exo video data does not improve this consistency likely due to conflicting priors.
* Finally, the authors propose View-GRPO, an RL based approch (along with a training dataset View30K) to enhance temporal reasoning while encouraging view-invariant video comprehension. This outperforms SFT and standard GRPO. View-GRPO includes three reward signals:
    1. Format reward (for answer structuring)
    2. Accuracy: irrespective of view, the final answer consistency is encouraged by this reward
    3. Reasoning reward: the model is encouraged to output reasoning traces similar to those of GPT5 - this balances cross-view temporal reasoning as well as certain view-specific details.
weaknesses: 1. Instead of enforcing cross-view consistency only through the final answer, the View-GRPO method relies on matching reasoning traces for cross-view consistency. This means that the consistency is enforced via the language space and not directly via visual correspondence. In some narrow cases this can be a limitation for examples where it is hard to describe certain visual elements in language.

2. There is still a lot of room for improvement in terms of consistency scores (Tab 3). While this highlights strength of the benchmark, it could strengthen the paper if some discussion is included on what the authors project as potential ways to improve performance (more ego data in pre-training? entirely new algorithms/architectures?).

3. (Minor) There is some overlap with prior work [1] in terms of problem formulation. However, it is not published and can be considered concurrent. Nevertheless, it may be nice to include a deeper discussion on the differences.

4. Some discussion on whether the proposed ViewGRPO affects performance on other standard video benchmarks is desirable. I am not sure if it is LoRA or full finetuning. In case of latter, such evaluation is much needed.

[1] EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs
Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, Jiangmiao Pang

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: This paper presents a valuable and timely contribution by defining and benchmarking the problem of view-invariant temporal understanding. The EgoExo-Con dataset and the accompanying analysis are strong points that will benefit the community.
The proposed View-GRPO method, while seems promising, has weaknesses in presentation.
soundness: 3
strengths: - EgoExo-Con benchmark is a significant contribution. The authors have been sourcing data from diverse datasets and performing careful filtering and human-backed refinement of queries to ensure they are unambiguous and visible from both perspectives.
- The paper provides a comprehensive evaluation of a wide range of Video-LLMs. The key finding that cross-view consistency scores are "barely over half their single-view performance" and that naive multi-view training fails to improve consistency, is a crucial insight that highlights a fundamental weakness in current architectures and training paradigms.
- The proposed View-GRPO method offers a promising direction for improving consistency.
summary: This paper investigates the ability of Video-LLMs to maintain consistent temporal understanding of events captured from different viewpoints. The authors introduce EgoExo-Con, a new benchmark consisting of synchronized egocentric and exocentric video pairs with refined natural language queries for temporal verification and grounding tasks. Through extensive experiments, they reveal that current SOTA models struggle significantly with cross-view consistency, often achieving scores that are much lower than their single-view performance. The paper also demonstrates that naively fine-tuning models on a mix of both viewpoints is insufficient and can even degrade performance. To address these shortcomings, the authors propose View-GRPO, a reinforcement learning framework that encourages models to develop view-specific reasoning chains while aligning their final temporal predictions, showing improved consistency over standard fine-tuning methods.
weaknesses: - The effectiveness of the proposed View-GRPO method is demonstrated only on the Qwen2.5-VL model family. While the results are positive, application to wider range of models would be necessary to make a stronger claim about the generalizability and robustness of the approach.
- Authors showed that the choice of judge model impacts performance. However, the issue of reliability and bias in judge is left for future work. Given its central role in the method's success, a more thorough analysis or ablation study (e.g., removing the reasoning reward) is wanted.
- The View-GRPO is an incremental approach built on top of GRPO. The novelty lies in its application to the cross-view consistency problem and the design of the reward function.


Decision: Reject