Ego-centric Predictive Model Conditioned on Hand Trajectories

## Abstract

Abstract In egocentric scenarios, anticipating both the next action and its visual outcome is essential for understanding human–object interactions and for enabling robotic planning. However, existing paradigms fall short of jointly modeling these aspects. Vision-Language-Action (VLA) models focus on action prediction but lack explicit modeling of how actions influence the visual scene, while video prediction models generate future frames without conditioning on specific actions—often resulting in implausible or contextually inconsistent outcomes. To bridge this gap, we propose a unified two-stage predictive framework that jointly models action and visual future in egocentric scenarios, conditioned on hand trajectories. In the first stage, we perform consecutive state modeling to process heterogeneous inputs—visual observations, language, and action history—and explicitly predict future hand trajectories. In the second stage, we introduce causal cross-attention to fuse multi-modal cues, leveraging inferred action signals to guide an image-based Latent Diffusion Model (LDM) for frame-by-frame future video generation. Our approach is the first unified model designed to handle both egocentric human activity understanding and robotic manipulation tasks, providing explicit predictions of both upcoming actions and their visual consequences. Extensive experiments on Ego4D, BridgeData, and RLBench demonstrate that our method outperforms state-of-the-art baselines in both action prediction and future video synthesis. Code is available at https://github.com/showlab/Ego-PM .

## Introduction

World models (Ha and Schmidhuber 2018; Wang et al. 2024b; Liu et al. 2024; Lai et al. 2025) should be capable of simulating how the physical world changes in response to complex environments through integrated processing of language, vision (Chen et al. 2023; Li et al. 2024), and audio (Ghosal et al. 2023; Yang et al. 2023). Humans naturally possess this ability: we can intuitively imagine how a specific physical action (by ourselves or others) will affect objects and the environment around us. In egocentric settings, particularly in human–object interaction tasks (Touvron et al. 2023; Brown et al. 2020; Chowdhery et al. 2022; Zhang et al. 2023), the human hands (or robot arms) are the primary instruments of action. It is therefore crucial to explicitly model the hand’s actions as a conditioning factor when predicting future states of the world.

Figure: Figure 1: Examples of egocentric predictive modeling. Given a sequence of past video frames and text prompt, our model first predicts the upcoming action and then generates future video frames conditioned on that action. Shown are results from (a) Ego4D, (b) and (c) BridgeData, and (d) the Meta-World simulation environment.
Refer to caption: x1.png

Despite the importance of jointly modeling actions and future vision, most existing approaches have significant limitations. On one hand, vision-language-action (VLA) (Kim et al. 2024) models only focus on predicting the next action given visual (and textual) inputs without visual prediction, which can hinder an agent’s understanding of the consequences of its actions. On the other hand, video prediction and generation methods (Lai et al. 2025; Hu et al. 2023) attempt to foresee future frames but often do so without explicit action conditioning.

To bridge this gap, we propose a unified two-stage Egocentric Predictive Model (Ego-PM) conditioned on hand trajectories, which unifies action prediction with future frame generation. In the first stage, we explicitly predict future hand trajectories via consecutive state modeling of the heterogeneous inputs (visual observations, language, and action history). In the second stage, we introduce causal cross-attention to fuse multi-modal cues, leveraging inferred action signals to guide frame-by-frame future video generation. Importantly, our method does not require any additional annotations at inference time: the model autonomously predicts the necessary hand trajectory as an intermediate step and then uses it to generate the future video frames. This design makes our approach both powerful and practical for real-world applications.

Our key contributions can be summarized as follows:

- •
We introduce the first unified model that explicitly predicts both upcoming actions and future visual frames, addressing tasks that prior approaches handled separately.
- •
We propose a novel Consecutive State Modeling (CoSMo) strategy to explicitly predict hand trajectories, and introduce a new causal cross-attention scheme to enhance visual synthesis conditions through the intermediate hand action representation. These strategies significantly improve the quality of hand trajectory predictions by leveraging historical information, which in turn leads to more accurate and consistent future frame generation.
- •
This is the first unified predictive model applicable to both human egocentric and robotic manipulation scenarios, as demonstrated by its strong performance on Ego4D, BridgeData, and RLBench.

## Related Work

#### Text-Image-Based Video Generation.

Text-image-based video generation has emerged as a promising field that bridges natural language processing (NLP) and computer vision, enabling models to generate video content from textual and visual descriptions. These methods typically process text and images separately using a text tokenizer and vision encoder, after which the tokens are integrated through a unified autoregressive model. A vision decoder then generates images based on the multimodal inputs.

Various approaches employ generative adversarial networks (GANs) or variational autoencoders (VAEs) for text-to-image synthesis (Souček et al. 2024), while others leverage diffusion models (Hu et al. 2023). For instance, GenHowTo (Souček et al. 2024) employs separate models for action state prediction and final state prediction, which generates frames in stages but results in redundant and computationally expensive training. Current text-image-based video generation methods face significant limitations in ego-centric contexts, as they often lack the first-person perspective and fail to incorporate action-conditioned predictions necessary for interactive, immersive experiences.

#### Robotics & VLA Models.

LLARVA (Niu et al. 2024) introduces a vision-action instruction tuning approach that uses structured natural language prompts to unify diverse robotic tasks, along with an auxiliary 2D “visual trace” prediction to align vision and action spaces. AdaWorld (Gao et al. 2025) is a world-model pretraining approach that learns to incorporate actions without explicit labels by extracting latent action representations from video. This&That (Wang et al. 2024a) couples deictic language commands (“this/that”) with pointing gestures for video generation in robotic planning. TORA (Zhang et al. 2025) adds controllable motion trajectories to a diffusion transformer for text-to-video generation. Notably, our model uses only simple hand position annotations during training as explicit action cues, and it requires no additional inputs at inference—predicting the hand trajectory on its own. Unlike previous trajectory-conditioned video generation approaches that often assume a static background, our model captures dynamic scene context and moving foreground objects, leading to more realistic future video predictions.

#### Ego-Centric Vision-Language Models.

Recent efforts in ego-centric vision have focused on understanding human actions and attention (Huang et al. 2020; Sudhakaran, Escalera, and Lanz 2019; Kazakos et al. 2019; Lai et al. 2023), modeling hand-object interactions (Ragusa, Farinella, and Furnari 2023; Liu et al. 2022; Goyal et al. 2022), and estimating human poses (Tome et al. 2020; Wang et al. 2023; Li, Liu, and Wu 2023) from a first-person perspective. In ego-centric video-language modeling, Lin et al. (Lin et al. 2022) introduced EgoVLP, a large-scale pretraining model for video-language tasks. For visual generation, Jia et al. (Jia, Liu, and Rehg 2022) leveraged GANs (Goodfellow et al. 2020) to generate future head motion in a hand forecasting task, while Zhang et al. (Zhang et al. 2017) used GANs to anticipate future gaze direction. Ye et al. (Ye et al. 2023) developed an affordance diffusion model to generate possible hand-object interactions from an object image in an egocentric view. More recently, LEGO (Lai et al. 2025) synthesized egocentric action frames from a given image and text prompt, but its lack of an action input prevents modeling human–environment interactions, and it struggles to maintain temporal coherence across consecutive frames, especially during dynamic hand movements. These limitations underscore the need for an egocentric world model that integrates multimodal inputs (vision, language, and action) to produce coherent, contextually relevant predictions from a first-person perspective.

## Methodology

### Problem Definition

Given a $n$-timestep egocentric video sequence $\mathcal{V}_{t:0\rightarrow n}=\{v_{0},v_{1},\dots,v_{n}\}$, a text prompt $\mathcal{T}$, and an action sequence $\mathcal{A}_{t:0\rightarrow n}$, our Ego-centric Predictive Model (Ego-PM) captures the multimodal context and predicts both future frames and upcoming actions. The goal of Ego-PM can be formulated as:

$$ $\displaystyle\max p\big{(}\{\mathcal{V}_{t:>n},\,\mathcal{A}_{t:>n},\,\mathcal{T}_{\mathcal{A}}\}\mid\{\mathcal{V}_{t:0\rightarrow n},\,\mathcal{A}_{t:0\rightarrow n},\,\mathcal{T}\}\big{)},$ (1) $$

where $\mathcal{T}_{\mathcal{A}}$ denotes a textual description of the predicted action. Ego-PM thereby predicts future frames $\mathcal{V}_{t:>n}$, future actions $\mathcal{A}_{t:>n}$, and an action description $\mathcal{T}_{\mathcal{A}}$ conditioned on the observed inputs.

Figure: Figure 2: Training framework of Ego-Centric Predictive Model (Ego-PM). In Stage I, a pre-trained LLM is fine-tuned via LoRA (Hu et al. 2022) to process multi-modal inputs (images, text, and actions). Using a Consecutive State Modeling strategy, the autoregressive model explicitly explores the hand trajectory representation by taking two adjacent states as input. In Stage II, a block-wise causal cross-attention mechanism enhances the visual and textual embeddings using the inferred action embedding from Stage I. These combined conditions are then projected into the latent space and used to train a latent diffusion model for predicting future egocentric frames.
Refer to caption: x2.png

### Overall Architecture

As previously noted, VLA models (Niu et al. 2024; Kim et al. 2024) handle action prediction without visual forecasting, whereas video generation methods (Lai et al. 2025; Hu et al. 2023) ignore action conditioning and may produce future frames inconsistent with the actions taken.

To bridge this gap, we propose a unified Egocentric Predictive Model (Ego-PM) conditioned on hand trajectories, which unifies action prediction with future frame generation. Given the challenges of training convergence and memory limitations, we adopt a two-stage training strategy similar to prior work (Lai et al. 2025; Liu et al. 2024). In the first stage, we explicitly predict future hand trajectories via consecutive state modeling of the heterogeneous inputs (i.e., visual observations, language, and action history). In the second stage, we introduce causal cross-attention to fuse multi-modal cues, leveraging inferred action signals to guide frame-by-frame future video generation. The overall architecture of Ego-PM is illustrated in Fig. [2](https://arxiv.org/html/2508.19852v2#Sx3.F2), and the following sections detail each component as well as our CoSMo strategy for consistent state prediction.

### Stage I: Explicit Action Modeling

Stage I explicitly models the upcoming action (i.e., the hand trajectory) using our consecutive state modeling strategy. Specifically, we first align embeddings of the image, text, and action inputs from previous states, and then sequentially predict an enhanced action description as well as an explicit action representation for the next state.

#### Vision Encoder.

We use a pre-trained CLIP visual encoder $\Phi_{V}$ to extract visual features from each frame. To align image features with the text embedding space, we apply a linear projection $\mathcal{F}$, yielding aligned visual tokens $\mathcal{F}(\Phi_{V}(\mathcal{V}))$ for each frame.

#### Action Intent Encoder&Decoder.

To equip the pretrained LLM with action-awareness, we design a lightweight action encoder that integrates action inputs into the token embedding space. We introduce a special token $\langle\text{ACT}\rangle$ to indicate the presence of an action, allowing the action embedding $\Phi_{A}(\mathcal{A})$ to be concatenated with vision and text tokens. During inference, the model’s action tokens are decoded via a small MLP-based decoder (matching the encoder) to reconstruct the original action signals. The action intent encoder and decoder are both 3-layer MLPs.

#### Autoregressive Model.

We build upon LLaVA (Chen et al. 2023) as the initialized autoregressive model, following a setup similar to LEGO (Lai et al. 2025). At each time step $t$, both text and action tokens are incorporated to enhance temporal understanding. The model is trained on interleaved sequences of vision, text, and action tokens, enabling autoregressive prediction with improved temporal coherence.

#### Consecutive State Modeling.

Effective temporal modeling and action–effect interaction are crucial for egocentric predictive modeling. We observe that prior work (Lai et al. 2025) relied solely on the current state to predict the next state, which is suboptimal for sequential predictions. To improve temporal coherence, we introduce a Consecutive State Modeling (CoSMo) strategy. CoSMo leverages both the current and previous states to predict the future state, providing a more robust representation of temporal dynamics. By conditioning on two consecutive time steps (states $t$ and $t-1$) instead of one, the model learns richer temporal dependencies that lead to more coherent and accurate predictions.

### Stage II: Action-Enhanced Frame Prediction

Existing text-based image generation methods (Hu et al. 2023; Souček et al. 2024) and world models (Liu et al. 2024; Wang et al. 2024b) utilize visual and textual information to predict future frames but lack action guidance. However, we posit that hand trajectory information is essential for human–object interactions. Therefore, explicitly integrating hand trajectory cues into the generation process should yield more detailed and accurate predictions. In this stage, we first enhance the multi-modal conditions (vision and text embeddings) with the inferred hand trajectory embedding. We then fuse these conditions to guide a latent diffusion model, which generates future video frames sequentially.

#### Action-Enhanced Multi-Modality Condition Fusion.

To form the vision condition for frame generation, we concatenate the current frame ($\mathcal{V}_{t=n}$) and previous frames ($\mathcal{V}_{t:<n}$), apply a linear projection $\mathcal{F}$, and then use self-attention and Casual Cross-Attention (CCA) to capture visual context:

$$ $\mathcal{C}_{\mathcal{V}}=\text{CCA}\Big{(}\text{Self-Attn}\big{(}\mathcal{F}(\Phi_{V}(\mathcal{V}_{0\rightarrow n}))\big{)}\Big{)}.$ (2) $$

In practice, as illustrated in Fig. [2](https://arxiv.org/html/2508.19852v2#Sx3.F2), we treat the action embedding from Stage I as the Query and perform block-wise CCA over the visual embeddings (as Key and Value). CCA constrains each query to attend only to key-value pairs from the same or later time steps, providing the historical information for future frame prediction.

For text conditioning, we use CLIP’s text encoder $\psi$ to obtain a feature embedding for the prompt $\psi(\mathcal{T})$. We concatenate this with the text embedding generated by the LLM in Stage I (denoted $\mathcal{T}_{\mathcal{A}}$) and apply self-attention followed by causal cross-attention similar to the vision branch:

$$ $\mathcal{C}_{\mathcal{T}}=\text{CCA}\Big{(}\text{Self-Attn}\big{(}\mathcal{F}([\psi(\mathcal{T}),\,\mathcal{T}_{\mathcal{A}}])\big{)}\Big{)}.$ (3) $$

To build the action condition, we project the Stage I action embedding into the diffusion model’s latent space using a linear layer $\mathcal{F}$:

$$ $\mathcal{C}_{\mathcal{A}}=\mathcal{F}\big{(}\mathcal{A}_{t:1\rightarrow n}\big{)}.$ (4) $$

Finally, we combine all conditioned inputs:
$\mathcal{C}=[\,\mathcal{C}_{\mathcal{V}},\,\mathcal{C}_{\mathcal{T}},\,\mathcal{C}_{\mathcal{A}}\,]$,
which is fed into the diffusion model’s UNet via cross-attention layers to guide iterative future frame predictions.

#### Frame Prediction.

We employ a latent diffusion model (LDM) (Rombach et al. 2022) to generate frames in the latent space of a pre-trained variational autoencoder (with encoder $\mathcal{E}$ and decoder $\mathcal{D}$). We follow the image-based LDM approach, generating future frames one at a time. Starting from the last observed frame $v_{n}$, we encode it as $z_{n}=\mathcal{E}(v_{n})$ and add noise over diffusion timesteps. The LDM network $\epsilon_{\theta}$ is trained to predict the added noise $\epsilon$ in $z_{t}$ at each diffusion step $t$, conditioned on our multimodal context $\mathcal{C}$ (i.e., $\mathcal{C}_{\mathcal{V}}$, $\mathcal{C}_{\mathcal{T}}$, $\mathcal{C}_{\mathcal{A}}$).

**Table 1: Ego4D single-step prediction results ($t=2$ predicted from $t=1$). We report frame metrics (EgoVLP, EgoVLP^+, CLIP, FID, PSNR, LPIPS) and action metric (Hand IoU). Lower values are better for metrics marked with $\downarrow$. *For a fair comparison, we finetune LWM with the same ego-centric dataset to minimize the domain gap.**
| Baseline | Frame Prediction | Action Prediction |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| EgoVLP | EgoVLP^+ | CLIP | FID$\downarrow$ | PSNR | LPIPS$\downarrow$ | Hand IoU |  |
| LWM^∗ | 62.31 | 76.54 | 77.89 | 24.95 | 11.73 | 38.12 | - |
| LEGO | 65.65 | 80.44 | 80.61 | 23.83 | 12.29 | 36.43 | - |
| Ours | 69.11 | 83.98 | 83.24 | 21.31 | 16.12 | 33.23 | 44.25 |

**Table 2: Ego4D consecutive prediction results ($t=3$ predicted using $t=1$ and the intermediate prediction for $t=2$). The same metrics are reported as in Table [1](https://arxiv.org/html/2508.19852v2#Sx3.T1). Bold numbers indicate the best performance, and $\downarrow$ indicates lower is better.**
| Baseline | Frame Generation | Action Prediction |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| EgoVLP | EgoVLP^+ | CLIP | FID$\downarrow$ | PSNR | LPIPS$\downarrow$ | Hand IoU |  |
| LWM^∗ | 61.13 | 75.43 | 76.47 | 25.23 | 11.23 | 38.94 | - |
| LEGO | 62.34 | 77.31 | 77.45 | 25.23 | 11.24 | 37.76 | - |
| Ours | 65.87 | 81.13 | 81.24 | 22.59 | 13.89 | 34.32 | 40.71 |

### Training Objectives

#### Stage I.

Previous approaches primarily guided predictions with vision and text, overlooking explicit supervision in the action space. To address this, we introduce an objective for consecutive state modeling that includes action supervision, improving the quality of future predictions:

$$ $\displaystyle\mathcal{L}_{\text{CoSMo}}=\mathcal{L}_{\text{lang,t=1}}+\lambda_{1}\mathcal{L}_{\text{act,t=1}}+\mathcal{L}_{\text{lang,t=2}}+\lambda_{1}\mathcal{L}_{\text{act,t=2}},$ (5) $$

where $\mathcal{L}_{\text{lang}}$ is the standard language modeling loss used in LLaMa (Touvron et al. 2023), and $\mathcal{L}_{\text{act}}$ represents the action loss with a hyperparameter $\lambda_{1}$.

For human activity dataset (Ego4D (Grauman et al. 2022)), we supervise predicted hand coordinates using a combination of L1 and GIoU losses:

$$ $\displaystyle\mathcal{L}_{\text{act}}(\hat{\text{b}},\text{b}_{\text{gt}})=\mathcal{L}_{\text{L1}}(\hat{\text{b}},\text{b}_{\text{gt}})+\lambda_{2}\mathcal{L}_{\text{GIoU}}(\hat{\text{b}},\text{b}_{\text{gt}}),$ (6) $$

where $\hat{b}$ and $b_{\text{gt}}$ denote predicted and ground-truth hand bounding boxes, respectively, and $\lambda_{2}$ balances the terms.
For robot data (BridgeData (Walke et al. 2023), RLBench (James et al. 2020)), we similarly supervise robot arm keypoints (7D pose) using L1 loss.

#### Stage II.

The diffusion process in Stage II is guided by a diffusion-specific loss function, minimizing the following latent diffusion objective:

$$ $\mathcal{L}_{\text{LDM}}=\mathbb{E}_{\mathcal{E}(\mathcal{V}),\mathcal{C},\epsilon\sim\mathcal{N}(0,1),t}\Big{[}\|\epsilon-\epsilon_{\theta}(z_{t},t,\mathcal{C})\|_{2}^{2}\Big{]},$ (7) $$

where $\epsilon$ represents the added noise, and $z_{t}$ is the noisy latent at timestep $t$. This loss encourages the model to denoise $z_{t}$ based on multimodal conditioning $\mathcal{C}$, yielding temporally coherent and contextually accurate predictions.

## Experiments

**Table 3: Quantitative comparison of different methods on the BridgeData V2 dataset. Frame prediction metrics include FID, FVD, PSNR, SSIM, and LPIPS, where lower FID/FVD/LPIPS and higher PSNR/SSIM indicate better performance. Generation-based models do not predict action trajectories (no action results in top block). VLA-based models do not generate frames (no visual metrics). Our model supports both future frame generation and action prediction.**
| Methods | Type | Frame Prediction | Action Prediction |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FID↓ | FVD↓ | PSNR↑ | SSIM↑ | LPIPS↓ | Success Rate↑ |  |  |
| SVD | Generation | 29.49 | 657.49 | 12.47 | 0.334 | 0.391 | - |
| StreamingT2V | Generation | 42.57 | 780.81 | 11.35 | 0.324 | 0.504 | - |
| DragAnything | Generation | 34.38 | 764.58 | 12.76 | 0.364 | 0.466 | - |
| This&That | Generation | 17.28 | 84.58 | 21.71 | 0.787 | 0.112 | - |
| OCTO | VLA | - | - | - | - | - | 20.0 |
| OpenVLA | VLA | - | - | - | - | - | 70.6 |
| Ours | Omni | 16.36 | 83.43 | 21.56 | 0.791 | 0.113 | 73.7 |

**Table 4: Success rates (%) on the RLBench Multi-Task benchmark. Each method is evaluated on 25 episodes per task, with each episode scored as 100 (success) or 0 (failure). We report the average success rate per task for each method.**
| Method | Additional annotation | open drawer | meat off grill | turn tap | put money | push buttons | sweep dustpan | slide block | close jar | screw bulb | Avg. Rate |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Image-BC (ViT) | 2D trajectories | 0 | 0 | 16 | 0 | 0 | 0 | 0 | 0 | 16 | 3.56 |
| LLARVA | 2D trajectories | 60 | 80 | 56 | 44 | 56 | 84 | 100 | 28 | 8 | 57.33 |
| RVT | 3D point cloud | 71.2 | 88 | 93.6 | 40 | 100 | 72 | 81.6 | 52 | 32 | 70.71 |
| SAM-E | 3D point cloud | 82.4 | 95 | 100 | 45 | 100 | 100 | 95 | 82 | 30 | 81.49 |
| Ours | No need | 64 | 70 | 60 | 56 | 56 | 84 | 96 | 32 | 12 | 58.89 |

We evaluate Ego-PM on both human egocentric (Ego4D) and robotic (BridgeData V2, RLBench) scenarios. We first describe our datasets and metrics, then present quantitative and qualitative results comparing Ego-PM to state-of-the-art baselines. We also provide ablation studies to examine the contributions of CoSMo and the action conditioning.

### Dataset

#### Human Activity (Ego4D).

We use the Ego4D egocentric video dataset (Grauman et al. 2022) focusing on the Pre-Conditioning (PRE-15) and Point-of-No-Return (PNR) moments. We define the start state $t=0$ at PRE-15, the intermediate state $t=1$ at PNR, and the final state $t=2$ as the frame after PNR (same time interval as between PRE-15 and PNR). The text narration serves as the prompt, and hand position tracks are used as action inputs during training.

Robot Manipulation (BridgeData & RLBench). We train and evaluate on BridgeData V2 (Walke et al. 2023), a human-to-robot demonstration dataset, and additionally test on RLBench (James et al. 2020), a simulated robotic manipulation benchmark. For BridgeData, we use the provided multi-step human demonstration videos and textual instructions. For RLBench, we evaluate on nine tasks (e.g., “open drawer”, “push buttons”) in a multi-task setting.

### Metrics

For future frame prediction, we report both perceptual and pixel-level metrics: EgoVLP and EgoVLP^+ alignment scores (Lin et al. 2022), CLIP image-text similarity, Fréchet Inception Distance (FID) (Heusel et al. 2017), Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al. 2018). Lower FID and LPIPS and higher SSIM and PSNR indicate better visual fidelity.

For action prediction, we use hand mask Intersection-over-Union (IoU) for Ego4D ( predicted vs. ground-truth hand regions) and Success Rate for BridgeData/RLBench (the percentage of successful task completions). Higher values indicate better action prediction accuracy.

**Table 5: Ablation study on Ego4D evaluating key components of Ego-PM: Consecutive State Modeling (CoSMo), action input format, and Causal Cross-Attention (CCA). Frame metrics include EgoVLP, EgoVLP^+, CLIP, FID, PSNR, and LPIPS (higher is better for EgoVLP/EgoVLP^+/CLIP/PSNR, lower is better for FID/LPIPS). Hand IoU measures action prediction (higher is better). Configs 1–4 use one-step prediction ($t=2$), and 5–8 use consecutive two-step prediction ($t=3$).**
| Row | State | Component | Frame Prediction | Action Prediction |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CoSMo | Action input | With CCA | EgoVLP | EgoVLP^+ | CLIP | FID$\downarrow$ | PSNR | LPIPS$\downarrow$ | Hand IoU |  |  |
| 1 | $t=2$ | ✗ | Plain text | ✗ | 65.76 | 81.03 | 80.01 | 23.71 | 12.36 | 36.89 | 39.71 |
| 2 | $t=2$ | ✗ | Enc&Dec | ✗ | 67.12 | 82.31 | 81.54 | 22.88 | 13.01 | 35.89 | 40.54 |
| 3 | $t=2$ | ✓ | Enc&Dec | ✗ | 68.32 | 83.44 | 82.53 | 22.12 | 14.10 | 34.31 | 41.43 |
| 4 | $t=2$ | ✓ | Enc&Dec | ✓ | 69.11 | 83.98 | 83.24 | 21.31 | 16.12 | 33.23 | 44.25 |
| 5 | $t=3$ | ✗ | Plain text | ✗ | 62.71 | 77.56 | 77.58 | 25.45 | 11.35 | 37.65 | 36.21 |
| 6 | $t=3$ | ✗ | Enc&Dec | ✗ | 63.15 | 78.53 | 79.13 | 24.34 | 12.56 | 36.56 | 37.36 |
| 7 | $t=3$ | ✓ | Enc&Dec | ✗ | 64.78 | 80.21 | 80.31 | 23.89 | 12.87 | 35.21 | 38.78 |
| 8 | $t=3$ | ✓ | Enc&Dec | ✓ | 65.87 | 81.13 | 81.24 | 22.59 | 13.89 | 34.32 | 40.71 |

### Consecutive State Prediction

We choose LWM (Liu et al. 2024) and LEGO (Lai et al. 2025) as benchmarks for both quantitative and qualitative comparisons.
Table [1](https://arxiv.org/html/2508.19852v2#Sx3.T1) and Table [2](https://arxiv.org/html/2508.19852v2#Sx3.T2) show Ego4D results for single-step and consecutive predictions, respectively.
Our Ego-PM significantly outperforms LWM and LEGO across all metrics. Notably, LEGO (the strongest baseline) lacks the ability to predict actions (Hand IoU not applicable), whereas Ego-PM predicts hand trajectories with high accuracy (44.25 IoU) while also achieving superior frame generation quality (e.g., FID 21.31 vs 23.83 for LEGO in single-step).

For the harder consecutive prediction (Table [2](https://arxiv.org/html/2508.19852v2#Sx3.T2)), all methods see performance drops, but Ego-PM maintains a clear advantage in every metric, highlighting its robustness in compounding predictions.

### Robotic Manipulation Results

Our model is not only effective on human-centric datasets such as Ego4D, but also generalizes well to robot manipulation datasets, including BridgeData V2 (Walke et al. 2023) and RLBench (James et al. 2020). To evaluate its versatility, we compare our model with two representative VLA-based models (OpenVLA (Kim et al. 2024) and OCTO (Team et al. 2024)) and four state-of-the-art video generation models (SVD (Blattmann et al. 2023), StreamingT2V (Henschel et al. 2025), DragAnything (Wu et al. 2024), and This&That (Wang et al. 2024a)).

On BridgeData (Table [3](https://arxiv.org/html/2508.19852v2#Sx4.T3)), our model is the only “omni-capable” approach that excels in both frame generation and action prediction. VLA baselines (OpenVLA, OCTO) predict actions well but cannot generate frames; generation models (SVD, StreamingT2V, etc.) produce frames but lack action outputs.
Ego-PM achieves the best of both: highest action success rate (73.7% vs 70.6% OpenVLA) and best frame metrics (FID 16.36 vs 17.28 next best).

Table [4](https://arxiv.org/html/2508.19852v2#Sx4.T4) reports RLBench multi-task success rates. Although some methods like SAM-E and RVT have higher overall success by leveraging heavy 3D inputs, our model (which uses no additional 3D sensors or ground-truth trajectories) still outperforms 2D-trajectory baselines and is competitive with methods relying on richer inputs. For tasks such as “sweep dustpan” and “slide block”, Ego-PM matches or exceeds the performance of more complex baselines, despite its simpler input requirements. This indicates a favorable trade-off: Ego-PM achieves strong generalization with far less annotation overhead.

### Ablation Studies

To analyze the contributions of our design choices, we perform controlled experiments on Ego4D. Table [5](https://arxiv.org/html/2508.19852v2#Sx4.T5) summarizes the results.

#### Effect of Action Encoding.

In table [5](https://arxiv.org/html/2508.19852v2#Sx4.T5), comparing row 1 vs. 2 (and 5 vs. 6), we see that replacing the naive “action as plain text” input with our dedicated action encoder–decoder yields notable improvements (e.g., Hand IoU from 39.71 to 40.54, FID from 23.71 to 22.88 at $t=2$). This validates that treating the action modality separately (instead of as additional text) is important. The plain-text approach fails to convey distinct action information to the model, whereas our encoded action tokens provide effective guidance.

#### Effect of CoSMo

Row 3 (and 7) introduce our Consecutive State Modeling strategy. Using two consecutive states during prediction (row 3) improves all metrics over row 2 (single-state), confirming that CoSMo enhances temporal coherence. EgoVLP^+ (video-text alignment) and PSNR both rise with CoSMo, and Hand IoU improves as well (from 40.5 to 41.4). Intuitively, looking at both $t=1$ and $t=0$ provides the model with motion context, leading to better anticipation of $t=2$.

Figure: Figure 3: Qualitative comparisons of generated future frames (with predicted actions) for a sample egocentric scenario. LWM (Liu et al. 2024) struggles due to domain gap (exo-centric training leads to unrealistic first-person outputs). LEGO (Lai et al. 2025) produces incomplete hand poses and loses coherence after one step. In contrast, Ego-PM generates accurate hand-object interactions that follow the prompt (e.g., reaching for the correct object), while also maintaining background context across frames and predicting the hand trajectory (green overlay).
Refer to caption: x3.png

#### Effect of Causal Cross-Attention.

Row 4 (and 8) add the causal cross-attention (CCA) fusion. This yields the best performance: e.g., Hand IoU jumps from 41.4 to 44.3 at $t=2$, and FID drops from 22.12 to 21.31. CCA particularly helps maintain quality into the second predicted frame ($t=3$), as seen by improvements from Row 7 to 8 (e.g., EgoVLP^+ 80.2 to 81.1, Hand IoU 38.8 to 40.7). This indicates that aligning the visual and textual tokens with the action query in a causal manner (ensuring proper temporal order) benefits both action and frame prediction.

#### Consecutive Predictions

To test the model’s ability to handle consecutive predictions, we use the predicted state at $t=2$ to predict the next state at $t=3$. The results, presented in rows 4-6 of Table [5](https://arxiv.org/html/2508.19852v2#Sx4.T5), show a noticeable performance drop across all baselines. This indicates that existing world models struggle with consecutive predictions due to compounding errors in earlier state predictions. However, our model consistently outperforms the baselines, showcasing its robustness and effectiveness in handling consecutive predictions.

### Visualization

We present qualitative results in Fig. [3](https://arxiv.org/html/2508.19852v2#Sx4.F3), comparing Ego-PM to baseline models for an Ego4D example. LWM’s output is clearly off (hand shape and context are implausible) due to its exocentric bias. LEGO does better initially but fails to render the hand properly and drifts by the second predicted frame. Ego-PM, however, generates a realistic hand interacting with the correct object (per the text prompt) and preserves the scene context (background remains consistent). Moreover, Ego-PM is the only model that provides an explicit hand trajectory prediction (green overlay), indicating where the hand will move. These visualizations underscore how Ego-PM more holistically understands “what will happen next” in first-person scenes.

### Implementation Details

By default, we use a 2-timestep history as input (two past frames and their actions). The auto-regressive model is initialized from LEGO’s pre-trained weights (or OpenVLA) for Ego4D (or BridgeData). We freeze the CLIP image encoder and fine-tune the projection layer and the LLM for 3 epochs using cross-entropy loss. The LDM is initialized from a Stable Diffusion checkpoint (Rombach et al. 2022). All frames are $256\times 256$ resolution. We set $\lambda_{1}=0.1$ and $\lambda_{2}=0.01$ in our losses based on validation performance.

## Conclusion

In summary, we have presented a novel egocentric predictive model conditioned on hand trajectories, capable of both action prediction and future video generation. By explicitly incorporating hand motion as an intermediate representation, our approach addresses the limitations of prior methods that separated action reasoning from visual forecasting. The proposed model not only learns to anticipate which action will occur in a first-person scenario, but also visualizes the consequences of that action in the form of future frames – all within a unified architecture.

Our contributions include introducing a simple yet powerful conditioning signal (hand position annotations) to guide the learning of action–effect relationships, and a consecutive state modeling strategy that effectively integrates heterogeneous inputs. Crucially, our model operates without requiring any manual action annotations at test time, making it practical for real-world deployment.
Extensive experiments on three distinct datasets – Ego4D (human egocentric videos), BridgeData (human-to-robot scenarios), and RLBench (robotic manipulation tasks) – validate the effectiveness of our model. It consistently outperforms state-of-the-art baselines in both the accuracy of predicted actions and the quality of generated future video sequences. These results underscore the generality of our approach, as the same model can be seamlessly applied to both human and robotic perspectives.
6——
Future work can build upon this foundation by exploring even richer conditioning signals and by extending the model to handle longer-horizon predictions and more complex multi-agent interactions.
Overall, our hand-conditioned egocentric predictive model opens new avenues for intuitive and interactive AI systems that understand and predict the coupling between actions and their effects in the world.

## Appendix I: Related Work

In the main paper, we briefly summarize relevant research on text-based video generation, robotics and vision-language-action (VLA) models, and egocentric vision-language models. In addition to these areas, we also provide a discussion of related world model approaches.

#### World Models.

World models (Ha and Schmidhuber 2018) enable intelligent agents to understand, predict, and make decisions based on interactions with their environments. In computer vision, world models are widely used for video prediction (Lotter, Kreiman, and Cox 2017; Lee, Strobel, and Finn 2018), which is critical for downstream tasks such as autonomous driving (Hu et al. 2023; Shi et al. 2023; Zheng et al. 2025; Tian et al. 2024) and robotics (Liang et al. 2022; Lu et al. 2025; Firoozi et al. 2023). These models generate plausible future frames by conditioning on the agent’s current state, actions, and environmental context, thereby enhancing temporal coherence and action prediction accuracy.

LWM (Liu et al. 2024) tackles long-sequence prediction via masked sequence packing to improve training efficiency. However, it directly concatenates video frames without explicitly modeling consecutive states. GAIA-1 (Hu et al. 2023) predicts future frames conditioned on visual inputs and action sequences, facilitating real-time event anticipation in autonomous driving. Despite their effectiveness, GAIA-1 and related models (Wang et al. 2024b) primarily operate from exocentric (third-person) viewpoints, which limits their capacity to model fine-grained egocentric details such as hand movements. While world models have made considerable progress, they still face challenges in egocentric and interactive tasks where a user-centered, first-person view is essential for immersive applications.

## Appendix II: Technical Details

#### Overview of Ego-PM.

An overview of the Ego-Centric Predictive Model (Ego-PM) is illustrated in Fig. [4](https://arxiv.org/html/2508.19852v2#Sx7.F4), inspired by the architecture of the original World Model (Ha and Schmidhuber 2018). The model consists of three core components:

- •
Multimodal Encoders extract features from visual, textual, and action inputs. Incorporating the action space is critical for accurate prediction.
- •
An Autoregressive Model learns cross-modal correlations to enhance temporal coherence and explicitly predict future hand trajectories.
- •
A Controller uses these representations to predict future frames, generate narrative descriptions, and anticipate upcoming actions.

Figure: Figure 4: Overview of Ego-PM architecture, comprising multimodal encoders, an autoregressive model, and controllers. The model predicts consecutive future states, including both frames and actions. Darker colors denote input states, and lighter colors indicate predicted states.
Refer to caption: x4.png

#### Training Details.

We use Bi-State Prediction to illustrate the CoSMo (Consecutive State Modeling) strategy. Given triplets of input data $\{\mathcal{V}_{t:1\rightarrow 2},\mathcal{A}_{t:1\rightarrow 2},\mathcal{T}\}$, the model learns to predict future states from consecutive inputs. We introduce a [MASK] token to represent a zero-initialized state at $t=0$. The model first predicts the state at $t=2$ using inputs from $t=0$ and $t=1$, then continues with $t=1$ and $t=2$ to predict $t=3$, and so on.

**Table 6: Comparison of baselines with respect to input modality and prediction capabilities. Our model uniquely supports egocentric inputs with multimodal conditions (text, visual, action) and predicts future frames, narrative text, and action movements.**
| Model | Input Modality | Prediction Capability |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Scene | Text | Action | Frame | Text | Action |  |
| LWM | Exocentric | ✓ |  | ✓ | ✓ |  |
| GAIA-1 | Exocentric | ✓ | Vehicle behavior | ✓ |  |  |
| WorldDreamer | Exocentric | ✓ | Vehicle behavior | ✓ |  |  |
| LEGO | Egocentric | ✓ |  | ✓ | ✓ |  |
| OpenVLA | Robot | ✓ |  |  | ✓ |  |
| PEVA | Human | ✓ | Body skeleton | ✓ |  |  |
| Ours (Ego-PM) | Omni | ✓ | Hand pose / Robot kinematics | ✓ | ✓ | ✓ |

**Table 7: Frame prediction results on the Nymeria dataset. Lower FID and LPIPS, and higher PSNR indicate better performance.**
| Model | Frame Prediction |  |  |
| --- | --- | --- | --- |
| FID$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ |  |
| PVEA-B (original, DiT-based) | 74.338 | 16.01 | 33.7 |
| PVEA-B (reproduced) | 78.56 | 15.61 | 35.8 |
| Ours (LDM-based) | 76.21 | 16.10 | 33.4 |

#### Comparison of Model Capabilities.

Table [6](https://arxiv.org/html/2508.19852v2#Sx7.T6) compares various models in terms of input modality and prediction ability. LWM (Liu et al. 2024) can process long exocentric videos and generate both frames and text. GAIA-1 (Hu et al. 2023) and WorldDreamer (Wang et al. 2024b) use exocentric views and vehicle behavior for frame prediction. LEGO (Lai et al. 2025), though not a full world model due to missing action prediction, remains a strong baseline for egocentric frame generation.

### Pseudo-code for Consecutive State Modeling (CoSMo)

To improve temporal consistency and action prediction in egocentric settings, we introduce CoSMo, which utilizes both current and previous states to predict future ones. Unlike prior work (Lai et al. 2025) that relies on only the current state, our method captures richer temporal dependencies.

## Appendix III: Additional Experiments

#### Human-Skeleton-Based Video Prediction.

Beyond hand trajectory modeling, we explore the use of full-body motion as an action representation. Specifically, we replace the original $4\times 2$ hand coordinates with 17 human keypoints (skeleton) to examine whether explicitly modeling richer action space improves video generation.

We use the Nymeria dataset (Ma et al. 2024), which contains egocentric RGB frames, 3D body pose annotations, and textual descriptions in real-world scenarios. We use PEVA (Bai et al. 2025) as the baseline and filter training samples with available text.

PEVA is a video-based DiT method that requires 3–15 continuous frames and corresponding skeleton inputs to predict 64 future frames. In contrast, our method (LDM-based) only requires two frames and a text prompt. We first predict the action representation, then generate future video frames iteratively using action embeddings.

Results in Table [7](https://arxiv.org/html/2508.19852v2#Sx7.T7) show that our method outperforms the PEVA baseline using the same training set, demonstrating that explicit action modeling enhances prediction quality.

Figure: Algorithm 1 Consecutive State Modeling (CoSMo)

## References

- Bai et al. (2025)
Bai, Y.; Tran, D.; Bar, A.; LeCun, Y.; Darrell, T.; and Malik, J. 2025.
Whole-Body Conditioned Egocentric Video Prediction.
*arXiv preprint arXiv:2506.21552*.
- Blattmann et al. (2023)
Blattmann, A.; Dockhorn, T.; Kulal, S.; Mendelevitch, D.; Kilian, M.; Lorenz, D.; Levi, Y.; English, Z.; Voleti, V.; Letts, A.; et al. 2023.
Stable video diffusion: Scaling latent video diffusion models to large datasets.
*arXiv preprint arXiv:2311.15127*.
- Brown et al. (2020)
Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33: 1877–1901.
- Chen et al. (2023)
Chen, W.-G.; Spiridonova, I.; Yang, J.; Gao, J.; and Li, C. 2023.
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing.
*arXiv preprint arXiv:2311.00571*.
- Chowdhery et al. (2022)
Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022.
Palm: Scaling language modeling with pathways.
*arXiv preprint arXiv:2204.02311*.
- Firoozi et al. (2023)
Firoozi, R.; Tucker, J.; Tian, S.; Majumdar, A.; Sun, J.; Liu, W.; Zhu, Y.; Song, S.; Kapoor, A.; Hausman, K.; et al. 2023.
Foundation models in robotics: Applications, challenges, and the future.
*The International Journal of Robotics Research*, 02783649241281508.
- Gao et al. (2025)
Gao, S.; Zhou, S.; Du, Y.; Zhang, J.; and Gan, C. 2025.
Adaworld: Learning adaptable world models with latent actions.
*arXiv preprint arXiv:2503.18938*.
- Ghosal et al. (2023)
Ghosal, D.; Majumder, N.; Mehrish, A.; and Poria, S. 2023.
Text-to-audio generation using instruction-tuned llm and latent diffusion model.
*arXiv preprint arXiv:2304.13731*.
- Goodfellow et al. (2020)
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020.
Generative adversarial networks.
*Communications of the ACM*, 63(11): 139–144.
- Goyal et al. (2022)
Goyal, M.; Modi, S.; Goyal, R.; and Gupta, S. 2022.
Human hands as probes for interactive object understanding.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 3293–3303.
- Grauman et al. (2022)
Grauman, K.; Westbury, A.; Byrne, E.; Chavis, Z.; Furnari, A.; Girdhar, R.; Hamburger, J.; Jiang, H.; Liu, M.; Liu, X.; et al. 2022.
Ego4d: Around the world in 3,000 hours of egocentric video.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 18995–19012.
- Ha and Schmidhuber (2018)
Ha, D.; and Schmidhuber, J. 2018.
World models.
*arXiv preprint arXiv:1803.10122*.
- Henschel et al. (2025)
Henschel, R.; Khachatryan, L.; Poghosyan, H.; Hayrapetyan, D.; Tadevosyan, V.; Wang, Z.; Navasardyan, S.; and Shi, H. 2025.
Streamingt2v: Consistent, dynamic, and extendable long video generation from text.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, 2568–2577.
- Heusel et al. (2017)
Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30.
- Hu et al. (2023)
Hu, A.; Russell, L.; Yeo, H.; Murez, Z.; Fedoseev, G.; Kendall, A.; Shotton, J.; and Corrado, G. 2023.
Gaia-1: A generative world model for autonomous driving.
*arXiv preprint arXiv:2309.17080*.
- Hu et al. (2022)
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022.
Lora: Low-rank adaptation of large language models.
*ICLR*.
- Huang et al. (2020)
Huang, Y.; Cai, M.; Li, Z.; Lu, F.; and Sato, Y. 2020.
Mutual context network for jointly estimating egocentric gaze and action.
*IEEE Transactions on Image Processing*, 29: 7795–7806.
- James et al. (2020)
James, S.; Ma, Z.; Arrojo, D. R.; and Davison, A. J. 2020.
Rlbench: The robot learning benchmark & learning environment.
*IEEE Robotics and Automation Letters*.
- Jia, Liu, and Rehg (2022)
Jia, W.; Liu, M.; and Rehg, J. M. 2022.
Generative adversarial network for future hand segmentation from egocentric video.
In *European Conference on Computer Vision*, 639–656. Springer.
- Kazakos et al. (2019)
Kazakos, E.; Nagrani, A.; Zisserman, A.; and Damen, D. 2019.
Epic-fusion: Audio-visual temporal binding for egocentric action recognition.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 5492–5501.
- Kim et al. (2024)
Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al. 2024.
Openvla: An open-source vision-language-action model.
*arXiv preprint arXiv:2406.09246*.
- Lai et al. (2025)
Lai, B.; Dai, X.; Chen, L.; Pang, G.; Rehg, J. M.; and Liu, M. 2025.
Lego: Learning egocentric action frame generation via visual instruction tuning.
In *European Conference on Computer Vision*. Springer.
- Lai et al. (2023)
Lai, B.; Ryan, F.; Jia, W.; Liu, M.; and Rehg, J. M. 2023.
Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation.
*arXiv preprint arXiv:2305.03907*.
- Lee, Strobel, and Finn (2018)
Lee, A.; Strobel, M.; and Finn, C. 2018.
Stochastic Adversarial Video Prediction.
In *Advances in Neural Information Processing Systems*.
- Li et al. (2024)
Li, F.; Zhang, R.; Zhang, H.; Zhang, Y.; Li, B.; Li, W.; Ma, Z.; and Li, C. 2024.
Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.
*arXiv preprint arXiv:2407.07895*.
- Li, Liu, and Wu (2023)
Li, J.; Liu, K.; and Wu, J. 2023.
Ego-Body Pose Estimation via Ego-Head Pose Estimation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 17142–17151.
- Liang et al. (2022)
Liang, P. P.; Lyu, Y.; Chhablani, G.; Jain, N.; Deng, Z.; Wang, X.; Morency, L.-P.; and Salakhutdinov, R. 2022.
Multiviz: Towards visualizing and understanding multimodal models.
*arXiv preprint arXiv:2207.00056*.
- Lin et al. (2022)
Lin, K. Q.; Wang, J.; Soldan, M.; Wray, M.; Yan, R.; XU, E. Z.; Gao, D.; Tu, R.-C.; Zhao, W.; Kong, W.; et al. 2022.
Egocentric video-language pretraining.
*Advances in Neural Information Processing Systems*, 35: 7575–7586.
- Liu et al. (2024)
Liu, H.; Yan, W.; Zaharia, M.; and Abbeel, P. 2024.
World model on million-length video and language with ringattention.
*arXiv preprint arXiv:2402.08268*.
- Liu et al. (2022)
Liu, S.; Tripathi, S.; Majumdar, S.; and Wang, X. 2022.
Joint hand motion and interaction hotspots prediction from egocentric videos.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 3282–3292.
- Lotter, Kreiman, and Cox (2017)
Lotter, W.; Kreiman, G.; and Cox, D. 2017.
Deep predictive coding networks for video prediction and unsupervised learning.
In *arXiv preprint arXiv:1605.08104*.
- Lu et al. (2025)
Lu, G.; Zhang, S.; Wang, Z.; Liu, C.; Lu, J.; and Tang, Y. 2025.
Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation.
In *European Conference on Computer Vision*, 349–366. Springer.
- Ma et al. (2024)
Ma, L.; Ye, Y.; Hong, F.; Guzov, V.; Jiang, Y.; Postyeni, R.; Pesqueira, L.; Gamino, A.; Baiyya, V.; Kim, H. J.; et al. 2024.
Nymeria: A massive collection of multimodal egocentric daily motion in the wild.
In *European Conference on Computer Vision*, 445–465. Springer.
- Niu et al. (2024)
Niu, D.; Sharma, Y.; Biamby, G.; Quenum, J.; Bai, Y.; Shi, B.; Darrell, T.; and Herzig, R. 2024.
Llarva: Vision-action instruction tuning enhances robot learning.
*arXiv preprint arXiv:2406.11815*.
- Ragusa, Farinella, and Furnari (2023)
Ragusa, F.; Farinella, G. M.; and Furnari, A. 2023.
StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 3635–3644.
- Rombach et al. (2022)
Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 10684–10695.
- Shi et al. (2023)
Shi, Y.; Jiang, K.; Li, J.; Qian, Z.; Wen, J.; Yang, M.; Wang, K.; and Yang, D. 2023.
Grid-centric traffic scenario perception for autonomous driving: A comprehensive review.
*arXiv preprint arXiv:2303.01212*.
- Souček et al. (2024)
Souček, T.; Damen, D.; Wray, M.; Laptev, I.; and Sivic, J. 2024.
Genhowto: Learning to generate actions and state transformations from instructional videos.
In *2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE.
- Sudhakaran, Escalera, and Lanz (2019)
Sudhakaran, S.; Escalera, S.; and Lanz, O. 2019.
Lsta: Long short-term attention for egocentric action recognition.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 9954–9963.
- Team et al. (2024)
Team, O. M.; Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; et al. 2024.
Octo: An open-source generalist robot policy.
*arXiv preprint arXiv:2405.12213*.
- Tian et al. (2024)
Tian, X.; Gu, J.; Li, B.; Liu, Y.; Wang, Y.; Zhao, Z.; Zhan, K.; Jia, P.; Lang, X.; and Zhao, H. 2024.
Drivevlm: The convergence of autonomous driving and large vision-language models.
*arXiv preprint arXiv:2402.12289*.
- Tome et al. (2020)
Tome, D.; Alldieck, T.; Peluse, P.; Pons-Moll, G.; Agapito, L.; Badino, H.; and De la Torre, F. 2020.
Selfpose: 3d egocentric pose estimation from a headset mounted camera.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*.
- Touvron et al. (2023)
Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*.
- Walke et al. (2023)
Walke, H. R.; Black, K.; Zhao, T. Z.; Vuong, Q.; Zheng, C.; Hansen-Estruch, P.; He, A. W.; Myers, V.; Kim, M. J.; Du, M.; et al. 2023.
Bridgedata v2: A dataset for robot learning at scale.
In *Conference on Robot Learning*. PMLR.
- Wang et al. (2024a)
Wang, B.; Sridhar, N.; Feng, C.; Van der Merwe, M.; Fishman, A.; Fazeli, N.; and Park, J. J. 2024a.
This&that: Language-gesture controlled video generation for robot planning.
*arXiv preprint arXiv:2407.05530*.
- Wang et al. (2023)
Wang, J.; Luvizon, D.; Xu, W.; Liu, L.; Sarkar, K.; and Theobalt, C. 2023.
Scene-aware Egocentric 3D Human Pose Estimation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 13031–13040.
- Wang et al. (2024b)
Wang, X.; Zhu, Z.; Huang, G.; Wang, B.; Chen, X.; and Lu, J. 2024b.
Worlddreamer: Towards general world models for video generation via predicting masked tokens.
*arXiv preprint arXiv:2401.09985*.
- Wu et al. (2024)
Wu, W.; Li, Z.; Gu, Y.; Zhao, R.; He, Y.; Zhang, D. J.; Shou, M. Z.; Li, Y.; Gao, T.; and Zhang, D. 2024.
Draganything: Motion control for anything using entity representation.
In *European Conference on Computer Vision*, 331–348. Springer.
- Yang et al. (2023)
Yang, D.; Tian, J.; Tan, X.; Huang, R.; Liu, S.; Chang, X.; Shi, J.; Zhao, S.; Bian, J.; Wu, X.; et al. 2023.
Uniaudio: An audio foundation model toward universal audio generation.
*arXiv preprint arXiv:2310.00704*.
- Ye et al. (2023)
Ye, Y.; Li, X.; Gupta, A.; De Mello, S.; Birchfield, S.; Song, J.; Tulsiani, S.; and Liu, S. 2023.
Affordance diffusion: Synthesizing hand-object interactions.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 22479–22489.
- Zhang et al. (2023)
Zhang, B.; Ge, Y.; Xu, X.; Shan, Y.; and Shou, M. Z. 2023.
Taca: Upgrading your visual foundation model with task-agnostic compatible adapter.
*arXiv preprint arXiv:2306.12642*.
- Zhang et al. (2017)
Zhang, M.; Teck Ma, K.; Hwee Lim, J.; Zhao, Q.; and Feng, J. 2017.
Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, 4372–4381.
- Zhang et al. (2018)
Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018.
The unreasonable effectiveness of deep features as a perceptual metric.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, 586–595.
- Zhang et al. (2025)
Zhang, Z.; Liao, J.; Li, M.; Dai, Z.; Qiu, B.; Zhu, S.; Qin, L.; and Wang, W. 2025.
Tora: Trajectory-oriented diffusion transformer for video generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*.
- Zheng et al. (2025)
Zheng, W.; Chen, W.; Huang, Y.; Zhang, B.; Duan, Y.; and Lu, J. 2025.
Occworld: Learning a 3d occupancy world model for autonomous driving.
In *European Conference on Computer Vision*, 55–72. Springer.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: Why is "Ours" bolded for LPIPS in Table 3 when it is outperformed by This&That?

How do you calculate the GT hand bounding boxes when Ego4D does not provide such poses out-of-the-box? Could the baseline Hand IoU in tables 2 and 3 be calculated by using that same method on the imagined future frames of the baselines?
soundness: 3
strengths: The design choices of the method are thoroughly ablated in Table 5. The method shows strong performance against baselines in Tables 2 and 3. The inclusion of both a robotic dataset and a human hand dataset showcases the usefulness and versatility of the method.
summary: The authors propose a method which, given the action history, visual input and language prompt of either an egocentric video or a robot task rollout video, simultaneously generates imagined action rollouts and, conditioned on these, imagined future video frames. The method makes use of a novel cross-attention scheme to condition a latent diffusion model for the future frame synthesis.
weaknesses: The submission could have benefited from the inclusion of more qualitative examples in the supplementary material, comparing the imagined and real future frames. This is crucial to assess the quality of the generations.

I believe Table 3 should be split into two separate tables, evaluating frame prediction and action prediction separately.

In Table 4, as it stands, the method is strongly outperformed by two baselines depending on 3D point clouds. An interesting comparison would be running the baselines without providing them the additionally required 3D point cloud (e.g. by passing dummy values), to assess the extra gain from its inclusion. I am willing to increase my score if a sensible answer is provided to this point.

In the paper, please clarify how you calculate the hand IoU. It took me a long time to find the detail that you use hand bounding boxes in the supplement. I believe this information is important enough to be included in the main paper. 

A bounding box is arguably far less useful than, for instance, a wrist pose or even hand pose, either in 2D or 3D. Thus, I would encourage the authors to study predicting more informative hand representations for egocentric videos.

Table 1 and Table 2 could benefit from the inclusion of more baselines.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: Please address any of the weaknesses identified above.
soundness: 2
strengths: The design is intuitive and leads to an architecture that achieves sota result when compared to few recent baselines.
summary: is a tow-stage framework for action anticipation and frame generation coupling VLM and LDM through Causal Cross-Attention (CCA). VLM predicts action trajectories that then condition LDM to generate future frames. CCA implements cross-attention with future masking, and is applied to encode and fuse video, text and action for conditioning future frame generation.
weaknesses: Causal Cross Attention, a main element upon which the contribution of the paper is build, is not particularly novel: standard cross-attention with causal masking.

Why the query in CCA is chosen as the embedding of previous action is not well motivated. It is just adopted this way.

There is no feedback loop that enforces generated frames to contain generated action trajectories following conditioning action trajectories.

Overall, there is no particularly significant novelty. The method is two-stage and not integrated, and there is no grounding loop in the diffusion with the action trajectory conditioning. Loss terms for training are standard or previously published.

It is not clarified in the paper why generating future frame in a 2-stage approach without grounding (see above) should be beneficial. For what should the generated video be used?

Tables are spread around the second part of paper, and the arrangement of frames in figure 3 could be improved. Figure 3 contains only four generated, sparse frames and this makes it difficult to assess the quality of generated future video.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Long-Horizon Generalization: The model is evaluated on very short-term predictions. Could the authors demonstrate or discuss its potential for longer-horizon prediction (e.g., 20 frames)? What are the main failure modes (e.g., object deformation, trajectory drift) when the prediction horizon extends, and how might the architecture be adapted to address them? 

Ablation on Action Representation: The action is represented as hand trajectory coordinates for Ego4D and 7D robot poses for Bridge/RLBench. How critical is the specific form of this action representation? Did the authors experiment with other representations, such as a more abstract latent action space (e.g., akin to AdaWorld), and if so, how did it impact performance? 

Causal Cross-Attention Analysis: The CCA module is a key innovation. Could the authors provide more analysis or visualization of the attention maps within the CCA? For example, when the action query attends to the visual keys, which parts of the historical frames (e.g., the object, the hand's current position) does it primarily focus on? 

Computational Cost and Latency: The paper would benefit from a discussion (and ideally, metrics) regarding the computational cost and latency of the full pipeline compared to the baselines (especially the faster VLA models and single-stage video predictors).
soundness: 3
strengths: Originality: The core originality lies in its novel problem formulation: the joint modeling of egocentric action prediction and future video generation within a single, unified framework. This is a distinct advance over prior work that addressed these tasks separately. 
Quality: The technical quality is high. The two-stage architecture is well-motivated and built upon strong, modern components (LLaVA, LDM). The experimental quality is exceptional, featuring rigorous evaluation on three diverse and challenging datasets (Ego4D, BridgeData V2, RLBench). 
Clarity: The paper is generally well-structured and clearly written. The logical flow from problem identification to solution and evaluation is easy to follow. 
Significance: The work is highly significant for the research community. It directly addresses a critical capability for embodied AI systems: understanding the coupling between an action and its perceptual consequences.
summary: This paper addresses a critical gap in egocentric AI: the disjointed modeling of future actions and their visual outcomes. This paper proposes Ego-PM, a novel two-stage framework that unifies action prediction and future video generation conditioned on hand trajectories. The core idea is to use predicted hand motion as an intermediate, guiding representation to ensure that generated future frames are both visually plausible and physically consistent with the intended action. Below are contributions： 
1.	First Unified Action-Visual Predictive Model: This is the first model capable of jointly predicting both the upcoming action (as a hand trajectory) and the future visual frames resulting from that action. 
2.	Novel Architectural Innovations: Consecutive State Modeling (CoSMo): In Stage I, the model predicts future actions by conditioning on two consecutive previous states. Causal Cross-Attention (CCA): In Stage II, the action embedding from Stage I serves as the Query to perform causal cross-attention over the visual and textual context (Keys/Values). 
3.	Demonstrated Generality and Practicality: The same model architecture, without task-specific modifications, achieves state-of-the-art or competitive performance on three distinct benchmarks: Ego4D (human egocentric videos), BridgeData V2 (real-world robot demonstrations), and RLBench (simulated robotic tasks). Crucially, the model requires no external action annotations at inference time, predicting the hand trajectory autonomously, which enhances its practical applicability.
weaknesses: Analysis of Error Propagation in Long-Horizon Prediction: The experiments primarily focus on predicting one or two frames into the future (e.g., t=2, t=3). A key challenge in predictive modeling is error accumulation over longer sequences. The paper would be significantly strengthened by an analysis of the model's performance degradation when generating longer video horizons (e.g., 20 frames). Does the CoSMo strategy provide robustness against compounding errors compared to baselines? 

Computational Efficiency and Latency: The proposed two-stage pipeline, involving a large autoregressive model and an iterative diffusion model, is computationally intensive. For real-world applications like robotic planning, inference speed is critical.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. In the experiments, the authors explored using one or two consecutive frames as context, what about using more time steps?

2. What is the advantage of CCA compared to bi-directional attention in your model?

3. The ablation study on consecutive predictions is not very clear. Please explain in detail.

4. In L456-457, the authors claimed the failure of LWM as its exocentric bias, however, the authors already fine-tuned LWM on egocentric data (L260). It is a bit confusing.
soundness: 3
strengths: 1. This paper combines action modeling with video generation in a single framework.

2. This model demonstrates consistent improvements across multiple benchmarks (Ego4D, BridgeData, RLBench) and modalities.
summary: The paper proposes Ego-PM, a unified two-stage egocentric predictive model designed to jointly predict actions and generate future visual frames, conditioned on hand trajectories. Extensive experiments have demonstrated the effect of each proposed component on multiple real-world and robotic datasets.
weaknesses: 1. One of the main contribution of this paper is the joint prediction of action and future frames, however, I do not see the qualitative/quantitative analysis showing the mutual benefit. 

2. It is not clear how the model fuses multi-modal information when not using CCA in Table 5. 

3. The text format could be further improved (L84-L85).

4. This paper misses some discussions with recent works on conditioned egocentric image/video generation [1][2][3].

[1] Luo et al. "Put myself in your shoes: Lifting the egocentric perspective from exocentric videos." ECCV 2024.
[2] Xu et al. "Egoexo-gen: Ego-centric video prediction by watching exo-centric videos." ICLR 2025.
[3] Liu et al. "Exocentric-to-egocentric video generation." NeurIPS 2024.


Decision: Reject