Whole-Body Conditioned Egocentric Video Prediction
Authors: Yutong Bai
Danny Tran
Amir Bar
Yann LeCun
Trevor Darrell
Jitendra Malik
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
World Models
Generative Model
Primary area: deep_learning
TL;DR: We introduce a diffusion-based video model that predicts egocentric futures from full-body 3D motion, enabling realistic and controllable first-person simulation.

Abstract: We train models to predict ego-centric video from human actions (PEVA), given the past video and an action represented by the relative 3D body pose. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, a large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with increasingly challenging tasks, enabling a comprehensive analysis of the model’s embodied prediction and control abilities. Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.

Whole-Body Conditioned Egocentric Video Prediction
Yutong Bai"
Danny Tran"
Amir Bar*
Yann LeCun 2,3
Trevor Darrell
Jitendra Malik 1,2
UC Berkeley (BAIR)
FAIR, Meta
New York University
Abstract
We train models to Predict Ego-centric Video from human Actions (PEVA), given the past video and an action represented by the relative 3D body pose. By condi- tioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with in- creasingly challenging tasks, enabling a comprehensive analysis of the model's embodied prediction and control abilities. Our work represents initial attempt to tackle the challenges modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of human
Introduction
Human movement is rich, continuous, and physically grounded (Rosenhahn et al. 2008 Aggarwal and Cail 1999). The way we walk, lean, turn, or reach- -oft-oofen subtle and coordinated directly shapes what we see from first-person perspective. For embodied agents to simulate and plan like humans, they must not only predict future observations (Von Helmholtz 1925). but also understand how visual input arises from whole-body action (Craik 1943). This understanding is essential because many aspects of the environment are not immediately visible-we need to move our bodies to reveal new information and achieve our goals.
Vision serves as a natural signal for long-term planning (LeCun 2022 Hafner et al. 2023 Ebert
al. 2018 Ma al. (022). We look our environment plan and act, using our egocentric view as predictive goal (Sridhar et al. 2024 Bar et al. (025). When we consider our body movements, we should consider both actions of the feet (locomotion and navigation) and the actions of the hand (manipulation), or more generally, whole-body control (Nvidia et al.) 2025 Cheng et al. 2024 He al. 2024b, Radosavovic et al. 2024 He al. 2024a] Hansen et al. 2024). ). For example, when reaching for an object, we must anticipate how our arm movement will affect what we see, even before the object comes into view. This ability to plan based on partial visual information is crucial for embodied agents to operate effectively in real-world environments.
Building model that can effectively learn from and predict based on whole-body motion presents several fundamental challenges. First, representing human actions requires capturing both global body dynamics and fine-grained joint articulations, which involves high-dimensional, structured data with complex temporal dependencies. Second, the relationship between body movements and
Equal contribution; Equal advising.
Projecct page: https://dammttran123 A 1o/PEVA
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

BPN
(a) Atomic Actions
(b) Counterfactuals
(c) Long Video
Figure 1: Predicting Ego-centric Video from human Actions (PEVA). Given past video frames and an action specifying desired change in 3D pose, PEVA predicts the next video frame. Our results show that, given the first frame and a sequence of actions, our model can generate videos of atomic actions (a), simulate counterfactuals (b), and support long video generation (c).
visual perception is highly nonlinear and context-dependent the same arm movement can result in different visual outcomes depending on the environment and the agent's current state. Third, learning these relationships from real-world data is particularly challenging due to the inherent variability in human motion and the subtle, often delayed visual consequences of actions.
To address these challenges, we develop a novel approach PEVA that combines several key innova- tions. First, we design a structured action representation that preserves both global body dynamics and local joint movements, using a hierarchical encoding that captures the kinematic tree structure of human motion. This representation enables the model understand both the overall body movement and the fine-grained control of individual joints. Second, we develop novel architecture based on conditional diffusion transformers that can effectively model the complex, nonlinear relationship between body movements and visual outcomes. The architecture incorporates temporal attention mechanisms to capture long-range dependencies and a specialized action embedding component that maintains the structured nature of human motion. Third, we leverage a large-scale dataset of syn- chronized egocentric video and motion capture data (Ma et al., 2024), which provides the necessary training signal to learn these complex relationships. Our training strategy includes random timeskips to handle the delayed visual consequences of actions and sequence-leve training to maintain temporal coherence.
For evaluation, we design hierarchical evaluation protocol to understand PEVA's 's capabilities across different levels of complexity. First, we assess its ability to predict immediate visual consequences through single-step step predictions. Second, we decompose complex human movements into atomic actions to test the model's ss understanding of how specific joint-level movements affect the egocentric view. Third, we examine the model's capability to predict long-term visual consequences by evaluating on extended time horizons, where the effects of actions may be delayed or not immediately visible. Finally, we explore the model's ability to serve as a world model for planning by using it to simulate actions and choose the ones that lead to predefined goal. This layered approach allows us to systematically analyze the strengths and limitations of our model, revealing both its capacity to simulate embodied perception and the open challenges that remain in bridging the gap between physical action and visual experience.
To conclude, we introduce PEVA, diffusion- model that predicts future egocentric video conditioned on whole-body motion. By grounding prediction in 3D whole-body movement, our model captures the intricate relationship between movement and visual perception. Our comprehensive evaluation framework demonstrates that whole-body control significantly improves video quality, semantic consistency, and simulating counterfactuals.

---

2
Related Works
World Models. The concept of "world model", an internal representation of the world used for prediction and planning, has a rich history across multiple disciplines. The idea was first proposed in psychology by Craik (1943). who hypothesized that the brain uses "small-scal models" of reality to anticipate events. This principle found parallel development in control theory, where methods like the Kalman Filter and Linear Quadratic Regulator (I.QR) rely on an explicit model of the system to be controlled (Kalman| 1960). The idea of internal models became central to computational neuroscience for explaining motor control, with researchers proposing that the brain plans and executes movements by simulating them first (Jordan 1996 Kawato et al.. 1987 Kawato 1999).
With the rise of deep learning, the focus shifted learning these predictive models directly from data. Early work in computer vision demonstrated that models could learn intuitive physics from visual data solve simple control tasks like playing billiards or poking objects (Fragkiadaki et al. 2015; Agrawal et al. 2016). This paved the way for modern, large-scale world models that predict future video frames conditioned on actions, enabling planning by "imagining" future outcomes (Ha
and Schmidhuber 2018 Hafner et al.) Liu et al. 2024 Li et al. 2022 Zhou et al., 2024 Yang et al. 2023 2024] Assran et al.] 2025). reinforcement learning, models like Dreamer have shown that
learning a world model improves sample efficiency (Hafner et al. 2023). Recent approaches have
used diffusion models for more expressive generation; for example, DIAMOND generates multi-step rollouts via autoregressive diffusion (Alonso et al., 2024). In the egocentric domain, Navigation World Models (NWM) used conditional diffusion transformers (CDiT) to predict future frames from planned trajectory (Bar et al. 2025). However, these models use uww-dimnsioonl controls and neglect the agent's 's own body dynamics. We build on this extensive line of work by conditioning video prediction whole-body pose, enabling more thysically-groundee simulation.
Human Motion Generation and Controllable Prediction. Human motion modeling has advanced from recurrent and VAE-based methods (Rempe et al. | Petrovich et al.)
[221] Ye et al.) 2023) to
powerful diffusion- generators (Tevet et | 2022] Zhang et al., 2024). These models generate
diverse, realistic 3D pose sequences conditioned on text (Hong et al. 2024 Guo et al. 2022 Dabral
ct al. 2023), audio (Ng et al. [2024, Dabral et al. 2023 Ao et al. 2023), and head pose (Li et al.
2023 Castillo et al.) 2023 Yi et al. 2025). Recent works like Animate Anyone (Hu et al.) 20233 and
human animations from a reference image and extensions like PhysDiff (Yuan et al.) (023) incorporate contact into the denoising loop. While prior works treat pose as the target, our model uses as input for egocentric video prediction, reversing the typical motion generation setup. This enables fine-grained visual control, bridging cose-conditioned video generation /Wu et al. 2023 Zhang et al. 2023) with embodied simulation. Unlike Make-a- Video (Singer et al.) 2022) Tune-A- Video (Wu et al. 2023)
MagicAnimate (Xu et al. 2023) generate
pose sequence. Physically-awan
which focus on text/image prompts, we condition directly on physically realizable body motion.
Egocentric Perception and Embodied Forecasting, Egocentric video datasets such as Ego4D (Gau- man et al. 2022), Ego-Exo4D (Grauman et al.) 2024) and EPIC-KITCHENS (Damen et al. 2018)
were used to study human action recognition, object anticipation (Furnari and Farinella 2020)) future
video prediction (Girdhar and Grauman (02T). and even animal behavior (Bar et al. (0224). To study pose estimation, EgoBody (Zhang et al. 2022) and Nymeria (Ma et al. (2224) provide synchronized egocentric video and 3D pose. Unlike these works, we treat future body motion as a control signal, enabling visually grounded rollout. Prior works in egocentric pose forecasting (Yuan and Kitani 2019) and visual foresight (Finn and Levine (017) show that predicting future perception supports downstream planning. Our model unifies these lines by predicting future egocentric video from detailed whole-body control, enabling first-person planning with physical and visual realism.
3 PEVA
In this section we describe our chhole-body-cooditionee ego-centric video prediction model. We start by describing how to represent human actions (Section 3.1). then move on to describe the model and the training objective (Section 3.2). Finally, we describe the model architecture in Section 3.3
3

---

loss
withing
Enooderr
cattn: to
Encoderr
context
Per
Encodder
MLP
Frame
AdaLN
clean
+nolee
attn. cattn.
7ho
fc
tokens
(shift, scale, = ((%..)
[XM) (b) Auto Regressive Training
(a) Random Time Skips
(c) Conditioning Types
Figure 2: Design of PEVA. To train on an input video, we choose a random subset of frames and encode them via fixed encoder (a). They are then fed CDiT that is trained autoregressively with teacher forcing (b). During the denoising process, each token attends to same-image tokens and cross-attends clean tokens from past image(s). Action conditioning is done via AdaLN layers.
3.1 Structured Action Representation from Motion Data
To effectively capture the relationship between human motion and egocentric visual perception, we define each action as nigh-dimensional vector encoding both global body dynamics (global translation via the root joint) and detailed joint articulations structured by the kinematic tree. We synchronize motion capture data with egocentric video frames using timestamps and convert global coordinates into local frame centered the pelvis, ensuring invariance initial position and orientation. Global positions are converted to local coordinates, quaternions to relative Euler angles, and joint relationships are preserved using the kinematic hierarchy. We normalize all motion parameters for stable learning: positions are scaled to [-1, 1] and rotations bounded within [-T, x]. Each action reflects the change between consecutive frames, allowing the model to learn how physical movements produce visual outcomes. Our motion representation follows the Xsens skeleton (Movella 2021 Ma et al. 2024), which shares the kinematic tree of SMPI. (Loper et al. 2023 but differs in joint set and ordering, and omits body shape parameters.
3.2 Ego-Centric Video Prediction for Whole-Body Control
Next, we describe our formulation of PEVA from the perspective of an embodied agent. Intuitively, the model is autoregressive diffusion model that receives input video and corresponding sequence of actions describing how the agent moves and acts. Given any prefix of frames and actions, the model predicts the resulting state of the world after applying the last action and considering other environment dynamics.
More formally, we are given a dataset D {(xo, ag IT, ar) of agents videos from egocentric view and their associated body controls, such that every r, E xx x3 is video frame and aj E Iddact an action in the Xsens skeleton ordering (Movella 2021 for the upper body (everything above the pelvis), representing the change in translation, together with the delta rotation of all joints relative to the previous joint rotation. We represent motion in 3D space, thus we have 3 degrees of freedom for root translation, 23 joints for the body and represent relative joint rotations as Euler angles in 3D space leaving dast = 3 23 x 3 72.
We start by encoding each individual frame s; enc(r,) into corresponding state representation, through a pre-trained VAE encoder (Rombach et al.) 2022). Given a sequence of controls dg, CT, goal is to build generative model that captures the dynamics of the environment:
T
P(so) t=0
P( (8t...,, 80, ar,....o)
(I)

---

To simplify the model, we factorize the distribution and make Markov assumption that the next
state is dependent on the last & states and a single past action:
P(s St: Su, ar. ... dn) P .... St-k+1 at-1)
(2)
We aim to train model parametrized by that minimizes the negative log-likelihood:
arg min i 8
log Po(so)
We model each transition Pe (s [/:4+1/s:: St. st-k+1, az) using a Denoising Diffusion Probabilistic Model (DDPM) (Ho al. 2020), which maximizes the (reweighted) evidence lower bound (ELBO) of the log-likelihood. For each transition, we define the forward diffusion process q(=+ S(+1) = N(2: v $1,$1 (1 II) where 2- is the noisy version of 8t+1 noise timestep т, and âr the cumulative product of noise scales. The reverse process is learned by training neural network to predict the noise given 2, and the conditioning context C2 , 4c).
Then denoising loss term for a transition is given by:
Csimple, t = (r,o~ [||= (vārs + vI- ā,e, ce, 7) [|]]]]]]
(3)
Where Simple, is the loss term corresponding to the unconditional generation of so. Additionally, we also predict the covariances the noise, and supervise them using the full variational lower bound loss Lab as proposed by (Nichol and Dhariwal 202T..
Hence the final objective yields (weighted) version of the ELBO for each term in the sequence:
C Caimple, + ACuIB,
Despite not being lower bound of the log-likelihood, the reweighted ELBO works well in practice for image generation with transformers (Nichol and Dhariwal 2021 Peebles and Xie 2023
The advantage of our formulation is that it allows training in parallelized fashion using causal masking. Given a sequence of frames and actions, we can train on every prefix the sequence in a single bowward-ackward pass. Next, we elaborate the architecture of our model.
3.3 Autoregressive Conditional Diffusion Transformer
While prior work in navigation world models (Bar et al. 2025) focuses on simple control signals like velocity and heading, modeling whole-body human motion presacents significantly greater challenges. Human activities involve complex, coordinated movements across multiple degrees of freedom, with actions that are both temporally extended and physically constrained. This complexity necessitates architectural innovations beyond standard CDiT approaches.
To address these challenges, extend the Conditional Diffusion Transformer (CDiT) architecture with several key modifications that enable effective modeling of whole-body motion:
Random Timeskips. Human activities often span long time horizons with actions that can take several seconds complete. At the same time, videos are a raw signal which requires vast amounts of compute to process. To handle video more efficiently, we introduce random timeskips during training (see Figure 2 and include the timeskip as an action to inform the model's prediction. This allows the model to learn both short-term motion dynamics and longer-term activity patterns. Learning long-tenn dynamics is particularly important for modeling activities like reaching, bending, or walking, where the full motion unfolds over multiple seconds. In practice, we sample 16 video frames from a 32 second window.
Sequence-Levè Training, Unlike NWM which predicts single frames, we model the entire sequence of motion by applying the loss over each prefix frames following Eq. 4 We include an example of this in Figure 2p. This is crucial because human activities exhibit strong temporal dependencies the way someone moves their arm depends on their previous posture and motion. We enable efficient training by parallelizing across sequence prefixes through spatial-only attention in the current frame
5

---

and past-frame-onfy attention for historical context (Figure 2 In practice we train models with sequences of 16 frames.
Action Embeddings. The high-dimensional nature of whole-body motion (joint positions, rotations, velocities) requires careful handling of the action space. We take the most simple strategy: con- catenate all actions in time into ID tensor which is fed to each AdaLN layer for conditioning (see Figure 2
These architectural innovations are essential for modeling the rich dynamics of human motion. By training on sequence prefixes and incorporating timeskips, our model learns to generate temporally coherent motion sequences that respect both short-term dynamics and longer-term activity patterns. The specialized action embeddings further enable precise control over the full range of human movement, from subtle adjustments to complex coordinated actions.
3.4 Inference and Planning with PEVA
Sampling procedure at test time. Given set of context frames (t. we encode these frames to get (st, ..., St st-++1) and pass the encoded context as the clean tokens Figure and pass in randomly sampled noise as the last frame. We then follow the DPPM sampling process to denoise the last frame conditioning on our action. For faster inference time, we employ special attention masks where we change the mask in Figure for within image attention to only be applied on the tokens of the last frame and change the mask for cross attention to context so that cross attention only applied for the last frame.
Autoregressive rollout strategy. To follow set of actions we use an autoregressive rollout strategy. Given an initial set of context frames we (It, ((_++++)) we start by encoding each individual frame get (S....., ...y St-k+1) 1) and add the current action to create the conditioning context c (st...., st-k+1, at). We then sample from our model parameterized by 0 to generate the next state: 81+1 = Po($t+1 (cs). We then discard the first encoding and append the generated st+1 and add the next action to produce the next context C++1 = ($1+1, St. St-k+1, 4++1). We then repeat the process for our entire set actions. Finally, to visualize the predictions, we decode the latent states to pixels using the VAE decoder (Rombach et al. 2022).
4 Experiments and Results
4.1 Experiment Setting
Dataset. We use the Nymeria dataset (Ma et al. 2024), which contains synchronized egocentric video and full-body motion capture, recorded in diverse real-world settings using an XSens system (Movella 2021). Each sequence includes RGB frames and 3D body poses in the XSens skeleton format, covering global translation and rotations of body joints. We sample body motions at 4 FPS. Videos are center-cropped and resized to 224x224. We split the dataset 80/20 for training and evaluation, and report all metrics on the validation set.
Training Details. We train variants of Conditional Diffusion Transformer (CDIT-S to CDIT-XXL, up to 32 layers) using context window of 3-15 frames and predicting 64-frame trajectories. Models operate on 2x2 patches and are conditioned on both pose and temporal embeddings. We use A (Ir=8e-5, betas=(0.9, 0.95), grad c0ip=10.0) and batch size 512. Action inputs normalized to [-1, for translation and [- n, x for rotation. All experiments use Stable Diffusion VAE tokenizer and follow NWM's hardware and evaluation setup. Metrics are averaged over samples per sequence.
4.2 Comparison with Baselines
To comprehensively evaluate our model, we compare PEVA with CDiT (Bar al. 2025) and Diffusion Forcing (Chen et al., 2024) along two key dimensions. First, to assess whether the model faithfully simulates future observations conditioned on actions, we evaluate the perceptual and semantic similarity, and action consistency of the generated frames. We use LPIPS (Zhang et al. 2018ab and DreamSim (Fu et al. 2023), to measure perceptual and semantic similarity. To assess action consistency, we evaluate how faithfully the model follows the intended actions by measuring camera motion, using Absolute Trajectory Error (ATE) and Relative Pose Error (RPF) (Sturm) et al. 2012), and 2D wrist keypoints, using Percentage of Correct Keypoints (PCK @0.2), precision,

---

recall, and accuracy. More details on the camera pose estimation and wrist position evaluation are in Supplementary Section 7 Second. to evaluate the overall quality and realism of the generated samples, we report FID (Heusel et al. 2017).
As shown in Table our model achieves better results on both action consistency and generative quality. Furthermore. Figure shows that our models tend to maintain lower FID scores than the baselines as the prediction horizon increases, suggesting improved visual quality and temporal consistency over longer rollouts. Qualitative results for 16 second rollouts can be seen in Figure
Figure 5 and Supplementary Figures 22 34 We implement Diffusion Forcing (DF*) on top of PEVA by applying the diffusion forward process to the entire sequence of encoded latents, then predicting the next state given the previous (noisy) latents. At test time, we autoregressively predict the next state as in PEVA, without injecting noise into previously predicted frames, like Chen et al.
(2024).
We additionally, compare PEVA against Cosmos (Agarwal et al. 2025) on wrist position accuracy with PCK @0.2. We find that whole-body actions as control signal leads to significantly finer control compared to text conditioning, with PEVA achieving an accuracy of 0.85 compared to 0.22 for Cosmos.
Table 1: Baseline Metrics. Comparison of baselines on single-step prediction 2 seconds ahead.
Model
LPIPS
DrcamSim
ATE
RPE (Trans)
RPE (Rot.) L
PCK00.21 1
Precision f
Recall
Accuracy
FID
DF
0.332
0. 0:2440005
0.464
0.422
36,514
0.750
0.914
0.582
0.679
73.022-000
0.313
0000
0.367
26.057
0.755
0.865
0.833
63.714 on
PEVA
0.3
0.19 193
0.274
0.266
13.527
0.791
0.923
0.888
0.871
62,293
4.3 Atom Actions Control
To evaluate PEVA's 's ability to follow structured physical control, we decompose complex mo- tions into atomic actions. We extract video seg- ments exhibiting fundamental movements-such as hand motions (up, down, left, right) and whole- body actions (forward, botete)-bssed on thresh- olded positional deltas. We sample 100 exam- ples per action type, and evaluate single-step pre- diction 2 seconds ahead. Qualitative results are shown in Figure Th and Figure and quantitative results in Table 2 See Supplementary Figure 21 for more qualitative examples of interaction rich atomic actions.
110
100
DP
GBU
Ours (X1.
ascooddee
Figure 3: Video Quality Across Time (FID).
Comparison of generation accuracy and quality as function of time (up to 16s). Qualitative results for 16-second rollouts can be seen in Figure Figure 5 and Supplementary Figures 22
34
Table 2: Atomic Action Performance. Comparison of models in generating videos of atomic actions evaluated on IPIPS.
Model
Navigation
Right
Forward
Roc.
Rot.
Right
Up
Dowa
Left
Right
DE*
0.39g0.011 0,314 006 27g o0d 292 009 0.306 005 3322 o08 323 000 304 006 3.5000007 g. 3 005 o. 296 o0e
CDT
0.349 004 0.284 000,,0 q. 0.24400000
20..0.00407777 9.2770. 004 u. 2..0 0.00 u 0. .00 205 0099 u. 2.277.000 w0w.0.0............... 2.207 0.206 ". o u.278 0.277
(XL)
0.3370.006
000 0.2777.00000 2007 0. 0.242 0007 2.244 0.000 2.257 804 2722 oos 2.30 nos 0os 0.271 o.0 0.2877.000 0.265 004, 250 oos
Ours (XXL
0.325
000 0.26600000 g 2.30..004
0.23600003 u. 24241 o 003 u.20 0:0000..447777 0.247 000 0.266 no7 u.240. 05000 q g.2620 004 u. 0.000
4.4 Ablation Studies
We first conduct ablation studies to assess the impact of context length, action representation, and model size in PEVA summarized in Table 3 First, increasing the context window from 3 to 15 frames consistently improves performance across all metrics, highlighting the importance of temporal context for egocentric prediction. Second, we compare two action embedding strategies encoding versus simple concatenation andd find that the latter performs competitively despite its simplicity, suggesting that our structured action representation already captures sufficient motion information. Third, model scale plays a significant role: larger variants from PEVA-S to PEVA-XXL show steady gains in perceptual and semantic fidelity. The gray-highlighted row denotes the default configuration in main experiments. See Supplementary Sections 8.1 ablations on action type, timeskips, and action conditioning.
8.2 and 8.3 for additional
7

---

Move Forward
Rotate Left
Rotate Right
Move Left Hand Up
Move Left Hand Down
Move Right Hand Up
Move Right Hand Down
Move Left Hand Left
Move Left Hand Right
Move Right Hand Left
Move Right Hand Right
Figure 4: Atom Actions Generation. We include video generation examples of different atomic actions specified by 3D-body poses.

---

R
N
OL
4
1
Figure 5: Generation Over Long-Horizon We include 16-second video generation examples.
Score: 0.623
Score: D. 62.244
Score: D.575
Figure 6: Planning with Counterfactuals. We demonstrate planning example by simulating multiple action candidates using PEVA and scoring them based on their perceptual similarity to the goal, as measured by LPIPS Zhang et al. (2188). We show that PEVA enables us to rule out action sequences that leads us the sink the top row, and outdoors in the second row.

---

Table Model Ablations. We evaluate the impact of different context lengths, action embedding methods, and model sizes on single-step prediction performance (2 seconds into the future).
Metries
Configuration
LPIPS 4
DreamSim
PSNR
FID 4
Context Length
frames
0.304
0.1990.003
16.469%.044
03.9660-421
7 frames
0.304
0.195
16.443
62.5400t35
frames
0.303
0.193
16.5110.061
62.293".671
Action Representation
Action Embedding (d = 512)
0.3170.003
0.2022.002
16.195
63. 63.101% 0.341
Action Concatenation
0.303
0.1939.002
16.511
62. 62.293
Model Size
PEVA-S
0.370
0.3270.002
15.743 080
101 001.300.50
PEVA-B
0.337
0.2469.002
16.0130.09
74.3381. 1.057
PEVA-L
0.30g
0.2022.001
16.4170.037
64.402"
PEVA-XL
0.303
0.193 .022
16.511 0.061
62. 62.293 0.671
PEVA-XXL
0.2988:002
0.186
16.5560.000
61.100%
4.5 Long-Term Prediction Quality
We evaluate the model's ability to maintain visual and semantic consistency over extended prediction horizons. As shown in Figure 5 and Supplementary Figures 22 34 PEVA generates coherent 16-second rollouts conditioned full-body motion. Table 3 reports DreamSim scores at increasing time steps, showing a gradual degradation from 0.178 (1s) to 0.390 (16s), indicating that predictions remain semantically plausible even far into the future. PEVA also demonstrates physical realism over longer sequences, as seen in Supplementary Figure 14
4.6 Planning with Multiple Action Candidates.
We first test localized head and hand counterfactuals in Supplementary Figure 15
We then demon- strate samples where PEVA enables planning with multiple action candidates in Figure Tp, Figure 6 We start by sampling multiple action candidates and simulate each action candidate using PEVA via autoregressive rollout. Finally, we rank each action candidate's 's final prediction by measuring LPIPS similarity with the goal image. We find that PEVA is effective in enabling planning through simulating action candidates.
and Supplementary Figures 18 20
Failure Cases, Limitations and Future Directions
While PEVA shows strong results in predicting egocentric video from whole-body motion, several limitations remain. First, our planning evaluation is preliminary, we only perform simulation- based selection over single-arm actions (see Supplementary Section (9). This demonstrates early planning capability but not full trajectory optimization. Extending PEVA to closed-loop or interactive environments an important next step. Second, the model lacks explicit conditioning on task intent or semantic goals, instead relying on image similarity. Third, long- long-horizon degradation arises from error accumulation in open-loop generation. We plan address this through feedback in interactive environments, or by incorporating text-conditioned supervision supervision to anchor predictions.
6 Conclusion
We introduced PEVA, a diffusion- based model that predicts egocentric video from detailed 3D human motion. Unlike prior work that low-dimensional control, PEVA leverages full-body pose sequences to simulate realistic and controllable visual outcomes. Trained on Nymeria, captures the link between movement and egocentric perception. Experiments show improvements in prediction quality, semantic consistency, and fine-grained control over strong baselines. Our hierarchical evaluation highlights the value of whole-body conditioning across short-term, long- horg-horizon, and atomic action tasks. While preliminary, our planning results demonstrate the potential for simulating embodied action consequences, moving toward more grounded models of perception and action.
10

---

Acknowledgment
The authors thank Rithwik Nukala for his help in annotating atomic actions. We thank Katerina Fragkiadaki, Philipp Krähenbühl, Bharath Hariharan, Guanya Shi, Shubham Tulsiani and Deva Ramanan for the useful suggestions and feedbacks for improving the paper; Jianbo Shi for the discussion regarding control theory; Yilun Du for the support on Diffusion Forcing; Brent Yi for his help in human motion related works and Alexei Efros for the discussion and debates regarding world models. This work is partially supported by the ONR MURI

---

References
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0353
2025.
Jake K Aggarwal and Quin Cai. Human motion analysis: A review. Computer vision and image understanding. 73(3):228 440, 1999.
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning poke by poking: Experiential learning of intuitive physics. Advances neural information processing systems, 29, 2016.
Eloi Alonso et al. Diamond: Diffusion as model of environment dreams. arXiv preprini 3.Xi::441.002444 2024.
Tenglong Ao, Zeyi 7hang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions Graphics (TOG), 42(4):1-18, 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable under- standing, prediction and planning. arXiv preprint wwXiv2550.099985 2025.
Amir Bar, Arya Bakhtiar, Danny Tran, Antonio Loquercio, Jathushan Rajasegaran, Yann LoCun, Amir Globerson, and Trevor Darrell. Egopet: Egomotion and interaction data from an animal's perspective. In European Conference on Computer Vision, pages 377-394. Springer, 2024.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15791-15801, 2025.
Angela Castillo, Maria Escobar, Guillaume Jeanneret, Albert Pumarola, Pablo Arbeláez, Ali Thahet, and Artsiom Sanakoyeu. Bediffusion: Diffusing sparse observations for full-body human motion synthesis. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 4221-4231, 2023.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing 221225, 2024.
Systems, 37:24081
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint urXiv:2402.16790 2024.
Kenneth JW Craik. The Nature of Explanation. Cambridge University Press, 1943.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mofusion: framework for denoising-diffin diffusion-bbssd motion synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9760-9770, 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic- kitchens dataset. In Proceedings of the European conference computer vision (ECCV), pages 720-736 2018.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 1rXiv:1812.00563 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEF, 2017.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. arXiv preprint arXiv: 2015.
Stephanic Fu, Netanel Tamir. Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.099: 2023.
Antonino Furnari and Giovanni Maria Farinella.
Istms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence, 43(11):4021-403 2020.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 13555- 2021.
12

---

Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
Kristen Grauman, Andrew Westhury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exood: Understanding skilled human activity from first-and third perspectives. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 19383-19400, 2024.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5152 5161, 2022.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint a8Xiv 10122, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations.
Danijar Hafner, Jurgis Pasukonis,, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint urXiv:2301.041000 2023.
Nicklas Hansen, Jyothir SV, Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models visual whole-body body humanoid controllers. arXiv preprint arXiv:2405. 18418, 2024.
Tairan Hc, Zhengyi Luo, Xialin Hc, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous uoman-to-humanodd whole-body telcoperation and learning. arXiv preprint ar Xiv:224006 08858, 2024a.
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi. Learning human-to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8944 8951. IEEE, 2024b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prohabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), pages 6840-6851, 2020.
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim. Yuting Ye, Richard Newcombe, Ziwei Liu, Lingni Ma.
Egolm: Multi-modal language model of egocentric motions. arXiv preprint arXiv:2409.181) 181227 2024.
Li Hu, Xin Gao, Peng Zhang. Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and
controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.11117 2023.
Michael I Jordan, Computational aspects of motor control and motor learning. Handbook of perception and action, 2:71-120, 1996.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Mitsuo Kawato. Internal models for motor control and trajectory planning. Current opinion in neurobiology. 9 (6):718-727, 1999.
Mitsuo Kawato. Kazunori Furukawa, and Ryoji Suzuki. A hierarchical neural-network. model for control and learning of voluntary movement. Biological cybernetics, 57:169-188 185, 1987.
Yann LeCun. path towards autonomous machine intelligence version 0.9. 2022-06-27. Open Review, 62(1): 1-62, 2022.
Jiaman Li, Karen Liu, Jiajun Wu, Ego-body pose estimation via ego-head pose estimation. CVPR, pages 17142-17151, 2023.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 112-123. PMLR, 2022.
Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and Yuke Zhu. Multi-task interactive robot flect learning with visual world models. arXiv preprint 22.2699, 2024.
13

---

Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundariex, Volume 2, pages 85 851-866. 2023.
Qianli Ma et al. Nymeria: massive collection of multimodal egocentric daily motion wild, arXiv preprint arXi::2406.099000 09905, 2024.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Oshert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint (rXiv:2210000030 2022.
Movella. MVN User Manual. Movella, 2021.
_Mauaal__aaaa__ pdf
Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjon Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1001-1010, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
J Bjorck Nvidia, Castaneda, Cherniadev, Da, R Ding, Fan, Y Fang. D Fox. Hu. Huang, et al. Gr00 nI: An open foundation model for generalist humanoid robots, arXiv preprint arXiv:2503. 14734, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages -205,, 2023.
Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10985-10995, 2021.
Ilija Radosavovie, Sarthak Kamat, Trevor Darrell, and Jitendra Malik. Learning humanoid locomotion over challenging terrain. arXiv preprint arXiv:2410.03654 2024.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings the IEEE/CVF international conference on computer vision, pages 11488-1 1499, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser. Björn Ommer. High-resolution image synthesis with latent diffusson models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
Bodo Rosenhahn, Reinhard Klette, and Dimitris Metaxas. Human motion. Understanding, Modeling, Capture, 2008.
Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89-112, 1997.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual. Oran Gafni, et al. Make-a-video; Text-to-vidco generation without text-video data. arXiv preprint arXiv:2209.1443 14792, 2022.
Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. 2024 IEEF International Conference on Robotics and Automation (ICRA), pages 63-70. IEEE, 2024.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573 580. IEEE, 2012.
Guy Tevet, Sigal Raab, Brian Gordon. Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.144 14916, 2022.
Hermann Von Helmholtz. Helmholtz's treatise on physiological optics. Optical Society of America, 1925.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5294-5306, 2025.
14

---

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to- video generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, fia Liu, Chenxu Zhang, Jiashi Feng. and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311 16498, 2023.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:23/0.06114, 1(2):6, 2023.
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Vidco as the new language for real-world decision making, arXiv preprint 2421:24022.11139 2024.
Vickie Ye. Georgios Pavlakos. Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In Proceedings of the IREF/CVF conference computer vision and pattern recognition, pages 21222-21232, 2023.
Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion an ego-sensed world. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7072-7084, 2025.
Ye Yuan and Kris Kitani. Ego-pose estimation and forccasting as real-time pd control. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pages 10082-10000 2019.
Ye Yuan, Jiaming Song. Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10082-10092, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- se--to-immee diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lci Yang, and Ziwei Liu. Motion- diffuse: Text-driven human motion generation with diffusion model. IEEE transactions on pattern analysis and machine intelligence, 46(6):41 15 15-4188, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018a. 8a.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018b.
Siwci Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Tacin Kwon, Mare Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. arXiv preprint arXiv:2204. 06953, 2022.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint 1..0022411.04983, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state that we trained model to perform whole- body-condittoned ego-centric video prediction, and evaluate it accordingly. The paper covers exactly those topics.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
16

---

Answer: [Yes]
Justification: All theorems, formulas, and proofs are numbered. Section mechanisms for action conditioned video generation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All model configurations and apper-prrametrss for training are detailed the main body. Fixed seeds are used for all runs.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use open datasets for evaluations which can be downloaded from their respective websites.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/uubll wwtpc//giiieeeeeeeeeeee hubbli//guddds//oeeeubbiiss SenieeSubmiss..e SttSSubbissoooooicccry. onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Dustification:Ther details are provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we report the standard error for all of our experimental results in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section and Section I1 our experiments
we detail the necessary GPUs needed to reproduce
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eeee.ss.ccppcppeeeeeee
Answer: [Yes]
Justification:
Our research conforms to the NeurlPS Code of Ethics completely.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has neither positive nor negative societal impacts as it is foundational research. There is no direct path any negative applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper presents foundational research and as such does present high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We appropriately cite the original papers for Nymeria.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, latasets.paperssit pwp.pppppwssic..
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper did not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification; This paper was not subject to IRB/HSR did not involve any crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only utilized for writing and editing. None of the core methods were derived from any LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
22

---

Supplementary Material
The structure of the Appendix is as follows: we start by providing additional details for our Camera Pose estimation evaluation and Wrist Position evaluation Section then provide additional ablation studies in Section 8 then some additional planning attempts with PEVA in Section 9 then include additional qualitative results in Section T0 and then include details about training and inference time in Section
Camera Pose Estimation and Wrist Position Evaluation
We assess how faithfully PEVA follows head and hand motion using two evaluations. Camera Pose Estimation: we extract egocentric camera poses from generated frames using VGGT (Wang et al. 2025) and compare them with ground truth using Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) for translation and rotation (Sturm et al. 2012). Wrist Pose Evaluation: we extract 2D wrist keypoints and compute PCKe0.2, precision, recall, and accuracy. These metrics jointly measure whether the generated videos accurately reflect the conditioned action signals, both globally (head-drivee motion) and locally (hand alignment).
8 Additional Abalation Studies
We conduct two additional ablation studies. First, we ablate our Whole-Body actions against actions that only include the Head and Hands in Section [811] Second, ablate Random Timeskips against fixed frame rates in Section 8.2 Third, we ablate the impact of our action conditioning in Section 8.3
8.1 Action Type Ablation
We conduct ablation studies the action type. We compare 72D) whole-body actions with only the 6-DoF poses of the head, left hand, and right hand relative to their previous states leaving dact 3
18 summarized in Table 4 for the camera pose and wrist position evaluation. Our Whole-Body actions perform better than only using the Head and Hands across almost all metrics. These results indicate that non-visible body joints, such as those in the torso and legs, meaningfully affect egocentric visual dynamics, such as through head stabilization, balance, and intent. We find that full- full-body conditioning improves both spatial grounding and long-term consistency.
Table 4: Action Type Ablations. We evaluate the impact of different Action Types single-step prediction performance (2 seconds into the future) on the camera pose and wrist positions.
Action Type
ATE
RPE (Translation)
RPE (Rotation). +
PCK
Precision f
Recall f
Accuracy
Head Hands
0.381
0.412
38.425
0.692
0.906
0.722
0.756
Whole-Body
0.351
0.346
26.578
0.877
0.890
0.907
0.858
8.2 Random Timeskip Ablation
We ablate our random timeskips against fixed frame rates summarized in Table 5 demonstrate that fixed-rate models tend to overfit to sepcific temporal patterns and degrade at longer horizons. In contrast, random timeskips encourage temporally invariant learning, improving both short and long term prediction quality.
The results
8.3 Influence of Action Conditioning on Visual Prediction
examine how the action signal influences vision prediction Figure 7 We find that the further we deviate from the ground-truth action, the further the visual prediction is from ground truth. This demonstrates the the model meaningfully conditions on the action for visual prediction, as opposed to simply relying solely on the visual context.
23

---

Table Random Timeskips Ablation. We evaluate the impact our Random Timeskips compared to various fixed frame rates measured on DreamSim J.
Method
1s
2s
4s
8s
16s
Avg
4 Hz
0.175
0.218
0.279
0.343
0.434
0.290
2 Hz
0.167
0.198
0.240
0.291
0.358
0.251
0.5 Hz
0.175
0.200
0.235
0.265
0.319
0.239
Random Skips
0.169
0.198
0.236
0.268
0.293
0.233
0.50
0.45
DreamSim
0.40
0.25
0.20
25
50
75
100
Action Noise Level (%)
Figure 7: Visual Quality Across Noise Levels. Evaluation of the visual quality as we add noise the the action. 0% is just the action whereas 100% is pure gaussian noise.
9 Some planning attempts with PEVA
Here we describe how to use a trained PEVA to plan action sequences to achieve a visual target. We formulate planning energy minimization problem and perform standalone planning in the same way as NWM (Bar et al. 2025) using the Cross-Fntroopy Method (CEM) (Rubinstein, 1997) besides minor modifications
representation and initialization of the action.
For simplicity, we conduct two experiments where we only predict moving either the left or right arm controlled by predicting the relative joint rotations represented as euler angles. For each re- spective arm we control only the shoulder, upper arm, forearm, and hand leaving our actions space 4 X 3 = 12 where have (A@shoulder: Abshoulder: D'shoulder .... A Ptoreamm: Aftorcamm A At/tercam)) We initialize mean (HAsibeeateer A HA Visunss: .... AA Mhteeem (4ammmm))) and variance 2 Sdhwem "0neemm 0 ('ommom as the mean and variance of the next action across the training dataset for these segments.
(o2 phamsstee 3. 0 O'
Table 6: Mean and Variance of relative rotation as euler angles (d), 0, vb) for arm segments computed across the training dataset.
Segment
Statistic
Right Arm
Left Arm
Mean
(0.0027,
0.0012,
-0.0015)
(0.0624, 0.0687, 0.1 0.1494) (0.0625, 0.0697, 0.1496)
Shoulder
Variance
(0.0010. - 0.0006, 0.0003)
Mean
(0.0107,
-0 -0.0011,
-0.0020)
(0.1119, 0.1647, 0.1791)
Upper Arm
Variance
0.0062.
-0.0001, -0.0013)
(0.0991, 0.1593, 0.1611)
Mean
(0.0068.
- -0.0035, 0.0077)
(0.1937, 0.2107, 0.2261)
Forearm
Variance
0.0036,
-0.0063, 0.0002)
(0.1791, 0.2012, 0.2186)
Mean
(0.0065.
0.0001,
0.004,)
(0.2417, 0.229, 0.2631)
Hand
Variance
0.0024,
-0.0032,
-0.0001)
(0.2126, 0.2237, 0.2475)
24

---

We assume the action is a straight continuous motion. Thus we repeat this action for our sequence length, in our case T 8 and optimize the delta actions. The time interval between steps is fixed at k 0.25 seconds. All other hyperparameters remain the same as in NWM (Bar et al. 2025)
9.1 Qualitative Results
Due to time constraints, we focus our investigation on arm movements argguabyy the most complex among body actions. While this remains an open problem, we present preliminary results using PEVA with CEM for standalone planning. This setting simplifies the nigh-dimensional control space while still capturing key challenges of full-body coordination.
Obsensatio
Prod
4
Figure In this case, we are able to predict sequence of actions that pulls our left arm in, similar to the goal.
Observation
Gool
GT
Hed AC:
Figure 9: In this case, we are able to predict sequence of actions that lowers our left arm, but not the same amount as the groundtruth sequence as we can see in the pose and hand the bottom of our rollout.
Observatio
Gool
G1
Hed AC:
Figure 10: In this case, we are able to predict sequence of actions that lowers our left arm that lowers the tissue. However, the goal image still has the tissue visible.
Hol
Figure 11: this case, we are able to predict a sequence of actions that raises our right arm to mixing stick. We see a limitation with our method as we only predict the right arm so we do not predict to move the left arm down accordingly.
10 More Qualitative Results
In this section, we present additional qualitative results following the same settings in the main paper. These results are organized into three parts: counterfactual simulations in Section TO. atomic action generation in Section 10.2 and long-horizon video generation in Section 10.3
25

---

Figure 12: In this case, we are able to predict a sequence of actions that moves our right arm toward the left but not quite enough. We see limitation with method as only predict the right arm so we do not predict any necessary additional body rotations.
Obsersation
Prod
Figure 13: In this case, we are able to predict sequence of actions that reaches toward the kettle but does not quite grab it as in the goal.
10.1 Additional Counterfactual Qualitative Results
In this section, include additional counterfactual qualitative results. First, demonstrate that model is able to capture the physical realism through a counterfactual example of continuing to pull on an open drawer and correctly follow real world physical constraints in Figure [4] Second, we include counterfactual examples of head and movement in Figure [5 Third, we include more
examples of using counterfactuals for visual planning in Figures T6 20
10.2 Additional Atomic Actions
We provide additional atomic action rollouts that are more interaction rich in Figure 21
10.3
Additional Long Video Generation
We provide additional examples of long video generation in Figures 22
34
11 Training and Inference Time
The model was trained for a total of 57.9 hours on 16 H100 nodes, each equipped with 8 GPUs. For inference, the average time per frame is 23728 + 207 ms, measured on a single A6000 GPU.
Figure 14: Physical Realism. When we repeatedly prompt the model to keep opening drawer, it correctly predicts that the drawer stays its fully opened position. This demonstrates an understand- ing of physical constraints.
26

---

Figure 15: Counterfactuals, We include counterfactuals on head movement and hand movement.
27

---

BIT 2
DOTT (1+
BKT: SwizI
BET: Je18s
Score:0.67
B.C. Dwwed
BAW% DHNaEK
Gool
Score: 0.510
BCT. 2
RLT. b6201
Score:
Figure Planning with Counterfactuals. PEVA allows us find a reasonable sequence actions to open the refrigerator in the third row. PEVA enables to rule out action sequences that grab the nearby plants and go to the kitchen and mix ingredients. PEVA allows us to choose the most correct action sequences that grab the box from the shelf.
(m-4)
(1-17)
(1-16)
Score: 0.564
Iepui
pon
pon G--81
Ben.
Score: 0.542
(t:88
s0n. t- 12)
pon. 1=166
Score: 0.413
Figure 17: Planning with Counterfactuals PEVA allows to rule out action sequences that lead to the light switch first row and the counter the second row. PEVA allows us find reasonable sequence of actions that opens the trash can and throws trash away in the third row.
28

---

Impur
gPe (mdd)
Oem Kio
gr 12
CMO (116)
Score: 0.585
pon: 22-41
pont (2-81
gon. (C- 12)
pon 16-161
Score: 0.617
Input
gen.
Score: 0.570
Figure 18: Planning with Counterfactuals. PEVA allows to rule out action sequences that lead to the counter in the first row and the kitchen utensils in the second row. PEVA allows us to find reasonable sequence of actions that opens the cabinet and reach in the third row.
Input
gon. (::4)
pon
gon. (t=12)
ANT. (1-16)
Score: 0.611
20-41
(2
Goal
Score: 0.528
Inpur
pen 20-41
gen.
pan- 16-161
Score: 0.651
Figure 19: Planning with Counterfactuals, PEVA allows rule action sequences that lead the reaching into the sink in the first row and opening the cabinet in the third row. PEVA allows us to find a reasonable sequence of actions that opens the fridge the second row.
29

---

Input
gen ba
pen (2:1))
eew. Ihe 12)
gen. (b:16)
Score: 0.686
p00% (2-44
(2--8
(2-12)
KOn. (b-16)
Goal
Score: 0.534
Ingurt
D00%
gen
8cn. 12-1
Score: 0.645
Figure 20: Planning with Counterfactuals, PEVA allows rule action sequences that lead the grabbing the curtains in the first row and grabbing the cabinet in the third row. PEVA allows us to find a reasonable sequence of actions that grabs the pillow the second row.
30

---

Open the
Open
Close
Close drawer
Reach out object
Reach out object
Walking downstairs
Walking downstairs
Open the door
Open the door
Getting peanut butter from the jar
Spread peanut butter on bread
Jumping
Pumping balloon
Standup from the ground
Folding clothes
Figure 21: Interactions. We include video generation of interaction rich actions specified by 3D-body poses.
31

---

Figure 22: Generation Over Long-Horizone We include 16-second video generation examples.
32

---

Figure 23: Generation Over Long-Horizons. We include 16-second video generation examples.
33

---

Figure 24: Generation Over Long- Horizoss. We include 16-second video generation examples.
34

---

Figure 25: Generation Over Horg-norizons. We include 16-second video generation examples.
35

---

Figure 26: Generation Over Horizoorizons. We include 16-second video generation examples.
36

---

Figure 27: Generation Over Long-Horizone We include 16-second video generation examples.
37

---

Figure 28: Generation Over Long-Horizone We include 16-second video generation examples.
38

---

Figure 29: Generation Over Long-Horizons. We include 16-second video generation examples.
39

---

Figure 30: Generation Over Long-norizons. We include 16-second video generation examples.
40

---

Figure 31: Generation Over Long-Horizons We include 16-second video generation examples.
41

---

Figure 32: Generation Over Long-Horizon We include 16-second video generation examples.
42

---

Figure 33: Generation Over Long-Horizon We include 16-second video generation examples.
43

---

Figure 34: Generation Over Long-norizons. We include 16-second video generation examples.
44


Reviews:
[{"review_id": "MBaqLQ8DlP", "reviewer": "Reviewer_6RQg", "summary": "This work proposes a novel model for generating egocentric videos conditioned on a past video segment and a sequence of human actions. The approach introduces random time skips, sequence-level training, and action embeddings to ensure the model captures both short-term motion dynamics and long-term motion semantics. Experiments demonstrate that the proposed PEVA model generates high-quality egocentric videos with full-body motion control.", "questions": "See weaknesses", "limitations": "This work presents a video diffusion model conditioned on motion. However, it does not account for actions involving the lower body (e.g., kicking a ball) or incorporate other control signals such as textual input.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well-written and effectively presents the details of the proposed model. The ablation study clearly highlights the contribution of each design choice.\n2. The experiments demonstrate that the model generates high-quality egocentric videos, outperforming baselines when conditioned on both atomic actions and long-term motion sequences.\n\nWeaknesses\n\n1. The proposed model builds upon existing diffusion-based methods by incorporating additional control signals, which limits the novelty of the contribution.\n2. The role and importance of the random time skips in the model are not clearly justified in the ablation study.\n3. There is no visualization of generated videos in the supplementary materials which would help illustrate the model.", "was_revised": true, "final_justification": "The rebuttal satisfactorily addressed my concerns about this paper. I would like to maintain my rating of accept.", "rebuttal": "We thank the reviewer for the constructive feedback and for acknowledging the clarity of our model presentation and experiments. We appreciate the suggestions and have addressed the raised concerns as follows:\n### **1. Novelty of the Contribution**\n\nIn this work we address a new and underexplored problem: **how embodied physical actions causally shape first-person visual experience over time**. This problem is challenging, and so far has only been tackled with either simple action space (e.g., navigation) or in synthetic environments (e.g., computer games).\n\nTo address this, we proposed **PEVA**, a novel autoregressive video model that generalizes to variable context lengths and can condition on physical actions. PEVA outperforms recent state-of-the-art action-conditioned diffusion models like **NWM** and **DF** across perceptual metrics such as **LPIPS** and **DreamSim**. Additionally, in a hand-controlled motion task, PEVA achieves **85% PCK\\@0.1**, compared to only **22%** for the text-based model **Cosmos**, highlighting its ability to capture precise low-level control that language struggles to specify.\n\nUnlike prior diffusion-based methods, which typically use flat or semantically abstract conditioning (e.g., text, labels, global motion), PEVA leverages **hierarchically structured full-body kinematic trajectories**—aligned with the human joint graph. This enables visual prediction driven by specific embodied actions (e.g., head tilts, shoulder rolls, fine arm motions), and supports:\n\n* Disentangled action-perception analysis\n* Atomic counterfactuals\n* Goal-conditioned planning in pixel space\n\nThe novelty lies not only in the modeling technique but in **proposing and solving a new class of controllable, embodied egocentric prediction problems**.\n\n---\n\n### **2. Justification for Random Time Skips**\n\nWe thank the reviewer for highlighting the need for clarification.\n\nTo evaluate the importance of random time skips, we conduct an ablation comparing models trained with fixed frame rates (**0.5 Hz, 1 Hz, 2 Hz, 4 Hz**) to our approach that was trained using **random time skips**. All models are evaluated on 1-step predictions at 1, 2, 4, 8, and 16 seconds using **DreamSim perceptual loss**.\n\n| **Method**            | **1s**    | **2s**    | **4s**    | **8s**    | **16s**   | **Avg**   |\n| --------------------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 4 Hz (fixed)          | 0.175     | 0.218     | 0.279     | 0.343     | 0.434     | 0.290     |\n| 2 Hz (fixed)          | 0.167     | 0.198     | 0.240     | 0.291     | 0.358     | 0.251     |\n| 0.5 Hz (fixed)        | 0.175     | 0.200     | 0.235     | 0.265     | 0.319     | 0.239     |\n| **Random Time Skips** | **0.169** | **0.198** | **0.236** | **0.268** | **0.293** | **0.233** |\n\nFixed-rate training often **overfits to specific temporal dynamics** and degrades at longer horizons. In contrast, **random time skips encourage learning temporally invariant dynamics** and improve generalization across both short- and long-term predictions.\n\nWe will clarify this motivation and experimental design more explicitly in the revised paper.\n\n---\n\n### **3. Visualization of Generated Videos**\n\nDue to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included **extensive visualizations in the supplementary material**—such as sampled frames, temporal rollouts, and attention maps—to illustrate the model’s behavior.\n\nWe hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the **camera-ready version**, and we are happy to provide additional video results in an **online repository** upon request if there is a change in the conference guidelines."}, {"review_id": "uUZCM0qdsb", "reviewer": "Reviewer_t9bj", "summary": "This paper proposes an interesting task that predicts future egocentric frames conditioned on human body trajectory. Specifically, conditioned on previous video frames and an action (representing human upper-body pose and root translation) for the current step, they trained a diffusion model to predict the video frame after executing the action. They showcased better results compared to baselines and also showed the potential of using the proposed model for planning tasks given a goal image.", "questions": "As described in the weakness part, there are some concerns about the current evaluation both quantitatively and qualitatively. In addition, it would be better to discuss whether 3-point poses (head+hand) are sufficient instead of full-body poses, clarify the confusion about the evaluation metrics and method description.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The task is interesting and has the potential to help with planning tasks. \n- The paper showcased promising results of generating future video frames conditioned on past frames and action trajectories. \n- They conducted various ablations to demonstrate the effectiveness of their proposed setting. \n- They also showed some preliminary results of using the proposed model for action planning given a goal image. \n\nWeakness: \nThere are some concerns about the current evaluation and presentation. \n- The paper focused on using full-body pose as an action condition to generate future video frames. How accurate is the predicted future frame? Could there be some metrics for evaluating the accuracy of following actions beyond image space evaluation? For example, if it’s generating a frame with visible hand, maybe there could be a metric that evaluates the wrist position error in 2D (extract 2D keypoints from generated frame and compare with GT frame)? And maybe can also do camera pose estimation using the generated video frames and compare with GT camera pose (camera pose should be directly related to head pose)?   \n- Lack of more qualitative results. It would be better to have a supplementary video to showcase the generated video frames conditioned on the full-body action conditions (for example, show two videos simultaneously, left shows full-body human pose, right side shows the generated egocentric video).  \n- Another question is that for generating egocentric video, it seems head pose and hand pose are enough? Head pose is directly related to the camera pose of egocentric video, and thus is important to have. Hand poses are also important since sometimes the hands are visible in the egocentric videos. However, it’s not clear whether we need other joints’ poses for egocentric video generation. The current submission lacks discussion about this.  \n- Some descriptions and presentations in the paper are not clear. For example, in Table 2, what’s the evaluation metric here? In section 3.3, what does random timeskips mean?", "was_revised": true, "final_justification": "The rebuttal has provided more evaluations for hand pose accuracy and camera pose accuracy, and ablations of using 3-point input only. It addressed my major concern. I decide to raise my score.", "rebuttal": "We sincerely thank the reviewer for their detailed and insightful review. Your feedback is incredibly valuable. Based on your review, we have conducted additional experiments for clarity. We discuss each point in depth below.\n\n### **1. Evaluation Beyond Image Space: Measuring Action-Following Accuracy**\n\nWe appreciate the reviewer’s suggestion to evaluate action-following accuracy beyond image-level metrics. In response, we introduce two additional evaluations:\n\n* **Wrist Position Evaluation**\n  We compute **PCK\\@0.2**, precision, recall, and accuracy on 2D wrist keypoints across generated frames to measure local hand alignment.\n\n* **Pose Estimation Evaluation**\n  We estimate egocentric camera pose using **VGG-T** \\[2] and compare it to ground truth using **ATE**, **RPE (Translation)**, and **RPE (Rotation)** \\[1].\n\n\nThese metrics directly evaluate whether the generated video faithfully reflects the conditioned action signal—both **locally** (e.g., visible hand positions) and **globally** (e.g., head-driven camera motion). **PEVA** outperforms all baselines, demonstrating its superior capacity for tracking embodied actions.\n\n#### **Wrist Position Evaluation across Models**\n\n| Model           | PCK\\@0.2  | Accuracy  | Precision | Recall    |\n| --------------- | --------- | --------- | --------- | --------- |\n| DF              | 0.750     | 0.679     | 0.914     | 0.592     |\n| CDiT            | 0.755     | 0.794     | 0.865     | 0.833     |\n| **PEVA (Ours)** | **0.791** | **0.871** | **0.923** | **0.888** |\n\n#### **Camera Pose Evaluation across Models**\n\n| Model           | ATE ↓     | RPE (Translation) ↓ | RPE (Rotation) ↓ |\n| --------------- | --------- | ------------------- | ---------------- |\n| DF              | 0.464     | 0.422               | 36.514           |\n| CDiT            | 0.386     | 0.367               | 26.057           |\n| **PEVA (Ours)** | **0.274** | **0.266**           | **13.527**       |\n\n---\n\n### **2. Necessity of Whole-Body Conditioning**\n\nWe thank the reviewer for raising this important question. To investigate whether head and hand pose alone are sufficient, we conduct an ablation study using only the 6-DoF poses of the head, left hand, and right hand (relative to their previous states).\n\nAlthough this subset captures visible body parts and head orientation, it leads to substantial degradation in both wrist and pose evaluations—highlighting the importance of full-body context.\n\n#### **Wrist Position Evaluation by Conditioning Type**\n\n| Conditioning  | PCK\\@0.2  | Accuracy  | Precision | Recall    |\n| ------------- | --------- | --------- | --------- | --------- |\n| Head + Hands  | 0.692     | 0.756     | **0.906** | 0.722     |\n| **Full Body** | **0.877** | **0.858** | 0.890     | **0.907** |\n\n#### **Camera Pose Evaluation by Conditioning Type**\n\n| Conditioning  | ATE ↓     | RPE (Translation) ↓ | RPE (Rotation) ↓ |\n| ------------- | --------- | ------------------- | ---------------- |\n| Head + Hands  | 0.381     | 0.412               | 38.425           |\n| **Full Body** | **0.351** | **0.346**           | **26.578**       |\n\nThese results indicate that **non-visible body joints**, such as those in the torso and legs, meaningfully affect egocentric visual dynamics—e.g., through head stabilization, balance, and intent. Full-body conditioning improves both **spatial grounding** and **long-term consistency**.\n\n---\n\n### **3. Lack of Qualitative Results / Videos**\n\nDue to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included extensive visualizations in the supplementary material—such as sampled frames, temporal rollouts, and attention maps —to illustrate the model’s behavior. We hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the camera-ready version and we are happy to provide additional video results in an online repository upon request if there is a change in the conference guidelines.\n\n---\n\n### **4. Clarity of Metrics and Terminology (e.g., Table 2 and Random Time Skips)**\n\nThank you for this valuable feedback.\n\n* **Table 2**: The evaluation metric is **LPIPS** (Learned Perceptual Image Patch Similarity), which measures perceptual distance between generated and ground truth frames. We will clarify this in the table caption and main text.\n\n* **Random Time Skips**: Instead of training with a fixed timestep (e.g., every 0.5 s), we **randomly sample time intervals** during training and condition the model on the time delta. This improves generalization across short- and long-term predictions.\n\nTo assess the effect of random time skips, we compare models trained at fixed frame rates (0.5 Hz, 1 Hz, 2 Hz, 4 Hz) against one trained with random time skips. We evaluate 1-step predictions at 1, 2, 4, 8, and 16 seconds using **DreamSim** perceptual loss.\n\n#### **Ablation: Random Time Skips vs Fixed Rate (↓ DreamSim)**\n\n| Method           | 1s        | 2s        | 4s        | 8s        | 16s       | Avg       |\n| ---------------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 4 Hz (fixed)     | 0.175     | 0.218     | 0.279     | 0.343     | 0.434     | 0.290     |\n| 2 Hz (fixed)     | 0.167     | 0.198     | 0.240     | 0.291     | 0.358     | 0.251     |\n| 0.5 Hz (fixed)   | 0.175     | 0.200     | 0.235     | 0.265     | 0.319     | 0.239     |\n| **Random Skips** | **0.169** | **0.198** | **0.236** | **0.268** | **0.293** | **0.233** |\n\nFixed-rate models tend to overfit to specific temporal patterns and degrade at longer horizons. In contrast, **random time skips** encourage **temporally invariant learning**, improving both **short- and long-term prediction quality**.\n\nWe will clarify this motivation and experiment in the revised paper.\n\n> \\[1] Sturm et al. *\"A benchmark for the evaluation of RGB-D SLAM systems.\"* IROS 2012\n\n> \\[2] Wang et al. *\"VGG-T: Visual Geometry Grounded Transformer.\"* CVPR 2025"}, {"review_id": "CxY2BteZHb", "reviewer": "Reviewer_pyh8", "summary": "This paper proposes a novel autoregressive conditional diffusion transformer model for egocentric video prediction conditioned on whole-body human motion. The authors design a structured 48-dimensional action representation encoding pelvis translation and upper-body joint rotations, capturing the kinematic hierarchy of the human body. The model is trained and evaluated on the Nymeria dataset, which contains synchronized first-person videos and precise 3D motion capture data. A hierarchical evaluation framework is introduced to assess short-term perceptual quality, long-term semantic consistency, and performance in control/planning tasks. This work advances embodied AI by linking detailed physical human motion with first-person visual predictions, showing promising results in a challenging real-world setting.", "questions": "1. Please provide quantitative results evaluating the model on unseen individuals with different body shapes or motion styles. For example, a cross-subject split in Nymeria with metrics such as FID or LPIPS would be valuable.\n2. Report the total training time, hardware used (GPU type/number), and average inference time per frame (in milliseconds). This information is crucial for assessing real-time applicability.\n3. The current 48D vector excludes lower-body joints. What was the rationale behind this choice? Did you conduct ablation studies comparing upper-body only vs. full-body conditioning?\n\n4. Although some failure cases are presented in the supplementary materials, could you please elaborate on the main challenges or issues that cause the current model to fail? Additionally, the main text mentions that the quality of long-term predictions deteriorates as the prediction horizon extends. Besides the inherent uncertainty of the future, could you explain what other factors contribute to the difficulty of long-term prediction? Finally, what potential directions or strategies could be pursued to improve long-term prediction performance in the future?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths \n1. The use of Nymeria, a large-scale dataset with synchronized egocentric video and 3D pose data, grounds the work in real-world complexity.\n\n2. The hierarchical encoding of full-body motion is well motivated and enables fine-grained conditioning of video prediction.\n\n3. The hierarchical evaluation protocol covers multiple aspects of prediction quality, including downstream control tasks, which is a strong point.\n\n4. Introducing whole-body conditioned egocentric video prediction with a diffusion transformer is a fresh and interesting direction.\n\n# Weaknesses\n1.  The paper lacks experiments testing generalization across unseen subjects or environments, which is critical for embodied video prediction. \n2.  No quantitative details on training or inference time are provided, making it difficult to assess effciency.\n\n3. The choice to exclude lower-body joints is not justified with experiments or analysis.", "was_revised": true, "final_justification": "This paper is well written and introduces a novel model for the egocentric prediction task that is very hard. By incorporating human body joints, the prediction tasks become more realistic and practically useful. I believe this represents a meaningful and promising direction for future research.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback, which helped us significantly improve the clarity and depth of our work.\n\n### **1. Generalization to Unseen Subjects or Environments**\n\nWe thank the reviewers for highlighting this important issue. We fully agree that evaluating generalization across subjects and environments is essential for embodied video prediction.\n\nWhile our main experiments focus on within-subject generalization on **Nymeria**, we took several steps to ensure meaningful diversity in motion and scenario distribution. First, our train/eval split is designed to cover distinct sets of action episodes. The evaluation set contains many tasks and motion patterns not seen during training, including interactions with objects (e.g., balloons, boxes, mats), fine-grained hand manipulations, and multi-stage sequences.\n\nFor example, the evaluation video `20231214_s0_jeremy_allen_act5_m10nnd` contains a full sequence of making a balloon animal, which involves a series of fine-grained, compositional actions that are not seen during training. These include:\n\n* **3.35–3.47**: pick and place, arranging materials\n* **4.59–5.02**: pumping air into the balloon\n* **5.25–5.28**: tying the balloon, stretching it\n* **6.09–6.15**: folding and twisting the balloon with both hands\n* **9.01–9.04**: additional pumping in a new context\n\nThese action combinations and their temporal structure are not present in training, yet the model generates coherent predictions—demonstrating generalization to unseen, complex, and multi-step interactions.\n\nThat said, while we believe the diversity in Nymeria is meaningful, it is inherently difficult to quantify how diverse is “diverse enough.” To further ensure generalization and provide a more systematic evaluation, we conduct **cross-subject experiments on EgoBody** \\[1], which features identities, body shapes, action styles, motion dynamics, and indoor environments that are entirely unseen during training. We evaluate **PEVA and baselines** on this dataset with single-step prediction, with completely novel scenes and activities. \n\n| Model    | LPIPS ↓   | DreamSim ↓ | FID ↓       |\n| -------- | --------- | ---------- | ----------- |\n| DF-XL    | 0.525     | 0.457      | 231.355     |\n| CDiT-XL  | 0.667     | 0.517      | 305.237     |\n| PEVA-XL  | **0.525** | **0.445**  | **210.070** |\n| PEVA-XXL | 0.531     | **0.422**  | **198.192** |\n\nPEVA-XL outperforms both DF-XL and CDiT-XL, matching LPIPS while achieving significantly better DreamSim and FID. The XXL variant further improves performance on DreamSim and FID.\n\nThese results suggest that PEVA produces **semantically coherent and perceptually realistic future predictions**, even under substantial distribution shift across subjects and environments.  \n\nDue to Program Chairs' guidelines, we are unable to include generated videos in the rebuttal, but we will include more visualizations of generated videos in the camera-ready version.\n\n---\n\n### **2. Use of Lower-Body Joints**\n\nThank you for pointing this out. The model does in fact use full-body joints, including the lower body. The confusion stems from an error in how the degrees of freedom (DoF) were reported—we mistakenly excluded the contribution of lower-body joints in the DoF count. We will correct and clarify this in the revised manuscript. **No joints were excluded** in training or inference.\n\n---\n\n### **3. Efficiency: Training and Inference Time**\n\nWe appreciate the request for computational details. The model was trained for a total of **57.9 hours on 16 H100 nodes**, each equipped with 8 GPUs. For inference, the **average time per frame is 23728 ms ± 207 ms**, measured on a single A6000 GPU. We will include these numbers in the final version to clarify the model’s efficiency and real-time applicability.\n\n---\n\n### **4. Main Failure Modes, Challenges in Long-Term Prediction, and Future Directions**\n\nThe model tends to fail in scenarios involving **ambiguous egocentric motion**, such as rapid head turns with no clear landmarks or **asynchronous movement of body parts**. These cases challenge the model’s ability to temporally disambiguate motion and infer global structure from a first-person view.\n\nMoreover, **long-horizon degradation** is driven by error accumulation: small inaccuracies in early timesteps compound over time, leading to drift or overshooting. This is a common limitation of open-loop generative models. While some of this degradation is due to future uncertainty, it also reflects the lack of **corrective feedback** during inference.\n\nIn future work, we plan to address this via training in **interactive environments** that allow feedback, or by incorporating **goal-conditioned** or **text-conditioned supervision** to help anchor predictions over extended rollouts.\n\n---\n\n### **5. Qualitative Visualization**\n\nDue to Program Chairs guidelines, we are unable to include generated videos in the rebuttal. However, we have included extensive visualizations in the supplementary material—such as sampled frames, temporal rollouts, and attention maps —to illustrate the model’s behavior. We hope these help convey the qualitative aspects of the generated sequences. We will include more visualizations of generated videos in the camera-ready version and we are happy to provide additional video results in an online repository upon request if there is a change in the conference guidelines.\n\n---\n\n> \\[1] Zhang, Siwei, et al. *Egobody: Human body shape and motion of interacting people from head-mounted devices*. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022."}, {"review_id": "fV7ss1k3fu", "reviewer": "Reviewer_XZaV", "summary": "This paper presents PEVA, a method for predicting future egocentric frames based on a sequence of prefix frames and corresponding body motion. PEVA incorporates several novel design choices, including Random Time Skips and Auto-Regressive Training, to enhance temporal modeling. The method is evaluated across diverse scenarios—such as atomic actions, counterfactual reasoning, and long-horizon video prediction—demonstrating its effectiveness across multiple applications.", "questions": "Please refer to weaknesses.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. This paper is the first to generate future egocentric frames conditioned solely on body motion, which is novel and valuable for the field.\n2. As explained in Sec.1, using body motions to predict future frames makes a lot of sense.\n3. The paper presents thorough experiments covering atomic actions, counterfactual scenarios, and long video sequences. The prediction results are generally convincing.\n4. The paper is well written and easy to follow, with clear organization and presentation.\n\nWeaknesses:\n1. It remains unclear how PEVA compares to text-conditioned video generation methods. The lack of comparison with such baselines leaves open the question of whether motion-based conditioning provides unique advantages.\n2. According to Tables 1 and 2, the performance improvement over DF and CDiT is marginal (approximately 0.01), which may not be statistically significant or practically meaningful.\n3. The distinction between the SMPL-X representation and the representation described in Section 3.1 is unclear. Both appear to encode relative joint rotations based on the kinematic tree, and the paper should clarify what differentiates them.", "was_revised": true, "final_justification": "Thanks for the responses. Overall, the authors have adequately addressed my main concerns. After reviewing the rebuttal and considering the comments from the other reviewers, I have decided to maintain my original rating.", "rebuttal": "We thank the reviewer for the clear and constructive comments, and appreciate the recognition of both the novelty of our approach and the breadth of our experiments. We appreciate the suggestions to improve the paper and we address these in depth below:\n### **1. Comparison to Text-Conditioned Video Generation Methods**\n\nThank you for the thoughtful comment. Our goal with **PEVA** is to explore action-conditioned egocentric video generation using physically grounded full-body motion actions that are given at every timestep. This is fundamentally different from text-conditioned methods such as **Cosmos** \\[1], which represent actions using text (e.g., “walk forward”) and thus lack precise and continuous physical control of the generated video.\n\nTo further validate the benefits of motion-based control, we conducted a comparison with Cosmos on a controlled hand-motion subset. We moved the hand in defined directions (e.g., “move the hand to top-right”, “move the hand to bottom-right”) and evaluated the alignment between intended and generated motions using **PCK\\@0.1**. PEVA achieves **85%**, while Cosmos reaches only **22%**.\n\nWhile text prompts are intuitive and flexible, they often lack the precision required for fine-grained, spatially grounded motion control. Our experiments suggest that translating subtle physical intentions into language is inherently ambiguous—especially when pixel-level accuracy is needed. Expressing such intent in natural language would require awkward approximations (e.g., “move the hand slightly upward at a 73-degree angle”), which are rarely interpretable or faithfully executable by text-conditioned models. This likely contributes to Cosmos's lower performance on tasks requiring precise action execution.\n\nWhile we are unable to include the visualizations at this stage, we will provide full results and qualitative comparisons in the final version. These preliminary findings suggest that **PEVA offers significantly more accurate and controllable motion grounding than text-conditioned alternatives**.\n\n---\n\n### **2. Performance Improvement Over Baselines (DF, CDiT)**\n\nWe respectfully disagree with the claim that the improvement is marginal. While LPIPS gains may appear small numerically, they are consistent across all horizons and paired with significantly better controllability.\n\nBelow is a comparison of 1-step **DreamSim perceptual loss** over various prediction horizons:\n\n#### **DreamSim 1-step Perceptual Loss Across Time Horizons**\n\n| Model           | 1s        | 2s        | 4s        | 8s        | 16s       | Avg       |\n| --------------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| DF              | 0.750     | 0.791     | 0.821     | 0.856     | 0.902     | 0.824     |\n| CDiT            | 0.755     | 0.794     | 0.812     | 0.840     | 0.881     | 0.816     |\n| **PEVA (Ours)** | **0.741** | **0.774** | **0.799** | **0.828** | **0.864** | **0.801** |\n\n---\n\n### **3. Clarification on SMPL-X vs. Our Representation**\n\nThank you for the request for clarification.\n\nOur motion representation is based on the **Xsens skeleton** \\[2]\\[3], which shares the kinematic tree structure with SMPL but differs in joint set, ordering, and lacks body shape parameters. **SMPL-X** extends SMPL with hand and facial joints and facial expression parameters, while our representation is a flattened **72D vector of joint angles and root translation**, without any mesh or expression components. We will revise Section 3.1 to clarify this distinction.\n\n---\n\n> \\[1] Agarwal, Niket, et al. *\"Cosmos world foundation model platform for physical AI.\"* arXiv preprint arXiv:2501.03575 (2025).\n\n> \\[2] Movella Xsens MVN Link Motion Capture, [https://www.movella.com/products/motion-capture/xsens-mvn-link](https://www.movella.com/products/motion-capture/xsens-mvn-link)\n\n> \\[3] Ma, Lingni, et al. *\"Nymeria: A massive collection of multimodal egocentric daily motion in the wild.\"* European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024."}]

Author rebuttal:

Meta-review: This paper introduces PEVA, a model designed to predict future ego-centric video conditioned on past video and the human agent's actions, which are represented by relative 3D body pose trajectories. The authors also contribute a hierarchical evaluation protocol to systematically assess the model's capabilities in embodied prediction and control.

The initial reviews for this paper were mixed, with some reviewers raising points that required further clarification. However, the authors provided a comprehensive rebuttal that effectively addressed the concerns. Following the author's response and subsequent discussion, all reviewers reached a consensus and are now in favor of accepting the paper.

I concur with the reviewers' final recommendation. This is an interesting and well-executed work that makes a valuable contribution toward the ambitious goal of creating ego-centric world models. The proposed method is sound and will likely benefit the community. The authors are encouraged to revise the final version of their manuscript to incorporate the suggestions raised by the reviewers and reflect the valuable points from the discussion period.
Decision: Accept (poster)