EgoTwin: Dreaming Body and View in First Person
Authors: Jingqiao Xiu
Fangzhou Hong
Yicong Li
Mengze Li
Wentao Wang
Sirui Han
Liang Pan
Ziwei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Egocentric Vision
Primary area: generative models
TL;DR: We propose EgoTwin, a diffusion-based framework that jointly generates egocentric video and human motion in a viewpoint consistent and causally coherent manner.

Abstract: While exocentric video synthesis has achieved great progress, egocentric video generation remains largely underexplored, which requires modeling first-person view content along with camera motion patterns induced by the wearer's body movements. To bridge this gap, we introduce a novel task of joint egocentric video and human motion generation, characterized by two key challenges: 1) Viewpoint Alignment: the camera trajectory in the generated video must accurately align with the head trajectory derived from human motion; 2) Causal Interplay: the synthesized human motion must causally align with the observed visual dynamics across adjacent video frames. To address these challenges, we propose EgoTwin, a joint video-motion generation framework built on the diffusion transformer architecture. Specifically, EgoTwin introduces a head-centric motion representation that anchors the human motion to the head joint and incorporates a cybernetics-inspired interaction mechanism that explicitly captures the causal interplay between video and motion within attention operations. For comprehensive evaluation, we curate a large-scale real-world dataset of synchronized text-video-motion triplets and design novel metrics to assess video-motion consistency. Extensive experiments demonstrate the effectiveness of the EgoTwin framework. Qualitative results are available on our project page: https://egotwin.pages.dev/.

Published as a conference paper at ICLR 2026
EGOTWIN: DREAMING BODY AND VIEW IN FIRST PERSON
Jingqiao Xiu-
Fangzhou Hong" Yicong Li Mengze Li
Wentao Wang
Sirui Han Liang Pan 1+
Ziwei Liu
National University of Singapore Nanyang Technological University
HHong Kong University of Science and Technology "Shanghai Al Laboratory
Reconstructed Scene
Initial View
Finter the room, open Deseription
Ego
Twin
Initial
Jointly Generated Egocentric Views & Human Poses
Figure 1: We propose EgoTwin, diffusion-baan -aasd framework that jointly generates egocentric video and human motion in a viewpoint consistent and causally coherent manner. Generated videos can be lifted into 3D scenes using camera poses derived from human motion via 3D Gaussian Splatting.
ABSTRACT
While exocentric video synthesis has achieved great progress, egocentric video generation remains largely underexplored, which requires modeling first-person view content along with camera motion patterns induced by the wearer's body movements. To bridge this gap, we introduce novel task of joint egocentric video and human motion generation, characterized by two key challenges: Viewpoint Alignment: the camera trajectory in the generated video must accurately align with the head trajectory derived from human motion; 2) Causal Interplay: the synthesized human motion must causally align with the observed visual dynamics across adjacent video frames. To address these challenges. we propose EgoTwwn, joint vidco-motion generation framework built the diffusion transformer ar- chitecture. Specifically, EgoTwin introduces head-centric motion representation that anchors the human motion to the head joint and incorporates cybernetics- inspired interaction mechanism that explicitly captures the causal interplay be- tween video and motion within attention operations. For comprehensive evalua- tion, we curate large-scale real-world dataset of synchronized text- text-video- video-motion triplets and design novel metrics to assess video-motion consistency. Extensive experiments demonstrate the effectiveness of the Tgg framework. Qualitative results are available on our project page:
pages.dess dev/
INTRODUCTION
Recent advances in deep generative models have delivered remarkable progress in exocentric (third-person video generation (Blattmann et al. 2023a Brooks et al., 2024 Yang et al. 2025). capable of producing photorealistic and temporally consistent videos conditioned various modal-
ities, with broad implications for multimodal learning (Xiu et al.) 2024 2025a b 2026). However,
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
egocentric (first-person) video synthesis remains largely underexplored. despite its increasing im- portance for wearable computing (Fiannaca et al. 2014), augmented reality (Ashtari et al. (020), and embodied agents (Nair et al. [022). In contrast to exocentric setups, where the camera static or externally controlled (Wang et al. 2224bb He et al. 2025), egocentric video captures the perspec- tive of moving individual, with the footage inherently entangled with the camera wearer's motion. In particular, head movements influence the camera's 's position and orientation, while full-body ac- tions affect the wearer's body pose and the surrounding scene, collectively shaping the egocentric recording. Therefore, to model body-dna body-driven dynamics in egocentric views, we argue that the visual stream must be generated in lockstep with the motion stream that drives it.
In this paper, we introduce a novel task of joint cedeo-motion generation that explicitly models ego- centric video together with the motion of the camera wearer. As illustrated in Figure T given a static human pose and an initial scene observation, our goal is to generate synchronized sequences of egocentric video and human motion, guided by the textual description. This task introduces two fundamental challenges beyond prior works: (1) Viewpoint Alignment. Throughout the sequence, the camera trajectory captured in egocentric video must precisely align with the head trajectory derived from human motion. This requirement naturally stems from the fact that the camera is rigidly mounted on the wearer's head (Engel et al., |023] Apple Inc. (203), causing head movement and camera motion to be tightly coupled. However, existing exocentric video generation methods typically employ a unidirectional niewpoint-conditionng strategy that synthesizes video based on predefined camera poses (Wang et al. 2024b; He et al. 2025). Such approaches are unsuitable for our setting, as the camera poses in egocentric video are not externally provided but are inher- ently determined by the wearer's head motion. As result, the camera poses must be generated concurrently with the human motion, necessitating a bidirectional interaction to ensure viewpoint alignment. (2) Causal Interplay. At each time step, the current visual frame provides spatial context that shapes human motion synthesis; conversely, the newly generated motion influences subsequent video frames. Take the "opening door" scenario in Figure [ as an example: egocentric observation informs the wearer of the door's location, which guides the wearer's 's action. In turn, the performed action can alter the body pose (c.g.. reaching for the doorknob). the camera pose (c.g., orienting to- ward the door), and the surrounding scene (e.g., the door gradually opening). These changes must be accurately reflected in subsequent video frames, thereby affecting future motion generation. This re- cursive dependency forms closed observation action loop between video and motion, highlighting the necessity of modeling their causal interplay over time.
To address these challenges, we propose Tin, a joint video-motion generation framework that generates egocentric videos with body-induced camera motion patterns while capturing the causal interplay between visual observations and human actions. Specifically. EgoTwin adopts a diffu- sion transformer backbone (Peebles& & Xie, 2023] Esser et al. 2024). with three modality-specific branches for text, video, and motion, respectively. To model the joint distribution, EgoTwin employs asynchronous diffusion in video and motion branches, which allows each modality to evolve on its timestep while maintaining cross-modal interaction. To facilitate accurate viewpoint alignment, we depart from the commonly used root-centric motion representation (Guo et al. 2022a), which ob- scures head pose within full-body motion and thus fails to expose the egocentric perspective to the video branch. Instead, we introduce head-centric motion representation that anchors the human motion to the head joint, allowing for direct alignment between the camera viewpoint of the gener- ated video and the head pose in the synthesized motion. To faithfully capture the causal interplay, we draw inspiration from the observation-
feedback loop in cybernetic systems (Agrawal et al.) 2016 Pathak et al., 2017). where observations shape actions and actions alter future observations. We implement this principle through a structured interaction mechanism: each video token attends to preceding motion tokens, capturing how current observations arise from past actions, while each motion token attends to current and upcoming video tokens, enabling the inference of actions based on perceived scene transitions. This bidirectional design allows motion-driven video synthesis and video-informed motion synthesis to evolve in synchrony.
To foster research in this field, we curate a large-scal dataset of real-world egocentric videos with human pose annotations from Nymeria (Ma et al.] (024). For evaluation, we extend beyond the individual quality of video and motion, and propose video-motion consistency metrics that quantify their cross-modal alignment. Extensive experiments demonstrate the effectiveness of Tgin
In summary, our contributions are fourfold:
2

---

Published as a conference paper at ICLR 2026
To the best of our knowledge, we are the first to explore the joint generation of egocentric video and human motion in a viewpoint consistent and causally coherent manner.
We identify the limitations of conventional root-centric motion representations in egocentric con- texts and reformulate head-centric approach that facilitates video-motion alignment.
We design triple-branch diffusion transformer featuring a video-motion interaction mechanism, supported by efficient three-stage training paradigm and versatile sampling strategies.
We propose video-motion consistency metrics and build a benchmark for evaluating joint video- motion generation, where our EgoTTwi demonstrates strong performance.
2 RELATED WORK
Video Generation. Video generation has witnessed significant advancements with the emergence of video diffusion models (Ho et al. 2020 Karras et al., 2022 Ho et al. 2022). A central research focus has been on text-to-video (T2V) generation and image-to-video (12V) generation, where mod- els synthesize coherent video sequences from textual prompts or static images. Early approaches (Blattmann et al., 2223a b) augment UNet-based text-to-image (T2I) models (Rombach et (022)) with temporal modeling layers to efficiently transform them to video generation models. Recent works (Brooks et al. 2024 Yang et al.) 2025) adopt cransformerbbasd architectures (Peebles & Xie 2023). achieving improved temporal consistency and generation quality. To incorporate camera control, representative methods (Wang et al. (0244)] He et al. (2025) inject camera parameters (c.g., extrinsic matrices or Plücker embeddings (Sitzmann et al.] 2021)) into pretrained video diffusion models (Xing et al. 2024] Guo et al., 2024b). These approaches rely on known camera trajecto- ries and encode them as input conditions. In contrast, our work considers fundamentally different setting where the camera trajectory is not available beforehand, yet the generated video must main- tain consistency with other synthesized content that is strongly correlated to the underlying camera motion. This key distinction renders existing methods inapplicable, necessitating a framework for controllable video generation that operates without predefined camera guidance.
Motion Generation. Generating realistic and diverse human motions from text remains longstand- ing challenge in computer vision and graphics, offering intuitive control of motion synthesis through natural language. Early works (Guo al., 2020; Petrovich et al. 202 Guo et al.) 2022a Petrovich
et al. 2022) employ temporal VAES (Kingma et al.] 2014) to capture temporal dependencies and
learn probabilistic mappings between language and motion. Recent advances have introduced pow-
erful generative modeling techniques to this field, including diffusion models Tevet et al. 2023
Zhang et al. 2024, Chen et al. 2023), autoregressive models (Guo et al. 2022b Zhang al. 2023
Jiang et al. 2023), and generative masked models (Guo et al. 2024a] Pinyoanuntapong et al. 2024
Meng et al. 2025). To comply with these frameworks, motion data is represented in different forms.
Diffusion- based methods typically operate on continuous vectors, either in the latent space of a VAE or directly from raw motion sequences. Autoregressive models, by contrast, often discretize motion into tokens using vector quantization techniques such as VQ-VAE (Van Den Oord et al. 2017) or RVQ-VAE [Lee et al. 2022). Generative masked models are flexible in this regard, accommodating both discrete and continuous representations depending on the loss function and model architecture.
Furthermore, several researchers (Hassan et al. 2021 Wang et al.] 2021: Huang et al.) 2023 Zhao
al. 2023) have investigated human motion generation within 31D scenes represented RGB point clouds. Others combine the above two tasks by simultaneously incorporating textual and scene in- formation (Wang et al. 2022 Cen et al. Wang et al. 20244a] Yi et al.) 2024). Our work differs from this line of research in how scene information is provided: instead of granting full scene access during motion synthesis, we observe the scene only once from the initial human pose and rely on a generative model to hallucinate scene observations as the human moves. In egocentric vision, stud- ies (Li et al. 2023 Yi et al.) [025 Hong et al. (025) focus on estimating the human motion from the egocentric video, while we explore this close relationship through joint generative modeling.
Multimodal Generation. Recent advances have expanded generative models from unimodal to multimodal generation. Specifically for diffusion models, |uan et al. (2023) introduces the first multimodal diffusion framework for synchronized audio-video generation. Other studies (Xu et al., Bao et al. 2023) design unified models capable of jointly generating text and images. In the domain of human motion, Li et al. (2024) pioneers the simultaneous generation of motion and frame- level language deseriptions that explain the generated motions. Despite these developments, the joint modeling of human motion and its corresponding egocentric views remains largely unexplored. To the best of our knowledge, take the first step in this direction, uncovering their tight coupling.
2023
3

---

Published as a conference paper at ICLR 2026
4u
4
8/
Timestep Embed
Video VAE Decoder
Motion VAE Decoder
VidtoVideo Feed Foward
Motion Feed Forward
Text AdaLN
Video AdaLN
Motion AdaLN
-D-
Text Block
Vidco Block
Text-Video Out Projection
Motion Out Projection
Dd
Text Block
Video Block
Motion Block
Joint Text-Vidoo Vidco-Mottin Attention
Video VAE Encoder
Motion VAE Encoder
Text Encoder
Text-Video QKV Projection
Motion QKV
Projection
Enter the notraation mom, firn right, and apen dear
Text
Video Adal.N N
Motion Adal.N f
AdaL.
Figure 2: Ewo features a triple-branch architecture (left), where the motion branch spans only the lower half of the layers used by text and video branches. Each branch has own tokenizer and transformer blocks (right), with shared weights across branches indicated by matching colors.
3 METHODOL.OGY
Problem Definition. Given initial human pose po E Jx3 in scene, an egocentric observation 70 E TR. xgg x3 from that pose, and a textual description of intended human actions in the scene, our goal is generate two synchronized sequences: (l) a human pose sequence p1:Nm
XJxx3 and (2) an egocentric view sequence 1 I1:N. E R XN.xHxWXx spanning the same duration. Here, is the number of human joints, H and W are the image height and width, Nm and N, are the number of frames in the pose and view sequences, respectively. This forms closed-loop generation paradigm where video and motion mutually and continuously influence each other over time.
Framework Overview. An overview of EgoTwin is shown in Figure 2 Text, video, and motion inputs are first encoded using a text encoder, video VAE encoder, and motion VAE encoder, re- spectively. These embeddings are then processed through the corresponding branches of diffusion transformer. Finally, the video and motion outputs are decoded by respective VAE decoders.
3.1 MODALITY TOKENIZATION
For the text and video modalities, we adopt T5-XXL (Raffel et al. 2020) as the text tokenizer and encoder, and 3D causal VAE (Yang et al. 2025) the video tokenizer. Specifically, the input text first tokenized and adjusted to a fixed length I, via truncation or padding, then encoded into text embeddings e E IR De The video frames are temporally and spatially compressed into latent representations E R No +1)x H xC, with a compression ratio of 4 x 8 8 and C. latent channels, which are subsequently patchified and unfolded into video embeddings X, x D. of sequence length L,, D; and D, denote the embedding dimension of text and video, respectively.
Motion Representation. Unlike the uniform representation for text and video, motion representa- tion exhibits great degree of diversity. Currently, the most widely adopted format in human motion generation is the overparameterized canonical pose representation (Guo et al., 2022a), which has become the default standard for popular datasets, including KIT-MI. (Plappert et 2016) and Hu- manML3D (Guo et al., 2022a). Formally, the human pose at each frame is defined as a tuple of (7 f=z T. j", J" j e ), comprising seven groups of features: root angular velocity along Y-axis fa, root linear velocities on XZ-plane j=z root height A local joint positions jr E IR3(/-1) and velocities jr E (R3((1)) in root space, joint rotations j" E ( in local space, and binary foot- ground contacts a' E Rd.. Motions are retargeted to a default human skeletal template and initially rotated to face the positive Z-axis.
However, the above root-centric representation not suitable for our task, as the critical informa- tion for alignment with the egocentric video. such as the pose of the head joint, is deeply buried in a multi-step kinematic calculation. Mathematically. recovering the head joint pose requires inte- grating root velocities to obtain the root pose, then applying forward kinematics (FK) propagate transformations through the kinematic chain to the head joint. Intuitively, this computation is too complex be precisely modeled by neural networks, experimentally substantiated in Section
4

---

Published as a conference paper at ICLR 2026
To address this issue, we propose head-centric motion representation that explicitly exposes ego- centric information. Specifically, we define the representation tuple (h", h², h², h2, j," j", j"",, where h" E R and h² E R are the absolute and relative rotation of the head joint, h2 E IR' and hP E IR3 are the absolute and relative position of the head joint. The terms jo and j are now pressed in head space, while j" retains its original meaning, Additionally, we normalize the initial head pose to zero translation and identity rotation, and set all first-order kinematic features zero in the initial frame. Our representation naturally resonates with egocentric video in at least two novel ways: 1) offers more accurate access to the head trajectory, which closely correlates with camera movement; 2) It more clearly informs the egocentric video how the body is observed egocentrically.
Motion Tokenization. Inspired by the Causal 3D CNN (Yu al. 2024), we build the motion VAE using 1D causal convolutions, where all padding is applied at the beginning of the convolutional axis. The encoder and decoder are symmetrically structured, each comprising two stages of 2x downsampling upsampling, interleaved with ResNet blocks (He ct al. 2016). The motion VAE is trained using a combination of reconstruction loss free and Kullback-Leible (KL) divergence regularization LKL weighted by λKL- To ensure that loss contributions are balanced across different groups regardless of their dimensions, we compute the VAE loss LVAe separately for the 3D head (h², hP),, 6D head (l",, h²), 3D joint (iP,j"), and 6D joint (j") components. The final loss averages these four items:
LVAE Σ
((o) where E {head header. joint joint E(Cco) (Cfel + kkL
(1)
Using the trained VAE, motion representations are tokenized into latents Zm E T "Tmh +1)xCmm with a 4x downsampling rate and C'm channels, and subsequently transformed into motion embeddings
Xm E B. x Dm, with Lm, as the sequence length and Dm. as the embedding dimension.
3.2 DIFFUSION TRANSFORMER
Our diffusion transformer extends MM-DiT (Esser et al. 2024), initially designed for text-to-image generation, to support text, video, and motion modalities. As illustrated in Figure 2 each branch consists of sequence of MLPs and applies adaptive layer normalization (AdaLN) in conjunction with a gating mechanism (Peebles & Xie 2023) to incorporate timestep information. The text and video branches are initialized from Cog VideoX (Yang al. 2025), with shared weights except for the AdaLNs. The motion branch corresponds to only the lower half of the layers in other branches, as essential visual cues for video-motion interaction, such as camera pose and scene structure, are primarily captured in the early layers of the video diffusion backbone. In contrast, the higher layers specialize in appearance details, which are less relevant motion. To further improve efficiency, the motion branch employs reduced channel dimensions, consistent with the lower representational complexity of motion relative to video. The embedding sequences from different modalities are projected to common dimensionality D and concatenated for joint attention operations (Vaswani al. 2017). This triple-branch architecture allows each modality to work in its own representational space while still attending to and interacting with the others.
Interaction Mechanism. The original MM-DiT framework includes only text and image modal- ities, where cross-modal consistency is enforced only at the global level, i.c., matching the entire image with the entire text suffices. However, our task demands ine-grained temporal synchroniza- tion between video and motion: each video frame must be temporally aligned with the corresponding motion frame. Although we incorporate sinusoidal positional encodings (Vaswani et al.] 2017) for both video and motion tokens, along with 3D rotary position embeddings (RoPE) (Su et al. (2024) for video tokens to provide absolute and relative position information, these mechanisms primar- ily capture intra-modal structure. Consequently, the inter-modal correspondence each time step remains implicit to the diffusion transformer, which may lead to globally consistent outputs that nevertheless lack frame-wise synchronization.
To address this challenge, we explicitly encode the causal interplay between video and motion by introducing a structured joint attention mask to the diffusion transformer. Given that human motion is typically captured at higher temporal resolution than egocentric video, we set the number of motion tokens to be twice the number of video tokens (i.e., Nr, = 2N,,, without loss of generality.
5

---

Published as a conference paper at ICLR 2026
Formally. we follow the notations in Cybemetics
Preserved
Blocked
(Agrawal et al. 2016; Pathak et al. 2017) to rewrite I' as the observation O and [P2011 2 the (chun- ked) action A', where E 10, N2 - -1]. According to the principles of forward dynamics: {0r, A*) O++1 and inverse dynamics: {0', {0', O++1] + A', video tokens cor- responding to O' can attend to motion tokens that corre- spond to A capturing how O' arise from A'-1, while motion tokens corresponding to A'' can attend video tokens that correspond to both O' and O*+1, enabling the inference of A based on scene transitions from O' to O++1, A special case is given to pa, which is allowed bilateral attention with 1 As demonstrated in Figure 3 apart from the aforementioned relationship, the remain- ing attention between video and motion is blocked, while all intra-modal attention, well as inter-modal attention related to text, are preserved.
Figure 3: Interaction mechanism.
Asynchronous Diffusion. We independently sample timesteps, l, and (m. between and T (maximum timestep), and add Gaussian noises e and fm associated with these timesteps to the la- tents 2, and 2m respectively. Each timestep is first encoded via a sinusoidal embedding, and MLP then processes two concatenated embeddings to produce a unified timestep embedding y, which serves as input the AdaLN layers. Our model consists of a video denoiser (= zim C, tu tim) and motion denoiser eg (2 24r fim tw), which are jointly optimized to simultaneously predict the noises added to the video and motion latents using the following objective:
DIT Peeoiem [1cr
(2 =m C, by Lm 1 [ ||<mn
'g <2/m (2/m 20 C, Ii to) 12] (2)
3.3 TRAINING AND SAMPLING
Training Paradigm. Our training schema comprises three stages: 1) Motion VAE Training, as de- scribed in Equation []] 2) Text-to-Motion Pretraining. Since the motion branch lacks pretrained weights for initialization, we pretrain it on the text-to-motion task using only text and motion embed- dings as input, while keeping the text branch frozen. Following classifier free guidance (CFG) (Ho) & Salimans 2022), we randomly discard the text embeddings with probability of 10% to model unconditional motion generation. By omitting the much longer video embeddings, we can leverage greater parallelism, which accelerates the training process. Critically, freezing the text branch not only preserves the pretrained text-to-video weights but also facilitates the integration of motion em- beddings into the pretrained text-video embedding space. 3) Joint Text-Vieeo-Be Video-Motion Training, as formulated in Equation (2). Video embeddings are incorporated in this final stage, and the model learns the joint distribution of video and motion conditioned on text. Again, text embeddings are randomly dropped with probability of 10% to model the unconditional video-motion generation.
Sampling Strategy. Benefiting from the joint distribution modeling, our framework supports not only joint video-motion generation conditioned on text (T2VM), but also unimodal generation, in- cluding video generation conditioned on text and motion (TM2V), and motion generation condi- tioned on text and video (TV2M). The CFG for TM2V sampling defined as follows:
2yd (2 20 C, t)
Gj (2 2T p, T (c)(2) Am C, t, T) Cp c5(2) 2 T))
(3) The CFG formula for TV2M sampling can be derived by exchanging the roles of and m Equa- tion 33. Here, Wt. Wv. and Wm denote the guidance scales for text, video, and motion conditions, respectively. For T2VM sampling, taking the motion branch as an example (with the video branch being analogous), its CFG formula is expressed as:
+wm (2) Ams G, t, - <) 2 G,t G,,,))
T Eg (='. 2 e, 1, 1) 6g m (zin zy &, 1, T)+w, (2 Zo e, C,I,T) 66 ( 2T (T (,,,))
(4) After sampling, latents from the video branch unpatchified to recover their original shape and then decoded by the 3D causal VAE decoder (Yang et al.) (025) to reconstruct the video, while latents from the motion branch are passed through the motion VAE decoder reconstruct the motion.
+w, (eT (2 2,, c, t,t) e/ (2 o,,,,,

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EVALUATION METRICS
Video Quality. We adopt Image Fréchet Inception Distance (I-FID) (Heusel et al. 2017) to evaluate the visual fidelity and realism of individual frames by measuring the distributional distance between the features of generated frames and those of real images. At the video level, we employ Fréchet Video Distance (FVD) (Unterthiner et al. 2018) to quantify temporal coherence and consistency across generated video sequences compared to real ones. Additionally, CLIP Similarity (CLIP- SIM) (Wu et al. 2021) is utilized to assess the semantic alignment and contextual relevance between generated video clips and textual prompts.
Motion Quality. We choose Motion Fréchet Inception Distance (M-FID) (Heusel et al. 2017) to as- sess the statistical similarity between the high-level features of generated motions and real motions. To evaluate the alignment between text and motion, we train a GRU-based text feature extractor and a GRU-based motion feature extractor, both sharing the same architecture
the evaluator in (Guo et al. 2022a). These models are optimized using a contrastive loss on GloVe (Pennington al. 2014) text embeddings and our motion representation described in Section 3.1] ensuring that matched text-motion pairs yield geometrically close feature vectors. Within this Icarned feature space, the text-to-motion Retrieval Precision (R-Prec) measured in terms of Top-3 retrieval ac- curacy. Meanwhile, the Multimodal Distance (MM-Dist) captures the average Euclidean distance between corresponding motion and text features.
Video-Motion Consistency. We propose to evaluate the consistency between generated egocentric videos and human motions from two aspects: 1) View Consistency: We first estimate the frame- wise camera poses of the generated egocentric videos using FROID-SLAN, (Teed & Deng, 2021) and extract the head joint poses from the generated human motions. Then, we align both trajec- tories at the first frame and apply Procrustes Analysis to determine the optimal scale factor that aligns the estimated camera trajectory with the extracted head trajectory. Finally, we compute the Translation Error (TransErr)
the average Euclidean distance between the corresponding cam- era and head positions, and the Rotation Error (RotErr) as the average angular difference between the corresponding camera and head orientations, using the same formulas as He et al.] (2025). 2) Hand Consistency: We detect the presence of the left and right hands, equipped with the motion capture device, in the generated egocentric videos. For the generated human motions, we compute the hand visibility from the perspective of a virtual camera mounted on the corresponding head joint with known intrinsics. Based on the presence and visibility analysis, we define the Hand F-Score (HandScore) as the average F-Score of left and right hands, where a True Positive means the hand is present in the video and visible from the head in motion, a False Positive means the hand is present in the video but invisible from the head in motion, and a False Negative means the hand is absent in the video but visible from the head in motion.
4.2 EXPERIMENTAL SETUP
Dataset. To overcome the limitations of using synthetic or small-scale real-world datasets for eval- uation, train and evaluate our model on Nymeria (Ma et al.) 2024). ; large-scale, real-device dataset that captures diverse people engaged in a wide range of daily activities across various indoor and outdoor locations. The dataset provides paired text-video-motte
data, including egocentric videos recorded with Project Aria glasses (Engel et al. (002)), full-body motions captured using the Xsens inertial motion capture system (Paulich al.] 2018), and motion narrations written by human annotators. All data are segmented into 5-second clips, yielding approximately 170K samples after filtering, which are split into training, validation, and test sets for the joint training stage. We ensure that both the individuals and environments in the test split remain unseen during joint training.
Baseline. Since no prior methods are capable of addressing our task, we propose a simple yet ef- fective baseline, VidMLD, that retains the architecture of EgoTTwin while removing all dedicated designs introduced in Section 3.1 and Section 3.2 In other words, VidMLD combines the state-of- the-art video diffusion model Cog VideoX (Yang et al. 2025) and the latent-space motion diffusion model MLD (Chen et al. 2023), both of which excel in unimodal generation, and connects them through the multimodal diffusion architecture MM-DiT (Esser et al. 2024) to enable joint genera- tion. We adopt the same three-stage training recipe described in Section 3.3 and employ the original classifier-free guidance (Ho & Salimans 2022) for sampling.
7

---

Published as a conference paper at ICLR 2026
Table I: Quantitative results of joint video and motion generation, evaluated by metrics covering video quality, motion quality, and vidco-motion consistency.
Video Quality
Motion Quality
Video-Motion Consistency
Method
I-FID + FVD 4 CLIP-SIM 1 M-FID + R-Prec ? MM-Dist + TransErr + RotErr HandScore f
VidMLD 157.86 1547.28
25.58
45.09
0.47
19.12
1.28
1.53
0.36
EgoTwin 98.17
1033.52
27.34
41.80
0.62
15.05
0.67
0.46
0.81
Prompt: Enter the recreation room, turn right, and open the door to the yard.
Prompt: Tum left walk into the kitchen, then turn towards the living area.
Condition
Condition
Qualitative results of joint video and motion generation, based textual prompt and initial frames of both video and motion.
Figure 4:
Implementation Details. In our experiments, videos are undistorted and resized to resolution of H - W 480, with each segment containing +1 = 41 frames FPS. The motion data adopts the Xsens skeleton with J 23 joints and consists of Nm + 81 frames per segment at 16 FPS. The video and motion latents have C. = 16 and Cm 64 channels, respectively. The embedding lengths for text, video, and motion L, 226, L 9900, and Lm. 21, with corresponding dimensions D. = D, = D = 3072, and Dm = 768. The hyperparameter λ in Equation ① is set to le-4. CFG scales and video branches have 42 layers, totaling approximately 5B parameters, with most shared across both branches. The motion branch comprises 21 layers, corresponding to the lower halves of the other two branches, and contains roughly 300M parameters.
w; 6 for text and w. W'm 4 for video and motion. The text
4.3
MAIN RESULTS
Quantitative Results. As shown in Table Tgo significantly outperforms the baseline method across all evaluation metrics, with especially pronounced improvements in video-motion consis- tency scores. The brute-force joint training of the video and motion generation models leads to poor alignment between the two modalities, resulting in notably lower video-motion consistency performance. In contrast, Tgn effectively captures the intrinsic correlation between the two modalities, achieving not only excellent cross-modal consistency but also enhanced single-modal generation quality through the mutually beneficial interaction between video and motion modalities.
Qualitative Results. We visualize several examples generated by Ego in Figure These samples illustrate that the video and motion streams not only adhere the textual descriptions for songle-modal generation but also evolve in strict cross-modal synchrony, particularly in terms of camera viewpoint and head pose, as well as in scene content and human action. We encourage read- ers to visit our project page (https:/ http://gotwiin www.ggii..agess...............ee dev/) / for richer generation examples.
4.4 ABLATION STUDIES
We present the extensive ablation studies in Table where each row corresponds to a specific ab- lation setting. All variants exhibit a consistent performance decline across all metrics compared to our full model (listed at the bottom), confirming the effectiveness each design. First, we replace our Motion Reformulation with the standard representation (Guo et al.) 2022a) commonly used in human motion generation research (""wo MR"). The resulting performance drop highlights the im- portance of our reformulation in exposing egocentric motion cues to the video, which fundamentally facilitates the aligmment between egocentric video and human motion. Next, we remove the Interac- tion Mechanism from the joint attention operations and instead apply full attention without masking
8

---

Published as a conference paper at ICLR 2026
Ablation results three designs: Motion Reformulation (MR), Interaction Mechanism (IM), and Asynchronous Diffusion (AD).
Table 2:
Video Quality
Motion Quality
Video-Motion Consistency
Variant
I-FID + FVD 4 CLIP-SIM M-FID + R-Prec f MM-Dist TransErr + RotErr HandScore f
w/o MR
134.27
1356.81
26.36
43.65
0.56
17.31
0.96
1.22
0.44
w/o IM
117.54
1237.58
27.10
44.01
0.59
15,87
0.85
0.89
0.57
w/o AD
109.73
1124.19
26.91
42.58
0.53
16,48
0.74
0.62
0.73
Ego
98.17
1033.52
27.34
41.80
0.62
15.05
0.67
0.46
0.81
Table 3: Comparisons between joint intoo-motion modeling and separate video/motion modeling.
Video Quality
Motion Quality
Method
I-FID
FVD
CLIP-SIM f
M-FID +
R-Pree ?
MM-Dist 4
VideoX 182.97
1793.79
24.90
CameraCtrl
120.48
1263.90
27.01
MLD
47.25
0.39
21.47
EgoTwin 98.17
1033.52
27.34
41.80
0.62
15.05
("w/o IM"). The observed degradation underscores its critical role in capturing causal relationships between video and motion, well as ensuring fine-graine temporal synchronization. Finally, substitute the Asynchronous Diffusion with a synchronous counterpart for video and motion latents, and accordingly simplify the sampling algorithm vanilla CFG ("w/o AD"). The performance de- cline validates its value for modeling comprehensive and diverse dependencies between video and motion modalities, and enabling precise textual control over the joint generation process.
4.5 IN-DEPTH ANALYSIS
To demonstrate the advantages joint modeling, we compare our joint generation (Text-to- Video- Motion) with separate generation approaches (Text-to- Video and Text-to-Motion), with results ported in Table 3 We also implement and compare against camera-controlled video generation method based on VideoX The substantial performance gains confirm that joint modeling sig- nificantly enhances the generation quality of each modality. This is corroborated by Table [L which shows that the stronger joint generation capability of Two yields notable improvements both video and motion metrics compared to VidMLD, which exhibits weaker joint modeling.
Beyond quantitative evaluation, we also provide qualitative analysis using the door-opening exam- ple in Figure [ Without modeling the state of the door at each timestep and its temporal evolution, the generated human motion often appears unrealistic, hand, without modeling the underlying human motion that drives the egocentric video, the resulting viewpoint shifts and scene dynamics tend to physically implausible. Moreover, relying solely on camera poses as external control signals fails to capture the underlying human motion in egocentric video, underscoring the necessity for internal motion modeling egocentric video generation.
lacks awareness of the door. On the other
4.6 PROSS-DATARE EVALUATION
Given the scarcity of high-qualing datasets synchronizing full-body human motion with egocentric video, our evaluation in T on the Nymeria dataset. Specifically, we establish a rigorous evaluation protocol by strictly partitioning the data to ensure that human subjects and interaction scenes in the test set are unseen during the joint training stage. To further assess the generalization capabilities of the video-motion correspondence captured by our model, we conduct a systematic dross-dttaset evaluation on Ego-Exo4D (Grauman et al.) 2024). However, a substantial discrepancy exists in skeletal representations: Ego-Exo4D annotations adopt the sparse 17-keypoint COCO for- mat (Lin et al. 2014), which structurally differs from the dense 23-joint Xsens skeleton required by model, rendering direct inference infeasible.

---

Published as a conference paper at ICLR 2026
Table 4: Cross-dataset results of joint video and motion generation, evaluated by metrics covering video quality, motion quality, and vidco-motion consistency.
Video Quality
Motion Quality
Video-Motion Consistency
Method
I-FID + FVD 4 CLIP-SIM 1 M-FID + R-Prec MM-Dist + TransErr + RotErr HandScore f
VidMLD 173.95 1738.15
24.27
52.18
0.37
23.09
1.46
1.78
0.32
EgoTwin 115.82 1242.60
26.13
48.34
0.50
20.16
0.70
0.51
0.79
Prompt: Open and close the kitchen cabinet.
Condition
Condition
Figure 5: Results of conditional generation. Left: motion generation conditioned on text and video; Right: video generation conditioned on text and motion.
To address this skeletal incompatibility, we design an optimization based retargeting pipeline. Specifically, first regress the sparse COCO keypoints to the SMPL parametric model (Loper et al. 2015), leveraging its inherent statistical shape and pose priors to reconstruct missing degrees of freedom and enforce anatomical plausibility. Subsequently, we retarget the resolved SMPL kine- matics to the Xsens rig via precise joint remapping and spline-based interpolation. Finally, we employ forward kinematics to propagate local joint angles into global joint positions and rotations. As evidenced in Table EgoTwwi not only significantly outperforms the baseline but also maintains superior sideo-motion consistency metrics. This validates that the video-motion correspondence learned by Tgn is robust and generalizes effectively to external datasets.
4.7 BROADER APPLICATIONS
Conditional Generation. Our joint distribution enables conditional sampling of one modality given another, using the CFG algorithm described Equation (3). As shown in Figure we can generate human motion conditioned on text and egocentric video (left), as well as generate egocentric video conditioned on text and human motion (right). Interestingly, textual descriptions are often ambigu- ous (e.g., they may refer to cabinets additionally condition on either motion or video provides greater control over the generation process, which further substantiates the strong coupling between video and motion in our model.
the left right side of the scene in Figure 5 the ability to
Scene Reconstruction. With jointly generated video and motion, we can effortlessly extract camera poses from human motion and directly integrate both modalities into 3D Gaussian Splatting (Kerbl et al. 2023) pipeline. As illustrated in Figure [ we reconstruct the 3D scene from the generated video and seamlessly position the synthesized human into it by aligning head poses with camera trajectories. The realistic spatial interactions exhibited, such as the feet on the ground and the right hand near the door handle, demonstrate strong spatiotemporal coherence between the generated egocentric videos and human motion.
CONCLUSION
We propose EgoTwin, diffusion-based framework that jointly generates egocentric video and hu- man motion in viewpoint consistent and causally coherent manner. Our method introduces a head- centrie motion representation and cybernetics-inspine efficient three-stage training paradigm and versatile sampling strategies. To evaluate our approach, we establish a comprehensive benchmark that includes large-scale dataset of rext-video-man -ottonn motion consistency metrics. Experiments demonstrate that TgoTwin deliv- ers promising results. We hope our work encourages further exploration of joint generative modeling for egocentric video and human motion, and lays a solid foundation for future research in this area.
interaction mechanism, supported by an
triplets and novel
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work funded in part by the HKUST Start-up Fund (R9911), Theme-based Research Scheme grant (T45-205/21-N), the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Govermment, and the research funding under HKUST- DXM Al for Finance Joint Laboratory (DXM25EG01).
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in Neural Information Processing Systems, 29, 2016.
Apple Inc.
Apple vision
pro. 2023.
URL https://www.apppn com/
apple Sople-mi vi s on- prop
Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K Chilana. Creat- ing augmented and virtual reality applications: Current practices, challenges, and opportunities. In ACM CHI Conference on Human Factors in Computing Systemx, pp. 1-13, 2020.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pp. 1692-1717, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprinl arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhom, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563- 22575, 2023b.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL httpt:://ppppaaaaeeeee aeteeaa..co/resseaee wwt. wwt./eseeeccc/////oo/eeee
video--eneraa ieteeeraaiinnmooooe
inn cn-mooemmlll s- aa-as-a- -as-asww wor d- simulators
Zhi Cen, Huaijin Pi, Sida Peng. Zehong Shen, Minghui Yang, Shuai Zhu, Hujun Bao, and Xiaowei Zhou, Generating human motion in 3d scenes from text descriptions. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1855-1866, 2024.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2021.
Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint 2rXiv:2308.13561 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
Alexander Fiannaca, Ilias Apostolopoulous, and Eelke Folmer. Headlock: A wearable navigation aid that helps blind cane users traverse large open spaces. In International ACM SIGACCESS Conference on Computers and Accessibility, pp. 19-26, 2014.

---

Published as a conference paper at ICLR 2026
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19383-19400, 2024.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In ACM International Conference on Multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152 5161, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng, Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 380-597. Springer, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1900-1910, 2024a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. In International Conference on Learning Representations, 2024b.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In IEEE/CVF International Conference Computer Vision, pp. 1.374-11344 2021.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang- Cameractrl: Enabling camera control for text-to-video generation. In International Conference on Learning Representations, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. JEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Jonathan Ho and Tim Salimans. 2rXiv:2207.12598 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022.
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning 3d scenes. In IEEF/VVF Conference Computer Vision and Puttern Recognition, pp. 16750-16761, 2023.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
12

---

Published as a conference paper at ICLR 2026
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4): 2023,
Diederik Kingma, Max Welling, et al. Auto-encoding variational bayes. International Confer- ence on Learning Representations, 2014.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shh Shin Han. Autoregressive image generation using residual quantization. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11523-118 11532, 2022.
Chuqiao Li, Julian Chibane, Yannan He, Naama Pearl, Andreas Geiger, and Gerard Pons-Moll. Unimotion: Unifying 3d human motion synthesis and understanding. In International Conference on 3D Vision, 2024.
Jiaman Li, Karen Liu, and Jiajun Wu. Ego-body pose estimation via ego-head pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17 17142-17151, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, pp. 2980-2988, 2017.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: skinned multi-person linear model. ACM Transactions Graphics, 34:1-16, 2015.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong. Juhyun Lec, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv: 08166008117 72, 2019.
Lingni Ma, Yuting Ye. Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collec- tion of multimodal egocentric daily motion in the wild. In European Conference on Computer Vision, pp. 445-465 Springer, 2024.
Zichong Meng, Yiming Xie, Xiaogang Peng, Zeyu Han, and Huaizu Jiang. Rethinking diffusion for text-driven human motion generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: univer- sal visual representation for robot manipulation. In Conference on Robot Learning. 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-super prediction. In International Conference on Machine Learning, pp. 2778-2787, 2017.
Monique Paulich, Martin Schepers, Nina Rudigkeit, and Giovanni Bellusci. Xsens mtw awinda: Miniature wireless pneetial-magneet motion tracker for highly accurate 3d kinematic applications. Xsens: Enschede. The Netherlands, pp. 1-9, 2018.
Georgios pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9826-9836, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In JEEF/CVF International Conference Computer Vision, pp. 4195 2023.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Conference on Empirical Methods Natural Language Processing, pp. 1532- 1543, 2014.
13

---

Published as a conference paper at ICLR 2026
Mathis Petrovich, Michael J Black, and Güi Varol. conditooodi 3d human motion synthesis with transformer vac. In JEEE/CVF International Conference on Computer Vision, pp. 10985- 10995, 2021.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference Computer Vision, pp- 480-497, 2022.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1546- 1555, 2024.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big Data, 4(4):236-22,, 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-1999 19325, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in Neural Information Processing Systems, 34: 111116558166660 2021.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. In International Conference on Learning Representations, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 11X.011812201117,,200 2018.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 30, 2017.
Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9401-9411, 2021.
Zan Wang. Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems, 35:14959-14977 2022.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move you say interact you Language-guide human motion generation with scene affordance. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 433-444, 2024a.
14

---

Published as a conference paper at ICLR 2026
Zhouxia Wang, Ziyang Yuan, Xintao Wang. Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM Special Interest Group on Computer Graphics and Interactive Techniques Conference, pp- 1-11, 2024b.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural deseriptions. arXiv preprint arXiv:2104.14866 2021.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang. Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In European Conference Computer Vision, pp. 399-417. Springer, 2024.
Jingqiao Xiu, Mengze Li, Wei Ji, Jingyuan Chen, Hanbin Zhao, Shin iicii Satoh, and Roger Zim- mermann. Hierarchical debiasing and noisy correction for cross-domain video tube retrieval. In ACM International Conference on Multimedia, pp. 9271-9280, 2024.
Jingqiao Xiu, Mengze Li, Zongxin Yang, Wei Ji, Yifang Yin, and Roger Zimmermann. Few-shot in- cremental learning via foreground aggregation and knowledge transfer for audio-visual semantic segmentation. In AAAI Conference on Artificial Intelligence, pp. 8788 8796, 2025a.
Jingqiao Xiu, Yicong Li, Na Zhao, Han Fang, Xiang Wang, and Angela Yao. Geometric alignment and prior modulation for view-guided point cloud completion on unseen categories. JEEE/CVF International Conference Computer Vision, pp. 27435-27444, 2025b.
Jinggiao Xiu, Can Wang, and Dong Xu. Milmsplat: A 2d mwlmmpoowett framework for 3d gaussian splatting understanding, generation, and editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In IEEE/CVF International Conference on Computer Vision, pp. 7754-7765, 2023.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations, 2025.
Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in ego-sensed world. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference on Computer Vision, pp. 246-263 Springer, 2024.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, al. Language model beats diffusion- tokenizer is key to visual generation. In International Conference on Learning Representations, 2024.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete rep- resentations. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14730- 14740, 2023.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 46(6):4115-411 2024.
Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In /EEE/CVF International Conference on Computer Vision, pp. 14738-14749, 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILS OF MOTION VAE
VAE Encoder. The encoder begins with a ID causal convolution (kernel size 3) that projects the 288-channel motion input to 512 channels, followed by four DownBlocks that progressively expand the channel dimension to 2048. The first two DownBlocks each downsample the temporal dimen- sion by factor of 2 using ID causal convolutions with a kernel size of 4 and stride of 2. Each DownBlock comprises four ResNet layers, with each layer including two ID causal convolutions (kernel size 3). GroupNorm, SiLU activation, and optional shorteut connections for channel align- ment. A MidBlock comprising two 2048-channel ResNet layers further processes the features. The encoder ends with a ID causal convolution (kernel size 3) that outputs 128 channels, representing the mean and log-variance of 64-dimensional motion latent space.
VAE Decoder. The decoder mirrors the encoder in reverse. starts with ID causal convolution (kernel size 3) that maps the 64-dimensional motion latent vector to 2048 channels, followed by MidBlock with two 2048-chamne ResNet layers. Four UpBlocks then progressively reduce the channel dimension to 512. The first two UpBlocks each upsample the temporal dimension by factor of 2 using ID nearest neighbor interpolation and ID causal convolution (kernel size 3). Each UpBlock includes five ResNet layers, structurally identical to those in the encoder. The decoder ends with a ID causal convolution (kernel size 3) that reconstructs the original 288-channel motion representation. Both the encoder and decoder contain approximately 200M parameters.
B DETAILS OF CONSISTENCY METRICS
Since the generated motion sequence has twice the temporal resolution of the video sequence, downsample it by selecting every other frame to match the length of the video sequence, Ne. before evaluating the consistency metries described below.
View Consistency. We denote the canonicalized egocentric camera trajectory and head joint trajec- tory (i.e., with each frame expressed relative to the first frame) as sequences of rotation matrices and translation vectors: RR, 7.] E R.N. x3x4 and Rm Tm)] E R,N, x3x4 respectively. Let 8 denote the optimal scale factor for scene scale alignment. The Translation Error (TransErr) is calculated as the scale-invariant L2 distance between the translation vector sequences T, and Tw:
TransErr || Tm
(5)
The Rotation Error (RotErr) is computed by comparing the rotation matrix sequences Rg and Rm:
[rr RB R
RotErr = arccos
(6)
2
where tr(-) denotes the trace a matrix. Since each sequence is normalized with respect to the first frame, errors are computed the average frame-wise over the remaining frames.
Hand Consistency. Because the camera wearer's hands are covered by motion capture gloves rather than bare skin, -fate-of-the-art hand landmark detection tools, such MediaPipe (Lugaresi et al. 2019) and HaMeR (Pavlakos et al. 2024), fail to detect the wearer's hands in video frames. To overcome this limitation, formulate the problem two independent binary classification tasks: one for detecting the presence of the left hand and the other for the right hand in each frame. Our hand classifier is built upon pretrained Vision Transformer (ViT) (Dosovitskiy et al.) (021) back- bone and is trained using supervision signals derived from human poses and camera intrinsics. To mitigate the issue of class imbalance between positive and negative samples, we adopt Focal Loss (Lin et al. 2017) as our classification objective, which is defined as:
2
LEtS Σ
aj (1 - pi)" log (pi),
where y is the focusing parameter, c E {0, 1) indicates the hand side, pf is the predicted probability for the target class and of is weighting factor associated with the target class t. Given the
16

---

Published as a conference paper at ICLR 2026
TransErr
RotErr
0.8
0.6
0.4
0.2
0.0
0
10
20
30
40
50
Figure 6: Head pose regression
epochs.
predicted probability p sigmoid(z) obtained from the output logits of the model RR, and the ground-truth labels E {0, 1) the values of pi and of are calculated as:
p
if y =1
a if y" = {
pf-
of
(8)
p
p
if yf 0
a
if y =0
In experiments, set of 0.80 for the left hand and af 0.75 for the right hand, and use focusing parameter of 7 2.
C DETAILS OF REGRESSION EXPERIMENT
To validate our insights in Section 3.1 we train GRU-based regression model that takes the root- centric motion representation sequences as input, supervised by an MSE loss against the ground- truth head pose sequences. As shown in Figure 6 both translation and rotation errors (TransErr and RotErr, see Section 4.1 for details) plateau at high levels, due to insufficient explicit cues for accurately modeling head pose.
D ADDITIONAL QUALITATIVE RESULTS
Joint Generation. Figure 7 presents additional examples generated by EgoTwin. Together with those in Figure 4 these results demonstrate the model's capacity to synthesize a variety of mo- tion types (e.g., walking, opening, closing, grabbing) across diverse environments (e.g., bedrooms, kitchens, living areas, outdoor yards) and involving a wide range of objects of interaction (e.g.. cabinets, clothing, doors, drawers). Video demonstrations winegotwin... pages. dev/
be found
project page:
Recall that in Figure 5 we demonstrate that when the textual descrip- tion is ambiguous concerning the referent objects involved in the interaction, conditional sampling strategies (i.e.. TV2M or TM2V) allow for fine-grained control over the generation process. An- other salient source of linguistic ambiguity arises from human interaction behaviors. For instance, while the prompt in Figure (egure@8cleen clearry specifies the target object of interaction (i.e., the pillow located on the right side of the sofa in Figure 8). leaves the manner of the human action unconstrained (e.g., whether the individual uses the left hand or right hand). such cases, incorporating additional conditioning signals can help disambiguate these subtleties.
Conditional Generation.
Failure Cases. The failure cases primarily stem from the physical implausibility of the generated videos. For instance, in the first example of Figure the cup displays abnormal deformation. This issue reflects broader challenge in current video generation models and is difficult to address through joint modeling with human motion, the motion signal typically captures only the global displacement of the object. Nevertheless, we generally observe strong correspondence between the generated video and motion.
17

---

Published as a conference paper at ICLR 2026
Prompt: Open the kitchen cabinet and take out mug.
Prompt: Look for something on the shelf the kitchen
Conditton
Condition
Prompt: Grab a piece of clothing from the floor.
Prompt: Walk down the hallway, then turn into the bedroom.
Condition
Condition
Prompt: Close the drawer, open the base cabinet, and then close it.
Prompt: Pick up the towels from the black loss them onto the bed.
Condition
Condition
Additional qualitative results joint generation, based on a textual prompt and initial frames of both video and motion.
Figure 7:
Prompt: Pick up the pillow the right side of the sofa.
Condition
Condition
Figure 8: Additional qualitative results of conditional generation. Left: motion generation condi- tioned on text and video; Right: video generation conditioned on text and motion.
E LIMITATIONS AND FUTURE WORKS
One major limitation of EgoTwin is its focus on full-body motion representation without incorpo- rating hand joints. This stems from the absence of hand motion capture data in our dataset, which restricts our model's ability to generate hand movements. While our model achieves reasonable consistency between generated video and motion, the synthesized videos may still exhibit artifacts that violate physical laws or lack 3D coherence, posing challenges for downstream applications.
Our future work could involve finetuning datasets that include hand motion to enable the gener- ation of coordinated body and hand motions alongside egocentric videos. Additionally, integrating 3D priors and physical constraints may enhance the realism of generated videos, thereby improving the fidelity the corresponding motion synthesis. Other avenues for improvement include increas- ing video resolution and extending the temporal context support longer sequences.
18


Reviews:
[{"review_id": "PyU1kMqr4l", "reviewer": "Reviewer_LqoE", "summary": "The paper introduces EgoTwin, a diffusion-based generative framework for jointly producing egocentric videos and human motion that are both viewpoint consistent (ensuring alignment between camera and head movement) and causally coherent (visual observations and human actions influence each other over time). To train and evaluate the model, the authors leverage the Nymeria dataset of synchronized text–video–motion triplets and introduce video–motion consistency metrics to assess alignment between generated video and motion.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed approach, built on a triple-branch diffusion transformer, effectively models text, video, and motion within a unified framework. The cybernetics-inspired interaction mechanism is interesting, which captures the bidirectional dependencies between visual and motion streams.\n\n- The paper is well-written, clearly structured, and easy to follow, with strong motivation and coherent presentation of technical details.\n\n- The evaluation is extensive and thorough, covering diverse quantitative and qualitative metrics that  demonstrate the model’s performance.", "weaknesses": "- The approach relies exclusively on the Nymeria dataset for training and evaluation. While Nymeria is a large and well-curated dataset, the generalization of EgoTwin to other egocentric or synthetic environments remains untested, like EgoExo4D.\n\n- The base model design choice for the text–video component relies on CogVideoX. While this is a strong foundation, it raises the question of whether using a more recent or higher-capacity base model could further improve video quality or multimodal alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KS8F3vHORJ", "reviewer": "Reviewer_9aiA", "summary": "This paper introduces EgoTwin, a generative approach that synthesizes ego-centric video and human poses jointly based on text instructions. It presents a novel motion representation for aligning viewpoints from a head-centric perspective. To model the relationships among text, motion, and video, the paper introduces an advanced interaction mechanism. Experiments on video quality, motion quality, and video-motion consistency underline the effectiveness of the proposed method.", "questions": "1. What does \"c\" represent in Figure 3?\n2. I don't fully understand why two different time steps for video and motion generation are necessary. It appears that a single time step would suffice for joint motion and video generation. What considerations influenced this design choice?\n3. Regarding unimodal sampling in Equation 3, I understand that \"T\" is set to zero on the left-hand side due to elimination. However, why is the variable \"T\" on the right-hand side? Should it be sampled over a grid of \"t\" and \"T\"?\n4. Similarly, in Equation 4, why does the left-hand side use both \"t\" and \"t,\" while the right-hand side uses \"t\" and \"T\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow, with the main idea clearly articulated. The implementation details are sufficient for reproduction.\n2. The design of head-centric motion tokenization is novel.\n3. The interaction mechanism is reasonable; it employs local temporal attention, which not only improves accuracy but also reduces computational load.", "weaknesses": "1. While head-centric motion tokenization may enhance head-centric evaluations, the quality of the whole body is not assessed. Can a full-body evaluation be included?\n2. A related paper is not cited: https://egoallo.github.io.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSs4ZW2cwP", "reviewer": "Reviewer_Dv46", "summary": "This paper proposed EgoTwin, a newly defined task that joins the generations of egocentric (first-person) video and human motion, addressing two core challenges: Viewpoint Alignment (ensuring camera trajectories in generated videos match head trajectories from human motion) and Causal Interplay (modeling influences between visual results and human actions). Formally, the authors proposed a multi-modal generation model to tackle this task, including a motion branch and a video generation branch. For the motion generation, a head-centric motion representation is proposed, while the Joint Text-Video-Motion Attention is used to unify both modalities. Experiments show the effectiveness of the proposed method.", "questions": "The authors should clarify the motivation and generalizability of the proposed approach, and providing more details about the Interaction Mechanism would be helpful.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. To the best of my knowledge, EgoTwin is the first work to explicitly model joint egocentric video and human motion generation.\n2. Several useful and reasonable techniques are proposed in this paper to handle this joint generation. The head-centric motion representation directly solves the limitation of root-centric representations. The attention mechanism is adjusted to improve the multi-modal learning and address causal interplay.", "weaknesses": "1. The main concern is the unclear motivation. The paper does not sufficiently justify why a complex triple-branch diffusion generation (with specialized motion generation/video interaction modules) is required, especially given recent advances in scaling video foundation models (e.g., Genie3) that can generate high-fidelity, interactive egocentric videos via implicit scene and motion modeling. EgoTwin heavily depends on explicit motion representation and cross-modal modules, and well-labeled data (Nymeria), which may not be necessary if foundation models can implicitly address these issues with less annotation and better generalization. While Nymeria is large (170K samples in 5s), the diversity and generalization are still questionable.\n\n2. The illustration of the Interaction Mechanism is chaotic and confusing. What is the meaning of \"human motion is typically captured at a higher temporal resolution than egocentric video\", and why set $N_m = 2N_v$? The presentation of $O$ and $A$ are not well defined. What is the relation among action $A$, observation $O$, pose $P$, and view sequence $I$? Figure 3 is also very confusing without a proper illustration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZZDMDYZ6ib", "reviewer": "Reviewer_ZiMJ", "summary": "This paper introduces EgoTwin, a diffusion framework for the novel task of jointly generating egocentric video and the wearer's full-body motion. The method tackles the key challenges of Viewpoint Alignment (matching camera to head) and Causal Interplay (the feedback loop between seeing and acting) . The architecture is a triple-branch (text, video, motion) Diffusion Transformer featuring two main contributions: a head-centric motion representation for direct alignment and a cybernetics-inspired attention mask to model causality. Using the Nymeria dataset and new consistency metrics, the paper demonstrates strong performance.", "questions": "This is a very strong and interesting paper. My main questions for clarification are the weaknesses listed above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces a novel and clearly defined problem of *joint egocentric video and human motion generation*, with strong motivation and potential applications in embodied AI and AR.\n- The head-centric motion representation is a well-motivated and technically sound reformulation that effectively exposes egocentric cues for video-motion alignment.\n- The model design—a triple-branch diffusion transformer with asynchronous diffusion and causal attention—is elegant and verified through systematic ablations.\n- Comprehensive evaluation, including novel *video-motion consistency metrics*, supports the technical claims. The qualitative results are convincing.", "weaknesses": "- The paper relies exclusively on the Nymeria dataset. While uniquely suited, this makes it difficult to assess model robustness. A brief justification for not using other large-scale egocentric datasets (e.g., EgoExo4D) would be helpful. \n- All data are segmented into 5-second clips. It's unclear if this is a dataset constraint (narration length) or a computational limit of the diffusion model. A discussion on this limitation and the path toward modeling longer action sequences would be beneficial.\n- The degree to which the text prompts are \"free-form\" is not analyzed. A description of the text diversity in the Nymeria dataset would be helpful to understand the model's linguistic generalization\n- The main paper's baseline comparison (Table 1) is limited to one method. The more informative comparison against strong unimodal baselines (video-only and motion-only) is in the appendix (Table 3). Maybe the authors could move Table 3 to the main paper to better highlight the benefits of joint modeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)