EgoTwin: Dreaming Body and View in First Person
Authors: name: Jingqiao Xiu; name: Fangzhou Hong; name: Yicong Li; name: Mengze Li; name: Wentao Wang; name: Sirui Han; name: Liang Pan; name: Ziwei Liu
Published: 2025-08-18 15:33:09

Abstract: While exocentric video synthesis has achieved great progress, egocentric
video generation remains largely underexplored, which requires modeling
first-person view content along with camera motion patterns induced by the
wearer's body movements. To bridge this gap, we introduce a novel task of joint
egocentric video and human motion generation, characterized by two key
challenges: 1) Viewpoint Alignment: the camera trajectory in the generated
video must accurately align with the head trajectory derived from human motion;
2) Causal Interplay: the synthesized human motion must causally align with the
observed visual dynamics across adjacent video frames. To address these
challenges, we propose EgoTwin, a joint video-motion generation framework built
on the diffusion transformer architecture. Specifically, EgoTwin introduces a
head-centric motion representation that anchors the human motion to the head
joint and incorporates a cybernetics-inspired interaction mechanism that
explicitly captures the causal interplay between video and motion within
attention operations. For comprehensive evaluation, we curate a large-scale
real-world dataset of synchronized text-video-motion triplets and design novel
metrics to assess video-motion consistency. Extensive experiments demonstrate
the effectiveness of the EgoTwin framework.