Title: PlayerOne: Egocentric World Simulator
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video generation

World model

Egocentric understanding
Abstract: We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real-scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and world-consistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

PlayerOne: Egocentric World Simulator

Abstract

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real-scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and worldconsistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

1 Introduction

World models [36, 28, 35, 13, 1, 34] have undergone extensive research due to their ability to model environmental dynamics and predict long-term outcomes. Recent breakthroughs in video diffusion models [30, 12, 17] have revolutionized this domain, enabling the synthesis of high-fidelity, action-conditioned simulations that forecast intricate future states. These advancements empower applications ranging from autonomous navigation in dynamic real-world environments to the creation of immersive, responsive virtual worlds in AAA game development. By bridging the gap between predictive modeling and interactive realism, world simulators are emerging as critical infrastructure for next-generation autonomous systems and game engines, particularly in scenarios requiring real-time adaptation to complex, evolving interactions.

Despite significant progress, this topic remains underexplored in existing research. Prior studies [9, 36, 7] predominantly focused on simulations within game-like environments, falling short of replicating realistic scenarios. Additionally, in their simulated environments, users are limited to performing predetermined actions ( i.e. , directional movements). Operating within the confines of a constructed world restricts the execution of unrestricted movements as in real-world scenarios. While some initial efforts [19, 26, 1] have been made toward real-world simulation, they mainly contribute to world-consistent generation without human movement control. Consequently, users are reduced to passive spectators within the environment, rather than being active participants. This limitation significantly impacts the user experience, as it prevents the establishment of a genuine connection between the user and the simulated environment.

Faced with these challenges, we aim to design an egocentric world foundational framework that enables the user being a freeform adventurer. Given a user-provided egocentric image as the world to be explored, it can enable the user to perform unrestricted human movements real-time captured by an exocentric camera and consistent 4D scene modeling in the simulated world. Specifically, we propose the first realistic egocentric world simulator termed PlayerOne. Starting from a diffusion transformer (DiT) model [23], we first extract the latent of an egocentric user-input image. Meanwhile, we select the real-world human motions ( i.e. , human pose or keypoints) as our motion representation. Considering the varying importance of different body parts in our task, the human motion sequence is partitioned into three groups ( i.e. , head, hands, feet and body) and fed into our part-disentangled motion injection to generate latents that can enable precise part-wise control. Additionally, we developed a joint scene-frame reconstruction framework that can progressively complete scene point maps during the video generation process to enable scene-consistent generation. The DiT model takes the concatenation of the first frame latent, motion latent, video latent, and the point map latent as input and conducts noising and denoising on both the video and point map latent. Notably, the point map sequence is not required during inference, ensuring practical efficiency. Moreover, to overcome the absence of publicly available datasets, we curate required motion-video pairs from existing egocentric-exocentric datasets using an automated pipeline designed to filter and retain high-quality data. A coarse-to-fine training strategy is also designed to compensate for the data scarcity. The base model is fine-tuned on large-scale egocentric text-video data for coarse-level generation, then refined on our curated dataset to achieve precise motion control and scene modeling. Finally, we distill our trained model [38] to achieve real-time generation. By integrating these innovations, PlayerOne advances the field of dynamic world modeling. Our contributions are summarized as follows:

We introduce PlayerOne, the first egocentric foundational simulator for realistic worlds, capable of generating video streams with precise control of highly free human motions and world consistency in real-time and exhibiting strong generalization in diverse scenarios.

We design a novel part-disentangled motion injection scheme to enhance fine-grained motion alignment, where a joint scene-frame reconstruction framework is introduced to guarantee world-consistent modeling in long-term video generation as well.

We construct an effective automatic dataset construction pipeline to extract high-quality motion-video pairs from existing egocentric-exocentric datasets, where a coarse-to-fine training scheme is also introduced to compensate for the data scarcity.

2 Related Work

Video generation. The rapid development of diffusion models [27, 14, 41, 3] has driven substantial advancements in video generation. Early researchers [10, 5] adapted existing text-to-image models to

enable text-to-video generation to compensate for the limited availability of high-quality video-text datasets. Subsequently, diffusion transformers based frameworks [37, 23, 17, 12, 30, 29] are proposed. When scaling-up training, they enable more highly realistic and temporally coherent generation results. Among them, HunyuanVideo [17] substitutes T5 with a Multimodal Large Language Model. LTXVideo [12] modifies the V AE decoder to handle the final denoising step and convert latents into pixels. Wan [30] introduces a full spatial-temporal attention to ensure computational efficiency.

World models. Existing world models [13, 35, 36, 7, 1, 34] can be roughly divided into two categories: 1) Agent learning targeted models, 2) World simulation models. For the former, they [13, 35, 34] aim at enhancing policy learning within simulated environments. Among them, Dreamer [13] and DayDreamer [35] solve long-horizon tasks from images purely by latent imagination. MuZero [34] runs the self-play of Monte Carlo tree search to build world models for Atari. Distinct from this direction, world simulation aims to model an environment by predicting the next state given the current state and action. These works focus on human interaction with neural networks through high-quality rendering, robust control, and strong domain generalization to real-world scenarios. With advances in video generation, high-quality world simulation with robust control has become feasible, leading to numerous works focusing on interactive world simulation [9, 26, 19, 36, 7, 1, 28, 6]. Among these works, WORLDMEM [36]. The Matrix [7] proposes the first world simulator capable of generating infinitely long real-scene video streams with real-time, responsive control. Matrix-Game [43] redefines video generation as an interactive process of exploration and creation. Cosmos [1] presents a general-purpose world model and a pre-training-then-post-training scheme. Aether [28] designs a unified framework with synergistic knowledge sharing across reconstruction, prediction, and planning objectives. However, these methods primarily focus on virtual game scenarios and are limited to specific directional actions, rather than facilitating high-degree-of-freedom motion control in real-world environments. To address these limitations, we target at developing a human motion driven realistic world simulator. Given an egocentric image, we can construct a real-scene world that immerses users as freeform adventurers with precise and unrestricted human motion control.

3 Method

In this section, we detail the methodology of PlayerOne. Sec. 3.1 introduces the relevant preliminaries and the overall pipeline. Sec. 3.2 presents the core of our proposed model, followed by Sec. 3.3 introducing our dataset construction and training strategy.

3.1 Overview

Video diffusion models [23, 15] consist of two key processes: a forward (noising) process and a reverse (denoising) process. The forward process gradually adds Gaussian noise, denoted as ϵ ∼ N (0 , I ) , to a clean latent sample z 0 ∈ R k × c × h × w , where k, c, h, and w represent the dimensions of the video latents. This transforms z 0 into a noisy latent z t . In the reverse process, a learned denoising model ϵ θ progressively removes the noise from z t to reconstruct the original latent representation.

As shown in Fig. 2, our method comprises two core modules: Part-disentangled Motion Injection (PMI) and Scene-frame Reconstruction (SR). In PMI, we use real-scene human motion as the motion condition to enable free action control for the user. The first frame is converted into z frame via a 3D VAE encoder. The human motion sequence is split into three parts based on varying importance, and each part is fed into a 3D motion encoder to obtain latents. These motion latents are concatenated into z motion ∈ R k × 3 × h × w . To improve view alignment, we transform the head parameters of the human motion sequence into a camera sequence, which is then fed into a camera encoder. The output is added to the noised video latents z video to inject view-change signals. In SR, we jointly reconstruct video frames and 4D scenes to ensure world-consistent generation in the context of long video generation. We render a point map sequence from the ground truth video and feed it into a point map encoder with an adapter to obtain z point ∈ R k × 64 × h × w . Finally, all latents and conditions are concatenated channel-wise. The training objective of our method can be expressed as:

Where t = 1 , . . . , T , α t 2 + δ t 2 = 1 . Since we only add noise to point map latents and video latents, thus z 0 = z video ⊗ z point . ⊗ denotes the channel-wise concatenation operation, U ( · ) represents a uniform distribution, and T denotes the denoising steps.

3D VAE

Encoder

Human Motion Sequence

First Frame

Noised

Video Latent

❄

Motion

Latent

Noised Point

Map Latent

❄

🔥

Trainable

Frozen

Part-disentangled

Motion Injection

🔥

First Frame

Latent

Di

ff

usion

Transformer

🔥

Scene-frame Reconstruction

🔥

Denoised Latent

Video/Point

❄

Ground Truth

Point Map

Encoder

Point Map

Sequence

Noised Point

Map Latent

Head

Hands

Body&Feet

Decompose

Camera

Encoder

🔥

Motion

Encoder

🔥

Noised

Video Latent

Motion  Latent

Part-disentangled Motion Injection

Scene-frame Reconstruction

Part-wise

Adapter

🔥

❄

Denoising

1 1

11

11

3.2 Model Components

Part-disentangled motion injection. Prior studies [19, 26, 7, 43] typically utilize camera trajectories as motion conditions or are constrained to specific directional movements. These restrictions confine users to passive 'observer' roles, preventing meaningful user interaction. In contrast, our approach empowers users to become active 'participants' by adopting real-world human motion sequences ( i.e. , human pose or keypoints) as motion conditions, allowing for more natural and unrestricted movement. However, our empirical analysis reveals that extracting latent representations holistically from human motion parameters complicates precise motion alignment. To address this challenge, we introduce a part-disentangled motion injection strategy that recognizes the distinct roles of various body parts. Specifically, hand movements are essential for interacting with objects in the environment, while the head plays a crucial role in maintaining egocentric perspective alignment. Accordingly, we categorize the human motion parameters into three groups: body and feet, hands, and head. Each group is processed through its own dedicated motion encoder, comprising eight layers of 3D convolutional networks, to extract the relevant latent features. This specialized processing ensures accurate and synchronized motion alignment. These latents are subsequently concatenated along the channel dimension to form the final part-aware motion latent representation z motion ∈ R k × 3 × h × w .

To further enhance the egocentric view alignment, we solely transform the head parameters of the human motion sequence into a sequence of camera extrinsics with only rotation values. We zero out the translation values in the camera extrinsics, assuming the head parameters are at the camera coordinate system's origin. Specifically, suppose the head parameter v = ( θ x , θ y , θ z ) , we first normalize the rotation axis as follows:

Then we construct the rotation matrix as follows:

Exocentric Video

Egocentric Video

Video Pool

Detection Model

Low Confidence

❌

SMPL Model

Automatic Data Filtering

3D Mesh

SMPL-X

Exocentric Video

OpenPose

2D Keypoints

2D Keypoints

Projection

Keypoint Distance

❌

✓

Small

Large

SMPL

Sequence

-------→

Where u × is the cross product matrix of u , which can be denoted as follows:

Then we use Plücker ray [40] to parameterize the camera extrinsics and then feed the output to an extra camera encoder, which shares a similar structure with the motion encoder. Then the latents from this encoder are added to the noised video latents to inject the view-change information.

Scene-frame reconstruction. While PMI enables precise control over egocentric perspective and motion, it does not guarantee scene consistency within the generated world. To address this limitation, we introduce a joint reconstruction framework that simultaneously models the 4D scene and video frames, ensuring scene coherence and continuity throughout the video. Specifically, it begins by employing CUT3R [31] to generate a point map for each frame based on ground truth video data, reconstructing the n-th frame's point map using information from frames 1 through n. These point maps are then compressed into latent representations using a specialized point map encoder [16]. To integrate these latents with video features, we implement an adapter composed of five 3D convolutional layers. This adapter aligns the point map latents with video latents and projects them into a shared latent space, facilitating seamless integration of motion and environmental data. Finally, we concatenate the latent representations from the first frame, the human motion sequence, the noised video latents, and corresponding noised point map latents. This comprehensive input is then fed into a diffusion transformer for denoising, resulting in a coherent and visually consistent world. Importantly, point maps are only required during the training phase. During inference, the system simplifies the process by utilizing only the first frame and the corresponding human motion sequence to generate world-consistent videos. This streamlined approach enhances generation efficiency while ensuring that the resulting environment remains stable and realistic throughout the entire video.

3.3 Training Strategy

Dataset preparation. The ideal training samples for our task are egocentric videos paired with corresponding motion sequences. However, no such dataset currently exists in publicly available repositories. As a substitute, we derive these data pairs from existing egocentric-exocentric video datasets through an automatic pipeline. Specifically, for each synchronized egocentric-exocentric video pair, we first employ SAM2 [25] to detect the largest person in the exocentric view. The background-removed exocentric video is then processed using SMPLest-X [39] to extract the SMPL parameters of the identified individual as the human motion. To enhance optimization stability, an L2 regularization prior is incorporated. We then evaluate the 2D reprojection consistency to filter out low-quality SMPL data. This involves generating a 3D mesh from the SMPL parameters using SMPLX [22], projecting the 3D joints onto the 2D image plane with the corresponding camera parameters, and extracting 2D key points via OpenPose [4]. The reprojection error is calculated by measuring the distance between the SMPL-projected 2D key points and those detected by OpenPose. Data pairs with reprojection errors in the top 10% are excluded, ensuring a final dataset of high-quality motion-video pairs. The refined SMPL parameters are decomposed into body and feet (66 dimensions), head orientation (3 dimensions), and hand articulation (45 dimensions per hand) components for each frame. These components are fed into their respective motion encoders. The dataset construction pipeline is illustrated in Fig. 3. As detailed in Tab. 1, our training dataset combines multiple publicly available datasets to ensure comprehensive coverage of diverse environmental contexts, action types, and intensity levels, thereby enhancing model generalization.

Ours

No Pretrain

Baseline

Action

Joint-Train

First Frame

First Frame

First Frame

Coarse-to-fine training. Though we can extract high-quality motion-video training data with our automatic pipeline, the limited scale of this dataset is insufficient for training video generation models to produce high-quality egocentric videos. To address this, we harness the extensive egocentric text-video datasets ( i.e. , Egovid-5M [33]). Specifically, we first fine-tune the baseline model using LoRA on large-scale egocentric text-video data pairs, enabling egocentric video generation with coarse-level motion alignment. Then we freeze the trained LoRA and fine-tune the last six blocks of the model with our constructed high-quality dataset to enhance fine-grained human motion alignment and view-invariant scene modeling, which can effectively address the scarcity of pair-wise data. Finally, we adopt an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher [38] to achieve real-time generation and long-duration video synthesis.

4 Experiments

4.1 Experimental Setting

Implementation details. We choose Wanx2.1 1.3B [30] as the base generator. We set the LoRA rank and the update weight of the matrices as 128 and 4 respectively and initialize its weight following [30]. The inference step and the learning rate are set as 50 and 1 × 10 -5 respectively, where the Adam optimizer and mixed-precision bf16 are adopted. The cfg of 7.5 is used. We train our model for 100,000 steps on 8 NVIDIA A100 GPUs with a batch size of 56 and sample resolution of 480 × 480. The generated video runs at eight frames per second, and we utilize 49 video frames (6 seconds) for training. After distillation, our method can achieve 8 FPS to generate the desired results. All the action videos in this paper are shot with the front camera.

Benchmark. Since there is no publicly available benchmark for our task, we construct a benchmark with 100 videos collected from Nymeria [21] dataset, which is not included for training. It consists of coarse-level motion descriptions for each sample and covers diverse realistic scenarios. Considering the information gap between the human motion sequence and the text, we further use Qwen2.5-VL [2] to enrich the caption to generate videos for the competitors for more fair comparisons.

Metrics. On our constructed benchmark, for evaluation of alignment with the given text descriptions, we calculate both CLIP-Score and DINO-Score, where LPIPS [42] is employed to evaluate the video fidelity of the generated video. Besides, we calculate the frame consistency to evaluate the temporal coherence and consistency of the generated video frames over time. We further utilize a 3D hand pose estimation model [24] to estimate the hand pose of the generated videos and use the results of

First Frame

First Frame

Ours

Entangled

ControlNet

Action

No Cam

the ground truth video as the labels. Afterward, we follow [24] to calculate two metrics: (1) Mean Per-Joint Position Error (MPJPE): the L2 distance between the predicted and ground truth joints for each hand after subtracting the root joint. (2) Mean Relative-Root Position Error (MRRPE): the metric distance between the root joints of the left hand and right hand.

4.2 Ablation Study

Investigation on coarse-to-fine training. We first evaluate several variants of our coarse-to-fine training scheme, as depicted in Fig. 4. Specifically, when inputting action descriptions into the baseline model without fine-tuning, the generated results exhibit noticeable flaws, such as hand distortions or the unexpected appearance of individuals. Similar issues can be observed when training with only motion-video pairs. We also explore jointly training with both large-scale egocentric videos and motion-video pairs. Specifically, when inputting egocentric videos, we set the motion latent values to zero and extract the latents of the text description to serve as the motion condition, where a balanced-sampling strategy is used as well. Despite this variant being capable of generating egocentric videos, it fails to produce results accurately aligned with the given human motion conditions. In contrast, our coarse-to-fine training scheme delivers much better outcomes compared to these variants.

Investigation on part-disentangled motion injection. Next, we conduct a detailed analysis of our PMI module. Specifically, three variants are included: ControlNet-based [41] motion injection, inputting motion sequences as a unified entity (the 'Entangled' scheme), and removing our camera encoder. As shown in Fig. 5, the ControlNet-based scheme suffers from information loss, preventing it from producing results that accurately align with the specified motion conditions. Similarly, the entangled scheme demonstrates comparable shortcomings. Furthermore, removing the camera encoder leads to the model's inability to generate view-accurate alignments. As depicted in Fig. 5, this variant fails to produce the corresponding perspective change associated with crouching. Ultimately, our PMI module successfully generates outcomes that are both view-aligned and action-aligned.

Investigation on scene-frame reconstruction. Additionally, we conducted a detailed analysis of the SR module, exploring three variants: omitting reconstruction, removing the adapter within the SR

Ours

Action

No Recon

DUSt3R

No Adapter

First Frame

First Frame

Action

Action

Action

First Frame

Result

Result

Result

First Frame

First Frame

First Frame

First Frame

First Frame

module, and substituting CUT3R [31] with DUStR [32] for point map rendering. As illustrated in Fig. 6, the absence of reconstruction results in the model's inability to generate consistently simulated results. Moreover, due to the distribution gap between latents of frames and point maps, training without the adapter leads to difficulty in loss convergence, causing noticeable distortions. Furthermore, after replacing CUT3R [31] with DUStR [32], our PlayerOne can also produce scene-consistent outputs, demonstrating its robustness to different point map rendering techniques.

Motion alignment. To verify the alignment capability with the given motion condition, we conduct experiments by generating world-simulated videos with the same first frame but different human motion sequences. Fig. 7 shows that our PlayerOne can accurately generate corresponding results according to different conditions and produce reasonable interactive changes.

Quantitative comparisons. We provide quantitative results on the core components of our PlayerOne in Tab. 2. All numerical results concur with the visualization outcomes. A significant performance improvement is observed when the model undergoes pre-training on large-scale egocentric text-video datasets. The introduction of PMI yields an additional accuracy boost, and it outperforms all of its variants. In addition, our designed filtering strategy maximizes performance as well by filtering noisy motion-video pairs. After removing the adapter, the performance suffers from a notable degradation due to the distribution gap between latents of video frames and point maps. By introducing our joint scene-frame reconstruction scheme, we achieve superior results across all metrics.

4.3 Comparison with State-of-the-arts

Quantitative comparison. Since there is no method sharing the same setting as ours, we selected two potential competitors for comparison: Cosmos [1] and Aether [28]. As shown in Tab. 3, our

Descriptions: First-person perspective, I stretched out my right hand from below and took out the milk in the middle of the freezer.

Descriptions: First-person perspective, I stretch out my hands to turn the steering wheel to the right.

First Frame

First Frame

Ours

Aether

Cosmos-14B

Cosmos-7B

Action

PlayerOne outperforms all the baselines by large margins, especially on the metrics of motion alignment. Notably, Cosmos [1] exhibits better generalization ability than Aether [28] by explicitly capturing general knowledge of real-world physics and natural behaviors. Besides qualitative results, we provide visualization comparisons in Fig. 8, where consistent superiority can be observed in diverse scenarios for both user interaction and world modeling.

User study. In Tab. 4, we report the comparison results of human preference rates. We let 20 annotators rate 25 groups of videos, where each group contains the generated video of each method and text description. And we provide detailed regulations to rate the results for scores of 1-4 from four views: 'Quality', 'Smooth', 'Fidelity', 'Alignment'. 'Quality' counts for whether the result is harmonized without considering fidelity. 'Smooth' assesses the motion consistency across the video. 'Fidelity' measures ID preservation and distortions within the video, while we use 'Alignment' to measure the alignment with the given text descriptions. It can be noted that our model demonstrates significant superiority across all the metrics, especially for 'Alignment', and 'Smooth'.

5 Conclusion

In conclusion, PlayerOne represents a significant advancement in interactive and realistic world modeling for video generation. Unlike conventional models that are restricted to particular game scenarios or actions, our PlayerOne can capture the complex dynamics of general-world environments and enable free motion control within the simulated world. By formulating world modeling as a joint process of videos and 4D scenes, our PlayerOne ensures coherent world generation and enhances motion and view alignment with the given conditions through part-disentangled motion injection. Experimental results demonstrate our superior performance across diverse scenarios.

Limitations. Despite the compelling outcomes, our performance in game scenarios is slightly inferior to realistic ones, likely due to the imbalanced distribution between realistic and game training data. It can be addressed by incorporating more game-scenario datasets in future research.

Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 62441615, 62422606, 62201484) and DAMO Academy via DAMO Academy Research Intern Program.

References

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv:2501.03575 , 2025. 2, 3, 8, 9, 23, 27

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv:2309.16609 , 2023. 6

Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer. Retrieval-augmented diffusion models. In NeurIPS , 2022. 2

Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. TPAMI , 2019. 5

Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv:2305.13840 , 2023. 2

Junhao Cheng, Yuying Ge, Yixiao Ge, Jing Liao, and Ying Shan. Animegamer: Infinite anime life simulation with next game state prediction. arXiv:2504.01014 , 2025. 3

Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. arXiv:2412.03568 , 2024. 2, 3, 4, 21

Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR , 2024. 6, 27

Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft. arXiv:2504.08388 , 2025. 2, 3, 21

Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR , 2024. 2

Zihui Guo, Yonghong Hou Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li. Ft-hid: A large scale rgb-d dataset for first and third person human interaction analysis. Neural Computing and Applications , 2022. 6, 27

Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv:2501.00103 , 2024. 2, 3

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv:1912.01603 , 2019. 2, 3

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arxiv:2006.11239 , 2020. 2

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv , 2022. 3

Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, and Andrea Vedaldi. Geo4d: Leveraging video generators for geometric 4d scene reconstruction. arXiv:2504.07961 , 2025. 5

Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv:2412.03603 , 2024. 2, 3

Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng. Egoexo-fitness: towards egocentric and exocentric full-body action understanding. In ECCV , 2024. 6, 27

Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Konstantinos N. Plataniotis, Sergey Tulyakov, and Jian Ren. Wonderland: Navigating 3d scenes from a single image. arXiv:2412.12091 , 2024. 2, 3, 4, 23

Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, and Richard Newcombe. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In ECCV , 2024. 6

Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In ECCV , 2024. 6, 27

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR , 2019. 5

William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv:2212.09748 , 2022. 2, 3

Aditya Prakash, Ruisen Tu, Matthew Chang, and Saurabh Gupta. 3d hand pose estimation in everyday egocentric images. arXiv:2312.06583 , 2024. 6, 7

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. In ICLR , 2025. 5

Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed worldconsistent video generation with precise camera control. arXiv:2503.03751 , 2025. 2, 3, 4, 23

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv:2112.10752 , 2021. 2

Aether Team, Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, and Tong He. Aether: Geometric-aware unified world modeling. arXiv:2503.18945 , 2025. 2, 3, 8, 9, 27

Genmo Team. Mochi 1. https://github.com/genmoai/models , 2024. 3

Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv:2503.20314 , 2025. 2, 3, 6, 27

Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arXiv:2501.12387 , 2025. 5, 8

Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR , 2024. 8

Xiaofeng Wang, Kang Zhao, Feng Liu, Jiayu Wang, Guosheng Zhao, Xiaoyi Bao, Zheng Zhu, Yingya Zhang, and Xingang Wang. Egovid-5m: A large-scale video-action dataset for egocentric video generation. arXiv:2411.08380 , 2024. 6, 27

Aurèle Hainaut Werner Duvaud. Muzero general: Open reimplementation of muzero. https: //github.com/werner-duvaud/muzero-general , 2019. 2, 3

Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. In CoRL , 2022. 2, 3

Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory. arXiv:2504.12369 , 2025. 2, 3, 21

Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformer. arXiv:2410.10629 , 2024. 3

Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In CVPR , 2025. 2, 6

Wanqi Yin, Zhongang Cai, Ruisi Wang, Ailing Zeng, Chen Wei, Qingping Sun, Haiyi Mei, Yanjun Wang, Hui En Pang, Mingyuan Zhang, Lei Zhang, Chen Change Loy, Atsushi Yamashita, Lei Yang, and Ziwei Liu. Smplest-x: Ultimate scaling for expressive human pose and shape estimation. arXiv:2501.09782 , 2025. 5

Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. In ICLR , 2024. 5

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2, 7

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. arXiv:1801.03924 , 2018. 6

Yifan Zhang, Chunli Peng, Boyang Wang, Puyi Wang, Qingcheng Zhu, Zedong Gao, Eric Li, Yang Liu, and Yahui Zhou. Matrix-game: Interactive world foundation model. arXiv , 2025. 3, 4

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to the contributions in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the limitations in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the assumptions and experimental results in Section 3 and Section 4.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to the implementation details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to the implementation details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to the experimental results in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to the implementation details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, it conforms to the requirements.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The scope of our research does not involve data or models with a high risk for misuse. We utilize publicly available, non-sensitive datasets and models that do not require special safeguards for responsible release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This question does not apply to our research as no new assets were introduced. Our study solely relies on existing, publicly available resources, and thus, no additional documentation for new assets is necessary.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This question does not apply to our research as it does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This question is not applicable to our research as no human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We only use the LLM for writing, editing, or formatting purposes.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Appendix Overview

This appendix complements the main paper with additional results, analyses, and implementation details:

Visualizations and reconstructed scenes. Qualitative roll-outs and scene reconstructions in diverse scenarios (Figs. 9, 10, 11-13, 14).

Implementation notes. Clarifications of the training/inference pipeline (world/scene consistency is preserved at inference), superiority over prior video diffusion models, and detailed distillation settings.

A More Experimental Results

Framework comparisons with prior works. Fig. 10 illustrates the framework comparison between our PlayerOne and other competitors. Prior studies [9, 36, 7] have mainly concentrated on simulations within game-like environments, yet they often fail to accurately replicate real-world scenarios. Within these simulated environments, users are typically restricted to performing predefined actions, such

🚗

⬆

⬇

➡

⬅

User

Limited Action

Video

Text / Camera Sequence

Observer

Observation

(No Interaction)

(a)

(b)

(c)

User

Human Action Capture

Real-time Interactive Video

PlayerOne

------→

Aether

Cosmos-14B Cosmos-7B

Ours

Ours

Aether

Cosmos-14B Cosmos-7B

Descriptions: First-person perspective, I stroll forward.

First-person perspective, I stretch out my left

hand and high-fived the man opposite me.

Action

Action

Descriptions: First-person perspective, I sit down and put

my hands on the chair.

Descriptions: First-person perspective, I stretch

my right hand to touch the dog.

as directional movements. This limitation confines user interactions to a constructed world, thereby

Aether

Cosmos-14B

Cosmos-7B

Ours

Descriptions: First-person perspective, I walked forward

and stretched out my left hand to hold the door handle.

Descriptions: First-person perspective, I stretched out my

left hand and high-fived the cat opposite me

Action

Aether

Cosmos-14B Cosmos-7B

Ours

Descriptions: First-person perspective, stretching out my right hand to shake hands

with the man opposite.

Descriptions: First-person perspective, stretch out the left hand to high-five the man

opposite.

Action

restricting the execution of freeform movements akin to those in real-world settings. Existing realistic world simulators [19, 26, 1] often focus solely on world-consistent generation, lacking mechanisms for human movement control. As a result, users are relegated to passive observers, rather than active participants, within the environment. This significantly impacts the user experience by hindering the

Aether

Cosmos-14B Cosmos-7B

Ours

Descriptions: First-person perspective, I extended my left hand to

shake hands with the woman opposite me.

Descriptions: First-person perspective, I stretched out my left hand and high-fived the golden retriever.

Action

formation of a genuine connection with the simulated world. In contrast to these limitations, our approach enables freeform motion control for users, enhancing their interactive experience.

Why separate processing contributes to precise alignment? Splitting the latent representation into three separate parts-head, hands, and body/feet-enables the model to more effectively capture the unique and often semi-independent motion characteristics of each region. Human body motion is inherently high-dimensional and highly articulated, with different parts frequently moving independently or exhibiting distinct dynamics. Encoding all motion information into a single, entangled latent can cause fine-grained cues-especially from subtle or fast-moving regions like hands or head-to be overshadowed by larger body movements, making precise alignment difficult. By disentangling the motion into part-specific latents, each body region's movement can be independently modeled and aligned with the corresponding input. This modular approach allows the model to better capture local nuances, improves compositionality, and enhances control over complex motions, resulting in higher motion fidelity and visual quality.

Latency between motion input and video output. After distillation, the measured end-to-end inference latency is approximately 119 milliseconds per generated-frame, corresponding to about 8.4 FPS. This latency is measured from the moment the motion input is received to the generation of the corresponding video frame, and includes all major processing steps.

Details of distillation. We strictly follow the Causvid distillation procedure in our implementation. During the distillation stage, we use the entire SMPL-Video paired dataset from the second phase of training. Specifically, we first generate 8,000 ODE pairs and train the student for 30,000 iterations using AdamW with a learning rate of 5 × 10 -6 . We then continue training with our asymmetric

DMDloss for another 30,000 iterations using AdamW with a learning rate of 2 × 10 -6 . A guidance scale of 3.5 is used, and we adopt the two time-scale update rule from DMD2 with a ratio of 5. The entire distillation process takes about 5 days on 32 A800 GPUs.

Details of training pipeline. We clarify that the world/scene consistency mechanism is not removed at inference. During training, a CUT3R-like reconstructor is used only to produce ground-truth pointmap sequences corresponding to each video frame; these point maps supervise our scene-frame reconstruction loss and are further encoded to build noised point-map latents. In both training and inference, the model inputs are always the initial egocentric frame and the human-motion sequence, while the prediction targets are the video frames together with the point-map sequence.

Concretely, let the raw video be V ∈ R B × K × 3 × H × W and its point maps P ∈ R B × K × N × 3 . After VAE encoding, the video becomes z 0 video ∈ R B × k × c × h × w with h = H/ 8 , w = W/ 8 , k = K/ 4 . A 3D encoder plus an adapter maps the point maps to z point ∈ R B × k × 64 × h × w , matching the video latent's spatio-temporal shape. Both z video and z point are noised to obtain z ′ video and z ′ point for denoising training. The motion sequence, passed through our motion encoder, yields z motion ∈ R B × k × 3 × h × w . The first-frame latent z first ∈ R B × 1 × c × h × w is repeated k times so that z k first ∈ R B × k × c × h × w . We then concatenate along channels:

A DiT backbone denoises z input into z output (same shape). Losses are computed between the predicted ( z ∗ video , z ∗ point ) and the ground-truth ( z video , z point ) latents via an MSE objective.

Inference. At test time there are no ground-truth point maps and thus no 3D point-map encoder is required. Instead, the network jointly predicts the video and its point-map latents autoregressively: both streams are initialized from noise and co-denoised frame by frame, conditioned on the initial frame and the motion sequence. Because the video and point-map streams are modeled with identical spatio-temporal layouts and are denoised in lockstep, the scene-consistency module remains fully active during inferenceno extra inputs are needed and no module is removed . In short, training uses point maps to supervise and align the latent spaces; inference keeps the joint two-stream denoising (video + point maps) to maintain world consistency throughout generation.

Superiority over previous video diffusion models. Our approach advances egocentric video generation and simulation in several complementary ways that prior video diffusion systems do not address. (i) Scene consistency and world modeling. Instead of treating frames almost independently-which often causes background flicker, object drift, and abrupt scene changes-our method explicitly reconstructs and maintains a unified 3D world along the entire sequence. We jointly predict the video stream and a point-map stream under an explicit camera representation, so every generated frame is geometrically aligned to an evolving scene state; this yields stable, immersive first-person experiences over long autoregressive roll-outs. (ii) Accurate and fine-grained first-person control. Existing methods commonly rely on a single, entangled pose/control vector and therefore cannot model the different roles of head, hands, and feet in egocentric interaction. By employing part-disentangled motion injection, our system exposes independent, precise controls for head orientation (viewpoint/world alignment), hand motion (object interaction), and foot placement (navigation realism), enabling faithful user-driven control and correct coupling between the actor and the rendered environment. (iii) Efficient and real-time generation. Rather than invoking heavy 3D encoders at inference, we align modalities in the latent space using lightweight adapters inside the scene-frame reconstruction pathway, and we deploy a distilled backbone for decoding. This design removes bulky inference-time modules and substantially reduces compute/memory cost, enabling high-quality egocentric video at interactive rates suitable for VR/AR and gaming scenarios. (iv) Comprehensive validation. We adapt several strong video-diffusion baselines to the egocentric setting under a unified protocol and evaluate them on a reconstructed 100-video benchmark and an extended 24 s horizon. Across perceptual (DINO/CLIP), geometric (MPJPE/MRRPE), and temporal/fidelity (FVD/LPIPS) metrics, our model consistently outperforms adapted baselines, confirming that the gains come from principled world modeling and fine-grained control rather than from model size alone. Collectively, these ingredients make PlayerOne a practical, accurate, and controllable egocentric world simulator, rather than a generic short-clip generator.

Limitation analysis on our model structure. From the view of the model, one current limitation of our model is the lack of explicit physical interaction modeling between the human body and the

environment. Structurally, our framework conditions video generation on SMPL motion sequences and a static point cloud reconstruction of the scene, but does not incorporate physical constraints or interaction modules such as collision detection, physics engines, or contact reasoning. Consequently, our model may generate unrealistic results, such as hands or body parts penetrating objects, or a lack of proper occlusion and collision feedback. We acknowledge this limitation and consider it a promising direction for future work.

More visualization results. Here we provide more simulated results in Fig. 11 and Fig. 12. In terms of first-person action alignment and world consistency, we have achieved outstanding results in both game and real-world scenarios. Additionally, we selected highly dynamic settings, such as driving scenes, where our method successfully models the world with high accuracy while maintaining excellent video fluidity. More visualization results can be referred in the submitted video.

Visualization of the scene reconstruction. This section presents a comprehensive visualization of the reconstructed scenes using our PlayerOne. As illustrated in Fig. 9, our approach adeptly reconstructs both scenes and video frames through a progressive methodology. This ensures not only inter-frame coherence but also overarching scene consistency across a diverse array of scenarios. Specifically, the method seamlessly integrates temporal and spatial elements to maintain visual congruity, even in complex environments. The robustness of our technique is further reflected in its capacity to adapt to varying scene dynamics and compositions, thereby offering a reliable framework for generating high-quality, consistent video outputs. Through these visualizations, the effectiveness of our PlayerOnein achieving smooth and coherent reconstructions is clearly demonstrated, highlighting its potential applications in advanced graphical simulations and interactive environments.

Visual and quantitative difference with the "No Adapter" and "No Recon" setting. The large performance gap for 'no adapter' comes from a mismatch in the feature spaces: our point map encoder is fixed, and its latent space is not directly aligned with the V AE-encoded video latent space. When these are simply concatenated without a dedicated adapter, the model struggles to reconcile the different statistics and semantics between the two types of latents, resulting in degraded video quality and scene breakdown. Our adapter module learns to map point map features into the appropriate latent space for effective fusion with video latents, thus improving both fidelity and scene consistency. This design is crucial for effective information transfer between geometry and appearance. For 'no recon', it is primarily due to two reasons:

Strong Pretraining: Our model is first pretrained on a large-scale egocentric video dataset. This pretraining stage may equips the model with a weak ability to model scene dynamics and maintain a certain level of spatial consistency, even before introducing explicit sceneframe reconstruction.

Lack of Direct Metrics: Most commonly used metrics (e.g., DINO, CLIP, FVD, LPIPS) are not designed to specifically measure scene (world) consistency across frames. These metrics primarily evaluate frame-wise visual similarity, semantics, or overall diversity, but do not directly capture temporal geometric consistency, which is the core advantage of our reconstruction module.

Definition of evaluation criteria. For user study, we evaluate the video quality from four views:

Quality: Defined as the overall visual harmony and realism of the video, regardless of whether the generated content is perfectly true to the input. Annotators were asked: 'Does the generated video look natural, coherent, and free from jarring artifacts?'

Smoothness: Measures how consistently the motion flows across frames, without abrupt transitions or temporal artifacts. Annotators were prompted: 'Are the motions and camera transitions fluid and continuous throughout the video?'

Fidelity: Refers to the preservation of the subject's appearance (identity, clothing, background) and the absence of distortions or glitches. Annotators were asked: 'Does the person look like the initial frame and is there minimal distortion?'

Alignment: Defined as how closely the generated motion matches the intended action described in the motion condition. For this criterion, annotators viewed both the generated

video and the motion condition signal (stick-figure or pose sequence) side-by-side for all 100 videos, and rated: 'Does the generated video accurately follow the given pose/motion signal in timing, type of action, and spatial positioning?'

B Broader Impact

The proposed PlayerOne for video generation, designed to facilitate freeform human motion control within environments created from user-provided images while producing world-consistent videos, demonstrates considerable potential across diverse domains. It is particularly adept at generating engaging and dynamic educational content, thereby fostering experiential learning through interactive simulations. Moreover, the model optimizes the production of high-quality, consistent visual content for films, television, and online media, dramatically reducing both production time and costs. It also enables the creation of interactive narratives, allowing users to influence the storyline through their interactions within the generated environments, thus enhancing user engagement and narrative immersion. Beyond these applications, the world model serves as a valuable tool for research on human behavior and interactions within controlled virtual settings, offering insights for fields such as psychology, sociology, and human-computer interaction. By integrating these capabilities, the proposed world model not only amplifies existing applications but also paves the way for novel research and development, significantly contributing to technological progress and societal advancement.

C Limitations & Discussion

While significant strides have been made in egocentric interaction and coherent world modeling, certain limitations persist. Despite the compelling outcomes, the performance in game scenarios is somewhat diminished compared to realistic scenarios, likely due to the disproportionate amount of realistic training data available. Moreover, in highly dynamic scenes, predictions may falter, reflecting the constraints inherent in the current base model. Future research endeavors could potentially overcome these challenges by investigating novel action representations, incorporating an expanded dataset for game scenarios, and adopting a more robust base model.

D License of assets

Datasets (Apache 2.0 License) Nymeria [21]/FT-HID [11]/EgoExo-Fitness [18] (Creative Commons Attribution 4.0 International), EgoExo4D [8]/Egovid-5M [33](MIT License).

Codes The official repository of Aether [28] (MIT License), the official repository of Cosmos [1] (Apache 2.0 License), the official repository of Wan2.1 [30] (Apache 2.0 License).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PlayOne, an egocentric world model that offers fine-grained control to generate egocentric videos conditioned on motion extracted from exocentric videos.\n\nThe paper proposes a novel part-disentangled motion injection method as its main contribution. The part-disentangled motion injection scheme encodes motion into three separate sets of body parts to obtain three separate latent representations. The authors claim this leads to improved video alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Controllable video generation via egocentric pose\n- Novel idea of using exocentric videos as a conditional signal\n- A pipeline for novel dataset generation\n\nWeaknesses:\n- The paper is not clearly written; it's hard to understand the architecture, the research gap, and the motivation behind the model design\n- Some claims are unsupported or lack proper citations\n- Comparison to state-of-the-art models like GEM appears to be missing\n- It remains unclear how the third-person exocentric view is used beyond motion extraction\n- The need for the proposed dataset is unclear, given the existence of datasets that already pair egocentric views with local body pose\n- Many concepts and ideas are introduced abruptly, making the paper difficult to follow\n- The novelty claims are questionable. The authors often claim to be the first, despite the existence of similar prior work\n\nQUESTIONS:\nWill the dataset be released?\n\nAll other short comments and Questions:\n\n1 - not the first egocentric simulator: GEM, EgoSim, EgoGen, ...\n\n16-18: sounds very LLM written, many words no real meaning, please be more direct what does this work offer the community? Why is it the first work?\n\n20-28: paragraph should be adapted for better understanding of the authors problem. How do the authors define a world model? \n\n25: citations needed or better explanation why these advancements help\n\n26: predictive modeling not explained yet, what is meant by that?\n\n27-28: again not clear why almost these rather novel and less established world models are critical infrastructure. citations or explanation needed\n\n29: unclear what this topic refers to\n\n30: there is a lot of works on realistic scenarios, e.g. GEM, VISTA\n\n31: why is that which works limit users in that sense?\n\n32: GEM seems to allow arbitrary poses drawn as skelton\n\n39-60: very hard to follow. the problem statement is still unclear, and also the research gap is not yet clear to me. Then i get swamped with many detailed information that are very hard to follow.\n\n61: still unclear why its the first, there are some before, so why is yours the first, maybe rather mention on how you improve on previous works\n\n67: do you release the dataset? do you release training code?\n\n106: is k number of frames?\n\n99: the method section can benefit a lot by clearly stating the problem statement. what are the inputs, what are the outputs\n\n109-120: very unclear how this paragraph is connected to its context, e.g. the previous paragraph\n\n109-110 the goal of the core modules is unclear and are introduced without motivation\n\n112: VAE is suddenly introduced. What does it encode? How is related to the human motion sequence? How is it connected to the 3D motion encoder\n\n114: why are the motion latents teh same shape as images? How is that ensured? Why is that high resolution/dimensional encoding needed?\n\n115: What is a camera sequence?\n\n117: how if a 4d scene represented?\n\n118: what is a point map sequence?\n\nEq 1: why is t in U(0,1) but not U(1, T)\n\n121: it is unclear where alpha^2 + delta^2 = 1 comes from\n\n130: where is that empirical analysis?\n\n132: why is splitting the latent into 3 separate latents allowing for better motion alignment? Are those body part poses not highly correlated and thus capturing it in a single latent should be superior?\n\n137-138: completely unclear why separate processing ensures this claim\n\n143: maybe write the axis angles different, i have never seen thetas (angles) being used to represent the rotation axis, seem like you are using euler angles and not a axis representation.\n\n152: still unclear how the 4D scene is represtanted\n\n154: CUT3r not really itnroduced before\n\n158: how is the alignment happening\n\n163-165: how is scene consistency insured during inference if the pointsmaps dont get updated?\n\n169: incorrect. There is nymeria, and also egoExo4D has sequences of egocentric video with ego motion gt, Also there is the EgoSim dataset\n\nMethod: how is the exoview used besides for motion extraction?\n\n4.2 why do the ablations studies come before the main results?\n\n232: do you ensure that the computational complexity of a joint motion encoder is the same as the 3 individual encoders? What are teh dimensions of the latents?\n\n4.3) why are other sota models like GEM not included?\n\nDiscussion and Limitations very short and not detailed enough.\n\nReferences:\n\nGEM - https://arxiv.org/html/2412.11198v1\nVISTA - https://arxiv.org/pdf/2405.17398\nEgoSim - https://proceedings.neurips.cc/paper_files/paper/2024/file/c1017d0a006d31dfbfd4cf1e9189d747-Paper-Datasets_and_Benchmarks_Track.pdf\nEgoGen - https://openaccess.thecvf.com/content/CVPR2024/papers/Li_EgoGen_An_Egocentric_Synthetic_Data_Generator_CVPR_2024_paper.pdf\nNymeria - https://arxiv.org/pdf/2406.09905\n\nLIMITATIONS:\nLimitations and Discussions are discussed very sparsely. The authors spend less than three lines to discuss the limitations", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method called PlayerOne that takes an egocentric image of a scene together with the exocentric video of a person doing some body motions, and provides as output a synthesized video where the actions from the person in the exo video are mapped into the ego frame.  Both the body movements (typically hands and forearms) plus the surrounding scene are aimed to appear in sync in the output video.  The method is a diffusion model that decomposes the body motion into body regions/parts, represents the camera rotation separately, and also inputs a scene representation for the ego frame using a point map encoder.  The DiT diffusion transformer model learns to condition the output video on these factors.   Results are quantified by the alignment between the output video and its ground truth text descriptions, as well as the hand pose error against the ground truth video.  Baseline models are Cosmos and Aether.  Qualitative examples are shown in the paper and supp video.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Interesting problem definition, which seems to offer a new twist on conditional video generation.  The approach could be used by artists to guide the ego content in video games or movies to a target interaction pose.  (The authors could spend a few sentences somewhere to motivate where they see this being most useful though.)\n\n-\tApproach seems well designed.\n\n-\tResults are nice looking.\n\n-\tWriting is overall good and flows well with informative figures (but see note below on details that were not clear.)\n\n- I consider this between BA and A, pending questions below being clarified.\n\nWeaknesses:\n\n-\tMotivation of technical design was not always apparent:\n\no\tFor example, why is the camera translation removed?  This seems like a limitation on which body motions can be realistically received as input and then shown in the output.  (around L140)  \n\no\tIt is not clear how the scene frame alignment step works.  How does the adapter have the signal to align the point map latents with video latents?  (L158)  What enforces the alignment, what is the loss.\n\n-\tThe related work coverage seems fine, but I would have liked more explicit and precise call outs about what is distinct and what is important+distinct in this work.  For example, it says the existing methods “primarily” focus on virtual game scenarios and are limited to specific directional actions.  If it is only “primarily”, then what are the exceptions and how do they compare to this model?  What do the authors mean exactly by “specific directional motions”?  Do any of the existing methods also condition on human body pose or is that entirely new?  If they do, what’s insightful about how it’s done here instead?\n\n-\tL130  about adopting real-world human motions as motion conditions allowing more natural movement – but the movement is still camera (head) motion, just now we see the body parts overlayed as well.\n\n-\tTechnical novelty – choices seem reasonable but not surprising.  For me, the interesting nature of the specific setting is what is strong about the paper.\n\n-\tI don’t think the steps towards end of Sec 3 are reproducible given what’s written, e.g., about the distillation and the finetuning.\n\n-\tI could not tell from the model design whether it’s expected that the input exo body poses are “projected” to the nearest visually suitable poses in the ego scene thanks to the diffusion model?  Is this intended and is it observed?  For example, if the person is making petting motions in the exo video and they would not align well with the way the dog is posed in the ego frame, is this corrected?  Or should the input poses be very precise to match the target scene (and is this why the self-taken videos are used for qual results?)\n\n-\tIn the results:\n\no\tIt is not explained why the Cosmos and Aether baselines were chosen and why they are complete set of baselines to support the paper’s claims.\n\no\tWhy is Wanx 2.1 the right base model?  How does that interact with the choice of baselines as well in terms of fairness, apples to apples.\n\no\tBenchmark samples from Nymeria – why only from this datatset?  Would seem more complete to sample unseen examples across all the datasets.\n\no\tThe exo videos in the supp seem to be taken by the authors, not from Nymeria.  Unclear how the qual and quant results match up.\n\no\tWhere do the text descriptions come from that are used to evaluate ground truth?\n\no\tWhat about arms or any other visible body parts for the MPJPE metric, vs. just hands?\n\no\tWhat if the hands are not visible in the frame of the proposed method’s output?  Hopefully those instances are not dropped from evaluation.\n\nMinor:\n\n-\tFragment L89 “Among those works…”\n\n-\tIn the supp video it would help to show the ego frame input for each one.  Also it would be nicer to show one video at a time throughout since a person can’t watch multiple videos at once.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\n-\tPlease see questions about clarity above, whose answers could help me better state the limitations.\n-\tThe authors comment on limitations at the end, though I think they could elaborate on aspects of the model design that may cause issues (not just the data shortage).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PlayerOne,  a novel world simulator that allows users to control human motion in real-time within dynamic, user-generated environments. By focusing on egocentric video generation and precise motion control, it creates interactive simulations with high visual and motion consistency. PlayerOne excels in both motion alignment and environmental interaction across diverse scenarios, outperforming previous models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper is well-motivated with a clear methodology and good presentation.\n- The key innovation of this paper is its task setting: generate high-fidelity, dynamic, ego-centric worlds where users can flexibly interact with. \n- Adequate qualitative visualizations in both the main paper (Fig 6,7,8) and the supplementary video.\n- High-quality data collection pipeline. As shown in Fig.3, the authors developed a data collection pipeline with the integration of pre-developed detection and pose estimation models and an automatic filtering scheme.\n\nWeaknesses\n- Lack of analysis for failure mode. Analyzing failure modes is a critical aspect of research in generative models to understand the weakness of the learned model in certain usages. However, such a discussion and visualization of the failure mode is missing.\n\n- Limited temporal horizon of generation. According to line 204-205 and the visualizations in the supplementary files, it seems that the proposed model can only synthesize a 6-second video sequence. It seems that current method does not support longer generation beyond the training horizon.\n\n- Insufficient details about distillation. According to line 195-196, the authors applied a distillation stage in the end of model training to improve the inference efficiency. However, details like how the distillation would affect the quality of generation and comparison between distilled and non-distilled model are not included.\n\nQUESTIONS:\nBased on the weaknesses above, the authors are encouraged to revise this article in the following aspects:\n1) Analyze the failure model of the learned model and provide some corresponding visualizations.\n2) Add more discussion about the temporal horizon of generation. Is generating longer videos necessary for this task? How the result  and visualization quality would be if we extend the inference horizon beyond the one in training (6s)? Will extending the horizon be one of a future directions?\n3) About the distillation. Please consider adding more comparisons to demonstrate how would the distillation stage affect the model performance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PlayerOne, a novel framework for egocentric world simulation that generates immersive videos aligned with real-world human motion. Given a single egocentric image and an exocentrically captured motion sequence, the system constructs a consistent 4D virtual world, enabling freeform and fine-grained human action control in the generated video. The proposed architecture builds upon a video diffusion transformer, with key innovations including a part-disentangled motion injection module for precise body part control and a joint scene-frame reconstruction framework to enforce long-term scene consistency. To address the lack of suitable training data, the authors design an automated pipeline to extract high-quality motion-video pairs from egocentric-exocentric datasets, combined with a coarse-to-fine training strategy leveraging large-scale egocentric text-video data. The system demonstrates strong performance across multiple metrics and outperforms relevant baselines such as Aether and Cosmos.\n\nSTRENGTHS AND WEAKNESSES:\n##  Strengths\n- Novelty \n\nThe idea of simulating egocentric immersive worlds controlled by real human motion is novel and underexplored.\n- Technical Contributions\n\nThe paper presents a comprehensive automatic data construction pipeline, a pragmatic contribution to the field.\n\n- Clarity\n\nThe paper is well-organized.\n\n## Weaknesses\n- Consistency in Long-Video Generation\n\nWhether using point clouds as implicit encodings to constrain the consistency of generated results is relatively weak. For the egocentric simulator task described in the paper, the user is expected to freely explore within the generated scene, which requires the model’s capability to generate long-term video sequences. It seems that the current point cloud encoding approach may not adequately ensure consistency across multiple concatenations when generating long videos.\n\n- Limited Discussion of Failure Cases:\n\nThe paper notes weaker performance in game-like scenarios but does not deeply investigate or visualize these limitations.\n\nQUESTIONS:\n- The length of video generated by the model in one pass, how long videos are generated through concatenation, and whether the point cloud encoding scheme can ensure consistency across the generated video sequences.\n\n- The paper claims real-time performance post-distillation, but only FPS is reported. Could the authors clarify the actual latency between motion input and video output? This would help gauge deployment feasibility in interactive settings.\n\nLIMITATIONS:\nThe paper notes weaker performance in game-like scenarios but does not deeply investigate or visualize these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This work proposes an egocentric realistic world simulator that takes a scene image and human motion as input to generate corresponding egocentric videos. The core of the method is a novel video diffusion model incorporating a Part-Disentangled Motion Injection module, which separately encodes motion signals from different body parts to guide video generation. Additionally, a Scene-Frame Reconstruction mechanism is introduced to maintain scene consistency. For training, the paper designs a pipeline to extract motion-video pairs from existing egocentric-exocentric video datasets and further leverage other egocentric datasets to train the video generation model.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This paper presents a novel video diffusion model featuring an innovative module, along with a well-designed dataset collection and training pipeline. The writing is clear and well-structured.

2. The experiments demonstrate significant improvements over the baseline, and the ablation studies effectively highlight the contribution of each component.

Weaknesses and Questions:

1. Could you elaborate on the decision to use 3 body parts? Why not 5 or 6, which might encode more detailed information about body movement?

2. In Figure 7, what causes the failure in the second motion condition?

3. Regarding the visualization results involving interactions with other people or animals—are there examples of actions beyond handshakes or basic interactions?

QUESTIONS:
See weakness.

LIMITATIONS:
The generated videos still exhibit noticeable artifacts, particularly during fast or complex motion. Additionally, the work does not explore the potential benefits of incorporating more images or scene information as input to the model.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02