EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
Authors: Siyuan Huang
Liliang Chen
Pengfei Zhou
Shengcong Chen
Yue Liao
Zhengkai Jiang
Yue Hu
Peng Gao
Hongsheng Li
Maoqing Yao
Guanghui Ren
Venue: NeurIPS 2025 poster ()
Keywords: Manipulation
robotics
World Models
Primary area: applications
TL;DR: We introduce EnerVerse, a generative robotics foundation model that constructs and interprets embodied spaces.

Abstract: We introduce EnerVerse,  a generative robotics foundation model that constructs and interprets embodied spaces. EnerVerse employs a chunk-wise autoregressive video diffusion framework to predict future embodied spaces from instructions, enhanced by a sparse context memory for long-term reasoning. To model the 3D robotics world, we adopt a multi-view video representation, providing rich perspectives to address challenges like motion ambiguity and 3D grounding. Additionally, EnerVerse-D, a data engine pipeline combining generative modeling with 4D Gaussian Splatting, forms a self-reinforcing data loop to reduce the sim-to-real gap. Leveraging these innovations, EnerVerse  translates 4D world representations into physical actions via a policy head (EnerVerse-A), achieving state-of-the-art performance in both simulation and real-world tasks. For efficiency, EnerVerse-A reuses features from the first denoising step and predicts action chunks, achieving about 280 ms per 8-step action chunk on a single RTX 4090. Further video demos, dataset samples could be found in our project page.

VNERV:RSE: Envisioning Embodied Future Space for Robotics Manipulation
Siyuan Huang 1.3, Liliang Chen? 1, Pengfei Zhou', Shengeong Chen', Yue Liao", Zhengkai Jiang", Yue Hu?, Peng G Hongsheng Li Maoqing Yao Guanghui Ren?
"STU AgiBot "Shanghai AI Lab *CUHK MMLab LLVNUS Lab
Project Page: stttps:/sstte google wwwe//eeeeeereeeeeeeeeeeeeeeeeee Email: yaomaoqing wwg.got.@iiiooocooooooooo renguanghui @ @ggiott.com
Abstract
We introduce ENERVERSE, a generative robotics foundation model that constructs and interprets embodied spaces. ENERVERSE employs chunk-wise autoregressive video diffusion framework to predict future embodied spaces from instructions, enhanced by a sparse context memory for long-term reasoning. To model the 3D robotics world, we adopt multi-view video representation, providing rich perspec- tives to address challenges like motion ambiguity and 3D grounding. Additionally, ENERVERSF-D, a data engine pipeline combining generative modeling with 4D Gaussian Splatting, forms a self-reinoorcing data loop reduce the sim-to-real gap- Leveraging these innovations, ENERVERS, ERSE translates 4D world representations into physical actions via a policy head (ENERV ERSE-A), achieving state-of-the-art performance in both simulation and real-world tasks. For efficiency, ENERV ERSE- A reuses features from the first denoising step and predicts action chunks, achieving about 280 ms per 8-step action chunk on single RTX 4090. Further video demos, dataset samples could be found in our project page.
Introduction
Creative Al in vision has achieved significant progress, especially in video generation, where models produce high-quality videos from human instructions [23] [5]] This success highlights the model's spatiotemporal imagination, enabling accurate forecasting of future frames. Similarly, robotic manipulation, fundamental task in embodied AI, needs accurate predictions of future actions based on language instructions interact with the physical world. Based on this sharing principle of future space prediction, one natural strategy is to align robotics action prediction with a video generation task to leverage video generation models' imagination capabilities for policy planning. Motivated by this, recent studies 4g (88 6 15 have conducted preliminary explorations by fine-tuning general video generation models on robotic manipulation videos to align feature representations with the robotics domain, and predict physical actions. However, such methods (38) often simply adapt ceneral-purpose video generation models embodied tasks, neglecting the substantial gap between their representation space and the dineessinnii,nll, temporally interconnected robotics environment, thereby hindering accurate action policy prediction. We do not claim a direct monotonic link between pixel-level video quality and control success. Rather, align the latent space to encode 3D, action-conditioned dynamics so that actions can reliably follow generated trajectories.
To bridge the gap, propose ENERVERSE, generative robotics foundation model designed to construct and interpret the robotics 4D (3D with time) world.
ENERVERSS we employ an autoregressive video diffusion framework that iteratively predicts the embodied future space based on a given instruction. Within this generative paradigm, we define a minimal unit of the future space
*t indicates project leader. indicates corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as 'chunk', and the model repeatedly predicts the next chunk to incrementally expand the space. Additionally, prevent model collapse and enhance the action planning capabilities, we design sparse context memory mechanism during training. Instead of relying on consecutive memory, this mechanism preserves essential prior content throughout the generation process in a non-redundant manner, theoretically allowing infinite-length sequence generation. While this design achieves stable 2D embodied video generation, it remains insufficient for 3D understanding.
A straightforward solution is di- rectly extending 2D video generation into niew video generation and mounting multiple cameras to provide more 3D cues [12]. However, adding cameras increases hardware costs, I/O bandwidth requirements, and system complexity. To eircumvent these chal- lenges, we argue that strong 3D generative prior learned during pre- training can effectively enhance the single-camera setup. During infer- ence, observations from the single camera can be wrapped and rendered into multiple views. To establish this generative prior, we pre-train ENER- VERSE multi-view video genera- tion and introduce a Multi- View Dif- fusion Generator Block. This block utilizes ray direction maps to encode camera information and employs tem- poral attention to seamlessly fuse multi-camera data. Emperically, pre- training with multi-view consistency supplies 3D prior that benefits even single-camera deployments via rendered auxiliary views.
(el Malti-V Vie Video Genaration
Rendened Views
EnerVerse
Ener' Verreeereee N
(b) Rebotic Action Pradiction
EnerVerse
EnerVerse-A
Act.
Recretmuction with Obs.
Enerl EnerVerse- Veree-D
PerVVerse*
Vid. Gen.
4005 Spri catlen
An overview of ENERVERSE. With camera ob- servations, we first obtain 3D reconstruction via depth warping, then multi rendered images. ENERVERSE (a) con- nects a video generator head (ENERVERSE-G) to produce multi-view videos. (b) attaches to a robotic action policy head (ENERVERSE-A) for action prediction, and (c) inte- grates with 4DGS to form data flywheel JENERVERSE-D) Sim2Real.
Figure 1:
Although the pre-trin pre-training stage requires substantial multi-camera data, acquiring precisely calibrated multi-camera observations with corresponding robotic actions is costly and labor-intensive, While simulators can generate abundant synthetic data, the Sim2Real gap remains significant challenge. To address this, we propose ENERVERSE a data engine combining a generative model with 4D Gaussian Splatting (4DGS). By leveraging the adaptability of the generative model and the spatial constraints of 4DGS, ENERVERSE-D establishes a data flywheel that narrows the Sim2Real gap.
Building on these designs, ENERVERSE effectively models and interprets the robotic environment in both 3D space and temporal dimensions. With this generative prior, we can directly translate the 4D) world (3D spatial with temporal information) representation into physical actions via policy head, as shown in Fig. [ allowing the robot to execute task instructions in real-world scenarios. As result, ENERVERSE-A attains state-of-the-art performance in both simulation and real-world deployments.
The contributions of this work are as follows: (1) A chunk-wise autoregressive diffusion architecture with sparse contextual memory capable of long-term grounding. (2) A multi-view diffusion generator that endows the model with 3-D spatial prior beneficial under single-camera deployment. (3) A HDGSSbasee data flywheel that supplies ecometry-consister multiview training data for robotics.
2 ENERVERST
ENERVERSE comprises several designs, including a chunk-wise autoregressive generation framework and the sparse memory design for embodied future space generation. We additionally integrate 4DGS to construct a data flywheel, referred to as ENERVERSE-DD, and a policy head to generate physical actions, referred as ENERVERSE-A

---

(o) Chunkwise Autoregressive Generation
(b) Mutll-View Diffusion
Prompt
Offiute Ganerots
Diffusion Generator
20 Cone.
Training stage
Target
M
10
20 Cont.
Diffusion Generator
Diffusion Generator
Chunk
Chunk ix]
Render: Image Letast
Trage
holie Lahast
Figure An overview of our chunk wise autoregressive generation approach and multi-view diffusion generator block. (a) During training, random clean frames from consecutive sequences combined with noisy frames to predict denoised latents. In inference, newly generated denoised frames become the next clean frames for subsequent steps, iterating until the EoS frame is detected. Only a single view of the autoregressive process is shown for clarity. (b) In the multi-view diffusion generator block, observational frames from Camera i Rendered View + 1 encoded with a VAE. Ray direction maps are concatenated with video latents, followed by conv layers and attention mechanisms.
2.1 Next Chunk Diffusion
Chunk-wise Autoregressive Generation. As shown in Fig. 2 the observed latent sequence repre- sented as o.:K -o!....... of*1 (VAE). Here, K denotes the number of observed frames, H x W represents the spatial resolution, C' the number of channels, and is the denoising step. Similarly, the latent representation of the rendered image is given by r!:' E J x xC For simplicity, we treat r as special case of o. The predicted latent sequence is denoted Z!:M = [=!....,,24 2M M The goal is to develop a video diffusion model that generates these predicted latents conditioned on op I.K and a textual prompt c, following the conditional probability: (= o. Here, O represents the parameters of the denoising network, which is defined as c0(2) C, of t). e is encoded by frozen T5 encoder and then projected with an MLP. The network is trained predict the ground truth noise from the noisy frame targets by optimizing the loss function:
&xxxxxxx; encoded by pre-trained Variational Autoencoder
a
min E, 1- co.
co (2] 1:M
o. .)
where is the sampled ground truth noise, and 0 denotes the learnable parameters. We follow 41 to implement the v-prediction. Instead of predicting the noise e,. the model predicts te defined as:
Ve OyC; o.Fp. Here, 0g vā. (signal scale) and a, =I- 1 a (noise scale), consistent with the forward process equation Z; = agPD + dec. After training, the denoised data Zg can be derived from random noise Zr through iterative denoising.
During inference, the diffusion generator takes both clean and noisy frames as input to produce M denoised frames. The newly generated frames serve as clean inputs for subsequent iterations, and this process repeats until detecting a predefined End-of-Sequence (EOS) frame. As the diffusion generation operates on latent frames, the L1 distance of each frame the EOS is computed, If this distance falls below a predefined threshold, generation is terminated. In practice, this threshold- baseed EOS detection is highly effectivve
Sparse Memory Mechanism. Instead of the conventional approach using consecutive frames as the clean frame context for chunk prediction during training, we propose using sparsely sampled frames as the clean frame context. This approach leverages the redundancy often present in video data, allowing approximately 80% of frames to be discarded without compromising training effectiveness. Additionally, the high frame-dropping ratio enhances the model's robustness, particularly handling out-of-distributien (OOD) scenarios such covariant shift problems commonly encountered in the

---

robot learning domain. From representation learning perspective, this randomized sampling strategy promotes deeper understanding chunk prediction, potentially outperforming methods that rely on continuous frames.
During inference, clean frames derived from observed rendered frames and denoised using sliding window approach. This technique ensures a smooth transition between observed and generated frames while improving efficiency and reducing GPU memory consumption.
2.2 4D Embodied Space Generation
Single-view video generation struggles to recover accurate 3D structure and resolve occlusions, which critical for embodied manipulation. We therefore extend the next-chunk diffusion model in Sec. [.1 into a multi-view video gencrator that is conditioned on camera geometry and learns cross-view consistency end-to-end. Given task prompt m-view observations O1:K R with per-view camera intrinsics and extrinsics, we encode each frame into VAE latents and directly predict future multi-view latents Z1:M R M #xXxC illustrated in Fig. To make the representation view-aware, we compute a per-view ray-direction map that encodes the camera intrinsics and extrinsics and concatenate it channel-wise with the image latents before the diffusion backbone, following ray-based conditioning 77 40). Cross-view geometric coherence is modeled using attention along the view dimension: reshape features attend across views corresponding spatial locations and apply spatial attention that preserves pixel-to-ray alignment. Temporal attention applied along the time dimension capture scene dynamies. During training, simulator data provide ground-truth camera parameters; for real-world videos we use estimated extrinsics relative to the robot base. At inference, we use the available extrinsics; when depth is available, we optionally depth-warp warp observed frames to synthesize auxiliary rendered frames that better match the multi-view training conditions.
Otvervatio
Genersted Mull-rinn Videos
4D Gountan Splathing
from Camera 3o
Multi-View Generalor
JD GS
Optinatation
from toaster
Benderel tragel
Add Notie & Reploce
Ficooed Rei daraed Trage
View Rendering
Figure 3: The pipeline for ENER as a data engine. Observation images from multiple cameras and rendered images are processed by the multi-view video generator produce denoised videos. These videos, along with their camera poses, are used in 4DGS for 4D) scene reconstruction. The reconstructed 3D content is rendered to generate nigh-precision images. These high-quality rendered images are iteratively refined and fed back into the pipeline.
Real-World Data Flywheels. To reduce reliance on costly fully calibrated multi-camera capture and to narrow sim-to-real gaps, we introduce an offline data flywheel, e.g., ENNEVVEREE-D, that leverages sparse real observations bootstrap geometry-consiseer the gap, as shown Fig. After pretraining the 4D base model (EnerVerse-G) as above, we fine-tune accept sparse multi-view inputs in the 4D latent space VXCXXXXXXHX x Specifically, among m views, at least & m robot-mounted cameras provide complete observation sequences; for these observed views we skip noise injection and use their clean latents as conditioning, while we apply the standard noisy-to-dee noisy-to-denoised diffusion to unobserved target views. Given sparse observations (e.g.. one full video), their camera parameters, and task prompt, the model produces denoised predictions for the missing views. We then reconstruct a 4D scene using 4D Gaussian Splatting (4DGS) from union of observed and generated multi-view videos and their poses. The 4DGS representation rendered to all target views to obtain higher-fidelity, geometry-consister frames. These renders can be re-noised and fed back through the multi view generator, followed by another 4DGS optimization step, yielding iterative loop that progressively reduces noise, improves reconstruction accuracy,
multi-view videos and progressively reduce

---

and tightens cross-view consistency. As the loop accumulates real view episodes, we further fine-tune the multi-view generator on the collected data.
2.3 From 4D Embodied Space to Physical Action
We further integrate a policy head into the diffusion generator, enabling the generation of actions after the extensive pretraining of future space generation. The policy operates on compact visual latent E extracted from the middle block of the UNet backbone at the first denoising step (the noisiest step), which reduces computation while retaining rich task-relevant cues. Visual inputs may be captured RGB frames or rendered views, as shown in Fig 4 both are encoded by the shared video backbone. The action head is stack of DiT blocks followed by linear projection the action space. We predict action chunks [58) for temporal consistency. Let Gc:+r-1 E TR denote a chunk of actions with d (delta position, rotation, and gripper openness). The denoising model fo estimates clean actions from noisy inputs using DDPM-style training:
Gut+T-1 < fo(c, or, a ho(E, a a
where E. is the cached visual latent from ENERVERSE-G and k E {1,.... K} } is the diffusion step. The training objective minimizes denoising MSE.
At inference, we compute E' once by passing (c, oz) through the video diffusion backbone and cache across the action denoising steps. The action head then iteratively denoises from a to a" for the current chunk. We use per-view, per-frame decoding for visuals, but policy conditioning is view-aggress awgrregged via E (mean over spatial dimensions within the UNet middle block). We provide more details in the Appendix.
Cog
RGB-Image
Depth Map
Render View
Render View
Figure 4: Render View 1 and Render View 2 are generated by rendering from point cloud recon- structed from RGB-Image using depth wrapping. The render views correspond camera views obtained by rotating the RGB camera view around the Z-axis by +30%
3
Experiments
To demonstrate the effectiveness of proposed method, we evaluate ENERVERSE in two different domains, e.g. video generation quality and robotic policy performance.
3.1 Experiment Settings
Training Data: We selected several public datasets characterized by well-defined task logic, including RT-1 4. Tlco-Play 39), ManiSkill [4] BridgeVI 46 LanguageTaable (27]. and RoboTurk (28] for pretraining. Furthermore, we constructed a dataset containing multi-view video ground truths using the Isaac Sim simulator [I The detailed dataset statistics could be found in Appendix. During pretraining only video frames were utilized for video generation training. For the policy planning task, fine-tuning with limited quantity of demonstration data from specific scenarios proved sufficient to attain stafe-of-the-art performance. To mitigate domain gaps encountered when training with heterogeneous data, we employed domain embeddings inspired by [47). Specifically, distinct domain embeddings were allocated to each sub-dataset. In subsequent space generation and policy planning, these embeddings were integrated with the diffusion timestep embeddings prior to input into the diffusion model. This methodology effectively alleviated conflicts arising from discrepancies in entities, task types, and visual styles.
Model Details. Our model is conducted based on UNet-based Video Diffusion Models (VDM) 53 and can be easily adapted to DiT 32 architectures. And image decoding occurs per-view and per-frame. In our experiments on generating embodied future spaces, we identified that chunk size significantly influences model performance. Comparative analyses utilizing chunk sizes of 1, 4, and 16 revealed that the model exhibited optimal robustness when employing chunk size of 8

---

Following the methodology outlined in [1 we introduced corruptive noise to the frames within the memory context. To alleviate degradation in autoregressive generation, the intensity of this noise was modulated in a cosine-related manner relative to the distance from the current moment. After pretraining the multiview video generation models, we performed generation learning with the action videos to achieve both visual and spatial adaptation. Subsequently, we fine-tuned the action policy head using action trajectories. Following this, we fine-tune the action policy head using the action trajectories. The action head adopts the Diffusion Policy (DP) architecture [10), with total of 190M parameters. For the DP head's condition, we utilized features extracted from the middle block of the UNet during the first denoising step and calculated the mean across the spatial dimensions. This resulted a final shape of T x C. where T is the video length and C' is the channel number. Rendered images have resolution of 512 x 320, and the action head predicts the delta pose. For ENERVERSF-DD we integrate 4D Gaussian Splatting using the official implementation [50]. with depth-based initialization when available and deformation depth of 1.
3.2 Comparison Results
Embodied Future Space Generation. Following AVID 38 we assess video generation quality utilizing the RT-1 4 dataset. To create comparable baseline, we fine-tune DynamicCrafter on the RT-1 dataset and run inference iteratively with FreeNoise [34] to enable long video generation(DC. FN). For evaluation, we generate 200 synthetic videos with varied lengths by conditioning the models on the initial frame and task instructions, subsequently comparing the generated videos against the ground truth using standard metrics such as PSNR and FVD. However, while these metries primarily evaluate visual quality, embodied tasks necessitate additional considerations, including semantic alignment with instructions, workspace consistency across frames, and motion continuity. To address these higher-order aspects, execute a user study involving robotics experts, assessing the generated videos based on semantic accuracy, frame consistency, and motion continuity.
Atomic Task
Long Task
Method
PSNR1
FVD4
Qualityt
Seman.t
Consist.
Continuity
Ability
DC-FN
25.42
445.94
54
97
92
80
x
ENERVERSE
26.1
404.65
97
Table 1: Performance comparison between DynamiCrafter (FN) and our proposed approach across Atomic Task metrics (Quantitative Comparison and User Study) and Long Task ability. The proposed method outperforms DynamiCrafter (FN) in most metrics, demonstrating its effectiveness in video generation and task performance.
Tab. D iilluttrate that our method substantially outperforms DynamicCrafter (FN) in both quantitative and qualitative evaluations. In terms of quantitative metrics, our approach achieves a higher PSNR and a lower FVD. These findings indicate that our method produces videos of superior visual quality and enhanced temporal dynamics. In the user study, our method secures a higher quality score and exceeds DynamicCrafe in motion continuity, which is essential for robotic manipulation tasks. Although both methods attain equivalent semantic accuracy, this suggests that our approach effectively preserves instruction aligmment while delivering superior overall performance. Moreover, our method uniquely accommodates long tasks, as evidenced by its successful execution of long- manipulation scenarios, whereas DynamicCrafter falters in this domain. We also provide qualitative comparison in Fig. 5)
Multi- View Generation Consistency. In this section, we qualitatively demonstrate the capability of ENERVERSE to generate multi-view videos of the same scene while ensuring consistency across views. Furthermore, each view attains high-quality image generation, thereby highlighting the robustness of our approach. As shown in Fig. 6 ENERVERSE could generate high-quality multi-view videos in both simulator and real-world settings.
Robotic Policy Evaluation on LIBERO Following the evaluation protocol in Open VLA 22 we
evaluate robotic policies using the LIBERO 26 benchmark, which consists of four distinct task suites: IIBER--Spatall. LIbeRO-Object, LIBERO-Goal, and LIBERO-Long. Each suite contains 10 tasks, each with 50 human demonstrations. For each task suite, separate policy model is fine-tuned. We compare our method against five baselines: Diffusion Policy [10] a direct action learning policy trained from scratch; Octo 44) transformer- based policy model fine-tunel on the target dataset;

---

Obveration
16
32 Dynami rafter wi EN
Task: PPick
plastic bottle firm
EOS
bottom
16
place
Figure 5: Qualitative comparison for single view video generation between ENER ENERVERSE and Dy- namiCrafter(FN) RT-1 dataset. Since ENERVERSF predict EOS frame at 42th frame for this task, we visualize up-to 42th frame sampled from both generated sequence. The sequences generated by DynamiCrafter(FN) did not maintain the logic and produce many hallucinations as the sequence grew. In contrast, the sequence generated by ENERVERSE was logically coherent, continuously and completely generating the future space of the entire task, and accurately predicting the EOS frame.
Sen
Taak:
clevbel The tonats scuce baska?
hes dioss tout plaze them the deng paote
Figure 6: Qualitative results for multi view generation LIBERO (left) and real-world manipulation data (right). The first generated view position is overlapped with a static mounted RGB camera and others are manually set. The consistency of objects across views is highlighted by red rectangle.
OLenVLA, 7B vision-language-zction (VLA) model fine-tuned on the target dataset; MDT 37 ,a diffusion wansformer-based policy with auxiliary MAE loss; MAIL [19], policy model with Mamba 133 in an encoder-decoder structure. Besides, we provide the results of MAIL with two S- RGB input with their official implementation. For evaluation, all models are tested across tasks using 50 rollouts per task, with results averaged over three random seeds. Experiments with conducted under three setups: single RGB image, and when RGB-D is available, RGB image with rendered view, and RGB image with 2 rendered views, shown in Fig. 4 The abbreviations denote different input modalities: S-RGB for Static RGB, G-RGB for Gripper RGB, S-RGBD for Static RGB-D, G-RGBD for Gripper RGB-D, P for proprioceptive arm position.
Model
Visual Input
Spatial
Object
Goal
Long
Avg.
Diffusion Policy
S-RGB
78.3
92.5
68.3
50.5
72.4
Octo
S-RGB
78.9
85.7
84.6
51.1
75.1
VLE
S-RGB
84.7
88.4
79.2
53.7
76.5
MDT
S-RGB.G-RGB
78.5
87.5
73.5
64.8
76.1
MAIL
S-RGB.G-RGB
74.3
90.1
81.8
78.6
81.2
MAIL
S S-RGB.S-RGB
76.0
90.0
82.0
78.0
81.5
ENERVERSE
S-RGB
92.1
93.2
73.0
84.1
ENERVERSE
S-RGBD - RGB with Render
93
95.0
81.0
73.0
85.5
ENERVERSE
S-RGBD - RGB with Render
91.2
97.7
85.0
80.0
88.5
Table 2: Evaluation results on the LIBERO benchmark across four task suites.
As shown in Tab. ENERVERSE achieves tate-offthhe-att performance across the LIBERO bench- mark, significantly surpassing all baselines. With single S-RGB input, it achieves an average score of 8. outperforming strong basclines.
7

---

Robotic Policy Evaluation on CALVIN CALVIN 29 is an open source simulated benchmark designed for learning long-horizon tasks. It consists of four distinct scenes (A. B. C, and D) and introduces the ABC-DD evaluation protocol, where models are trained on environments A, B, and C and evaluated on environment D. The primary evaluation metric the success sequence length, which measures the ability to complete five consecutive subtasks within a sequence. Notably, CALVIN's whereas inference requires performing sequential tasks without
training data is trajectory-basee explicit task transition signals. This discrepancy between training and inference introduces challenges, as our model relies on memory. Nevertheless, we strictly adhere to the evaluation protocol and do not task begins, making policy inference more demanding. This limitation
reset the memory when does not affect other models, as they do not utilize memory. Despite these challenges and limited input signals, model achieves competitive performance, shown in Table 3
Method
Input
2
3
4
5
Avg. Len.
RoboFlamingo 25
S-RGB, G-RGB
82.4
61.9
46.6
33.1
23.5
2.47
GR-1 [5]]
S-RGB, G-RGB, P
85.4
71.2
59.6
49.7
40.1
3.06
3D Diffuser 20
S-RGBD, G-RGBD, P
92.2
78.7
63.9
51.2
41.2
3,27
SUSIE 2
S-RGB
87
69.0
49.0
38.0
26.0
2.69
ENERVERSE
S-RGB
90.8
73.0
57.3
43.7
35.6
3.00
Table The comparisons with state-of-the-an approaches on Calvin (ABC - D).
3.3 Further Studies
In this section, we explore several key design choices for ENERVERSE. First, we examine the significance of the proposed sparse memory mechanism, which plays critical role in both policy learning and video generation. Second, we discuss the training strategy utilized in ENERV ERSE.. Third, analyze the alignment between the predicted action spaces and visual spaces through attention map analysis. Finally, we introduce the real-world experiment setup.
Effectiveness of Sparse Memory Mechanism. We evaluate the effectiveness of our sparse memory mechanism both policy learning and video generation. The evaluation is conducted on the LIBERO- Long task suite, as this suite involves significantly longer task execution steps, requiring the policy to exhibit strong long-range memory and task reasoning capabilities. The evaluation is performed with single visual input. As shown in Tab. 4 the absence of the sparse memory results in significant performance degradation, with the policy achieving only 30.8 compared to 73 when the sparse memory mechanism is applied. Similarly, Fig. Fes-Zdeen demonstrates that when the video generator operates without sparse memory, the model experiences unexpected collapse and fails to recover in out-of- distribution (OOD) scenarios. In contrast, the sparse memory mechanism ensures robust performance while also saving computational resources.
Observation
16
EnerVerse Generated consecutive memory
Task: 'Turn the stove."
EnerVerse Gemerated w/ sparse memory Figure 7: Ablation results for context memory mechanism in video generation. Providing history information to the generation model with consecutive context (first line) often leads to unexpected model collapse while the model with sparse memory (second line) shows robust performance and mush computing resources.
Training Strategy Analysis. To analyze the impact of different training strategies on robotic policy learning, we trained four robotic policies on the LIBERO-Spatial task suite using the following approaches: (1) training the entire ENERVERSE from scratch using only policy loss optimization; (2) training the entire ENERVERSE in (1) but initialized with pretrained weights from general video generator, e.g. DynaniCrafter(DCC (33, which trained with the general natural videos; (3) co-training ENERVERSE ERSE by optimizing both the robotic policy action loss and the video generation

---

Setup
w/o Sparse Memory
Sparse Memory
LIBERR-LLon-SV
30.8
73
Table 4: Sparse Memory analysis on LIBERO-Long.
loss simultaneously; and (4) the default two-stage training strategy, where the video generator is pretrained first, followed by fine-tuning ENERVERSS ERSE using only robotic policy loss optimization.
Strategy
All-Scratch
With DC Pretrain.
One-Stage Co-Train
Two-Stage Finetune
LIBERO-Spatial
Failed
79
86.3
92.1
Table 5: Performance comparison different training strategies on the LIBERO-Spatial task suite,
As shown in Tab. 5 training Verse from scratch without loading pretrained weights failed to converge, underscoring the importance of robust initialization. Another possible reason for this failure could be the relatively limited training data compared the number of network parameters. Initializing with pretrained weights improved performance (79), while jointly optimizing the policy loss and video generation loss in a one-stage co-training setup further increased performance to 86.3. This demonstrates that the video generation task enhances policy learning. Our default Two- Stage Fine-tuning strategy, which involves pretraining the video generator followed by fine-tuning ENERVERSE with policy loss optimization, achieved the best performance.
Effectiveness of Additional Rendered Views. With the expressive pretrained generative space prior, our method using a single S-RGB camera already achieves SOTA performance. When a single S-RGBD camera is available, we can incorporate additional rendered views, with the original RGB image as input to the model. These additional views not only bring the input setting closer to the training configuration but also enable the policy model to better leverage the pretrained generative space prior. Notably, the configuration RGB with 1 Render outperforms MAIL's 's 2 S-RGB setup, both in terms of overall performance and gains compared to a single RGB input, demonstrating that the performance improvement is not solely due to the additional visual inputs. Incorporating RCB with 2 Render yields even greater gains by mitigating occlusions, as illustrated in Fig. 4
3D Video vs. 4D Space for Robotics. We provide a direct comparison between attaching diffusion policy head to base single-view video generator (DynamiCrafterr) and our 4D variant (Ener Verse-A). As shown in Table[ō the base video model underperforms substantially (79.0) relative to ENER V ERSE- (92.1), highlighting the benefits of 4D extensions. We hypothesize that the cross-view consistency learned during multi-view pretraining provides stronger geometric priors, which help the model better understand spatial relationships and occlusions. Further, even when tested with single RGB-D inputs, ENER V ERSE benefits from additional rendered views at inference. These findings underscore the ability to incorporate additional rendered views test time further enhances performance, showcasing the practicality and effectiveness of our approach.
Model
Multi-view Pre-Train
Input at Test
SR
DynamiCrafter + DP
No
S-RGB
79.0
EnerVerse-A
Yes
S-RGB
92.1
VeerVeeseAA
Yes
S-RGB with Render 93.0
Table 6: Direct comparison between a base single-view video generator with a policy head and approach on LIBERO-Spatee Multi-view pretraining substantially improves SR even with single-view inputs.
Real-World Experiments. To evaluate the manipulation capabilities of EEESVE, we con- ducted real-world experiments using commercial robotics the tasks of Block Placing, Plastic Objects Sorting, and Fruit Sorting. For further details, please refer to Appendices A
More Discussions and Experiments. We provide additional discussions and experiments in the appendices: pretrained model performance (Appendices (). aligmment between action and visual spaces (Appendices D). robustness against OOD samples (Appendices(B EB). model architecture and computational overhead (Appendices and G and visual samples validating the data engine's effectiveness (Appendices

---

Related Works
Video Generation Models. Diffusion-based video generation models have made notable progress,
especially in text-to-video (T2V) generation 3 42 Early T2V approaches [56] 9
35 16] build
on text-to-image (T2I) priors by introducing temporal modules trained on video data. Dynamic-
Crafter [53] reuses motion priors from T2V diffusion models in an image-to-video (12V) context.
Recent works [23] 59 explores replacing U-Nets with Diffusion Transformer (DiT) [33], and [36] uses the multi-camera poses information to extend the video generation into the 3D world modelling. Other studies [T] incorporate causal mechanisms to generate longer sequences or extend video gecooegnrrttion models into world modeling by forecasting future states 17
In this paper, we mainly adopt DynamicCrafter as our base I2V framework due to its open-source availability and widespread use. We also ensure compatibility with modern DiT architectures, although that is not main focus here.
5 48
Video Pretraining for Robotics. GR-2 6 presents generalizable robot manipulation framework that pretrains on large-scale internet videos, then fine-tunes on both video generation and action prediction for robotic trajectories. LAPA (55] uses non-robot action videos for representation learning, mapping discrete latent actions (via VQ- VAE) to robotic manipulation tasks through a VLA model. SEER 45 further explores inverse dynamies pretraining to boost performance. AVID [38] employs DynamicCrafter [53] as its foundation, using an adapter for the robotics domain. VidMan [49], based on OpenSora [59], focuses environment prediction before action generation but is limited to 2D image space. In contrast, we propose generating long-seuu unn-sequence futures via a novel data generation engine, capturing richer motion information vital for robotics.
Recent progress allows reconstruction of dynamic scenes from 2D videos using 3D Gaussian Splatting (GS) 21 and Neural Radiance Fields (NeRF) 30). Prior approaches approxi- 4D volume with sets of 4D Gaussians [54] jointly optimizing geometry
Generation.
mate the
and motion in canonical space [50). More recent advancements 24 employ customized sampling
for multi-view video diffusion models, particularly for single dynamic objects. DimensionX [3] leverages multiple LoRAs [8] for diverse camera motions, while Cat4D [52] uses single multi-view diffusion model to generate videos for dynamic 3D reconstruction. By contrast, our method produces videos from multi-views tailored for robotic manipulation tasks. In offline data flywheel stage, GS complements video generation models to mitigate the sim-to-real gap, enhancing geometric consistency and reducing hallucinations from the generative models.
Conclusions
In conclusion, ENERVERSE is generative robotics foundation model that tackles multi-view video generation and long-range policy execution by modeling embodied future spaces. With sparse con- textual memory and chunkwise autoregressive architecture, VeErE ERSE enhances spatial reasoning pipeline, combining generative modeling with 4DGS,
and task adaptability. The bridges the sim-to-real gap, reducing reliance real-world data. Integrated with a policy head, ENERVERSE-A achieves state-of-the-an performance in manipulation tasks, both in the simulator environment and the real-world settings.
Limitations. Due to the high dynamics and rich object interactions in robotics tasks, video generation models inevitably produce artifacts, discussed in App. B However, advancements in the video modeling community are expected to improve this. Additionally, while we provide an initial attention map analysis, further exploration is needed to better understand how video generation guides action policy learning. Finally, the current rendered views are derived from heuristically set camera poses, which may not be optimal. Integrating Next-Ber Best View methods [57] could address this limitation.
10

---

References
[1] Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng. Min Zhao, Shilong Liu, Yaole Wang. and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233 2024.
[2] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint 2ato:3331..00333 2023.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.5512,, 2023.
[4] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0663 2022.
[5] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[6] Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024.
[7] Eric Ming Chen, Sidhanth Holalkere, Ruyu Yan, Kai Zhang, and Abe Davis. Ray conditioning: Trading photo-consistency for photo-realism multi-view image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23242-23251, 2023.
[8] Guikun Chen and Wenguan Wang- A survey 3d gaussian splatting. arXiv preprint MxXiv:241100390 2024.
[9] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang. Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation, 2023.
[10] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 027836492441233668 2023.
[11] Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models. arXiv preprint arXiv:2406. 10981, 2024.
[12] Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations. arXiv preprint arXiv:2406.085445 2024.
[13] Albert Gu and Tri Dao. Mamba: Linear-tine time sequence modeling with selective state spaces. arXiv preprint arXiv:23 2023.
[14] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023.
[15] Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. arXiv preprint 1xXi:2241..1//777 2024.
[16) Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 20Xiv:2307.0475,,00 2023.
11

---

[17] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: 1: A generative world model for autonomous driving. arXiv preprini arXiv:2309. 17080, 2023.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106. 2021.
[19] Xiaogang Jia, Qian Wang, Atalay Donat, Bowen Xing, Ge Li, Hongyi Zhou, Onur Celik, Denis Blessing, Rudolf Lioutikov, and Gerhard Neumann. Mail: Improving imitation learning with mamba. arXiv preprint 4rX1v22406.08234, 2024.
[20] Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402. 10885, 2024.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 1394)) 2023.
[22] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint (XXx::2406.092444 2024.
[23] Weijie Kong, Qi Tian. Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
[24] Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. arXiv preprint arXiv:2406.08653 2024.
[25] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang. Ya Jing, Weinan Zhang, Huaping Liu, et al. Vlsion-language foundation models as effective robot imitators. arXiv preprint arXiv: 1 2023.
[26] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
[27] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
[28] Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1048- 1055. IEEE, 2019.
[29] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L., 7(3):7327-7344 2022.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[31] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 87):3774737 2023.
[32] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023.
12

---

[33] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[34] Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling, 2023.
[35] Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint 2xXi:24022004224, 2024.
[36] Xuanchi Ren, Tianchagg Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-Davit Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3e: 3d-informed world- consistent video generation with precise camera control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[37] Moritz Reuss, Ömer Erdinç Yağmurlu, Fabian Wenzel, and Rudolf Lioutikov. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024.
[38] Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models world models. arXiv preprint arXiv:2410. 12822, 2024.
[39] Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. 2022.
[40] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, UIs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- frree novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229 0238, 2022.
[41] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 200iv:2202.00512, 2022.
[42] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[43] Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimensionx: Create any 3d and 4d scenes from single image with controllable video diffusion. arXiv preprint arXiv:24/ 1.04928, 2024.
[44] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint 24Xiv:2405. 12213, 2024.
[45] Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. hmtp::/aaxiisorgdds22411.11.10.0 2024.
[46] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[47] Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceetiive-vssaal learning with heterogeneous pre-trained transformers. arXiv preprint arXiv:2409.20537, 2024.
[48] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards welldderld world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023.
[49] Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, and Xiaodan Liang. Vid- man: Exploiting implicit dynamics from video diffusion model for effective robot manipulation. arXiv preprint arXiv:2411.01153 2024.
13

---

[50] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310-20320, 2024.
[51] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-tra pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023.
[52] Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. arXiv preprini arXiv:2411. 1.1613,, 2024.
[53] Jinbo Xing. Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang. Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In European Conference on Computer Vision, pages 399-417. Springer, 2025.
[54] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642, 2023.
[55] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11755 2024.
[56] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qing, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models, 2023.
[57] Xuechao Zhang, Dong Wang, Sun Han, Weichuang Li, Bin Zhao, Zhigang Wang, Xiaoming Duan, Chongrong Fang, Xuelong Li, and Jianping He. Affordance-driven next-best-view planning for robotic grasping. arXiv preprint wrXiv:2309005533 2023.
[58] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[59] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024.
[60] Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, al. Is sora world simulator? comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405 03520,, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and Introduction Section, we clearly demonstrate the contribu- tion and scope of the proposed ENERVERSE.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the main text, we outlined the limitations related to our work. In Appendix, we further discussed the limitations of our article, hoping to guide more future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have described the relevant details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: This may be temporary, and we are working hard to promote the process of open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
/tips.cccppblii... wwcppbblccgguuieeeeo http:/gddese/eee/ htt////ooSSumiiseeeeeeeeee stibbbmmsssoooooccccc. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Section g and App. we clearly demonstrated experimental settings,
including model architecture, training settings, evaluation settings, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high cost of pre-training, we do not repeat the same experiments. But we perform multiple independent runs with different random seeds for the policy evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In App. we have provided sufficient information on the computer resources needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:://ourrps..ccpppiit////ees
Answer: [Yes]
Justification: We guarantee that the research conducted in the paper complies with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our license name CC-BY 4.0. We will cite the original works and properly acknowledge the authors of any existing assets (e.g., code, models, datasets) used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppccc/ooooeereeee/ /ct/////oeeeeeccc//22////.. for what should or should not be described.
21

---

A Real-World Robotic Experiments
To evaluate the manipulation capabilities of ENERVERSE-A, we conducted real-world experiments. The robot is instructed to place blocks into designated compartments of foam worktable, requiring accuracy due to the tight fit and visual similarity between the foam and table, as shown in Figure
Compared with the general "Pick and Place" task, this task has additional challenges:
The robot must follow natural language instructions, such as "Row One, Column Two," to identify the required compartment.
The compartments are only slightly larger than the magnet blocks, transforming the pick- and-place task into highly precise "insertion" operation.
The magnet blocks are relatively heavy, requiring the robot gripper to grasp near the center of the block ensure stability during manipulation.
Correspondingly, we define four evaluation metrics:
Grasp: Indicates whether the robotic gripper holds the suitable part of the block and transfers
it stably during manipulation. It has binary values: 0 for failure, 1 for success.
Determines whether the robot places the block into possible compartment. A score of O indicates failure, indicates a perfect placement, and 0.5 indicates that the block has some collisions with the foam during manipulation.
Place:
Instruction Following:
Evaluates whether the robot places the block into the desired compartment as instructed. It has binary values: 0 for failure, for success.
The overall Success is calculated as the product of the individual factors. The policy was executed five times for each compartment, and the average scores are presented in Table ENERVERSE-A demonstrates strong performance in most target positions. However, fails to handle positions (3, 2) and (3.3). We hypothesize that this limitation arises because these positions are located near the boundary of the robot's action space, making them challenging to reach. We provide the VLE [22] results. Our method outperforms OpenVLA both grasp and place subtasks, demonstrating superior spatial understanding. The place subtask, in particular, is challenging due to compartments being only slightly larger than the blocks, requiring precise spatial understanding and target localization, which highlights the benefits of our method's 4D space prior. However, OpenVL. shows better cerructionfollowiing ability, likely due to its LLM part (our CLIP text encoder). Demonstration videos are provided in the supplementary materials.
In addition to the block placement task, we conducted experiments sorting transparent plastic objects and fruit sorting. Demonstration videos for these experiments are also included in the supplementary materials.
69
(8) Block Placement
(b) Fruit Sorting
Figure 8: Real-world experimental setup.
22

---

Target Position
Grasp
Place
Ins. Following
Success
(1.1)
1
(1,2)
1
(1,3)
0.8
1
0.8
(2,1)
0.7
0.7
(2,2)
1
1
(2,3)
0.8
0.8
(3,1)
0.7
0.7
(3,2)
1
0
0
(3.3)
0
OpenVLA-Avg
0.89
0.61
0.96
0.61
ENERVERSE-AVE
1.0
0.89
0.78
0.67
Table 7: Performance of the robotic system in placing blocks into designated compartments. The task demands high precision due the tight fit and visual similarity between the foam and table.
Further Discussions on the Tasks Types and Video Quality in the Real-Worke Settings.
Integrating physical knowledge, such as kinematics and dynamics, into generative models remains a significant challenge, particularly for complex robotic manipulation tasks (60). However, believe that large-scale, high-quality pretraining data can significantly enhance physical modeling capabilities, including tasks involving deformable object manipulation. For instance, we provide video generation Our approach is not limited to simple pick-and-place tasks. is capable of modeling tasks that require kinematic constraints, such as articulation (e.g., turning button or opening drawer). Additional video results are provided in the supplementary materials.
results for a cloth-folding task in Fig. /
Figure 9: Generation Results on the Cloth Folding Task.
While visual artifacts in the generated videos (e.g., surface penetration or snappy transitions) are present, these imperfections have minimal impact on robotic task execution. our framework, the generated videos primarily serve 4D spatiotemporal prior, which is further refined during fine-tuning. This is supported by our real-world robot experiments, where task performance remains robust despite the presence of these artifacts.
23

---

Init. Obs
Init. Obs
Figure 10: Generation Results with Pre-Trained Model on LIBERO.
C Discussion on the Pretrained Model Performance
We present the pretrained model's generation results on the LIBERR-Object split in Fig. TO The gen- crated videos exhibit significant artifacts, with the scenes collapsing after several frames. We attribute this issue to the domain gap between the LIBERO dataset and the datasets used for pretraining.
Additionally, we directly fine-tuned the pretrained model on LIBERO-Object actions without adapting it to the LIBERO video generation task. As shown in Table this approach results in substantial performance degradation for the final policy model.
Method
LIBERO-Object
ENERVERSE
93.2
ENERV ERSE w/o Gen-Adaption
85
Table 8: Comparison of policy performance w/wo LIBERO video generation task adaption.
D
Attention Map Analysis
To further analyze the alignment between the predicted action space and the visual space, including the visual observations cached by our Sparse Memory Mechanism and the generated future space, we visualized the attention maps from the first several layers of the Cross-Attention Block in our policy head.
Fig- TT illustratess attention maps from different heads and layers, showcasing the model's hierarchical focus and the impact of our proposed embodied future space generation in facilitating robust action prediction. In Fig. II a), attention distributed almost entirely across the future space, reflecting the model's 's ability to leverage sparse memory conditions and generated predictions from the outset. In contrast, Fig. d) shows the attention sharply focused on the sparse memory space, with minimal reliance on the generated future space, indicating that the model has transitioned to memory- based reasoning. Interestingly, Figures ([]c,e) demonstrate that the model effectively integrates information from both the sparse memory space and the predicted future space. Moreover, these attention maps reveal that earlier decision steps tend to prioritize sparse memory, while later action steps shift focus to the generated future space. These results validate that our generative pretraining effectively enhances the model's ability to integrate temporal information, align predicted actions with future visual contexts, and make robust decisions.
24

---

te Head Loyer
DO read Leyer
00 Heed Lower
Heat IS, Loyer
Head
Dred. Action spaze
Sperte
Conl
Cane
Figure 11: Attention maps from different heads and layers of the model. The y-axis (Query) represents the predicted action space (8 steps), while the x-axis (Key-Value) spans Sparse Memory (first columns) and predicted future space (last 8 columns). Bright yellow indicates high attention, showing how the model focuses on memory (left) and future predictions (right) when generating actions.
E Robustness against OOD Samples
To evaluate generalization out-of-distribttion (OOD) samples, we designed three experiments:
Changing the floor texture for unseen scenes.
Altering container textures for unseen containers.
Training on all LIBERO splits ("train-all, test-all") and evaluating on each split simultane-
ously.
The first two experiments required no retraining and were conducted on the LIBERO-Object split. As shown in Table 2 model demonstrated strong generalization and robustness. For the "train-all, test-all" experiment, the performance (87.63 Avg) improved compared to single-split training (84.1 Avg). We attribute this improvement shared textures and spatial layouts across splits, which enable better learning from the larger mixed dataset.
Method
Seen
Uns. Scene Texture
Delta
Uns. Cont. Texture
Delta
Open VLA (S-RGB)
88.4
64.9
23.5
82
6,4
Ours (S-RGB)
93.2
93.1
0.1
93.0
0.2
Ours (RGB with 2 Render)
97.7
96.4
1.3
97.5
0.2
Table 9: Performance comparison across seen and unseen scenarios with texture variations.
F
Computational Overhead
During the action-related fine-tuning training stage, using LIBERO-Sparie as example, the single S-RGB setting requires A100 GPUs for approximately 20 hours during the video generation adaptation stage and an additional 12 hours for the action learning stage.
For video generation inference, the single-view setting consumes approximately 12 GB of GPU memory, while generating three views requires about 13.5 GB. The generation of single video chunk takes around 20 seconds per view.
In action inference, the single S-RGB setting uses 10.6 GB of GPU memory, whereas the three-view configuration requires 12 GB. Action inference for single S-RGB setting takes approximately 300 ms per action chunk, with default chunk size of eight frames.
G Further Details on the Model Architecture
The main architecture based on DynamiCrafter [53]. with extensions support multi-view processing using the Ray Camera Map and Spatial Attention. No additional specialized designs were introduced; instead, operations were conducted in a latent space. Specifically, the input latent has shape of BCVTHW, where is the batch size, C is the channel, V is view number, and T stands for timestamp. This input latent reshaped as follows:
(BT) (V HW) C for spatial attention.
25

---

Dittusion Model
Down blocks
Up blocks
middel block
bevthw
Pooling
btc
MLP
btc
noisy actions
DiT
Linear
actions pred
inner dim: 1152
Figure 12: The Construction of the Action Policy Head.
Hyperparameter
Configuration
Diffusion Setup
Diffusion steps: 1000; Noise schedule: Linear; 30 0.00085; 3r 0.0120
Sampling Parameters
Sampler: DDIM; Steps:
Input
Video resolution: 320 x 512; Chunk size: 8; Encoded with VAE Language prompt c, tokenized with T5
Camera parameters encoded with ray direction map (L118 in main text)
UNet Temporal Attention
Latent image channels: Ray map channels: s-spape: x 64 4
Attention resolutions: {64, 32, 16); Head channels: 64; Conv layers: 4
Temporal kemeel size: 3, 1, 1; downscales: 40 x 64 + 20 x 32 - 10 x 16
Spatial Attention
Attention resolutions: [64, 32, 16); Head channels: 64; Conv layers: 4
Video Training
Learning rate: x Optimizer. Adam; Batch/GPU single-view):
Batch/GPU (multi-view).
Parameterization: u-prediction; Max steps: 100,000: Gradient clipping: 0.5 (norm)
Policy Training Number of Parameters
Same as vidco training, but with preplc-prediction parameterization
Base model (DynamiCrafter): 1.4B; Policy head (DiT blocks): 190M; VAE
(frozen): 83.7M
Table 10: Training details and hyperparameters used in our experiments.
HBV for temporal attention.
(BVT) CHW before decoding.
The image decoding occurs per view and per frame. Features (BCVTHW) were extracted before the U- Net's middle block, followed by Pooling and an MLP obtain a BTC feature vector for conditioning the denoising process. The action head consists of 18 DiT blocks, with denoised latents passed through a linear layer action predictions. As mentioned in Section 2.3 the header predicts delta pose. Actions are represented as 7-dimensional vector in pose space: delta position (z, y. z),
rotation (roll, pitch, yaw), and gripper openness. A simple diagram is shown Fig. T2 We further provide the hyperparameters in Table I0
26

---

H More Details on Training Data
Pretraining Datasets. We pretrain on heterogeneous embodied datasets with clear task logic:
RT-1 4 Language Table [27]. Bridge [4), Robo'Turk 228 ManiSkill
[1], and an Isaac Sim
dataset [3I].
Summary statistics:
RT-1: 3.7M frames, 87K episodes, egocentric, real robot.
Language Table: 7.0M frames, 442K episodes, front-facing, real robot.
Bridge: 2.0M frames, 25K episodes, egocentric, real robot.
RoboTurk: 72K frames, 1.9K episodes, front-facing, real robot.
ManiSkill: 4.0M frames, 30K episodes, front-facing, simulation.
Isaac Simulator: 3.0M frames, 40K episodes, egocentric + 8 stird-person views, simulation.
At the time of this work, public embodied datasets predominantly provide single third- perd-person or egocentric views, which are insufficient for training multi-view generation. LIBERO contains fewer than 500 trajectories, making inadequate for robust multi-view learning. To bridge this gap, constructed a multi-view dataset in Isaac Sim with ground-truth camera parameters. The simulator corpus contains about 40K episodes across 8 tasks spanning industrial and home scenarios, with diverse object layouts, lighting, and camera poses. Task list:
place trash into dustbin;
pick fruit into basket;
pick toy into box;
insert pen;
place bag;
open drawer;
place fruits;
arrange workpieces.
We provide visual examples of data collected from the simulator Fig. 13 with additional videos available in the supplementary material. At the time of this work, all available embodied datasets provided only single third-person camera views, which are insufficient for multi-view generation tasks. Furthermore, the evaluation benchmark LIBERO contains fewer than 500 trajectories, which is inadequate for training a multi-view generation model. Collecting real-world multi-view data directly prohibitively expensive and sabor-intensive.
Figure 13: Visual Examples from the Simulator Collected Data.
I Visual Samples for our Data Engine
We provide visual samples from our Data Engine in Fig. 14 As shown the figure, using ENER- VERSE-D the data engine results in fewer artifacts and clearer boundaries.
Furthermore, we conducted additional experiments on the "arrange workpieces" task, where a robotic arm manipulates gears and boxes a tabletop with frequent self-occllssioss. Following the data- flywheel setting, given the task description and one complete head-camera video, the goal to generate the corresponding video from target view. We evaluated 50 generated episodes under two
27

---

Only 4DGS
EnerVerse-D
Figure 14: Visual Samples for Data Engine.
settings: (i) Without SEGSS-direcly running the ENERVERSE-D video generation pipeline; (ii) With 4DGS-firs first generating an initial video via ENERVERSE-D, then applying the 4DGS pipeline to render target views, re-noising the renders, and feeding them back into ENERVERSE for refinement. Two blinded human experts assessed diffusion-induced hallucinations in the generated videos. The assessment result shows that integrating 4DGS reduces hallucinations by 40% relative the baseline without 4DGS in scenarios with self-occlusions, quantitatively demonstrating the value of 4DGS in mitigating generative artifacts.
28


Reviews:
[{"review_id": "CBrGTuuS8a", "reviewer": "Reviewer_C5W7", "summary": "This paper presents EnerVerse, a robotics foundation model that is capable of generating future frames conditioned on past observations + task context, and using these future frames for action prediction to solve the task. In particular, EnerVerse uses a chunk-wise autoregressive diffusion model with sparse memory for future prediction and incorporates multi-view video generation for better 3D spatial understanding. This video generation model can be extended to action generation (EnerVerse-A) by adding a diffusion-based action head. To facilitate better training of the multi-view frame generation + action prediction, the paper proposes a data synthesis framework for multi-view video generator based on sparse observations in simulation. Experiments are performed for video generation quality and robotic task execution, and show promising improvements over prior work.", "questions": "My initial assessment is that this paper is incomplete and not yet ready to be accepted. My primary concerns are to do with the paper clarity and experimental section. I have listed the questions along with the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n## Strength 1\n I liked some of the experimental aspects of the work.\n* Real-world qualitative results are useful to understand the sim2real generalizability of the work.\n* Several relevant robotic action policies are compared against in Tables 2 and 3, strengthening the claims of the paper and demonstrating the usefulness of high-quality multi-view video generation for robotic tasks.\n* Additional experiments are presented to study the value of different components of the system, which are helpful to understand their impact on the robotic tasks (sparse memory in Table 4, training strategy in Table 5, and utility of multi-view rendering in Table 2).\n\n## Strength 2\nThe related work section was good and helpful to understand the relevant literature on video generation and robotics with foundation models.\n\n## Strength 3\nI appreciate that the authors have included qualitative videos of robotic task executions in the supplementary material. \n\n# Weaknesses\n\n## Weakness 1:  Paper writing clarity is poor\nThe proposed approach was quite difficult to understand given the amount of partial / missing information in the main paper. This makes it hard for a reader to grasp the idea and the core contributions.\n* **Q1:** L93 - The velocity term used for prediction is not described (in fact, the variable $v$ is used without any explanation or definition)\n* **Q2:** Section 2.2 4D Model - Lots of things are unclear here.\n\t* What is the exact architecture of the \"Multi-View Video Generator\" in Figure 3? Some details are mentioned in text (like \"spatial attention is applied ...\" in L121-122), but these are very vague.\n\t* How exactly is the 4D model trained?\n\t* What are the inputs, outputs and losses during training?\n\t* What are the sources of training data for 4D training?\n\t* How are the multi-view camera extrinsics decided during training and inference?\n* **Q3:** Section 2.2 Real-World Data Flywheels - The mechanism and motivation behind the data generator is vague.\n\t* L127 - \"utilizes sparse observations ...\" - where are these sparse observations coming from? How are these sparse observations being utilized?\n\t* L128 - What is the base model EnerVerse here? Is it the model trained for generating a single video from past video + task context? \n\t* L129 - How is EnerVerse \"fine-tuned to be capable of receiving a complete offline observation sequence\"? \n\t* L129 - 131 - \"When inferring, .... used to construct a 4D ...\" -- What does inference mean here? What are the inputs and outputs? How are they used to construct a 4D gaussian representation?\n* **Q4:** Section 2.3\n\t* How is the model trained for action prediction? \n\t* What are the loss functions?\n\t* Does the diffusion happen along with the video diffusion or are they separate?\n\t* Does the action diffusion model use past actions as inputs?\n* **Q5:** L183 - Unclear how the DynamiCrafter baseline works in this context. \n\t* DynamiCrafter is trained to accept an image and text context to generate short videos. How exactly is it adapted as a baseline to this setting with long video generation?\n\t* Does DynamicCrafter baseline get the advantage of the same training data as EnerVerse (see L154 - 156)?\n\n## Weakness 2: Experiment section feels incomplete\nWhile I liked some aspects of the experiment design, there are a lot of things missing / unclear. \n* **Q1:** Why is DynamiCrafter the only video generation baseline? Why are other models not evaluated? AVID (https://arxiv.org/html/2410.12822v2), I2VGen-XL (https://arxiv.org/abs/2311.04145), Stable Video Diffusion (https://arxiv.org/abs/2311.15127), Motion-I2V (https://dl.acm.org/doi/abs/10.1145/3641519.3657497), LivePhoto (https://link.springer.com/chapter/10.1007/978-3-031-72649-1_27), PhysGen (https://link.springer.com/chapter/10.1007/978-3-031-73007-8_21), etc. to name a few. \n* **Q2:** Multi-View Generation is stated as a key contribution (L74), but there is only a small qualitative analysis in Figure 6 to evaluate this. Why are there no quantitative evaluations and comparisons to other baselines for novel view synthesis?\n* **Q3:** One of the core assumptions here is that better video generation should result in better robotic task performance (L23 - 25). This has been completely neglected in the experiments. Ideally, multiple video generation baselines must have been evaluated in Table 1, and their corresponding performance on Tables 2 and 3 must be reported. \n* **Q4**: Tables 2 and 3 do not report standard deviations in performance over multiple trials --- how significant are the differences in performances between methods? Are they within noise margins?\n* **Q5:** The results in Tables 2 and 3 are a bit confusing because different rows use different visual inputs. It is hard to get an apples-to-apples understanding of the relative performances between methods. Is it possible to equip other methods with the depth sensor (i.e., S-RGBD)? Is it possible to equip EnerVerse with G-RGB?\n* **Q6:** Why are baselines across Tables 2 and 3 different? \n* **Q7:** L277 - Real-world experiments are only qualitative. Can we get quantitative results and comparisons to a subset of baselines in the real-world?\n* **Q8:** L312 - What is the impact of the gaussian-splatting based approach to reduce the Sim2Real gap? This has not been studied in the experiments.\n* **Q9:** There are no baselines that leverage multi-view representations. The idea of using multiple views to enhance model representations and robotic task performance is not new (e.g., MVWM: https://arxiv.org/pdf/2302.02408, RoboHorizon: https://arxiv.org/abs/2501.06605). How do these compare to the proposed approach?", "was_revised": true, "final_justification": "I thank the authors for their patience and efforts in this process. After multiple rounds of discussion with the authors, several points were clarified about the methodological details. Authors also pointed to existing experiments from appendix to address some concerns (e.g., quantitative real-world results of OpenVLA vs. EnerVerse) and performed new experiments to address weaknesses in the experimental section (listed below).\n* Standard deviations were included for OpenVLA and EnerVerse on LIBERO\n* An additional video generative method (LTX) was included in addition to DynamiCrafter. The proposed EnerVerse method improves on the video generation quality with both LTX and DynamiCrafter. This further strengthens the significance of EnerVerse.\n* 3D video generation (DynamiCrafter) was compared against 4D scene generation (EnerVerse) on robotics tasks (LIBERO-Spatial), and the value of 4D scene generation was clearly demonstrated for robotics control.\n* To address the lack of common baselines between LIBERO and CALVIN, OpenVLA was evaluated on both benchmarks. EnerVerse outperforms in both cases.\n* Impact of 4DGS was quantitatively studied through a very small-sample human study of hallucinations occurring with and without 4DGS. The authors have further promised to extend this analysis to impact for other experiments in the paper.\n\nWhile I still have concerns about the final paper requiring significant changes to address the clarity concerns and add new experiments, I feel more confident to take the leap of faith that these will be incorporated into the paper effectively. Therefore, I am raising my rating to **Accept**. \n\n**A request to the authors for the final paper version:** Since this was a short rebuttal turnaround period, I expect not all new experiments may have been run to completion / may have been smaller scale than needed. I hope the authors can have a more complete version of these experiments for the final paper. Some examples below:\n\n* **LTX experiments:** *However, due to time and computational constraints during the first-round discussion period, we were unable to complete the full training of all variants. In this round discussion, we managed to provide the complete results. However, both LTX variants were trained for only ~30k steps, which is fewer than the fully trained DynamiCrafter setup due to the time limitations*\n\n* **Human study for 4DGS impact:** *For evaluation, we asked two human experts to assess the generated videos specifically for diffusion-induced hallucinations.* --- it would be nice to have a larger sample size of human experts\n\n* **Other quantitative studies for 4DGS:** *We acknowledge the need for more comprehensive quantitative analyses. To further support the benefits of 4DGS, we plan to conduct additional experiments on other tasks and metrics in the revised version of our paper.*", "rebuttal": "We thank the reviewer for their valuable comments. We have addressed the concerns on methodological clarity and experimental completeness in detail below, with some additional information already included in the appendix. We will revise and polish the main paper to ensure clarity.\n\n> Q1: velocity term\n\nAs mentioned in L93, we follow [1] to implement the v-prediction. Instead of predicting the noise $\\epsilon_t$, the model predicts $v_t$, defined as:\n\n$v_t = \\alpha_t \\epsilon_t - \\sigma_t x_0$\n\nHere, $\\alpha_t = \\sqrt{\\bar{\\alpha}_t}$ (signal scale) and $\\sigma_t = \\sqrt{1 - \\alpha_t^2}$ (noise scale), consistent with the forward process equation $x_t = \\alpha_t x_0 + \\sigma_t \\epsilon_t$.\n\n> Q2: 4D Model Details\n\n Due to space limitations, we moved the model architecture details to **Appendix L591–L604**. Additional details are as follows:\n\n* **Input and Output:** The model takes as input a task prompt c, $m$-view observation images($m \\ge 1$)$O_t$, and their camera parameters, and outputs the predicted $m$-view videos.\n* **Loss:** We use the v-prediction diffusion loss to supervise $m$-view image generation, as noted in Q1.\n* **Dataset:** As stated in L155–L158, we use both single-view academic datasets and self-constructed multi-view datasets from the simulator. For single-view videos, camera extrinsics (relative to the robot base) are manually and approximately estimated; for simulator videos (examples in L606), GT parameters are available. During inference, we use the extrinsics provided by the robot hardware. We plan to release the simulator rendering scripts and estimated extrinsics.\n\n> Q3: Mechanism on data flywheels.\n\nAfter pretraining the 4D base model (e.g., EnerVerse-G) on the dataset mentioned in Q2, the model takes as input multi-view images (with camera parameters) and a task prompt, and outputs predicted multi-view videos. Following the standard diffusion process, we add noise to the latent features from all views. However, in the **data flywheel**, at least one camera (n ≪ m) mounted on the robot provides full visual observations. For these views, we skip noise injection and instead use their latents as guidance for generating other views. We refer to these as **sparse observations**.\n\nTo support this input format, we fine-tune the base model. In the 4D latent space of shape CVTHW, no noise is applied to latents of the n observed views (e.g., C₁THW). During inference (see L153–Fig.4), for examlpe, we use one full video, two rendered initial frames, three camera parameters, and a task prompt. The model then generates denoised videos for the two target views. These, along with the original observed video and parameters, are fed into 4DGS to construct the 4D Gaussian representation. Then rendered frames are obtained from GS, then iteratively.\n\n> Q4: Policy Model Trianing.\n\nAfter obtaining EV-G (L139), we attach a policy head $h\\_\\theta$, composed of stacked DiT blocks, to enable action generation, resulting in EnerVerse-A. The goal of EnerVerse-A ($f\\_\\theta$) is to estimate the clean action from a noisy one:\n\n$\na_t^0 \\leftarrow f_\\theta(c, O_t, a_t^k, k) = h_\\theta(E, a_t^k, k),\n$\n\nwhere the latent vector \\$E\\$ is extracted fromthe U-Net backbone in EnerVerse-G. We train \\$f\\_\\theta\\$ by minimizing the denoising mean-squared error:\n\n$$\n\\mathcal{L}(\\theta) := \\text{MSE} \\left( a_t, f_\\theta(c, O_t, \\sqrt{\\bar{\\alpha}^k} a_t + \\sqrt{1 - \\bar{\\alpha}^k} \\epsilon, k) \\right), \\tag{2}\n$$\n\nwhere \\$k \\sim \\text{Uniform}({1, \\ldots, K})\\$. As described in L146, we predict a chunk of actions in one shot to improve temporal consistency and reduce error accumulation by decreasing decision frequency.\n\nDuring inference, \\$f\\_\\theta\\$ takes $O\\_t\\$ and \\$c\\$, which pass through the video diffusion backbone once to compute the latent vector \\$E\\$. This \\$E\\$ is cached and reused across multiple denoising steps for action generation, and no past actions are used.\n\n> Q4:  DC for long-video generation.\n\nIn L183, we primarily adopt FreeNoise [3], a tuning-free paradigm for extending video length via noise rescheduling. Its core mechanism involves local noise shuffling, constructing long-range correlated noise sequences, and applying window-based attention fusion. In our implementation, the second inference stage takes the results from the first stage as input, and FreeNoise is applied at each inference stage.\n\n> Q5: Does DynamicCrafter get the advantage of the same training data as EnerVerse?\n\nFor a fair comparison, the EnerVerse reported in Tab 1 and Fig 5 is trained solely on RT-1. Based on the generation results, we attribute the performance gains primarily to our model’s architectural design.\n\n> Q6:  DC the only video baseline\n\nOur work targets world models for embodied agents on single-GPU or edge-GPU platforms, where video generation is not the core focus. While recent video diffusion models (e.g., I2VGen-XL, Stable Video Diffusion, LivePhoto, PhysGen) achieve high visual quality, they are often unsuitable for our setting—being too large (I2VGen-XL), reliant on extra components like simulators (PhysGen), or closed-source (LivePhoto). In contrast, **DynamiCrafter** is a well-maintained open-source model with public weights and serves as the basis for AVID. Considering our compute and latency constraints, we selected DynamiCrafter for reproducibility.\n\nSimilar to prior works such as VPP[4] and VidMan [5], which adopt a single video model due to high training cost, we additionally selected **LTX** [6] for further analysis because of its efficiency and favorable inference speed for robotic tasks. We report PSNR results showing **EnerVerse-LTX (27.8) > EnerVerse-DC (26)**, likely benefiting from LTX’s advanced architecture and higher-quality pretraining, and demonstrating the flexibility of our method to different video models. These results and the discussion of additional models will be included in the revised paper.\n\n> Q7: No other baselines...(Multi-view) \n\nThe availability of open-source multi-view video generation models with publicly released weights is limited. Most SOTA methods are either closed-source, require substantial computational resources for fine-tuning, or depend on bespoke rendering pipelines, making them impractical as baselines under realistic constraints.\n\nFurthermore, the objectives of embodied video generation differ significantly from general video synthesis. Embodied scenarios demand high scene consistency and task-relevant motion—factors not well captured by conventional evaluation metrics. Given the lack of suitable multi-view baselines, we compare against SOTA single-view generators: Kling-1.6, Hailuo, and Open-Sora, under identical textual and visual conditions. In a human preference study with 20 annotators, EnerVerse received 75 votes, outperforming Kling (64), Hailuo (30), and Open-Sora (1).\n\nWhile EnerVerse may be less competitive in general-purpose video generation, it is more aligned with the requirements of embodied video synthesis. Our primary objective is to validate the utility of multi-view prior synthesis for robotic policy learning. Accordingly, we present qualitative results to establish feasibility and leave more comprehensive quantitative evaluations—including those involving real-robot policy improvements—for future work as more suitable baselines become available.\n\n>Q8:  assumption better video ... better robotic task performance (L23)\n\nTo clarify, in L23 we do not assume a naïve one-to-one mapping whereby any uplift in general video-generation metrics (e.g. FVD, SSIM) will automatically yield better robotic control. Rather, our claim is that representational alignment—i.e., adapting the video model’s latent space so that it encodes 3D, action-conditioned dynamics brings gains.\n\n> Q9: not report standard deviations\n\nThe selected baselines, such as GR-1 and SUSIE, did not report standard deviations. To ensure a consistent comparison, we also omitted them in the main paper. Here, we provide results comparable to OpenVLA.\n\n| | Spatial | Object | Goal | Long | \n|-----------|-----------|--------|-|----|\n| OpenVLA | 84.7 ± 0.9    | 88.4 ± 0.8   | 79.2 ± 1.0 | 53.7 ± 1.3 | \n|Our | 92.1 ± 0.7 | 93.2 ± 0.9 |  78.1 ± 1.1 | 73.0 ± 1.2 |\n\n>Q10: use different visual inputs in Tab 2/3\n\nOur method with a single S-RGB input outperforms all other approaches, including those using S-RGB or S-RGB + G-RGB, demonstrating its effectiveness. Current most visual-policy or VLA models do not support depth input, and our current pipeline requires a static camera for G-RGB.\n\n>Q11: baselines across Tab 2 and 3 are different.\n\nIn policy learning, CALVIN and LIBERO are two representative large-scale benchmarks, and most prior works evaluate on only one of them. To better demonstrate the effectiveness of our method, we conduct experiments on both.\n\n>Q12: quantitative results and comparisons to a subset of baselines in the real-world\n\nIn Tab 6(L531), we provided the quantitative results for ours and OpenVLA.\n\n>Q13: The impact of GS\n\nWe provide the visualization results in the Appendix. The main goal of GS is to enhance the geometric consistency and reduce video diffusion hallucinations (especially under robotics arm - object occlusions). We will provide more visual examples and deeper analysis in the revised version.\n\n>Q14: No baselines that leverage multi-view.\n\nIn Tab 2, we provide the MAIL with two S-RGB results, and many baselines with S-RGB+G-RGB. MVWM focuses more on the masked autoencoder, while RoboHorizon emphasizes more on the LLM's usage.\n\n[1]Progressive distillation for fast sampling of diffusion models\n\n[2]Learning fine-grained bimanual manipulation with low-cost hardware\n\n[3]Freenoise: Tuning-free longer video diffusion via noise rescheduling\n\n[4] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations\n\n[5] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation\n\n[6] LTX-Video: Realtime Video Latent Diffusion"}, {"review_id": "DEtOPX0oEP", "reviewer": "Reviewer_hqTx", "summary": "This submission focuses on learning a multi-view latent space for robotic manipulation using a world modeling objective. The paper proposes three systems: EnerVerse, a diffusion world model that takes patches from prior images and predicts future patches based on a context variable, EnerVerse-A, a diffusion transformer policy trained on top of the EnerVerse latent space, and EnerVerse-D, a system for using Gaussian Splatting to generate novel views used to train the world model. Overall, this submission describes a large system. EnerVerse is tested in its ability to predict future frames and in terms of being able to learn generalizable policies. Experiments are performed in LIBERO and CALVIN in simulation as well as on a real-world AGIBOT platform.", "questions": "Could you add substantially more details to your description of EnerVerse world model and EnerVerse-D?\n\nWhat is the conditioning variable $c$? Is it possible to condition the world model on actions?\n\nIs it possible to use the world model to generate synthetic training data for the policy?", "limitations": "The authors describe inaccuracies of video prediction methods and heuristic camera pose selection as the main limitations of their work in Section 5.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n**Novel sparse memory mechanism.** It is proposed to subsample patches from each frame in the video history so that a longer history can be provided to the model. This idea is helpful both because it allows for a longer history and also because it serves as a data augmentation method (by randomizing which patch is selected) leading to better generalization. This design choice is ablated in Table 4, showing that having longer memory leads to a major improvement in policy learning success rate. Further, Figure 7 shows a convincing example of the world model remembering that a stove was turned on at a prior time step.\n\n**Strong multi-view video generation results.** Figure 6 depicts an example of multi-view video generation that demonstrates object consistency across multiple views in a simulated and a real-world scene. This is supported by a quantitative evaluation in Table 1.\n\n**Strong policy learning results.** The authors demonstrate that policies trained based on the latent space of EnerVerse achieve state-of-the-art success rates on the LIBERO benchmark. LIBERO specifically tests generalization to different objects, layouts, goals and background and this is a strong indication that EnerVerse learns generalizable features. Further, CALVIN results in Table 3 are on par with strong baselines.\n\n## Weaknesses\n\n**The method description is incomplete.**\n* EnerVerse: It is stated that EnerVerse is based on DynamiCrafter, but the paper does not actually specify the size of the EnerVerse model (in the number of parameters) or any training details (training steps, optimizer setting, dataset size, etc.). The only training details provided are about the EnerVerse-A policy head. Further, I believe the paper does not specify what is the content of the context variable $c$ that is used to condition the video prediction model.\n* EnerVerse-D: Section 2.2 describes the high-level idea of generating multiple views using a video prediction model, training a 4D Gaussian splat on this sequence and generating novel views from the Gaussian splat to further train the video prediction model. However, the submission does not contain specific details as to how this system was implemented. This limits the impact of this paper, as it is unlikely that researchers could reproduce the system from its description.\n\n**The experiments description is incomplete.** On line 157, it is stated “Furthermore, we constructed a dataset containing multi-view video ground truths using the Isaac Sim simulator [31].” The submission does not contain further elaboration as to the size of the simulated dataset or the diversity of the generated scenes. Therefore, it is unlikely that researchers could independently reproduce this training setup.\n\n## Minor comments:\n* Line 86: define $J$.\n* Line 93: define $v$.", "was_revised": true, "final_justification": "I think this submission contains novel methodological contribution as well as significant and interesting results. The main drawback is that the submission describes a large and complicated data generation and robot learning system, and the authors provide limited details about (a) the simulated dataset used to train the model and (b) the \"data flywheels\" used to generate additional data. This was also pointed out by Reviewer C5W7. The authors provided more details in the rebuttal but the discussion is still at a fairly high level. Therefore, the community may benefit from some ideas in the paper, but it is highly unlikely that the system described in this paper will be replicated in future works.", "rebuttal": "We appreciate the reviewer's thoughtful feedback. We have thoroughly addressed the concerns regarding methodological clarity and experimental coverage in the responses below, and note that some supporting details are already included in the appendix. We will revise the main paper to improve clarity and ensure all aspects are clearly presented.\n\n >Q1: More Training details\n\nWe thank the reviewer for this valuable question. And we provide the training details and hyperparameters in the following table, and this table will be added to the main paper in the revised version.\n\n| Hyperparameter       | Configuration                                                                                                 |\n|----------------------|---------------------------------------------------------------------------------------------------------------|\n| **Diffusion Setup**  | - Diffusion steps: 1000  - Noise schedule: Linear  - beta_0: 0.00085  - beta_T: 0.0120 |\n| **Sampling Parameters** | - Sampler: DDIM  - Steps: 500                                                           |\n| **Input**            | - Video, resolution: 320 x 512  - Chunk size: 8  - Encoded with VAE |\n|                      |- Language prompt c, tokenized with T5  |\n|                      | - Camera Parameters, encoded with ray direction map (L118 in main text) |\n| **UNet**             |  - Latent image channels: 4  - Ray map channels: 6  - z-shape: 40 x 64 x 4 |\n| **Temporal Attention** | - Attention resolutions: 64, 32, 16  - Head channels: 64  - Conv layer num: 4  - Temporal kernel size: 3,1,1  -> 40 x 64 -> 20x32 -> 10x16 |\n| **Spatial Attention** | - Attention resolutions: 64, 32, 16  - Head channels: 64  - Conv layer num: 4 |\n| **Video Training**       | - Learning Rate: 5x 10e-5  ; - Optimizer: Adam ; - Batch size per GPU (single-view): 8 ; - Batch size per GPU (multi-view): 1 |\n|                      |  - Parameterization: v-prediction  ; - Max steps: 100,000 ; - Gradient clipping: 0.5 (norm)|\n| **Policy Training** | Similar to video training, but with sample-prediction parameterization|\n| **Num of Parameters** | - Base model: DynamiCrafter 1.4B  - Policy Head: 190M (DIT Blocks)   - VAE: 83.7M(frozen)|\n\n>Q2: Details on the dataset\n\nWe sincerely thank the reviewer for this question. And we provide the pre-training dataset details as follows. And we will add these information in our main text in the revised version.\n\n\n| Dataset         | Frames   | Episodes   | Camera View                     | Category    |\n|-----------------|-------------------|------------|---------------------------------|-------------|\n| RT-1            | 3.7M              | 87K        | Egocentric                      | Real robot  |\n| Language Table  | 7.0M              | 442K       | Front-facing                    | Real robot  |\n| Bridge          | 2.0M              | 25K        | Egocentric                      | Real robot  |\n| RobotTurk       | 72k               | 1.9k       | Front-facing                    | Real robot  |\n| ManiSkill       | 4.0M              | 30K        | Front-facing                    | Simulation  |\n| Issac Simulator | 3M                | 40k        | Egocentric + 8 Third Views     | Simulation  |\n\n>Q3: context variable c\n\nIn Line 89, we descibe the c as the textual prompt. In our implementation, this textual prompt is encoded by a frozen T5-base and an MLP is used to project the output to the token space.\n\n>Q4: Details on EnerVerse-D implementation.\n\nAfter pretrain the EnerVerse-G, it is capable to receive m-view images (with the camera paramters), task prompt and then produce the predicted m-view videos. Following the standard diffusion process, we add noise to the latent from all m-view images. However, in the data flywheel setting, we have at-least one (N) view camera mounted on the robot has complete and clean visual observation images, and we did not need to predict the future frames for this view and thus *NO* noise is appied on the correponding latents. Instead, those visual observation will serve as a guidance for other views video generation. We finetune our base model to let it capable to receive such input format. In view of 4D latent space with dimension of CVTHW, for tensor coming from the camera images input, e.g. C1THW, we did not apply noise.\n\nDuring inference, we use clean N-view camera images and their camera extrinsics to obtail M-view rendered images with the depth wraping, as shown in Figure 4. These M+N view videos with text prompt c then go through the diffusion model to obtain the cleaner denoised M-view images. After that, these M+N view videos and their cooresponing camera parameters are used to construct the 4DGS representation. We use the officaial implementation of 4DGaussian [1], and followed most default settings, the only change is that we use the depth information to do the initialization process and the depth of deformation model is set to 1. The 4DGS model produces higher-precision, geometrically consistent 4D representations. Then, we again render M-view images from the 4DGS to obtain higher-quality observations. These M+N video videos then agian go through diffusion process and 4DGS process to iteratively improve the quality.\n\n>Q4:  The submission does not contain further elaboration as to the size of the simulated dataset or the diversity of the generated scenes. \n\nWe thank the reviewer for this valuable suggestions. The reason why constructed that simulated dataset is that, At the time of this work, all available embodied datasets provided only single third-person camera views, which are insufficient for multi-view generation tasks. We provided visual examples for the simulator collected data in Appendix Figure 13, and this dataset is with about 40K episodes. The collected episodes is with 8 tasks, and crossing industrial and home scenarios. The task descriptions are as following, and we plan to release some splits in the near future to enable other researchers could independently reproduce this training setup.\n\n- Task Descriptions:\n    1. place trash into dustbin\n    2. pick fruit into basket\n    3. pick toy into box\n    4. insert pen\n    5. place bag\n    6. open drawer\n    7. place fruits\n    8. arrange workpieces \n\n>Q5: Symbol defination in Line 86  and Line 93.\n\nThe **J** in Line86 describes the frame number of rendered images. We used the **J** for rendered views $r_t^{1:J}$ instead of **K** used for $o_t^{1:K}$ is that the rendered views and the obseravtion camera images could have different frames. For an example, in the EnerVerse-D, the camera obervations have complete offline videos, in such case $K=T$, while the rendered views could only have one frame.\n\nWe follow [2] to implement the v-prediction. Instead of predicting the noise $\\epsilon_t$, the model predicts $v_t$, defined as: $v_t = \\alpha_t \\epsilon_t - \\sigma_t x_0$.\n\n>Q6:  Is it possible to condition the world model on actions?\n\nWe thank the reviewer for this insightful question. While our current framework focuses on generating videos and subsequently predicting action policies from text instructions and image inputs, we believe it is easy to adapt to support action-conditioned video generation. Although this is not the primary setting explored in our work, the extension is conceptually straightforward: if physical action sequences can be represented in text or visual modalities, they can be directly integrated into our framework, enabling an action-to-video generation process.\n\nMoreover, we view our framework's ability to ground text prompts and visual observations as a foundational capability for building action-conditioned video models. We consider this a promising direction [3] and plan to explore it in future work.\n\n\n>Q7: Is it possible to use the world model to generate synthetic training data for the policy?\n\nThank the reviewer for this valuable questions. It is a promising and active research domain to use world model to generate synthetic training data for the policy learning as done ine [4], where the model generates both video frames and action lables. For this work, we chose to focus more on the action output, e.g. the robot policy learning.\n\n## References\n\n[1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering\n\n[2] Progressive distillation for fast sampling of diffusion models \n\n[3] 1X World Model: Evaluating Bits, not Atoms\n\n[4] DreamGen: Unlocking Generalization in Robot Learning through Video World Models"}, {"review_id": "d8MMmODDdT", "reviewer": "Reviewer_J1q3", "summary": "This paper proposes *EnerVerse*, a generative foundation model for robotic manipulation that predicts embodied future spaces via a chunk-wise autoregressive video diffusion framework augmented with sparse memory. It introduces Free Anchor Views (FAVs) for flexible multi-view 3D scene modeling and integrates a 4D Gaussian Splatting pipeline (EnerVerse-D) to reduce the sim-to-real gap. Coupled with a policy head (EnerVerse-A), the approach achieves state-of-the-art performance on manipulation benchmarks, demonstrating robust long-horizon reasoning and real-world deployment capabilities.", "questions": "1. Could the authors clarify why they claim that “wrist-mounted cameras complicate policy learning”? Are there any supporting studies or empirical evidence for this assertion?\n2. Does multi-view generation induce severe hallucinations in long-sequence tasks, and if so, does this directly degrade action generation quality？", "limitations": "See weakness and questions.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strength:**\n\n- The paper presents a comprehensive view from data generation to video synthesis and action planning, featuring a novel chunk-wise autoregressive diffusion model with sparse contextual memory for embodied future reasoning.\n- The integration of 4DGS optimization into multi-view video generation establishes an effective data flywheel, leading to high-quality, realistic videos.\n- The proposed FAV-based embodied future space approach, linked with policy planning, enhances spatial understanding and demonstrates clear potential for robotics applications.\n\n**Weakness:**\n\nWhile the proposed work is promising, further real-world robotic experiments are necessary. These experiments would bolster the evidence that the data generation pipeline effectively mitigates the sim-to-real gap, which is one of the primary motivations for this research.", "was_revised": true, "final_justification": "After reading the rebuttal and the other reviews, I am raising my score to 6. The paper provides convincing evidence that a world model can be trained and used for manipulation. The extensive real‑world experiments help mitigate the ambiguity common in simulation benchmarks (as noted by Reviewer C5W7). I also appreciate the use of multiple cameras to build the world model; this is a more realistic setup, whereas much prior work assumes a single camera. Overall, I recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for recognizing the contributions of our work and for providing constructive comments. We have carefully considered your suggestions and provide detailed, point-by-point responses below.\n\n>Q1: further real-world robotic experiments are necessary.\n\nWe thank the reviewer for this valuable suggestions. In the current submission version, we conducted real-world experiments on the tasks Block Placing, Plastic Objects Sorting, and Fruit Sorting as shown in *Section A in the Appendix*. And we are trying to add new real-world experiments including *Bin Picking*， *Package Sorting*, etc. The visual results will be updated in our revised version paper.\n\n>Q2: Clarify why they claim that “wrist-mounted cameras complicate policy learning”? \n\nWe thank the reviewer for this questions. This statement is coming mainly from our setting, where the input for the model is the text prompt c, observation images o and the related inaccurate camera parameters. Those camera paramters would serve as the condition information for the multi-view generation diffusion process, then the denoised feature vector will guide the final cation planning, e.g. the pose of the EEF.  If we feed the images from the wrist camera, we also need to input its camera parameters. However, its extrinsic parts has directly replected the EEF's pose, which is slightly violating our settings. Besides, in our practical training experiments, we found that when we use both third-view images and wrist images for the training process, we would result in degraded generation performance, and the final policy performance is also suboptimal. As a result, our framework is more suitable to receive multi static view image input, similar to [1]\n\n>Q3: Does multi-view generation induce severe hallucinations in long-sequence tasks, and if so, does this directly degrade action generation quality？\n\nWe appreciate the reviewer’s concern. In fact, our multi‐view generator is built with cross‐view attention and trained on synchronized multi‐view video data, which strongly enforces geometric constraints and thus reduces hallucinations consistent with prior work on multi‐view consistency[1,2]. As shown in Figure 6, our model achieves high consistency across views, confirming its cross‐view coherence even over long sequences. Crucially, this consistency translates into better action generation: on the LIBERO-Long splits, a policy using only single-view S-RGB rollouts achieves 73 % success, whereas augmenting with two additional rendered views boosts success to 80 %. The improvement is largest compared with other splits, demonstrating that our multi-view conditioning enhances (not degrades) downstream control performance by mitigating hallucinated dynamics. Furthermore, in practice, unlike purely video generation setup,\n\n## References \n[1] RVT: Robotic View Transformer for 3D Object Manipulation\n\n[2] Zero-1-to-3: Zero-shot One Image to 3D Object\n\n[3]GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image"}, {"review_id": "gUYpwjC7au", "reviewer": "Reviewer_y6Qo", "summary": "The paper introduces EmerVerse, a generative robotics foundation model designed to construct and interpret embodied spaces for robotic manipulation tasks. The model uses a chunk-wise autoregressive video diffusion framework enhanced by a sparse context memory mechanism for long-term reasoning. It also integrates a multi-view video representation to address 3D grounding challenges and a 4D Gaussian Splatting (4DGS)-based data engine (EmerVerse-D) to bridge the sim-to-real gap. The policy head (EmerVerse-A) translates 4D world representations into physical actions, achieving good performance in simulation and real-world tasks.", "questions": "1. What is the core technical contribution beyond combining existing techniques?\n\n2. How much does each component (sparse memory, multi-view, 4DGS) contribute to performance?\n\n3. How sensitive is performance to different camera poses?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The chunk-wise autoregressive diffusion framework with sparse memory is a well-executed approach for long-term reasoning in robotic tasks. The multi-view diffusion generator and 4DGS-based data engine are well-motivated solutions to key challenges in robotics.\n\n2. The paper provides extensive experiments across multiple benchmarks (LIBERO, CALVIN) and real-world tasks, demonstrating superior performance over existing baselines. Quantitative and qualitative results strengthens the claims.\n\n3. The paper is well-written and clearly presents its methodology and results. \n\n---\n\n**Weaknesses**\n\n1. **Technical Novelty**: While EmerVerse demonstrates impressive engineering efforts and strong empirical results, its technical novelty is limited as it primarily combines existing techniques—video diffusion, multi-view rendering, and 4D Gaussian Splatting. The work’s strength lies in its system-level integration and scaling, but the core methodology does not significantly advance beyond established paradigms. The authors are suggested to clarify the key technical novelty compared to existing works.\n\n2. **Complexity & Computational Inefficiency**: While EmerVerse demonstrates strong empirical performance, its complexity and computational inefficiency raise concerns about practicality and novelty. The pipeline combines multiple heavyweight components—autoregressive video diffusion, multi-view rendering, 4D Gaussian Splatting, and a policy head—without clear justification for why this level of complexity is necessary. Inference is slow (20 seconds per multi-view chunk), making real-world deployment challenging. Crucially, the paper lacks ablation studies to show whether simpler, faster baselines could achieve comparable results, leaving the impression of an over-engineered solution rather than a fundamentally novel advance. \n\n3. **Lack of Baseline Comparisons**: The paper notably omits comparisons with recent state-of-the-art 3D-aware video generation methods like Gen3C [1], which achieves precise camera control and world consistency through explicit 3D representations. A direct ablation against 3D-consistent generators would better contextualize its contributions.\n\n\n[1] Ren, X., Shen, T., Huang, J., Ling, H., Lu, Y., Nimier-David, M., Müller, T., Keller, A., Fidler, S. and Gao, J., 2025. Gen3c: 3d-informed world-consistent video generation with precise camera control. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 6121-6132).", "was_revised": true, "final_justification": "While some complexity remains and direct comparisons to very recent 3D video methods are limited, the overall strong empirical results and thorough rebuttal support a borderline accept.", "rebuttal": "We greatly appreciate the reviewer’s recognition of our work and the constructive feedback provided. We have thoroughly reviewed your comments and address each point with detailed responses below.\n\n>Q1: While EmerVerse demonstrates impressive engineering efforts and strong empirical results, its technical novelty is limited as it primarily combines existing techniques—video diffusion, multi-view rendering, and 4D Gaussian Splatting. \n\nWe thank the reviewer for acknowledging our engineering efforts and strong empirical results. However, our work goes beyond a simple combination of existing components. Our central contribution is the introduction of a video-generative robotic world model that leverages video prediction to model the robot’s future visual latent space. This model is pretrained on large-scale robotic data, enabling effective domain adaptation from general video generation to robotic world.\n\nThe learned latent space forms the basis for policy learning. By attaching a lightweight, plug-and-play action head, we map the visual latent representation to actionable motor commands. This results in a unified framework that unifies robot sense and policy learning within a coherent pipeline, purposefully designed for real-world robotic manipulation. Each module is carefully introduced to support robotic world modeling, rather than assembled through ad-hoc integration.\n\n\n\n\n>Q2: While EmerVerse demonstrates strong empirical performance, its complexity and computational inefficiency raise concerns about practicality and novelty. \n\nWe thank the reviewer for acknowledging that our framework could achieve strong empirical performance. However, we have to point out the 4DGS is not used for our data flywheel components which is generally an offline procedure. And during policy inference, the video generation process is not necessary. Instead, as stated in Line 144, we only use the feature vector after the first video denoising step and this feature vector is then cachaed and used as the condition feature vector for multiple action denosing steps. The action is predicted with the action chunks (chunk size is 8 in our setting, means we predict 8 step actions from one inference). We achieve 8-step action prediction within 280 ms on a real-world robotic platform equipped with an NVIDIA RTX 4090, which is comparable to the current mainstream VLA model, which is also generally about 200ms - 350ms （for example, OpenVLA[1] is about 180ms, DexVLA[2] is about 310ms）. Furthermore, we demostrate its effectivess with our real-world robotic manipulation tasks.\n\n\n\n>Q3: Lack of Baseline Comparisons: The paper notably omits comparisons with recent state-of-the-art 3D-aware video generation methods like Gen3C...\n\nWe thank the reviewer for this suggestions. The concurrent work Gen3C [3] is open-sourced in GitHub on about Middle June, which is already after the submission deadline of the conference. Moreover, the goals of embodied video generation differ substantially from those of general video generation. Embodied settings place greater emphasis on scene consistency and task-relevant motion, which are not well captured by conventional video generation metrics. Due to the lack of multi-view baselines, we instead compared against SOTA single-view video generators: Kling-1.6, Hailuo I2V-01live, and Open-Sora-2.0. These models were conditioned on the same textual and visual prompts as our method to generate robot-centric videos. We then conducted a human preference study involving 20 annotators, who rated the outputs across dimensions such as scene consistency and object motion. Our method received 75 scores overall, outperforming Kling (64), Hailuo (30), and Open-Sora (1). Besides, in Gen3C, the camera moves around the target object, and the target object is generally with few movements, which differs greatly from our robotic manipulation tasks, where we use static cameras and robotic arm moves. We will add further discussions and comparsions in our revised versions.\n\n\n>Q4: How much does each component (sparse memory, multi-view, 4DGS) contribute to performance?\n\nThank you for this question. We’ve carried out ablations to isolate the impact of each component:\n\n- Sparse Memory: As shown in Figure 7 (video rollouts) and Table 4 (policy learning), adding sparse memory substantially improves long-horizon generation quality and yields a significant boost in task success.\n- Multi-view: Table 2 compares single-view versus multi-view rollouts on LIBERO: incorporating multi views reduces raises success rates, especially on the LIBERO-10 splits. \n- 4DGS: We provided some inintal visualization in the Appendix, Figure 14. We also conducted some qualitative analysis， and finding that 4DGS would enhance the cross-view geometric consistency, and  dramatically reduces diffusion-induced hallucinations (especially under robotics arm - object occlusions). We will enrich the revised manuscript with additional visualizations of 4DGS’s effects and deeper analysis.\n\n>Q5: How sensitive is performance to different camera poses?\n\nWe appreciate the reviewer's insightful question regarding the sensitivity of our method's performance to varying camera poses. To address this, we would like to highlight two key points that demonstrate the robustness of our approach against camera pose variations: Firstly, in the LIBERO benchmark, the front view camera mounted positions differ across different task scenarios. Furthermore, as noted in Lines 207 and 225, our policy evaluation leverages both the LIBERO and CALVIN benchmarks. Notably, the camera poses in these two benchmarks are fundamentally different.  The consistent performance of our method across these distinct setups directly validates its generalization capability to camera pose variations. Secondly, in our real-world experiments, we intentionally avoided labor-intensive camera calibration throughout the multi-day evaluation process. Instead, we relied solely on the default parameters provided by the robot's hardware client, which inherently contain non-negligible noise. The stable performance observed under such uncalibrated conditions further confirms the robustness of our method to practical camera pose inaccuracies.\n\n\n## References\n[1] OpenVLA: An Open-Source Vision-Language-Action Model\n\n[2] DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control\n\n[3]  Gen3c: 3d-informed world-consistent video generation with precise camera control"}]

Author rebuttal:

Meta-review: The paper initially received mixed reviews. Two reviewers leaned toward rejection while two supported acceptance. The reviewers agreed that the sparse memory mechanism is novel, the multi-view video generation approach improves performance, and the empirical results are strong, covering multiple benchmarks and real-world experiments. At the same time, some reviewers felt the writing was unclear and lacked important experimental details. They also noted that the method is complex, computationally inefficient, and may not scale well. In addition, comparisons with other video generation methods were missing, and the level of technical novelty was questioned.

In their rebuttal, the authors addressed most of these concerns, leading all reviewers to support acceptance. Some reviewers, however, remained concerned that significant revisions would be needed to integrate the rebuttal into the final version, and that the method’s complexity and reproducibility remain unresolved issues.

The AC agrees with the reviewers’ consensus and finds that the paper makes a valuable contribution, supported by strong empirical results that justify acceptance. At the same time, the AC emphasizes the importance of addressing the remaining concerns, and asks the authors to carefully integrate their rebuttal into the final version and provide sufficient details, including releasing code if possible, to ensure reproducibility.
Decision: Accept (poster)