COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
Authors: Yining Shi
Kun Jiang
Qiang Meng
Ke Wang
Jiabao Wang
Wenchao Sun
Tuopu Wen
mengmeng yang
Diange Yang
Venue: NeurIPS 2025 poster ()
Keywords: Generative Models
Autonomous Driving
Occupancy Forecasting
Occupancy Generation
ControlNet
Primary area: deep_learning
TL;DR: This paper proposes to add scene-centric forecasting as an effective control to enhance occupancy world models.

Abstract: World models are critical for autonomous driving to simulate environmental dynamics and generate synthetic data.
Existing methods struggle to disentangle ego-vehicle motion (perspective shifts) from scene evolvement (agent interactions), leading to suboptimal predictions. Instead, we propose to separate environmental changes from ego-motion by leveraging the scene-centric coordinate systems. In this paper, we introduce COME: a framework that integrates scene-centric forecasting Control into the Occupancy world ModEl. Specifically, COME first generates ego-irrelevant, spatially consistent future features through a scene-centric prediction branch, which are then converted into scene condition using a tailored ControlNet. These condition features are subsequently injected into the occupancy world model, enabling more accurate and controllable future occupancy predictions. Experimental results on the nuScenes-Occ3D dataset show that COME achieves consistent and significant improvements over state-of-the-art (SOTA) methods across diverse configurations, including different input sources (ground-truth, camera-based, fusion-based occupancy) and prediction horizons (3s and 8s). For example, under the same settings, COME achieves 26.3% better mIoU metric than DOME and 23.7% better mIoU metric than UniScene. These results highlight the efficacy of disentangled representation learning in enhancing spatio-temporal prediction fidelity for world models. Code is available at https://github.com/synsin0/COME.

COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
Yining Shil. Kun Jiangl.2 Qiang Meng', Ke Wang',
Jiabao Wang',
Wenchao Sun Tuopu Wen Mengmeng Yang Diange Yang
'School of Vehicle and Mobility, Tsinghua University State Key Laboratory of Intelligent Green Vehicle and Mobility Kargobot Inc *Nakkai University
Abstract
World models are critical for autonomous driving to simulate environmental dy- namics and generate synthetic data. Existing methods struggle to disentangle ego-vehicle motion (perspective shifts) from scene evolvement (agent interactions), leading to suboptimal predictions. Instead, propose to separate environmen- tal changes from ego-motion by leveraging the scene-centric coordinate systems. In this paper, introduce COME: framework that integrates scene-centric forecasting Control into the Occupancy world ModEl. Specifically, COME first generates ego-irrelevant, spatially consistent future features through scene-centric prediction branch, which are then converted into scene condition using tailored ControlNet. These condition features subsequently injected into the occupancy world model, enabling more accurate and controllable future occupancy predic- tions. Experimental results the nuScenes-Occi dataset show that COME achieves consistent and significant improvements over state-of-the-art (SOTA) methods across diverse configurations, including different input sources (ground- truth, camera-based, fusion-based occupancy) and prediction horizons (3s and 8s). For example, under the same settings, COME achieves 26.3% better mloU metric than DOME 4 and 23.7% better mloU metric than UniScene [[]] These results highlight the efficacy of disentangled representation learning in enhane- ing spatio-temporal prediction fidelity for world models. Code is available at htttp:://githut www. www.cco..sss.eeeeeee caamm/yynnnn////
Introduction
World models designed to discern the current state of the environment and predict subsequent states based on executed actions. This predictive ability of world models not only enables the assessment of decision-making consequences but also facilitates the generation synthetic data, which serves as crucial resource for training, testing, and simulation in autonomous systems. Consequently, world models have emerged as focal point of research in the field of autonomous driving, garnering substantial attention from both academia and industry.
The synthetic data generated by world models take various forms represent future scenes. These include 2D videos 6 [T
24 25], 3D lidar point clouds [ I [ and 3D occupancy
26 Irrespective of the representation format, the changes in the appearance of future scenes predicted by world models are predominantly governed by two key factors: 1) Ego-vehicle motion: The movement of the autonomous vehicle alters its viewing perspective,
grids 355
g
33
27
29
Work done during internship at Kargobot Inc. Corresponding authors: Diange Yang, Kun Jiang, Mengmeng Yang
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Egn-centric
Occupuncy Sequenee
Final Prediction
Large gap
Occupancy World Model
Scene -ontric
Forocasting
Control
Trjjectory
gap
COME
consisient
Scene Conditiun
ControlNet
Scene-centric
Sceno-centric prodiction
Figure 1: COME with both scene-centric and ego-centric representation. Compared to ego-centric evolution, scene-centric prediction shows smaller gap in the context of temporal evolution. COME uses scene-centric prediction results as an important guidance to enhance occupancy world model. leading to dynamic changes in perceived spatial features (e.g., perspective shifts, occlusions). 2) Scene evolvement: Natural changes in the environment, such agent interactions (e.g., pedestrian movements, vehicle collisions) and background updates (c.g.. traffic light changes, weather variations). Notably, these two sources of change exhibit distinct characteristies. In wide range of real-worhh scenarios, the background environment may remain relatively stable, with minimal changes over time (see Fig. D Instead, it is the motion of the ego-vehicle and the resulting shifts in the viewing perspective that contribute significantly to the dynamic changes in the world model's representation.
ofate-of-the-art (SOTA) world models, however, rely on neural networks to implicitly learn the intertwined effects of these factors, often resulting in suboptimal spatial consistency. As case study, the SOTA model DOME 4 achieves an average 27.10 10 mloU for 3-second occupancy prediction using ground-trutt trajectories. In contrast, when predictions are formulated in scene-centric coordinate system (where static backgrounds are decoupled from ego motion), vanilla U-Net achieves 39. 19.12 mloU-a 44% improvement. This disparity underscores the critical lack of explicit control over spatial consistency in existing methods, motivating the need for disentangled representation learning.
To address this, we propose COME, three-stage framework that leverages scene-centric coordinates to separate ego motion from scene dynamics (see Fig. 2 1) Pose-conditioned generative diffusion stage: A diffusion- bassed model with spatiotemporal Diffusion Transformer generates future occu- pancy maps by iteratively denoising future occupancy latents, using historical occupancy latents and pose or BEV layouts as conditional inputs. 2) Fixed-view forecasting stage: By transforming past and future frames into common coordinate system, this stage mitigates ego- ego-motion effects, enabling non-interactive occupancy prediction without explicit scene flow estimation. 3) Forecasting guided diffusion stage: The COME ControlNet design acts as the mechanism, transferring knowledge from fixed-view forecasts to variable-view generation. Structured as trainable copy of the generative model's first half, the ControlNet injects scene-centric condition features into the latter half via skip connections, enhancing generative realism and temporal consistency.
We test under multiple configurations nuScenes dataset and Waymo open dataset, and COME outperforms SOTA baselines by significant margins, with scene-centric settings demonstrating the largest gains. Key contributions include: 1) A divide-and-conquue strategy for occupancy world models, decomposing complex spatio-temporal prediction into ego-motion-agnostic and scene- dynamic components: 2) COME ControlNet, which leverages scene-centric forecasts guidance to improve generative fidelity and spatial consistency; 3) Empirical validation of disentangled representation learning, establishing state-of-the-ard performance challenging autonomous driving benchmark.
2 Related Works
2.1 World Model in Autonomous Driving
leverage 2D video representations, offering great scalability due to the easy accessibility of camera data. However, the lack of 3D geometry understanding limits their fidelity in autonomous driving applications. contrast, LiDARRbasedd representation 9
Visual world models 24 25
31 37] provides rich geometric comprehension but falls short in semantic-aware generation. 26 have emerged as a compelling alternative for world modeling, due to their ability to encode both geometric and semantic information
Recently, occupancy representation 35

---

simultaneously. Occupancy as intermediate representation also serves as strong geometric prior condition for downstream generation of driving videos 28 and LiDAR point clouds[T]]
Some occupancy-based approaches [3 33 leverage occupancy flow to forecast future scenarios.
While achieving promising results, they often depend on additional annotations and struggle to produce imaginative predictions. Recent works have shifted toward generative frameworks [35] 27 26]: for example, OecWorld [35] employs two-stage pipeline, first tokenizing occupancy with
VQ-VAE [23] and then predicting ego motion and scene evolution autoregressively. Inspired by the success of large language models, some works [27 26 further enhance the model interpretability during generation. Recent methods 4 [1 employ diffusion transformers (DiTs), demonstrating strong generative capabiliiiess for world modeling. In this work, we similarly adopt the DiT paradigm while introducing control features inspired by ControlNet [4]] - a pioneering framework for conditional generation in 2D images using multi-modal inputs (e.g., depth maps, edges, or sketches). By incorporating explicit control signals, our method achieves superior spatial consistency and prediction accuracy, establishing new state-of-the-arr performance for occupancy world models.
2.2 4D Occupancy Forecasting
Occupancy world models and 4D occupancy forecasting methods share a common paradigm of predicting future occupancy states. The key distinction is that occupancy forecasting typically targets short-term scene evolution and does not involve predicting future ego trajectories. Oce4Cast [5] and Cam4DOce [17] establish LiDAR- -bade and camera- based benchmarks, respectively, and propose baseline models using temporal recurrent networks such as ConvLSTM [9]. To address the high cost of 4D occupancy annotation, 4D-Occ-Forecastiee future occupancy and employs differentiable depth rendering for self-supervised learning. Building upon this, Vidar [32] and UnO [2] introduce latent rendering and continuous 4D fields, respectively. to further enhance self-superrised learning for 4D occupancy prediction and related downstream tasks. In our work, the scene-centric prediction module adopts similar paradigm to 4D occupancy forecasting predicting future occupancy while explicitly factoring out the influence of ego trajectory. This design establishes a spatially consistent control prior that effectively guides the learning process.
[10] leverages future point clouds as proxies for
Methodology
This section presents our COME, a framework that integrates scene-centric forecasting Control into the Occupancy world ModEl. We detail the three main components of COME (illustrated in Fig. 2 in Secs. 3.1 to 3.3 Finally, Sec. 3.4 describes the training objectives and the overall training pipeline.
3.1 Occupancy World Model
We first describe our modified baseline world model, which is capable of performing the generation task independently. Following DOME [4, our model leverages diffusion Transformers (DiTs) for the superior fine-grained and imaginative generation compared to auto-regressive counterparts 35 27 26.. Given historical observations X1:1. ego-vehicle states Pı:t and other possible inputs (e.g.. BEV layouts), the occupancy world model predicts future occupancy & while optionally forecasting future trajectories P++1:t4r. We treat future trajectories - whether ground-truth or predicted by planning module- - as available inputs for the generative model for two key reasons: (I) It ensures comparability with existing methods, since some directly generate with ground-truth trajectories generation, which is essential
while others predict by themselves. (2) It enables for world models to produce diverse and controllable outputs.
Our model architecture comprises two principal components. The first is a collection of input encoders that transform historical observations into compact representations for efficient processing during the diffusion stage. We adopt the trajectory encoder from DOME 4 to encode trajectories, and apply a max-poot layer for BEV layouts when available. For the occupancy data, we utilize a Occupancy Variational Auto-Encoder
VAE) consisting of an encoder network qo (z|x) and a decoder network po(x(z)) Given input occupancy x E RBxWxD, qa (z) encodes it into a continuous latent variable z (a(x))). Conversely, px(x is able to reconstruct the occupancy from the latent variable. This design enables the VAE to compress high-dimensional occupancy data into compact latent space, facilitating more efficient processing in the generative model.
3

---

Occupancy World Model
Pose Time
Spatial Artention
Tempooral Attention
Spatial Artention
Tanporaal Arcention
Sputial Artession
Temporal Attontion
Spotial Ateesion
Tempocal Attontion
VAR Engoder
FAF
HISHT
Decoder
Kist
1/2
N/2
Reesicer
Semne-contric
Forecasting
N/2
semv
Gaag
Sparial Attention
Temporal Attentonn
Sputial Attetion
Temporal Alfention
Elementoise Add
Zeco Cooy
Zoro Cony
Zero Cony
Zero Coay CoDY
VAS Emnoder
Concellatitte
CORY
Transformation
Pose Time
Masking
ControlNer
Figure 2: The proposed COME framework comprises three main modules: (1) an Occupancy World Model that predicts future occupancy using historical observations and other inputs (e.g., poses, time steps, BEV layouts): (2) Scene-centric Forecasting Module that produces spatially consistent scene predictions by eliminating the influence of ego motion; and (3) the COME ControlNet which converts the scene conditions from the forecasting module into control features that are subsequently injected into the world model for controllable and geometrically coherent occupancy generation.
The second component is the spatial-temporal Diffusion Transformer modified from the DOME which consists of alternating stacked spatial and temporal layers. To reserve interface for later feature conditioning, we introduce skip connections between every early and late layer pair, similar to the architectural designs in UNet [18] and HunyuanDiT [3]. For example, the last spatial block takes the first spatial block as a skip input. For each block the second half, the input and the skip are concatenated in channel dimension and undergo simple MLP for feature fusion.
The model processes latent features encoded from occupancy data, along with optional inputs, e.g. BEV layouts if available. Trajectories are converted into trajectory condition and injected into each spatial-temoraa block, enabling trajectory-aware generation. In the end, the outputs are decoded through (z)x x) into occupancy predictions conditional to various future time steps and ego poses.
3.2 Scene-centric Forecasting Module
Future occupancy prediction represents a complex temporal modeling challenge that re- quires reasoning about two key factors: (1) the natural evolution of the scene (e.g., moving objects, infrastructure changes), and (2) the planned trajectory the ego-vehicle. Our foundation world model described in Sec. 3.1 treats the future trajectory of the inherent nature of scene evolution, leading to spatially inconsistent scene evolution in the limitation also confirmed in later experiments. To address this, pro- pose a scene-centric forecasting module that produces more coherent seene evolution and generates scene-conditioned inputs for the world model.
condition but has limited understanding
generated occupancy
In practice, the scene exhibits relatively smaller changes when represented in a unified coordinate frame compared
ego-centric occupancy representation. In Occ3D-nuScenes static voxels account for over 92.7% of all occupied voxels, suggesting that forecasting the scene in a shared coordinate system simplifies the task without requiring complex designs. In our implementation, find that a simple UNet [18] suffices for high-quality future predictions, where the skip connections effectively preserve static voxels, while the multi- scale structure captures dynamic elements.
To generate module inputs, we transform the historical occupancy sequence X1.t into unified coordinate frame at timestamp l. Each frame x; is transformed using ego-vehicle states: x = T(x;, Pi, Pt) for 1,2,.... This transformation implemented by initializing an empty grid x IR DxW and populating each voxel via nearest-neighbor sampling from x,, using

---

the coordinate transformation from to Pi- The resulting aligned sequence x is stacked into Dt H W tensor, which serves as the module input.
Subsequently, our UNet produces an output tensor of dimensions Dr x H x W, which is decoded to obtain the future occupancy sequence 8141+47- These predictions, formulated in the coordinate frame of ego-pose pt, are then transformed to the vehicle's future poses via 9; = T (t.PPt.)))))) pt). Each resulting occupancy Y; encoded by the Occ-VAE encoder, producing the final scene conditions 2; qo (z) P:), for = + 1, ... +T
3.3
COME ControlNet
The COME ControlNet, which encodes scene conditions } E RC xW into implicit control features, consists of NV blocks inspired by HunyuanDiT As shown in Fig. 2 the first N/ 2 blocks trainable copies of the corresponding spatial-temporal blocks from the occupancy world model, while the remaining blocks are the zero convolution layers. Each of these layers outputs control Cx XW, for N/2 + 1,.... .N, where T denotes the number of future
feature {cī] } 1=1 frames, and C, H1, W represent the channel depth, height, and width dimensions, respectively.
However, we observe that [_= is not directly suitable as control prior for world model. This limitation arises because the forecasting module, due to its simple structure, lacks sufficient capacity for imagining future states. Consequently, its predictions in historically unobserved regions may bring noise, degrading the world model's s generative performance - a finding corroborated by late experiments. Thus, filtering unreliable features from control features is crucial for robust generation.
To address this, we propose visibility masking strategy based on 3D spatial relationships. For each future timestamp E 1,++,, feature c,, namely the ý; predicted by the scene-centric forecasting module. We then construct binary voxel mask m, Pxx where a value of indicates that the corresponding voxel center in y, is observable in historical occupancy data, and 0 otherwise. This allows us to derive an invisibility mask M, E 2. xWi that quantifies the reliability of each spatial feature in c/:
+ r). we first trace the root source of the control
D 5 (w+1))bb Σ
M,h, w) =1 D
Σ
Σ P-w--W
mi u, <E
(1)
8
d=1 u=h-d
where 5, HH W/W1 and is pre-defined threshold. In practice we set = 0.5. The I is an indicator function to check whether proportion of historically observed voxels within each pillar (corresponding feature location (h, w) in c²)) is large enough. Then, we suppress unreliable features via element-wine weien multiplication: c, f- c. @ M, Finally, these refined control fcatures are injected into the world model through residual addition at corresponding layers, ensuring robust and controllable generation. The control features are added to skip features and concatenated together to the input features for each blocks in the second half of the world model.
3.4 Training Pipelines
Inspired by ControlNet 34 we employ a multi-stage training pipeline as (1) In stage 1, the occupancy world model is trained with configurations introduced in DOME 4. This stage establishes a strong foundational occupancy generation ability. (2) In stage 2, we train the UNet-based forecasting module using simple cross-entropy loss. (3) In stage 3, freezee all other modules and exclusively train the parameters of ControlNet. Our multi-stage training strategy not only optimizes training efficiency but also ensures controllable generation, demonstrated in our ablation studies.
4 Experiments
We elaborate our experimental settings in Sec. 4.1 the quantitative and qualitative results of the proposed COME framework in Secs, 4.2 a2dd and extensive ablation studies in Sec. 4.4
4.1 Experimental Setup
Dataset and metrics Most experiments are conducted on the widely used Oce3DD-nuScenes 22 benchmark, which offers 3D occupancy labels for 18 categories based on the large-scale nuScenes
5

---

Table 1: 4D occupancy generation performance under various settings. Each setting varies in terms of input modality and whether the ego trajectory (ego traj.) is predicted (Pred.) by planning module or provided as ground truth (GT). "Avg." " indicates the average performance across 1s, 2s, and 3s horizons. The best performance in each setting is highlighted in bold.
mloU (%) f
IoU
(%)
Method
Input
Ego traj.
1s
2s
38
Avg.
1s
2s
3s
Avg.
OceWorld-D
35
Camera
Pred.
11.55
8.10
6.22
8.62
18.90
16.26
14.43
16.53
OceWorld-T
35
Camera
Pred.
4.68
3.36
2.63
3.56
9.32
8.23
7.47
8.34
OccWorld-S
35
Camera
Pred.
0.28
0.26
0.24
0.26
5,05
5,01
4.95
5.00
OccWorld-F
Camera
Pred.
8.03
6.91
3.54
6.16
23.62
18.13
15.22
18.99
OccI.LaMA
Camera
Pred.
10.34
8.66
6.98
8.66
25.81
23.19
19.97
22.99
OccVAR g
Camera
Pred.
17.17
10.38
7.82
11.79
27.60
25.14
20.33
24.35
DFFT-Occ World
[X]
Camera
Pred.
13.38
10.16
7.96
10.50
19.18
16.85
15.02
17.02
Occ-LLM-S 27
Camera
Pred.
11.28
10.21
9.13
10.21
27.11
24.07
20.19
23.79
RenderWorld-S
Camera
Pred.
2.83
2.55
2.37
2.58
14.61
13.61
12.98
13.73
COME (Ours)
Camera
Pred.
25.57
18.35
13.41
19.11
45.36
37.06
30.46
37.63
DOME 4
Camera
GT
24.12
17.41
13.24
18.25
35.18
27.90
23.44
28.84
COME (Ours)
Camera
GT
26.56
21.73
18.49
22.26
48.08
43.84
40.28
44.07
Cosy&Paste 35
3D-Occ
Pred.
14.91
10.54
8.52
11.33
24.47
19.77
17.31
20.52
OccWorld R5
3D-Occ
Pred.
25.78
15.14
10.51
17.14
34.63
25.07
20.18
26.63
OccLLaMA 26
3D-Oee
Pred.
25.05
19.49
15.26
19.93
34.56
28.53
24.41
29.17
OccVAR S
3D-Oce
Pred.
27.96 21.75
16.47
22.06
38.73
29.50
24.86
31.03
Render World
29
3D-Occ
Pred.
28.69 18.89
14.83
20.80
37.74
28.41
24.08
30.08
Occ-LLM 2
3D-Occ
Pred.
24.02 21.65 17.29
20.99
36.65
32.14
28.77
32.52
DFFT-OccWorld
3
3D-Oce
Pred.
31.68
21.29 15.18
22.71
40.28
31.24
25.29
32.27
COME (Ours)
3D-Oee
Pred.
30.57 19.91
13.38
21.29
36.96
28.26
21.86
29.03
DOME [
3D-Oce
GT
35.11
25.89
20.29
27.10
43.99
35.36
29.74
36.36
COME (Ours)
3D-Oce
GT
42.75
32.97
26.98
34.23
50.57
43.47
38.36
44.13
UniScene-Fore
JDD-Occ21,.Bx.MM
GT
35.37
29.59
25.08
31.76
38.34
32.70
29.09
34.84
COME (Ours)
JD-Oce(2D.Box.Mag
GT
45.98
38.57 33.28 39.28
52.11
46.73 42.65
47.16
dataset. The annotations cover a spatial range of [-40m, -40m, -3.2m, 40m, 40m, 3. 3.2m] with a voxel resolution of [0.4m, 0.4m, 0.4m], resulting in 200 200 x 16 voxel grid per frame. The dataset is split into 700 training. 150 validation, and 150 test driving sequences, each lasting 20 seconds.
We also
Boc3D-Waymol 22) benchmark based the Waymo Open Dataset [1] (WOD), which has 3D occupancy labels for 16 categories. The spatial range and voxel resolution are the same as on nuScenes dataset. The dataset is split into 798 training and 202 validation driving sequences. To align with the task settings on nuScenes, we down-sample to select one frame every 0.5 seconds.
We adopt geometric Intersection over Union (IoU) and semantic mean Intersection over Union (mloU) as evaluation metrics. Results are reported at each future timestamp, along with the average performance over all timestamps on the validation set, in line with previous works.
Implementation details. Unless otherwise specified, our world model predicts future occupancy over a 3-second horizon at 2 frames per second, conditioned on 4 frames of historical occupancy data, following the protocol established in OceWorld (35). As outlined earlier, the COME framework is trained in multiple stages: (1) iiffiusioo-basee World Model. We adopt the pre-trained Occ-VAE from DOME 4 and train the diffusion-bar based world model for 2000 epochs with total batch size of 128 and learning rate of 2e-4. (2) Scene-centrie Forecasting Module. This module is trained for 12 epochs using total batch size of 32 and the CBGS resampling strategy [36]. (3) COME ControlNet. This component is trained for 1000 epochs with total batch size of 64. All models are trained on H20 GPUs and use a learning rate le-4 is not stated specifically. Please refer to supplementary materials for additional implementation details, including network architectures and statistics, training hyperparameters, and planning trajectory configurations.
4.2 Quantitative Evaluation
Main results on Occ3D-nuScenes. the occupancy world modeling domain, existing methods differ significantly in their experimental setups. To enable a fair and comprehensive comparison,

---

Table 2: Comparisons of 4D occupancy forecasting performance between DOME and different stages of COME on Occ3D-Waymo dataset.
mIoU
(%) t
IoU
(%) f
Method
1s
2s
3s
Avg
2s
3s
Avg
DOME
31.62
24.82
19.65
25.36
51.24
43.17
37.47
43.96
COME Stage I: World Model
34.45
26.28
22.50
27.74
51.16
43.24
38.44
44.28
COME Stage 2: Scene-Centric Forecasting
47.37
40.85
36.31
41.51
59.69
52.27
45.60
52.52
COME Stage 3: ControlNet
39.85
34.65
30.58
35.03
55.53
50.09
45.53
50.38
COME+ Stage 3: ControlNet
47.29
40.80
36.44
41.51
60.63
53.98
48.62
54.41
adapt our proposed COME framework to match these various settings and report the main results in Tab. [
The first setting uses camera images as input and predicted ego trajectories from a planning module.
In this case, we employ a modified BEVDet 7 to convert camera inputs into occupancy predictions. Among prior works, VcR previously achieved the best performance with an average mloU of 1 11.79 and an average IoU of 24.35, In contrast, our COME achieve 19.1 11 mloU and 37.63 IoU, significantly surpassing the previous best results by 62. % and 54.5%, respectively.
When ground-trun ego trajectories are used with the same camera input, our method further improves performance to 22.26 mloU and 44.07 IoU, with gains of 3.15 and 6.44, respectively. In this setting, our method also outperforms the state-of-ther 34.6% in IoU, highlighting the robustness of COME.
DOME by a significant margin 22.0% in mloU and
In the third setting, we adopt ground-truth occupancy inputs but use predicted ego trajectories. Here, the best prior mloU (22.71) is achieved by DFFT-OccWorld while the highest IoU (32.52) is obtained by Occ-LLM. It is noteworthy that both methods incorporate additional cues such as occupancy flow or language information into their frameworks. In comparison, relying solely on trajectory conditioning without such external information, still achieves competitive results of 21.29 mloU and 29.03 IoU, We hypothesize that the slightly lower performance is due to COME's strong dependency on trajectory input: since it generates occupancy strictly conditioned on predicted trajectories, it may be more sensitive to planning errors. We further validate the hypothesis in the ablation study. Nevertheless, our method remains among the top-performing approaches, demonstrating its effectiveness even in this challenging setup.
COME, despite
Under the configuration with both ground-truth occupancy and ground-truth trajectories, COME achieves 34.23 mloU and 44.13 IoU, showing strong performance when free from upstream prediction
errors. COME outperforms the state-of-the-ari DOME by 26.3% in mIoU and 21.3% in IoU, Furthermore, with BEV layouts, COME's performance increases further to 39.28 mloU and 47.16 IoU. COME outperforms the state-of-the-arr UniScene-Fore by 23.7% in mIoU and 35.4% in IoU.
Results on Occ3D-Waymo. On Occ3D- Waymo, we retrain OCC-VAE, UNet DOME COME world model, and COME ControlNet on Occ3d- Oce3d-Waymo, initializing from nuScenes-traine components to save cost and time, with unchanged model sizes and training schedules.
We report the default COME model and COME variant (COME+) with the same structure and with stronger geometric controls from the scene-centric forecasting module, differing by additionally replacing noisy BEV latents with UNet-encoded latents in future visible areas during denoising.
The Occ-VAE reconstruction quality on Boc3D-Wayme is mloU = 74.88 and LoU = 82.57.
The results of Oce3D- Occ3D-Waymo are shown in Table 2 our reproduced DOME achieves similar performance as in nuScenes dataset. COME world model has marginally better performance with DOME by 2.38 mIoU and 0.32 IOU. Scene-centric forecasting module outperforms the generative world model by a large margin (63% mloU and 19.5% IoU). COME achieves 38. 1% better mloU and 14.6% better loU compared to DOME COME+, which relies more heavily on condition features, maintains the same 3-s average mloU with Scene-centric Forecasting module and further improves 3-s average loU to 23.7% improvement.
7

---

Table 3: Long-term 4D occupancy generation performance. Ground-truth 3D occupancy and trajectories are used as imputs. The best results are highlighted in bold. "Avg." denotes the average performance over the second to 8 second horizon.
mloU (%)
Method
2s
38
48
5s
6s
7s
88
Avg.
DOME 30.10 21.35
17.36
14.86
12.61
11.03
10.00
9.34
15.83
COME (Ours)
33.78
24.57
21.35
18.25
15.84
13.85
12.99
11.96
19.07
IoU (%) 1
Method
38
6s
Avg.
DOME
39.04
31.20
27.14
24.73
22.32 20.28
19.05
17.97
25.21
COME (Ours)
44.20 36.25
32.86
30.03
26.93 24.70 23.30
21.44
29.96
Table 4: Model performance in various stages. Here, "Visible", "Invisible" and "All" refer to voxels that are observed, unobserved, and the total set of voxels based on historical occupancy, respectively.
mloU (
ToU (%)
Model
Input
Visible Invisible
All
Visible Invisible
All
Stagel: World Model
3D-Occ (4f)
25.68
5.81
23.58
35,96
13.60
32.61
Stage2: Ceene-Centric Forecasting
3D-Occ (4f)
42.74
0.09
39.12
55.08
0.31
48.00
Stage3: ControlNet
3D-Oce (4f)
40.06
5.56
34.23
51.12
14.95
44.13
Stagel: World-Model
(D-Occ(2I,.oo,MM))
32.92
11.74
30.04
39.52
16.20
35,43
Stage2: Scene- Cenne-Centicc Forecasting
3D-Occ(21)
41.65
0.08
37.93
54.50
0.27
47.18
Stage3: ControlNct
3D-Occ(2I)Box..o
42.75
14.85
39.28
52.78
20.48
47.16
Long-term Occupancy Generation The ability to generate long-term predictions crucial for world models, particularly in the context of autonomous driving. To evaluate this capability, we extend the prediction horizon from 3 seconds consistently outperforms baselines across all timestamps and on average, for both mloU and IoU metrics. Specifically, COME achieves average mloU and IoU scores of 19.07 and 29,96, surpassing DOME by 20.5% and 18.8% respectively.
seconds and present the results in Tab. Our method, COME,
We further discuss in the supplementary material that the best practice of masking strategy for balancing quantitative and qualitative results to pose the invisibility mask on control features.
4.3 Qualitative Results
Fig. 3 presents visualizations comparing ground-truth occupancy with predictions from DOME and our method. We observe that DOME suffers from object category inconsistency (top example) and sudden object disappearance (bottom example). In contrast, COME produces results with improved spatial consistency, highlighting the efficacy the scene condition injection. Additional qualitative results, including analyses of input variants, model architectures, and component contributions, provided in the supplementary material for comprehensive evaluation.
4.4 Ablation Study
Model performances across various stages. In Tab. 4 we analyze the predicted occupancy across our three stages, evaluating visible, invisible, and all voxels. Consistently, the world model performs better in previously invisible areas, while scene-centric forecasting excels in observed regions. This validates our motivation: the generative world model exhibits strong imaginative capabilities but under-utilizes the 3D spatial consistency of the driving scene. In contrast, scene-centrie forecasting achieves high accuracy in observed areas but lacks generative flexibility, limiting its imagination applicability novel-view synthesis under diverse trajectories.
In the third stage, we leverage the ControlNet to combine the strengths the two modules. Stage three model significantly improves performance in invisible regions compared to the forecasting module. Although the overall mloU experiences slight decline, the model gains imaginative capabilities and broader adaptability. With extra BEV layout inputs, COME demonstrates further

---

Condition Frame
Future 1-s
Future 2-s
Future 2.5-s
Future 3-5
Ground- Truth
DOME
COME (Ours)
Ground
DOME
COME (Ours)
Figure Qualitative results of 3-s 4D occupancy generation.
enhanced generation of invisible scenes and achieves the best overall performance. These results showcase effectiveness of our proposed framework.
Effects of training setups. Tab. presents an ablation study on different training configurations within the proposed COME framework. We first train ControlNet with and without freezing the world model in the final stage, and present the results in Tab. Sa It can be observed that training ControlNet while keeping the world model frozen yields significantly better results, confirming the importance of aligning with the tne-ttunig strategy validated in the original ControlNet 34 paper.
In Tab, 5b explore two model sizes for both the generative model and its corresponding Control- Net. Across both settings, the introduction of ControlNet consistently improves performance by substantial margin. These gains are particularly pronounced under limited computational budgets, where mloU increases from 7.78 to 32.00 and loU from 18. 14 to 42.03 surpassing even the standalone world model with 1375.4 GFLOPS. These results demonstrate that integrating ControlNet significantly enhances the generative capability of the model even if the generation model is small.
Effects of inference setups. In Tab. 6 we investigate how different inference configurations affect model performance. We begin by analyzing the impact of the number of denoising steps during the diffusion process, as shown in Tab. 6a] Increasing the number of steps consistently improves generative performance. The results indicate a clear positive correlation between performance and the number of denoising steps, with satisfactory results achieved when performing least 10 steps.
Next, we examine model performance using different sources of predicted occupancy and trajectories. In Tab. 6b replace ground-truth -tunt occupancy with predictions from a vision-only model, BEVSterco

---

Table Ablations on different training setups. "WM" and "ControlNet" denote the world model and ControlNet in our COME, respectively.
(a) Effects trainable parameters in the final stage.
(b) Model performances under different model sizes.
Small IoU
Base
Model
Trainable modules
mloU
loU
mloU
GFLOPS
mloU
IoU
GFLOPS
ControlNet
34.23 14.13
WM
7.78
18.14
147.8
23.49
32.36
1375,4
WM&ConttrolNet
28.67
41.07
+ControlNet
32.00
42.03
222.7
34.23
44.13
2066.6
Table 6; Ablations on various inference configurations.
(a) Effects of the de- noising steps at the in- ference stage.
(b) Model performance when ground- truth occupancy is replaced with dif- ferent occupancy generators. Here, "C" and "L" denote inputs from cam- era and LiDAR sensors, respectively.
(c) Effects the used trajectories. "align." is the alignment operation of the ground-truth occupancies.
#step
mloU
lnU
Pose2D
Yaw
Align.
mIoU
loU
4.41
14.47
GT
GT
34.23
44.13
Model
Input
mloU
IoU
4,95
16.17
Pred.
25.90
35.21
33.42
44.35
BEVStereo [2
22.26
44.07
Pred. Pred.
Pred.
21.29
29.03
20
34.23
44.13
EFFOcc 20
LC
26.75
50.49
Pred.
34.00 43.69
(mloU 42.54), and a fusion model, EFF-Oce (mloU 54.08). The results show that stronger occupancy inputs lead to better generative outcomes. Tab. shows that gradually replacing the ground-truth pose and yaw with predicted values leads noticeable drop in model performance. However, when we remove the influence of ego pose during evaluation by aligning predicted and ground-truth occupancy to a same predicted future waypoint coordinate- the degradation is minimal, with only 0.23 drop in mloU and a 0.44 drop in IoU, This phenomenon suggests that the performance degradation primarily stems from misaligned trajectories, while the generated scene quality remains relatively unaffected by trajectory errors. Further discussion is provided in our supplementary material.
Conclusion
We introduce COME, framework that enhances generative occupancy world models through scene- centric forecasting control. By explicitly decoupling ego-motion effects from scene evolution, COME first generates spatially consistent, ego-invariant control features, which are then integrated into the occupancy world model for more accurate and controllable future predictions. Extensive experiments the large-scale Oce3ID-nuScenes dataset demonstrate the state-of-the-art performances across multiple settings, validating the effectiveness our approach for occupancy world model.
To motivate future work, we outline few limitations based on our current comprehension: (1) The introduction of control modules to the base generative model increases computational complexity. Although our approach achieves superior performance than the baseline with lower overhead, further optimization to reduce computations remains valuable for real-time applications. (2) Our current multi-stage training pipeline could be streamlined. An end-to-end training scheme may improve efficiency while maintaining or enhancing model performance.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China(52394264, 22372414.52472444 Beijing Natural Science (oundation(23L1000) 100. 1000) and Beijing Munici- pal Science and Technology Commission (Z241 122110000052401, Z241 100000524000) This work was also sponsored by Tsinghua University-DiD Joint Research Center for Future Mobility and Tsinghua University-Toyot Joint Research Center.
10

---

References
[1] Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupaney fields for pereeption and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14487-14496, 2024.
[2] Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14487-14496, 2024.
[3] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 11621-1166 2020.
[4] Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Taming diffusion model into high-fidelity controllable occupancy world model. arXiv preprint arXiv:2410. 10429, 2024.
[5] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY.
USA, 2020. Curran Associates Inc.
[6] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:23009 17080, 2023.
[7] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790 2021.
[8] Bu Jin, Xiaotao Hu, Yupeng Zheng. Xiaoyang Guo, Qian Zhang, Yao Yao, Diming Zhang, Xiaoxiao Long, Wei Yin, et al. Ocevar: Scalable 4d occupancy prediction via next-scale prediction.
[9] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 1124,14, 2023.
[10] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1124-1 2023.
[11] Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding. Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1 11971-11981, 2025.
[12] Yinhao Li, Han Bao, Zheng Ge. Jinrong Yang. Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1486-1494, 2023.
[13] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint 2xXiv:200500874 2024.
[14] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14864 14873. 2024.
[15] Xinhao Liu, Moonjun Gong, Qi Fang, Haoyu Xie, Yiming Li, Hang Zhao, and Chen Feng. Lidar-based 4d occupancy completion and forecasting. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1100-11099 IEEE, 2024.
[16] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEF/CVP International Conference on Computer Vision (ICCV). pages 9992-10002, 2021.
[17] Junyi Ma. Xieyuanli Chen, Jiawei Huang, Jingyi Xu, Zhen Luo, Jintao Xu, Weihao Gu, Rui Ai, and Hesheng Wang. Cam4doce: Benchmark for camera-only 4d occupancy forecasting in autonomous driving applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21486 21495, 2024.

---

[18] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asissted imlervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. pages 234-241. Springer, 2015.
[19] Xingjian Shi, Zhourong Chen, Hao Wang. Dit- Yeung. Wai- Wai-Kin Wong, and Wang-chun Woo. Con- volutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information proessing systems, 28, 2015.
[20] Yining Shi, Kun Jiang, Ke Wang, Kangan Qian, Yunlong Wang, Jiusi Li, Tuopu Wen, Mengmeng Yang. Yiliang Xu, and Diange Yang. Effoce: A minimal baseline for efficient fusion-based 3d occupancy network. arXiv preprint arXiv:2406.0706 2024.
[21] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurélien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens. Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2443 1441-2451,222 2020.
[22] Xiaoyu Tian, Tao Jiang. Longfei Yun, Yucheng Mao, Huitong Yang. Yue Wang. Yilun Wang. and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36:64318 64330, 2023.
[23] Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6309-63 6318, Red Hook, NY, USA, 2017. Curran Associates Inc.
[24] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards word-rere world models for autonomous driving. In European Conference on Computer Vision, pages 55-72. Springer, 2024.
[25] Yugi Wang, Jiawei He. Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14749-14759, 2024.
[26] Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Oc- cllama: An wangpaanyy-lnguage--ee arXiv:2409.03272. 2024.
generative world model for autonomous driving. arXiv preprint
[27] Tianshuo Xu, Hao Lu, Xu Yan, Yingjie Cai, Bingbing Liu, and Yingcong Chen. Oce-Ilm: Enhancing autonomous driving with occupancy-basee large language models. arXiv preprint arXiv:2502.066. 064499 2025.
[28] Tianyi Yan. Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng-zhong Xu, and Jianbing Shen. Drivingsphere: Building high-fidelity world for closed-loop simulation. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27531-2755 127541,, 2025.
[29] Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Haiyang Liu, Jingwen Liu, Haozhe Wang. Zhe Wang, Yan Wang, Fabio Remondino, et al. Renderworid: World model with self- supervieed 3d label. In 2025 IEEF International Conference Robotics and Automation (ICRA), pages 6063-6070. IEBE, 2025.
[30] Yu Yang, Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving the occupancy world: Vision-centrie 4d occupancy forecasting and planning via world models for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9327-9335. 2025.
[31] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14673 14684, 2024.
[32] Zetong Yang. Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673-14684 2024.
[33] Haiming Zhang, Ying Xue, Xu Yan, Jiacheng Zhang, Weichao Qiu, Dongfeng Bai, Bingbing Liu, Shuguang Cui. and Zhen Li. An efficient occupancy world model via decoupled dynamic flow and image assisted training. arXiv preprint arXiv:2412.13772. 2024.
12

---

[34] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pages 3836-3847, 2023.
[35] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang. Borui Zhang. Yueqi Duan, and Jiwen Lu. Oceworld: Learning a 3d occupancy world model autonomous driving. European conference on computer vision, pages 72-72. Springer, 2024.
[36] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming and Gang Yu. Class-bard balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv: 09492, 2019.
[37] Zyrianov. Henry Che, Zhijian Liu, and Shenlong Wang, Lidardm: Generative lidar simulation generated world. In 2025 IEEE International Conference on Roboticy and Automation (ICRA), pages 6055-6062. IEEE, 2025.
13

---

A Technical Appendices and Supplementary Material
A.1 Broader Impacts
Our work proposes generative occupancy world model with scene-centric forecasting as control, achieving state-of-lhe-art occupancy generation performance under multiple settings. This advancement leads to safer understanding, forecasting, generation and simulation of the driving scenes, which are crucial for real-world applications of autonomous driving (AD). Consequently, the most significant positive impact of our work is the enhancement of safety in AD systems.
However, the biggest negative socictal impact of this work, with any component of AD systems, is the safety concem. Autonomous driving systems are directly related human lives, and erroneous hallucinating predictions or generation can lead to hazardous understanding of the driving environment. Therefore, increasing the accuracy of occupancy generation to address false generation and illusions will require substantial follow-up efforts.
A.2 Licenses for involved assets
The project code is built on top of the codehas<[provided by DOME [. which is subject to the Apache License Version 2.0. Our experiments are conducted on the OccAD-nuScenes [22] which provides occupancy labels for the nuScenes dataset B and Dee3-o [2]] which provides occupancy labels for the Waymo open dataset [21]. Oce3D is licensed under MIT license, nuScenes is licensed under the CC BY-NC-SA 4.0 license and Waymo open dataset is licensed under the Apache License Version 2.0.
A.3 Discussion on the Design Philosophy of External Occupancy ControlNet
this paper, we use a direct concatenation method for pose and BEV layout control, while we employ an external ControlNet for controlling scene-centric forecasting. We propose some explanations of the design philosophy.
Most importantly, We hope to remain the original generation capabilities of the occupancy world model unchanged. ControlNet only provides additional guidance to enhance the original world model, while direct concatenation makes the scene-centric forecasting downgrade the capability of the world model a completion or inpainting model that inpaints future invisible voxels with the context of visible voxels. On the other hand, the model easily learns shortcut that outputs forecasting results directly on visible voxels. This shortcut learning may greatly hurt the multi-modal nature of both the generative model and the forecasting task. In contrast, ControlNet may be added to certain regions of the space or certain frames on the sequences, with user-defined masks and enjoy greater flexibility.
indispensable component for generation. This will
Morcover, poses (in the form waypoints [a: y, yaw)) and bev layouts (in the form of semantic maps) are low-dimension conditions that empirically fit direct concatenation while occupancy sequences latents are high-dimension conditions that empirically fit external ControlNet.
A.4 Discussion on Evaluation Metrics under End-to-end Planning Setting
Early autoregressive world models 35 generates both future waypoints and future occupancy sequences at the predicted waypoints' coordinate at the same time. The evaluation between generated occupancies and ground-truit occupancies couples the similarity between the planning trajectory and the expert trajectory, and the similarity of generation. Our ablation experiment illustrates that both the translation error of the trajectory and the yaw angle error of the trajectory can greatly reduce final IoU metric.
order to factor out the influence of trajectory quality from the generation evaluation. we propose to reset the origin of the ground-truth occupancy according to the rotation matrix the planning trajectory, and then evaluate the difference between the generated occupation and the ground-truth occupation. In this setting, we demonstrate that the generation performs similarly well when conditioned by the planning trajectory.
A.5
More Implementation Details
A.5.1
Environment Setup
The proposed algorithm runs in the python3.9 and torch2.5.1 environment and is expected to be compatible with the torch2.x environment. The environment needs to have mmev 2.x and minde3dd 1.1.x installed, and it is basically the same the environment configuration scheme of WccWorld 35 and DOME[] We use AdamW optimizer for all experiments.
https://github.
com htw.hauww..oooguuggggoge con/gusongen//O0E
14

---

Condition Frame
Future 1-s
Future 2-s
Future 3-s
Ground-Truth Turn Left
COME Turn Left
Ground-Truth Turn Right
COME Turn Right
Ground-Truth Go Straight
COME Go Straight
Figure 4: Visualization examples demonstrate the pose control alignment ability of COME generation. For different driving commands such as Go Straight, Turn Left and Turn Right, COME well follows the pose control and generate similar scenarios compared to ground-truth.
15

---

Futurn -
Future ou.s
Future 0-
Figure 5: Visualization examples of occupancy generation with BEV layouts. COME generates
occupancy sequences that well follows the BEV layout control.
Future
Future
Future
Future
Future 5-s
Future
Ground-Truse
Ground- Truth
COME World Model
GOME-Worid Model
COME Scene-Centric Forecasting
COME Scene-Centric Forecasting
COME ControlNet
COME ControlNet
Figure 6: Visualization examples demonstrate the spatiotemporal consistency of differnet stages in COME for generating static environments. During the 8-second long generation process, COME world model mistakenly links drivable areas as intersections with the main street, but under the guidance of scene-centric forecasting, COME maintains the background consistency of road structures during generation.
16

---

Condition Frame
Future 1-s
Future 2-s
Future 3-8
Bround-Truth
COME-3DOco-Inpe
COME- BEME--EEStereo-IIput
COME-EFFOcc-Inpue
Figure 7: Visualization examples that takes 3D-Oce, vision-based BEVDet and fusion-based EFFOcc as occupancy sequences input.
Condition Frame
Future 1-s
Future 2-s
Future 3-s
Ground-Truthi Truth
COME-All-Control
COME-Mask Invisible
Figure 8: Visualization examples that uses different masking strategies during training and inference. Models with invisibility masks generally achieve much better qualitative results results.
17

---

Condition Frame
Future 1-s
Future 2-s
Future 3-S
Ground-Trun -ruth
COME-Smi Ammal World Model
COME-Sma ControlNet
Figure 9: Visualization examples that uses small models. In this right turning example, the small model shows accurate pose control and good scene completion results.
Condition Frame
Future 3-8
Future 6-s
Future 12-s
Future 16-s
Future 20-s
Figure 10: Visualization examples that uses repeated roll-out to generate super-long scenarios (20 second) with free trajectories.
18

---

Normal BEV Layout
Adverse BEV Layout
t=1s
t=2s
t=3s
Scene-0024
Figure 11: Visualization examples that uses adverse BEV layouts as inputs. In adverse BEV condition input (shown in right), all objects are re-placed to the centers which are half of the distance of the original centers of normal BEV layouts (shown in left).
A.5.2 Networks Setup
Continuous VAE use the same VAE with DOME which consists a encoder and a decoder. The class embedder has the expansion of 8. The VAE has the compression ratio of 64. The encoder down- sampling and decoder upsampling ratios are set as [1, 2, 4, 8j. The base channel is set as 64. The num- ber of residual blocks is 2. The attention resolution is 50. The shapes of intermediate feature maps [[200, 200]. [100, 100,, [50, 50), [25, 25]].
Scene-centric forecasting The class embedder has the expansion of 32. We use UNet with five-stage encoder and decoder. The encoder downsampling and decoder upsampling ratios are set as [1, 2, 4, 8, 16) and the channel dimensions are 256, 512, 1024, 1024, 1024 for each downsampling stride. The hase channel is set as 256. The number of temporal convolutional layers is set as 2. We use InterpConv for upsampling operations.
COME world model and ControlNet We have two model sizes. By default, we use the model size similar to DIT-XL. The number of attention head is as 12, the hidden size is set 768. The number of layers (also named depth of world model) is set as 28. We also set smaller model. The number attention head is set as the hidden size is set as 384. The number layers is set as 12. The mip_ratio is set as 4, The patch size set as 1. The topk is set as 10. For different size of world models, corresponding controlnct has half of the depth of the world model and uses the same parameters in each block.
Network statistics. The following statistics is tested with the standard task of generation future three second occupancy sequences with four-frame occupancy history. The scene-centric forecasting module has 27.31 M parameters and 737.76 GFLOPs. The base world model has 362.3 parameters and 1375.38 GLOPS. The base ControlNet has 158, 49M parameters and 691.23 GLOPS. The base world model has 45.83M parameters and 147,85 85 GLOPS. The base ControlNet has 17.27M parameters and 74.86 GLOPS.
A.5.3 Running Parameters Setup
Diffusion parameters. We use DDPM(5] as the default diffuser. By default, we use 1000 denoising steps for training and 20 denoising steps for inference. We also find in the ablation that 10 denoising steps for inference only very slightly drops the final performance. The guidance scale is set as 7.5. If the historical frame number is 4, the conditional frames in training may be []. [0], [0, 1]. [0, 1, 2, [0, 1, 2, and the conditional frames in inference is [0, 1, 2,3,3 3). The possibility of using pose as condition in training set as 0.9.
A.5.4 Optional Inputs
Sensor Inputs. For vision track, we use the officially released checkpoint of BEVDe BEeStereo, with Swin Transformer [6] base and image size 512 x 1408 as input and I historical frames, achieves 3D occupancy prediction mlou of 42.45. For LiDAR-camera fusion track, we use officially released checkpoint of EFFOcc[20). EFFOcc, with Swin Transformer [66 base and image size 512 x 1408 as input, achieves 3D occupancy prediction mlou of 54.08. As 3D occupancy models are trained with camera mask on nuScenes dataset, also camera mask during occupancy generation evalaution.
19

---

Table 7: Results on COME with different control masking strategies.
mloU (%) t
IoU (%)
Masking Strategy
1s
2s
3s
Avg.
1s
2s
3s
Avg.
No Mask
38.33
35,05
29.90
36.19
51.67
45.32
40.90
45,97
Mask Condition
43.25
34.05
28.82
35.37
50.86
43.78
39.05
44.56
Random Dropout
40.32
30.55
36.69
31.97
48.78
41.78
25.04
42.42
Mask Control
42.75
32.97
26.98
34.23
50.57
43.47
38.36
44.13
Planning trajectory Input COME does strongly bind the world model to planning, instead, COME controlled by the ego future trajectory as input. compare with other world models with planning, we train a unsupervised simple imitation learning planning framework built upon BEV-Planner 14. The planning module takes multi-view images as sensor imput and outputs six waypoints including 2d translation and relative yaw angle compared the current frame. The only supervision is the expert trajectory with ground-truth yaw angles. The planning module has an 3-8 average error 0.48m under BEV-Planner open-loop metrie.
The vast majority planning algorithms are centered around the current coordinate of the ego vehiele, without including past or future perspectives, so more complicated planning modules can be placed as downstream or a parallel heads which integrates with the ceene-centric forecasting module. In this way, planning trajectories can be integrated as control conditions into the generative model and theControlNet. We leave better planning modules considering scene-centric forecasting as future research.
BEV Layout Input The pre-processing for BEV layouts the same UniScene [T] 3D bounding boxes of annotated objects are splatted the BEV plane. The static layouts are computed with polygons the high-definition maps.
A.6
More Quantitative Results
A.6.1 Generation with Different Masking Strategies
Tab. 7 demonstrates how masking strategies affect the final quantitative results. We find that model without any masking performs the best mloU and IoU metrics, but it has strong tendency to generate invisible areas free. On the other hand, the model with masked control has much better visualization performance, but the quantitative results are lower than model without masking. In the main paper, we use model with masked control by default.
We find the possible reason for the formation of a trend that opposes quantitative results and qualitative results, is that the deductions for incorrect generation of invisible areas (generated content that does not match the true value) are higher for non-generation of invisible areas (where all areas that need to be generated are left blank) under the existing metries.
We also try with different masking strategies. We try perform masks conditions before COME ControlNet rather than masks on controls after COME ControlNet, but the qualitative results are poor, so we not pose the masks in condition features. We test with random dropout of masking in training stage, this operation helps better visualization quality but decreases the quantitative results. Finally, we find the best practice to balance quantitative results and qualitative results, that is to fixed invisibility mask control features after COME ControlNet both training and inference stages. As a result, we report this model and variants the main paper.
A.7 More Visualizations
We show more visualization results including the generation results with different driving commands and
future trajectoriesFig 4 with BEV layouts as input (Fig. 5). comparison between results across stages (Fig. [).
comparison between different sensor inputs (Fig. 7. comparison between different masking strategies (Fig.
visualization with small models (Fig. 19. super-long occupancy video generation (Fig. T0p. generation with costumed adverse BEV layouts (Fig.
Fig. demonstrates that COME can well align occupancy generation results with different driving commands (tum Left, turn right, go straight) and pose control. The accurate pose control credits to the explicit transformation modelling guidance from ccene-centric forecasting.
Fig. 5 demonstrates that COME can well align occupancy generation results with BEV layouts as condition.
Fig. demonstrates an example of how scene-centric forecasting helps ego-centric generation. With COME world model ego-centric generation alone, after long generation time, path appears at the roadside where
20

---

there is originally separating strip, forming intersection, which is inconsistent with previous observations. The scene-centric forecasting easily learns the static nature of the scenario. With scene-centric forecasting as control, COME ControlNet generates correct road structures avoid inconsistency caused by COME world model.
Fig. demonstrates that the generation from sensor inputs are similarly well for the same seene. When the 3D occupancy quality is better, the generation results are also better.
Fig. 8 compares the generation results with different masking strategies. If we add full-space control regions, the model has an increasing lendency of generation invisible areas free. If we mask control features invisible (COME-Mask Invisible), the generation results more satisfactory on most cases. We find that the performance the visualization not always consistent with the quality the fact that the metric was too restrictive. Specifically, when we mask the control of unseen areas, it very reasonable to have free generation of unseen areas, but the reported metrics decrease because it is very likely exert control over the whole world, there higher
metric, which may be due to
that free generation not the tendency set the unseen area free, which in turn improves the quantitative results.
the truth value.
Fig. shows an example that the smaller model achieves cqually good generation results with ControlNct. The ego vehicle turning at a large angle. The small world model has limited generation quality. However, with ControlNet, the small COME demonstrates better pose control and satisfactory generation results.
Fig. 10 shows some examples that the model roll-outs several times to generate super-long vidcos. This roll-out mechanism is similar to DOME except that we use COME ControlNet for the first roll and use COME world model without ControlNet for the next rolls. This shows the fleixibility of framework with and without ControlNet guidance the same time.
Fig. MD shows the simulation ability COME world model to adverse BEV layouts as inputs. this case, we try to reduce the center of each object box to half distance and create an adverse busy intersection. The 3D occupancy sequences can accurately respond to the BEV layout condition. A potential issue is conflicts between the Scene-Centrie Forecasting module and BEV-layout control conditions (e.g., other vehicles' hehavior conflicting with historical inertia). We find elegant fix, that is mark adversarial iimulation-focusse agent areas, mask the corresponding Scene-Centric Forecasting results, ensuring only BEV-layout covers these regions to maintain global physical consistency and local customization.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make claims Abstract and Introduction (Sec. T. which match results Experiments (Sec. 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations the current approach in Conclusion (Sec.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does include theoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed information needed to reproduce our results in the main manuscript (Sec.4.. [4.1) and the supplementary material (Appendix A.5). A.5)) We upload codes parts of supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
Open access to data and code
the data and code, with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Question: Does the paper provide open
Answer: [Yes]
Justification: We submit our codes as supplemental material and provide instructions for reproducing our results. The data is public available.
Guidelincs:
The answer means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines ftttps://nips.cc/puue guides/Codedee https///ooooouuoiiiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
23

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttss://niss.cc/ppin
Buzdee//DodeSuummissioooPPo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: We describe experimental setting and details the main paper (Sec. and supplementary material (Appendix A.S)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- lion about the statistical significance of the experiments?
Answer: [No]
Justification: Due the limited compute resources high computation expenses of generative
models, we cannot afford to run experiments several times to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric етror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Iustification:
We describe the computer resources Sec.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Fthies htttp:://eeurig::i http:a/ppp.......bee////eeeeeuue
Answer:
[Yes]
Justification:
Our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer means that authors have not reviewed the NeurlPS Code of Fihics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential positive socictal impacts in the section of "Broader impacts" (Appendix(A..) A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimatc point out that improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it not needed point out a generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected"?
Answer: [Yes]
Justification:
We credit the existing assets in Appendix A.2
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed instruction of codes supplementary materials.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work dose not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our work dose involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- blc), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it important, original, or non non-standard component
core methods this research? Note that the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer I.L policy should or should not described.
for what
27


Reviews:
[{"review_id": "EPPy8k6Fak", "reviewer": "Reviewer_2AyB", "summary": "COME introduces a three-stage occupancy world‑model forecasting framework for autonomous driving: (1) it rigidly transforms ego‑centric historical occupancy grids into a fixed, scene‑centric coordinate frame and uses a lightweight UNet to predict short‑term future occupancy; (2) it inversely reprojects these predictions and encodes them into latent “scene‑condition” features; and (3) it injects those features via a ControlNet into a diffusion Transformer—filtered by a visibility mask—to produce long‑horizon occupancy forecasts with enhanced spatiotemporal consistency and controllability. On the nuScenes‑Occ3D benchmark, COME outperforms the state‑of‑the‑art DOME", "questions": "1)\tInference Latency & Throughput: What are the end‑to‑end inference times and achievable frame rates for 10, 20, and 40 denoising steps\n2)\tVisibility Mask Sensitivity: How sensitive are final forecasts to the visibility‑mask threshold \\varepsilon ? Please provide an ablation study over a range of \\varepsilon values.\n3)\tHow is the performance of DOME in Tables 2, 3b and 4c? Is the conclusion consistent with COME ablation study?", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1)\tSignificant performance gains: COME achieves 20–40% improvements in mIoU and IoU over DOME across multiple input modalities and forecast horizons on nuScenes‑Occ3D.\n2)\tClear decoupling of ego‑motion and scene dynamics: The scene‑centric branch removes vehicle‑induced transforms, allowing the model to focus solely on environmental evolution.\n3)\tEnhanced controllability and robustness: Visibility‑mask filtering of injected scene features mitigates noise from unobserved regions, stabilizing long‑horizon forecasts.\nWeakness\n1)\tIncreased computational cost: Compared to Baseline, COME’s combined UNet + DiT + ControlNet pipeline raises GFLOPS from ~1,375 to ~2,066, challenging real‑time deployment. Provide a comparison with the GFLOPS of DOME.\n2)\tNarrow evaluation scope: Experiments are confined to nuScenes‑Occ3D.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recoginition of clarity and experimental results. We sincerely thank the revieweer for the insightful comments. We supplement more comparisons between proposed COME and the baseline DOME\\[1], including computational cost and latencies, ablations on hyper-parameter, area partition, ego trajectories. Moreover, Occ3d-Waymo\\[2] is not used by prior occupancy world models as a 4D occupancy benchmark. We supplement the comparison of DOME and COME on Occ3d-Waymo. Below are our responses to other specific concerns:\n\n* **\\[W1]**:Increased computational cost: Compared to Baseline, COME’s combined UNet + DiT + ControlNet pipeline raises GFLOPS from \\~1,375 to \\~2,066, challenging real‑time deployment. Provide a comparison with the GFLOPS of DOME.\n\n**\\[A1]**: In comparison to the baseline DOME, our COME approach incorporates an additional lightweight UNet and a relatively heavy ControlNet, naturally leading to an increase in computational cost under the same world model settings. To highlight the effectiveness of our method, we introduce COME-small, a streamlined variant with only 222.7 GFLOPs—substantially less than both DOME and the full COME model. Specifically, we reduce the number of attention heads and hidden dimensions by half and decrease the transformer blocks from 28 to 12. As shown in Table. 1, COME-small outperforms DOME even with 80% fewer computations, demonstrating efficiency without compromising performance. We sincerely thank the reviwer for the insightful comments and will include the results in our new draft.\n\n| **Model**                    | **Parameters (M)** | **GFLOPs** | **3s-AVG mIoU** | **3s-AVG IoU** |\n| ---------------------------- | -------------- | ---------- | --------------- | -------------- |\n| DOME                         | 345.74         | 1271.62    | 27.1            | 36.36          |\n| COME                         | 566.83         | 2066.6     | 34.5            | 44.26          |\n| COME-Small ($\\varepsilon=0.5$) | 74.61          | 222.7      | 24.00           | 32.12          |\n| COME-Small($\\varepsilon=1.0$)  | 74.61          | 222.7      | 31.09           | 42.23          |\n\nTable 1: Parameters and computational complexity of DOME and COME variants.\n\n* **\\[W2]**:Narrow evaluation scope: Experiments are confined to nuScenes‑Occ3D.\n\n**\\[A2]**: In our initial draft, COME was evaluated solely on occ3d-nuscenes following the settings of the baseline approaches. However, we acknowledge the feedback from reviewers (2AyB and Gif3) that evaluating on a new benchmark would enhance the study. Consequently, we have extended our evaluation to include occ3d-waymo. Please refer to our response to the reviewer Gif3 for our detailed settings and experimental results. In summary, COME consistently outperforms the baseline in this new benchmark.\n\n\n\n* **\\[Q1]**:Inference Latency & Throughput: What are the end‑to‑end inference times and achievable frame rates for 10, 20, and 40 denoising steps.\n\n**\\[A3]**: Thanks for the comment. We report the results on Table. 2. In a nutshell, the overall time is proportional to the number of denoising steps, and lower latency can be achieved with smaller models.&#x20;\n\n| Model      | Denoising Steps | End-to-end Latency(ms) | Per-step world model latency (ms) | Per-step controlnet latency (ms) |\n| ---------- | --------------- | ---------------------- | --------------------------------- | -------------------------------- |\n| DOME       | 20              | 1104                   | 36                                | 0                                |\n| COME       | 10              | 860                    | 38                                | 15                               |\n|  COME          | 20              | 1635                   | 38                                | 15                               |\n|   COME         | 40              | 3216                   | 38                                | 15                               |\n| COME-small | 10              | 354                    | 17                                | 7                                |\n|     COME-small       | 20              | 599                    | 17                                | 7                                |\n|      COME-smal      | 40              | 1132                   | 17                                | 7                                |\n\nTable 2: Latencies of models with different denoising steps.\n\n* **\\[Q2]**: Visibility Mask Sensitivity: How sensitive are final forecasts to the visibility‑mask threshold \\varepsilon ? Please provide an ablation study over a range of \\varepsilon values.\n\n**\\[A4]**: Table. 3 below presents the ablation study on the ($\\varepsilon$) values. When ($\\varepsilon = 0$), meaning only visible BEV patches are used for control, the performances are not satisfying. Conversely, when ($\\varepsilon = 1$), utilizing all BEV patches, the model achieves peak performances. However, as illustrated in Fig. 8 of our draft, this setting leads to a diminished ability to imagine invisible regions due to the injection of control signals. Therefore, we employ ($\\varepsilon = 0.5$) in our paper for a balance between model performance and imaginative ability.\n\n| $\\varepsilon$ | 3s-AVG mIoU | 3s-AVG IoU |\n| ----------- | ----------- | ---------- |\n| 0.00        | 29.09       | 37.24      |\n| 0.25        | 34.21       | 43.91      |\n| 0.5         | 34.50       | 44.26      |\n| 0.75        | 34.71       | 44.46      |\n| 1.00        | 36.91       | 45.53      |\n\nTable 3: Ablations of hyper-parameter $\\varepsilon$ of COME.\n\n\n\n* **\\[Q3]**: How is the performance of DOME in Tables 2, 3b and 4c? Is the conclusion consistent with COME ablation study?\n\n**\\[A5]**: Yes, the conclusion aligns with our ablation study. We provide additional results in the table below. In the first two tables (Table. 4 and Table. 5), our implemented base world model currently performs worse than DOME despite having similar structures, as we only introduced extra skip connections. However, the COME counteract this issue, resulting in significantly better performance compared to DOME. We anticipate that if our base world model could replicate DOME's performance, our COME would further amplify its advantages.\n\nTable. 6 illustrates the performance across different trajectories. Both COME and DOME exhibit similar trends when settings are modified. When ground-truth occupancies align with predicted trajectories, DOME experiences a more significant decline (-2.77 mIoU and -2.73 IoU) in performance compared to COME (-0.23 mIoU and -0.44 IoU), demonstrating COME's superior geometric consistency under perspective shifts.\n\n| Model                                  | Input       | mIoU    |           |        | IoU     |           |       |\n| -------------------------------------- | ----------- | ------- | --------- | ------ | ------- | --------- | ----- |\n|                                        |             | Visible | Invisible | All    | Visible | Invisible | All   |\n| DOME                                   | 3D-Occ (4f) | 29.62   | 6.45      | 27.10  | 40.65   | 16.08     | 36.36 |\n| COME Stage1: World Model               | 3D-Occ (4f) | 25.68   | 5.81      | 23.58  | 35.96   | 13.60     | 32.61 |\n| COME Stage2: Scene-Centric Forecasting | 3D-Occ (4f) | 42.74   | 0.09      | 39.12  | 55.08   | 0.31      | 48.00 |\n| COME Stage3: ControlNet                | 3D-Occ (4f) | 40.06   | 5.56      | 34.23  | 51.12   | 14.95     | 44.13 |\n\n&#x20;Table 4: Ablation studies of model performance in various stages (Tab. 2 in manuscript).&#x20;\n\n| Model            | mIoU  | IoU   | GFLOPS  |\n| ---------------- | ----- | ----- | ------- |\n| DOME             | 27.10 | 36.36 | 1271.62 |\n| COME World Model | 23.49 | 32.36 | 1375.4  |\n| COME ControlNet  | 34.23 | 44.13 | 2066.6  |\n\nTable 5: Model performances under different computational complexity. (Tab. 3(b) in manuscript).&#x20;\n\n| Model | Pose2D | Yaw   | Align | mIoU  | IoU   |\n| ----- | ------ | ----- | ----- | ----- | ----- |\n| DOME  | GT     | GT    | No    | 27.10 | 36.36 |\n| COME  | GT     | GT    | No    | 34.23 | 44.13 |\n| DOME  | Pred.  | GT    | No    | 17.42 | 26.25 |\n| COME  | Pred.  | GT    | No    | 25.90 | 35.21 |\n| DOME  | Pred.  | Pred. | No    | 20.62 | 29.58 |\n| COME  | Pred.  | Pred. | No    | 21.29 | 29.03 |\n| DOME  | Pred.  | Pred. | Yes   | 24.33 | 33.63 |\n| COME  | Pred.  | Pred. | Yes   | 34.00 | 43.69 |\n\nTable 6: Effects of the used trajectories.. (Tab. 4(c) in manuscript).&#x20;\n\n\n\n\\[1] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.\n\n\\[2] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, NeurIPS 2023."}, {"review_id": "HKjypWIBKc", "reviewer": "Reviewer_exSF", "summary": "Autonomous driving systems increasingly depend on accurate 3D world models to anticipate future scene dynamics and support safe decision-making. This paper introduces COME (Control into the Occupancy world ModEl), a novel approach for forecasting 3D occupancy maps in simulation. Unlike prior methods such as DOME that rely on ego-centric coordinates, COME adopts a scene-centric coordinate system, significantly reducing prediction errors from vehicle perspective shifts. The approach consists of three main components: (1) a diffusion-style transformer world model that predicts future occupancy maps conditioned on historical pose and auxiliary data, (2) a UNet-based forecasting module that restructures input data into scene-centric coordinates and forecasts future vehicle states, and (3) a ControlNet that injects scene-level conditions from the forecasting module into the world model for controllable prediction. On the nuScenes-Occ3D benchmark, COME demonstrates substantial performance improvements over existing methods, supported by extensive ablation studies and qualitative analysis. Overall, the innovative use of scene-centric coordinates and robust results make COME a significant contribution to the autonomous driving field.", "questions": "1. 161-163: Can you explicitly explain what inputs the scene-centric forecasting module receives and what it is predicting? Additionally, how are these predictions propagated to future timesteps? This section is not entirely clear.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Novel approach to scene occupancy synthesis for ego-centric sequence problems.\n- The authors recognize that scene generation is significantly more difficult when using an ego-centric coordinate system, where small changes in vehicle orientation can dramatically alter the scene. Instead, they propose a scene-centric coordinate system, ensuring that perspective shifts from the ego-vehicle do not substantially impact the scene, which reduces the need for constant scene reconstruction as the vehicle moves.\n- The method leverages ControlNet to inject control signals into the Occupancy World Model, facilitating scene synthesis over time. The key observation that 92.7% of occupied voxels are static means the model needs to update only a small portion of the scene. This insight, combined with ControlNet, reduces computational complexity for temporal updates.\n- A masking operation is introduced to ignore control signals from ControlNet in regions not historically observed, since such areas tend to be noisy and difficult to predict accurately.\n2. SOTA or near SOTA performance on multiple configurations on the Occ3D-nuScenes benchmark dataset for 4D occupancy generation.\n- Performance gains shown in Table 1 are substantial, with IoU time averages exceeding the prior SOTA by 13.28%.\n3. Comprehensive evaluation and ablation studies.\n- Multiple useful experiments (including longer sequence forecasting up to 8s, masking strategies, and qualitative ablations) are included, even if some are in the appendix.\n- The ablation study clarifies the importance of each component and demonstrates performance drops due to issues such as misalignment of 2D pose and yaw for predicted future waypoints.\n- It also highlights the benefit of the ControlNet-style training protocol, emphasizing training one component at a time.\n\n**Weaknesses**\n1. Unclear motivation and applicability to real-world scenarios.\n- The introduction lacks a clear explanation of why generating occupancy maps is critical for autonomous driving.\n- More detail is needed on why perspective shifts from ego-centric systems are problematic, ideally with statistics or realistic examples.\n2. Some lack of clarity and organization in the writing.\n- 46-57: This section is difficult to follow, especially for readers unfamiliar with the methods. These details would fit better in the methods section.\n- 58-61: Discussion of input sources and prediction horizons is misplaced in the introduction.\n- 39-45, 66-68: Repeatedly foreshadowing SOTA results feels redundant.\n- 153-163: The purpose and operation of the scene-centric forecasting UNet are not explained clearly enough.\n3. Minor editing issues:\n- 134: Periods for e.g.\n- 543: Confusing sentence.\n\nOverall, the method is interesting and delivers strong experimental results, but the motivation and presentation could be improved.", "was_revised": true, "final_justification": "The authors have done an admirable job addressing our concerns. Notably, they evaluated their method on an additional benchmark dataset, as requested by reviewers 2AyB and Gif3 — an effort I especially appreciate. They also responded thoroughly to my comments regarding clarity and motivation. Most importantly, they substantiated many of their claims with experimental results, reinforcing the advantages of COME over the baseline DOME. Overall, I find the proposed approach compelling and believe it will be valuable to the community, which is why I am recommending acceptance.", "rebuttal": "We thank the reviewer for recoginition of novelty and experimental results. We sincerely thank the reviewer for pointing out clarification issues and suggesting potential improvements to the introduction and methodology sections. We have revised our draft accordingly to enhance its comprehensiveness. Below, please find our responses to other concerns:\n\n\n\n* **\\[W1.1]**: Unclear motivation and applicability to real-world scenarios. The introduction lacks a clear explanation of why generating occupancy maps is critical for autonomous driving.\n\n**\\[A1.1]**:&#x20; Thank you for pointing this out. We will add explanations in the revised draft. Occupancy is widely recognized as a key intermediate representation for understanding drivable space in autonomous driving, effectively handling spatial perception of general obstacles. The generative occupancy world model has multiple potential applications:\n\n(1) Predicting future occupancy grid maps based on planned trajectories and validating collision risks for the current trajectory \\[1].\n\n(2) Serving as a critical intermediate result for synthesizing simulation data \\[2]\\[3]. Generated occupancies can be rendered to the image perspective via Gaussian splatting \\[4], acting as accurate geometric controls for driving video generation.\n\n\n\n\n\n* **\\[W1.2]**: More detail is needed on why perspective shifts from ego-centric systems are problematic, ideally with statistics or realistic examples.\n\n**\\[A1.2]**:&#x20; Perspective shifts in ego-centric systems are problematic due to the challenge of relocating visible dynamic agents and static environments. We classify scenes as having perspective shifts based on whether the vehicle moved more than 5 meters in 5-second prediction clips. We test DOME \\[5] and COME in these scenarios, with quantitative results in Table 7. DOME performs well in ego-static scenes (where other agents are static) but poorly in ego-dynamic ones, achieving only 47% mIoU and 52.9% IoU (vs. ego-static). In contrast, COME shows larger improvements in ego-dynamic scenes (36.3% mIoU, 26.88% IoU) than in ego-static ones (8.8% mIoU, 4.2% IoU), demonstrating the effectiveness of our scene-centric forecasting control for the world model.\n\n\n\n| Model | Scenarios | 1s-mIoU | 2s-mIoU | 3s-mIoU | 3s-AVG mIoU | 1s-IoU | 2s-IoU | 3s-IoU | 3s-AVG IoU |\n| ----- | --------- | ------- | ------- | ------- | ----------- | ------ | ------ | ------ | ---------- |\n| DOME  | All       | 35.11   | 25.89   | 20.29   | 27.10       | 43.99  | 35.36  | 29.74  | 36.36      |\n| DOME  | Static    | 56.89   | 47.73   | 40.20   | 48.27       | 67.44  | 61.29  | 53.56  | 60.76      |\n| DOME  | Dynamic   | 30.60   | 21.44   | 16.32   | 22.79       | 39.40  | 31.07  | 25.90  | 32.12      |\n| COME  | All       | 42.75   | 32.97   | 26.98   | 34.50       | 50.57  | 43.47  | 38.36  | 44.13      |\n| COME  | Static    | 61.27   | 52.32   | 45.27   | 52.95       | 68.66  | 64.15  | 57.80  | 63.53      |\n| COME  | Dynamic   | 39.15   | 29.60   | 24.46   | 31.06       | 47.05  | 39.82  | 35.42  | 40.76      |\n\nTable 7. Model performance on scenes classified according to ego movement.&#x20;\n\n\n* **\\[W2]:** Some lack of clarity and organization in the writing: 46-57: This section is difficult to follow, especially for readers unfamiliar with the methods. These details would fit better in the methods section. 58-61: Discussion of input sources and prediction horizons is misplaced in the introduction. 39-45, 66-68: Repeatedly foreshadowing SOTA results feels redundant. 153-163: The purpose and operation of the scene-centric forecasting UNet are not explained clearly enough.\n\n**\\[A2]:** Thanks for pointing out paragraphs that are hard to understand. We will revise the manuscript based on the instructions above. For l46-57, we will simplify the module introduction and place the original paragragh in methodology. For l58-61, we will place the test details in the experiment section. For l39-45, we will delete the redundant description. For l153-163, we will thoroughly rewrite this section to clarify its purpose and operation.\n\n* **\\[W3]:** Minor editing issues: 134: Periods for e.g. 543: Confusing sentence.\n\n**\\[A3]:** Thanks for pointing out editing errors, we will check and correct editing errors in the manuscript.  Line 543 should be revised as: We try to perform masks on conditions before COME ControlNet rather than masks on controls after COME ControlNet, but the qualitative results are poor, so we do not pose the masks in condition features. \n\n\n* **\\[Q1]:** 161-163: Can you explicitly explain what inputs the scene-centric forecasting module receives and what it is predicting? Additionally, how are these predictions propagated to future timesteps? This section is not entirely clear.\n\n**\\[A4]: &#x20;**&#x54;he scene-centric forecasting module takes historical occupancy sequences and ego poses as input. Historical occupancy sequences are transformed to the current ego pose, resulting in semantic occupancy sequences of shape \\[D\\_t, H, W, L] under the same coordinate system. The module outputs future scene-centric occupancy sequences of shape \\[D\\_τ, H, W, L], which are concatenated with the input sequences to form a tensor of shape \\[D\\_t + D\\_τ, H, W, L]—serving as the input condition feature for COME ControlNet.\n\n\n\n\\[1] Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving, AAAI 2025.\n\n\\[2] Uniscene: Unified occupancy-centric driving scene generation, CVPR 2025.\n\n\\[3] Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation, CVPR2025.\n\n\\[4] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, ACM Transactions on Graphics (TOG) 2023.\n\n\\[5] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024"}, {"review_id": "dtRdyVSnAu", "reviewer": "Reviewer_hsxA", "summary": "This paper proposes a simple scene-wise forecasting module by leveraging the observation that a significant portion of voxels in the occupancy world model are static and shared across sequential scenes. The output of this module is then used as a condition for the COME ControlNet, resulting in improved performance. The proposed approach is evaluated through comparisons of 4D occupancy generation performance using various input modalities, including camera inputs and occupancy inputs.", "questions": "1. L158 and L160: Definitions for D_t and D_τ are missing. -> This creates confusion regarding how the transformed sequence x' is handled. Are all sequential voxels concatenated into x' ?\n2. Would directly using the transformed voxels as conditions yield a similar effect? This approach may be analogous to the use of warping in the image domain to leverage information from previous frames, as shown in [1].\n[1] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n1. This paper proposes a method that leverages the consistency of scene-centric representations to construct scene conditions.\n2. The proposed method demonstrates significant performance improvements over the baselines across various input settings.\n\n[Weaknesses]\n1. In Table 1, when comparing methods that utilize camera input, the same occupancy prediction model should be used across all baselines for a fair comparison. To properly attribute the performance improvement to the proposed method, a fair comparison using an identical occupancy prediction backbone is necessary.\n2. Given that the scene-centric forecasting module relies on a large proportion of static voxels. In that case, what is the specific advantage of using predicted future occupancy prediction from a simple Unet architecture as a condition? \n3. Since the output of the scene-centric forecasting module is used as a condition, concerns arise about how moving objects are accounted for. In a world model, rather than simply maintaining the static regions of a scene, it is important to predict how the scene will evolve. It is unclear how the proposed method addresses this aspect.", "was_revised": true, "final_justification": "All of my concerns have been adequately addressed.", "rebuttal": "We thank the reviewer for recognition of experimental results. We sincerely thank the reviewer for suggestions to enhance originality clarification. We supplement comparison experiments under identical camera and ego trajectory inputs, experiments with varying input conditions to demonstrate the effectiveness of scene-centric forecasting for downstream generation, and statistics on dynamic categories.\n\nWe respectfully disagree with originality. Prior works (DOME \\[10], OccWorld \\[1]) remain in ego-centric coordinates, suffering performance drop under perspective shifts. Our novelty lies in recognizing that \\*\\*short-term scene-centric forecasting of moving regions is sufficient and necessary\\*\\*; we are the first to formalize this as a ControlNet condition. The ablation in Table 10 shows that removing this component leads to a 31 % performance drop, confirming our nontrivial contribution and novelty. Below are our responses to other specific concerns:\n\n* **\\[W1]**: The same occupancy prediction model with camera input should be used across all baselines for a fair comparison. \n\n**\\[A1]**: Thank you for highlighting the need to address experiments with potentially different setups and inputs. We prepare two occupancy predictors that are the same as prior works and demonstrate COME's performance in Table 8. COME with both inputs receives the best performances.\n\n| Model              | Occupancy Predictor           | Image Backbone        | 3s-AVG mIoU | 3s-AVG IoU |\n| ------------------ | ----------------------------- | --------------------- | ----------- | ---------- |\n| OccWorld \\[1]      | TPVFormer \\[2]                | N/A                   | 8.62        | 16.53      |\n| OccLLaMA \\[3]      | FB-OCC \\[4]                   | N/A                   | 8.66        | 22.99      |\n| OccVAR \\[5]        | FB-OCC                        | N/A                   | 11.79       | 24.35      |\n| Occ-LLM \\[7]       | FB-OCC                        | N/A                   | 10.21       | 23.79      |\n| COME (Ours)        | FB-OCC                        |                   ResNet-50 | 16.09       | 33.02      |\n| DFIT-OccWorld \\[6] | BEVDet Series (Not Specified) | N/A                   | 10.50       | 17.02      |\n| RenderWorld \\[8]   | BEVStereo \\[9]                | Swin-B                | 2.58        | 13.73      |\n| COME (Ours)        | BEVStereo                     | Swin-B                | 19.11       | 37.63      |\n\nTable 8. Comparisons of Camera-input occupancy world models\n\nWe add a quantitative comparison between DOME and COME with strictly the same input, as shown in Table. 9. Compared to DOME, COME has 17.5% mIoU and 8.5% improvement under BEV-Planner\\[11] ego trajectories, and 17.3% mIoU and 15.7% IoU improvement under ground-truth ego trajectories. &#x20;\n\n| Model | Occupancy Predictor | Image Backbone | Ego Trajectory | 3s-AVG mIoU | 3s-AVG IoU |\n| ----- | ------------------- | -------------- | -------------- | ----------- | ---------- |\n| DOME  | BEVStereo           | Swin-B         | BEV-Planner    | 16.15       | 34.68      |\n| COME  | BEVStereo           | Swin-B         | BEV-Planner    | 19.11       | 37.63      |\n| DOME  | BEVStereo           | Swin-B         | GT             | 18.97       | 38.17      |\n| COME  | BEVStereo           | Swin-B         | GT             | 22.26       | 44.07      |\n\nTable 9. Comparison with aligned camera input and ego trajectory.\n\n* **\\[W2]**: Given many static voxels, what is the specific advantage of using predicted future occupancy prediction from a simple Unet architecture as a condition?\n\n**\\[A2]**: The scene-centric forecasting module handles not only static voxels but also dynamic agents, retaining static voxels while accurately predicting future dynamic voxels. A key advantage is that pre-processing rigid body coordinate transformations simplify predicting from ego-centric to scene-centric views. Results for dynamic categories (Table. 11) demonstrate the module’s prediction ability.\n\nWe add an experiment (Table 10) where the COME world model is controlled solely by static voxels. This yielded marginal improvements (0.37 mIoU, 1.23 IoU), showing static voxels as controls are less effective than the proposed UNet module.\n\n| Model            | Condition     | 1s-mIoU | 2s-mIoU | 3s-mIoU | 3s-AVG mIoU | 1s-IoU | 2s-IoU | 3s-IoU | 3s-AVG IoU |\n| ---------------- | ------------- | ------- | ------- | ------- | ----------- | ------ | ------ | ------ | ---------- |\n| COME-World-Model | N/A           | 30.54   | 21.89   | 17.74   | 23.39       | 38.99  | 31.38  | 26.66  | 32.34      |\n| COME-ControlNet  | Static Voxels | 30.82   | 22.24   | 18.21   | 23.76       | 40.28  | 32.33  | 28.11  | 33.57      |\n| COME-ControlNet  | UNet          | 42.75   | 32.97   | 26.98   | 34.23       | 50.57  | 43.47  | 38.36  | 44.13      |\n\nTable 10. Model performance when COME world model is controlled by static voxels.\n\n\n\n* **\\[W3]**: How moving objects are accounted for in a world model?\n\n**\\[A3]**: The scene-centric forecasting module perceives static surroundings and strongly predicts moving objects’ evolution, providing a geometric prior for generative models. We compare moving object prediction across the stage-1 world model, stage-2 scene-centric forecasting, and stage-3 COME ControlNet, listing IoU metrics for moving categories below. Table 11 reports the average IoU of moving categories across 6 future frames (0.5s to 3.0s). The UNet module outperforms DOME in predicting all moving categories, so using UNet as a prior significantly enhances COME-ControlNet’s dynamic object prediction.\n\n| Model                          | Bicycle 3-s AVG IoU | Bus 3-s AVG IoU | Car 3-s AVG IoU | Construction Vehicle 3-s AVG IoU | Motorcycle 3-s AVG IoU | Pedestrian 3-s AVG IoU | Trailer3-s AVG IoU | Truck3-s AVG IoU |\n| ------------------------------ | ------------------- | --------------- | --------------- | -------------------------------- | ---------------------- | ---------------------- | ------------------ | ---------------- |\n| DOME                           | 24.15               | 24.76           | 27.17           | 29.15                            | 15.04                  | 11.81                  | 26.71              | 27.12            |\n| COME-World-Model               | 21.12               | 24.88           | 24.02           | 21.75                            | 10.84                  | 9.72                   | 26.16              | 24.65            |\n| COME-Scene-Centric Forecasting | 30.71               | 38.92           | 37.04           | 39.70                            | 30.03                  | 32.53                  | 41.43              | 42.89            |\n| COME-ControlNet                | 29.14               | 33.55           | 31.83           | 33.84                            | 28.34                  | 23.11                  | 34.49              | 30.75            |\n\nTable 11. Comparison of model performance on dynamic categories on nuScenes dataset. \n\n* **\\[Q1]**: L158 and L160: Definitions for D\\_t and D\\_τ are missing. -> This creates confusion regarding how the transformed sequence x' is handled. Are all sequential voxels concatenated into x' ?\n\n**\\[A4]**:  We will thoroughly rewrite the UNet section. D\\_t denotes the number of input historical frames, and D\\_τ the number of output future frames; both are set based on the task formulation. The transformation from x to x' is a simple rigid transformation: each voxel’s center point in x is converted to the current frame’s coordinates, resulting in x' having the same shape as x.\n\n* **\\[Q2]**: Would directly using the transformed voxels as conditions yield a similar effect? This approach may be analogous to the use of warping in the image domain to leverage information from previous frames, as shown in \\[1]. \\[1] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion\n\n**\\[A5]**: We add an experiment that all visible voxels are transformed and then inputted to COME ControlNet as conditions. Table 12 shows this variant improves marginally over the COME world model (0.88 mIoU, 0.81 IoU) but is far less effective than the UNet module.\n\n\n\n| Model            | Condition          | 1s-mIoU | 2s-mIoU | 3s-mIoU | 3s-AVG mIoU | 1s-IoU | 2s-IoU | 3s-IoU | 3s-AVG IoU |\n| ---------------- | ------------------ | ------- | ------- | ------- | ----------- | ------ | ------ | ------ | ---------- |\n| COME-World-Model | N/A                | 30.54   | 21.89   | 17.74   | 23.39       | 38.99  | 31.38  | 26.66  | 32.34      |\n| COME-ControlNet  | Transformed Voxels | 31.31   | 22.66   | 18.85   | 24.27       | 39.82  | 31.86  | 27.76  | 33.15      |\n| COME-ControlNet  | UNet               | 42.75   | 32.97   | 26.98   | 34.23       | 50.57  | 43.47  | 38.36  | 44.13      |\n\nTable 12. Model performance when COME world model is controlled by all transformed voxels.\n\n\n\\[1] OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving, ECCV2024.\n\n\\[2]Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction, CVPR2023.\n\n\\[3] OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving, Arxiv 2024.\n\n\\[4] FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation, Arxiv 2023.\n\n\\[5] OccVAR: Scalable 4D Occupancy Prediction via Next-Scale Prediction, Openreview 2024.\n\n\\[6] An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training, Arxiv 2024.\n\n\\[7] Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models, ICRA2025.\n\n\\[8] RenderWorld: World Model with Self-Supervised 3D Label, ICRA2025.\n\n\\[9] BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo, AAAI2023.\n\n\\[10] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.\n\n\\[11] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving, CVPR2024."}, {"review_id": "znEIJ3P2NG", "reviewer": "Reviewer_Gif3", "summary": "This paper proposes a world model for 3D occupancy forecasting.The key idea is to disentangle the evolution of the scene from that of the ego-vehicle motion. The method consists of three parts: 1) scene-centric forecasting using a unet architecture on historical occupancy. 2) World-model inference predicting future occupancy conditioned on historical occupancy, trajectory, or BEV layout. The paper demonstrates strong performance on the nuScenes 3D-Occ dataset.", "questions": "Is it possible to verify that the method works on more diverse driving scenarios?\nHow does the model handle user-specified changes to the BEV-layout?\nHow does the model handle adversarial scenarios or scenarios where a crash is imminent?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "_Strengths_:\nThe idea of separating ego-motion from scene evolution is elegant and intuitive, and the authors demonstrate that this design choice leads to improved forecasting performance. I think the method is well-motivated and clearly presented. The paper is generally well-written and easy to follow. The experimental results are strong, with competitive performance on nuScenes 3D occupancy prediction benchmarks. Also, the ablation studies are fairly informative and help validate the contributions of the method’s components.\n\n_Weaknesses_:\nThe method is only evaluated on nuScenes, which is relatively constrained in terms of driving diversity. Ego-motion is often limited to driving in-lane following the road, and the environment contains few truly complex or rare events. This limits insight into how the model might generalise to more diverse or interactive settings. Furthermore, the paper does not explore the generation of fictional or counterfactual scenarios, which is a key promise of using world models for training and evaluation data. For example, it is unclear whether the model could simulate a crash scenario or how it would react to structural changes in the environment (e.g., the insertion or removal of a lane or a pedestrian crossing). It would strengthen the paper to evaluate or at least qualitatively discuss the model's capacity for reasoning about interventions or rare events, which are crucial for planning and decision-making.", "was_revised": true, "final_justification": "The authors resolved the concerns I had and I think the updated paper will be a good contribution to the community.", "rebuttal": "We thank the reviewer for recognition of novelty, writing and experimental results. We sincerely thank the reviewer for suggesting evaluating the proposed method under more diverse driving scenarios and adversarial conditions. We have supplemented initial verification on Occ3D-Waymo\\[1] and are currently working with the new dataset. Additionally, we have added analyses explaining how to ensure generated results closely align with input condition changes to meet simulation needs.\n\n* **\\[W1]**: The method is only evaluated on nuScenes, which is relatively constrained in terms of driving diversity. Ego-motion is often limited to driving in-lane following the road, and the environment contains few truly complex or rare events. This limits insight into how the model might generalise to more diverse or interactive settings.&#x20;\n\n**\\[A1]**: nuScenes has limited diversity in ego-motion and complex objects, so we add experiments on the more complex Waymo Open Dataset (WOD). Occ3d-Waymo, built on WOD perception data, includes 798 training and 202 test sequences—richer and more diverse than nuScenes’ training scenes—with 10Hz sensor data and annotations. Our task is the same as nuScenes: predict 3s of future using 2s of past occupancy data at 0.5s intervals.\n\nWe retrain OCC-VAE, UNet, DOME \\[2], COME world model, and COME ControlNet on Occ3d-Waymo, initializing from nuScenes-trained components to save cost and time, with unchanged model sizes and training schedules.\n\nCurrently we report a COME variant with the same structure and with stronger geometric controls from the scene-centric forecasting module, differing by additionally replacing noisy BEV latents with UNet-encoded latents in future visible areas during denoising. This variant converges faster, yields slightly better quantitative results but risks reducing prediction multimodality. Due to computational limits, the default COME model requires a few more training days (Current status: mIoU=29.35 and IoU=46.46 in epoch 20/1000,  already surpassing DOME and COME world model).\n\nVAE reconstruction quality: mIoU=74.88, IoU=82.57.\n\nThe results of Waymo Occ3D are shown in the following tables. In general comparison (Table. 13) , our reproduced DOME achieves similar performance as in nuScenes dataset. COME world model has marginally better performance with DOME by 2.38 mIoU and 0.32 IoU. Scene-centric forecasting module outperforms the generative world model by a large margin (63% mIoU and 19.5% IoU). COME ControlNet, which built on control features provided with UNet module, maintains the same 3-s average mIoU and further improves 3-s average IoU to 23.7% improvement.\n\n| Method                                  | 1s-mIoU | 2s-mIoU | 3s-mIoU | 3s-AVG mIoU | 1s-IoU | 2s-IoU | 3s-IoU | 3s-AVG IoU |\n| --------------------------------------- | ------- | ------- | ------- | ----------- | ------ | ------ | ------ | ---------- |\n| DOME                                    | 31.62   | 24.82   | 19.65   | 25.36       | 51.24  | 43.17  | 37.47  | 43.96      |\n| COME Stage 1: World Model               | 34.45   | 26.28   | 22.50   | 27.74       | 51.16  | 43.24  | 38.44  | 44.28      |\n| COME Stage 2: Scene-Centric Forecasting | 47.37   | 40.85   | 36.31   | 41.51       | 59.69  | 52.27  | 45.60  | 52.52      |\n| COME Stage 3: ControlNet                | 47.29   | 40.80   | 36.44   | 41.51       | 60.63  | 53.98  | 48.62  | 54.41      |\n\nTable 13. Comparisons of general 4D occupancy forecasting performance between DOME and different stages of COME on Waymo Occ3D dataset.&#x20;\n\n\n\nWe also compare the per-category 4D-occupancy forecasting performance of moving objects in Waymo Occ3D dataset in Table. 14. COME outperforms DOME on all moving categories by a large margin.&#x20;\n\n| Method | Vehicle IoU | Bicyclist IoU | Pedestrain IoU | Bicycle mIoU | Motorcycle-IoU |\n| ------ | ----------- | ------------- | -------------- | ------------ | -------------- |\n| DOME   | 38.28       | 16.83         | 21.15          | 24.34        | 9.9            |\n| COME   | 50.98       | 31.85         | 41.63          | 36.78        | 27.75          |\n\nTable 14. Comparisons of 4D occupancy forecasting performance of moving categories between DOME and COME on Waymo Occ3D dataset.&#x20;\n\nTo demonstrate the prediction of truly complex or rare events, we hope to further train and test our method on industrial-level private autonomous driving data and corner cases.\n\n* **\\[W2]**: Furthermore, the paper does not explore the generation of fictional or counterfactual scenarios, which is a key promise of using world models for training and evaluation data. For example, it is unclear whether the model could simulate a crash scenario or how it would react to structural changes in the environment (e.g., the insertion or removal of a lane or a pedestrian crossing). It would strengthen the paper to evaluate or at least qualitatively discuss the model's capacity for reasoning about interventions or rare events, which are crucial for planning and decision-making.\n\n\n\n**\\[A2]**: Thanks for pointing out the potential application of COME. World models for simulation typically use obstacle box positions and maps as control conditions, generating content that adapts to changes in these conditions. Our BEV Condition control scheme mirrors UniScene \\[3], which yields good rendering with customized BEV layout. Unlike UniScene, our method adds external controls for physics-consistent prediction, with adaptive scene-centric forecasting control (details in A3.2).\n\n\n\nWe test static (by randomly removing half map patches) and dynamic (by halving box-to-car center distances) scene distortions. In both, COME prioritizes BEV layout guidance rather than conditions from the scene-centric forecasting module, generating occupancy maps qualitatively matching the layouts. Per new NeurIPS guidelines, results can’t be visualized but will be added to supplements in revision.\n\n\n\n* **\\[Q1]**: **\\[Q1.1]: Is it possible to verify that the method works on more diverse driving scenarios? **\\[Q1.2]: How does the model handle user-specified changes to the BEV-layout? **\\[Q1.3]: How does the model handle adversarial scenarios or scenarios where a crash is imminent?\n\n\n\n**\\[A3.1]**: We verify the method on the Waymo Open Dataset, which has more diverse driving scenarios.\n\n**\\[A3.2]**: The model accepts user-specified changes or adversarial scenarios via BEV-layout modifications, validated in UniScenes. A potential issue is conflicts between the Scene-Centric Forecasting module and BEV-layout control conditions (e.g., other vehicles' behavior conflicting with historical inertia). An elegant fix: mark adversarial simulation-focused agent areas, mask the corresponding Scene-Centric Forecasting results, ensuring only BEV-layout covers these regions to maintain global physical consistency and local customization.\n\n**\\[A3.3]**: COME's simulation model takes adversarial BEV layouts (e.g., from adversial traffic simulators like Challenger \\[4]) as input conditions. When agents crash, COME merges them into a single occupancy instance. World models for recognizing and predicting such instances are left for future work.\n\n\n\n\\[1] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, NeurIPS 2023.\n\n\\[2] Dome: Taming diffusion model into high-fidelity controllable occupancy world model, arxiv 2024.\n\n\\[3] Uniscene: Unified occupancy-centric driving scene generation, CVPR 2025.\n\n\\[4] Challenger: Affordable Adversarial Driving Video Generation"}]

Author rebuttal:

Meta-review: The paper proposes COME, a scene-centric control framework for occupancy world models, which disentangles ego-motion from scene dynamics. The method achieves significant performance gains on nuScenes and, following reviewer requests, also on Occ3D-Waymo. The key strength lies in its novel scene-centric formulation and strong empirical improvements across multiple configurations. While concerns were raised about evaluation scope and computational overhead, the rebuttal provided additional experiments (including Waymo results and a lightweight variant) that convincingly addressed these issues. All four reviewers found the rebuttal convincing and maintained positive recommendations.
Decision: Accept (poster)