Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
Authors: Xiang Li
Pengfei Li
Yupeng Zheng
Wei Sun
Yan Wang
yilun chen
Venue: ICLR 2025 Poster (2025)
Keywords: Autonomous Driving; Occupancy; World Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Understanding world dynamics is crucial for planning in autonomous driving. Recent methods attempt to achieve this by learning a 3D occupancy world model that forecasts future surrounding scenes based on current observation. However, 3D occupancy labels are still required to produce promising results. Considering the high annotation cost for 3D outdoor scenes, we propose a semi-supervised vision-centric 3D occupancy world model, **PreWorld**, to leverage the potential of 2D labels through a novel two-stage training paradigm: the self-supervised pre-training stage and the fully-supervised fine-tuning stage. Specifically, during the pre-training stage, we utilize an attribute projection head to generate different attribute fields of a scene (e.g., RGB, density, semantic), thus enabling temporal supervision from 2D labels via volume rendering techniques. Furthermore, we introduce a simple yet effective state-conditioned forecasting module to recursively forecast future occupancy and ego trajectory in a direct manner. Extensive experiments on the nuScenes dataset validate the effectiveness and scalability of our method, and demonstrate that PreWorld achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks.

Published as a conference paper at ICLR 2025
SEMI-SUPERVISEE VISION-CENTRIA 3D OCCUPANCY WORLD MODEL FOR AUTONOMOUS DRIVING
Xiang Li, Pengfei Li, Yupeng Zheng, Wei Sun, Yan Wang", Yilun Chen" Institute for Al Industry Research (AIR), Tsinghua University 1 1-x21 @ma i ts tainghua edu. en; wangyanBair.. wtt.eearrrriiiiiiiee tainghua...... edu.cn
ABSTRACT
Understanding world dynamics is crucial for planning in autonomous driving. Re- cent methods attempt to achieve this by learning 3D occupancy world model that forecasts future surrounding scenes based on current observation. However, 3D occupancy labels are still required to produce promising results. Considering the high annotation cost for 3D outdoor scenes, we propose a semi-supervised vision-centric 3D occupancy world model, PreWorld, to leverage the potential of 2D labels through a novel two-stage training paradigm: the self-supervised pre- training stage and the superssue fine-tuning stage. Specifically, during the pre-training stage, we utilize an attribute projection head to generate different at- tribute fields of scene (e.g., RGB, density, semantic), thus enabling temporal supervision from 2D labels via volume rendering techniques. Furthermore, we in- troduce simple yet effective condicoondti forecasting module recursively forecast future occupancy and ego trajectory in a direct manner. Extensive exper- iments on the nuScenes dataset validate the effectiveness and scalability of our method, and demonstrate that PreWo achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks
(a) Self Supervieede 3D Occupancy Model
Supervision
Input Images
Volume Features
Semantic Occupancy
Labels
Occupancy Network
Occupancy
Head
Input Images
Semantic Occupancy
Future Occupancy
3D Occupancy Labels
3D Occupancy
Forecasting
Model
Module
(b) Fully-Supervised 3D Occupancy World Model
Input Images
Volume Features
Future Volume Features
Future Occupancy
Occupancy
Forecasting
Occupancy Head
Network
Module
(c) Semi- Seeii-Sueervieed 3D Occupancy World Model (Ours)
Figure 1: (a) Self-Supervised 3D Occupancy Model be trained using solely 2D labels supervision. However, it lacks the capability to forecast future occupancy. In contrast, (b) Fully- Supervised 3D Occupancy World Model can forecast future occupancy, but it relies 3D oc- cupancy labels for meaningful results due to its indirect architecture, which employs frozen 3D occupancy model. To tackle these challenges, our (c) Semi-Supervised 3D Occupancy World Model, featuring 2D rendering supervision and an end-to-end architecture. can forecast future oc- cupancy straightly from image inputs while taking advantage 2D labels.
"Corresponding authors.
Coodes and models be accessed http:///////ettto////////et//rrr

---

Published as a conference paper at ICLR 2025
INTRODUCTION
3D scene understanding forms the cornerstone of autonomous driving, exerting a direct influence on downstream tasks such as planning and navigation. Among various 3D scene understanding tasks (Wang et al. [022] et al. 2022a Wei et al. 2023 Jin et al.
2024). .11 Occupancy Prediction plays crucial role in autonomous systems. Its objective is to predict the semantic occupancy of each voxel throughout the entire scene from limited observation. To this end, some previous methods (Liong et al. 2020 Cheng al. 202T] Xia et al. 2023) prioritize LiDAR input modality due to its robust performance in capturing accurate geometric information. Nevertheless, they are often considered hardware- expeenive. Consequently, there has been a shift towards vision-centric
solutions in recent years (Zhang et al. 2023c Li et al. 2023a Zheng et al. 2024).
Despite significant advancements in aforementioned methods, they primarily focus on enhancing better perception of the current scene. For advanced collision avoidance and route planning, au- tonomous vehicles need to not only comprehend the current scene but also forecast the evolution of future scenes based on the understanding of world dynamics. Therefore, 4D Occupancy Forecasting has been introduced to forecast future 3D occupancy given historical observations. Recent works have aimed to achieve this by learning 3D occupaney world model (Zheng al. 2023 Wei al., 2024). However, when processing image inputs, these methods follow an circuitous path, as shown in Fig (b). Typically, a pre-trained 3ID occupancy model is employed to obtain current occupancy, which is then fed into a forecasting module to generate future occupaney. The forecasting module includes a tokenizer that encodes occupancy into discrete tokens, an autoregressive architecture to generate future tokens, and a decoder to obtain future occupancy. Information loss is prone to occur in such repeated encoding and decoding processes. Hence, existing methods heavily rely on 3D occupancy labels as supervision to produce meaningful results, leading to notable annotation costs.
In contrast to 3D occupancy labels, 2D labels are relatively easier to acquire. Recently, employing purely 2D labels for self-supervised learning has shown some promising results in 3D) occupancy prediction task, as illustrated Fig [ (a). By utilizing volumetric rendering, RenderOce (Pan al,) 2024) employs 2D depth maps and semantic labels to train the model. Methods like SelfOcc (Huang) ct al. 2024) and OccNerf (Zhang et al. 2023a) take a step further, using only image sequences as supervision. However, there have not been similar attempts in 4D occupancy forecasting task.
Based on the above observations, we propose PreWorld, a semi-supervised vision-centric 3D oc- cupancy world model, designed to fulfill the utility of 2D labels during training, while achieving competitive performance across both 3D occupancy prediction and 4D occupancy forecasting tasks, as shown in Fig | (c). To this end, we propose novel two-stage training paradigm: the self- supervised pre-training stage and the fully-supervised fine-tuning stage. Inspired by RenderOcc, during the pre-training stage, we introduce an attribute projection head to obtain diverse attribute fields of current and future scenes (e.g., RGB, density, semantic), facilitating temporal supervision through 2D labels using volume rendering techniques. Moreover, we propose simple yet effective state-con catte-conditoneed forecasting module, which allows us to simultaneously optimize occupancy net- work and forecasting module, and directly forecast future 3D occupancy based on multi-view image inputs in an end-to-end manner, thus avoiding possible information loss.
To demonstrate the effectiveness of PreWo we conduct extensive experiments on the widely used
benchmark (Tian et al. (024) and compare with recent methods using both 2D and/or 3D supervision. Experimental results indicate that our approach can yield competitive performance across multiple tasks. For 3D occupancy prediction, PreWorld outperforms the pre- vious best method OccFlowNet (Boeder et al. 2024) with an mloU of 34.69 over 33.86. For 4D occupancy forecasting, Pre World sets the new SOTA performance, outperforming existing methods OccWorld (Zheng et al. 2023) and LcCLLaMM (Wei et al., 2024). For motion planning. World yields comparable and often better results than other vision-centric methods (Hu et al. 2022 Jiang et al. 2023 Tong et al. 2023). Furthermore, we validate the scalability of our two-stage training paradigm, showcasing its potential for large-scale training.
Our main contributions are as follows:
A sumi-super vision-centric 3D occupancy world model, PreWorld, which takes ad- vantage of both 2D labels and 3D occupancy labels during training.
2

---

Published as a conference paper at ICLR 2025
A novel two-stage training paradigm, the effectiveness and scalability of which has been validated by extensive experiments.
A simple yet effective saate-conditionel forecasting module, enabling simultaneous opti- mization with occupancy network and direct future forecasting based visual inputs.
Extensive experiments compared to SOTA method, demonstrating that method achieves competitive performance across multiple tasks, including 3D occupancy prediction, 41) occupancy forecasting and motion planning.
RELATED WORK
2.1 3D OCCUPANCY PREDICTION
Due to its vital application in autonomous driving, 3D occupancy prediction has attracted consider- able attention. According to the input modality, existing methods can be broadly categorized into LiDAR- -baeed and vision-centric methods. While LiDAR-based methods excel in capturing geomet- ric details (Tang al. 2020 Ye et al. 2021 2023), vision-centric methods have garnered growing interest in recent years due to their rich semantic information, cost-effecttiveness, and case of de- ployment (Philion & Fidler 2020 Liu et al. 2023 Ma et al. (024). However, these methods focus solely on understanding the current scene while ignoring the forecasting of future scene changes. Therefore in this paper, follow the approach of OccWorld (Zheng al. 2023) and endeavor to address both of these tasks in a unified manner.
2.2 WORLD MODELS FOR AUTONOMOUS DRIVING
The objective of world models is to forecast future scenes based on action and past observations (Ha
Schmidhuber] 2018 In autonomous driving, world models can be utilized to generate synthetic data and aid in decision making. Some previous approaches (Hu et al., 2023a Gao et al. 2023 Wang et al., 2024) aim to generate image sequences of outdoor driving scenarios using large pre- traindd generative models. However, relying on 2D images as scene representations leads to the lack of structural information. Some works (Khurana et al.) 2022 2023 Zhang et al., (023b) tend to generate 3D point clouds, which on the other hand, fail to capture the semantic of the scene.
Recent attempts have emerged to generate 3D occupancy representations, which combine an un- derstanding of both semantic and geometric information. The pioneering OceWorld (Zheng et al.) 2023) introduces the 3D occupancy world model that, employing an autoregressive architecture, can forecast future occupancy based current observation. Taking it step further, OccLLaMA (Wei ct al. 2024) integrates occupancy, action, and language, enabling 3D occupancy world model to pos- sess reasoning capabilities. However, when it comes to vision-centric approaches, they both adopt an indirect path, requiring the usage of pre-trained 3D occupancy models for current occupancy prediction, succeeded by arduous encoding-decodtee manner poses challenges in model training, thus necessitating 3D occupancy labels as supervision to yield effective results. Considering this, we explore straightforward way directly forecast future occupancy using image inputs.
process forecast future occupancy. This
2.3 SULF-SUPERVISED 3D OCCUPANCY PREDICTION
While 3D occupancy provides rich structural information for training, it necessitates expensive and laborious annotation processes. contrast, 2D labels are more readily obtainable, presenting an opportunity for self-supervised 3D occupancy prediction. Recently, some works have explored using Neural Radiance Fields (NeRFs) (Mildenhall et al., 202T) to perform volume rendering of scenes, thereby enabling 2D supervision for the model. RenderOcc (Pan al. (024) tends to use 2D depth maps and semantic labels for training. Despite significant performance gaps compared to existing methods, SelfOce (Huang et al. 2024) and OceNeRF (Zhang et al. 2023a have made meaningful attempts, aiming to solely utilize image sequences for self-supervised learning.
On the contrary, self-supervised approaches have not yet been observed in the realm of 4D occu- pancy forecasting tasks. Although OccWorld (Zheng et al., (023) offers a self-supervised setting, it merely relies existing self-supervised 3D occupancy model to produce current occupancy
3

---

Published as a conference paper at ICLR 2025
Self-Supervised Pre-training Stage
Multi-View Images
Pr+1
40D
A RGB čytf Semantic S1+f
Attribute Projection Head p
Forecasting Module FF
Density ō
State State-Condi Cond-oioiiiiiddd
Fr+2
Ar+/
A00 2D Rendering Supervision
Occupancy
Network N
Fr+1
Future Attribute Fields
Volume Rendering
Occupancy Head H
3D Occupancy Supervision
Volume Features
Future
Volume Features
Future Semantic Occupancy Pr.f Fully-Supervised Fine-tuning Stage
Recursive Forecasting
Figure 2: The architecture of our proposed PreWorld. Firstly, volume features are extracted from multi-view images with occupancy network. Subsequently, state-conditioned forecasting module is employed to recursively forecast future volume features using historical features. In the self-supervised pre-training stage, volume features projected into various attribute fields and supervised by 2D labels through volume rendering techniques. In the fully-supervised fine-tuning stage, the attribute projection head longer participates in the computations, occupancy predictions are directly obtained via an occupancy head and supervised by 3D occupancy labels.
without engaging in novel endeavors, and it also suffers from subpar performance, Different from OceWorld, we attempt to directly supervise future scenes using 2D labels, thereby optimizing our performance both 3D occupancy prediction and 4D occupancy forecasting tasks simultaneously.
METHOD
3.1
REVISITING 4D OCCPUANCY FORECASTING
For the vehicle at timestamp T. vision-centric 3D occupancy prediction task takes views of im- ages Sr = {rl 12 (?.....77" J as input and predicts current 3D occupancy Yz E xXxxxZxx as output, where (X, Y, 2) denote the resolution of the 3D volume and C represents the number of semantic categories, including non-occupied (Huang et al., 2023 Zhang et al.] 2023c] Liu et al., 2023 Pan 2024). A 3D occupancy model O typically comprises occupancy network N and an occupancy head R. The process of occupancy prediction can be formulated as:
Fr = N(S)), Yr = H(Fr),
(1)
where N extracts 3D volume features Fr. RXxxxZxD from 2D image inputs (D denotes the dimension of volume features), and H serves as a decoder to convert Fr into 3D occupancy.
Vision-centric 4D occupancy forecasting task, the other hand, utilizes image sequence of } as input, aiming at forecasting 3D occupancy of future f frames (Zheng al. 20233 Wei et al., 2024). A 3D occupancy world model W attempt achieve this by adopting an auto-regressive manner:
past & frames (Sr,
(2) To this end, W employs available 3D occupancy model O to predict 3D occupancy of past frames (Yr, Fr-s). and leverages scene tokenizer J, autoregressive architecture A and decoder D to forecast future 3D occupancy. After obtaining historial occupancy, W encodes 3D occupancy into discrete tokens {2T,..., =T through Subsequently, A is utilized to forecast future token 21+1 based on these tokens, which is then input into D to generate future occupancy Y++1- Formally, the process of occupancy forecasting can be formulated as follows:
Yr+1 = W(S Sr.
Yr,.., Y (((T))..,((STr.)))
2T. 2T -((T() J(.r)..., J(Y
(3)
2T+1 = A(2T, k)), Yr+1 = D(2T41).
4

---

Published as a conference paper at ICLR 2025
Here, need to mention that O is pre-trained and frozen during training. For example, Oce- World (Zheng et al. 2023) utilizes FPrmer (Huang et al. 2023) as O, while OccLLaMA (Wei
al. 2024) chooses FB-OCC (Li et 2023c).
3.2 STATE-CONDITIONED FORECASTING MODULE
Different from these approaches, we tend to a more straightforward path, which enables us to op- timize 3D occupancy model and forecasting module simultaneously. Specially, we employ state- conditioned forecasting module IJ instead of the combination of JJ A and D, as illustrated in Fig We formulate our approach of occupancy forecasting as follows:
Fr. = N(Sr, Sr-1,... ST Frt1 - F(Er), Fr+1 = H((F)+1)
(4)
where we leverages N' extract volume features Fr from temporal images, IF to directly forecast future volume features Fr+1 and H to transform Fr+1 into future occupancy Yr+1-
Without loss of generality, our forecasting mod- ule is simply composed of two MLPs. We demonstrate that even without intricate design, this simple architecture can still achieve com- parable and even superior results to state-of- the-art methods. This design showcases that previous practice of solely optimizing the fore- casting module during training has its limita- tions. By simultaneously optimizing the cupancy network and forecasting module, 3D occupancy world models achieve stronger performance. Additionally, our module can op- tionally incorporate ego-state information such as speed, acceleration and historical trajectories into the network. In Section 4.3 we demon- strate that this approach can further enhance the forecasting capabilities of the model.
State Conditioned Forecasting Module
Future Ri
State Tokens
Downscaling Pooling
Volume Features
Future
Volume Features
The proposed state-conditioned fore- casting module is simply composed of two MLPs. Ego states be optionally integrated into the network, as denoted by the dashed arrows.
Figure 3:
Furthermore, this architecture brings an additional benefit for us. Given that previous forecasting modules encode scenes into discrete tokens, they cannot directly supervise future predictions with 2D labels via volume rendering, done by self-supervised 3D occupancy models (Zhang et al. 2023a; Huang et al., 2024). Since our module preserves the volume features of future scenes, provides opportunity to train 3D occupancy world models in self-supervised manner.
3.3 TEMPORAL 2D RENDERING SELF-SUPERVISION
Attribute Projection. Inspired by Pan et al. 2024). we transform the temporal volume feature sequence of current and future f frames {F}
[Fr, Stt....,FFrt) into temporal attribute
fields {A]; through an attribute projection head P:
{Å}. ((,, =
(5)
where ē XXXXXXXI, XXXxxXX and ē Xxxxxxx denote the density, semantic and RGB fields of the 3D volume, respectively, In implementation, P comprises several MLPs, which is validated to be a simple yet effective method (Boeder et al. 2024).
Ray Generation. Given the intrinsic and extrinsic parameters of camera timestamp we can extract set of 3D rays (r), where each ray r originates from camera j and corresponds pixel of the image F/- Additionally, we can utilize ego pose matrices to transform rays from adjacent n frames current frame, enabling better capture of surrounding information. These rays collectively constitute the set (r); utilized for supervising À; (ō;, (6,,8;,64) Rx)
5

---

Published as a conference paper at ICLR 2025
Volume Rendering. For each {r), sample M points {um 1M along the ray. Then the rendering weight ((u) each sampled point um can be computed by:
m-
T(um) = exp(
Σ ds(up)5p), w(um) = T(um)( - exp( -i--z (um))mm))
(6)
where T(um) denotes the accumulated transmittance until tmm and 6m Um++1 Um denotes the interval between adjacent sampled points. Finally, the 2D rendered depth, semantic and RGB predictions (d (r), (r), (r),,?((r)) can be computed by cumulatively summing the products of the values corresponding each point along the ray and their respective rendering weights:
M
(P)
w Um Um: 32
(r)
Σ (uum) (umm) (r)
Σ weum (umm)
Temporal 2D Rendering Supervision. After acquiring 2D rendered predictions (deD with 3D ray set {r) the temporal 2D) rendering loss can be formulated as:
1 (2D)
22D
T+
L2D Σ dep RDD) + Asem sse , + >RGBL ,DD)
(8)
i=
where (dDD, 188 .DDD) represents 2D depth map, semantic label and RGB of corresponding pixels.
3.4 TWO-STAGE TRAINING PARADIGM
Training Scheme. As illustrated in Fig 2 our training scheme for PreWorld includes two stages: In the self-supervised pre-training stage, as illustrated in Section 3.3 we employs the attribute pro- jection head P to enable temporal supervision with 2D labels. This approach allows us to leverage the abundant and easily obtainable 2D labels, while preemptively optimizing both the occupancy network N and forecasting module F. In the subsequent fine-tuning stage, utilize a occupancy head H to produce occupancy results and use 3D occupancy labels for further optimization.
Training Loss. For pre-training stage, we employ temporal 2D rendering loss Lnn as formulated in Eq. Specially, we utilize SILog loss and cross-entropy loss from Pan et al. (2024) as & dep and Ssems respectively, and LI loss as LGBB For fine-tuning stage, we employ Tocal loss L f. lovasz-softmax loss &i and scene-class affinity loss Eeem and _ following the practice of Li
al. (202cc). Therefore, the total loss function for fine-tuning stage can be represented as follows:
LAD Aghy + Aehi + X ysem L gsem + > 2000,comm
(9)
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Dataset and Metrics. Our experiments are conducted on the Oce3D-nuSeenes benchmark Tian al. 2024), which provides dense semantic occupancy annotations for the widely used nuScenes dataset CCasarr et al., 2020). Each annotation covers a range of [-40- ~ 40m, -40 ~ 40m, -1 ~ 5.4m] around the ego vehicle. The ground-truth semantic occupancy is represented as 200x 200 16 3D voxel grids with 0.4m resolution. Each voxel is annotated with 18 classes (17 semantic classes and 1 free). The official split for training and validation sets is employed. Following common practices, we use mloU and loU as the evaluation metric for 3D occupancy prediction and 4D occupancy forecasting tasks, and use error and collision rate for motion planning task.
Implementation Details. We use the identical network architecture for all the three tasks, yet for the non-temporal 3D occupancy prediction task, we omit temporal supervision and losses ac- cordingly. We adopt BEVStereo (Li et al. 2023b) as the occupancy network N. only replacing its detection head with the occupancy head JL from FB-OCC Li et al. (2023c) to produce occupancy prediction. For training, set the batch size to 16, Adam as the optimizer, and train with

---

Published as a conference paper at ICLR 2025
Table 1: 3D occupancy prediction performance on the Occ3D-nuScenes dataset. GT represents
the type of labels used during training. The best and second-bes besst performances are represented by bold and underline respectively.
20lestaan
salfic CH00
1800022000
SANU (50
Stydd
salkr
Blck
4r. SUT
Methed
GF
Sel0Oce
1014
0.15
0.66
12.54
0.00
0.80
2,10
8.25
55
36.30
54
9.30
OccNeRd
0.23
1.84
RenderOcc
2D
5.69
27.56
14
19.91
20
96
12.42
12.14
18.34
20.1
18.94
68.85
33.05
42.01
43.94
17.36
22.61
23:93
OceFooNe
Moxder d 0l. 1024
2D
1.60
27.5
26
00
34,00
32,00
20
40
25
90
18.80
20.00
26
00
28,
70
00
20
57
39.50
29
26
28.42
MonoSonne
76
4.93
567
3
5,00
TPVRmmer
40.78
17
19.99
88
27.83
DEVDel [lon
K0/4
3D
4.39
30.31
0.23
26
36,47
12
10.34
10.36
6.26
8.93
23.65
52.27
24.61
26.06
22.31
15.04
15.10
19.38
OccFormer Zain Welmeu
3D
5.94
30.29
12
32
34
99
17
14 44
16
17.22
9.27
10
26
36
99
30
96
94
32.73
6.76
6.97
21.93
RFVFermer Frel 25
3D
5.85
83
17
87
40
42
7.36
23
8K
81
20 98
22
18
50
70
55
35
28
36
36 08
28.06
20.04
17.69
26.88
Reada
CTF-0w
24.52
22.72
SpreseOce
30.00
DocFkwNir (Rivalr 2 11. 2024
2D+50
8.00
37.60
26
42. 10
42,
42,50
21 60 20.20 22, 50 25.70
20.70
34, 40
64.90
37.30
44
30
20.20
34,30
32.50
33.86
PreWorld (Ours)
10.83
26.35
42.15
34.4%
29.54
33,95
Pre-ttriining
2013 11.81 45.01
26.20
43.32
47.71
24.23 31,29 27.41
27.68 30.62
63.71
37.54
29.36 29:20 34.69
learning rate of 1 x 10 All the hyperparameters À in the loss functions have been set to 1.0. For 3D occupancy prediction task, PreWorld undergoes 6 epochs in self-supervised pre-training stage and 12 epochs in fully-supervised fine-tuning stage. For 4D occupancy forecasting and motion planning task, Pre undergoes 8 epochs in self-supervised pre-training stage and 18 epochs in fully-supervised fine-tuning stage. All experiments are conducted on NVIDIA A100 GPUs.
4.2 RESULTS AND ANALYSIS
Table 2: 4D occupancy forecasting performance on the Oec3D-nuScenes dataset. The latest vision-centric approaches of OccWorld (Zheng et al., 2023) and OccLLaMA (Wei et al., (024) are taken baselines for fair comparison. Aux. Sup. represents auxiliary supervision apart from the ego trajectory. Avg. reprersents the average performance of that in 1s, 2s, and 3s. The best and second- best performances represented by bold and underline respectively.
mioU (%)
ToU (5) f
Method
Aux. Sup.
1s
2s
3s
Avg.
1s
2s
3s
Avg.
OceWorld-S OccWorld-T
None
0.28
0.26
0.24
0.26
5.05
5.01
4.95
5.00
Semantic LiDAR
4.68
3.36
2.63
3.56
9.32
8.23
7.47
8.34
OceWorld-D
3D Oce
11.55
8.10
6.22
8.62
18.90
16.26
14.43
16.53
OccLLaMA-F
10.34
8.66
6.98
8.66
25.81
23.19
19.97
22.99
PreWorld (Ours)
3D Oce
11.69
8.72
6.77
9.06
23.01
20.79
18.84
20.88
+ Pre-training
2D Labels & 3D Oce
12.27
9.24
7.15
9.55
23.62
21.62
19.63
21.62
3D Occupancy Prediction. We first compare the 3D occupancy prediction performance of our World model with the latest methods on the Occ3D-nuScenes dataset. As shown in Ta- ble [ PreWorld achieves an mIoU of 34.69, surpassing the previous state-of-the-arl method, Oc- cFlowNet (Boeder et al. 2024). which has an mIoU of 33.86, as well as other methods using 2D, 3D, or combined supervision. This highlights the effectiveness of PreWorld in perceiving the current scene. Additionally, the proposed 2D pre-training stage boosts performance by 0.74 mloU, with im- provements observed across nearly all categories, both static and dynamic. These results underscore the importance of the proposed 2D pre-training stage for enhanced scene understanding.
In Figure 4 we further compare the qualitative results of Pre World with the latest fully-supervised method SparseOcc (Liu et al. (003) and self-supervised method RenderOce (Pan et al., (024). Ren- derOcc can project scene voxels onto multi-view images to obtain comprehensive supervision from various ray directions, thus capturing abundant geometric and semantic information from 2D labels. However, as shown in the last column, it struggles in predicting unseen regions and understanding the overall scene structure. On the other hand, SparseOcc excels in predicting scene structures. Yet owing to insufficient supervision for small objects and long-tailed objects from 3D occupancy labels, it often encounters information loss when predicting objects like poles and motorcycles, as
7

---

Published as a conference paper at ICLR 2025
Ground Truth
PreWorld (Ours)
SparseOce
RenderOce
dri. sur
car
truck
terrain
vegetation
sidewalk
manmade
pedestrian
bicycle
barrier
cons.
veh
bus
trailer
motorcycle
traffic cone
other flat
Figure 4: Qualitative results of 3D occupancy prediction on the Occ3D-nuScenes validation
set. The holistic structure and fine-grained details of the scene are highlighted by orange boxes and red boxes respectively. Compared with existing fully-supervised methods and self-supervised methods, PreWorld can obtain better scene structure and capture finer local details.
Table 3: Motion planning performance on the Ous3D-nuSeenes dataset. The latest vision- centrric approaches of World (Zheng al.) (023) and OccLLaMA (Wei al., (024) are taken base- lines for fair comparison. represents training and inference with ego state information introduced. The best and second- best performances are represented by bold and underline respectively.
(m)
Collision
Rate (%)
Method
Aux. Sup.
2s
3s
Avg.
1s
2s
3s
Avg.
ST-P3 Hu
2022
Map Box Depth
1.33
2.11
2,90
2.11
0.23
0.62
1.27
0.71
UniAD el al. 202-5b
Map Box & Motion Track & 3D
0.48
0.96
1.65
0.05
0.17
0.71
0.31
VAD Jiang el al. 2013
Map & Box Motion
0.54
1.15
1.98
0.04
0.39
1.17
OceNel [oog al. 2023
Map & Box 3D Oce
1.29
2.13
2.99
2.14
0.21
0.59
1.37
OccWorld-st
None Semantic LiDAR
0.67
1.69
3.13
1.83
0.19
1.28
4.59
2.02
OccWorkd-Tt
0.54
1.36
2.66
1.52
0.12
0.40
1.59
0.70
OccWorld-D!
3D Oce
0.52
1.27
2.41
1.40
0.12
0.40
2.08
0.87
OccLLaMA-FF
3D Oce
0.38
1.07
2.15
1.20
0.06
0.39
1.65
0.70
PreWorld (Ours)
3D Occ
0.44 1.22
2.32
1.34
0.19
0.57
2.65
1.14
+ Pre-ttrining
2D Label & 3D Oce
0.41 1.16
2.32
1.30
0.50
0.88
2.42
1.27
PreWorld (Ours)
3D Oce
0.22 0.31
0.41
0.31
0.36
0.52
0.73
0.54
+ Pre-training
2D Label & Oce
0.22 0.30
0.40
0.31
0.21
0.66
0.71
0.53
shown in the second and the last row. In contrast, our model is initially pre-trained with 2D labels, thereby gaining sufficient understanding of the scene geometry and semantics. In the fine-tuning stage, the model is further optimized using 3D occupancy labels, enabling PreWorld to better predict scene structures. Consequently, PreWorld performs comparably to SparseOce in holistic structure predictions but exhibits clear advantage in predicting fine-grained local details, underscoring the superiority of our training paradigm.
4D Occupancy Forecasting. Table 2 presents the 4D occupancy forecasting performance of Pre- World compared to existing baseline models, Oce World et al.] 2023) and OceLLaMA Wei et al. 2024). When using only 3D occupancy supervision, our method achieves the highest mloU over the future 3-second interval, outperforming the baselines. This demonstrates the effectiveness of our cooperative training approach for both occupancy feature extraction and forecasting modules
8

---

Published as a conference paper at ICLR 2025
in end-to-end manner. Similar to the results for 3D occupancy prediction, incorporating the 2D pre-training stage further improves both mloU and IoU across all future timestamps. This highlights how pre-training provides valuable geometric and semantic auxiliary information from dense 2D image representations. Given that 2D labels are more readily available than costly 3D occupancy annotations, the performance boost from the two-stage training paradigm of PreWorld noteworthy.
Motion Planning. The motion planning results are further compared in Table Without incorpo- rating ego-state information, our model performs comparably to occupancy world models and even some well-designed planning models. When ego-state information is utilized following the same configuration as Wcelorld and OccLLaMA (indicated in gray), our method achieves SOTA perfor- mance with significant improvements, further enhanced by the pre-training stage. Since PreWorld follows a direct training paradigm, taking the original images as input and producing planning re- sults, the impact of ego-state is notably different from that in world model baselines. We attribute this difference to the "shortcut" effect observed in prior work (Zhai et al. 2023 Li al., 2024). We leave the detailed analysis of the relationship between input ego-state, forecasted occupancy, and planning outcomes for future investigation.
4.3 ABLATION STUDY
Ablation study of different data scale utilized in pre-training and fine-tuning stage.
Table
Table 4: Ablation study of different supervision attributes utilized in pre-training stage.
Fine-tuning
Pre-training
mloU (%) f
RGB
Depth
Semantic
mIoU (%) 1
150 Scenes
X
18.66
700 Scenes
25.02 (+6.36)
33.95
X
31.99
34.11 (+0.16)
450 Scenes
700 Scenes
33.37 (+1.38)
V
34.43 (+0.48)
34.69 (+0.74)
X
33.95
700 Scenes
450 Scenes
34.28 (+0.33)
700 Scenes
34.69 (+0.74)
Effectiveness of Pre-training, The effectiveness of different supervision attributes of the 2D pre- training stage is analyzed in this section. As noted earlier, the benefits of pre-training are consistent across both 3D occupancy prediction and 4D occupancy forecasting. Therefore, conserve compu- tational resources, we perform ablation experiments on the 3D occupancy prediction task. Table I shows that RGB, depth, and semantic attributes are progressively added during the pre-training stage, the final mIoU results steadily improve. This demonstrates the effectiveness of the three 2D supervision attributes, with even the simplest RGB attribute providing boost in performance.
To validate the scalability of our approach, we conduct ablation studies the data scale used in both pre-training and fine-tuning stages, shown in Table[5] Firstly, the introduction of the pre-training stage consistently improves performance across all fine-tuning data scales, where larger pre-training scale leads to better results. Secondly, when the fine-tuning dataset is small (150 scenes), which means costly 3D occupancy labels are limited, the pre-training stage significantly boosts the mloU from 18.66 to 25.02. Thirdly, with pre-training, the model fine- tuned on smaller dataset (450 scenes) achieves comparable performance to model without pre- training but fine-tuned on larger dataset (700 scenes), with mloU of 33.37 and 33.95, respectively. These results highlight the effectiveness and scalability of our two-stage training paradigm.
Scalability of Pre-training.
Model Components. We perform ablation studies on the effectiveness of various components in our approach for 4D occupancy forecasting, as shown in Table For comparison, we first present a Copy&Paste baseline, which simply copys the current occupancy prediction results our best 3D occupancy prediction model and calculates the mloU between these results and the ground truth of the future frames. This serves as a lower bound for PreWorld, showcasing the performance of model without any future forecasting capabilities. The results in row 1 and row 2 demonstrate that our proposed forecasting module has effectively equipped the model with future forecasting capabilities. By introducing this straightforward design, the model can produce non-trivial results

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of different components in our approach. The Copy&Paste employs our best model for 3D occupancy prediction task. Ego denotes using ego-state information during training. SSP denotes self--upervised pre-training for model. TS denotes trajectory supervision.
mloU (%r
f
loUi (%)
Method
Ego
SSP
TS
Is
2s
3s
Avg.
1s
2s
3s
Avg.
Past&
9.76
7.37
6.23
7.79
20.44
17.73
16.20
18.12
11.12
7.73
5.89
8,25 (+0.46)
22.91
20.31
17.84
20.35
(+2.23)
11.17
8.54
6.83
8.85 (+1.06)
23.27
20.83
18.51
20.87 (+2.75)
PreWorld
11.69
8.72
6.77
9.06 (+1.27)
23.01
20.79
18.84
20.88 (+2.76)
11.58
9.14
7.34
9.35 (+1.56)
23.27
21.41
19.49
21.39 (+3.27)
12.27
9.24
7.15
9.55 (+1.76)
23.62
21.62
19.63
21.62 (+3.50)
Table 7: Ablation study of joint training. All results in the table are obtained utilizing ego-state information. Traj, 2D and 3D denote ego trajectory, 2D labels and 3D occupancy labels, respectively.
Supervision
L2 (m) J
Collision
Rate (%)
Traj 2D
3D
1s
2s
3s
Avg.
1s
2s
3s
Avg.
0.20
0.34
0.80
0.45
0,50
0.62
0.90
0.67
0.22
0.31
0.41
0.31
0.36
0,52
0.73
0.54
v
0.22
0.30
0.40
0.31
0.21
0.66
0.71
0.53
and achieve significant performance enhancements, particularly evident in the IoU metric. Addition- ally, incorporating ego-state information and employing self-supervised pre-training further enhance both mloU and loU, shown in row 3 and row 5. These findings underscore the importance and contribution of each component in our approach.
Joint Training. We further demonstrate the effectiveness of joint training. As shown in the row 4 and row of Table 6, when simultaneously optimizing both 4D occupancy forecasting and mo- tion planning tasks, the forecasting capabilities of PreWorld are further enhanced. The introduc- tion of trajectory supervision has improved model performance regardless of the utilization of self- supervised pre-training. with an increase from 8.85 and 9.35 to 9.06 and 9.55 in mloU, respectively. Furthermore, joint training has also enhanced the planning capabilities of our model. As shown in T compared to the model supervised solely by ego trajectory, model supervised using both ego trajectory and 3D occupancy labels exhibits significant improvement in both L2 error and collision rates, while the introduction of 2D labels further elevates the model performance. These results collectively demonstrate that jointly training 4D occupancy forecasting and motion planning tasks, as opposed to training them separately, provides additional performance benefits for the model.
CONCLUSION
In this paper, we propose PreWorld, a semi-supervised vision-centric 3D occupancy world model for autonomous driving. We propose novel two-stage training paradigm that allows our method to leverage abundant and casily accessible 2D labels for self-supervised pre-training. In the subsequent fine-tuning stage, the model is further optimized using 3D occupancy labels. Furthermore, we troduce simple yet effective conei-oondi forecasting module, which addresses the challenge faced by existing methods in simultancously optimizing the occupancy network and forecasting module. This module reduces information loss during training, while enabling the model directly forecast future scenes and ego trajectory based on visual inputs. Through extensive experiments, we demonstrate the robustness of World across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks. Particularly, we validate the effectiveness and scalability of our training paradigm, outlining viable path for scalable model training in autonomous driving scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project is supported by National Science and Technology Major Project 00022D0015500 Lenovo Research, and Wuxi Research Institute Applied Technologies, Tsinghua University under Grant 20242001 120.
REFERENCES
Simon Boeder, Fabian Gigengack, and Benjamin Risse. Occflownet: Towards self self-supervised occupancy estimation via differentiable rendering and occupancy flow. arXiv preprint MiXiv:2402.12792, 2024.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631, 2020.
Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3991-4001, 2022.
Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. P Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12547-12556, 2021.
Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 1003/0020, 2018.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamic Shot- ton, and Gianluca Corrado. Gaia-1: generative world model for autonomous driving. arXiv preprint ar -xXx:23399170088 2023a.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junehi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning. In European Confer- ence on Computer Vision, pp. 533-549, Springer, 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17853-17862, 2023b.
Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11799 2021.
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for viseon-based 3d semantic occupancy prediction. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9223-9232, 2023.
Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. Selfoce: Self-supervised vision- based 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19946-19956. 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 1340-8350, 2023.
Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie Yan, Haiyang Sun, et al. Tod3cap: Towards 3d dense captioning in outdoor scenes. In European Conference Computer Vision, pp. 367-384. Springer, 2024.

---

Published as a conference paper at ICLR 2025
Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Dif- ferentiable raycasting for self-super occupancy forecasting. In European Conference on Computer Vision, pp. 353-369 Springer, 2022.
Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1116-1124, 2023.
Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA), pp. 4628-4634. IEEE, 2022a.
Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9087-9098, 2023a.
Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1486-1494, 2023b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's s-eee-view representation from multi-camera images via spatiotempo- ral transformers. In European conference on computer vision, pp. 1-18. Springer, 2022b.
Zhiqi Li, Zhiding Yu, David Austin, Mingsheng Fang, Shiyi Lan, Jan Kautz, and Jose M Alvarez. Fb-occ: 3d occupancy prediction based on forward-backward view transformation. arXiv preprint arXiv:2307.01 2023c.
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14864- 14873, 2024.
Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion- based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934, 2020.
Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang. Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. arXiv preprint arXiv:2312.17116 2023.
Qihang Ma, Xin Tan, Yanyun Qu, Lizhuang Ma, Zhizhong Zhang, and Yuan Xie. Cotr: Compact oc- cupancy transformer for vision-based 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19936-1998 19945, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Hongwei Xie, Bing Wang, Li Liu, and Shanghang Zhang. Renderoce: Vision-centric 3d occupancy prediction with 2d ren- dering supervision. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12404-12411 IEEE, 2024.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Fision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 110- Springer, 2020.
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Search- ing efficient 3d architectures with sparse voxel convolution. In European conference on computer vision, pp. 685-702. Springer, 2020.
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng. Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 8406-8415, 2023.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Con- ference on Robot Learning, pp. 180-191. PMLR, 2022.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14749-14759, 2024.
Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Ocellama: An locupancy-langugge-atiie generative world model for autonomous driving. arXiv preprint wwXiv:2409.03272, 2024.
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 21729-21740, 2023.
Zhaoyang Xia, Youquan Liu, Xin Li, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, and Yu Qiao. Scpnet: Semantic scene completion on point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17642- 11661, 2023.
Donggiangzi Ye, Zixiang Zhou, Weijia Chen, Yufci Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarmultinet: Towards a unified multi-task network for lidar perception. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3231-3240, 2023.
Maosheng Ye, Rui Wan, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Drinet++: Efficient voxel- as-point point cloud segmentation. arXiv preprint arXiv:2111.088
2021.
Jiang-Tian Zhai, Ze Feng. Jinhao Du, Yongqiang Mao, Blang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 10430, 2023.
Chubin Zhang. Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen Lu. Ocenerf: Self-supervised multi- camera occupancy prediction with neural radiance fields. arXiv preprint arXiv:2312.09244 2023a.
Lunjun Zhang, Yuwen Xiong. Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Learn- ing unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01017. 2023b.
Yunpeng Zhang, Zheng Zhu, and Dalong Du. Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1433-9443, 2023c.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning 3d occupancy world model for autonomous driving. arXiv preprint arXiv:2311.16038, 2023.
Yupeng Zheng, Xiang Li, Pengfei Li, Yuhang Zheng, Bu Jin, Chengliang Zhong, Xiaoxiao Long, Hao Zhao, and Qichao Zhang. Monooce: Digging into monocular semantic occupancy prediction. Xrviv preprint XrXiv:2403.0876. 2024.
13

---

Published as a conference paper at ICLR 2025
A MORE EVALUATIONS
A.1 3D OCCUPANCY PREDICTION WITH RAYIOU
To address the inconsistent depth penalty issue within the mloU metric, SparseOce (Liu al. 2023 introduces a novel metric, RayloU, designed to enhance the evaluation of 3D occupancy model performance. In order to demonstrate the robustness of our approach as 3D occupancy model across various metrics, we opt to evaluate PreWorld on the 3D occupancy prediction task using RayloU as metric, and compare the results with existing methods this section.
Table 3D occupancy prediction performance on the Occ3D-nuSeenes dataset. We use RayloU as the evaluation metric (Liu et al., 2023). The best and second-best performances are represented by bold and underline respectively.
Method
RayloU
RayloU Im. 2m, 4m
BEVFormer(4f)
32.4
26.1
32.9
38.0
RenderOce
19.5
13.4
19.6
25.5
SimpleOcc
22.5
17.0
22.7
27.9
BEVDet-Occ (2f)
29.6
23.6
30.0
35.1
BEVDee-Occe-Long (8f)
32.6
26.6
33.1
38.2
OceFlowNet
32.6
25.6
33.3
38.8
FB-OCC (16f)
33.5
26.7
34.1
39.7
SparseOce (8f)
34.0
28.0
34.7
39.4
SparseOce (16f)
36.1
30.2
36.8
41.2
PreWo (Ours) + Pre-training
36.4
30.0
37.2
41.9
38.7
32.5
39.6
44.0
As shown in Table 8] PreWo achieves RayloU of 38.7, outperforming the previous SOTA method SparseOce (Liu et al. 2023) by 2.6 RayloU. Comparing to purely 3D occupancy super- vision, the proposed self-supervised pre-training stage provides a significant boost in RayloU from 36.4 38.7, which reaffirms the effectiveness of our two-stage training paradigm for Pre World. Altogether, Table I and showcases the strong performance of PreWorld across various metrics.
More importantly, the introduction of RayloU explains the reason why our PreWo does not out- perform the baseline in certain categories. As shown in Table 1 these situations are predominantly focused on large static categories. For instance, in categories like manmade and sidewalk, the perfor- mance of Pre World is surpassed by RenderOce (Pan et al. 2024). SparseOcc points out that common practice in mloU computation involves the utilization of visible masks, which only accounts for vox- els within the visible region, without penalizing predictions outside this area. Consequently, many models can achieve higher mloU scores by predicting thicker surfaces for large static categories. As demonstrated in the last column of Figure RenderOce, despite lacking an understanding of the overall scene structure, manages to attain higher scores in these categories through this strategy.
On the contrary, due to RayloU considering the distance between voxels and the ego vehicle during computation, the model cannot gain an advantage by predicting thicker surfaces under this evalua- tion metric. Therefore, we believe that RayloU performance in predicting large static categories. As shown in Table 8, when using RayloU as the evaluation metric. the scores of both RenderOce and OccFlowNet (Boeder et al.) 2024) have de- creased. While OccFlowNet outperforms SparseOce in the mloU metric with 33.86 over 30.90, its performance notably lags behind SparseOce in terms of RayloU. These results indicate that the performance of our World is not inferior in categories; rather, model tends to generate more reasonable predictions, which can be reflected in the RayloU metric.
more reasonable metric for comparing model
Likewise, we can explain why pre-training leads to decline in model performance in certain cat- egories. It can be observed that these instances also primarily focus on large static categories. For example, after pre-training there is a significant mloU performance decline in the driveable surface category. Based on the previous analysis, we showcase the RayloU performance for the models on large static categories with and without pre-training.
14

---

Published as a conference paper at ICLR 2025
Table 9: Detailed 3D occupancy prediction performance of the large static categories on the OccSDensSeenes dataset. We use RayloU as the evaluation metric (Liu et al. 2023). GT represents the type of labels used during training. The best performances are represented by bold.
Dri. Sur
other fltt
sidewalk
manmade
vegetation
barrier
terrain
Metric
Method
GT
RayloU
PreWorld
3D
30.0
39.4
56.4
24.2
25.5
23.8
32.9
23.2
RayloU Im
+ Pre-training
2D+3D
32.5
40.1
57.8
29.8
27.2
27.3
35.1
25.9
PreWorld
3D
37.2
64.4
29.2
30.7
31.3
43.4
36.0
RayloU2
+ Pre-training 2D+3D
39.6
45.4
65.9
34.3
32.7
34,6
44.7
37.8
PreWorld
3D
41.9
46.6
72.4
33.2
35.6
38.1
49.7
47.3
RayloU
+ Pre-training
2D+3D
44.0
47.4
74.3
38.3
37.7
41.1
50.5
47.2
As shown Table 9 the pre-trained model surpasses the model without pre-training on almost all large static categories across all thresholds. The results under the RayloU metric indicate that pre- training steers the model towards predicting more plausible scene structures, rather than leading to performance decline. In conclusion, helieve that the results under RayloU metric validate the effectiveness of pre-training and better showcase the robust prediction capabilities of our Pre World.
A.2 HOW PRE-TRAINING WORKS?
Due to time constraints, when conducting experiments on smaller datasets in Table models fine- tuned on 150 scenes and 450 scenes are trained for 24 and 18 epochs, respectively, while the model on the full dataset is trained for 12 epochs. Considering the ratio of data reduction to extended training time, we believe that we not allocate sufficient additional training time for experiments on the smaller datasets. Therefore in this section, to delve into how pre-training benefits the model, we extend the training duration across various settings to obtain more comprehensive experimental results, as presented in Table 10
Table 10: The extended ablation study of different data scale utilized in pre-training and fine- tuning stage. The best performances are represented by bold.
Epoch
Fine-tuning
Pre-training
12
18
24
36
48
11.18
13.85
18.66
29.30
30.26
30.00
150 Scenes
700 Scenes
13.01
21.83
25.02
31.65
31.56
30.98
25.54
31.99
32.89
33.32
450 Scenes
700 Scenes
29.52
33.37
34.19
34.08
X
33.95
33.99
700 Scenes
450 Scenes
34.28
34.15
700 Scenes
34.69
34.89
As shown in the results, we believe that pre-training has benefited the model in key aspects: on one hand, pre- pre-training accelerates the convergence of the model; on the other hand, pre-training con- tinues enhance the model performance after convergence, thereby improving the data efficiency. Taking models fine-tuned on 150 scenes as an example, it can be observed that during the first 24 epochs, employing pre-training accelerates the convergence. Subsequently, both models have con- verged, with the pre-trained model still maintaining an advantage in prediction performance.
Furthermore, it can be observed that pre-training leads to a 0.87 mloU improvement for the model fine-tuned on 450 scenes, while it results in 0.90 mloU improvement for the model fine-tuned on 700 scenes. We believe this situation is still related to the reasons analyzed in Section that is, for large static categories, existing evaluation metric does not adequately reflect the actual performance of the model. Therefore, we have detailed the corresponding mloU for large static categories and small objects in Table T1
15

---

Published as a conference paper at ICLR 2025
Table 11:
Detailed 3D occupancy prediction performance of different data scale utilized in pre-
training and fine-tuning stage.
mloU Large Static
Fine-tuning
Pre-training
Overall
Small
150 Scenes
X
30.26
35.82
24.08
700 Scenes
31.65 (+1.39)
37.18 (+1.36)
25.50 (+1.42)
450 Scenes
X
33.32
38.40
26.29
700 Scenes
34.19 (+0.87)
39.04 (+0.64)
27.44 (+1.15)
700 Scenes
x
33,99
39.83
26.98
700 Scenes
34.89 (+0.90)
40.72 (+0.89)
27.96 (+0.98)
It can be observed that the mloU for large categories does not always effectively reflect the perfor- mance improvement of the model. For the model fine-tuned with 450 scenes, pre-training leads to 0.64 increase in mloU for large categories, while the model fine-tuned with 700 scenes sees an in- crease of 0.89. In contrast, the increase mIoU for small objects can better reflect the effectiveness of pre-training, aligning with the expectations: 2D pre-training yields more significant performance improvements for smaller 3D fine-tuning datasets. order better showcase the effectiveness of pre-training, we use RayloU as the evaluation metric, and the results obtained are follows:
Table 12: Detailed 3D occupancy prediction performance of different data scale utilized in pre-training and fine-tuning stage. We use RayloU as the evaluation metric (Liu et al. 2023).
Large Static RayloU
Fine-tuning
Pre-training
RayloU
Overall
Im
2m
4m
150 Scenes
X
29.5
32,5
26.2
32.9
38.5
700 Scenes
33.4 (+3.9)
36.6
30.2
37.0
42.5
450 Scenes
X
35.1
37.9
31.5
38.3
44.0
700 Scenes
37.8 (+2.7)
40.2 (+2.3)
33.1
40.6
46.8
700 Scenes
X 700 Scenes
36.8
39.5
32.4
39.9
46.2
39.0 (42.2)
(+1.9)
34.7
41.9
47.6
As shown in Table 12, when using RayloU as the evaluation metric, the improvements in overall RayloU and RayloU for large categories follow a similar trend, indicating that as the scale of the 3D fine-tuning dataset increases, the benefits 2D pre-training do indeed gradually diminish.
A.3 SELE-SUPERVISED 4D OCCUPANCY FORECASTING AND MOTION PLANNING
Instead of generating occupancy predictions through the occupancy head H. we support an alter- native approach by utilizing the attribute projection head P. Specially, by setting a threshold value т for the 3D volume density field ā of the scene, we can determinate whether a voxel is occupied. Subsequently, the semantic occupancy of the voxel un can be formulated as:
Ý (ux) argmax(5(rx)). ( 2T,
(10)
where ş denotes the semantic field of the scene, and we regard Uk as non-occupied if (v)) <..
In this manner, words, PreWorld is capable engaging in self-supervised tasks as well. Therefore, to validate its performance as self-supervised 3D occupancy world model, compare it against state-of-the- art supervuper methods on both 4D occupancy forecasting and motion planning tasks on the dataset (Tian et al. 2024), denoting Prelor...S.
also obtain occupancy predictions during the pre-training stage. In other
4D Occupancy Forecasting. Table 13 presents the 4D occupancy forecasting performance of PreWorld-S compared to previous self self-supervised approach of OceWorld (Zheng et al. 2023). In comparison to WccWordd.S, our approach yields significant outcomes. The loU over the future 3- second interval nearly doubles, while the average future mloU demonstrates an remarkable increase of over 1300%, soaring from 0.26 to 3.78. These results highlight the superiority of method
16

---

Published as a conference paper at ICLR 2025
in self-supervised learning and open up possibilities for future research the architecture of self-supervised 3D occupancy world models.
Table 13: Self-supervised 4D occupancy forecasting performance on the Occ3D-nuSceme dataset. We take the self-supervised vision-centric approach of OccWorld (Zheng al., 2023) as baseline for fair comparison. Aux. Sup. represents auxiliary supervision apart from the ego tra- jectory, Avg. reprersents the average performance of that in 1s, 2s, and 3s. The best and second-best performances are represented by bold and underline respectively.
mIoU (%) f
IoU (%)
f 3s
Method
Aux. Sup.
1s
2s
38
Avg.
1s
2s
Avg.
OccWorld-S
None
0.28
0.26
0.24
0.26
5.05
5.01
4.95
5.00
PreWorld-S (Ours)
2D Labels
4.36
3.72
3.27
3.78
9.49
9.17
8.90
9.19
Table 14: Self-supervised motion planning performance on the OcceD-nuScenes dataset. We take the self-supervvsed vision-centie approach of OccWorld (Zheng al., 2023) baseline for fair comparison. represents training and inference with ego state information introduced. The bestperformances are represented by bold.
L2(m) (%)
Collision Rate (%) |
Method
Aux. Sup.
1s
2s
38
Avg.
1s
2s
3s
Avg.
OccWorld-S
None
0.67
1.69
3.13
1.83
0.19
1.28
4.59
2.02
PreWorld-S (Ours)
Labels
0.66
1.49
2.60
1.58
0.57
1.26
2.23
1.35
PreWorid-S (Ours)
2D Labels
0.20
0.61
1.57
0.79
0.57
0.64
1.42
0.88
Motion Planning. As illustrated in Table 14 PreWorld-S significantly surpasses OccWorld-S on both metrics even without the incorporation ego-state information. When ego-state information introduced (indicated in gray), the performance of our self-supervised approach has received no- table enhancement, yielding results comparable or outperforming those fully-supervised methods such OccWorld-D. These findings once again demonstrate the effectiveness of our approach.
B MORE VISUALIZATIONS
We provide additional visualized comparison in this section.
Fig 5 shows more qualitative results of 3D occupancy prediction task compared with the latest fully-supervised method SparseOcc (Liu et al. (023) and sulf-supervised method RenderOcc (Pan et al. 2024), further substantiating the robustness of our PreWorld model and the effectiveness of our novel two-stage training paradigm. The red boxes highlight fine- mne-grained details of the 3D occupancy predictions and the ground truth, while the orange boxes mark holistic structure of an area within the preserving
scene. Compared prior approaches, PreWorld demonstrates superior performance RenderOcc the structural information of the scene and capturing fine-grained details. In contrast, struggles with comprehending the scene structure accurately and exhibits inaccurate predictions for unsupervised occluded regions. SparseOce, on the other hand, fails to effectively predict small objects like poles and long-tailed objects like construction vehicles, resulting in detail loss. These findings are consistent with the observations in the main text.
In Fig we further provide a detailed showcase of the prediction results for both visible and oc- cluded regions. Consistent with the quantitative analysis in Section A. it can be observed that RenderOcc (Pan et al. 2024) tends to predict thicker surfaces for large static categories. However, while this approach may lead to higher mloU scores, its predictions for occluded regions are chaotic, indicating a lack of true understanding of the scene structure. On the contrary, our PreWorld makes more cautious predictions for occluded regions, demonstrating more comprehensive understanding of the holistic scene structure.
17

---

Published as a conference paper at ICLR 2025
Ground Truth
PreWorld (Ours)
SparseOce
RenderOce
dri.
car
truck
terrain
vegetation
sidewalk
manmade
pedestrion
bicycle
barrier
cons.
veh
bus
trailer
motorcycle
traffic cone
other flat
Figure 5: More qualitative results of 3D occupancy prediction on the OcceD-nuScenes valida- tion set. The holistic structure and fine-grained details of the scene are highlighted by orange boxes and red boxes respectively.
Ground Truth
PreWorld (Ours)
RenderOec
dri. sur
car
truck
Terrain
vegetation
sidewalk
marmade
pedestrian
bicycle
barrier
cons.
weh
bus
traller
motorcycle
traffic cone
other flat
Figure 6: More qualitative results of 3D occupancy prediction on the Occ3D-nuScenes vali- dation set. The shaded area represents occluded regions where the voxels are not included in the evaluation. In contrast to RenderOcc, our Pre World makes more cautious predictions for occluded regions, tending to preserve the overall structure of the scene.
18


Reviews:
[{"review_id": "TNH6yXGKPU", "reviewer": "Reviewer_z3jn", "summary": "This paper presents a semi-supervised vision-centric 3D occupancy world mdoel, PreWorld. It leverages 2D labels through two-stage training, self-supervised pretraining and supervised finetuning. In the pre-training stage, the method utilizes an attribute projection head to enable temporal supervision from 2D labels. The method also includes a state-conditioned forecasting module.", "questions": "1. \"Our Copy&Paste\"  in Tab.6 is confusing for me. What is the setting of row 2 in Tab. 6 and what is difference between row 2 and copy & paste?\n2. How to correctly interrept the result in Tab.3? The method can achieve a comparable result even without ego-state information? Can you elaborate more about the result for ours with ego-state information.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The self-supervised pre-training is simple and intuitive. The method projects predicted volume using attribute projection head, then use future 2D frames as supervision signal. It is effective by upgrading training strategy without introducting extra modules for inference.\n2. The experiments are extensive to show superior performance in occupancy prediction and forecasting tasks.\n3. The paper writing is clear, it is easy to read and follow.", "weaknesses": "1. The improvement with and with-out pre-training seems not very significant. The module sometimes hurt performance (In tab.1). \n2. The ego state-conditioned module lacks of sufficient analysis and comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29JHl0tDKL", "reviewer": "Reviewer_GcMx", "summary": "The paper presents \"PreWorld,\" a semi-supervised model aimed at improving 3D occupancy prediction and 4D forecasting for autonomous driving. It leverages both 2D and 3D data in a two-stage training approach, including a self-supervised pre-training phase using 2D labels and a fully-supervised fine-tuning phase with 3D labels. The authors introduce a state-conditioned forecasting module that directly predicts future occupancy and vehicle trajectory. Tests on the Occ3D-nuScenes dataset show PreWorld achieves competitive results.", "questions": "The proposed two stages can be trained jointly, why train them in two stages? Can the authors provide any motivation on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provides a thorough evaluation of different methods on the Occ3D-nuScenes dataset. The experiments shows that the proposed model and training method can achieve competitive results across 3D occupancy prediction, 4D forecasting, and motion planning tasks.", "weaknesses": "1. The contributions are largely incremental. While the state-conditioned forecasting module (Section 3.2) enables volume feature preservation for rendering, this approach aligns closely with existing practices in similar tasks​ [1][2]. The self-supervision in Section 3.3 extends volume rendering to 4D but follows typical methodologies in related work​ [3].\n2. The figures are difficult to interpret due to insufficient captions. For instance, Figures 2 and 3 lack details, making it unclear how the components operate or interact. The difference between solid and dashed arrows in Figures 1, 2, and 3 is not explained. Figure 2 leaves confusion about whether the two stages are jointly or sequentially trained.\n3. The authors claim 2D rendering as self-supervision, but the best result comes from utilizing depth and semantic labels, which is not self-supervision, and invalidates \"semi-supervision\" in the title.\n4. The performance improvement is minimal. For example, the mIoU improvment for 3d occupancy prediction is only about 2% compared to OccFlowNet. And there is no significant improvement in other experiments and metrics\n\n[1] Khurana, et al. \"Point cloud forecasting as a proxy for 4d occupancy forecasting.\" CVPR 2023.\n\n[2] Liu, et al. \"Lidar-based 4d occupancy completion and forecasting.\" IROS 2024.\n\n[3] Yang, et al. \"Emernerf: Emergent spatial-temporal scene decomposition via self-supervision.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VW8EF8VlSB", "reviewer": "Reviewer_fN92", "summary": "The paper propose a semi-supervised method called Pre-World for occupancy forecasting problem. Pre-world first train the occupancy network in a self-supervised manner with the pre-training goal to render future image observation. Then Pre-world train the occupancy network with temporal occupancy labels to further improve the performance. Experiment results on NuScenes dataset show that the proposed method to some extend improve the performance compared to training from scratch.", "questions": "1)  Is it possible for the authors to provide experiments on training longer for both randomly initialization and pre-trained ones until they both converge in the downstream task?\n\n2) Could the authors provide analysis on the detailed category performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow. The proposed Preworld is able to utilize unlabeled data for pre-training and improve the performance of the downstream tasks. The experiments are extensive.", "weaknesses": "1) The performance improvement seems not that significant. Generally, there are two kinds of improvement that pre-training can bring. The first one is to accelerate convergence in the downstream task. The second one is to improve sample efficiency in the downstream task, that is to improve the performance at convergence. The reviewer are concerned about whether the improvement presented in the experiment part is related to the acceleration of convergence.\n\n2) It can be found in Table 1 that after pre-training, ious of some categories like driving surface and manmade drop. Also, compared to previous method, performance on some categories including manmade and vegetation are lower. But there is no analysis about these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tkpevUsm8c", "reviewer": "Reviewer_BYPK", "summary": "The paper proposes PreWorld, a semi-supervised, vision-centric 3D occupancy model that leverages 2D labels through a novel two-stage training approach. In the self-supervised pre-training stage, an attribute projection head is used to enable temporal supervision from 2D labels, while the fully-supervised fine-tuning stage introduces a state-conditioned forecasting module to directly predict future occupancy and ego trajectory.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a comprehensive approach, from the overall architectural design to thorough experimental validation:\n\na). The paper is well-written and easy to follow, presenting a method that is both well-motivated and clearly explained.\n\nb). The proposed state-conditioned forecasting module is interesting.", "weaknesses": "a). Although PreWorld leverages 2D labels to reduce the cost of 3D annotations, the paper only utilizes the nuScenes dataset for training, which does not fully demonstrate the method’s advantages. The semi-supervised approach should be highlighted by showcasing its benefits in terms of data efficiency.\n\nb). Different tasks in occupancy predictions (4D occupancy forecasting, motion planning...) may have interdependent effects, where optimizing one module could potentially negatively impact others. The paper should provide a clear discussion on how to achieve a balance among these tasks and address potential interactions between the modules.\n\nc). PreWorld’s two-stage training process may increase model complexity. This could lead to higher computational demands, potentially affecting the feasibility of real-time application in autonomous driving. The paper should further validate the feasibility of this method in terms of real-time inference performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)