ODG: Occupancy Prediction Using Dual Gaussians
Authors: Yunxiao Shi
Yinhao Zhu
Hong Cai
Shizhong Han
Jisoo Jeong
Amin Ansari
Fatih Porikli
Venue: NeurIPS 2025 poster ()
Keywords: occupancy prediction
Gaussian splatting
autonomous driving
Primary area: applications

Abstract: Occupancy prediction infers fine-grained 3D geometry and semantics from camera images of the surrounding environment, making it a critical perception task for autonomous driving. Existing methods either adopt dense grids as scene representation which is difficult to scale to high resolution, or learn the entire scene using a single set of sparse queries, which is insufficient to handle the various object characteristics. In this paper, we present ODG, a hierarchical dual sparse Gaussian representation to effectively capture complex scene dynamics. Building upon the observation that driving scenes can be universally decomposed into static and dynamic counterparts, we define dual Gaussian queries to better model the diverse scene objects. We utilize a hierarchical Gaussian transformer to predict the occupied voxel centers and semantic classes along with the Gaussian parameters. Leveraging the real-time rendering capability of 3D Gaussian Splatting, we also impose rendering supervision with available depth and semantic map annotations injecting pixel-level alignment to boost occupancy learning. Extensive experiments on the Occ3D-nuScenes and Occ3D-Waymo benchmarks demonstrate our proposed method sets new state-of-the-art results while maintaining low inference cost.

ODG: Occupancy Prediction Using Dual Gaussians
Yunxiao Shit
Yinhao Zhu
Shizhong Han
Jisoo Jeong"
Amin Ansari
Hong Cail
Fatih Porikli
Quualcomm Al Research" $Qualcomm Technologies, Inc
Youxshh,yinhaae shizhan, jisoj amina, htw.aaaaaaaaa.roallleeeeee. www.:t/w.uaacccmm..ooommm
Abstract
Occupancy prediction infers fine-grained 3D geometry and semantics from camera images of the surrounding environment, making it critical perception task for autonomous driving. Existing methods either adopt dense grids as scene repre- sentation which is difficult to scale to high resolution, learn the entire scene using a single set of sparse queries, which is insufficient to handle the various object characteristics. In this paper, present ODG, hierarchical dual sparse Gaussian representation to effectively capture complex scene dynamics. Building upon the observation that driving scenes can be universally decomposed into static and dynamic counterparts, we define dual Gaussian queries to better model the diverse scene objects. We utilize hierarchical Gaussian transformer predict the occupied voxel centers and semantic classes along with the Gaussian parameters. Leveraging the real-time rendering capability of 3D Gaussian Splatting, we also impose rendering supervision with available depth and semantic map annotations injecting pixel-level alignment boost occupancy learning. Extensive experiments on the Dec3D-nuScene and Occ3D- --Wyoooo benchmarks demonstrate our pro- posed method sets new state-of-the-ee results while maintaining low inference cost.
1 Introduction
3D spatial understanding forms the foundation of autonomous systems such self-driving cars. 3D object detection [57, 23, 34, 33] has been the primary task that outputs bounding boxes to capture different entities in the scene. Concise as box representation is, it cannot deal with out-of-vocabulart or irregularly-shan shaped objects (e.g. trash can on the side of road, excavator with arms deployed) which is critical for driving safety. Calling for unified 3D representation that handle such cases, 3D occupancy adopts a voxel grid to partition the scene and jointly predicts the occupancy state and semantic labels of each voxel, which provides critical information for downstream planning [22].
Previous approaches have explored regular grids like voxel [58, 63), BEV ground plane [31, 61], and tri-perspective plane [25] to represent the scene followed by dense classification. Such approaches do not take into account the fact that most the voxels are empty [3, 50] and allocate equal resource for each one, which inevitably results in severe inefficiency. To overcome such drawbacks, another line of research [47, 53] formulate the task of 3D occupancy as direct set prediction, effectively predicting 3D occupancy set of sparse points from sparse latent vectors. Such sparse representation avoids spending resource to model empty regions and improves scalability. Recent works [27, 4, 10] utilize 3D Gaussians instead of points which have more spatial context, and also leverage the real-time rendering of 3D Gaussian Splatting (3DGS) [28, 29] to either complement learning from occupancy ground-truth instead directly learn occupancy from 2D labels [59, 21] without relying on 3D annotations. Such sparse query-based methods have shown great promises for occupancy prediction.
"Qualcomm Al Research is an initiative of Qualcomm Technologies, Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

But Is it realty sensible to rely on a single set of queries to predict everything within driving scene? We observe that driving scenes can be universally decomposed into static background (e.g. roads, buildings, etc.) and dynamic agents (e.g. vehicles, pedestrians, etc.), and argue that the dynamic agents should have resources dedicated to them given their importance occupancy prediction. Hence, we introduce dual Gaussian query design, consisting of two sets of Gaussian queries, to model the static and dynamic parts of the scene for better capturing complex scene dynamics. To establish communication between queries, we propose simple and effective attention scheme to achieve this. Meanwhile for Gaussian-based representation, it is critical to have a sufficient number of Gaussians in order have enough capacity represent the scene. But existing methods [27, 4] utilize a single transformer which can only handle a smaller number of Gaussians. Therefore we propose to predict Gaussians in a coarse-to toorse-t--fin manner with series of transformer layers, enabling the use of a much larger number of Gaussians and thereby increasing model capacity. In addition to learning from 3D occupancy ground-truth, generate the depth and semantic maps for each camera view at every stage, which are supervised by the corresponding 2D labels and improves model consistency. Our contributions can be summarized as follows:
leverage the efficient rendering capabilities of 3DGS to
We propose novel dual-query architecture comprising two distinct sets of Gaussian queries to separately model the static and dynamic parts of the scene. A cross query attention is also introduced to establish effective interaction between queries, enhancing 3D occupancy prediction.
Dual Gaussian Query Design:
Hierarchical Coarse-to-Fing Refinement: We refine the Gaussian properties in a hierar- chical coarse-to-fine to-ffiee fashion, allowing a much larger number of 3D Gaussians to be utilized, effectively increasing model capacity and expressiveness.
Multi-Stage Rendering Supervision: We enforce depth and semantic consistency through multi-stage real-time rendering using 3D Gaussian Splatting. This allows supervision from 2D labels across views, improving spatial coherence and prediction accuracy.
State-of-the-Art Performance: Extensive experiments on the Occ3D [50] benchmark demonstrates that our proposed method sets new state-of-the-art results in 3D occupancy prediction, while maintaining competitive inference runtime.
2 Related Works
2.1 3D Occupancy Prediction
3D occupancy prediction partitions the 3D space with a voxel grid and simultaneously estimates the occupancy state and semantic label of each voxel. Earlier methods [7, 31, 25, 58, 63] utilize dense grids (e.g. voxel, BEV) as scene representation and learns 3D occupancy from ground-truth data. Given the high cost [56] of curating occupancy annotations, [62, 41, 5, 44] advocates the idea of using 2D labels projected from LiDAR or generated by vision foundation models [59, 21] to train 3D occupancy networks eliminating reliance on direct 3D occupancy annotations. Several works [8, 62, 24, 36] also explore self-supervised learning based on the photometric consistency between neighboring frames to learn 3D occupancy. Mcanwhile, multiple 3D occupancy benchmarks [3, 58, 50, 51, 13, 56, 64] have been created based on existing datasets [17, 16, 6, 45].
2.2 Set Prediction with Transformers
Direct set prediction with transformers [52, 14] for perception tasks was first introduced in DETR [9]. Follow-up works like [57, 37, 34, 33) further advanced this technique for 3D object detection and obtained impressive results. Witnessing such success, [47, 53] adapted such set prediction paradigm for 3D occupancy prediction and demonstrated competitive alternative to the common pipeline of explicit space modeling (e.g.. dense grids) followed by classification. We adopt such paradigm but different from previous works [34, 47, 53] that only use one set of queries assuming scene homogeneity, we define two distinctive sets of queries to better handle the dynamic agents in the scene.
2.3 Neural Rendering and 3D Gaussian Splatting Neural rendering [48] aims learn 3D representations from 2D data and experienced significant growth over the past few years [49]. Neural Radiance Fields (NeRF) [40, 1, 2] is one such technique

---

State Qlaeten
Stetic Flwenc
such Jayers
Static 3D Coussians
Egn Mittion Comp
Feature Bampling
Cross Amamiion
Self Adurnon
Statt QUITTe
30 Gaussiars Joytmmic stute!
Duad- Churrry
Dynamic
Gau-skar Deccoske
3D points W class scores
Seupervbaion
-00 occ.
Mu comera Videos
Dynamiic
Moxion
CM
Ego Motion
Comp.
Feature Sempling
Ciroes Attentio
Aftersion
Seel
Decoder
class scores
DOX
Dynamic
Dynamic Branch
Overview of proposed ODG, where we model the dynamic and static elements of the scene with two separate sets of Gaussian queries. Dual-query attention aggregates information across dynamic and static queries. These queries then decoded into 3D Gaussians, 3D points, well as 3D bounding boxes (from dynamic queries only), which are supervised by ground-trudi depth semantic maps through rendering, ground -rund- 3D occupancy, and ground-truth bounding boxes, respectively.
Figure 1:
and has achieved impressive results. However, NeRI-based methods suffer from slow training and high memory cost. Recently, 3D Gaussian Splatting (3DGS) [28, 29] emerged rasterization pipeline which drastically reduced rendering time while also achieving incredible rendering quality, Earlier methods [60, 19] focused on improving per-scene rendering quality. Later generalizable
3DGS [11, 12, 46] were proposed to enable reconstructing large-scale scenes in feed-forward
manner. Given that 3D occupancy also aims to reconstruct the scene, latest research begin exploring
3D Gaussians for occupancy prediction [26, 27, 10, 4]. GaussianFormer [26] and GaussRender [10] voxelizes 3D Gaussians which incurs high compute cost. GaussTR [27] and GaussianFlow
[4]
employs a single transformer to predict Gaussian parameters from sparse queries, which can only handle small number of Gaussians limiting model capability. In contrast, our method predicts Gaussians in hierarchical coarse-to-fing fashion allowing much larger number of Gaussians, effectively resulting in higher learning capacity.
3 Method
In this section, we present our proposed 3D occupancy approach, ODG, in which we adopt dual Gaussian query design to capture the respective dynamic and static elements in the scene, as discussed in Sec. 3.2. We model object motions for the dynamic Gaussians (Sec. 3.3) and leverage attention enable feature interaction between the dual queries (Sec. 3.4). Finally, we describe the training objectives in Sec. 3.5. Fig. provides an overview of ODG, Next, we first provide quick recap the problem of 3D occupancy prediction.
3.1 Problem Definition
Given an ego-vehicle at time T', the task of 3D occupancy prediction takes N, multi-camera images (with & x N, optional history frames where k 2 0). I (1:) Tho=1 and corresponding camera parameters input, and predicts 3D semantic voxel grid O fci, c2. )c)
cc) H x x where H, W, Z denotes the grid resolution and C is the number of classes. Formally, 3D occupancy prediction be defined as
O == G(V). == =
where F()) consists of an image backbone that extract multi-camert features and transforms them into scene representation V. G(.) is another neural network that maps V to final occupancy predictions. Dense grids (e.g. voxel grid [58, 63], BEV/TPV grid [31, 25)) common choices for V which are difficult to scale and cannot differentiate different object scales. Inspired by successes in object detection [9, 57], more recent works [35, 53] cast 3D occupancy prediction direct prediction with transformers, without explicitly building V. At its core, a set of learnable queries Q and 3D points P are initialized regress point locations and corresponding semantic class scores C simultaneously. Such paradigm can be described as
min DP, + (C, P.C
(2)
3

---

where (Pg. g: Cg) is the constructed ground-truth set for occupied voxels with P CC = Vi being the number of occupied voxels. Each Pg E P, represents the voxel center coordinate and cg C the class label. Correspondingly, {P, CJ denotes predictions where each Pi E P and E C is predicted by query 9 E Q. Pp()) and (a(+) are geometric and semantic distances respectively.
3.2 Dual Dynamic and Static Gaussian Queries
Instead of predicting occupancy as point sets from one set of learnable queries as in Eq. 2, we define dual Gaussian queries as shown in Fig. which are detailed below.
Formally, a standard 3D Gaussian g is parameterized by set of properties
g (,,
where ja E IR' is the mean, E IR is the scale, IR is the quaternion and [0, 1] is the opacity. We define two sets of Gaussian queries for static and dynamic objects, each query contains a set of Gaussians and feature vector
GO
{g] (g),)) Q" = (q:) o E RRM G = ( Q" [/ ERN,
(g)
where S, D is the number of static and dynamic Gaussian queries, and N, M is the embedding dimension of query features, K₂ is the number of Gaussians per query in stage É. Here we set N = M for simplicity but they can be different.
For dynamic query g" whose intended purpose is to model the dynamic agents in the scene, what additional information should they have on top of Eq. 3 that can effectively accomplish this? We observe that the box representation from 3D object detection [57, 34, 33] is a good candidate that tailored to capture dynamic objects. Therefore we expand the standard definition in Eq. 3 and define our dynamic Gaussian queries g" as
g" Bb b [1, (,,.h,& h, Uxy Vy, Uz].
(5)
where I, w. are the spatial dimensions of the 3D box and 0 its rotation. Ux Bys Us is the velocity vector. Given for driving scenes, motion along the z-axis is negligible hence we set Uy = 0 and treat it as constant.
We initialize Gaussian means 8yy and g% from 1/00, 1] for both types of queries and the box attributes b for g" but leave the rest of the Gaussian properties uninitialized. Instead we predict them using separate MLPs with their means and query features
{s r 0²) (G Q*),
(G Q*).
(6)
where & denotes respective MLPs. To enable an sufficient number of Gaussians, unlike previous methods [27, 4] which only utilizes a single transformer that maintains the same number of points over layers, we predict Gaussians in a hierarchical coarse-to-fin fashion through a series of transformer- based layers T akin to [34, 53]
d.b C.d. G C Q = (e( jusd 12 t,, t1),
G O C = Te(G. 1- Q2 1221;:000 T,, +r, $a),
(8)
where 1 S L with L being the number of layers used. For each layer Tr, it takes as input static Gaussian means G G and query features Q from the previous layer, and predict the current static Gaussian means G :.. E Sx Kxx3 and corresponding class scores C Sxxxx where K < K2 is the number of Gaussians per query at each stage which enables coarse-to-fine prediction. For dynamic Gaussians, besides decoding Gd DDx and C% DDxxxx
also parallelly predict box attributes and class scores G E Dx 10 and dd.b E DxC as shown in Eq. 7. We note that we do not perform coarse-to- to-see- prediction for boxes given the relatively small number of boxes that are typically present in the scene. The rest of the Gaussian properties is predicted by separate MLPs & as defined in Eqs. 6.
We aim to reduce the spatial artifacts of 3D occupancy through projective constraints. To this end, we render depth and semantic maps of each camera view at current time (keyframe) which effectively

---

enforces geometric and semantic consistency. We leverage 3D Gaussian splatting [28, 29] which offers real-time rendering:
G D3 = > Toodi S, = Tiac
G
(9)
where p indicates a certain pixel location and G D S the number of Gaussians. d; is the opacity and T, is accumulated transmittence. C; E [0, 1| is the "semantic probability" of the i-th Gaussian and dę is the depth of the i-th Gaussian. Dp. Sp are the rendered depth and semantic values at location p. We refer readers to [28] for further details regarding 3D Gaussian splatting.
3.3 Motion Modeling of Dynamic Gaussians
For each Gaussian g with g., representing its position in 3D space, it first samples 3D points following [34, 53]. Then we sample image features by projecting sampled 3D points onto each image feature plane using available camera extrinsics and intrinsics, and aggregate corresponding image features. Under the setting of having history frames, it is critical to move the Gaussians according its motion to sample features correctly. For driving scenes we consider two types of motions: 1) ego-motion which is the motion of the ego-vehicle as it navigates across the scene, which effectively camera motion, and 2) object-motion which describes how the dynamic agents themselves move in the scene.
For static Gaussians, compensating ego-motion enough. For the motion of dynamic Gaussians, adopt the approach of approximating instantaneous velocity with average velocity in a short-time window as in [34], and warp sampling points to previous timestamps t-, using the velocity vector [Va, 0y] from the dynamic Gaussian queries (Eq. 5)
E-i = Z0 - Uy (lo - (-:),
(10)
V- y/o Vy (to - t-1),
(11)
where to is the current timestamp. We note that on the z-axis, we assume there is no velocity give the nature of driving. Then we warp each dynamic Gaussian using camera motion
2..., P
where P-,, Po are the ego poses at timestamp t- and to.
3.4 Attention across Dynamic and Static Queries
To enable effective interaction between dynamic Gaussian queries Qº and static Gaussian queries Q", we first concatenate their features representations. We then apply Self-Attention [52] the combined features, allowing for rich information exchange cross both query types. We refer to this mechanism as Dynamic-am and-Static (DaS) Attention. This approach not only facilitates self-attention within the dynamic and static queries individually but also enables bidirectional cross-attention between them, enhancing the integration of dynamic and static information.
Q Self-AAtention(CCnaatnaaee Q" Q")),
(12)
Q Q.D, Q' = Qp.n+8
(13)
where: here is the index operator.
3.5 Loss Functions
We supervise predicted Gaussian means G.y. and corresponding class scores C with Chamfer distance [15] and focal loss [32]
I
Locc = CD PO)
+ Σ CD( PO)) FocalLoss((C, C
(14)
E=1
where CD(G), PO) encourages initial Gaussian means to capture global pattern of underlying data as pointed out in [53]. For the simplicity of notation, we do not differentiate between static and dynamic Gaussians in Eq. 14.
5

---

For box predictions by dynamic Gaussians, 21 loss is used to supervised box attributes defined in Eq. and focal loss is used supervise corresponding box class labels
Lbox Σ CilC BB)) + (ocalLosssC. C. 8=1
(15)
where B, Co.5 denotes ground-trum truth. Label assignment is done using the Hungarian algorithm [30] during training. Hence, the 3D Loss can be written as
Lad = Loee + Asa Coors
(16)
where Aad is the weighting factor.
For rendered depth and semantic maps from Gaussians at all stages, we supervise depth with L loss and semantics with cross-entropy loss
f, Σ D) C)($ 2=1
(17)
where D. S are the rendered depth and semantic maps with D, S being the ground-truth. Here we project LiDAR points with their ground-truth semantic labels that are available from datasets [6, 45] onto each camera view to obtain D,S. S. Therefore the final loss can be written as
L = Lad + XCr,
(18)
where À is the weighting factor.
4 Experiments
4.1 Experiment Setup
Datasets: We evaluate our model on the Occ3D benchmark [50] which bootstraps the nuScenes [6] and Waymo-Open [45] dataset.' nuScenes consists of 1,000 scenes with a split of 700/150/ 150 for training, validation and testing. OcSeDDssSeenes annotates 3D occupancy ground-trute truth providing 17 semantic classes. The voxel grid range is [-40m, -40m, - 1m, 40m, 40m, 5.4m] along the X, Y and Z axis with grid resolution of 200 x 200 x 16 and voxel size of 0.4m. The original image resolution is 900 x 1600. Waymo Open [45] has 798 training scenes and 202 validation scenes. Occ3D-Waymo provides 3D semantic occupancy ground-truth of 15 semantic classes with class being General Object (GO). The voxel grid resolution and voxel size the same as Occ3D-nuSeenes. On Waymo, the original image resolution is 1280 x 1920 for the front, front-left and front-right cameras. For the side-left and side-right cameras, the original image resolution is 1040 x 1920.
Evaluation Metrics: We evaluate our model under the mloU and RayloU [47] metric:
|P a Gl
RayloU ErER [P, n Grl
mloU
GUUG
ΣTER I.,
where P, G' are the set of occupied voxels in prediction and ground-truth respectively. R. is the set of all emulated LiDAR rays, and Pr,, G,. are the sets of occupied voxels intersected by ray r in prediction and ground-truth respectively.
Implementation Details: We implement our proposed method in PyTorch [42]. Following previous works [35, 53, 4], we use ResNet-50 [20] as image backbone to extract multi-camera image features.
On nuScenes, we resize input images to the resolution of 256 x 704. On Waymo, all input images are resized and padded to 640 x 960. For Ours-tiny, we set number of static Gaussian queries $ 500
and number of dynamic Gaussian queries = 100. For Ours-large, we set S 4000 and = 800,
respectively. We use L 6 transformer layers to conduct coarse-to-fin fine prediction. We set Aga 0.2
to balance box loss Coo and occupancy loss Loce. For rendering loss L,, we set À = 0.05 for stage = 1, 6, and À 0.01 for the rest. We use AdamW [38] as the optimizer with weight decay of 0.01.
We train all our models with an initial learning rate of x 10 and decays with Aonealnn [39] schedule. For experiments
Waymo, we sample 20% of the data matching practices in previous works [53, 50]. Unless otherwise specified, we train all our models with a global batch size of for 100 epochs using NVIDIA A100 GPUs. During inference, we adopt the standard practice and make use of the camera visibility masks provided by the dataset [50] and only evaluate in unoccluded regions. Inference runtime measured on single idle A 1000 GPU with PyTorch fp32 backend.
"uSSeees is under a CC BY-NC-SA 4.0 license and Waymo license terms can be found here: https: /wwwmoo wwt.://oo///oo/nneeeeeeeeeeeeeee term///////..e.//..//

---

Table 1: 3D semantic occupancy results Occ3D-nuScenes validation [6, 50]. Cons. Veh stands for "Construction Vehicle" and Dri. Sur stands for "Drivable Surface". note that for fair comparison, both ODG-T and ODG-L here are trained without using future frames. Bold/Underline: Best/second best results. Vindicates self- supervised methods.
Podention
Te. P0
Method
adoU
TIITA
RaykUU
FRS
Rendertic [8T]
76.11
4.84
11.73
00.71
27.67
76.09
13.87
15.
17
17.84
19
23.25
63.
36-07
66.21
64.66
19 56
7177
19.1
3.9
GancResder
21.86
15
96
11.28
37.5
[77)
Spa (47)
30.1
34.5
17.3
Saa scoc (300 (47)
301
12.5
ORUS-L
23.13
55.38
73.13
47.01
Danssea [4]
1663
7.23
0.35
11.55
17.94
4.5
0.33
8.51
16.05
3.00
11:80
63.99
11.11
12
14
12.55
16.07
10.3
76.84
ODG-L
0DG
X00%
Dicycle
Motorcycle
Vegetation
Visualization of ODG prediction on the [50, 6] validation set. The ODG can capture all the vehicles gloomy rainy day.
Figure 2:
4.2 Evaluation Results
In this section, we report evaluation results on the Occ3D benchmark [50] and compare with latest state-of-the-art methods.
nuScenes Our results on Occ3D-nuScenes summarized in Tab. One achieves new state-of-the art results in terms of both mloU and RayloU, while maintaining competi- tive inference speed even when compared latest efficient approaches. Specifically, ODG-T (8f) achieves an mloU of 35.54 with a RayloU of 39.2, outperforming OPUS-T (8f) who has an mloU of 33.2 (-2.34) and RayloU of 38.4 (-0.8), while still having an inference runtime of 20.1 FPS. Similarly, ODG-T also easily outperforms both SparseOcc (8f). SparseOcc (16f) and GaussRender with significant margins. Meanwhile, our heavy variant ODG-L sets new best result eventually obtaining mloU of 38.15 with RayloU of 42.3, surpassing previous best with definitive margin of + +1.98 and +1.1, respectively.
that method
is worth noting that given our specific design to attend to the dynamic agents in the scene, we show significant improvement when examining the key dynamic object classes. As shown in Tab. for the classes of Bus, Car, Construction Vehicle (Cons. Veh), Motorcycle, and Truck, ODG-L carries a sigmificant lead of +4.13 for mloU, once again demonstrating the efficacy our proposed strategy of handling dynamic agents.
Table 2: Occupancy prediction results over key dynamic object classes on Oec3D-nuSeenes [50] validation set. ODG achieves consistent improvement across all dynamic categories. Bold/Underline Best/second best results.
Coos. Veh
Motorcycle
Trick
Method
mloti
Bas
GaussRender [10]
33.69
43.76
46.37
19.49
25.2
33.65
OPUS-T (80 [53]
32.68
39.76
45.25
23.41
21.80
33.20
OPUS [53]
40.95
47.24
35.28
ODG-T (8f)
36.22
46.75
49.33
25.79
23.63
35.61
ODG-L (8))
38.77
48.77
52.09
26.79
28.05
38.17

---

Table 3: 3D semantic occupancy results Oec3D-Waymo
Waymo validation set [45, 50]. GO stands for "General Object". Traf. Light stands for "Traffic Light" and Cons. Cone stands for "Construction Cone".
Vellole
Tral Light
Coo Clane
Blayde
Moongyele
Ruiliing
Welboble
Method
5kg0
RayloU
IPS
BEND (DS)
9.88
0.13
13.06
3,17
10.15
7,80
5.85
4.62
0.94
1.49
0.00
1.27
10.06
2.35
48.15
34.12
Ror/oo [M]]
16:26
1.48
17:16
13.87
13.84
1.7
9.62
12.2
13.09
0.00
13.38
11.66
6.7%
14.97
$1.61
TPVRemer (25)
16:76
1.80
17.86
12.08
567
13.64
5.40
8.90
9.55
14.79
0.32
13.82
11.44
73.3
51.49
4.6
CTF-00x (50)
18:33
6.26
36,00
14.66
8.32
15.44
10.53
11.7%
13,62
16.45
0665
18.63
17.3
8.20
67.09
42.98
2.6
OPUS-L (53)
19.00
4.66
27.07
10.30
6.53
18,66
6.41
11.44
10.40
12.90
0.00
18.73
18.11
7.46
72.56
50.31
24.7
8.5
ODG4
21.35
5.00
31.34
22.4
19J16
15.34
6.09
12.51
12.73
13.59
0.00
21.49
17:80
8.37
78.19
56.28
25,9
5.6
DDG
Redestrian
Bicyclist
Pola
Biyccle
Building
Thee Trunk
Welkable
Vehicle
Sign
That Light
Cons. Cone
Motorcycle
Vegetation
Rood
Figure 3: Visualization of ODG prediction on the Doc3D-Wayme [50, 45] validation set.
Waymo We further evaluate our ODG the Occ3D- -aymo dataset and the results are presented in Tab. We note that Doc3D-Waymo a much less well evaluated occupancy benchmark especially for camera-only methods, given its challenging conditions (e.g. almost no view overlap between cameras). We train ODG-L of with 20% of the training data following practice in previous methods [53, 54]. To the best of our knowledge, the most comprchensive source which compiles vision-only methods are from [50, 53], hence we use them for comparison. is clear from Tab. 3 that our ODG obtains definitive lead of +2.35 for mloU and +1.2 for RayloU when compared to OPUS. Upon further examining Tab. 3 one can see that ODG establishes far surpass the second -est under class Vehicle(+3 +.225), Bicyclist(+3... +3.0l) and Pedestrians( (estiianss++8.900 +8.91), which are all critical traffic agents, once again proving the superiority of our modeling of dynamic objects.
4.3 Visualization
We showcase visualizations of predicted occupancy of ODG on nuScenes and Waymo in Fig. and Fig. 3. In Fig. 2, under gloomy rainy weather where the ground-truth is sparse especially in object-centric categories (e.g. cars in this case), ODG effectively outputs dense predictions capturing all the cars in the scene. In Fig. 3, ODG also captures all the traffic agents. We attribute this attractive behavior to our dynamic Gaussian queries dedicated to modeling the dynamic scene agents.
4.4 Ablation Studies
In this section, we conduct multiple ablation studies to analyze the effects of various components in our proposed ODG. For all our ablation studies, adopt ODG-T and train on the Oce3D-nuScenes for 24 epochs.
Temporal Alignment As pointed in Sec. 3.3, in order to sample features correctly from the history frames, it is critical to move the Gaussians according to their respective motions before projecting them onto image feature plane. Hence, here we study the effects of performing different type of motion compensation. As shown in Tab. 5b, if we merely correct ego-motion (camera-motion for the dynamic Gaussians, both mloU and RayloU suffers a noticeable drop of -0.71 and -0.5 respectively. This demonstrates it is essential to compensate object motion for dynamic agents.
Query Interaction Given we have dynamic and static Gaussians modeling the scene, is important to make them aware of each other. We analyze the effect of different attention mechanisms in Tab. 5a. our experiments, we first tried performing cross attention with dynamic query features serving as queries, and static query features keys and values, which gave straight-forwan forward way to concatenate the dynamic and static query features together, and perform self attention top of concatenated features. This lead to further better results shown in Tab. 5a.
modest improvement. A more

---

Table 4: Impact of different components inside ODG model performance.
Motion compensation
Query attention
Rendering Sup
mloU
RayIoU, 1m
RayloU2
RayloU
30.80
27.9
36.6
42.0
31.78
28.7
37.3
42.6
32.13
29.3
37.8
32.82
31.1
40.5
43.8
Table 5: Ablation studies on components related to dynamic Gaussian queries.
(a) Effects of Query Attention.
(b) Effects of Motion Compensation.
Query Attention
mIoU
RayloU
Ego Comp. Dyn. Comp
mloU
RayloU
Cross Attn
31.95
36.3
31.17
35.7
Self Concat Attn
32.13
36.7
31.78
36.2
We posit that running self attention on all features in exhaustive manner makes all queries become aware of each other therefore facilitating more information flow, leading to improved results.
Rendering Supervision To reduce spatial artifacts through projective constraints, we leverage 3D> Gaussian Splatting and render depth and semantic maps each camera view. As shown in Tab. 4, this resulted in noticeable improvement both in mloU and RayloU, with mloU+0.69 and RayloU+0.7 70.00 By injecting the auxiliary supervision signals from LiDAR points projected onto each camera view (keyframes only, constrained by sensor calibration), it effectively enforces geometric and semantic consistency especially during the early stages of rendering, which helps the model learn more effectively for the regions that the 3D occupancy ground -ruth ambiguous or noisy. It is worth noting that during inference, we turn off rendering hence incurring no extra computation cost, reaping the benefit with zero increase in inference latency.
We summarize the effect of the different components in our proposed method in Tab. 4. is evident that by progressively enabling different modules in ODG, the model performs increasingly well, validating the soundness of the designs that we incorporated into our system.
5 Conclusions and Discussions
In this paper, we present ODG, novel approach to 3D occupancy prediction based on dual Gaussian queries. ODG defines two distinctive sets of queries consisting of dynamic and static Gaussian queries, aiming to better model dynamic scene agents. To make ODG attend to moving objects, we expand the standard 3D Gaussian properties of dynamic queries with 3D bounding box attributes, which effectively guides queries to dedicate resource to capture complex scene dynamics. Self attention utilized to establish connection between dynamic and static queries. To enable sufficient model capacity, ODG adopts coarse-to coarse-to-fine prediction paradigm when it comes to predicting the Gaussian parameters, which allows large number of Gaussians to utilized. To reduce spatial artifacts in 3D occupancy, we enforce projective constraints through rendering depth and semantic maps to each camera view leveraging 3D Gaussian Splatting, supervised by projected LiDAR points. Our extensive experiments on the Oce3D nuSSenes and Oce3D- -aymo benchmark demonstrates ODG sets new state-of tfate-of-the-ar the--art results while maintaining highly competitive efficiency.
Limitations. However, as promising as ODG is, it does not come without limitations. Readers might have observed that currently the Gaussian parameters other than mean (namely scale, rotation and opacity) is only optimized through rendering loss, rather than for instance, aggregating nearby Gaussians to get occupancy and learn from occupancy ground as well, albeit such aggregation incurs significant cost. Therefore exploring ways to improve optimization of Gaussians would be an interesting research direction.
Broader Impacts. Our proposed ODG provides more accurate 3D occupancy prediction while maintaining inference efficiency, which is beneficial to safe, energy-efficient autonomous driving. Furthermore, given the versatility and efficiency of our ODG, we think it is promising venue towards building a unified perception and prediction module for autonomous driving, which will improve the homogeneity of the entire stack that will in turn lower overall operating cost, yielding economic benefit. On the other hand, during the development of this work, several experiments did not make into the paper (e.g. due to lack of time), which incurred extra energy consumption.

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualle and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5855-5864, 2021.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5470-5479, 2022.
[3] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickiti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference computer vision, pages 9297-9307, 2019.
[4] Simon Boeder, Fabian Gigengack, and Benjamin Risse. Gaussianflowoce: Sparse and weakly supervised occupancy estimation using gaussian splatting and temporal flow. arXiv preprini arXiv:2502. 17288, 2025.
[5] Simon Boeder, Fabian Gigengack, and Benjamin Risse. Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.
[6] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-1 11611, 2020.
Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3991-4001, 2022.
[8] Anh-Quan Cao and Raoul de Charette. Scenerf: Self-supervised monocular scene reconstruction with radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9387-9398, 2023.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages
Springer, 2020.
[10] Loick Chambon, Eloi Zablocki, Alexandre Boulch, Mickael Chen, and Matthieu Cord. Gaussrender: Learning 3d occupancy with gaussian rendering. arXiv preprint arXiv:2502. 050000 2025.
[11] David Charatan. Sizhe Lester Li. Andrea Tagliasacchi. and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457-19467, 2024.
[12] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In Furopean Conference on Computer Vision, pages 370-386. Springer, 2024.
[13] OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving. https://githin htttpp/://gitaaa...///ppeeeiiiii wwwpeerrieeaabaeeeaaa..eeaeeeeee
2023.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
[15] Haoqiang Fan, Hao Su, and Leonidas Guibas. point set generation network for object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[16] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):123 10:1231-12333 -237, 2013.
[17] Andreas Geiger, Philip Leaz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and puttern recognition, pages 3361- IEEE, 2012.
[18] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 2021.
10

---

[19] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 5354-5363, 2024.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Mu Hu, Wei Yin. Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang. Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: versatile monocular geometrie foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404. 15506, 2024.
[22] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-ore -oriente autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853-17862 17862, 2023.
[23] Junjie Huang, Guan Huang, Zheng Thu, Yun Ye, and Dalong Du Bevdet: High-performance multi-camera 3d object detection in bird-eyc-view. arXiv preprint arXiv:212.11 /1790 2021.
[24] Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. Selfore: Self-supervised vision- based 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19946-19956, 2024.
[25] Yuanhui Huang. Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9223-9232, 2023.
[26] Yuanhui Huang. Wenzhao Zheng. Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction. In European Conference on Computer Vision, pages 376-393. Springer, 2024.
[27] Haoyi Jiang. Liu Liu, Tianheng Cheng, Xinjie Wang. Tianwei Lin, Zhizhong Su, Wenyu Liu, and Xinggang Wang. Gausstr: Foundation model- aligned gaussian transformer for self--uper spatial understanding. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[28] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendcring. ACM Trans. Graph., 42(4): (144))139-11 2023.
[29] Bemhard Kerbl, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3d gaussian representation for real-time rendering of very large datasets. ACM Transactions on Graphics (TOG), 43(4):1-15, 2024.
[30] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(122):3397, 1955.
[31] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's s-eye-veww representation from multi-camera images via spatiotemporal trans- formers. In European conference computer vision, pages 1-18. Springer, 2022.
[32] Tsung-Yr Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2999-3007, 2017.
[33] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse tempolll fusion. arXiv preprint arXiv:2211.1058) 2022.
[34] Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High- berehhpeffrmancc sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18580 18590, 2023.
[35] Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang, Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. In Proceedings of the European Confernece on Computer Vision, 2024.
[36] Yili Liu, Linzhan Mou, Xuan Yu, Chenrui Han, Sitong Mao, Rong Xiong, and Yue Wang. Let oce flow: Self supervieee 3d occupancy flow prediction. The Conference on Robot Learning (CoRL), 2024.
[37] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr; Position embedding transformation for multi-view 3d object detection. arXiv preprint arXiv:2203.05625, 2022.
[38] Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100/ 2017.
11

---

[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv: 03983, 2016.
[40] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, Communications of the ACM, (6(1))99-106, 2021.
[41] Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang. Xiaoqi Li, Hongwei Xie, Bing Wang. Li Liu, and Shanghang Zhang. Rendemec:: Vision-centric 3d occupancy prediction with 2d rendering supervision. In 2024 IEFE International Conference on Robotics and Automation (ICRA), pages 12404-12411 IEEE, 2024.
[42] A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 1912.0173., 2019.
[43] Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3500 2020.
[44] Yunxiao Shi, Hong Cai, Amin Ansari, and Fatih Porikli. H3o: Hyper-efficient 3d occupancy prediction with heterogeneous supervision. IEEE International Conference on Robotics and Automation (ICRA), 2025.
[45] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patmaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[46] Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 10208-10217, 2024.
[47] Pin Tang. Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren. Bailan Feng, and Chao Ma. Sparseoce: Rethinking sparse latent representation for basion-based semantic occupancy prediction. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15035- 15044, 2024.
[48] Ayush Tewari, Ohad Fried, Justus Thics, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin- Tomas Simon, Jason Saragih, Matthias Nicßner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pages 701-727. Wiley Online Library, 2020.
[49] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[50] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang. Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36. 2024.
[51] Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8406-844 8415, 2023.
[52] A Vaswani. Attention is all you need. Advances Neural Information Processing Systems, 2017.
[53] Jiabao Wang, Zhaojiang Liu, Qiang Meng, Liujiang Yan, Ke Wang, Jie Yang, Wci Liu, Qibin Hou, and Mingming Cheng. Opus: Occupancy prediction using sparse set. In Advances in Neural Information Processing Systems, 2024.
[54] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEFE/CVF international conference on computer vision, pages 3621-3631, 2023..
[55] Sinong Wang, Belinda 7 Li, Madian Khahsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[56] Xiaofeng Wang. Zheng Zhu, Wenbo Xu, Yumpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17850- 17859, 2023.
12

---

[57] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference Robot Learning. pages 180-191. PMLR, 2022.
[58] Yi Wei, Linqing Zhao, Wenzhao Zheng. Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera occupancy prediction for autonomous driving. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 11729-21740, 2023.
[59] Wei Yin, Chi Zhang, Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shoo metric 3d prediction from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9043-9053, 2023.
[60] Zehao Yu, Anpei Chen, Binbin Huang. Torsten Sattler, and Andreas Geiger. Mip-splatting Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19447-19456, 2024.
[61] Zichen Yu. Changyong Shu, Jiajun Deng, Kangjie Lu, Zongdai Liu, Jiangyong Yu, Dawei Yang. Hui Li, and Yan Chen. Flashoec: Fast and memory-effeient occupancy prediction via channel-to-height plugin. arXiv preprint arXiv:2311. 12058, 2023.
[62] Chubin Zhang, Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen Lu. Ocenerf: Sele-super multi-camera occupancy prediction with neural radiance fields. arXiv preprint arXiv:2312.09243, 2023.
[63] Yunpeng Zhang, Zheng Zhu, and Dalong Du. Oceformer: Dual-path transformer for vision-bass basood 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9433-9443, 2023.
[64] Benjin Zhu, The Wang, and Hongsheng Li, nucraft: Crafting high resolution semantic occupancy for unified 3d scene understanding. In European Conference on Computer Vision, pages 125-141. Springer, 2024.
13

---

Appendix
We provide more experiment analysis regarding various aspects ODG.
A. Coarse-to-fine Prediction of Gaussian Parameters
We present details of the coarse-to- to-fine refinement used in ODG. For notation simplicity, we denote motion compensation, feature sampling, cross and self attention, along with dual query-attention presented in Figure as single layer Ti, rest of the notations are the same as defined in Section 3.2. To further enhance clarity, since coarse-to-fine refinement doesn't it apply to bounding box attributes (i.e. Kn = throughout all the layers TD,, we have omitted them in Algorithm below.
Input: images features F; static Gaussian queries (G O dynamic Gaussian queries (G Qa);
Output: refined Gaussian queries (Gi, Q class predictions Cf refined dynamic Gaussian queries (G Q class predictions Cy.
Function ODGRefine(F, Gū- Q6: GØ Q
G 12 Kox3 +- (00 1), Ko =1 Gd E Dx ((,, 1), K, =1
// initialize Gaussian means with uniform distribution; rest of the Gaussian parameters left uninitialized for <- 1 to do
# static
G Qi Ci = Gi 1 Qi- Ci-1)
G (G: Qi) rest of Gaussian params predicted by MLP
// G "n.
E 1R5xK Mi.1 Sx Cf E 5 KiXO, _K-1 < K₁ # coarse-to-fing
# dynamic
Gd 1 Q Cf = Ti :n.-1: Qi C%-1)
G pcd = (G : Qf") // rest of Gaussian params predicted by MLP
// G Sul E ER Dx Kix3 pe,s E R xxXXKix0 Of E R Dx KiXC Ki-i Kt
# coarse-to-fine
end
return
(Gi, Q2 Ct), (G Q C1)
Algorithm 1: Coarse-to-fine refinement in ODG.
B. Runtime Efficiency
We profiled ODG-L at inference time with DeepSpeed [43]. Results are summarized in Table 6a below.
Table 6: Runtime analysis of ODG-L (FP32).
Runtime of different components in ODG-L
(b) Runtime of different components in ODG-L transformer.
Component
Runtime (ms)
Percentage
img_ hackhone (ReNet50)
33.82
16.58%
Component
Runtime (ms)
Percentage
img neck (HPN)
9,83
4.82%
Self-attention
70.77
44.14%
ODG-L tramerormer
160.32
78.59%
Point sampling
25.88
16. 14%
Cross-attention
24.06
15.01%
FFN
8.55
5.33%
We can see that the transformer part takes up most of the inference cost in ODG-L, We provide further profiling results on ODG-L transformer in Table 6b above. Evidently query self-attention takes up almost half the runtime. One straight-away optimization can be replacing self-attention with efficient attention schemes such linear attention [55] or state space models (SSMs) [18]. We plan look into this as part of future work.
14

---

C. Evaluation of Prediction from Each Layer
To provide further insight of the coarse-to- scheme in ODG, we evaluate predictions of all layers of ODG-T against ground-trum occupaney. The results are summarized in Table below. In the beginning since the prediction is coarse, the model behaves poorly in terms of mloU and RayloU. Then with our coarse-to-fine refinement, ODG starts gradually learning the scene and gives progressively better results, which validates the effectiveness of our coarse-to-fine refinement design.
Table 7: Evaluation each layer's prediction ODG-T.
ODG-T # Predicted Gaussians
mloU
RayloU im
RayloU2m
RayloUam
RayloU
layer
600
0.35
1.9
2.8
3.8
2.8
layer
2400
3.17
9.5
13.5
16.3
13.0
layer 9600
18.68
25.9
32.6
36.6
31.7
layer 12800
26.48
29.9
37.9
42.2
36.6
layer 38400
31.02
30.5
39.9
43.3
37.9
layer
76800
32.82
31.1
40.5
43.8
38.5
D. Ablation Supervision Signals
In Table 8, we show how ODG performs when there is only occupancy labels available. Compared to full supervision with bbox and rendering labels, ODG suffers slight drop in terms of mloU and rayloU, but still delivers strong performance, which validates our design.
Table 8: Ablation on supervision signal.
Supervision
mloU
rayloU
Occupancy only
31.46
36.1
Occupancy & bbox & rendering supervision
32.82
38.5
E. Effect of Query Composition
We study the effect of utilizing different types of queries. The results are summarized in Table
Table 9: Effect of query composition on model performance.
Method
Queris
mloU
RayloU
FPS
OPUS-T [53]
600 (static)
33.2
38.4
22.4
OPUS-L [53]
4800 (static)
36.2
41.2
7.2
ODG-T
500 (static)+100 (dynamic)
35.5
39.2
20.1
ODG-L
4000 (static)+800 (dynamic)
38.2
42.3
4.9
ODG-T"
600 (dynamic)
34.8
38.9
17.6
ODG-L."
4800 (dynamic)
37.6
42.1
3.7
One can see that by taking into account object motion, ODG definitely outperforms baseline OPUS [53] which only considers ego motion. Interestingly, when treat all queries dynamic (denoted as ODG-*), still performs better than OPUS. We attribute this gain to the object detection task present in our system which effectively improves dynamic object predictions, and through rendering constraint we also learn better 3D geometry.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the paper, we have provided sufficient details for reproducing our work, including detailed design and experiment setups.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: We will aim to release the code. which will be subject to our institution's ' review and approval. The datasets used our experiments are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. "nips.cc/ppppp cccppblic/guu getggguiicccsssai wwweesummissioPpooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
In this paper, we have specified all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In our experiments, we use established benchmarks and training & testing protocols, which do not include reporting error bars or statistical significance tests. the literature, it is not a common practice to report error bars in the 3D occupancy prediction problem considered this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided information on the compute resources in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics ftttps://neuripet pttp:s///errppp...pppb.btt httppuublc//thhiccuudeeeeee..eee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics, and our research conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not introduce data or models that have a high risk for misuse and as such, poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; In the paper, we have cited the original papers that produced the data or models, and have mentioned/provided links to the licenses and terms of use. We have properly respected the licenses and terms of use of the data and model used in this work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httpaa/:///upps...eee http://erppp..c/ooeeeeeeeeeeeeee / for what should or should not be described.
22


Reviews:
[{"review_id": "FUyiEZGc8f", "reviewer": "Reviewer_cwc4", "summary": "This paper proposes the ODG framework for occupancy prediction. First, it uses priors of static and dynamic objects in the real world to initialize Gaussians for modeling. Second, the method designs a hierarchical model structure to improve modeling accuracy. Finally, multiple supervision signals are adopted during training. This approach enhances the accuracy of occupancy prediction without sacrificing inference speed.", "questions": "1. Regarding Weakness 1: Please elaborate on the design of the coarse-to-fine hierarchical structure.\n2. Regarding Weakness 2: Please provide detailed visual comparisons with other methods.\n\nNote: As I am not an expert in this field, I will also refer to suggestions from more specialized reviewers during the rebuttal stage.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. **Clear writing:** The paper is clearly written, allowing readers to easily follow the author’s logical flow.\n2. **Superior performance:** The method outperforms state-of-the-art approaches across all listed test datasets.\n3. **Good efficiency:** The proposed framework ensures competitive inference speed with minimal parameter overhead, balancing performance and efficiency.\n\n## Weaknesses\n1. **Model design details:** The authors highlight \"Hierarchical Coarse-to-Fine Refinement\" as a core contribution in the paper’s significance section, but the methodological description lacks detailed elaboration. Further clarification is expected.\n2. **Visualization experiments:** The paper only provides visualization results of the proposed method. Visual comparisons between ODG and other methods are needed for a comprehensive evaluation.", "was_revised": true, "final_justification": "The authors have provided a thorough response to our first question during the rebuttal period. However, due to changes in the conference policies, they were unable to address the second question. We are confident that the authors can supplement the relevant experimental details in subsequent stages, and thus we propose to maintain the \"Borderline accept\" rating.", "rebuttal": "We thank reviewer cwc4 for the constructive feedbacks. We address the reviewer's questions and comments below.\n\n$\\textbf{Design of coarse-to-fine hierarchical structure}$:\n\nWe present our coarse-to-fine refinement scheme in **Algorithm 1** below. \nFor notation simplicity, we denote motion compensation, feature sampling, cross and self attention, along with dual query-attention presented in Figure. 1 in the main paper as a single layer $\\hat{\\mathcal{T}}_l$, rest of the notations are the same as defined in the main paper. To further enhance clarity, as described in the main paper, since coarse-to-fine refinement doesn't apply to bounding box attributes (i.e. $K_0=1$ throughout all the layers $\\hat{\\mathcal{T}}_l$), we have omitted them in **Algorithm 1** presented below.\n\nWe note that due to OpenReview's limited TeX support, we had to declare transient notations such as $p, \\text{inputs}^s_{p}, \\text{ and }\\text{inputs}^d_{p}$ to facilitate algorithmic flow.\n\n**Algorithm 1: Coarse-to-fine refinement in ODG**\n\n$\\text{Input}: \\text{image features } \\textbf{F}; \\text{static Guassian queries } \\textbf{G}_0^s, \\textbf{Q}_0^s; \\text{dynamic Guassian queries } \\textbf{G}_0^d, \\textbf{Q}_0^d$\n\n$\\text{Output}: \\text{refined static Guassian queries } \\textbf{G}_L^s, \\textbf{Q}_L^s, \\text{ class predictions }\\mathbf{C}^s_L; \\text{ refined dynamic Guassian queries } \\textbf{G}_L^d, \\textbf{Q}_L^d, \\text{ class predictions }\\mathbf{C}^d_L$\n\n$\\text{Function } \\text{ODGRefine}(\\mathbf{F}; \\textbf{G}_0^s, \\textbf{Q}_0^s; \\textbf{G}_0^d, \\textbf{Q}_0^d):$\n\n$\\quad\\mathbf{G}^s_{:\\mu,0}\\in\\mathbb{R}^{S\\times K_0\\times 3} \\leftarrow \\mathcal{U}(0, 1), K_0=1$\n\n$\\quad\\mathbf{G}^d_{:\\mu,0}\\in\\mathbb{R}^{D\\times K_0\\times 3} \\leftarrow \\mathcal{U}(0, 1), K_0=1$\n\n$\\quad$// initialize Gaussian means with uniform distribution; rest of the Gaussian parameters left uninitialized\n\n$\\quad\\text{for }l\\leftarrow 1 \\text{ to } L \\text{ do }$\n\n$\\quad\\quad p=l-1$\n\n$\\quad\\quad\\text{inputs}^s_p=[\\mathbf{G}^s_{\\:\\mu,p}, \\mathbf{Q}^s_{p}, \\mathbf{F}]$\n\n$\\quad\\quad\\mathbf{G}_{\\:\\mu, l}^s,\\mathbf{Q}^s_l,\\mathbf{C}^s_l=\\hat{\\mathcal{T}}_l(\\text{inputs}^s_p)$ \n\n$\\quad\\quad\\mathbf{G}^s_{\\mu:,l}=\\Phi(\\mathbf{G}^s_{:\\mu,\\ell}, \\mathbf{Q}^s_l)$ // rest of Gaussian params predicted by MLP $\\Phi$\n\n$\\quad\\quad\\text{// }\\mathbf{G}^s_{\\:\\mu, l}\\in\\mathbb{R}^{S\\times K_l\\times 3}, \\mathbf{G}^s_{\\mu\\:, l}\\in\\mathbb{R}^{S\\times K_l\\times 10}, \\mathbf{C}^s_l\\in\\mathbb{R}^{S\\times K_l\\times C}, K_p < K_l$, # coarse-to-fine\n\n$\\quad\\quad\\text{inputs}^d_p=[\\mathbf{G}^s_{\\:\\mu,p}, \\mathbf{Q}^d_{p}, \\mathbf{F}]$\n\n$\\quad\\quad\\mathbf{G}_{\\:\\mu, l}^d, \\mathbf{Q}_l^d,\\mathbf{C}^d_l=\\hat{\\mathcal{T}}_l(\\text{inputs}^d_p)$\n\n$\\quad\\quad\\mathbf{G}^d_{\\mu:,l}=\\Phi(\\mathbf{G}^d_{:\\mu,\\ell}, \\mathbf{Q}^d_\\ell)$ // rest of Gaussian params predicted by MLP $\\Phi$\n\n$\\quad\\quad\\text{// }\\mathbf{G}^d_{\\:\\mu, l}\\in\\mathbb{R}^{D\\times K_l\\times 3}, \\mathbf{G}^d_{\\mu\\:, l}\\in\\mathbb{R}^{D\\times K_l\\times 10}, \\mathbf{C}^d_l\\in\\mathbb{R}^{D\\times K_l\\times C}, K_p < K_l$, # coarse-to-fine\n\n$\\quad\\text{return }[\\mathbf{G}^s_l, \\mathbf{Q}^s_l, \\mathbf{C}^s_l], [\\mathbf{G}^d_l, \\mathbf{Q}^d_l, \\mathbf{C}^d_l]$\n\n$\\textbf{Visual comparison with other methods}$: \n\nWe have prepared such visual comparisons and will include them in the final version like in Fig. 2 and Fig. 3 to further demonstrate the superiority of our proposed ODG. Unfortunately, this year's NeurIPS rebuttal format does not support uploading a PDF for the rebuttal and as such, we are not able to share visualization results during the rebuttal stage."}, {"review_id": "5IGtkKP7pB", "reviewer": "Reviewer_JcRJ", "summary": "The article introduces ODG (Occupancy Prediction Using Dual Gaussians), a novel method for occupancy prediction in autonomous driving. This approach integrates 3DGS and models the scene by dividing it into two categories—static and dynamic—in a hierarchical manner. It employs a gaussian transformer to predict gaussian parameters and leverages the rendering capability of 3DGS to obtain supervisory signals. The method achieves SOTA performance across multiple benchmarks.", "questions": "1. ODG-L achieves better accuracy but at the cost of significantly reduced inference speed (4.9 FPS vs. 20.1 FPS). Can the authors provide more insight into what components contribute most to this slowdown? Are there practical ways to improve the efficiency of ODG-L without sacrificing too much performance?\n2. The ablation studies (Sec. 4.4) are conducted after training for only 24 epochs, while the final model is trained for 100 epochs. Could the conclusions drawn from these ablations change when evaluated over longer training periods? For example, do certain components become more or less important with extended training?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nThis paper proposes a novel 3D occupancy prediction method called ODG, which introduces a dual Gaussian query architecture to separately model static and dynamic elements in the scene, demonstrating strong technical innovation. The method leverages a hierarchical Gaussian Transformer to perform coarse-to-fine predictions, effectively enhancing the model's representational capacity. Experimental results show that ODG achieves SOTA performance on both the Occ3D-nuScenes and Occ3D-Waymo datasets while maintaining low inference cost, highlighting the effectiveness and practicality of the approach.\n\nWeaknesses:\n\n1. The coarse-to-fine prediction mechanism in Figure 1 is not intuitively illustrated.  \n2. The inference speed of ODG-L is only 4.9 FPS, indicating that its superior performance comes at a significant cost of efficiency, which may hinder its deployment in real-world applications.  \n3. The ablation studies were conducted by training for only 24 epochs on Occ3D-nuScenes, while the final model was trained for 100 epochs; therefore, the validity of generalizing the conclusions remains uncertain.", "was_revised": true, "final_justification": "The authors have fully addressed my concerns through a clear algorithmic description of the coarse-to-fine mechanism, detailed efficiency profiling, and ablation studies extended to 100 epochs, which validate the robustness of their design choices. The suggestion to improve the clarity of Figure 1 remains as a minor recommendation, but it does not affect the technical soundness of the work. The paper presents a novel and effective approach to 3D occupancy prediction with strong experimental results. Given the significance of the contribution and the thoroughness of the rebuttal, I maintain my rating of 4.", "rebuttal": "We thank reviewer JcRJ for the constructive feedbacks. We address the reviewer's questions and concerns below.\n\n$\\textbf{Coarse-to-fine mechanism}$: \n\nWe present our coarse-to-fine refinement scheme in **Algorithm 1** below. For notation simplicity, we denote motion compensation, feature sampling, cross and self attention, along with dual query-attention presented in Figure. 1 in the main paper as a single layer, rest of the notations are the same as defined in the main paper. To further enhance clarity, as described in the main paper, since coarse-to-fine refinement doesn't apply to bounding box attributes (i.e. $K_0=1$ throughout all the layers $\\hat{\\mathcal{T}}_l$), we have omitted them in **Algorithm 1** presented below.\n\nWe note that due to OpenReview's limited TeX support, we had to declare transient notations such as $p,\\text{inputs}^s_p,\\text{ and }\\text{inputs}_p^d$ to facilitate algorithmic flow.\n\n**Algorithm 1: Coarse-to-fine refinement in ODG**\n\n$\\text{Input}: \\text{image features } \\textbf{F}; \\text{static Guassian queries } \\textbf{G}_0^s, \\textbf{Q}_0^s; \\text{dynamic Guassian queries } \\textbf{G}_0^d, \\textbf{Q}_0^d$\n\n$\\text{Output}: \\text{refined static Guassian queries } \\textbf{G}_L^s, \\textbf{Q}_L^s, \\text{ class predictions }\\mathbf{C}^s_L; \\text{ refined dynamic Guassian queries } \\textbf{G}_L^d, \\textbf{Q}_L^d, \\text{ class predictions }\\mathbf{C}^d_L$\n\n$\\text{Function } \\text{ODGRefine}(\\mathbf{F}; \\textbf{G}_0^s, \\textbf{Q}_0^s; \\textbf{G}_0^d, \\textbf{Q}_0^d):$\n\n$\\quad\\mathbf{G}^s_{:\\mu,0}\\in\\mathbb{R}^{S\\times K_0\\times 3} \\leftarrow \\mathcal{U}(0, 1), K_0=1$\n\n$\\quad\\mathbf{G}^d_{:\\mu,0}\\in\\mathbb{R}^{D\\times K_0\\times 3} \\leftarrow \\mathcal{U}(0, 1), K_0=1$\n\n$\\quad$// initialize Gaussian means with uniform distribution; rest of the Gaussian parameters left uninitialized\n\n$\\quad\\text{for }l\\leftarrow 1 \\text{ to } L \\text{ do }$\n\n$\\quad\\quad p=l-1$\n\n$\\quad\\quad\\text{inputs}^s_p=[\\mathbf{G}^s_{\\:\\mu,p}, \\mathbf{Q}^s_{p}, \\mathbf{F}]$\n\n$\\quad\\quad\\mathbf{G}_{\\:\\mu, l}^s,\\mathbf{Q}^s_l,\\mathbf{C}^s_l=\\hat{\\mathcal{T}}_l(\\text{inputs}^s_p)$ \n\n$\\quad\\quad\\mathbf{G}^s_{\\mu:,l}=\\Phi(\\mathbf{G}^s_{:\\mu,\\ell}, \\mathbf{Q}^s_l)$ // rest of Gaussian params predicted by MLP $\\Phi$\n\n$\\quad\\quad\\text{// }\\mathbf{G}^s_{\\:\\mu, l}\\in\\mathbb{R}^{S\\times K_l\\times 3}, \\mathbf{G}^s_{\\mu\\:, l}\\in\\mathbb{R}^{S\\times K_l\\times 10}, \\mathbf{C}^s_l\\in\\mathbb{R}^{S\\times K_l\\times C}, K_p < K_l$, # coarse-to-fine\n\n$\\quad\\quad\\text{inputs}^d_p=[\\mathbf{G}^s_{\\:\\mu,p}, \\mathbf{Q}^d_{p}, \\mathbf{F}]$\n\n$\\quad\\quad\\mathbf{G}_{\\:\\mu, l}^d, \\mathbf{Q}_l^d,\\mathbf{C}^d_l=\\hat{\\mathcal{T}}_l(\\text{inputs}^d_p)$\n\n$\\quad\\quad\\mathbf{G}^d_{\\mu:,l}=\\Phi(\\mathbf{G}^d_{:\\mu,\\ell}, \\mathbf{Q}^d_\\ell)$ // rest of Gaussian params predicted by MLP $\\Phi$\n\n$\\quad\\quad\\text{// }\\mathbf{G}^d_{\\:\\mu, l}\\in\\mathbb{R}^{D\\times K_l\\times 3}, \\mathbf{G}^d_{\\mu\\:, l}\\in\\mathbb{R}^{D\\times K_l\\times 10}, \\mathbf{C}^d_l\\in\\mathbb{R}^{D\\times K_l\\times C}, K_p < K_l$, # coarse-to-fine\n\n$\\quad\\text{return }[\\mathbf{G}^s_l, \\mathbf{Q}^s_l, \\mathbf{C}^s_l], [\\mathbf{G}^d_l, \\mathbf{Q}^d_l, \\mathbf{C}^d_l]$\n\n$\\textbf{Efficiency analysis for ODG}$: \n\nWe profiled both ODG-L and ODG-T using DeepSpeed. The results are summarized Table 1 and 2 below. \n\n**Table. 1 ODG-L profiling.**\n| component       |  runtime (ms)    |    percentage    |\n| -------- | ------- | -------- |\n|    img_backbone (ResNet-50)    |    33.82    |   16.58\\%     |\n|    img_neck (FPN)    |    9.83    |   4.82\\%     |\n|    ODG-L transformer  |    160.32   |   78.59\\%     |\n\n**Table. 2 ODG-T profiling.**\n| component       |  runtime (ms)    |    percentage    |\n| -------- | ------- | -------- |\n|    img_backbone (ResNet-50)    |    12.30    |   24.88\\%     |\n|    img_neck (FPN)    |    3.44    |   6.96\\%     |\n|    ODG-T transformer    |    33.71  |   68.15\\%     |\n\nWe can see that in both ODG-L and ODG-T, the transformer part takes up most of the inference cost. We provide further profiling results on ODG-L transformer in Table 3 below. One can see that query self-attention takes up almost half the runtime. One straight-away optimization can be replacing self-attention with efficient attention schemes such as linear attention [1] or state space models (SSMs) [2], which would improve inference runtime. We will look into this as part of future work.\n\n**Table. 3 ODG-L transformer profiling.**\n| component       |  runtime (ms)    |    percentage    |\n| -------- | ------- | -------- |\n|    Self Attention    |     70.77   |   44.14\\%     |\n|    Point Sampling    |   25.88   |   16.14%     |\n|    Cross Attention  |    24.06   |   15.01\\%     |\n|     FFN                |         8.55         |        5.33\\%           |\n|    ... |     ...     |       ...      |   ... |\n\n[1] Wang, Sinong, et al. \"Linformer: Self-attention with linear complexity.\" arXiv preprint arXiv:2006.04768 (2020).\n\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n$\\textbf{Ablation studies for 100 epochs}$: \n\nWe have now conducted all ablation experiments with ODG-T and trained for 100 epochs on nuScenes. The results are listed in Table 4 below. We can see under longer training time our ODG still consistently improves, which validates the efficacy of our designs. The conclusions on the effectiveness of our proposed components do not change when training for 24 or 100 epochs.\n\n**Table 4. Ablation studies of different components of ODG-T. Trained for 100 epochs on nuScenes.**\n| Motion compensation    | Query Attention | Rendering Sup | $\\text{mIoU}$ | $\\text{RayIoU}_{1m}$ | $\\text{RayIoU}_{2m}$ | $\\text{RayIoU}_{4m}$ | $\\text{RayIoU}$ |\n| -------- | ------- | -------- | -------- | -------- | -------- | -------- | -------- | \n|   |     |  | 32.71 |30.0 | 39.6 | 42.7 | 37.4 |\n| yes |     |  | 33.97 | 31.1 | 40.3 | 43.5 | 38.5 |\n| yes  | yes    |  | 34.83 | 31.4 | 40.8 | 44.1 | 38.8\n| yes  | yes    | yes | 35.54 | 31.8 | 41.2 | 44.5 | 39.2 |"}, {"review_id": "LnOUVNFoKc", "reviewer": "Reviewer_M77i", "summary": "The paper proposes ODG, a method that employs a hierarchical dual sparse Gaussian representation for 3D occupancy prediction. It explicitly separates dynamic and static queries, and refines Gaussian parameters in a coarse-to-fine manner. Notably, the approach leverages supervision from 3D bounding boxes through a box encoder, as well as 2D labels via a rendering process. These innovations lead to strong performance on the Occ3D benchmark.", "questions": "- Does each dynamic query correspond to a single semantic class? If not, applying motion compensation may be questionable. If so, how are queries handled when they fail to match any 3D bounding box during the Hungarian matching process?\n- How would the model perform if only occupancy labels were used for supervision?\n- [minor] In Table 1, the best and second-best results are not correctly highlighted.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:  \n- The writing and figures in paper are clear.\n- The paper presents extensive experimental results, demonstrating their high accuracy and fast inference speed.\n\nWeaknesses:  \n- The novelty of the proposed model is limited. The separation of dynamic and static modeling is straightforward, and the coarse-to-fine refinement mechanism has already been explored in prior works [3, 4]. Additionally, the use of supervision via rendering shares conceptual similarities with approaches such as RenderOcc [1].\n- The method leverages extra supervision signals, including 3D bounding boxes and 2D image labels. As a result, the comparison with prior methods that do not utilize such annotations may be unfair.\n\n[1] RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision    \n[2] Fully Sparse 3D Panoptic Occupancy Prediction  \n[3] Opus: occupancy prediction using a sparse set", "was_revised": true, "final_justification": "Although some high-level concepts are similarly presented in related works, the paper introduces its own novel designs. The experiments included in the rebuttal demonstrate that ODG can perform well without utilizing extra labels. Therefore, I raise the rating into borderline accept.", "rebuttal": "We thank reviewer M77i for the constructive feedbacks. We address the reviewer's questions and comments below.\n\n$\\textbf{Contributions on dynamic and static modeling:}$\n\nDesigning an occupancy prediction network that properly considers both dynamic and static objects is not trivial and requires several careful designs. First, it is critical to create a proper learning scheme for dynamic queries, as velocity ground truth is not available. Without proper motion modeling and learning, one cannot correctly aggregate information across time steps. In addition, it is also important to introduce some form of feature interaction between dynamic and static queries in an efficient way, without which the occupancy prediction performance would not be good and/or significant additional computation cost would be incurred. Moreover, our explicit 3D representation using 3D Gaussians also makes it more efficient and feasible to incorporate motion of dynamic objects, as compared to commonly used dense representations like birds-eye-view or 3D voxels which would require expensive 2D/3D warping.\n\n$\\textbf{Comparing to OPUS and SparseOcc}$:\n\nWhile OPUS [1] and SparseOcc [2] leverage some forms of coarse-to-fine design, their models miss certain important factors and/or require some cumbersome designs. For instance, OPUS is a query-based method using points as scene representation, which cannot directly be used for rendering. In addition, OPUS does not consider the motion of dynamic objects at all, which makes the modeling incomplete as there are almost always dynamic objects in a driving scene. SparseOcc employs a series of sparse voxel decoders to filter out empty grids and predict occupancy state of retained voxels, which is inefficient. In contrast, we predict 3D Gaussians in a streamlined fashion without any pre-defined locations and takes dynamic objects into account. Our proposed ODG introduces a new design offering more complete modeling, higher efficiency, as well as the possibility of leveraging additional supervision, which goes beyond incremental improvements.\n\n[1] Wang, Jiabao, et al. \"Opus: occupancy prediction using a sparse set.\" Advances in Neural Information Processing Systems. 2024\n\n[2] Liu, Haisong, et al. \"Fully sparse 3d occupancy prediction.\" European Conference on Computer Vision. 2024.\n\n$\\textbf{Difference from RenderOcc}$: \n\nWhile RenderOcc also leverages neural rendering to improve occupancy prediction, it is NeRF-based which carries a significantly higher cost in terms of training time and memory required. In addition, RenderOcc adopts BEVStereo4D [1] as feature representation, which incurs a much higher inference runtime as shown in Table 1 in our paper. In contrast, in ODG our query-based representation enables a much more efficient scene representation, while also leveraging Gaussian splatting resulting in much faster rendering and significantly reduced training and inference cost. \n \n[1] Li, Yinhao, et al. \"Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.\n\n$\\textbf{Leveraging additional supervision signals}$: \n\nWe would like to point out that being able to efficiently leverage additional supervision signals (when available) is part of our contributions. Previous works either cannot incorporate these supervisions, e.g., OPUS [1] does not support rendering, or does it in a very expensive way, e.g., RenderOcc uses volume rendering which is both computationally and memory expensive. Moreover, many of these existing works do not model dynamic objects, making it not possible to properly apply the extra supervisions like rendering loss.\n\nAlso, we note that we do not use more ground-truth information as compared to other recent occupancy prediction methods. In order to create the occupancy labels, 3D object segmentation ground truths are used (see Occ3D [3]) which is actually more expressive than 3D bounding boxes. We obtain 2D sparse depth and semantic maps by projecting from LiDAR and the ground-truth 3D segmentation, which have also been used to create occupancy labels in existing Occ3D benchmark. \n\n[1] Wang, Jiabao, et al. \"Opus: occupancy prediction using a sparse set.\" Advances in Neural Information Processing Systems. 2024\n\n[2] Liu, Haisong, et al. \"Fully sparse 3d occupancy prediction.\" European Conference on Computer Vision. 2024.\n\n[3] Tian, Xiaoyu, et al. \"Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving.\" Advances in Neural Information Processing Systems. 2023\n\n$\\textbf{Whether each dynamic query corresponds to a single semantic class}$: \n\nWe do not impose such a constraint explicitly in the model. However, each dynamic query is implicitly encouraged to capture one object in our learning framework. This is because when motion compensation is applied to a dynamic query, the corresponding set of points move the same way, and we perform motion compensation for multiple temporal steps. In this way, the set of points corresponding to the same query learn to match with the same dynamic object in the scenes, as they always move together. We can provide a visualization of this in the revised paper (unfortunately this year's NeurIPS does not support uploading a PDF for rebuttal).\n\n$\\textbf{If only occupancy labels are used for supervision}$: \n\nWe conducted experiment with ODG-T and trained for 24 epochs on nuScenes where only occupancy labels are used in training. The results are listed in Table 1 below. We can see when only using occupancy labels as supervision, ODG-T experienced a minor drop in performance but is still performing respectably well, validating the robustness of our designs.\n\n**Table 1. Ablation study on different supervision signals with ODG-T on nuScenes**\n| Supervision    | mIoU | RayIou | \n| -------- | ------- | ------- | \n| occupancy only  |  31.46   | 36.1\n| occupancy \\& bbox \\& rendering supervision |  32.82  | 38.5\n\n$\\textbf{Highlighting error in Table 1}$: \n\nWe have corrected the errors and the changes will be reflected in the final version."}, {"review_id": "QGwMKfQ43N", "reviewer": "Reviewer_XGZs", "summary": "This paper tackles 3D occupancy prediction through a detr-like query-based approach. The paper proposes using two sets of queries: one set handling the static environment, and another set handling the dynamic objects. These two types of queries are handled differently, where the static environment is ego-motion compensated over time, and the dynamics predict their own speed to also compensate for object movement.\nThey further utilise a hierarchical approach where the number of Gaussians increases progressively through the transformer-based neural network. They further leverage Gaussian splatting techniques to use rendering as an auxiliary task.\nThe paper shows strong performance on both nuScenes and Waymo datasets.", "questions": "- Is it possible to show that the network learns how to disentangle the scene?\n- Can the velocity estimations be evaluated and visualised?\n- Can the effects of the paper's design choices be further validated?", "limitations": "Yes. Could probably add some visualizations of failure cases to sup.mat.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "_Stengths_\n- Decomposition via Query Types: The paper uses a disentanglement of the scene into static and dynamic components via separate query sets. It is intuitive and aligns well with the physical structure of driving environments.\n- Hierarchical Occupancy: The use of progressively increasing Gaussians through the transformer layers is an interesting design choice that (I suppose) incrementally improves the resolution further down in the network. Seems intuitive that this builds a bias in the network to focus on the overall structure first and then focus on the fine-grained details.\n- Auxiliary Rendering Task: Incorporating Gaussian rendering as an auxiliary supervision signal adds an extra layer of regularization and interpretability, further improving the model’s structure.\n- Strong Empirical Evaluations: The method achieves competitive performance on both nuScenes and Waymo datasets, suggesting good generalizability across benchmarks. Also show performance on both IoU and RayIoU.\n\n\n_Weaknesses_\n- Lack of visualisations for Decomposition: Given that the static/dynamic decomposition is a central contribution, it would be valuable to include qualitative visualisations demonstrating how well the model separates and models each component. Is it possible to visualize the scene given the static queries alongside the dynamic queries?\n- Lack of evaluation of hierarchical occupancy: Nothing verifies that the network actually starts with focusing on high-level occupancy and then refines it through the transformer layers. Showing metrics or visualisations from different layers would improve the understanding of the behaviour of the model.\n- Query Efficiency Comparison Missing: It would strengthen the paper to compare the number of queries used with other query-based occupancy prediction methods. This would contextualise the efficiency and scalability of the proposed approach. Furthermore, would it benefit the paper to show that it’s the separation between static and dynamic that make the method work well. For instance, what happens if all queries (the static ones too) are treated like the dynamic queries?\n- Velocity estimation evaluation: Since learning how to compensate for the moving objects it would be very interesting to evaluate (both quantitatively and qualitatively) how well these match with the ground truth. Understanding how well the model learns velocity/flow would significantly increase my takeaways from the paper.\n- Inconsistent Metric Reporting: For instance, Table 4 lacks mean RayIoU over thresholds, which is used elsewhere. Consistent metric reporting across all tables would make interpretation clearer and comparison more fair.\n- Ablation Limitations: The ablation studies do not fully disentangle the contributions of rendering supervision, motion compensation, and attention mechanisms. For instance, an ablation with only rendering supervision (without motion compensation or query attention) would help isolate its specific contribution. \nIt’s unclear how stable the dynamic motion compensation is (the improvement is rather close to variations due to training noise in Table 5). Does the learned behaviour vary significantly across training runs? Reporting results across multiple seeds would clarify this.\nAn analysis of how the number of temporal input frames affects performance would clarify the model’s temporal robustness.\n- Writing and notation polishing: The writing feels a bit rushed and could need some extra polishing. The equation and notations could also be improved. For instance, are (12)-(13) really necessary? Some definitions missing: P is not defined in eq (16)....", "was_revised": true, "final_justification": "Adding the evaluation on intermediate layers as the resolution increases highlights the models capability to incrementally refine the occupancy. I also appreciate the experiment on understanding the velocity of moving objects and robustness to stochasticity in the training process and think these improves the paper substantially.\n\nWhile not ground-breaking, this paper makes a relevant contribution to the field.", "rebuttal": "We thank reviewer XGZs for the constructive feedbacks. We address the reviewer's questions and comments below.\n\n$\\textbf{Visualize the scene with static and dynamic queries prediction alongside each other}$: \n\nWe have prepared such visualizations and will include them in the final version to further demonstrate the superiority of our proposed ODG. Unfortunately, this year's NeurIPS rebuttal format does not support uploading a PDF for the rebuttal and as such, we are not able to share visualization results during the rebuttal stage.\n\n$\\textbf{Metrics of predictions from each layer}$: \n\nWe evaluated predictions of all 6 layers of ODG-T against ground-truth occupancy and computed metrics. The results are summarized in Table 1 below. We can clearly see that in the beginning since the prediction is too coarse, the model behaves poorly in terms of mIoU and RayIoU. Then with our coarse-to-fine refinement, ODG starts gradually learning the scene and gives progressively better results, which validates the efficacy of our coarse-to-fine refinement design.\n\n**Table 1. Evaluation of predictions of all 6 layers of ODG-T on nuScenes.**\n| ODG-T    | #Predicted Gaussians | mIoU   | $\\text{RayIoU}_{1m}$ |  $\\text{RayIoU}_{2m}$  | $\\text{RayIoU}_{4m}$ | $\\text{RayIoU}$ |\n| -------- | ------- | -------- | ------- | -------- | ------- | ------- |\n| layer 1 | 600    | 0.35  | 1.9    | 2.8    | 3.8    | 2.8    | \n| layer 2 | 2400     | 3.17 |   9.5  | 13.5  | 16.3  | 13.0    |\n| layer 3   | 9600    | 18.68    | 25.9 | 32.6   | 36.6  | 31.7  |\n| layer 4   | 12800    |  26.48   | 29.9    | 37.9  | 42.2 | 36.6   |\n| layer 5   | 38400  | 31.02  | 30.5  | 39.9  | 43.3   | 37.9  |\n| layer 6   | 76800  | 32.82    | 31.1    | 40.5  | 43.8  | 38.5  |\n\n$\\textbf{Query efficiency comparison}$: \n\nWe compare with OPUS [1] which is a SotA query-based method for occupancy. The results are summarized in Table 2 below. We note that to ensure a fair comparison, the results of OPUS are directly obtained from their paper. It is clear that with the same number of total queries, ODG outperforms OPUS with significant margin, while only incurring a minimal overhead. For treating all queries as dynamic (denoted as ODG* in Table 2), we can see that even though ODG experienced a minor drop in terms of mIoU and RayIoU, it still definitively outperformed baseline OPUS. We attribute this gain to the object detection task present in our system which effectively improves dynamic object predictions, and through rendering constraint we also learn better 3D geometry.\n\n**Table 2. Query efficiency comparison.**\n| method    | #queries | mIoU    | RayIoU | FPS\n| -------- | ------- | -------- | ------- | ------- |\n| OPUS-T  | 600 (static)  | 33.2 | 38.4  | 22.4\n| OPUS-L | 4800 (static)   |  36.2 | 41.2   | 7.2\n| ODG-T | 500 (static) + 100 (dynamic)  |  35.5 | 39.2  | 20.1\n| ODG-L | 4000 (static) + 800 (dynamic) |  38.2  | 42.3  | 4.9\n| ODG-T* | 600 (dynamic)  |  34.8 | 38.9  | 17.6\n| ODG-L* | 4800 (dynamic) |  37.6  | 42.1  | 3.7\n\n[1] Wang, Jiabao, et al. \"Opus: occupancy prediction using a sparse set.\" Advances in Neural Information Processing Systems 37 (2024): 119861-119885.\n\n$\\textbf{Velocity estimation evaluation}$: \n\nSince our velocity is approximated by ego vehicle at different image timestamps and there is no velocity ground-truth, hence direct quantitative evaluation is not feasible in this case. However, we validate the effectiveness of the dynamic queries predictions through the following surrogate experiment on nuScenes: given current timestamp $t$ (keyframe), we denote occupancy ground-truth $O_t$ and predictions $O_t^p$. We denote the immediate previous timestamp $\\hat{t}$ (also keyframe, where occupancy GT is available) and its occupancy GT $O_{\\hat{t}}$ and predictions $O_{\\hat{t}}^p$. Then we warp prediction at current time $O_t^p$ to previous time using ego-motion for static queries and velocity for dynamic queries and denote as $O_{t\\rightarrow\\hat{t}}^p$. We evaluation both $O_{\\hat{t}}^p$ and $O_{t\\rightarrow\\hat{t}}^p$ against ground-truth $O_{\\hat{t}}$ at time $\\hat{t}$. The results are summarized in Table 3 below:\n\n**Table 3. Surrogate experiment of evaluating velocity estimation.**\n| Pred    | GT | mIoU    | RayIoU |\n| -------- | ------- | -------- | ------- |\n| $O_t^p$  | $O_t$  |  32.82 | 38.5  |\n| $O_t^p$|  $O_{\\hat{t}}$  | 26.77 |  36.1  |\n| $O_{t\\rightarrow\\hat{t}}^p$  | $O_{\\hat{t}}$  |  30.74 |  37.4  |\n\nWe can clearly see that due to motion misalignment if we directly evaluate prediction $O_{t}^p$ against GT $O_{\\hat{t}}$ at previous time, there is a significant degradation in terms of mIoU and RayIoU. But if we warp current prediction $O_t^p$ to previous time $O_{t\\rightarrow\\hat{t}}^p$ using estimated velocities of the queries, such misalignment is corrected and we obtain significantly improved results. This demonstrates the usefulness of our estimated velocity.\n\nWe note that we will add velocity flow visualization in the final version, given during the rebuttal stage it is not feasible for us to share any type of rich media such as images and videos.\n\n$\\textbf{Ablation on only using rendering supervision}$: \n\nWe conducted such ablation experiment with ODG-T (trained for 24 epochs on nuScenes, consistent with Table 4 in main paper) and the results are summarized in Table 4 below:\n\n**Table 4. More ablation studies of ODG-T including only using rendering supervision. Trained for 24 epochs on nuScenes.**\n| Motion compensation    | Query Attention | Rendering Sup | $\\text{mIoU}$ | $\\text{RayIoU}_{1m}$ | $\\text{RayIoU}_{2m}$ | $\\text{RayIoU}_{4m}$ | $\\text{RayIoU}$\n| -------- | ------- | -------- | -------- | -------- | -------- | -------- | -------- | \n|   |     |  | 30.80 | 27.9 | 36.6 | 42.0 | 35.5 |\n|  |     | yes | 31.29 | 28.3 | 36.9 | 42.2 | 35.8|\n| yes |     |  | 31.78 | 28.7 | 37.3 | 42.6 | 36.2 |\n| yes  | yes    |  | 32.13 | 29.3 | 37.8 | 43.1 | 36.7\n| yes  | yes    | yes | 32.82 | 31.8 | 41.2 | 44.5 | 38.5 |\n\nwe can see that even with only rendering supervision, our method is able to achieve a modest improvement, validating the effectiveness of our design.\n\n$\\textbf{Inconsistent metric reporting on RayIoU}$:\n\nWe have corrected such inconsistency by adding $\\text{RayIoU}$ as last column in Table 4 in the main paper. The changes will be reflected in the final version.\n\n$\\textbf{Stableness of improvement coming from dynamic motion compensation}$: \n\nWe launched another three separate experiments with distinctively different seeds to validate the stability of improvement. The results are summarized in Table 5 below. We can see that the metrics yielded across runs with different seeds are very stable, demonstrating the validity of the improvement from dynamic motion compensation. To facilitate reproducing, experiments were conducted using PyTorch 2.1.2, NVIDIA Driver 535.161.08 with CUDA 12.1 on A100 GPUs.\n\n**Table 5. Multiple experiments of ODG-T with different seeds to validate stableness of improvement coming from dynamic motion compensation. Trained for 24 epochs on nuScenes.**\n| Seed    | Ego Comp | Dyn. Comp  | mIoU | RayIoU |\n| -------- | ------- | ------- | ------- | ------- |\n| 1024 (paper)  |  yes |  |  31.17   |  35.7   |\n| 1024 (paper) | yes    | yes  | 31.78    | 36.2\n| 42   |  yes |   | 31.17   |  35.7   |\n| 42 |  yes  | yes | 31.78    | 36.2\n| 3407 |  yes  | | 31.18  | 35.7     |\n|  3407  | yes  | yes   | 31.80   | 36.2\n| 2025  | yes   |  | 31.17   |  35.7   |\n| 2025  | yes  | yes | 31.78    | 36.2\n\n$\\textbf{Writing and notation polishing}$: \n\nWe have condensed Eq.12-13 into a single-line text description and added the definition of $P$ in Eq. 16. The changes will be reflected in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a dual-query architecture comprising of two distinct sets of Gaussian queries to separately model the static and dynamic parts of the scene. A cross query attention is also introduced to establish interaction between queries, enhancing 3D occupancy prediction.    

Strength:  
Writing is good (cwc4, M77i), the performance is good (cwc4, JcRJ, M77i, XGZs), good balance between performance and computational complexity (cwc4, JcRJ, M77i), technical innovation is strong (JcRJ).

Weakness:   
More implementation details are needed (cwc4, JcRJ), more detailed results are needed (XGZs). Novelty is not strong or ground-breaking (M77i, XGZs).   

After rebuttal, this paper receives 4444. All the reviewers are satisfied with authors' response and give positive feedback. The AC agrees with reviewers, although not ground-breaking, this paper makes a relevant contribution to the field with its own novel designs. Please include the additional results and modifications here into later versions.
Decision: Accept (poster)