OPUS: Occupancy Prediction Using a Sparse Set
Authors: JiaBao Wang
Zhaojiang Liu
Qiang Meng
Liujiang Yan
Ke Wang
JIE YANG
Wei Liu
Qibin Hou
Ming-Ming Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Occupancy Prediction; 3D Perception; Transformer
Primary area: machine_vision

Abstract: Occupancy prediction, aiming at predicting the occupancy status within voxelized 3D environment, is quickly gaining momentum within the autonomous driving community. Mainstream occupancy prediction works first discretize the 3D environment into voxels, then perform classification on such dense grids. However, inspection on sample data reveals that the vast majority of voxels is unoccupied. Performing classification on these empty voxels demands suboptimal computation resource allocation, and reducing such empty voxels necessitates complex algorithm designs. To this end, we present a novel perspective on the occupancy prediction task: formulating it as a streamlined set prediction paradigm without the need for explicit space modeling or complex sparsification procedures. Our proposed framework, called OPUS, utilizes a transformer encoder-decoder architecture to simultaneously predict occupied locations and classes using a set of learnable queries. Firstly, we employ the Chamfer distance loss to scale the set-to-set comparison problem to unprecedented magnitudes, making training such model end-to-end a reality. Subsequently, semantic classes are adaptively assigned using nearest neighbor search based on the learned locations. In addition, OPUS incorporates a suite of non-trivial strategies to enhance model performance, including coarse-to-fine learning, consistent point sampling, and adaptive re-weighting, etc. Finally, compared with current state-of-the-art methods, our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.

OPUS: Occupancy Prediction Using a Sparse Set
Jiabao Wang's Zhaojjang Liu Qiang Meng'. Liujiang Yan Ke Wang", Jie Yang",
Wei Liu?, Qibin Hou Ming-Ming Cheng1.4
IVCIP, College of Computer Science, Nankai University
Shanghai Jiao Tong University 'KargoBot Inc. 4NKIARI, Shenzhen Futian https://gi //tgggtthh...../. com/ Joaa///aan199//PPSS
Abstract
Occupancy prediction, aiming predicting the occupancy status within voxelized 3D environment, quickly gaining momentum within the autonomous driving community. Mainstream occupancy prediction works first discretize the 3D envi- ronment into voxels, then perform classification on such dense grids. However, inspection on sample data reveals that the vast majority of voxels is unoccupied. Performing classification on these empty voxels demands suboptimal computa- tion resource allocation, and reducing such empty voxels necessitates complex algorithm designs. To this end, we present a novel perspective on the occupancy prediction task: formulating it as streamlined set prediction paradigm without the need for explicit space modeling or complex sparsification procedures. Our proposed framework, called OPUS, utilizes transformer encoder-decoder archi- tecture to simultaneously predict occupied locations and classes using set learnable queries. Firstly, employ the Chamfer distance loss to scale the set- to-set comparison problem to unprecedented magnitudes, making training such model end-to-end a reality. Subsequently, semantic classes are adaptively assigned using nearest neighbor search based on the learned locations. In addition, OPUS incorporates suite of non-trivial strategies to enhance model performance, includ- ing coarse-to-fin fine learning, consistent point sampling, and adaptive re- re-weighting, etc. Finally, compared with current state-of- off-the-ar art methods, our lightest model achieves superior RayloU on the - dataset at near x FPS, while our heaviest model surpasses previous best results by 6.1 RayloU.
1 Introduction
Compared with well-established box representations [8, 23, 20, 39, 29, 48, 32], voxel based occu- pancy [16, 37, 10, 38, 33] can provide finer geometry and semantic information for the surrounding scene. For example, it is not straightforward to use bounding boxes to describe vehicles with doors open or cranes with outriggers deployed. While occupancy can naturally describe such uncommon shapes. Thus occupancy prediction is quickly gaining traction in the autonomous driving community.
Recent approaches [3, 46, 27, 16, 34] the task predominantly rely on dense data representation, with a direct one-to-one correspondence between feature points and physical voxels. It has come to attention that the vast majority of physical voxels is empty. For instance, in SemanticKITTI [1], approximately 67% of all voxels are empty, while in Occ3D-nuSeenes [38], this proportion exceeds 90%. Such sparse nature of occupancy data renders the direct dense representation undeniably inefficient, as majority of the computation is allocated towards empty voxels. Alternative sparse latent representations have been explored alleviate such inefficiency, such as the Tri-Perspective View representation [37, 9] or reduced solution spaces [21, 10], leading to notably reduced computational costs. However, these approaches still treat occupancy prediction as a classification problem specific locations, necessitating complex intermediate designs and explicit modeling of 3D spaces.
"Equal contribution. *Correponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

prediction
P
Fa
ground-rruh
Chamfer distance
classification loss
s: (2, Cy
aligned semantic labels
set prediction problem. For each scene, we predict a set of point positions P and set of the corresponding semantic classes C. With the ground-truth set of occupied voxel positions Pg and classes Cg. we decouple the set-to-set matching task into two distinet components: (a) Enforcing similarity in the point distributions of P and 9 using the Chamfer distance. (b) Aligning the predicted classes C with the ground-truths C = (T,, TP Cg). where + generates a set of classes for points P based on those of the nearest ground-truth points.
Figure The occupancy prediction is approached
In this work, we instead formulate the task as a direct set prediction problem, where we regress occupied locations and classify corresponding semantic labels in parallel. Our proposed framework termed OPUS leverages a transformer encoder-decoder architecture featuring: (1) an image encoder to extract 2D features from multi-view images; (2) set of learnable queries to predict occupied locations and semantic classes; (3) a sparse decoder to update query features with correlated image features. Our OPUS eliminates the need for explicit space modeling complex sparsification procedures, offering a streamlined and elegant end-to-end solution. However, a key challenge lies in matching predictions with ground-truths, especially given the unordered nature of predicted results. We argue that the Hungarian algorithm [12], although widely adopted in the DETR families [5, 47, 28, 22, 13, 42], is not suitable for this task. Having (nn²) time complexity and (n?) space complexity, the Hungarian algorithm is unable to handle a substantial number of voxels. In our experiments, associating two sets with 10K points each, the Hungarian algorithm consumes approximately 24 seconds and 2,304Mb of GPU memory on a 80G A100 GPU. In reality, the voxel number can go up to 70K in the Oce3D-nuSeenes [38] dataset. Thus directly applying the Hungrian algorithm for set-to-set matching is infeasible in the occupancy prediction context.
But is accurate one-to-one association truly necessary for occupancy prediction? We recognize that the goal of one-to-one correspondence between prediction results and ground-truth annotation is to obtain supervision signals, essentially complete, precise point locations, and accurate point classes. The heavylifting of one-to-one association can be entirely avoided if we can obtain such supervision signals elsewhere. Therefore, we propose to decouple the occupancy prediction task into two parallel subtasks, as illustrated in Fig. 1. The first task obtains supervision predicted point distributions with ground-truths, a task achievable through the Chamfer distance loss, well-established technique for point clouds [6, 30]. The second task obtains supervision classes by assigning semantic labels to predicted points. This is accomplished by assigning each point the class of its nearest neighbor in the ground-truthe It's noteworthy that all operations involved can be executed in parallel and are highly efficient on GPU devices. As a result, a single matching in Occ3D-nuScenes can be processed within milliseconds, with negligible memory consumption. With a time complexity of (n²) and space complexity of Onz), our formulation breaks the ground for large-scale training for the occupancy prediction models.
point locations by aligning
point
In addition, we propose several strategies to further boost the performance of occupancy prediction in our end-to-end sparse formulation, including coarse-to-fine learning, consistent point sampling, and adaptive loss re-weighting, On Occ3D-nuScenes, all our model variants easily surpass all prior work, verifying the efficacy and effectiveness of the proposed method. Especially, our most lightweight model achieves 3.3 absolute RayloU improvement compared with SparseOce [21] while operating more than 2x faster. The heaviest configuration ultimately achieves a RayloU of 41.2, establishing a new upper bound with a 14% advantage. Our contributions are summarized as follows:
To the best of our knowledge, for the first time, we view the occupancy prediction as a direct set prediction problem, facilitating end-to-end training of the sparse framework.

---

Several non-trivial strategies, including coarse-to to-fee learning, consistent point sampling, and adaptive re-weighting. are further introduced for boosting the performance of OPUS.
Extensive experiments Occ3D-nuScenes reveal that OPUS can outperform state-of-the-art methods in terms of RayloU results, while maintain a real-time inference speed.
2
Related work
2.1 Occupancy prediction
Occupancy prediction entails determining the occupancy status and class of each voxel within a 3D space. This task has recently become foundational perception task in autonomous driving and raises great interests from both academic and industrial communities. Conventional methods [3, 46, 9, 27, 16, 41, 38, 4] typically employ the continuous and dense feature representation, which, however, suffer from computational redundancy due to the inherent sparsity of occupancy data. In addressing this issue, Tang et al. [37] compresses the dense feature using the Tri-Perspective View representation for model efficiency. Recently, several transformer- based approaches [21, 10, 14] with sparse queries have emerged. For example, OccupancyDETR [10] conducts object detection followed by assigning each object with one query for occupancy completion. VoxFormer [14] generates 3D voxels from set of sparse queries, corresponding to occupied locations identified through pre-task of depth estimation. Meanwhile, SparseOcc [21] employs a series of sparse voxel decoders to filter out empty grids and predict occupied statuses of retained voxels each stage. While these approaches have succeeded in reducing computational costs, they often necessitate multi-stage processes and intricate space modeling. In contrast, our method directly applies sparse queries to regress the occupied locations without pre-defined locations, facilitating an elegant and end-to-end occupancy prediction.
2.2
Set prediction with transformers
The concept of directly predicting sets with Transformers was initially introduced by DETR [5]. where set of sparse queries generates unordered detection results with feature and object interactions. By viewing the object detection as direct set prediction problem, DETR eliminates the need for complex post-processing, enabling end-to-end performance. Following DETR, numerous variants [47, 28, 22, 13, 42, 45, 36] have been proposed for performance improvements and efficient training. The effectiveness of the sparse-query-basee paradigm has also been validated in 3D object detection [43, 23, 18-20, 40], where 3D information is encoded into the queries. For example. DETR3D [43] employs sparse set of 3D object queries index 2D features, linking 3D positions to multi-view images using camera transformation matrices. PETR [23] generates 3D position-awe aware features by encoding 3D position embedding into 2D image features, enabling queries to directly aggregate features without the 3D-to-2D projection. Sparse4D [18] further advances sparse 3D object detection by refining detection results with spatial-temporal feature fusion. Despite of the great success, set prediction with Transformers remains restricted primarily to object detection, where the query number are typically small due to the limited object number in scene. Extending this approach to occupancy prediction poses a big challenge due to the substantially larger number of queries required.
Methodology
In this part, we first recap current query-based sparsification approaches for occupancy prediction in Sec. 3. Then, Sec. 3.2 describes our formulation that views the task direct prediction problem. Finally, we detail the proposed OPUS framework in Sec. 3.3.
3.1 Revisiting query- occupancy sparsification
Transformers with sparse queries offer a promising avenue for tackling the inherent sparsity in occupancy representation. A notable approach to reduce the number of queries is allocating each query patch of voxels rather than a single voxel, as presented in PETRv2 [24]. However, this method still generates a dense prediction of the 3D space, thus failing to efficiently address the redundancy issue. Alternatively, VoxFormer [14] and SparseOce [21] allocate sparse queries exclusively to occupied voxels. VoxFormer employs a depth estimation module to identify potentially occupied voxels, while SparseOcc utilizes multiple stages to progressively filter out empty regions. Nonetheless, their sparsification processes rely accurately recognizing the occupancy status of voxels and therefore suffer from the cumulative emrors. Moreover, their pipelines necessitate intricate intermediate descriptions of the 3D space, hindering seamless end-to-end operation.
3

---

The dilemmas of current approaches significantly stem from treating the task as classification problem, where each query is confined specific physical region for classifying the semantic labels. This constraint severely limits query flexibility, preventing adaptive focus on suitable areas. To address this, we propose to remove this restriction by allowing each query to autonomously determine its relevant area. In the end, we view occupancy prediction as direct set prediction problem, where each query predicts point positions and semantic classes, simultancously.
3.2 A set prediction problem
At the core of our work lies the conceptualization of occupancy prediction as a set prediction task. We denote the Vg occupied voxels in the ground-truth as {Pgg Cg where |P, p -ol=ic [Cgl Vg. For each entry in (Py: Cg) E (Tp Cg Pg represents the 3D coordinates of voxel center, while cg stores the semantic class of the corresponding voxel. Given the predictions {P, C) of V points, our primary challenge is to devise an effective strategy for set-to-set matching. In other words, we must determine how to supervise the training of unordered predictions with the ground-truth data. One alternative is to adopt the Hungarian algorithm. However, our previous discussions and experiments in the appendix reveal its scalability limitations. Rather than pursuing one-to-one associations between the predictions and ground-truths we recognize the matching essentially aims at accurate locations and classes in predictions. This motivates to decouple the task into two parallel objectives: (1) Encouraging the predicted locations to be precise and comprehensive. (2) Ensuring the predicted points are assigned with proper semantic classes from the ground-truth labels.
The first objective focuses on aligning distributions between predicted and ground-truth points, task achievable through the Chamfer distance loss which is well-proved in the field of point clouds [6, 30, 11, 44]:
CD(P, = PP
Σ D(p.P)) PEp
I Σ D(P,. where D(x, Y) min P.
|x yll1-
(I)
ye
PoEl'
Minimizing Chamfer distance leads to similar distributions of predictions and fround-truths, enabling direct learning of occupied voxels without necessitating knowledge of their orders.
Concerning the second objective, although direct comparison between C and Cg is invalid due to their correspondence to different locations, we can leverage the spatial locality properties of voxels to find a proxy. Nearby points belonging to the same object usually carry the same semantic labels, thus we propose assigning each predicted point the class of its nearest neighbor voxel in the ground-truth:
( = arg min (c
|Ip plla,
p PEP..
Here, Ć is the updated classes that are prepared to supervise the learning of the predicted C.
It's noteworthy that computations of both Eq. (1) and Eq. (2) can be executed efficiently and in parallel on GPU devices. As a result, a single matching can be swiftly processed within milliseconds, enabling feasibility of the large-scale training by viewing the occupancy prediction task direct set prediction problem. Next, we delve into the specifics of the proposed OPUS framework.
3.3 Details of OPUS
This part describes OPUS framework, as illustrated in Fig. 2. Initially, image features are extracted from multi-view images. And a set of learnable queries Q, point positions P. and scores C are initialized. Subsequently, these query features and prediction outcomes are fed into a sequence of decoders, undergoing iterative refinement through correlation with image features. At each stage, predicted positions and scores are supervised by the ground-truths, facilitating end-to-end training for the entire framework. can be observed that our most important structure the sequence of multiple decoders. Therefore, we next provide detailed description to the inputs/outputs of the decoders and how features are aggregated and updated within the decoders.
Denote the set of learnable queries, point positions, and point scores as (Qo. Pu. Co} before feeding into decoders, and as {Q,, P,, C.} for the outcomes of the i-th decoder. The length of these sets is all Q, which corresponds to the number of queries. Each query feature qi E Q. e (0, 1.... 6} has a channel size C, set to 256 in our implementation. To reduce the number of queries, which is a bottleneck for model efficiency, each query 4 predicts R2 points rather than a single one. Consequently, P; E IP, and c, E C, have shapes of Q R₁ x 3 and Q x R₂ x N, respectively. Here, N represents the number of semantic classes.
Notations.

---

Inttidiiaation
Consistint Point Sumpling
Queries
positions
Adaptiive Mixing
Decoder Layer
image feuzures
R. point
Aduptive Self Attention
Image Encoder
Addd Norm
Decoder Layeri i
Food--Forward Nenwork
poin
Addd & Norm
ground truth
image fouture park
Decoder Layer 6
query path
Q x Raponss
we-ihitteed losser
mulli-camera imagas
Figure 2: OPUS leverages a transformer encoder-decoder architecture comprising: (1) An image encoder extract 2D features from multi-view images. (2) A series of decoders refine the queries with image features, which are correlated via the consistent point sampling module. (3) A set of learnable queries to predict locations and classes of occupancy points. Each query obeys a coarse- to-fine rule, progressively increasing the number of predicted points. In the end, the entire model is trained end-to-end using our adaptively re-weighted set-to-set losses.
Coarse-to-fine prediction. High-level semantic information can be difficult to predict accurately from just low-level features. Therefore, instead of attempting to predict occupancy for the entire 3D environment, we allow the model to predict "sparse" occupancy results in early stages, as shown in Fig. 2. To achieve this, we follow coarse-to-fine strategy, gradually increasing the number of points generated from one query. In other words, we always have R S R3 for i E {1,2, ...,,...,00 .6].
It's noteworthy that the Chamfer distance has another advantage even when the number of predictions is smaller than that of the ground-truths, the assignment won't collapse into a local shape of the ground-truths. This is because the Hungarian algorithm could assign the predictions to any subset of the ground-truths due to its lack of distribution constraints. In contrast, the Chamfer distance maintains global perspective, considering the overall distribution of points rather than enforcing a strict one-to-one correspondence. This ensures that the predicted points are more evenly distributed and representative of the actual 3D environment, even when fewer in number.
the Hungarian algorithm here:
Details of the decoder. Our decoder is analogous to that Spa [20], a performant and sparse object detector. For a given query q-1 Qi-1 and its corresponding point locations Pi-1 E 1P the i-th decoder first aggregates image features through a consistent point sampling, new scheme elaborated in subsequent discussion. Subsequently, the query feature is updated into q, with the adaptive mixing of image and query features, along with the self-attention among all queries, mirroring operations in SpV In the end, prediction module, comprising only Linear, LayerNorm, and ReLU layers, generates the semantic classes c; (size R² x N) and the position offsets Ap, (size R² x 3). As the Ap; cannot be directly added to P;-1 due to dimension misalignment, first compute the mean of p along the first dimension and then duplicate the results by R, times into -1 The final position p; is computed as p. p;-1 + Ap:.
Consistent point sampling. The feature sampling method utilized in SparseBEV is not applicable for our method as it is specifically designed for detection inputs. Therefore, we propose novel process of Consistent Point Sampling (CPS), aiming at sampling 3D points and aggregating features from M image features. Given input {q, p) E QQ, PJ. sample S points and find their respective coordinates in the m-th image feature by the following equation:
Cm Tmr, where m, + ②(q) Op:
where Tm represents the projection matrix from current 3D space into the m-th image's 's coordinates. ((q) generates S 3D points from the query feature q using a linear layer. mp and on denote the mean
5

---

and standard deviation, respectively, of the R points in p. It's worthy to note that we re- weight the predicted offsets ((q) with the standard deviation op to inherent the dispersion degree from previous predictions. In essence, we tend to sample more aggressively if the input contains diverse points, and sample points in a narrower range otherwise. This operation can evidently enhance the prediction performance, as demonstrated in our experiments.
Not all coordinates in Cm. are feasible since the sampled points might not be visible within the corresponding camera. Therefore, we generate a mask set Vm where the s-th value is if Ca,m is valid and 0 otherwise, for s (1,2....., ,S} and m (1, .M. Next, we aggregate information from image features {Fm M for the later adaptively mixing stage. Specifically, we have
M ΣΣ Ws.m Us, (Bmm Cs,m).
1
(4)
M m==
where Va.m denotes the s-th element in Wm and Ca is the coordinates of the s-th point r, mapped into the m-th image feature. The operation B refers to the bilinear interpolation. Ws,1 is the weight for the rs on the m-th image feature, generated from the query feature q by linear transformation.
The training loss with adaptively re-weighting. The training object of our framework is supervise the learning of (L" C. with the ground-trutt {IP Cg Point positions can be trained with Eq. (1). However, the original Chamfer distance loss focuses on the overall similarity of point distributions, neglecting whether each individual is good enough. This leads to unsatisfactory performance, as observed in our experiments. To cope with this issue, we employ a simple but effective re- weighting strategy to emphasize erroneous points, and modify the Chamfer distance loss as follows:
CD,(P,P,)
= IP|
1 Σ Dx(p.Po)
+ IPol
Σ EyE
DR(Po:P),
DEP
(5)
where DR(x, (d) d with d
min yet
- y-ll--
Here, (dd) is the re-weighting function penalizing points with large distance the closest ground- truths. In our implementation, we use a step function of W (d) being 5 if 2 0.2 and otherwise.
For the classification, we first generate the target classes C, for C, using Eq. (2). Subsequently, the semantic classes can be trained with the conventional classification losses. In our implementation, we adopt the focal loss [17] with mannually searched weights on different categories and denote the modified loss as FocalLoss In the end, the training objective of the proposed OPUS becomes
LopUs CDn(Fo, Pa) + Σ (CDn(FF,, P2) + (oc, C.)),
(6)
where CD, Pp) explicitly encourages initial points Po to capture a general pattern of the dataset.
Experiments
4.1 Experimental setup
Dataset and metrics. All models are evaluated on the Oce3D-nuSceens [38] dataset, which pro- vides occupancy labels for 18 classes free class and 17 semantic classes) on the large-scale nuScenes [2] benchmark. Out of the 1,000 labeled driving scenes, 750/150/150 are used for train- ong/validatiiontestinng respectively. The commonly used mIoU metric utilized for evaluation. Recently. SparseOcc [21] points that that overestimation can easily hack the mloU metric and pro- poses RayloU as remedy. Therefore, following their work, we also report the RayloU results under different distance thresholds at 1, 2, and 4 meters, denoted as RayloU Im RayloU 2m and RayloU 4m respectively. The final RayloU score is the average of these three values.
Implementation details. Following previous works [21, 16, 8], we resize images to 704 x 256 and extract features using ResNet50 [7] backbone. We denote a series of models as OPUS-T, OPUS-S, OPUS-M and OPUS-L, with 0.6K, 1.2K, 2.4K and 4.8K queries, respectively. In each model, all queries predict an equal number of points, totalling 76.8K points in the final stage. The sampling number in our CPS is 4 for OPUS-T and 2 for other models. Please refer to Appendix D.2 for more details of different models. All models are trained on 8 nvidia 4090 GPUs with a batch size of using the Adam W [26] optimizer. The learning rate warms up to 2 in the first 500 iterations and then decays with a Cosine Annealing [25] scheme. Unless otherwise stated, models in main results are trained for 100 epochs and those in the ablation study are trained for 12 epochs.

---

Table Oceupancy prediction performance on Occ3D-nuSeenes [38]. "8f" and "16f" denote models fusing temporal information from 8 or 16 frames, respectively. Baseline results are directly copied
from their corresponding papers or the SparseOcc [21]. FPS results are measured on an A A10 GPU.
Methods
Backhone
Image Size
mloU
RayloU Im
RayloU
RayloU
RayloU
FPS
RenderOce [31]
Swin-B
1408 x 512
24.5
13.4
19.6
25.5
19.5
BEVFormer [15]
R101
1600 x 900
39.3
26.1
32.9
38.0
32.4
3.0
BEVDet-Oce [8]
R50
704 256
36.1
23.6
30.0
35.1
29.6
2.6
BEVDet-Ooc (80) [8]
R50
704 384
39.3
26.6
33.1
38.2
32.6
0.8
FB-Occ (16f) [16]
R50
704 256
39.1
26.7
34.1
39.7
33.5
10.3
SparseOce (8F) [21]
R50
704 256
28.0
34.7
39.4
34.0
SparseOce (160) [21]
R50
704 256
30.6
29.1
35.8
40.3
35.1
12.5
OPUS-T (8f)
704 256
31.7
38.4
OPUS-S (8))
R50
704 256
34,2
32.6
39,9
44.7
39.1
20.7
OPUS-M (8f)
R50
704 256
35.6
33.7
41.1
46.0
40.3
13.4
OPUS-I. (8f)
R50
704 256
36.2
34.7
42.1
46.7
41.2
7.2
FB-Occ (160)
SpurseOce (80
OPUS (8f)
ground truth
mlott-39.11 1, R335
mloU--9.66 Raylot
mloti-356 6, Raylol1-40.3
Figure 3: Visualizations of occupancy predictions. Best viewed in color.
4.2 Main results
Quantitative Performances. In this part, we compare OPUS with previous state-of-the methods on the Occ3D-nuSeenes dataset. Our methods not only achieves the superior performances in terms of RayIoU and competitive results in mIoU, but also demonstrates commendable real-time performance. As depicted in Tab. 1, OPUS-T T (8f) reaches 22.4 FPS, significantly faster than dense counterparts and nearly 1.3 times the speed of sparse counterpart SparseOce (8f). Despite using only 7 history frames, its 38.4 RayloU result easily outperforms other models, including FB-Occ (1 with RayloU of 33.5(- and SparseOce (16f) with RayloU of 35.1 (-3.3). Similarly, OPUS-S (8f) and OPUS-M (8f) achieve good balance between performance and efficiency. The heaviest version of OPUS ultimately achieves an RayloU of 41.2, surpassing the previous best result by notable margin of 6.1.
With the same total number of points predicted, we vary the query number and correspondingly change the number of points from each query, leading to different versions of OPUS. can be observed that increasing the query number decreases the FPS values from 22.4 to 7.2, while simultancously boosts model performance in terms of mloU and RayloU. The OPUS-M (8f), with 2.4K queries, strikes a balance by achieving comparable RayloU while maintaining competitive FPS.
Despite the vulnerability of mloU metric to overestimation manipulations [21], our OPUS attains a comparable mloU of 36.2, significantly bridging the gap between dense and sparse models in this metric. These results under different metrics collectively demonstrate the superiority of our OPUS.
Visualization. We visualize the predicted occupancy in Fig. 3. It can be observed that FB-Occ tends to produce denser results compared to sparse methods. Though seems complete in the 3D environment, its predicted occupancy results are severely over-estimated, especially for the far areas. The overestimation may hack the mloU metric [21], while heavily penalized by RayloU that primarily considers the first occupied voxels along rays. Consequently, FB-Occ achieves the best mloU of 39.1 but the worst RayloU value. On the other hand, SparseOcc occasionally exhibits discontinuous predictions with false negatives, especially in long distances. This is attributed SparseOce's gradual removal of empty voxels, making erroneous filtering in early stages accumulates and contributes to the final false predictions. In contrast, our OPUS maintains more continuous prediction thanks to its end-to-end approach, resulting in more reasonable visualization.
4.3 Ablation study and visualizations
This part details our ablation study and visualizations using the OPUS-M (8f) model.
7

---

Table 2: Model performances with different combinations of proposed strategies.
CDR
FocalLoss
CPS
Coarse-to-fine
mloU
RayloU
RayloU
RayloU
RayloU
17.4
23.6
29.7
34.3
29.2
23.7 (6.31)
23.9
30.7
35.6
30.1 (0,9t)
25.1 (1.4f)
25.2
32.3
37.0
31.5 (1.41)
25.5 (0.4t)
26.0
33.1
37.9
32.3 (0.8t)
27.2 (1.71)
26.1
33.3
38.4
32.6 (0.31)
stage
stage
stage
stage 6
(a) baseline
coarse-to-fine
ground -rith
Figure 4: Visualizations of the coarse-to-fine predictions.
Effects of the proposed strategies in OPUS. In our work, we introduce adaptive re- weighting for the Chamfer distance loss and focal loss, along with consistent point sampling, and coarse-to- prediction strategies. We examine the impacts of these strategies as shown in Tab. 2. Without bells and whistles, OPUS achieves a baseline 17.4 mloU and 29.2 RayloU. Replacing the original CD loss into our revision CDR significantly boosts the mloU and RayloU by 6.4 and 0.9, respectively, demon- strating the importance of focusing on erroneous predicted locations in this task. The FocalLoss further improves both metrics by 1.4. Incorporating the term or in Eq. (3) further enhances mloU and RayloU by 0.4 and 0.8, demonstrating the efficacy of considering previous point distribution in the current sampling process. The proposed coarse-to- -oo---ie query prediction gradually increases the number of points across the stages. This scheme not only reduces computations in early stages but also notably benefits model performance, particularly in mloU, which is increased by 1.7. These results highlight the cumulative benefits of each component, showcasing how their integration leads to substantial performance gains.
Visualization on the coarse-to-fin prediction. We visualize the prediction results at different stages Fig. 4.
baseline scenario depicted in Fig. 4(a), where all decoders regress same number of points, we observe inconsistent point distributions across stages and numerous false negative predictions in long distances, as highlighted by circles. This may be attributed to the difficulty of learning the fine-grained occupancy representations in the early stages, impeding the efficient training of the entire framework. In contrast, our coarse-to-fine strategy significantly alleviates the learning difficulty in early stages, thereby leading to improved model performances. As a result, the point distributions are more consistent among different stages, and the final predictions exhibit much fewer false negatives, illustrated in Fig. 4(b).
Visualizations of predicted points. In Fig. 6, we select few queries and visualize their predicted points. Notably, most queries exhibit tendency to predict points with con- sistent classes, or even from the same instance, as depicted in Fig. 6(a)-gg). An interesting observation is that the pre- dicted points tend to exhibit diverse distributions in classes with large volumes, such drivable surfaces and sidewalks. Conversely, for objects with limited sizes, such as traffic cones, motorcycles, and cars, the points distributed more closely with respect to the instance size. The patterns can be further verified by Fig. 5, where we present the standard deviations of points from queries with three chosen classes. These results highlight the efficacy of our model in adapting its predictions to the distinct spatial characteristics various object classes.
Class Vom es fralii modcreeek
Density
videxati
122
1:46
1.0
(mz)
Figure 5: Distributions of standard deviations of points from one query.
As we do not explicitly constrain points from one query to have the same class, it's conceivable that one query could yield points of different classes. We found this phenomenon commonly occurs the boundaries between objects. However, even when classes vary, these points are still closely distributed, as depicted in Fig. 6(h)-(j).
8

---

Table 3: Comparison of various treatments on initial locations po "Grid" and "Random" indicate that points sampled uniformly in BEV space and randomly in the 3D space, respectively "Optimized" means that points are randomly initialized but supervised with ground tround-truthhs via the CDR loss.
Type
mloU
RayloU im
RayloU
RayloU en
RayloU
Grid
22.8
22.2
28.9
33,9
28.3
Random
23.1
23.6
30.5
35.6
29.9
Optimized
23.7
23.9
30.7
35.6
30.1
(a) toffic
(b) matoreycie
drivable surface
sidewaik
monmade
borrier
ridewalk & drnable
terrain drivable
drivable
query points
barrier
bicycle
Ins
truiler
trick
moturcycle
pedestrion
traffic come
construction
drivable
other
sidewalk
ferrain
manmade
vegetation
Figure 6: Visualizations of points generated from different queries. Best viewed in color.
Influence of treatments on the initial points. Tab. 3 compares three different treatments on the initial points po Grid initialization divides the BEV space into evenly-distributed pillars and orderly assigns pillar centers as the initial locations, a method utilized in BEVFormer [15]. Random initialization assigns each location with uniform distribution the 3D space. After initialization, Fn remains learnable during training. On top of the random initialization, our OPUS further add supervisions of the ground- truth distributions po0 (i.e.. CD P2g) in Eq. (6)). The results in Tab, 4 show that random initialization outperforms grid initialization. achieving an mloU of 23.1 compared to 22.8, and RayloU of 29.9 compared to 28.3. This improvement is likely due the fact the random initialization provides more diverse 3D distribution. Furthermore, the introduced supervision results additional improvements of 0.6 on mloU and 0.2 on RayloU. These results reveal the efficiency of the random initialization and the additional supervision on the initial locations.
Visualization of the self-attention. For better understanding what query points attend to, we visualize top 10 query points with highest self attention weights for each query. We project them on 2D image for better visualization. Here are some interesting finding from Fig. 7. Generally speaking, tends to attend to neighbouring query points. For example, the sidewalk query in the first image and the car query the second image, allow local information from neighbouring query points flow into, enabling the query to capture detailed local information. Additionally, the query maintains the ability to attend to semantic related locations even they are not very close to the query point. For instance, the vegetation query in the first image of Fig. 7 not only attends to the stem of the tree, but also the grass, indicating that the query can capture semantically related information for more accurate predictions. Another notable observation is that query sense serrain-related information. For instance, in the third image, the sidewalk query points attend to other points along a straight line following the road's edge, highlighting the model's '' ability to understand scene related structure of the environment.
query
Figure 7: the self-attention in decoders. for each pivot (marked as x), query points with top 10 attention weights are shown by circles, with sizes proportional to weights. best viewed in color.

---

Table 4: Comparisons between different sparsification strategies.
Model
Q
RayloU
RayloU
RayloU,.
RayloU
FPS
SparseOce
(400016000660000
28.4
34,9
39.6
34.3
17.3
PETR
2500
256
24.4
31.0
36.3
30.6
13.8
OPUS
2400
32
31.7
38.8
43.4
38.0
13,4
Table 5: Performance the Waymo-Oce3D dataset.
Motorcycle
General
Vchicle
Bieyclist
Tíc. light
Bieycle
Building
Vegetaion
Trectrunk
Sklewalk
RuyloUJ
Sign
Pole
Cons.
Road
mloU
Model
BEVDet
0.13
13.06
2.17
10.15
7,80
5.85
4.62
0.94
1.49
0.0
7.27
10.06
2.35
48.15
34.12
9.88
TPVFormer
3.89
17.86
12.03
13.64
8.49
9.95
14.79
0.32
13.82
11.44
588
73.3
51.49
16.76
BEVFomee
3.48
17.18
13.87
5.9
13.84
2.7
9.82
12.2
13.99
0.0
13,38
11.66
6.73
74.97
51.61
16.76
4.6
CTV-Occ
6.26
28.09
14.66
8.22
15.44
10.53
11.78
13662
16.45
0.65
18.63
17.3
8.29
67.99
42.98
18.73
266
OPUS-L
4.66 27.07 19.39
18.66
6.41
10.40
12.90
0.0
18.73
18.11
7.46
72.86
19.00
Comparisons between different sparsífication strategies In Tab. 4, we compare OPUS to two other models with different sparsification strategies. The first baseline is SparseOce, which achieves sparsification by filtering out empty voxels at various cascade stages. Following PETRv2 [24]. the second baseline is pillar-patch based method that partitions the 3D space into small number of pillar-patches. We use 50 x 50 queries with each corresponding to the classification of neighbouring x x 16 voxels. For fair comparison, all these models are trained for 100 epochs. model achieve best results after sufficient training with a RayloU score of 38.0, 38.0,far outperforming SparseOce with RayloU score of 34.3. On the other hand, our model can also runs in a real-timm tile speed. These results demonstrates the superiority of our sparification procedure.
contrast, our
Comparisons on the Waymo-Oec3D dataset. We further simply implement OPUS on the Waymo- Occ3D [35] dataset to explore the generalization and robustness of OPUS. As Waymo-Oec3D is not commonly used
standard benchmark for vision-centric approaches, the only vision-based based method we found with reported results on this dataset is the Occ3D paper, which evaluates BEVDet, TPVFormer, BEVFormer, and the newly proposed CTF-Occ [38]. We trained the OPUS-L (If) on 20% of the dataset for a fair comparison with these baselines. As reported in Tab. 5, despite fine-tuning the training configurations, OPUS-L already achieves 19.0 mIoU, outperforming all previous methods. Moreover, OPUS-L also reaches 8.5 FPS on the Waymo- OocnD dataset, which is around 3 times the speed of CTF-Occ and 2 times the speed of BEVFormer.
Conclusions and limitations
This paper introduces a novel perspective on occupancy prediction by framing a direct set predic- tion problem. Using transformer encoder-decoder architecture, the proposed OPUS directly predicts occupied locations and classes in parallel from a set of Iearnable queries. The matching between predictions and ground truths is accomplished through two efficient tasks in parallel, facilitating end-to-end training with a large number of points in this application. In addition, the query features are enhanced via list of non-trivial designs (i.e., coarse-to-fing learning, consistent point sampling, and loss re- se-weighting), and therefore leads to boosted prediction performances. Our experiments on the Occ3D-nuScee -nuScenes benchmark demonstrate that OPUS surpasses all prior arts in terms of both accuracy and efficiency, thanks to the sparse designs in our framework.
However, the proposed OPUS also comes with new challenges, particularly regarding the convergence speed. The slow convergence may potentially be alleviated by drawing lessons from follow-up works of DETR, which have largely addressed the convergence issue of the original DETR. Another challenge is that while sparse approaches typically achieve higher RayloU compared to dense counterparts, they often struggle with the mloU metric. Improving the mIoU performance while maintaining superior RayloU results is a promising direction for future works. Moreover, despite conducting experiments vision-only datasets, our core formulation is directly applicable to multi- modal tasks as well. We leave the multi-moda occupancy prediction as future work.
10

---

Acknowledgments
This research was supported by NSFC (NO. 62225604, NO. 62276145), the Fundamental Research Funds for the Central Universities (Nankai University, 070-63223049) and partially supported by NSFC (No. 62376153, 62402318, 24Z990200676), Computations were supported by the Supercom- puting Center of Nankai University (NKSC).
References
[1] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke. Cyrill Stachniss, and Jurgen Gall. Semantickiti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9297-9307, 2019.
[2] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IELE/CVF conference on computer vision and pattern recognition, pages 11621-1 2020.
[3] Anh uuh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3991 1001, 2022.
[4] Qinglong Cao, Yuntian Chen, Ding Wang, Zhenggin Xu, Chao Ma, Xiaokang Yang, and Shiyi Chen. Vision-informed flow field super-resolution with quaternion spatial modeling and dynamic fluid convolution. Physics of Fluids, 36(9), 2024.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 219-229. Springer, 2020.
[6] Haoqiang Fan, Su, and Leonidas Guibas. point set generation network object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[8] Junjie Huang, Guan Huang, Zheng 7hu, Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112. 2021.
[9] Yuanhui Huang. Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vaseon-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9223-9232. 2023.
[10] Yupeng Jia, Jie He, Runze Chen, Fang Zhao, and Haiyong Luo. Occupancydetr: Making semantic scene completion as straightforward as object detection. arXiv preprint 2xXiv:2300008000 2023.
[11] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1116-1124 2023.
[12] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83 97, 1955.
[13] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr. Accelerate detr training by introducing query denoising. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 136 13619-13627, 2022.
[14] Yiming Li, Zhiding Yu, Christopher Choy, Chaowci Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9087-9098. 2023.
[15] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's e-ee-vieww representation from multi-camert images via spatiotemporal trans- formers. In European conference computer vision, pages 1-18. Springer, 2022.
11

---

[16] Zhiqi Li. Zhiding Yu, David Austin, Mingsheng Fang. Shiyi Lan. Jan Kautz, and Jose M Alvarez. Fb-oce: 3d occupancy prediction based on howaadd-bckwaad view transformation. arXiv preprint arXiv:2307.01992 2023.
[17] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[18] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv preprint 1. 2022.
[19] Xuewu Lin, Tianwi Lin, Zixiang Pei. Lichao Huang, and Zhizhong Su, Sparse4d v2: Recurrent temporal fusion with sparse model. arXiv preprint or 2305. 14018, 2023.
[20] Haisong Liu, Yao Teng. Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-performanee sparse 3d object detection from multi-camera videos. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18580-18590, 2023.
[21] Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang. Jia Zeng. Li Chen, and Limin Wang. Fully sparse panoptic occupancy prediction. arXiv preprint arXiv:2312. 1711111 2023.
[22] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang. Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint 2rXiv:2211.23229 2022.
[23] Yingfei Liu, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference Computer Vision, pages 531-548. Springer, 2022.
[24] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: unified framework for 3d perception from multi-camera images. In Proceedings of the IFFF/CVF International Conference on Computer Vision, pages 3262-3272, 2023.
[25] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv: 1608.06600.398.. 2016.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[27] Qihang Ma, Xin Tan. Yanyun Qu, Lizhuang Ma, Zhizhong Zhang, and Yuan Xie. Cotr: Compact occupancy transformer for vision-based 3d occupancy prediction. arXiv preprint arXiv:2312.01919, 2023.
[28] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF international conference computer vision, pages 3651-3660, 2021.
[29] Qiang Meng, Xiao Wang, JiaBao Wang, Liujiang Yan, and Ke Wang. Small, versatile and mighty: A range-view perception framework. arXiv preprint arXiv:2403.0033 2024.
[30] Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In Conference on Robot Learning, pages 1444-1454. PMIRR, 2022.
[31] Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Li Liu, and Shanghang Zhang Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision. arXiv preprint MeXiv:2309009500 2023.
[32] Yining Shi, Jingyan Shen, Yifan Sun, Yunlong Wang. Jiaxin Li. Shiqi Sun, Kun Jiang. and Diange Yang. Sren3d: Sparse r-enn 3d for compact convolutional multi-view 3d object detection and tracking. arXiv preprint arX6::2206. 14451, 2022.
[33] Yining Shi, Kun Jiang, Ke Wang, Kangan Qian, Yunlong Wang, Jiusi Li, Tuopu Wen, Mengmeng Yang. Yiliang Xu, and Diange Yang. Effoce: A minimal baseline for efficient arXiv preprint arXiv:2406.07044 2024.
3d occupancy network.
[34] Yining Shi, Jiusi Li, Kun Jiang, Ke Wang. Yunlong Wang, Mengmeng Yang. and Diange Yang. Panosse: Exploring monocular panoptic 3d scene reconstruction for autonomous driving, In 2024 International Conference on 3D Vision (3DV), pages 1219-1228. IEEE, 2024.
12

---

[35] Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF conference un computer vision and pattern recognition. pages 1446-2454, 2020.
[36] Zhiqing Sun, Shengcao Cao. Yiming Yang, and Kris M Kitani. Rethinking transformer based set prediction for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3611 I 136200, 2021.
[37] Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, and Chao Ma. Sparseoce: Rethinking sparse latent representation for vision- based semantic occupancy prediction. arXiv preprint arXi::2404, 09502, 2024.
[38] Xiaoyu Tian, Tao Jiang. Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang. Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36, 2024.
[39] Jiabao Wang, Qiang Meng, Guochao I Liu, Liujiang Yan, Ke Wang, Ming-Ming Cheng, and Qibin Hou. Towards stable 3d object detection. In Eumpean conference computer vision. Springer, 2024.
[40] Shihao Wang. Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3631, 2023.
[41] Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yumpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 17850- 17859, 2023.
[42] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer- based detector. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 2567-2575, 2022.
[43] Yue Wang. Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning. pages 180-191. PMLR, 2022.
[44] Zetong Yang. Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673-14684, 2024.
[45] Hao Zhang. Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations, 2022.
[46] Yunpeng Zhang, Zheng Zhu, and Dalong Du Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9433 9443, 2023.
[47] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2020.
[48] Ziyue Zhu, Qiang Meng. Xiao Wang, Ke Wang, Liujiang Yan, and Jian Yang. Curricular object manipula- tion in baser- object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 1125-1135, 2023.
13

---

Appendix
A
Broader impacts
Our work proposes an end-to-end paradigm for occupancy prediction, achieving state-of-the-a RayloU performance with fast inference speeds. This advancement can lead to real-time and precise occupancy outcomes, which are crucial for real-world applications autonomous driving (AD). Consequently, the most significant positive impact of our work is the enhancement of safety and response speed in AD systems.
However, the biggest negative societal impact of this work, with any component of AD systems, is the safety concern. Autonomous driving systems are directly related to human lives, and erroncous predictions decisions lead to hazardous outcomes. Therefore, increasing the accuracy of occupancy outcomes and developing complementary methods to address false predictions will require substantial follow-up efforts.
B Licenses for involved assets
Our code is built on top of the codebase' provided by Sp [20], which is subject to the MIT license. Our experiments are conducted on the Occ3D -nueeensSe [38] which provides occupancy labels for the nuScenes dataset [2]. Boc3D-nuScemt is licensed under the MIT license, and nuScenes is licensed under the CC BY-NC-SA 4.0 license.
C Complexity analysis
In this part, we provide a detailed analysis of the time and space complexity involved in matching m predictions with n ground-truths,
Hungarian algorithm. The Hungarian algorithm's 's core involves finding augmenting paths for min(m, n) iterations. Each iteration can be visualized as attempt to improve the current match- ing by finding the shortest augmenting path in the residual graph, which has a complexity of O(max(m m, m,n)3 2) using with Dijkstra's algorithm. Consequently, the time complexity for the Hungar- ian algorithm O O(min(m. n.) max(m, n)
Mcanwhile, the Hungarian Algorithm necessitates computing cost matrix of size to store the costs linked with each potential assignment. Throughout the matching process, the tracked labels and matched pairs each demand O(min (m, n)) space. Hence, the final space complexity is O(m n).
Our method. Our method employs the Chamfer distance loss, which involves computing pair- wise distances and determining the smallest distance for each point. The first step requires a time complexity of (m x n) and the next step requires O(m x n) as well. The assignment of semantic labels can re-use the results of previous nearest search, therefore requires additional computations. In the end, the time complexity is Omm x n.).
For each point in set, the algorithm needs to keep track of the minimum distance to any point in the other set. This can be done using single variable per point, resulting in ((m) and O(n) in the respective directions. Semantic label assignment, meanwhile, incurs space complexity of (()). Collectively, this sums up to O(2m + nt).
Comparison of the two methods. In conclusion, when m and Hungarian algorithm exhibits time complexity of O(n²) and space complexity of O(n²). whereas method demonstrates significantly improved efficiencies with complexities of O(n²) and (()), respectively. This represents notable reduction in both time and space requirements, making it more efficient solution for large-scale applications.
comparable in scale, the
D Additional experiments.
D.1 Comparison of Hungarian matching and our method
Tab. presents comparison of the duration and GPU utilization when matching two point clouds with the same number of points. It is evident that the Hungarian algorithm exhibits scalability issues. For instance, when the point number is 10K, it consumes approximately 24 seconds and 2,304Mb of
14

---

Table 6: Comparison Hungarian algorithm and our label assignment scheme.
Number of Points
Time (ms)
GPU (Mb)
Hungarian Algorithm
Ours
Hungarian Algorithm
Ours
100
0.52
0.12
39
14
1,000
78.34
0.13
10,000
24,216.35
1.25
2,304
100,000
28.85
39
GPU memory for single matching. Scaling up to 100K points renders the matching infeasible due to CUDA memory constraints, even on an 80G A100 GPU.
In contrast, label assignment method achieves remarkable efficiency, requiring only about 1 1.25ms and 28.85ms for 10K and 100K points, respectively. Furthermore, the GPU memory consumption during training is negligible. These findings reveal the practicality and efficacy of our label assignment approach, particularly for the occupancy prediction where point counts can easily exceed 10K.
Table Configurations for different models.
point number
Model
Q
s1
s2
s3
s4
s5
s6
OPUS-T
600
4
64
128
OPUS-S
1200
2
1
16
32
64
OPUS-M
2400
2
1
16
32
OPUS-L
4800
2
1
8
16
16
D.2 Detailed configuration for different versions.
In this section, we detail the settings of various versions of our model, as shown in Tab. 7, each tailored to prioritize different aspects of performance and speed. Our fastest model OPUS-T utilizes only 0.6K queries, with each query sampling 4 points in images. The number of predicted points are 1, 4, 16, 32, 64 and 128 for 6 stages, respectively. This configuration ensures a rapid processing time while maintaining competitive performance. Other versions of our model, such OPUS-S. OPUS-M, OPUS-L, sample 2 points CPS module, progressively double the number of queries and adjust the number of predicted points accordingly to balance speed and accuracy. All these models predict the same amount of points in the end.
Table 8: Performance with different points predicted.
Model
point number
mloU
RayloU im
RayloU,, km
RayloU 4m
RayloU
28.4
22.2
29.5
34.8
28.8
27.2
26.1
33.3
38.4
32.6
OPUS-M
16
22.8
28.1
35.3
40.2
34.5
16.4
27.4
34.6
39.6
33.9
D.3 Effects of various refined points number in last layer.
Tab. 8 assesses the impact of varying the number of predicted points in the last layer. We use OPUS-M as our model for this experiment. As shown in the table, mloU steadily rises as the number points increase from 8 to 64, going from 16.4 28.4. This trend is expected since increasing the number of points generally leads to higher mloU by covering more voxels, as mloU penalizes false negative (FN) heavily. However, the RayloU results peak when model predicting 16 points and decline with further increasing points. This decline occurs partly because adding more points beyond a certain extent introduces noise, which negatively impacts RayloU, which emphasizes first occupied voxels along the ray.
D.4 Predictions across different distances
We report the RayloU of FB-Oce and OPUS at different ranges in Tab. It is evident that OPUS demonstrates more pronounced advantage in nearby areas than at far distances. This could be attributed to the phenomenon pointed out by SparseOce: dense approaches tend to overestimate the surfaces, especially in nearby areas.
15

---

Table Performance across different distances.
Model
overall
Om ~ 20m
20m ~ 40m
> 40m
FB-Oce OPUS-L
33.5
41.3
24.2
12.1
41.2
49.10
31.15
13.73
E
Additional qualitative analysis
E.1 Differences between SparseOce and OPUS
View perspective of occupancy prediction. The fundamental difference lies in the perspective of occupancy prediction. As depicted in the main draft, all previous methods, including SparseOce [2 l], treat occupancy prediction as a standard classification task. OPUS, however, pioneers a set prediction viewpoint, offering novel, elegant, and end-to-end sparsification approach.
Multi-stage vs. end-to-end sparsification procedure. SparseOce generates sparse occupancy by gradually discarding voxels through multiple stages. The discarding of empty voxels at early stages irreversible, leading to obvious cumulative errors, as illustrated in Fig. Conversely, OPUS circumvents complex filtering mechanisms by directly predicting a sparse set, resulting in more coherent outcomes.
Detailed model design. In terms of a more detailed perspective of the structure, there are also many differences such as:
Query number. In NuSeene-Occ3D, SparseOce necessitates 32K queries in its final stage. OPUS, by comparison, operates with 0.6K-4.8K queries for occupancy prediction, capitalizing on its flexible nature and contributing to its fast inference pace.
Coarse-to-fine procedure. SparseOcc's coarse-to-fine strategy involves progressively filtering empty voxels and subdividing occupied voxels into finer ones. In contrast, OPUS interprets coarse-to-fine as the escalation in number of predicted points across stages.
Learning objective. Our learning target encompasses predicting both semantic classes and occupied locations, simultaneously. The latter is a new objective introduced by OPUS, achieved through a modified Chamfer distance loss.
E.2 Analysis of relationships between mIoU, RayloU and driving safety.
Our OPUS-L (8f) has achieved state-of-the-ar RayloU of 41.17, outperforming the previous sparse model SparseOce by 6.07 and the dense model FB-Occ by 7.7. The mloU gap between sparse and dense methods is also reduced from 8.5 in SparseOcc to 3.0 OPUS. However, the implications of this gap on safety remain ambiguous. This concern is particularly pertinent in the context of autonomous driving, and we would like clarify this follows:
Risks dense predictions. The biggest issue of dense predictions is the discrepancies between evaluation metries and real-world scenarios. As shown in Fig. 8, evaluation metrics only consider voxels within the camera mask, which derived from camera parameters and ground truth. However, in real-world applications, failing to filtering out over-estimated voxels. From Fig. 8 and and Fig. 3, dense methods can misidentify occupied voxels, even close to the ego vehicle. These errors are overlooked during evaluation but pose significant safety hazards real-world scenarios. In contrast, OPUS suffer much less from this issue as it does not over-estimate occupancy.
only produce view mask based camera intrinsics and extrinsics,
The depth errors of OPUS much smaller than FB-Oce. Fig. 9, we compare the depth errors of FB-Occ and OPUS along camera rays. OPUS demonstrates lower depth errors across all scenes, despite its relatively low mloU performance. Given the significance of the first occupied voxel for safety, OPUS's precision in this regard enhances safety rather than detracting from it.
In conclusion, while it necessary to minimize the mloU gap between sparse and dense methods, our analysis indicates that mloU might not fully represent potentially hazardous situations. Therefore, it would be more rational to take both mloU and RayloU into consideration for the occupancy task.
16

---

Mali-inaaes
Prolicticn
Mallicanaao
Prodiction
sogy
(a) Evalusaton for FB-Ore
(b) Real-world usage for FB-Ope
View Mark
prodee
Likeeperrive shead ago vitice
Mulii-nngee
Spano Producion
Firel Piolicion
Safety threat From Fi-Ore
Real-world usage Tor OPUS
Figure 8: Illustration of safety threat due discrepancies between evaluation metrics and real-worts scenarios. (a) Before evaluation, the camera visibility mask is first generated according to camera intrinsies and extrinsics. Then, the dense prediction will be masked get the final prediction for evaluation. (b) For real-world usage, we cannot have camera visibility reasoning without knowing the ground-truth occupancy. We can only generate the view mask from camera intrinsies and extrinsics. which fails to filter out the over-estimated voxels from dense models. (c) Plenty of false positive predictions are made close to the ego vehicle, marked by the symbol of red star. These erroneously predicted voxels are filtered during evaluating mloU, but could cause hazardous safety issue. (d) The OPUS produces sparse occupancy predictions and suffers much less from the over-estimation. Consequently, no such safety threat occurs in this scenario. Best viewed in color.
FB-Oxe
RUMA:
Rayalt 15.24
OPUS
1528
DERE 2007
HEUSR 28.7
BAAE 2151
(8)
(6)
(0)
(d)
Figure 9: The predicted maps of FB-Occ and OPUS. When compared with FB-Occ, OPUS has lower mloU and higher RayloU results, and achieves evidently smaller enrors. Best viewed color.
17

---

E.3 Occupancy predictions of different methods.
In Fig. 10, We further provide more visualizations of occupancy predicted by FB-Occ, SparseOce. and proposed OPUS. As shown in Fig. 3 and Fig. 10, common OPUS failure mode is the prediction of scattered and discontinuous surfaces at long distances. Another is the presence of holes in predicted driving surface, a phenomenon also observed in SparseOce due to the sparsity properties.
(a) FB-Occ
(b) SparseOce
(c) OPUS
(d) ground-truth
Figure 10: Visualizations of occupancy predicted by FB-Occ, SparseOcc and the proposed OPUS.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Question:
Answer: [Yes]
We have described our contributions and scope explicitly in both the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe our limitations in the section "Conclusions and Limitations" in main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
19

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided implementation details in our "Experiments" section. More- over, our codes and configurations are directly attached in the supplementary for directly reproducing results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
20

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we provide the codes and configurations in the supplementary. One can easily reproduce our results after preparing the Oce3D dataset as required.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: gwtg///dd//e////e Sowps//SoSeeStiieiooooooo.o
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Yes, our training/test details are detailedly presented in our "Experimental Setup" section.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive in this paper.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computation resources are detailed in the "Experimental Setup" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
22

---

9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the cetp:///pppttttttcccc.u..eeeeeee
NeurIPS Code of Ethics https:
Answer: [Yes]
Justification: Our work adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the societal impacts of our work in the "Broader Impacts" section in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The involved data/models does not pose high risk for misuse.
Guidelines:
23

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We details licenses for existing assets in "Licenses for Involved Assets" section in appendix and cites related papers in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URI..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The readme file in our code provides details about the training, license and limitations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "0xEdpWwcZN", "reviewer": "Reviewer_5U1g", "summary": "This paper introduces OPS, a novel framework that treats occupancy prediction as a set prediction problem. The approach leverages a transformer encoder-decoder architecture and Chamfer distance loss to align predicted and ground-truth points. The model improves performance with strategies like coarse-to-fine learning, consistent point sampling, and adaptive re-weighting. OPS achieves superior RayIoU and faster FPS compared to state-of-the-art methods on the Occ3D-nuScenes dataset. In summarize, this paper is technically sound and shows strong and convincing performance on a commonly used dataset.", "questions": "1. Can authors provide results on the Waymo split? Or at least can authors provide a convincing explanation why such experiments cannot be conducted on Waymo? As far as I know, Occ3D-Waymo is on a par with Occ3D-nuScenes in terms of data scale, thus computation should not be a significant concern. \n\n2. Can authors comment on the difference between SparseOcc and OPS? I'd like to understand both the detail differences and the high-level differences.", "limitations": "No potential negative societal impact is detected by reviewer.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper formulates occupancy prediction as a sparse set prediction problem, which is technically sound and interesting. Indeed, predicting a sparse set introduces several efficiency and memory benefits, as long as a significant performance improvement. \n\n2. Several proposed techniques, including coarse-to-fine learning, consistent point sampling, and adaptive re-weighting, are effective and have the potential to generalize to even other methods. Also, these techniques are validated by extensive ablation studies on the nuScenes dataset. \n\n3. Experiments are carefully designed and adequate ablation studies are provided, though only on the nuScenes split.", "weaknesses": "1. In a high level, the proposed method is similar to SparseOcc, although implementation details can be different. \n\n2. Although this method has strong performance on Occ3D-nuScenes dataset and several ablation studies are provided, I am still concerned that this method can overfit to this relatively small dataset. Can authors provide further evidences on the Occ3D-Waymo split?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Please refer to the global response to all reviewers for our experiments on Occ3D-Waymo. In summary, the proposed OPS demonstrates its generality with superior mIoU results and fast inference speed. Below, we discuss the differences between SparseOcc and OPS:\n- **View perspective of occupancy prediction.** The fundamental difference lies in the perspective of occupancy prediction. As depicted in lines 33-37 and 122-130, all previous methods, including SparseOcc, treat occupancy prediction as a standard classification task. OPS, however, pioneers a set prediction viewpoint, offering a novel, elegant, and end-to-end sparsification approach, as pointed out by Reviewer Zbj3.\n- **Multi-stage vs. end-to-end sparsification procedure.** SparseOcc generates sparse occupancy by gradually discarding voxels through multiple stages. The discarding of empty voxels at early stages is irreversible, leading to obvious cumulative errors, as detailed in lines 276-280 and illustrated in Fig.3.  Conversely, OPS circumvents complex filtering mechanisms by directly predicting a sparse set, resulting in more coherent outcomes.\n- **Detailed model design.** In terms of a more detailed perspective of the structure, there are also many differences such as\n  - Query number: In Occ3D-Nuscene, SparseOcc necessitates 32K queries in its final stage. OPS, by comparison, operates with a mere 0.6K-4.8K queries for occupancy prediction, capitalizing on its flexible nature and contributing to its fast inference.\n  - Coarse-to-fine procedure: SparseOcc's coarse-to-fine strategy involves progressively filtering empty voxels and subdividing occupied voxels into finer ones. In contrast, OPS interprets coarse-to-fine as the escalation in number of predicted points across stages.\n  - Sampling process: The feature sampling in SparseOcc is deterministic, with each query anchored to a specific location. In contrast, the query locations are learnable in OPS. Therefore, we propose consistent point sampling to dynamically and efficiently gather features from input images.\n  - Learning objective: Our learning target encompasses predicting both semantic classes and occupied locations, simultaneously. The latter is a new objective introduced by OPS, achieved through a modified Chamfer distance loss.\n\nIn conclusion, we believe that SparseOcc and OPS are markedly different in both fundamental and detailed designs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGoZawdSZ5", "reviewer": "Reviewer_TqRW", "summary": "This paper considers the problem of occupancy prediction from multi-view images for autoonomous driving. One of the main challenges in the occupancy prediction task is the high computational demand incured by discretizing the 3d space. Traditional methods typically predict the occupancy of each voxel individually and assign it a semantic class. The key idea of this work if to reduce computational burden by formulating occupancy prediction as a set prediction task. The authors employ the Chamfer distance in the loss term and propose a novel point sampling process, termed Consistent Point Sampling, to make the SparseBEV decoder applicable. In experiments on the Occ3d-nuScenes dataset, their method demonstrates superior performance with regard to the RayIoU metric while being computationally more efficient than the state-of-the-art methods.", "questions": "* From the writing, it is not really clear to me, whether some of the competing methods are also set based? I would not see why the Hungarian algorithm would be needed otherwise. If those methods are set-based, the writing might clarify this. From the abstract it seems like competing approaches perform classification of each voxel individually.\n* In the right most panel of fig 2, why is \"Consistent Point Sampling\" written in a green box while everything else is in grey?\n* What is meant by a set of learnable queries? Are these simply the queries that would be used in the traditional transformer except that they are now learned?", "limitations": "To the extent that I can understand it, the authors have properly addressed the limitations of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe the proposed general approach is interesting and the work seems well evaluated (to the extent that I can judge). The paper is largely well-written and seems easily understandable for someone working in that space. I also want to applaud the authors for committing to making their code publicly available.", "weaknesses": "This topic seems to be traditionally mainly covered in Computer Vision venues. And while the contribution of this work may also be of interest to the ML community, the writing needs to provide more background and context to be more accessible to the NeurIPS community. E.g. the readers at NeurIPS may not be familiar with the broader tasks and some more specialized architectures such as SparseBEV.\n\nThe work contains numerous writing and grammar issues. Some examples are mentioned in the minor comments section below. However, the work should undergo a careful proofread in its entirety.\n\nSome of the mathematical notation seems off. E.g. {$\\mathbb{P}^g,\\mathbb{C}^g$} is denoted as a set containing two sets while it is treated as a set cotaining tuples of point locations and semantic classes. Some more minor math notation issues are given in the comments below.\n\nThe first bullet point in the contribution list is somewhat broad and focuses more on the properties of the method. It does not mention some of the new key technical ideas underlying the method such as CPS. I would recommend fully rephrasing the contribution list and focusing on what is technically new (or at least for the first time applied in the context of occupancy prediction). Simply mentioning that there are several strategies that are introduced to boost performance makes the reader wonder what these strategies are.\n\nMinor Comments:\n* l.3 \"discretize 3D environment\" -> \"discretize the 3D environment\"\n* l.33/34 \"Alternative sparse latent representations has been explored\" -> \"Alternative sparse latent representations have been explored\"\n* l. 37 What is meant by \"necessitating complex intermediate designs and explicit\"? Complex architecture?\n* l.42/43 \"Our OPS eliminates\" should be \"Our method eliminates\" or \"OPS eliminates\"\n* l.48 \"unable to tacke tremendous voxels\" maybe something like \"unable to handle  a very high number of voxels\"\n* l.69/70 \"all our model configurations easily surpass all prior arts\" -> \"all our model variants easily surpass all prior work\".\n* l.83 \"This task recently becomes a foundational perception task in autonomous driving\" -> \"This task has recently become a foundational perception task for autonomous driving\"\n* Consider using $N_g$ instead of $N^g$ for the number of occupied voxels in ground truth. The latter looks like $N$ to the power of $g$. Same for all other occurrences of $g$ in a superscript.\n* If $\\mathbf{c}^g$ denotes a semantic class, it should not be defined as element of $\\mathbb{R}$. This is not necessarily wrong but may be confusing as it is likely represented as an integer if not a one-hot vector?\n* Using $N$ to represent the number of semantic classes may be confusing given that $N^g$ is used to represent the number of occupied ground truth voxels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive suggestions and feedback. We will release our full code and models once the paper is made public. Below are our responses to specific comments:\n- **Writing issues.** We greatly appreciate meticulous suggestions on grammar and notations, which indeed help elevate paper's quality. We have revised our draft accordingly and will conduct a comprehensive proofreading to ensure clarity and precision.\n- **Providing more background and context.** Thanks for the advice. We will enrich the background descriptions in sections of related works, methodology, and appendix. For specified architectures in SparseBEV[1], we will provide explanations to guide readers to the original paper, delineating which modules are newly designed and which are adapted from SparseBEV.\n- **Details about compting methods.** As mentioned in lines 74-75, OPS is the first set-based approach for occupancy prediction. All baselines perform voxel-wise classification, resulting in predictions that are inherently ordered according to the physical locations of voxels, thus differing from set-based methods whose predictions are unordered. We will provide further clarification on this distinction in Section 4.2.\n- **Why the Hungarian algorithm would be needed.** The primary challenge of a set-based approach lies in associating the unordered prediction set with ground-truths. As detailed in lines 46-52, while the Hungarian algorithm is effective for association in object detection, it is not suitable for occupancy prediction, which motivates development of OPS.\n- **Consistent Point Sampling in Fig.2.** We highlight the \"consistent point sampling\" in Fig.2, as it is newly introduced in OPS. Other components, such as \"adaptive mixing,\" are inherited from SparseBEV[1]. We will clarify this in the caption.\n- **Learnable queries.** The learnability of queries in Transformers is contingent upon the specific application. In standard Transformers for tasks such as machine translation, queries are derived from the input data and are not learnable. However, in many Transformer-based models (*e.g.*, SparseBEV[1] and DETR[2]), queries are initialized randomly and are optimized during training. They enable the model to dynamically focus on various aspects of the input data.\n\n[1] Sparsebev: High-performance sparse 3d object detection from multi-camera videos, CVPR23.  \n[2] End-to-end object detection with transformers, ECCV20.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnAoWSeo5o", "reviewer": "Reviewer_Zbj3", "summary": "This paper focuses on the sparsity property in occupancy prediction, given that most voxels are occupied. In order to reduce computation costs on empty voxels, this paper introduces a set prediction paradigm to explicitly model sparsity. OPS, the proposed framework, utilizes the encoder-decoder architecture to jointly reconstruct and classify point clouds. Chamfer distance is used as the supervision. For experiments, OPS outperforms state-of-the-art occupancy methods by 4.9 RayIoU.", "questions": "See Weakness.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Good motivation. The introduction of sparsity as the representation of scenes is with a great motivation.\n\n2. Novel architecture. Using the point cloud + set prediction approach for occupancy is novel, elegant, and makes sense.\n\n3. Strong performance. The proposed OPS with RayIoU achieves significantly better performance and inference speed compared to the current state-of-the-art. Although the authors mentioned the current limitation is the need for longer epochs, it is not a major issue, as most set-prediction methods face this challenge.", "weaknesses": "1. Error citation. I think the \"sparseocc\" mentioned frequently in this paper should be [1], instead of [2].\n\n    [1.] Fully sparse 3D panoptic occupancy prediction\n\n    [2.] SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction\n\n2. Is there a typo in Line 245? The query count for OPS-tiny is only 0.6k, while for OPS-S, OPS-M, and OPS-L, the query counts are as high as 12K, 24K, and 48K respectively.\n\n3. If the query count is as high as 48K, self-attention would not be able to handle such a large query size. How did the authors manage to do this? The design details are not elaborated on.\n\n4. Since the authors have replaced the basic representation of occupancy with point cloud, they should discuss the differences with point cloud forecasting tasks, such as 4D-Occ [3] and ViDAR [4], especially ViDAR paper which also constructs point clouds from visual images. Discussions on [3] and [4] are suggested to be included in the main paper.\n\n    [3.] Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting\n\n    [4.] Visual Point Cloud Forecasting enables Scalable Autonomous Driving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and feel honored for being approbated on our motivation, novelty and performances. Below, please find our responses to the weaknesses:\n- **Error citation.** Thanks for bringing this to our attention. We have thoroughly reviewed the cited works and identified an incorrect citation in line 71, which has been duly corrected. In the remaining instances (lines 35, 88, 92, 122, 239, 243, 268, 274, and Tab.1), we have confirmed that the correct paper[1] has been cited.\n- **Typos in line 245.** We appreciate the reminder and have made the corrections. The query numbers for OPS-S, OPS-M, and OPS-L are indeed 1.2K, 2.4K, and 4.8K. The small number of queries is a key factor for the fast inference speed of OPS. More detailed configurations of different models can be found in Tab.5 in the appendix.\n- **Discussions on point cloud forecasting tasks.** We have included comparisons between OPS and works[2,3] in the appendix. Regrettably, due to page limits, we were unable to incorporate this discussion into the main text.\n\n[1] Fully sparse 3D panoptic occupancy prediction, ECCV24.  \n[2] Point cloud forecasting as a proxy for 4d occupancy forecasting. CVPR23.  \n[3] Visual point cloud forecasting enables scalable autonomous driving, CVPR24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "STpSSwNwBb", "reviewer": "Reviewer_iFoi", "summary": "The paper presents OPS, a novel framework for occupancy prediction in autonomous driving. It formulates the task as a direct set prediction problem, using a transformer encoder-decoder architecture to predict occupied locations and classes simultaneously. This approach eliminates the need for explicit space modeling or complex sparsification. The main results of the paper demonstrate that the OPS framework outperforms previous state-of-the-art methods in occupancy prediction on the Occ3D-nuScenes dataset. It achieves superior RayIoU scores, a metric designed to address the overestimation issues of the traditional mIoU metric. The paper also includes an ablation study highlighting the effectiveness of the various strategies incorporated in OPS, such as adaptive re-weighting, consistent point sampling, and coarse-to-fine prediction. These strategies contribute to improved performance, particularly in terms of mIoU and RayIoU.", "questions": "Please check the weakness section, especially W1 and W3.", "limitations": "Yes, the limitations have been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **[S1] Clarity:** The paper is well-written and easy to follow. The problem formulation, methodology, and experimental results are clearly presented. Overall, the figures are informative and complement the text well.\n- **[S2] Quality:** The proposed OPS framework is thoroughly described, and the experimental results on the Occ3D-nuScenes dataset shows merits of the proposed method in terms of RayIoU. The ablation studies further validate the contribution of each component in the framework.", "weaknesses": "- **[W1] Technical soundness:** The paper proposes to use Chamfer distance as the main objective function, which is known to be sub-optimal. Although several strategies including FocalLoss, coarse-to-fine refinement have been proposed, it makes the system overly complicated and potentially unfair for comparison. For example, it is unclear whether the baseline methods can benefit from the proposed strategies as well. Please comment this in the rebuttal.\n\n- **[W2] Limited Evaluation:** The experimental evaluation is solely conducted on the Occ3D-nuScenes dataset. While this is a standard benchmark, evaluating the method on additional datasets, such as SemanticKITTI or Waymo Open Dataset, would provide a more comprehensive assessment of its generalizability and robustness.\n  - Reference: [NewRef1] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving\n\n- **[W3] mIoU Performance:**\n  - [W3.1] Although OPS excels in RayIoU, its performance on the mIoU metric lags behind some dense models. Given that mIoU is a widely used metric in occupancy prediction, addressing this weakness would make the method more appealing to a broader audience. In comparison, RayIoU is only introduced in a recent paper that has not been peer reviewed. \n\n  - [W3.2] As the paper is focusing on autonomous driving applications, it is unclear whether mIoU performance can cause safety-critical problems for downstream tasks such as behavior prediction and planning. It would be good to discuss this aspect in the rebuttal and next version of the paper.\n\n  - [W3.3] In Table 3, the reference number for FB-OCC is wrong. \n\n- **[W4] Lack of Qualitative Analysis:** While the paper provides some visualizations, a more thorough qualitative analysis of the predictions would be insightful. Analyzing failure cases, comparing predictions across different classes and distances, and examining the impact of the proposed strategies on the quality of predictions would provide a deeper understanding of the method's strengths and weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We have carried out experiments on Occ3D-Waymo, detailed in the global response to all reviewers. Our results underscore the generalizability of our OPS. We have also corrected the reference number for FB-Occ in Tab.3. Below are our responses to other specific concerns:\n- **[W1] Chamfer distance is sub-optimal.** As stated in lines 46-52 and Sec.C in the appendix, the Hungarian algorithm is optimal but not scalable for the occupancy task. In spite of the sub-optimal property, Chamfer distance is justified by its computational efficiency and satisfactory outcomes accroding to our experiments.\n- **[W1] Proposed strategies lead to complicated system.** We introduce four strategies in total, as shown in Tab.2. The re-weighting losses does not alter model structures. CPS is computationally similar to the sampling scheme in SparseBEV[1]. The coarse-to-fine approach, meanwhile, streamlines the computation in the initial phases. Therefore, these strategies do not complicate the system building on top of SparseBEV. \n- **[W1] Unclear if the baseline methods can benefit from the proposed strategies.** \n  - The re-weighted CD loss and CPS are tailored to our set prediction framework and are not directly applicable to other occupancy prediction methods. \n  - The coarse-to-fine scheme is also adopted by CTF-Occ[4] and SparseOcc[2]. However, each method has its uniquely crafted designs, which are not intended for external use.\n  - Regarding the classification loss, our main baselines, SparseOcc[2] and FB-Occ[3], incorporate more complex designs than our re-weighted focal loss. SparseOcc, for instance, integrates focal and Dice losses, while FB-Occ sums up four distinct classification losses. We are not sure if their performances can be improved by simply replacing their focal loss with ours, given the composite nature of their loss functions.\n- **[W1] Proposed strategies lead to potentially unfair comparison.** Strategies not exclusively tailored for OPS (CD loss and CPS), have comparable or even more complex alternatives in SparseOcc. In addition, we humbly believe that the entirety of the strategies and other components should be evaluated together for a single approach. Therefore, we respectfully assert that the comparisons are fair.\n- **[W3.1] mIoU vs. RayIoU.** \n  - Our mIoU are further enhanced: As detailed in the general response, OPS-L(8f) now achieves 36.14 mIoU, marking a 3.74 improvement over our prior submission and 6 mIoU lead over SparseOcc (8f). The gap between dense and sparse methods has been largely reduced to 3 mIoU.\n  - RayIoU also matters: RayIoU, introduced by SparseOcc[2], has gained acceptance through peer review at ECCV and is utilized in recent literature[2,5,6] and a CVPR workshop[7]. As illustrated in their Fig.4, Fig.5 and Sec. 4.1, the mIoU can be hacked by predicting a thicker surface, a common occurrence with dense methods trained on visibility masks. This could account for the higher mIoU observed in dense methods, despite poorer visualizations. In contrast, RayIoU is not vulnerable to that over-estimation.\n- **[W3.2] Relatively low mIoU performance may cause safety-critical problems.** Please refer to our global response.\n- **[W4] More qualitative analysis.** We thank the reviewer for the kind suggestion and have incorporated additional analysis into our draft. Here are the key points:\n  - Failure cases: As shown in Fig.3 and Fig.8, a common OPS failure mode is the prediction of scattered and discontinuous surfaces at long distances. Another is the presence of holes in predicted driving surface,  a phenomenon also observed in SparseOcc due to the sparsity properties.\n  - Impact of proposed strategies: Tab.4 demonstrates the impact of our proposed strategies. Tab.3, Tab.6 and Tab.7 examine other facts that could affect model performance. Fig.4 Fig.5 and Fig.6 uncover some underlying mechanisms of the proposed OPS.\n  - Predictions across different distances: We report the RayIoU of FB-Occ and OPS at different ranges in the following table. It is evident that OPS demonstrates a more pronounced advantage in nearby areas than at far distances. This could be  attributed to the phenomenon pointed out by SparseOcc: dense approaches tend to overestimate the surfaces, especially in nearby areas.\n|model|overall|0-20m|20-40m|>40m|\n|-|-|-|-|-|\n|FB-Occ|33.5|41.3|24.2|12.1|\n|OPS-L|41.2|49.10|31.15|13.73|\n  - Predictions across different classes: The per-class RayIoU results are presented as follows. OPS outperforms FB-Occ across all classes. The top 5 most improved classes are driving surfaces, other flats, sidewalks, man-made structures, and vegetation, indicating the background categories can benefit most from our model.\n|Method|others|barrier|bicycle|bus|car|c.veh.|motor.|ped.|cone|trailer|truck|surface|flat|sidewalk|terrain|manmade|vege.|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|FB-Occ|5|44.9|26.2|59.7|55.1|27.9|29.1|34.3|29.6|29.1|50.5|44.4|22.4|21.5|19.5|39.3|31.1|\n|OPS-L|10.9|46.2|29.6|65.5|58.4|29.7|31.1|35.8|33.8|34.7|52.7|68.6|37.3|35.1|37.5|50.1|43.1|\n\n[1] Sparsebev: High-performance sparse 3d object detection from multi-camera videos, CVPR23.  \n[2] Fully sparse 3D panoptic occupancy prediction, ECCV24.  \n[3] Fb-occ: 3d occupancy prediction based on forward-backward view transformation, arXiv23.  \n[4] Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving, Neurips24.  \n[5] CascadeFlow: 3D Occupancy and Flow Prediction with Cascaded Sparsity Sampling Refinement Framework.  \n[6] Panoptic-FlashOcc: An Efficient Baseline to Marry Semantic Occupancy with Panoptic via Instance Center, arXiv24.  \n[7] The Autonomous Grand Challenge at the CVPR 2024 Workshop.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their constructive comments and are privileged by their praise regarding our motivation (5U1g, TqRW, Zbj3), novelty (Zbjs), writing (TqRW, iFoi), and experiments (5U1g, Zbj3).  We'd like to first mention that OPS performances have further improved since our last submission, as detailed in the table below. In a nutshell, all OPS(8f) variants achieve substantial boosts in mIoU (+2.64 to +3.90) and RayIoU (+1.77 to +2.27). These improvements result from tuning hyperparameters and fixing post-processing, without altering any network structures. All our codes and checkpoints are well-organized and will be released once the paper is made public.

||mIoU|RayIoU|newmIoU|newRayIoU|
|-|-|-|-|-|
|OPS-T(8f)|30.6|35.9|33.24(+2.64)|38.40(+2.50)|
|OPS-S(8f)|31.2|37.3|34.24(+3.04)|39.07(+1.77)|
|OPS-M(8f)|31.7|38.0|35.60(+3.90)|40.27(+2.27)|
|OPS-L(8f)|32.4|38.9|36.14(+3.74)|41.17(+2.27)|

**Experiments on Occ3D-Waymo.** We'd like to clarify the comments about Occ3D-Waymo from reviewers 5U1g and iFoi. Initially, our draft did not evaluate OPS on the Occ3D-Waymo, as it is not commonly used as a standard benchmark for vision-centric approaches. The only vision-based method we found with reported results on this dataset is the Occ3D paper, which evaluates BEVDet, TPVFormer, BEVFormer, and the newly proposed CTF-Occ. During the rebuttal phase, we trained the OPS-L (1f) on 20% of the dataset for a fair comparison with these baselines. Despite not fine-tuning the training configurations, OPS-L already achieves 19.0 mIoU at 8.5 FPS, outperforming the baseline methods. We are grateful for the reviewers' suggestions and will incorporate the results into our revised draft.

||**mIoU**|**RayIoU**|**FPS**|general|vehicle|bicyclist|ped.|sign|tfc.light|pole|Cons.cone|bicycle|motorcycle|building|vegetaion|Treetrunk|road|sidewalk|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|BEVDet|**9.88**|**-**|**-**|0.13|13.06|2.17|10.15|7.80|5.85|4.62|0.94|1.49|0.0|7.27|10.06|2.35|48.15|34.12|
|TPVFormer|**16.76**|**-**|**-**|3.89|17.86|12.03|5.67|13.64|8.49|8.90|9.95|14.79|0.32|13.82|11.44|5.8|73.3|51.49|
|BEVFormer|**16.76**|**-**|**4.6**|3.48|17.18|13.87|5.9|13.84|2.7|9.82|12.2|13.99|0.0|13.38|11.66|6.73|74.97|51.61|
|CTF-Occ|**18.73**|**-**|**2.6**|6.26|28.09|14.66|8.22|15.44|10.53|11.78|13.62|16.45|0.65|18.63|17.3|8.29|67.99|42.98|
|OPS-L|**19.00**|**24.7**|**8.5**|4.66|27.07|19.39|6.53|18.66|6.41|11.44|10.40|12.90|0.0|18.73|18.11|7.46|72.86|50.31|

**Safety concerns.** Our OPS-L(8f) has achieved a state-of-the-art RayIoU of 41.17, outperforming the previous sparse model SparseOcc[1] by 6.07 and the dense model FB-Occ[2] by 7.7. The mIoU gap between sparse and dense methods is also reduced from 8.5 (in SparseOcc) to 3.0. However, as noted by Reviewer iFoi, the implications of this gap on safety remain ambiguous. This concern is particularly pertinent in the context of autonomous driving, and we would like to clarify this as follows:
- **Risks of dense predictions.** The biggest issue of dense predictions is the huge discrepancies between evalutation metrics and real-world scenarios. As shown in Fig.1 in the attached one-page pdf, evaluation metrics only consider voxels within the camera visiblity mask, which is derived from camera parameters and ground truth. Detailed precedure for generating the mask can be found in Occ3D[3]. However, in real-world applications, we can only produce view mask based on camera intrinsics and extrinsics, failing to filtering out over-estimated voxels. From this example and and Fig.3 in our paper, dense methods can misidentify occupied voxels, even close to the ego vehicle. These errors are overlooked during evaluation but pose significant safety hazards in real-world scenarios. In contrast, OPS suffer much less from this issue as it does not over-estimate occupancy.
- **The depth errors of OPS is much smaller than FB-Occ.** In Fig.2 in the attached one-page pdf, we compare the depth errors of FB-Occ and OPS along camera rays.  OPS demonstrates lower depth errors across all scenes, despite its relatively low mIoU performance. Given the significance of the first occupied voxel for safety, OPS's precision in this regard enhances safety rather than detracting from it. This aligns with Fig. 9 in SparseOcc, which shows that training FB-Occ without the camera visiblity mask results in poorer mIoU but lower errors.

In conclusion, while it is necessary to minimize the mIoU gap between sparse and dense methods, our analysis indicates that mIoU might not fully represent potentially hazardous situations. Therefore, it would be more rational to take both mIoU and RayIoU into consideration for the occupancy task.

[1] Fully sparse 3D panoptic occupancy prediction, ECCV24.  
[2] Fb-occ: 3d occupancy prediction based on forward-backward view transformation, arXiv23.  
[3] Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving, Neurips24.

Meta-review: This paper addresses occupancy prediction from a new persepctive via a sparse set. Without explicitly modelling the full 3D space set, since there are many empty voxels, this work simultaneously predict the occupied locations as a streamlined set prediction paradigm.

All reviewers reach concensus that the manuscript has been greatly improved after rebuttal. Most concerns raised by reviewers have been resolved. AC read the paper, rebuttal, revised experiments in pdf. Please incorporate all comments in the camera-ready version.
Decision: Accept (poster)