OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries
Authors: Yuhang Lu
Xinge ZHU
Tai Wang
Yuexin Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Scene Understanding; Occupancy Prediction; Octree
Primary area: machine_vision

Abstract: Occupancy prediction has increasingly garnered attention in recent years for its fine-grained understanding of 3D scenes. Traditional approaches typically rely on dense, regular grid representations, which often leads to excessive computational demands and a loss of spatial details for small objects. This paper introduces OctreeOcc, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. In particular, we incorporate image semantic information to improve the accuracy of initial octree structures and design an effective rectification mechanism to refine the octree structure iteratively. Our extensive evaluations show that OctreeOcc not only surpasses state-of-the-art methods in occupancy prediction, but also achieves a 15%-24% reduction in computational overhead compared to dense-grid-based methods.

OctreeOce: Efficient and Multi-Granularity Oceupancy Prediction Using Octree Queries
Yuhang Lu Shanghai Tech University luyh2@shanghaitech edu. en
Xinge Zhu The Chinese University of Hong Kong zhuxinge123@gaal1 com
Tai Wang"
Yuexin Ma ShanghaiTech University mayuexin@shanghaitedd edu, cn
Shanghai AI Laboratory taiwang me@gmail...mmmmmmmmmmmmmmmmmmmmm com
Abstract
Occupancy prediction has increasingly garnered attention in recent years for its fine-grained understanding of 3D scenes. Traditional approaches typically rely dense, regular grid representations, which often leads to excessive computational demands and loss of spatial details for small objects. This paper introduces OctreeOce, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. In particular, incorporate image semantic information improve the accuracy of initial octree structures and design an effective rectification mechanism refine the octree structure iteratively. Our extensive evaluations show that OctreeOce not only surpasses state-of-ther methods in occupancy prediction, but also achieves 15% 24% reduction in computational overhead compared dense-gre -ii--griddbsaee methods.
Introduction
Holistic 3D scene understanding is pivotal aspect of stable and reliable visual perception system, especially for real-world applications such as autonomous driving. Occupancy, as a classical repre- sentation, has been renascent recently with more datasets support and exploration in learning-bass approaches. Such occupancy prediction tasks aim partitioning the 3D scene into grid cells and predicting semantic labels for each voxel. It is particularly an essential solution for recognizing irreg- ularly shaped objects and also enables the open-set understanding tasks, like prediction and planning.
(. further benefiting downstream
Existing occupancy prediction methods 2 3 4 5 66) typically construct dense and regular grid representations, same as the ground truth. While such approach is intuitive and direct, it overlooks the statistical and geometric properties of 3D environments. In fact, the 3D scene is composed of foreground objects and background regions with various shapes and sizes. For example, the space occupied by larger objects, such buses, are considerably more extensive than that taken up by smaller items like traffic cones (Fig. la). Consequently, employing uniform voxel resolution to depict the scene proves to be inefficient, leading to computational waste for larger objects and lack of geometry details for smaller ones. Considering the large computation cost aforementioned
*Corresponding authors. This work was supported by NSFC (No.6220617) 173). Shanghai Frontiers Science Center of Human-conn centered Artificial Intelligence (ShangHAD) MoE Key Laboratory of Intelligent Perception and Hluman-Machin Collaboration (KL.I,HuMaCC), ShanghaiTeed University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

10
40
Barrier
Bicycle
Bus
Car
Construction Vehicle
Motorcycle
Pedestrian
Truffic Cone
Trailer
Truck
Instance Size Inbalance
(b) 3D and 2D Visualization of Octree Structures
Scale difference of various categories and octree representation. (a) compares the average space occupied by different object types, indicating varying granularities needed for differ- ent semantic regions. (b) demonstrates the advantage of octree representations, enabling specific granularities for different objects and even parts objects, reducing computational overhead while retaining spatial information.
Figure 1:
works, some recent works attempted to mitigate heavy memory footprint by utilizing other representations, such as 2D planes in TPVFormer ( and coarse voxels in PanoOcc (8) or modeling non-empty regions by depth estimation (9). However, these methods suffer from the loss of spatial information because of too coarse representations accumulated estimated depth errors.
To reduce the computational overhead and meanwhile improve the prediction accuracy, we propose to use octree (IO) representation for the occupancy prediction, which can flexibly adapt to objects and semantic regions with various shapes and sizes. As tree-based data structure, recursively divides the 3D space into eight octants, thus allowing coarse spatial partition for large regions and ene-grained processing for small objects complex details (Fig. [b). Incorporating octree representation, propose OctreeOce, an efficient and gratiigraauulritty method for occupancy prediction. It constructs octree queries by predicting the octree structure from the stored features of leaf nodes each level of the tree. However, directly predicting 3D) structure from 2D images is challenging due to the lack of depth and occlusion issucs. To address this problem, we first propose Semantic-guides Octree Initializatinn that incorporates image semantic information to produce more accurate initial structures. And then, we devise an Iterative Structure Rectification module that predicts new octree structure from the encoded query to rectify the low-confidence region of the original prediction, further improving the prediction precision.
Our extensive evaluations against state-of- ofute-of-th--ase the-art art occupancy prediction methods show that OctreeOce outperforms others on nuScenes and SemanticKITTI datasets, reducing computational overhead by 15% 24% for dense-grid- -red---id--dddd methods. Ablation studies further validate the effectiveness of each module within our method. Our contributions can be summarized as follows:
We introduce OctreeOcc, 3D occupancy prediction approach based on multi- griinlarattl octree queries. This method facilitates spatial sparsification, significantly decreasing the number of voxels needed to accurately depict a scene, yet retains critical spatial details.
We propose a semantic-guides octree initialization module and an iterative structure recti- fication module, which empower the network with robust initial setup and the ability to dynamically refine the octree, leading to more efficient and effective representation.
Comprehensive experiments demonstrate that OctreeOce achieves state-of-the-an perfor- mance and reduces computational overhead, highlighting the feasibility and potential of octree structures in 3D occupancy prediction.
Related Work
2.1 Camera-based 3D Perception
Camera-based perception has gained significant traction in recent years due its ease of deploy- ment, cost-efficttiveness, and the preservation of intricate visual attributes. According to the view

---

transformation paradigm, these methods can be categorized into three distinct types. LSS-base
explicitly lift multi-view image features into 3D space through depth prediction. Another category of works( 16. 17: 18) implicitly derives depth information by querying from 3D to 2D. Notably, projection free methods 20 21 (2) have recently demonstrated exceptional performance. While commendable progress has been made in detection, this approach compromises the comprehensive representation of the overall scene in 3D space and proves less effective in recognizing irregularly shaped objects. Consequently, there is burgeoning interest in methodologies aimed at acquiring a dense voxel representation through the camera, facilitating a more comprehensive understanding of 3D space.
approaches(II 2 12
13
14 (5)
2.2 3D Occupancy Prediction
3D occupancy prediction involves the prediction of both occupancy and semantic attributes for all voxels encompassed within three-dimensional scene, particularly valuable for autonomous vehicular navigation. Recently, some valuable datasets (23
25) have been proposed, boosting more and in this field. Most of the pioneers a camera- based approach using 3D UNet architecture. OccDepth(28) improves 2D-to-3D geometric projection using stereo-depth information. OccFormer(6) decomposes the 3D processing into the local and global trans cansformer pathways along the horizontal plane. SurroundOce(4) achieves fine-grained results with multiscale supervision. Symphonies(5) introduces instance queries to enhance scene representation.
24
more research works (26 1 27 28
4
5
7
9
29 30
32
33 34)
research focuses on dense voxel modeling. MonoScene
Nevertheless, owing to the high resolution of regular voxel representation and sparse context dis- tribution in 3D scenes, these methods encounter substantial computational overhead and efficiency issues. Some approaches recognize this problem and attempt to address it by reducing the number of modeled voxels. For instance, TPVFormer() models the three-view 2D planes and subsequently recovering 3D spatial information from them. However, its performance degrades due to the lack of 3D information. PanoOcc(S) initially represents scenes at the coarse-graine level and then upsamples them to the fine-grained level, but the lack of information from gaase-erained modeling cannot be adequately addressed by the up-sampling process. VoxFormer()) mitigates computational complexity by initially identifying non-empty regions through depth estimation and modeling only those specific areas. However, the effectiveness of this process is heavily contingent on the accuracy of depth estimation. In contrast, our approach provides different granularity of modeling for different regions by predicting the octree structure, which reduces the number of voxels to be modeled while preserving the spatial information, thereby reducing the computational overhead and maintaining the accuracy.
2.3
Octree-Rased 3D Representation
The octree sIructure(I0) finds widespread use in computer graphics for rendering or reconstruction (35
36 37 (38), owing to its spatial efficiency and GPU compatibility, Researchers have extended its
utility to efficient point cloud learning and related tasks 40 41 42). OctFormer(433 and OcTr
(44) utilize multi-granulan granularitt features of octree to capture a comprehensive global context, thereby enhancing the efficiency of understanding point clouds at the scene level. Furthermore, certain studies (46) adopt octree representation for effectively compressing point cloud data. These works have highlighted the versatility and effectiveness of octree-based methodologies in point cloud analysis and processing applications. However, unlike constructing an octree from 3D point clouds, we are the first to predict the octree structure of 3D scene from images, which is more challenging owing to the absence of explicit spatial information inherent in 2D images.
3 Methodology
In this section, we introduce details of our efficient and multi granularity occupancy prediction method OctreeOcc. After defining the problem and providing an overview of our method in Sec. [3.] and 3.22 we introduce key components of our method in order. In Sec. 3.3 we outline how we define octree queries and transform dense queries into octree queries. Next, we utilize image semantic priors Once the initialized
to construct high-quality initialized octree structure, as detailed in Sec. 3.4
octree query is obtained, we encode and refine the octree structure in Sec. describe how to supervise the network.
3.5
Finally, Sec. 3.6
3

---

Ocnee Encoder
Tempoca Self Attenionn
Imsse Cress Alteraus
licniive Stuaclree Rectilication
Baaiial Qway
Mullosiew Emages
Baashhon
Ing Featurcs
bones Quay
Senamtic Guided Ocave Inchollronion
8 8
Ocnee Decode:
prsestio
kaligayy
Dake Mat
Vocel Quecy
Colaacas Bugios
Octree
Occipancy Prediaction
Figure 2: Overall framework of OctreeOcc. From multi-view images, we extract multi-scale features using an image backbone. The initial octree structure is derived from image segmentation priors, transforming dense queries into octree queries. The octree encoder refines these queries and rectifies the octree structure. Finally, we decode the octree queries to obtain occupancy predictions. The diagram of the Iterative Structure Rectification module shows the octree query and mask in 2D (quadtree) form for better visualization.
3.1 Problem Setup
Camera-based occupancy prediction aims to predict the present occupancy state and semantics of each voxel grid within the scene using input from multi-view camera images. Specifically, consider a set of N multi-view images / {1 E IR HxW x Ji=1 together with camera intrinsics = (Ti E 14x41N Ji=1 as input, and the objective of the model K = (K₃ E 12x33,N and extrinsics T'
to predict the 3D semantic voxel volume O E {wo, W1, .... wc) XxXxxxxx where H. W indicate the resolution of input image and X,Y. Y. Z denote the volume resolution (e.g. x00x 16). The primary focus lies in accurately distinguishing the empty class (wn) and other semantic classes (w) Wc) for every position in the 3D space, which entails the network learning both the geometric and semantic information inherent in the data.
3.2 Overview
Given set of multi-view images I = {4 E R fi-1s we extract multi-view image fea- tures FF = (Fr E @Hxxxxoa Ji=1- Simultaneously, we randomly initialize the dense voxel query Qdense E RXXYXZxC To enhance computational efficiency, we transform Qdenee into sparse representation Quctree NxC leveraging octree structure information (i.e. octree mask) derived from segmentation priors. During encoding, we utilize Qoetree to gather information, including temporal fusion and sampling from image features F, while also rectifying the octree structure. Upon encoding Qoctrees to conform to the output format, we convert it back to Quenge and apply a Multi-Layer Perceptron (MLP) to obtain the final occupancy prediction O RXxY x2xX Here, H, W indicate the resolution of input image and X, Y. Z denote the volume resolution. N means the number of octree query .C denotes the feature dimension and K indicates the number of classes.
3.3 Octree Query
Given that objects within 3D scenes exhibit diverse granularities, employing dense queries 6 4) overlooks this variation and leads inefficiency. To address this, propose sparse and multi- granularity octree queries, leveraging the octree structure. This approach creates adaptable voxel representations tailored semantic regions different scales.
Octree Mask. To effectively construct the octree query, it's essential to understand its underlying structure. An octree partitions each node into eight child nodes within 3D space, each representing

---

equal subdivisions of the parent node. This recursive process begins with the initial level and proceeds with gradual splitting. At every level. a voxel query serves either as a leaf query it remains unsplit or as a parent query if it undergoes division. We obtain this geometric information by maintaining 2 }/=-1 where X, Y, denote the ground truth's 's volume resolution. The L denotes the depth of the octree, representing the existence of L different resolutions of queries, with - 1 splits being performed from the top to the bottom of the octree. The value in the octree mask represents the probability that voxel at that level requires split, which is initialized through segmentation priors (Sec. 3.4), rectified during query encoding (Sec. 3.5). and supervised by octree ground truth (Sec. 3.6)
learnable octree mask, denoting as M, = {M] E
R
Transformation between octree query and dense voxel query. During the query encoding process, we prioritize efficiency by leveraging octree queries. This involves transforming the initial dense queries Qennse into octree queries Qoetree using learned structural information. To determine the octree structure, we need to binarise the learned octree mask. Since most of the voxels in the scene at various resolutions do not necessitate splitting, neural network-based prediction binarization susceptible to pattern collapse, tending to predict all as non-split, leading to a decrease in performance. To mitigate this issue, we introduce a manually defined query selection ratio, where a subset of voxels with the highest probability of splitting is selected for division through the top-k mechanism.
The transformation from Qdense Q octree begins the finest granularity, we downsample Qdense to each level through average pooling and retain queries that do not require splitting (leaf queries) with the assistance of the binarized octree mask. This process iterates until reaching the top of the octree. By retaining all leaf queries, we establish Qoetree E R.NCC where N = N1 + N2 represents the total count of leaf queries, indicates the depth of octree. Conversely, applying the inverse operation of this process allows the conversion of Quntroe back into Qdense for the final output. Further details are provided in Appendix.
Nz
3.4 Semantic-Guided Octree Initialization
Predicting octree structure from initialised query via neural network can yield sub-optimal results due to the inherent lack of meaningful information in the query. To overcome this limitation, we employ the semantic priors inherent in images as crucial references. Specifically, we adopt UNe to segment the input multi-view images and obtain the segment map Ssgg = (Fee ggg E R xx >
We then generate sampling points p (p: x3xXxx2 with each point corresponding to the center coordinates of dense voxel queries. Subsequently, we project these points onto various image views. The projection from sampling point p; (si, Mis zi) to its corresponding 2D reference point (uig; Viy) on the j-th image view is formulated as:
T (p:) (.y)
(1)
where (p:) denotes the projection of the i-th sampling point at location p; the j-th camera view. We project the point p: onto the acquired semantic segmentation map [sg through the described projection process. To ensure the prioritization of crucial areas such as foreground objects and buildings in the initial structure, we adopt an unbalanced weight assignment method. Here, the highest weight is allocated to sampling points projecting onto foreground arcas, with decreasing weights assigned to points projecting onto buildings or vegetation, and the lowest weight designated for points projecting onto roads, among others. Subsequently, the voxel's weight determined as the average of the weights of all its sampling points. During this process, we determine the weights of each voxel at the finest granularity, denoted as W Xxxx2 Subsequently, average pooling to downsample W to each level of the octree, resulting in an initial octree mask M, {M* A 3 Here, X, Y. and Z represent the resolution of the ground truth volume, while L denotes the depth of the octree. Further details are provided in the Appendix.
employ
3.5 Octree Encoder
Given octree queries Qoetree and extracted image features F, the octree encoder updates both the octree query features and the octree structure. Referring to the querying paradigm in dense query- based methods 25), we adopt efficient deformable attention(48) for temporal A and image cross.)ttention((CAA....
5

---

Figure 3: Illustration of octree structure rectification. The left figure shows the initially predicted octree structure, while the right figure displays the structure after rectification. It's evident that the rectification module improves the consistency of the octree structure with the object's shape.
In accurately representing the driving scene, temporal information plays a crucial role. By leveraging historical octree queries Qi-1, we align it to the current octree queries by the ego-vehicle motion transformation. Given historal octree queries Qt-1 E NR,,C,, current octree query a located at p (x, y, =), the TSA is represented by:
M
TS Q.-1)= DeformAttnlq.p Qt-1);
where M1 indicates the number of sampling points. Implementing within voxel-based self- attention ensures that each octree query interacts exclusively with local voxels of interest, keeping the computational cost manageable.
Image cross-attention is devised to enhance the interaction between multi-scale image features and octree queries. Specifically, for an octree query q. we can obtain its centre's 3D coordinate (ar y, =) as reference point Refz.g.z. Then we project the 3D point to images like formula | and perform deformable attention:
ICA(FF)
Σ Σ BEN
DeformAlln(y, FalalllggsFaaaae (ReeRfz.g.)) Fn),
N
where denotes the camera view, m indexes the reference points and Ma is the total number of sampling points for each query. F,, the image features of the n-th camera view.
Iterative Structure Rectification Module. The initial octree structure, derived from image segmenta- tion priors, may not precisely match the scene due to the potential segmentation and projection errors. Nonetheless, the encoded octree query captures erucial spatial information. Thus, the predicted octree structure based on this query complements and rectifies the initial structure prediction, allowing us to mitigate limitations arising from segmentation and projection issues.
Specifically, we partition the octree structure into two parts: the high-confidence regions and the low-confidence regions, Fig. 2
shows. By sorting the octree split probability values stored in the octree mask in descending order and selecting the top k% of regions at each level, we can identify the locations of high-confidence regions. For these regions, predictions are relatively more accurate and no additional adjustments are required in this iteration. For regions where confidence remains low, we first extract the query features corresponding to those areas by utilizing the index low-confidence regions, denoted as Q (Q)... E R NiC | where Nj represents the number of low-confidence queries in level I. We then employ MLP to predict octree split probabilities from Qier- Subsequently. we apply weighted sum with the previous split probability predictions low-confidence regions to obtain rectified predictions. These refined predictions are concatenated with the preserved probability predictions high-confidence regions, culminating in the generation of the final rectified octree mask. It is worth noting that, due to the iterative nature of structure updates, regions initially considered high confidence may not necessarily remain unchanged, as they might be partitioned into low-confidence regions in the next iteration. More details shown in Appendix.

---

3.6
Loss Function
To train the model, we use focal loss focals lovasz- softmax loss Lis, dice loss diice, affinity loss Lecul and L scal from MonoScene In addition, we also use focal loss to supervise the octree prediction. The overall loss function L focnt + Lis Liscn L Lgeo + Leem + Loctree-
seal
Experiments
In this section, first introduce the datasets (Sec. evaluation metrics (Sec. 4.2). and imple- mentation details (Sec. 4.3). Subsequently, we evaluate our method on 3D occupancy prediction and semantic scene completion tasks (Sec. 4.4) demonstrate its effectiveness. Additionally, conduct extensive ablation studies and provide more analysis (Sec. 4.5) of our method.
4.1 Datasets
Ree3D-nuScenese re-annotates the nuScenes dataset(49) with precise occupancy labels derived from LiDAR scans and human annotations. It includes 700 training instances and 150 validation instances, with occupancy spanning -40m to 40m in X and Y axes, and -Im to 5.4m in the Z-axis. Labels are divided into 17 classes, with each class representing a volumetric space of 0.4 meters in each dimension, plus 1 18th "free" category for empty regions. The dataset also provides visibility masks for LiDAR and camera modalities.
SemanticKITTIC comprises 22 distinct outdoor driving scenarios, with focus areas located the forward trajectory the vehicle. Each sample in this dataset covers a spatial extent ranging from [0.0m, -25.6m, -2.0m, 51.2m, 25.6m, 4.4m], with a voxel granularity set at [0.2m, 0.2m, 0.2m]. The dataset consists of volumetric representations, specifically in the form of 256x256x32 voxel grids. These grids undergo meticulous annotation with 21 distinct semantic classes. The voxel data is derived through a rigorous post-pun proscesonsssing procedure applied to Lidar scans.
4.2 Evaluation metrics
Both 3D occupancy prediction and semantic scene completion utilize interseetioon-vver-miio (mloU) over classes as evaluation metrics, calculated as follows:
TP.
mloU
+P.
where TPc, FP., and FN, correspond to the number of true positive, false positive, and false negative predictions for class c; and C' is the number of classes.
4.3 Implementation Details
Based on previous research, we set the input image size 900x 1600 and employ ResNet101- 5C as the image backbone. Multi-scale features are extracted from the Feature Pyramid Network with downsampling sizes of 1/8, 1/16, 1/32, and 1/64. The feature dimension C is set to 256. The octree depth is 3, and the initial query resolution is 50x50x4. We choose query selection ratios 20% and 60% for the two divisions. The octree encoder comprises three layers, each composed of TSA, ICA, and Iterative Structure Rectification (ISR) modules. Both Mi and M2 are set to 4. TSA, we fuse four temporal frames. In ISR, the top 10% predictions are considered high-confidence in level 1, and 30% in level 2. The loss weights are uniformly set to 1.0. For optimization, we employ Adam(53 (53) optimizer with learning rate of 2e-4 and weight decay of 0.01. The batch size is 8, and the model is trained for 24 epochs, consuming around 3 days on 8 NVIDIA A100 GPUs.
4.4 Results
3D Occupancy Prediction. In Tab. we compare our method with other SOTA occupancy prediction methods on Oce3d-nus validation set. The performance FBOCC(3 relies on open-source code, which evaluate after ensuring consistency in details (utilizing the same backbone, image resolution.
7

---

Table I: 3D Occupancy prediction performance on Occ3D-nuScenes dataset. **" denotes training with the camera mask.
Mathod
Irage
Iruge
Reserrae
CNTR8-23
0.50
7:31
0.35
16-1
7.43
15.36
BecSer323
6305-1600
BE
23.67
0.6
38.78
6.88
1641
61/0
13.34
16.59
18:15
1735
19466
27.70
48.95
27.33
24.08
35.36
15.41
14.66
635-1660
CNTR6232
18.30
03.58
14.54
10.30
11.01
18.5
16.12
840%
ICE
31:5
33:38
34:0
13:41
50.94
6.7
4.6
Reckartii
5136 110
X3RA31
36:11
136
11.33
10723
3767
36/15
13.07
16.3
13
17
33.00
23.35
63.30
36
85
1136
18.56
76755
23.1
RaNaans
535
42.3
11.53
53.48
00.8
53.02
25.23
33.36
3335
83.01
08.3
34
58.04
43.06
40.00
R50000
190%
15.11
12.10
59.33
31.31
31.39
11.35
38.78
10.34
5838
17.7
11.66
Our"
RaNail
520-1800 1800
N0A
44/0
11.56
50.70
29.05
5363
56.77
30.30
30.17
93.65
39.90
30.76
43.87
85.77
08.52
35.45
58.06
49.53
46.3
Table 2: 3D Semantic Scene Completion performance on SemanticKITTI dataset.
3000
24lling
SHAMIO
Refereace
M0
LM
30V22
29.61
6.70
43:68
19.23
4.38
0.00
10.31
18.33
0.00
0.00
0.00
0.R1
13.56
0.22
20.54
0.00
0,00
0.63
1.21
0.00
0:00
AICNG
CypR:2 20
20.59
6.31
43.55
20-35
11.07
0.07
12:94
14.71
4.53
0.00
(.00)
QJM)
15.37
2.90
25-71
0.00
81.0
0U03
2.52
0.06
0.00
Ne Pg)
AAA731
0/5
53.49
11.34
11:94
0.00
15.05
34.05
2.41
0.00
0(0)
6.15
16.30
0,50
3.94
1.45
MonuSrzar
cyPR
16.06
1::05
55.47
36.73
14:27
0.56
14.09
15.36
6.68
0.1
8:45
1-43
11.89
1.81
79.4
1.86
1.30
0.00
1.81
4.16
2.36
TPVTomer
cyPR
19:1
11:34
55.99
30.0
13.88
33.81
3166
0.9
616
4.35
16.95
30.38
11.9
0U03
5.91
1.33
Occhverrrtt
10CV/22
58384
06.36
19.1
01:1
6.40
15.00
15,33
0.51
1.19
5.53
19.55
2.78
0.80
2.55
Symphonsa
CyPR24
02
14,09
55.37
27.58
15.25
08.85
31.64
36.66
30:44
2.54
282
13.89
25.71
6.48
30.67
3.52
2.34
0.00
8.40
0.57
5.76
and excluding CBGS for a fair comparison. Performance for other methods are reported in a Our approach demonstrates superior performance mIoU compared to them, particularly excelling in foreground classes such as barriers, cars, and buses, as well as in scene structure classes like manmade and vegetation. This highlights that processing scenes using multiple granularities aligns better with the scene characteristics, enhancing overall expressiveness.
series of (26 8 23
Moreover, we evaluate the efficiency of our approach by comparing it to alternative methods utilizing diverse query forms, as depicted in Tab 3 The results indicate that our approach not only surpasses these methods in terms of performance but also demonstrates significantly reduced memory usage and lower latency compared to dense queries, approaching the levels observed with 2D queries. Clearly, sparse octree queries effectively mitigate computational overhead while ensuring robust performance. Fig. 4 displays qualitative results obtained by our methods and some other methods, illustrating that approach comprehensively understands the structure of the scene, showcasing superior performance in scene understanding.
3D Semantic Scene Completion. To better evaluate the effectiveness of our approach, we conduct comparative experiments for the Semantic Scene Completion (SSC) task. As demonstrated in Tab 2
we compare our results with those of other SSC methods on the SemanticKITTI validation set. Our model demonstrates a more accurate perception of space due to octree construction and correction, outperforming others in IoU metric for geometry reconstruction. Additionally, for specific semantic classes such as car and vegetation, we achieve superior results, attributed to the enhanced treatment of objects facilitated by multi- gratularalrrtty modeling. Additionally, Tab also indicates that our method consumes fewer computational resources than other dense query-based methods.
Table 3: Comparison of query form and efficiency with SOTA methods on the Occ3D-nuSeenes
(left table) and SemanticKITTI (right table) datasset.
Methods
Query Form
mloU
Latency
Memory
Methods
Query Form
loU
mloU
Latency
Memory
BEVFormer 16
2D DEV
23.67
302ms
25100M
TPVFormet
2D tri-plane
35.61
11.36
179ms
23000M
Formert 8
tri-plane
341ms
OcdFormer 0
172ms
22/00M
3D voxel
43
502ms
35000M
VorFor
3D voxel
44.02
12.35
177ms
23200M
FBOOC 3
3D voxels & 2D DEV
43.41
463ms
31000M
Symphonics 1
3D voel
41.44
13:44
187ms
22000M
Ours
Query
44,02
386ms
26500M
Query
44.71
13.12
162ms
19000M
8

---

Table 4: Ablation experiments of Modules on
Table Comparison of octree structure
-nccennneeenss val set.
quality at different stages.
Octire Query Sem.Ini.
mãeU
$4.17
Lanency
266
27290M
Menooyy
Stage
level lo
level
54.91
218
18300M
Inirialized unbolancedd mesignment
45.79
33.60
60
56.3
214
(7900M
Initallized w. unbalanced usignment
$7.34
51.28
55.68
227
18500M
(6)
/
37,40
224
18500M
After the Ist Recetiiation After the 2nd Rectification
62.27
00.13
53.95
$6.79
Table 6: Ablation for different octree depth Occ3d-
Table 7: Ablation for the choice of query selection ratio on Occ3d- nuScenes val set.
nuScenes vol set.
Foon
Octaa Desa
Quary Rextatio
xdxcl
Mutwey
28.51
12h
74300
Selection Ruso
mioU
Lateray
Merioyy
30
yank 100 x-5
35.21
20icm
T12CM
54.47
207ma
16303M
30
Ostee Qwer
x13 -22- 40 1800-66
15 six shan 160h denani 160x
13.35
3485
153
M00
14R0M
36.73
255ms
225rr
21003M
15003M
1590M
37.40
224ns
18500M
4.5 Ablation Study and More Analysis
In this subsection, we perform ablation studies and analysis experiments on the Occ3d-nus validation set to assess the effectiveness of our proposed modules. All the experiments are conducted on the NVIDIA A40 GPU with reducing the input image size to 0.3x.
Effectiveness of Octree Queries. To validate the superiority of octree queries, we maintained consistent TSA and ICA settings while removing the proposed octree structure initialization and rectification modules. This facilitated a comparison with the baseline employing dense queries of size 100 100x 16. As illustrated in Tab. (b), experimental results consistently demonstrate our outperformance, with notable 0.8 mIoU advantage, despite achieving memory saving of approximately 9G. This underscores the adeptness of octree prediction in allocating queries with varying granularities diverse semantic regions. Moreover, the constructed octree queries exhibit adaptability to various object shapes, thereby optimizing the utilization of computational resources.
Effectiveness of Semantic-guided Structure Initialization module.To highlight the significance of the initial octree structure, we replaced the Semantic-guiee guideed Octree Initialization module with an MLP predicting the octree strueture from randomly initialized queries. This results in 1.7 mloU performance decrease, highlighting the inaccuracy of structurally predicted information from the initialized query due the absence of valid information coding. Incorporating semantic priors proves crucial as they enhance the quality of the initialized octree, thereby improving overall model performance. Meanwhile, Tab. 5)a) evaluates the effectiveness of the initial octree structure, which shows that assigning different initial weights to voxels based on their semantic regions improves the octree structure by focusing on the scene's effective areas.
Effectiveness of Iteritive Structure Rectification module. We perform an ablation study on the Iterative Structure Rectification module, shown in Tab, (d).. The incorporation of this module has led to noticeable improvements in performance. Meanwhile, Tab, Tab.55(b) shows this rectification gradually rectifies areas where structural predictions incorrect. Consequently, this refinement contributes to the efficiency of octree query expression, positively impacting overall performance.
Discussion on the depth of octree. Tab. pressenss experiments on octree depth variations. (a)-(d) show performance with varying 3D query sizes, while (c)-(h) depict octree query performance with different depths and initial resolutions. Comparing (c) to (c) and (f) to (d) reveals approach achieves comparable performance to dense queries, significantly reducing resource consumption. (g) shows that the predetermined octree depth should not be excessive. While reducing memory usage, the imperfect predictions of octree splitting result in accumulated errors, leading to performance degradation as the depth increases.
Discussion on query selection ratio of each level in the octree. In Tab. we present results for different query ratios at various octree levels. Results shows that inadequate queries result in an imperfect scene representation, especially for detailed regions (a.b vs e). Conversely, excessive queries impact computational efficiency, particularly for coarse-grained regions with empty spaces (c

---

vs d and c,d vs e). Optimizing query numbers based on scene object granularity distribution ensure effective processing of semantic regions of different sizes.
CAM FRONT LEFT
CAM FRONT
CAM FRONT RIGHT
CAM BACK LEFT
LEFT
CAM BACK
CAM BACK RIGHT
PanoOcc
FBOCC
Ours
Banier
Blcycle
Coun
Meter
Fed.
Inack
Toman
Marrrade
Vigetation
Figure Qualitative results on Oce3D-nuScenes val set, where the resolution of the voxel predictions is x00x 16.
Conclusions
In conclusion, our paper introduces OctreeOce, novel 3D occupancy prediction framework that addresses the limitations of dense-grid representations in understanding 3D scenes. OctreeOec's adaptive utilization of octree representations enables the capture of valuable information with variable granularity, catering to objects of diverse sizes and complexities. Our extensive experimental results affirm OctreeOcc's capability attain state-of-the-an performance in 3D occupancy prediction while concurrently reducing computational overhead.
Limitation. The quality of the octree ground truth depends the accuracy of the occupancy ground truth. Current occupancy ground truth comes from sparse lidar point clouds and surface reconstruction, leading low-quality results for some frames, which affects the octree construction.
References
[1] Z. Tan, Z. Dong, C. Zhang. W. Zhang, H. Ji, and H. Li, "Ovo: Open-vocabulary occupancy," arXiv preprint arXiv:2305.160 16133, 2023.
[2] Huang, G. Huang, Z. Zhu, Y. Yun, and D. Du, "Bevdet: High-performance multi-camert 3d object detection in bird-eye-view," arXiv preprint arXiv:2112.11799) 2021.
[3] 7. Li, 7. Yu, W. Wang, A. Anandkumar, T. Lu, and J. M. Alvarez, "Fb-bev: Bev representation from forward-backward view transformations." " in Proceedings of the IEEF/CV International Conference Computer Vision, pp. 6919-6928, 2023.
[4] Y. Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu, "Surroundoce: Multi-camera 3d occupancy prediction for autonomous driving," " in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 21729-21740, October 2023.
[5] H. Jiang, T. Cheng, N. Gao, H. Zhang, W. Liu, and X. Wang. "Symphonize 3d semantic scene completion with contextual instance queries," arXiv preprint wrXiv:2306. 15670, 2023.
[6] Y. Zhang, 7. Zhu, and Du, "Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction," arXiv preprint wtXxv:2304.033111 2023.
10

---

[7] Y. Huang, W. Zheng, Y. Zhang. J. Zhou, and J. Lu, "Tri- "Iri-perspective view for vision-based 3d semantic occupancy prediction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9223-9233 2023.
[8] Y. Wang, Y. Chen, X. Liao, L. Fan, and Zhang, "Panooce: Unified occupancy representation for camera-based 3d panoptic segmentation, arXiv preprint arXiv:2306. 10013, 2023.
[9] Y. Li, Z. Yu, C. Choy, C. Xiao, M. Alvarez, S. Fidler, C. Feng, and A. Anandkumar, "Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion," in Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9087-9098, June 2023.
[10] D. Meagher, "Geometric modeling using octree encoding," Computer graphics and image processing, vol. 19, no. 2. pp. 129-147, 1982.
[11] J. Philion and S. Fidler, "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d," in Computer EccinnEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210, Springer, 2020.
[12] Y. Li. H. Bao, Z. Ge. J. Yang, J. Sun, and Z. Li, "Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo," 2022.
[13] Y. Li, Z. Ge, G. Yu, J. Yang. Z. Wang. Y. Shi, J. Sun, and Z. Li, "Bevdepth: Acquisition of reliable depth for multi-view 3d object detection," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 1477-1485, 2023.
[14] Z. Liu, H. Tang, A. Amini, X. Yang. H. Mao, D. L. Rus, and S. Han, "Bevfusion: Multi-task multi-sensor fusion with unified bird's s-yye view representation," in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2781, IEEE, 2023.
[15] J. Park, C. Xu, S. Yang, K. Keutzer, K. Kitani, M. Tomizuka, and W. Zhan, "Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection," arXiv preprins 2rXiv:2210.0244
2022.
[16] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai, "Bevformer: Learning bird's s-eye-veew representation from multi-camera images via spatiotemporal transformers." in European conference on computer vision, pp. 1-18, Springer, 2022.
[17] C. Yang. Y. Chen, H. Tian, C. Tao, X. Zhu, Z. Zhang, G. Huang, H. Li, Y. Qiao, L. Lu, el al., "Bevformer v2: Adapting modern image backbones to bird's s-eye'-eee-view recognition via perspective supervision," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17830-17833 2023.
[18] Y. Jiang, L. Zhang, Z. Miao, X. Zhu, Gao, W. Hu, and Y.-G. Jiang, "Polarformer: Multi- camera 3d object detection with polar transformer." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 1042-1050, 2023.
[19] Y. Liu, T. Wang, X. Zhang, and J. Sun, "Petr: Position embedding transformation for multi-view 3d object detection," in European Conference on Computer Vision, pp. 531-548, Springer, 2022.
[20] Y. Liu, J. Yan, F. Jia, S. Li, A. Gao, T. Wang, and X. Zhang, "Petrv2: A unified framework for 3d perception from multi-camera images," " in Proceedings of the IEFF/CVE International Conference Computer Vision, pp. 3262-3272, 2023.
[21] S. Wang. Y. Liu, T. Wang, Li, and Zhang, "Exploring object-centric temporal modeling for efficient multi-view 3d object detection," arXiv preprint arXiv:2303.1 1926, 2023.
[22] X. Lin, T. Lin, Z. Pei, L. Huang, and 7. Su, "Sparse4d: Multi-view 3d object detection with sparse seapial-temporal fusion," arXiv preprint arXiv:22/1.1055: 2022.
[23] X. Tian, T. Jiang. I.. Yun, Wang, Y. Wang, and H. Zhao, "Oce3d: A large-scale 3d occupancy prediction benchmark for autonomous driving," arXiv preprint arXiv:2304.14365 14365. 2023.

---

[24] X. Wang, Z. Zhu, W. Xu, Y. Zhang, Y. Wei, X. Chi, Y. Ye, D. Du, J. Lu, and X. Wang, "Openoccupancy: large scale benchmark for surrounding semantic occupancy perception." arXiv preprini
2023.
[25] C. Sima, W. Tong. T. Wang, L. Chen, S. Wu, H. Deng, Y. Gu, L. Lu, P. Luo, D. Lin, and H. Li, "Scene as occupancy," 2023.
[26] M. Pan, J. Liu, R. Zhang, Huang, X. Li, L. Liu, and S. Zhang, "Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision," 2023.
[27] A. Cao and R. de Charette, "Monoscene: Monocular 3d semantic scene completion," " in Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3091-490911 June 2022.
[28] R. Miao, W. Liu, M. Chen, Z. Gong, W. Xu, C. Hu, and S. Zhou, "Occdepth: A depth-aware method for 3d semantic scene completion," arXiv preprint arXiv:2302. 13540, 2023.
[29] Y. Huang, W. Zheng, B. Zhang, J. Zhou, and J. Lu, "Selfoce: Self-supervised rision-based 3d occupancy prediction," 2023.
[30] Q. Ma, X. Tan, Y. Qu, I.. Ma, 7. Zhang, and Y. Xie, "Cotr: Compact occupancy transformer for vision-based 3d occupancy prediction," arXiv preprint arXiv:331.009119 2023.
[31] Z. Yu, C. Shu, J. Deng, Lu, Liu, J. Yu, D. Yang, Li, and Y. Chen, "Flashoce: Fast and memory-effiiient occupancy prediction via channel-to-heighee plugin," arXiv preprint arXiv:2311 /2058 2023.
[32] Z. Ming, J. S. Berrio, M. Shan, and S. Worrall, "Inversematrixvt3d:: An efficient projection matrix-based approach for 3d occupancy prediction," arXiv preprint arXiv:2401.12422 2024.
[33] H. Zhang, X. Yan, D. Bai, J. Gao, P. Wang, B. Liu, S. Cui, and Z. Li, "Radoce: Learning cross-modality occupancy knowledge through rendering assisted distillation," arXiv preprint arXiv:2312.1 18229 2023.
[34] S. Silva, S. B. Wannigama, R. Ragel, and G. Jayatilaka, "S2tpvformer: Spatio-temporal tri- perspective view for temporally coherent 3d semantic occupancy prediction," " arXiv preprint arXiv:2401. 13785, 2024.
[35] C. Häne, S. Tulsiani, and J. Malik, "Hierarchical surface prediction for 3d object reconstruction," in 2017 International Conference on 3D Vision (3DV), pp. 412-420, IEEE, 2017.
[36] C. H. Koneputugodage, Y. Ben-Shabat, and S. Gould, "Octree guided unoriented surface reconstruction." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16717-16726, 2023.
[37] J. Tang, Chen, J. Yang, B. Wang. S. Liu, B. Yang, and L. Gao, "Octfield: Hierarchical implicit functions for 3d modeling." arXiv preprint w:Xiv:21/1.0000 2021.
[38] P.-S. Wang, Y. Liu, and X. Tong. "Dual octree graph networks for learning adaptive volumetric shape representations," " ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1-15, 2022.
[39] P.-.. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong, "O-enn: Octree-based convolutional neural networks for 3d shape analysis," ACM Transactions On Graphics (TOG), vol. 36, no. 4, pp. 1-11, 2017.
[40] P.-S. Wang, C.-Y. Sun, Y. Liu, and Tong, "Adaptive o-cnn: A patch- deep representation of 3d shapes," ACM Transactions Graphics (TOG), vol. 37, no. 6. pp. 1-11, 2018.
[41] H. Lei, N. Akhtar, and A. Mian, "Octree guided cnn with spherical kernels for 3d point clouds." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9631-9640, 2019.
[42] M. Tatarchenko, A. Dosovitskiy, and T. Brox, "Octree generating networks: Efficient convolu- tional architectures for high-resolution 3d outputs," in Proceedings of the IEEE international conference computer vision, pp. 2088-2096, 2017.
12

---

[43] P.-S. Wang, "Octformer: Octree-based transformers for 3d point clouds," arXiv preprint arXiv:2305.03045, 2023.
[44] C. Zhou, Y. Zhang, J. Chen, and D. Huang, "Octr: Octree-based transformer for 3d object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5166-510 5175, 2023.
[45] C. Fu, G. Li, R. Song, W. Gao, and S. Liu, "Octattention: Octree-based large-scale contexts model for point cloud compression." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 625-633, 2022.
[46] Z. Que, G. Lu, and D. Xu, "Voxelcontext-net:: An octree based framework for point cloud compression." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
[47] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," " in Medical Image Computing and Computer-Assisted Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, Springer, 2015.
[48] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and Dai, "Deformable detr: Deformable transformers for end-to-end object detection." arXiv preprint arXiv:2010.04/55 2020.
[49] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: multimodal dataset for autonomous driving," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
[50] Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, "Se- mantickitti: A dataset for semantic scene understanding of lidar sequences," in Proceedings of the IEEE/CVF international conference on computer vision, pp. 9297-9307, 2019.
[51] Dai, H. Qi, Xiong, Li, G. Zhang, H. Hu, and Wei, "Deformable convolutional networks," in 2017 IEEE International Conference Computer Vision (ICCV), pp. 764-773, 2017.
[52] T.-.. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2117-2125, 2017.
[53] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," " 2017. 2017.
[54] B. Zhu, 7. Jiang, X. Zhou, 7. Li, and G. Yu, "Class-balanced grouping and sampling for point cloud 3d object detection," arXiv preprint arXiv:/906 1908.0949, 2019.
[55] J. Huang and G. Huang. "Bevdet4d: Exploit temporal cues in multi-camera 3d object detection," " arXiv preprint N.N:::2003.10554 2022.
[56] L. Roldao, R. de Charette, and A. Verroust-Blondet "Lmsenet: Lightweight multiscale 3d semantic completion," in 2020 International Conference on 3D Vision (3DV), pp. 111-119, IEEE, 2020.
[57] Li, K. Han, Wang, Y. Liu, and X. Yuan, "Anisotropic convolutional networks for 3d semantic scene completion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3351-3359, 2020.
[58] X. Chen, K.-Y. Lin, C. Qian, Zeng, and H. Li, "3d sketch-aware semantic scene completion via semi-supervise structure prior," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4193-4202, 2020.
[59] X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui, "Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from completion." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3101-3109, 2021.
13

---

Appendix
A
More Details
In this section, we provide detailed explanations of our proposed modules.
For the Semantic-Guided Octree Initialization, our approach commences with acquiring semantic segmentation labels for images by projecting occupancy labels onto the surround-view images. Subsequently, a UNet is trained using these labels. The initialization process entails randomly initializing dense queries, where each query's 's center point serves reference point projected onto the sunge-view images. If a reference point is projected onto a ground pixel (i.e., driveable surface, other flat, or sidewalk), the probability increases by 0.1. Conversely, if projected onto background pixel (excluding ground classes), the probability increases by 0.5. Projection onto a foreground pixel increases the probability of requiring split that position by 1.0. This process assigns split probability to each query. and the octree mask is constructed through average pooling, capturing split probabilities at different query levels. After obtaining the octree mask, designate the top 20% confidence queries as parent queries in level I, while the remaining queries become Leaf' queries and remain unsplit. Moving to level 2, after splitting the parent queries into octants, the top 60% confidence positions are selected as new parent queries, and the remainder as leaf queries. By storing leaf queries each level, we construct a sparse and multi graluiigaaularttyy octree structure for queries.
In Iterative Structure Rectification, at level 1. we retain predictions for the top 10% of positions with confidence. For the remaining positions, a 2-layer MI. is utilized to predict probabilities. These new probabilities are blended with the existing probabilities, with a weight distribution of 60% for the new probabilities and 40% for the old ones. The top 10% of positions with the new probability values are identified as the required splits, shaping the structure of the new level Similarly, at level 2. predictions for the top 30% of positions with confidence are preserved. For positions not in the top 30%, probabilities are predicted using a 2-layer MLP. The new probabilities are computed by merging them with the original probabilities, with an even weight distribution of 50% for each. The top 30% of the new probability values are then selected as the positions necessitating splitting, delineating the structure of the new level 2.
B Octree node index calculation
The hierarchical structure of the octree, particularly the assignment of queries to respective levels, is determined based the octree mask M, and the query selection ratio denoted as = { ....,,a--11 These ratios govern the number of subdivisions at each level, thereby defining the hierarchical organization of the octree. The procedure is as follows. For level queries with M¹ values within the top a' percentile are identified as candidates for octant subdivision. Subse- quently, within octants that have undergone one previous subdivision at the next level. queries are once again selected based on their values falling within the top a² percentile, initiating another round of subdivision. This process continues iteratively until reaching the final level of the octree.
Simultaneously,, exploiting the octree structure facilitates the direct conversion of sparse octree queries into dense queries to align with the desired output shape. For a query Gottre at level 1 with the index (a, b, c). the indexes of its corresponding 8L-1 children nodes in level L are determined by (a x 24-1 /offfett b x 24-1 + bof e 24-1 faet), where doffscts boffeet, and Coffset are independent, ranging from 0 to 2 Here, L denotes the depth of the octree. During this process, we allocate the feature of foctree to all of these positions. By iteratively applying this procedure to all queries at each level, we effectively transform Qoctree into Qdense-
C Octree Ground Truth Generation
We derive the octree ground truth from the semantic occupancy ground truth. Specifically, for a voxel at level 1 the octree, we identify its corresponding 84-1 voxels in the semantic occupancy ground truth. If these voxels share the same labels, we deem the voxel at level / unnecessary to split (assigned a value of 0): otherwise, it necessitates division (assigned a value of 1), as the current resolution insufficient to represent it adequately. Through this process, each voxel at each level is assigned
14

---

binary value of 0 or 1. Then obtain the octree ground truth Goctree (G E E J=1 Here, represents the depth of the octree, while X, Y, and Z denote the volume resolution of the semantic occupancy ground truth. Goctree is employed supervise the octree mask using focal loss, facilitating the network in learning the octree structure information.
D
More discussion of octree initialization
Given that the FloSP method outlined in MonoScene incorporates 3D to 2D projection oper- ation, similar to our initialization approach, we additionally adapted this method for comparison. Specifically, employed FloSP extract 3D voxel features from 2D image features. Subsequently, we applied a Multi-Layer Perceptron (MLP) to predict the splitting probability of each voxel, replac- ing the randomly initialized queries used in the original ablation experiments. The results indicate that, although this operation outperforms predictions from randomly initialized queries, it is still constrained by insufficient information, resulting in decline in overall performance.
Table 8: More ablation of octree initialization
Initialization Method
mloU
(a))
Randomly initialised queries
34.91
(b)
Voxel features from FLoSP
35.72
Semantic-Guided Octree Initialization
37.40
E Analysis of Various Usage of Octree.
As classic technique, octree is employed in various tasks 42 35 36
37
38)
Despite differences
in addressed problems, we compare our method with (GN(42) which proposes an octree- based upsampling approach. We keep the similar setup in Tab, (b) but substitute the deconvolution decoder with OGN's octree decoder. Results in Tab. iiddicate that employing octree solely in the decoder fails to mitigate excessive computational costs and yields sub-optimal performance, mainly due to the high query count during encoding.
Table 9: Comparison with another octree method.
mloU
Latency
Memory
baseline
34.10
266 ms
27200M
OGN(42)
33.39
212 ms
24300M
Ours
37.40
224 ms
18500M
F More Visualization
Fig. 5 shows additional visualizations of proposed OctreeOcc. Evidently, our approach, leveraging the multi-granularity octree modeling, demonstrates superior performance particularly the categories of truck, bus, and manmade objects.
Fig. illlustrats the results of occupancy prediction alongside the corresponding octree structure. For clarity in visualization, we employ distinct colors to represent voxels at various levels of the octree prediction, based on their occupancy status. For improved visualization, only portion correctly corresponding to the occupancy prediction is displayed, rather than the entire octree structure, ensuring clarity and focus on the relevant information. Level (voxel size: 0.4m x 0.4m x 0.4m) is depicted in light gray, level 2 (voxel size: 0.8m x 0.8m x 0.8m) in medium gray, and level (voxel size: 1.6m x 1.6m x 1.6m) in dark gray. It's worth noting that level voxels, predominantly situated in free space and within objects, might be less intuitively discernible. Nonetheless, this image underscores the efficacy of octree modeling, which tailors voxel sizes different semantic regions, enhancing representation accuracy.
15

---

CAM FRONT LEFT
CAM FRONT
CAM FRONT RIGHT
CAM BACK LEFT
CAM BACK
CAM BACK RIGHT
PanoOcc
FBOCC
Ours
GT
Bigycle
Bas
Car
Cana Vola
Moter
Pel.
Th
Tnder
Tuxh
Driee Sal..
Flx
Sdowi
Tomin
Mancade
Vegeation
Figure 5: More visualization on Oce3D-nuScent validation set. The first row displays input multi-view images, while the second row showcases the occupancy prediction results of PanoOce(8). FBOCC(), our methods, and the ground truth.
16

---

CAM FRONT LEFT
CAM FRONT
CAM FRONT RIGHT
CAM BACK LEFT
CAM BACK
CAM BACK RIGHT
Occupancy Prediction
Octree Prediction
Occ Pred:
Barrier
Bkcycle
Bas
Car
Const.
Veh.
Monor
Ped.
Taa. C.
Trailee
Trick Drive, Sur. Pir
S.devalk
Teman
Monnade Vegetation
Octree Pred:
Level
Level.2
Level
Figure 6: Visulization of octree structure. The first row displays input multi-view images, while the second and third showcase the occupancy prediction results and the corresponding octree structure prediction results.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract has mentioned the motivation of the paper to reduce the computa- tional overhead using octree technique, the details of the methodology and the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The conclusion section describes the limitations of the methodology
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification:
Our paper does not cover theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include all parameter settings, optimizer details, training resources, etc., for the model in the paper. Additionally, we outline how each experiment was conducted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will make the code publicly available upon acceptance of the paper to advance the field.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guudees/Codeeuuiii..
caapssionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplcccuua.eeaeaecccuee
Stteee//CooSSSemsssS.Si.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.3 shows all of the model details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The field of occupancy prediction that we study does not usually require eтror bars, confidence intervals, or statistical significance tests, and we have done extensive comparative and ablation experiments to demonstrate the validity of the method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The resources, time, etc. required for model training are provided in the Implementation Details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://wwerrppe.ccccupbb//////a
Answer: [Yes]
Justification:
Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our approach will facilitate the advancement of self-driving perception al- gorithms with less computational resource requirements that are practical real-world deployments.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our approach does not involve such data with pre-trained models. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
For the datasets used with the methods comparison, we have cited them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have not submitted new assets. If the paper is accepted, will make the code public soon.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "70tyUqkygA", "reviewer": "Reviewer_Mixn", "summary": "The authors aim to tackle the problem of high memory usage in dense occupancy prediction for 3D scenes. They introduce OctreeOcc, a method that uses octree structures to make predictions more efficiently. Experimental results show that OctreeOcc reduces computational load and achieves competitive performance.", "questions": "Q1: The proposed method seems to be quite time-consuming to train. Can the authors provide a comparison of training times with other methods? This is especially relevant given the emphasis on efficiency throughout the paper.\n\nQ2: There is some confusion regarding the reported memory consumption in the experiments. For instance, the original VoxFormer [1] paper reports memory usage of less than 16GB, but the authors here report around 23GB. What accounts for this significant discrepancy? Are there differences in the settings or configurations used?\n\nQ3: The implementation details provided focus on the nuScenes dataset. Could the authors include the specifics of the implementation for the SemanticKITTI dataset as well?\n\nQ4: The paper lacks results or discussions on other commonly used datasets, such as SSCBench-KITTI360 [2]. Including these would provide a more comprehensive evaluation of the proposed method.\n\n[1] VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion\n\n[2] SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views", "limitations": "The authors have identified a significant limitation in their work: the dependency on the quality of the occupancy ground truth (GT). This reliance can lead to suboptimal performance if the GT data, derived from sparse LiDAR point clouds and surface reconstruction, is not accurate or complete.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The authors address the significant issue of high memory consumption in dense occupancy prediction, highlighting the importance of finding more efficient solutions.\n\nS2: The use of octree representation for occupancy prediction is a novel and effective approach, offering adaptive granularity for various object shapes and regions.\n\nS3: Experimental results show some reduction in memory usage and latency, demonstrating the efficiency of the proposed method.", "weaknesses": "W1: The proposed method seems difficult to optimize and relies on segmentation and historical information.\n\nW2: The results on the SemanticKITTI dataset do not achieve state-of-the-art performance, and the reduction in memory usage is not very significant.\n\nW3: The training process is complex and costly, requiring three days on 8 A100 GPUs, which seems less efficient compared to previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **The proposed method seems difficult to optimize and relies on segmentation and historical information**\n\n   Regarding optimization, our method uses the same optimizer settings as other methods and is trained for 24 epochs as well. Under **the same training conditions**, our approach achieves SOTA performance on multiple benchmarks.\n\n   Our segmentation component is integrated into the model as a whole and is used to obtain a more accurate initialization. Importantly, **the input to the framework remains unchanged**, continuing to use surround-view images. As shown in Table 4 of the original paper, even without semantic-guided initialization, our approach still outperforms the baseline (34.17 vs. 35.88) while reducing memory consumption (27200M vs. 18500M).\n\nFurthermore, incorporating historical information is **standard** in all occupancy prediction methods, including ours. Including historical information is optional, and using only the current frame does not affect the model's inference.\n\nW2. **Results on the SemanticKITTI**\n\n   In our experiment on SemanticKITTI, we used the same set of octree hyperparameters as for nuScenes, achieving SOTA IoU, which highlights the effectiveness of our approach in capturing the overall spatial structure.\n\nFor smaller objects (e.g., bicyclists, poles), our IoU is limited by the three-level octree structure. The 32 × 32 × 4 resolution of the first level may not fully capture small objects, leading to early information loss. This represents a **trade-off** between performance and computational resources; increasing the number of queries generally improves performance but significantly raises memory overhead.\n\nRegarding memory overhead reduction, since model sizes for different methods on SemanticKITTI are small, the potential for memory reduction is limited. However, our memory reduction percentage on SemanticKITTI remains considerable.\n\nOverall, the current results sufficiently validate the effectiveness of our approach. We will continue to fine-tune the hyperparameters for SemanticKITTI and explore better trade-offs to achieve improved performance across different datasets.\n\nW3&Q1. **The training time**\n\n   We present the results of the comparison of the training time of each model on the nuScenes dataset, which show that our method **does not significantly increase training time** compared to other methods.\n\n   | Method    | Training Time |\n   | --------- | ------------- |\n   | BEVFormer | 61h           |\n   | PanoOcc   | 67h           |\n   | FBOCC     | 63h           |\n   | Ours      | 71h           |\n\nQ2. **Memory consumption of VoxFormer**\n\n   The memory consumption reported in the VoxFormer paper accounts only for the stage-2. For a fair comparison, our calculations include both the stage-1 and stage-2. Additionally, the memory consumption we report for our method also includes the segmentation part of the model. \n\nQ3. **Implementation details for SemanticKITTI**\n\n   The setup for SemanticKITTI remains essentially the same as in the previous method. We use ResNet50 as the backbone with an image size of 1220x370. The Adam optimizer is employed, with a learning rate of 2e-4 and a weight decay of 0.01. The octree-related hyperparameters are consistent with those set for the nuScenes dataset. We will include these details in the final version.\n\nQ4. **Results on SSCBench-KITTI360**\n\nWe provide experiment results on SSCBench-KITTI360, and with the same efficient framework and settings, our method also outperforms other methods in both metrics.\n| Method    | IoU   | mIoU  |\n| --------- | ----- | ----- |\n| TPVFormer | 40.22 | 13.64 |\n| OccFormer | 40.27 | 13.81 |\n| VoxFormer | 37.76 | 11.91 |\n| Ours      | 40.89 | 14.03 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRxZcrbnvD", "reviewer": "Reviewer_5724", "summary": "This paper introduces OctreeOcc, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. The authors incorporate image semantic information to improve the accuracy of initial octree structures and design an effective rectification mechanism to refine the octree structure iteratively. Extensive evaluations show that OctreeOcc not only surpasses state-of-the-art methods in occupancy prediction, but also achieves a 15% − 24% reduction in computational overhead compared to dense-grid-based methods.", "questions": "please refer to weaknesses.", "limitations": "NaN", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- OctreeOcc utilises octree to represent 3D space, which is novel and no one has done it before as far as my knowledge.\n\n- Owing to the design of octree queries, OctreeOcc improves the accuracy of occupancy prediction with the latency also decreasing.\n\n- Extensive experiments on Occ3D and SemanticKitti demonstrate the effectiveness of OctreeOcc.", "weaknesses": "- The pipeline is quite complex and heavily relies on manual operations, such as the selection strategy in Iterative Structure Rectification Module. My concern is that the complicated design may reduce the generalization.\n\n- The updating processing of octree queries is not very clear. In my understanding, if the octree mask changes, the newly generated octree queries cannot directly match the original octree queries.\n\n- Different octree queries may have different granularity, but, as shown in Eql (3), all octree queries seem to interact with image features only referring to their centers, which ignoring the influence of granularity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **Too many manually set hyperparameters affect generalisability**\n\n   Octree-related hyperparameters focus on selecting the top-k operation’s K during octree query sparsification and rectification. \n\n   These hyperparameters are statistically derived from the dataset and are not difficult to determine.\n\n   The experiments conducted on two datasets in the original paper used **the same set of octree hyperparameters**. Table 2 shows that applying the octree hyperparameters from nuScenes to SemanticKITTI resulted in SOTA performance on the IoU metrics and performance comparable to SOTA on the mIoU metrics. These experiments validate the generality of our approach.\n\nW2. **Updating processing of octree queries**\n\n   When the octree mask changes, the strategy for adjusting queries in the affected region is to first convert to a dense form and then adapt to the new sparse structure based on the updated octree mask.\n\nW3. **Octree queries focus only on their centers, neglecting granularity effects**\n\n   Thank you for the suggestion. We have compared the sampling scheme you mentioned (as shown in the Table below), where we assign different numbers of reference points to queries of varying granularity during cross-attention. For example, the finest granularity query uses only its centroid, while the medium granularity query samples four additional points around its centroid. \n\n   This approach can lead to performance gains but also increases memory overhead due to the effective increase in the number of queries. Our original design **balances memory overhead with performance**; regions with coarse query granularity are likely part of the same object, so sampling features only at the query's center is adequate. Overall, finding better ways to utilize octree queries will be our next step in enhancing our work.\n\n   | Sampling Method      | mIoU  | Memory |\n   | -------------------- | ----- | ------ |\n   | Original             | 37.40 | 18500M |\n   | More sampling points | 38.12 | 23100M |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTBJ5qPLOd", "reviewer": "Reviewer_NcWJ", "summary": "This paper introduces OctreeOcc, aiming to tackle the heavy computational demands of the dense and regular grid representations employed by the previous methods. Instead of randomly initializing the octree structure, OctreeOcc incorporates the semantic priors of images as guidance. The octree structure are further updated iteratively to correct the potential errors.", "questions": "Please see above weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Octree representation is a good solution to the heavy computation burden of the dense grid representation. OctreeOcc presents the details of the network components clearly.\n2. OctreeOcc provides detailed ablation experiments.\n3. OctreeOcc achieves good balance between the accuracy and the efficiency.", "weaknesses": "1. The image resolution and visible mask should be marked for fairly comparing with other methods. To my understanding, using the visible mask or not results in significant performance differences.\n2. This paper mentioned temporal fusion in line 131. Is the temporal information obtained similar to the bevformer? Besides, how many frames are employed? (e.g., 4 frames in the PanoOcc)\n3. The metrics of the Symponies in the tables should be updated. The performance differences are acceptable because Symponies uses stereo information (If OctreeOcc were to rely solely on monocular input).\n4. Dose the Memory presented in the tables refer to training memory? To my understanding, the inference memory for most of the methods are highly lower than this.\n5. In Table 6, query form of octree attend to the image features in a hierarchical manner, what is the number of the attention layers of the 3D voxel?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **The image resolution and visible mask should be marked**\n\n   In Table 1 of original paper, we use \"$\\star$\" to indicate whether a method uses a camera mask during training. Methods marked with \"$\\star$\" are the latest SOTA methods.\n\n   The image sizes used in training for each method are as follows:\n\n   | Method    | Image Resolution |\n   | --------- | ---------------- |\n   | BEVFormer | 900 × 1600       |\n   | TPVFormer | 900 × 1600       |\n   | OccFormer | 896 × 1600       |\n   | CTF-Occ   | 900 × 1600       |\n   | RenderOcc | 512 × 1408       |\n   | BEVDet4D  | 512 × 1408       |\n   | PanoOcc   | 900 × 1600       |\n   | FB-OCC    | 640 × 1600       |\n   | Ours      | 900 × 1600       |\n\n   All methods use settings close to full-size images within their respective ranges, ensuring a fair comparison.\n\nW2. **Temporal Fusion Details**\n\n   Yes, the temporal fusion method is the 3D version of BEVFormer’s TSA, using a fusion of four frames, consistent with PanoOcc.\n\nW3. **The metrics of the Symponies in the tables should be updated**\n\n   Thanks for your reminding, we will follow the number in its updated version and update it in our final version. \n\nW4. **Memory presented in the tables**\n\n   Yes, the table shows memory consumption during training.\n\nW5. **The number of the attention layers of the 3D voxel**\n\n   As shown in implementation details, octree encoder consists of three layers, each composed of TSA, ICA, and ISR modules.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uy3vTBWf3y", "reviewer": "Reviewer_mQ4T", "summary": "This paper proposes a new occupancy predictor that utilizes the octree structure to reduce the total number of occupancy query. The octree tree is initialized with semantic information and recursively rectified, so that it can accurately represent the structure of the driving scenes.", "questions": "1. It is not clear how the high-confidence and low-confidence regions are determined in Iterative Structure Rectification. Is the confidence related to the split probability?\n2. How does OctreeOcc handle the conflict between the octree queries initialized by current semantic information and the octree queries aligned from the previous frame?", "limitations": "The authors have described the limitation of this work, and there is no potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work gives a new way to represent occupancy instead of voxel feature or dense query, which is enlightening for later works.\n2. The prediction precision of OctreeOcc exceeds FB-Occ and PanoOcc, indicating a new SOTA. The visualization results also demonstrate its ability to predict fine structures.", "weaknesses": "1. OctreeOcc requires extra segmentation model to provide the reference for the initialization of octree queries, which reduces the efficiency gain. More details of the segmentation model should be published.\n2. The ablation study shows that the accuracy is largely improved after using the extra semantic information. A fair comparison between OctreeOcc without extra semantic information and other methods is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **More details of the segmentation model**\n\n   Segmentation models are trained using homologous data and then integrated into the network as fixed components, being inferred together rather than as additional separate modules. In this setup, a UNet architecture is used, where the encoder part is shared with the backbone's ResNet, and the decoder consists of a three-layer deconvolution. This design minimizes additional memory overhead.\n\n   Thank you for the suggestion. We will include these details in the revised version.\n\n\nW2. **Comparison between OctreeOcc without extra semantic information and other methods**\n\n   The **input and output** of our network are **the same** as those of other methods (i.e., input surround-view images and output occupancy prediction result). We trained a sub-network using the same data and incorporated it into the overall framework without relying on additional semantic information. The segmentation network is an integral part of our model, not a separate component. Theoretically, **we do not use extra information**.\n\n   Semantic segmentation is useful for initializing the octree structure; however, even without this component, our network's performance remains comparable to the dense voxel method (as shown in the table below).\n\n   | Method                   | mIoU  | Memory |\n   | ------------------------ | ----- | ------ |\n   | PanoOcc                  | 42.13 | 35000M |\n   | FB-OCC                   | 43.41 | 31000M |\n   | Ours                     | 44.02 | 26500M |\n   | Ours w/o sem-guided init | 42.08 | 25700M |\n\n\nQ1. **How the high-confidence and low-confidence regions are determined**\n\n   As shown in Section 3.5 of original paper, a split probability is maintained throughout the process, dividing high and low confidence areas using this probability and top-k. In this context, confidence actually refers to the split probability.\n\n\nQ2. **How does OctreeOcc handle the conflict between the octree queries initialized by current semantic information and the octree queries aligned from the previous frame?**\n\n   During temporal fusion, the octree query of the historical frame must first be converted into the same structure as the current frame during ego-motion conversion to ensure that fusion can be performed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript presents a novel efficient method for occupancy prediction in LIDAR scans. The proposed method uses octrees to reduce the computational footprint of previous methods while retaining competitive generalization performance. The reported improvements are quite incremental, but future work may lead to further improvements.
Decision: Accept (poster)