Context and Geometry Aware Voxel Transformer for Semantic Scene Completion
Authors: Zhu Yu
Runmin Zhang
Jiacheng Ying
Junchen Yu
Xiaohai Hu
Lun Luo
Si-Yuan Cao
Hui-liang Shen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Transformer
Semantic Scene Completion
Depth Estimation
Occupancy Prediction
Primary area: machine_vision

Abstract: Vision-based Semantic Scene Completion (SSC) has gained much attention due to its widespread applications in various 3D perception tasks. Existing sparse-to-dense approaches typically employ shared context-independent queries across various input images, which fails to capture distinctions among them as the focal regions of different inputs vary and may result in undirected feature aggregation of cross-attention. Additionally, the absence of depth information may lead to points projected onto the image plane sharing the same 2D position or similar sampling points in the feature map, resulting in depth ambiguity. In this paper, we present a novel context and geometry aware voxel transformer. It utilizes a context aware query generator to initialize context-dependent queries tailored to individual input images, effectively capturing their unique characteristics and aggregating information within the region of interest. Furthermore, it extend deformable cross-attention from 2D to 3D pixel space, enabling the differentiation of points with similar image coordinates based on their depth coordinates. Building upon this module, we introduce a neural network named CGFormer to achieve semantic scene completion. Simultaneously, CGFormer leverages multiple 3D representations (i.e., voxel and TPV) to boost the semantic and geometric representation abilities of the transformed 3D volume from both local and global perspectives. Experimental results demonstrate that CGFormer achieves state-of-the-art performance on the SemanticKITTI and SSCBench-KITTI-360 benchmarks, attaining a mIoU of 16.87 and 20.05, as well as an IoU of 45.99 and 48.07, respectively. Remarkably, CGFormer even outperforms approaches employing temporal images as inputs or much larger image backbone networks.

Context and Geometry Aware Voxel Transformer for Semantic Scene Completion
Zhu Yu'
Runmin Zhang'
Jiacheng Ying' Junchen Yu!
Xiaohai Hu
Lun Luo
Si-Yuan Cao
Hui-Liang Shen
Zhejiang University
Ningho Innovation Center, Zhejiang University
University of Washington
"HAOMO.AI Technology Co., Ltd.
http://gtthub.ommm///bbiiigCCCoo
Abstract
Vision-based Semantic Scene Completion (SSC) has gained much attention due its widespread applications in various 3D perception tasks. Existing sparse- to-dense approaches typically employ shared context-independent queries across various input images, which fails to capture distinctions among them the focal regions of different inputs vary and may result in undirected feature aggregation of cross-attention. Additionally, the absence of depth information may lead points projected onto the image plane sharing the same 2D position or similar sampling points in the feature map, resulting in depth ambiguity. In this paper, we present novel context and geometry aware voxel transformer. It utilizes a context aware query generator initialize context-dependent queries tailored to individual input images, effectively capturing their unique characteristics and aggregating information within the region of interest. Furthermore, extend deformable cross- attention from 2D to 3D pixel space, enabling the differentiation of points with similar image coordinates based on their depth coordinates. Building upon this module, we introduce neural network named CGFormer to achieve semantic scene completion. Simultaneously, CGFormer leverages multiple 3D representations (i.e., voxel and TPV) to boost the semantic and geometric representation abilities of the transformed 3D volume from both local and global perspectives. Experimental results demonstrate that CGFormer achieves tfate-of-the-a art performance the SemanticKITTI and SSCBench- KITTI-360 benchmarks, attaining a mIoU of 16.87 and 20.05, as well as IoU of 45.99 and 48.07, respectively. Remarkably, CGFormer even outperforms approaches employing temporal images as inputs or much larger image backbone networks.
1 Introduction
Semantic Scene Completion (SSC) aims to jointly infer the complete scene geometry and semantics. It serves as a crucial step wide range of 3D perception tasks such as autonomous driving [11, 24], robotic navigation [42, 15]. mapping and planning [34]. SSCNet [40] initially formulates the semantic scene completion task. Subsequently, many LiDAR- based approaches [5, 51, 38] have been proposed, but these approaches usually suffer from high-cost sensors.
Recently, there has been shift towards rision-based SSC solutions. MonoScene [3] lifts the input 2D images to 3D volumes by densely assigning the same 2D features to both visible and occluded regions, leading many ambiguities. With advancements in bird's -eie--eee (BEV) perception [24, 54], transformer-based approaches [13, 47, 23] achieve feature lifting by projecting 3D queries from 3D
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Context-independent Queries
Context Context-dependent Queries (b) CGFormer (ours)
(a) VoxFormer
Figure 1: Comparison of feature aggregation. (a) VoxFormer [23] employs set of shared context- independent queries for different input images, which fails to capture distinctions among them and may lead to undirected feature aggregation. Besides, due to the ignorance of depth information, multiple 3D points may be projected to the same 2D point, causing depth ambiguity. (b) Our CGFormer initializes the voxel queries based on individual input images, effectively capturing their unique features and aggregating information within the region of interest. Furthermore, the deformable cross- cooss-attention is extended from 2D to 3D pixel space, enabling the points with similar image coordinates to be distinguished based on their depth coordinates.
space to image plane and aggregating 3D features through deformable attention mechanisms [61]. Among these, VoxFormer [23] introduces sparse-to-dee to-see-t- architecture, which first aggregates 3D information for the visible voxels using the depth-based queries and then completes the 3D information for non-visible regions by leveraging the reconstructed visible areas starting points. Building upon VoxFormer [23]. the following approaches further improve performance through self-distillation training strategy [43], extracting instance features from images [14], or incorporating an conde--ond cross- attsntnt module [60].
Despite significant progress. existing sparse-to-dense approaches typically employ shared voxel queries (referred to as context-independent queries) across different input images. These queries defined set of learnable parameters that independent of the input context. Once the training process is completed, these parameters remain constant for all input images during inference, failing to capture distinctions among various input images, as the focal regions of different images vary. Additionally, these queries also encounter the issue of undirected feature aggregation in cross- attention, where the sampling points may fall in irrelevant regions. Besides, when projecting the 3D points onto the image plane, many points may end with the same 2D position with similar sampling points in the 2D feature map, causing a crucial depth ambiguity problem. An illustrative diagram presented in Fig. ((a).
In this paper, propose context and geometry features. We observe that context-dependent query tend to aggregate information from the points within the region of interest. Fig. presents example of the sampling points of the context- dependent queries. Thus, before the aggregation of cross-attention, we first utilizes a context aware query generator predict context-aware queries from the input individual images. Additionally, we extend deformable cross-attention from 2D to 3D pixel space, which allows points ending with similar image coordinates to be distinguished by their depth coordinates, illustrated in Fig. 1(b). Furthermore, we propose a simple yet efficient depth refinement block to enhance the accuracy of estimated depth probability. This involves incorporating a more precise estimated depth map from a pretrained stereo depth estimation network [39], avoiding the heavy computational burden as observed in StereoScene [16].
voxel transformer (CGVT) lift the 2D)
Based on the aforementioned module, we devise a neural network, named as CGFormer. To enhance the obtained 3D volume from the view transformation we integrate multiple representation, iee., voxel and tri-perspective view (TPV). The TPV representation offers a global perspective that encompasses more high-level semantic information, while the voxel representation focuses more the fine-grained structures. Drawing from the above analyses, we propose a 3D local and global encoder (LGE) that dynamically fuses the results of the voxel- based and TPV-based branches, to further enhance the 3D features from both local and global perspectives.
Our contributions can be summarized as follows:
We propose a context and geometry aware voxel transformer (CGVT) to improve the performance of semantic scene completion. This module initializes the queries based on the

---

context of individual imput images and extends the deformable aross-attention from 2D to 3D pixel space, thereby improving the performance feature lifting.
We introduce simple yet effective depth refinement block to enhance the accuracy of estimated depth probability with only introducing minimal computational burden.
We devise a 3D local and global encoder (LGE) to strengthen the semantic and geometric discrininability of the 3D volume. This encoder employs various 3D representations (voxel and TPV) to encode the 3D features, capturing information from both local and global perspectives.
Benefiting from the aforementioned modules, our CGFormer attains of-te-of-the-ee art results with a mloU of 16.63 and an IoU of 44.41 on SemanticKITTI, as well as mloU of 20.05 Notably, our method even surpasses methods
and an loU of 48.07 on SSCBench-KKTTT1366 employing temporal images as inputs or using much larger image backhone networks.
2 Related Work
2.1 Vision-based 3D Perception
Vision-based 3D perception [24, 25, 31, 20, 12, 45, 62, 44, 19, 58, 48, 52, 53, 56] has received extensive attention due to its ease of deployment, cost-effectivennsss and the preservation of intricate visual attributes, emerging as crucial component in the autonomous driving. Current research efforts focus on constructing unified 3D representations (e.g., BEV, TPV, voxel) from input images. LiftSplat [36] lifts image features by performing outer product between the 2D image features and their estimated depth probability to generate a frustum-shaped pseudo point cloud of contextual features. The pseudo point cloud features then splatted to predefined 3D anchors through voxel-pooling operation. Building upon this, BEVDet [12] extends LiftSplat to 3D object detection, while BEVDepth [21] further enhances performance by introducing ground truth supervision for the estimated depth probability. With the advancements in attention mechanisms [4, 61, 57, 33, 45], BEVFormer [24, 54] transforms image features into BEV features using point deformable cross-attention [61]. Additionally, many other methods, such as OFT [37], Petr [28, 29], Inverse Perspective Mapping (IPM) [10], have also been presented transform 2D image features into a 3D representation.
2.2 Semantic Scene Completion
SSCNet [40] initially introduces the concept of semantic scene completion, aiming to infer the semantic voxels. Following methods [38. 18, 51, 5] commonly utilize explicit depth maps or LIDAR point clouds inputs. MonoScene [3] is the pioncering method for directly predicting the semantic occupancy from the input RGB image, which presents FLoSP module for 2D-3D feature projection. StereoScene [6] introduces explicit epipolar constraints to mitigate depth ambiguity, albeit with heavy computational burden for correlation. TPVFormer [13] introduces a tri-perspective view alternative the BEV representation. The
(TPV) representation describe the 3D scene, elements fall into the field of view aggregates information from the image features using deformable cross-attention [24, 61]. Beginning from depth-based [23] sparse proposal queries, VoxFormer [23] construct the 3D representation in a coarse-to-fine manner. The 3D information from the visible queries diffused the overall 3D volume using deformable self-attention, akin the masked autoencoder (M) HASSC [43] introduces a self-distillation training strategy to improve the performance VoxFormer [23], while MonoOcc [60] further enhance the 3D volume with an image- conditioned cross-attention module. Symphonize [14] extracts high level instance features from the image feature map, serving as the key and value the cross- cross-attention.
3 CGFormer
3.1 Overview
As shown in Fig. 2, the overall framework of CGFormer composed of four parts: feature extraction to extract 2D image features, view transformation (Section 3.2) to lift the 2D features to 3D volumes,

---

Foature Extraction
Transfomation
Local and
Decoding Hoad
product
Contert
Co Cancattonat:
Contert and Geometry Avere Vonel Transformer
Dopti mndictionn
Vidtic Tokan
Desth
Realh
Inviibic Tokon
bases branch
Context and Geometry Aware Vizel Transformer
Reiphorooo
Deptth Net
Contzot
Probobility
Legtt Aap
Belamnak:
Image
Duti
Saff S100100
Wishooood
Quanry Connerstor
Query Proprool
Proteciity
Vo
2
(b)
(c)
Figure 2: Schematics and detailed architectures of CGFormer. (a) The framework of the proposed CGFormer for camera-based semantic scene completion. The pipeline consists of the image encoder for extracting 21) features, the context and geometry aware voxel (CGVT) transformer for lifting the 2D features to 3D volumes, the 3D local and global encoder (LGE) for enhancing the 3D volumes and a decoding head to predict the semantic occupancy. (b) Detailed structure of the context and geometry aware voxel transformer. (c) Details of the Depth Net.
3D encoder (Section 3.3) to further enhance the semantic and geometric discriminability of the 3D features, and decoding head to infer the final result.
The image encoder consists of backbone network for extracting multi-scale features and a feature pyramid network (FPN) to fuse them. We adopt F E R to represent the extracted 2D image feature, where C' is the channel number, and (H, W) refers the resolution.
Image Encoder.
Depth Estimator. In alignment with VoxFormer [23], we use an off-the-shelff stereo depth estimation model [39] to predict the depth Z(u,e) for each image pixel (u, r). The resulting estimated depth map is then employed to define the visible voxels located on the surface, serving as query proposals. Additionally, it is also used to refine the depth probability for lifting the 2D features.
3.2 View Transformation
A detailed diagram of our context and geometry aware voxel transformer is presented in Fig. 2(b). The process begins with a context aware query generator, which takes the context feature map to generate the context-dependent queries. Subsequently, the visible query proposals are located by the depth map from the pretrained depth estimation network. These proposals then attend to the image features to aggregate semantic and geometry information based on the 3D deformable cross-attention. Finally, the aggregated 3D information is further diffused from the updated proposals to the overall 3D volume via deformable self-attention.
Context-dependent Query Generation. Previous coarse-to-fine -innee approaches [23, 43, 46, 60) typically employ shared context-independent queries for all the inputs. However, these approaches may overlook the differences among different images and aggregate information from irrelevant areas. In contrast, CGFormer first generates context-dependent queries from the image features using a context aware query generator. To elaborate, the extracted F2D is fed to the context net and the depth net [21] to generate the context feature C #XXCC CC and depth probability D IR H WXD respectively. Then the query generator f takes C' and D as inputs to generate voxel queries Vo E xxxxxxxxC'
Vo= D),
(I)
where (X, Y, 2) denotes the spatial resolution of the 3D volume. Benefiting from the initialization, the sampling points of the deeexxtdeeennent queries usually locate within the region of interest, An example of the deformable sampling points is displayed in Fig. 3.
Depth-based Query Proposal. Following VoxFormer [23], we determine the visible voxels through the conversion of camera coordinates to world coordinates using the pre-estimated depth map, except that we do not employ an additional occupancy network [38]. A pixel (u, w) will be transformed to 3D point (z, y, 2). utilizing the camera intrinsic and extrinsic matrices (K 11x4 E e R RAX1).

---

(a) RGB
(b) CGFormer (ours)
(c) Ground Truth
Figure 3: Visualization of the sampling locations for different small objects. The yellow dot represents query point, while the red dots indicate the locations of deformable sampling points. The sampling points of the context-dependest query (a) tend to be distributed within the regions of interest. Beneficial from this, CGFormer achieve better performance than previous methods.
The synthetic point cloud is further converted into a binary voxel grid map M, where each voxel is masked as 1 if occupied by at least one point and 0 otherwise. The query proposals are selected based on the binary M as
Q Vo(M).
(2)
3D Deformable Cross-Attention The query proposals then attend the image features aggregate the visual features of the 3D scene. We first expand the dimension of C from 2D to 3D pixel space by conducting the outer product between D and C. formulated F C @ D. Here, the operator & refers to the outer product conducted at the last dimension. Taking F as key and value, for a 3D query Op located at position p, the 3D deformable cross-attention mechanism can be formulated
Ô 3D-DDAA( F,P) Σ A, ((p)))) + Ap),
where indexes the sampled point from total of N points, P(p) denotes camera projection function to obtain the reference points in the 3D pixel space, A, E [0, is the learnable attention weight, and W denotes the projection weight. Ap R is the predicted offset to the reference point and (F,, Pa(p) + Apa) indicates the trilinear interpolation used to sample features the expanded 3D feature map F. Q denotes the updated query proposals. For simplicity, we only show the formulation of single-head attention.
Deformable Self-Attention. After several layers of deformable cross-attention, merge Q and the invisible elements of Vq. indicated as 3 The information the updated query proposals is diffused to the overall 3D volume via deformable self-attention and aDD indicates the updated 3D volume. This process of a query located at position p can be formulated as
E.D DS FDD .) Σ A. p Ap).
Depth Net. The depth probability serves the inputs of both the context aware query generator and 3D deformable cross-attentinn whose accuracy greatly influences the performance. The results of single image depth estimation are usually unsatisfactory. Incorporating epipolar constraint can help estimate more accurate depth map. However, computing correlation to achieve this may introduce much computation burden during the estimation of the semantic voxels. Instead, we introduce simple yet effective depth refinement strategy. The detailed architecture of the depth net is shown
5

---

in Fig. 2(c). We first estimate monocular depth feature DM from F The depth map generated from the stereo method [39] is encoded as stereo feature Ds by several convolutions. These two features are further processed by cross-attention blocks for information interaction. To save memory, the window size of aots-aatention is restricted within a range of the nearest neighboring pixels [7], which can be formulated as
DM (DM. Ds, Ds),
(5)
Dg = (Ds. DM, DM),
where DM and Ds denote the enhanced DM and Ds, and da denotes the neighborhood attention [7]. We set the window size to 5 5. Finally, these two volumes are concatenated and fed to estimate the final depth probability. This strategy can boost the accuracy of the depth probability lot with only minimal computation cost.
3.3
3D Local and Global Encoder
The TPV representation derived by compressing the voxel representation along its three axes. TPV focuses more on the global high-level semantic information, whereas the voxel emphasizes more on the details of intricate structures. We combine these two representations to enhance representation capability of the 3D features. As shown in Fig. 2(a), our 3D local and global encoder consists of the voxel-based and TPV-based branches. With the input 3D volume, the two branches first aggregate features in parallel. The dual-brance -all-branch outputs are then fused dynamically and fed to the final decoding block.
Voxel-based Branch. The voxel-based branch aims to enhance the fine tne-grained structure features of the 3D volume. We adopt lightweight ResNet [9] with 3D convolutions to extract multi-scale voxel features from F and merge them into F3D voxel E RXXXxZZC using 3D feature pyramid network.
TPV-based Branch. To obtain three-plane features of the TPV representation, we apply spatial pooling the voxel features along three axes. Directly performing MaxPooling may cause the loss of the fine-grained details. Considering the balance between performance and efficiency, we apply the group spatial to channel operation [50, 25, 62] to transform the voxel representation to TPV representation. Specifically, we split the voxel features along the pooling axis into K groups and apply the MaxPooling within each group. Then we concatenate each group features along the channel dimension and use convolutions to reduce the channel dimension into C. Denote the 2D view features as F E Rxxxxx xc, F E Rxxxxx Fij E R.Y x the above process can be formulated as (
F XY Conv (Concat (ConnctCConcat (Pooling(z.))
F.D
Convyz (anoicat(Pooinn)) (Pooling (x.)) (F3D)
1- )),
(6)
F X = (Coooooat(PPoollnnnnnn (Poolin 30D )
After obtaining the TPV features, we employ a 2D image backbone to process each plane to produce corresponding multi-scale features, which are then fused by a 2D feature pyramid network. For simplicity, we leverage F F&D
x Fiy to represent the outputs of these networks, as well.
Dynamic Fusion. The dual-path outputs are fused dynamically. We generate aggregation weights W E R xxxx2x4 from FAD using a convolution with a softmax on the last dimension. The final 3D feature volume F3D computed
F Finl Σ OF
where W; E RXxYxEx1 is a slice of the W and F is an element of the set FD F Fy,, F.D X2. J. The operation C refers to the element multiplication and the dimen- sion broadcasting of the features from TPV-based branches is omitted in the equation for conciseness. FåD inal is finally fed to the decoding block to infer the complete scene geometry and semantics. Please refer the supplementary material for more detailed schematics of the 3D local and global encoder.
3.4 Training Loss
To train the proposed CGFormer, cross-entropy loss weighted by class frequencies Lee is leveraged to optimize the network. Following MonoScene [3], we also utilize affinity loss Laea and Ceem to

---

Table 1: Quatitative results on SemanticKITTI [1] test set.
represents the reproduced results in [13, 59]. The best and the second best results are in bold and underlined. respectively.
sidewakk
parking
othere ggd.
bull building
bicycle
motoneyel golo
othee-seh
vegetation
bicyclos
midonnyssio
(oad
trunk
fance
pole
Method
IoU
mlaU
MonoScene" [3]
34,16
11.08
54.70
27.10
24.80
5.70
14.40
18.90
3.30
0.50
0.70
4.40
14,90
2.40
19.50
1.00
1.40
0.40
11.10
3.30
2.10
TPVFormer [13)
31.25
11.26
55.10
27.20
27.40
6.50
14.80
19.20
3.70
1.00
0.50
2.30
13.90
2.60
20.10
1.10
2.40
11.00
2.90
1.50
SumvundOoe [47]
56.90
1.20
OcdFormer [59]
34.53
12.32
55.90
30.30
31.50
6.50
15.70
21.50
1.20
1.50
1.70
3.20
16.80
3.90
21.30
2.20
1.10
0.20
11.90
3.80
3.70
IAMSSC [19]
13.74
12.37
51.00
25.50
24.70
6.90
19.20
21.30
3.50
1.10
0.60
3.90
22.70
5.80
19.40
1.50
2.90
0.50
11.90
5.30
4.10
MoxForme--S S [23]
42.95
12.20
53.90
25.30
21.10
5.60
19.80
20.80
3.50
1.00
0.70
3.70
22.40
7.50
21 30
1.40
2.60
0.20
11.10
5.10
4.90
VorFormesT [23]
13.41
1.60
24
1.10
DepoSSC [55]
27.25
5.78
0.98
23.37
2.79
12.94
5.87
Symghoonize [14]
42.19
15.04
58.40
29.30
26.90
11.20
24.10
23,60
3.20
3.60
2.60
5.00
24,20
10.00
23 10
3.20
1.90
2.00
16.10
7.20
8.00
HASSC-S [43]
43.40
13.34
$4.60
27.70
23,80
6.20
21.10
22.80
4.70
160
1.00
3.90
23.80
8.50
23
160
4.00
0.30
13.10
5.80
5.50
HASSC-T [13]
12.87
14.38
55.30
29.60
25.90
11.30
23.10
23,00
1
1,50
24.80
9.50
26 50
1.40
3.00
14,30
7.00
7.10
StareSoae [[6]
61.50
10.70
2.20
7.00
H2QFomer [46]
44.20
13.72
56.40
28.60
26.50
4.90
22.80
23,40
4.50
0.80
0.90
4.10
24.60
9.10
23.
1.20
2.50
0.10
13.30
6.40
6.30
I10ffvmar-T [16]
13.52
14.60
57.90
30.40
30,00
6.90
24.00
23.70
5.20
0.60
1.20
5.00
25.20
10.20
25
1.0
0.10
0.00
14.60
7.50
9.30
MonoOge-S [60]
13.80
55 20
27.80
25.
15.10
9.70
40
23.20
5.00
2.20
1,50
5.40
24.00
8.70
23.00 00
1.70
2,00
0.20
13:40
5.80
6.40
MonoOas-1, [60]
7.30
CGFormer (0urs)
16.33
64.30 34.20
12.00
25.80 26.10
3.0 1,50
24.50
29.30
1.70
3.60
18.70
8.70
Table 2: Quantitative results SSCBench-KITTISS test set. The results for counterparts are provided in [22]. The best and the second best results for all camera-based methods are in bold and underlined, respectively. The best results from the LiDAR- methods are in red.
bigyole
motorcycle
truck
other-ech
person
rood
parking
sidewalk
others othenggad gmd
building
vegotation
terran
pole
ttal -ign
otber
other- obj..
Method
LJDAR-based methods
SSCNa [40]
53.58
95
0.17
10.29
0:07
65
77
6.77
28.87
0.78
0.75
8.69
0.67
LMSCNe [5k]
47.35 13.65
20:91
0.00
0.00
0.26
0.58
0.00
62.95
13.51
33.51
0.30
43.67
0.33
40.01
26.80
0.000
0.00
3.63
0.00
Comeng-fured methods
MonoSeene [3]
45.35
11.38
3.32
26.15
4.20
3.09
TPVRomer [13)
40.22
13.64
21.
1.09
1.37
8.06
2.57
2.38
52.99
11.99
31.07
3.78
34.83
4.80
30.08
17.52
7.46
5.86
5.48
2.70
Octoomer [59]
40.27
13.81
22.58
0.66
0.26
9.89
3
2.77
$4.30
13,44
31.53
3.55
36
4.80
31,00
19:51
7.77
8.51
6.95
4.60
VoxHomer [23]
2.80
3.79
2.43
IAMSSC [19]
41.80
12.97
18.53
2.45
1.76
5.12
3.92
3.09
47.55
10.56
28.35
1.12
33.53
6.28
29.17
15.24
8.29
7.01
6.35
1.19
DepliSSC [55]
A0.85
14.28
21
90
3.36
4.30
11.51
4.56
2.92
50.88
12.89
20.27
2.49
37.53
5:22
29.61
21.59
5.97
7.71
5.24
3.51
Symphonies [14]
44,12
18.58
30.02
1.85
5.90
25.07
12.06
8.20
54.94
13.83
32.76
6.93
35
8.58
38.33
11.52
14.01
9.57
14.44
11.28
CGFonnnr (ours)
48.07
17.59
6.63
63.85
17.15
40.72
5.53
42.75
52
38.80
24.94
17.45
10.18
6.22
optimize the scene-wise and class-wise metrics (gcometric IoU and semantic mloU). We also employ explicit depth loss La [24] to supervise the estimated depth probability of the depth net. Hence, the overall loss function can be formulated as
L= daLa + Lee + Lyco + L
(8)
where we set the weight of depth loss Aa to 0.001.
4
Experiments
4.1
Quantitative Results
We conducted experiments to evaluate the performance of our CGFormer with the latest state-of- the-art methods on the SemanticKITTI [1] and CCCBBecch-KITT---0 [22]. Refer to the supplement material for information of datasets, metrics and detailed implementation details.
We list the results on SemanticKITTI hidden test set in Table 1, CGFormer achieves a mloU of 16.63 and an IoU of 44.41. Notably, CGFormer outperforms all competing methods in terms of mIoU, demonstrating its superior performance. Regarding IoU, CGFormer surpasses all other methods except DepthSSC [55]. While ranking 2nd in terms of IoU, CGFormer exhibits only a slight marginal difference of 0.17 in comparison to DepthSSC [55]. However, CGFormer significantly outperforms DepthSSC [55] by a substantial margin of 3.52 in terms of mloU, Additionally, compared
to Vox [23], HASSC-T [43], H2GFormer-t [46], and MonoOcc-L. [3], although they utilize temporal stereo image pairs as inputs larger image backbone networks, our CGFormer exceeds them a lot in terms of both metrics. For specific instances, CGFormer achieves the best or second-best performance on most of the classes, such road, sidewalk, parking, and building.
7

---

Table 3: Ablation study of the architectural components SemanticKITTI [1] validation set. CGVT: context and geometry aware voxel transformer. LGE: local and global encoder. 3D-DCA: 3D deformable attention. CAQG: context aware query generator. LB: local voxel-based branch.
Txy, zyz, zxz: planes of the TPV-based branch. DF: dynamic fusion. There are 32M predefined parameters.
CGVT
LGE
Method
IoUt
mioU
Params (M)
Memory (M)
3D-DCA CAQG
1XY
Tyz
1xz
Baseline
37.99
12.71
76.57
13222
(a)
40.14
86.17
15150
(b)
42.86
86.19
15488
(C)
16.41
93.78
17843
(d)
44.63
16.54
122.42
19188
(e)
45,46
16.38
122.12
19024
(0)
45.53
16.74
122.12
18912
(g)
45.71
16.49
122.12
18912
(h)
45.99
16.87
122.42
19330
Table 4: Ablation on the choices of context
Table 5: Ablation on the depth refinement block of the depth net.
aware query generator.
Memory
w/o CAQG
40.14
14.34
86.17
15150
who sterpo feature wio neighteee haod atmn.
116.44
More attention layers
41.83
14,43
86.97
21.996
45.72
16.26
122.32
19260
FLuSP [3]
41.54
14.66
86.19
15907
w SterenSeene [16)
45.55
16.49
156.50
22347
Pooling
15458
model
122.42
19330
To demonstrate the versatility of our model, we also conduct experiments on the SSCBench-KITTL 360 [40] dataset and list the results Table published camera- basdd methods by margin of 3.95 IoU and 1.67 mloU. Notably, CGFormer outperforms the two LiDAR-bare methods in terms mloU. The above analyses further verifies the effectiveness and excellent performance of CGFormer.
is observed that CGFormer surpasses
4.2 Ablation Study
We conduct ablation analyses the components of CGFormer on the SemanticKITTI validation set.
Ablation on the context and geometry aware voxel transformer (CGVT). Table presents detailed analysis of various architectural components within CGFormer. The baseline can be consid- ered as a simplified version of VoxFormer [23], without utilizing the extra occupancy network [38] to generate coarse occupancy masks. It begins with an image encoder to extract image features, depth-based query proposal layer define the visible queries, the deformable cross-attention layer to aggregate features for visible areas, the deformable self-attention layer to diffuse features from the visible regions to the invisible ones. Extending the cross-attention to 3D deformable cross-attention (a) brings notable improvement of 1.63 mloU and 2.15 IoU. The performance is further enhanced by giving the voxel queries good initialization by introducing the context ware query generator (b).
Ablation on the local and global encoder (LGE). After obtaining the 3D features, CGFormer incorporates multiple representation refine the 3D volume form both local and global perspectives. Through experimentation, we validate that the boxel-based and TPV-based branches collectively contribute to performance improvement with suitable fusion strategy. Specifically, we compate the results of solely utilizing the local voxel-based branch (c), simply adding the results of dual branches (d), and dynamically fusing the dual-branch outputs (h), as detailed Table The accuracy is notably elevated to an loU of 45.99 and mloU of 16.87 by dynamically fusing the dual-branch outputs. Additionally, we conduct an ablation study on the three TPV planes utilized in the TPV-based branch (c.f.g). The results demonstrate that any individual plane improves performance compared to the model with only the local branch. Combining the information from all three planes into the TPV representation achieves superior performance, underscoring the complementary nature of the three TPV planes effectively representing complex 3D scenes.
Ablation on the context aware query generator. We present the ablation analysis of the context- aware query generator Table We remove the CAQG and increase the number attention layers, where the results of the previous layers can be viewed as initialization of the queries for the later layers. This configuration (6 cross-attention layers and self-attention layers) significantly improves IoU but only marginally lifts mloU, and it requires much more training memory. Employing

---

(a) RGB
(b) VoxFormer [23] (c) OccFormer [59] (d) CGFormer (ours)
(e) Ground Truth
Figure 4: Qualitative visualization results on the CemanticKITTI [1] validation set.
FLoSP [3] with the occupancy-aware depth module [35] can improve performance without much additional computational burden. By replacing with the voxel pooling [36], the model achieves the best performance. Thus, employ voxel pooling
context aware query generator.
Ablation on the depth refinement block. Table presents analyses of the impact of each module within the depth net. By removing the stereo feature Ds and employing the same structure as BEVDepth [21], we observe performance drop of 1.42 loU and 1.79 mloU. When incorporating stereo feature D but fusing it with simple concatenate operation without using the neighborhood attention. the model achieves a mloU of 45.72 and an IoU of 16.26. These results emphasize that deactivating any component of the depth net leads to decrease of the accuracy of the full network. Additionally, we replace the depth refinement module with the dense correlation module from StereoScene [16]. Compared this, our depth refinement module achieves comparable results while using significantly fewer parameters and less training memory.
4.3 Qualitative Results
Fig. 4 presents visualizations of predicted results the SemanticKITTI validation set obtained from MonoScene(3), VoxFormer [23], OccFormer [59], and our proposed CGFormer. Notably, CGFormer outperforms other methods by effectively capturing the semantic scene and inferring previously invisible regions. The predictions generated by CGFormer exhibit distinctly clearer geometrie structures and improved semantic discrimination, particularly for classes like cars and the overall scene layout. This enhancement is attributed to the precision achieved through the context and geometry aware voxel transformer applied in our proposed approach. In contrast, the instances generated by the comparison methods appear to be influenced by depth ambiguity, resulting in vague shapes.
5 Conclusions
In this paper, we present CGFormer, novel neural network for Semantic Scene Completion. CG- Former dynamically generates distinct voxel queries, which serve as good starting point for the attention layers, capturing the unique characteristics of various input images. To improve the accuracy of the estimated depth probability, we propose simple yet efficient depth refinement module, with minimal computational burden. To boost the semantic and geometric representation abilities, CG- Former incorporates multiple representations (voxel and TPV) encode the 3D volumes from both local and global perspectives. We experimentally show that our CGFormer achieves state-of-the-art performance on the SemanticKITTI and CCCBench-KITTI-. benchmarks.

---

Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant No. 2023YFB3209800, in part by Zhejiang Provineial Natural Science Foundation of China under Grant No. LD24F020003, and in part by the National Natural Science Foundation of China under Grant No. 62301484.
References
[1] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jürgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9297-9307,
2019. 7, 8, 9, 14, 15, 16, 17, 18
[2] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern
Recognition, pages 4009-4018 8, 2021. 14, 15
[3] Anh-Quan Cao and Raoul de Charette. Monoscene: Monocular 3d semantic scene completion.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 3981-3991, 2022. 3, 6, 7, 8, 9, 14, 16, 17, 18
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, pages
2020. 3 [5] Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Liu Bingbing. S3enet: A sparse semantic scene completion network for lidar point clouds. In Conference on Robot Learning, pages 1168-2161, 2021. 3,
[6] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 3354-3361, 2012. 14
[7] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6185-6194, 2023. 5,6
[8] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 16000-16009, 2022. 3
[9]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 770-778, 2016. 6, 14
[10] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan. Roberto Cipolla, and Alex Kendall. Fiery: Future instance prediction in bird's-eye-vinn from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 35273-15282 2021.
[11] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853-17862, 2023.
[12] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint orXiv:2112.11771 2021. 3
[13] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9223-9232, 2023. 1, 1, 7, 14,
15, 16
[14] Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Wenyu Liu, and Xinggang Wang. Symphonize 3d semantic scene completion with contextual instance queries. Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 3, 7, 14, 15, 16 [15] Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie Yan, Haiyang Sun, al. Tod3cap: Towards 3d dense captioning in outdoor scenes. arXiv
preprint arXiv:2403. 19589, 2024. 1
[16) Bohan Li, Yasheng Sun, Xin Jin, Wenjun Zeng. Zheng Zhu, Xiaoefeng Wang, Yunpeng Zhang, James Okae, Hang Xiao, and Dalong Du. Stereoscene: Bev-assisted stereo matching empowers 3d semantic scene completion. arXiv preprint arXiv:2303. 13959, 2023. 2, 3, 7, 8, 9, 15, 16
10

---

[17] Hongyang Li, Hao Zhang, Zhaoyang Zeng, Shilong Liu, Feng Li, Tianhe Ren, and Lei Zhang. Dfa3d: 3d deformable attention for 2d-to-3d feature lifting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6684-6693, 2023. 3
[18] Jie Li, Kai Han, Peng Wang, Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2020. 3
[19] Pengfei Li, Ruowen Zhao. Yongliang Shi, Hao Zhao. Jirui Yuan, Guyue Zhou, and Ya-Qin Zhang. Lode: Locally conditioned eikonal implicit scene completion from sparse lidar. In IEEE International Conference on Robotics and Automation, pages 2269-8276. IEEE, 2023. 3 [20] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhanc- ing depth estimation in multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1486-1494, 2023. 3
[21] Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1477-1485, 2023. 3, 4, 9
[22] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang. Zhiheng
Li, Tao Jiang. Fisher Yu, Yue Wang. Hang Zhao, Zhiding Yu, and Chen Feng. Ssebench: A large-scale 3d semantic scene completion benchmark for autonomous driving. arXiv preprint
arXiv:2306. 00000, 2023. 7, 14, 16
[23] Yiming Li, Zhiding Yu, Christopher B. Choy, Chaowei Xiao, José M. Álvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based
3d semantic scene completion. In Proceedings of the IEEE/CVE Conference
Computer
Vision and Puttern Recognition, pages 9087-9098, 2023. 2, 4, 5, 7, 8, 9, 14, 15, 16, 17, 18 [24] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye--ieer representation from multi-camera images via spatiotemporal transformers. In Proceedings of the European Conference on Computer Vision
pages 1-18, 1-18, 2022. 3.7
[25] Tingting Liang. Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework.
Advances in Neural Information Processing Systems, pages 10421-10434, 2022. 3, 6
[26] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine
Intelligence, pages 3292-3310, 2022. 14
[27] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 2117-2125, 2017. 14
[28] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. arXiv preprint arXiv:2203.0500
2022. 3
[29] Yingfei Liu, Junjie Yan. Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.00255 02250, 2022. 3
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021. 14 [31] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird' sird'--eeevvie representation. In IFFE International Conference on Robotics and Automation, pages 2774-2781, 2023. 3
[32] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17110: 101, 2017. 14
[33] Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning
Du, Qingqing Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint
ar.4iv:2304.m 08069, 2023. 3
[34] Jianbiao Mei, Yu Yang, Mengmeng Wang, Junyu Zhu, Xiangrui Zhao, Jongwon Ra, Laijian
Li, and Yong Liu. Camera-based 3d semantic scene completion with sparse guidance network.
arXiv preprint 2=Xxi:2332.0572 2023. 1
[35] Ruihang Miao, Weizhou Liu, Mingrui Chen, Zheng Gong, Weixin Xu, Chen Hu, and Shuchang Zhou. Occdepth: A depth-aware method for 3d semantic scene completion. arXiv preprint arXiv:2302.1 1344..3...1 2023. 9
[36] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, pages 194-210, 2020. 3, 9
11

---

[37]
Thomas Roddick. Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint arXiv: 1811,0818,, 2018. 3
[38]
Luis Roldão, Raoul de Charette, and Anne Verroust-Blondet. Lmsenet: Lightweight multiscale
3d semantic completion. In Proceedings of the International Conference on 3D Vision, pages
111-119, 2020. 1, 3, 4, 7, 8, 16
[39]
Faranak Shamsafar, Samuel Woerz, Rafia Rahim, and Andreas Zell. Mobilestereonet: Towards lightweight deep networks for stereo matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2417-2426, 2022. 2, 4,5, 15
[40] Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas A. Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 190-198, 2017. 1,
3. 7, 8, 16
[41] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pages 6105-61 2019. 14 [42] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang. Yue Wang. Yilun Wang. and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. In Advances in Neural Information Processing Systems, pages 1-186330. 2023. 1 [43] Song Wang, Jiawei Yu, Wentong Li, Wenyu Liu, Xiaolu Liu, Junbo Chen, and Jianke Zhu. Not all voxels are equal: Hardness- werednes--awareeeeee semantic scene completion with self-distillation. arXiv
preprint arXiv:2404. 11558, 2024. 2, 3, 4, 7, 16
[44] Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17850-17859, 2023. 3
[45] Yue Wang. Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao,, and Justin M. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191, 2021. 3
[46] Yu Wang and Chao Tong. H2gformer: Morizzontal-togllooe voxel transformer for 3d semantic scene completion. In Proceedings of the AAAI Conference on Artificial Intelligence, pages
5722-5730, 2024. 4, 7, 16
[47] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce:
Multi-camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 21729-21722
2023. 1, 7, 14, 16
[48] Yuan Wu, Zhiqiang Yan, Zhengxue Wang, Xiang Li, Le Hui, and Jian Yang. Deep height decoupling for precise vision-based 3d occupancy prediction. arXiv preprint ar (xxx:2409.00977
2024.3
[49] Haihong Xiao, Hongbin Xu, Wenxiong Kang, and Yuqiong Li. Instance-aware monocular 3d semantic scene completion. IEEE Transactions on Intelligent Transportation Systems, 2024. 7,
16
[50] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. Mbev: Multi-camera joint 3d detection and segmentation with unified birds-eye-view representation. arXiv preprini arXiv:2204. 00500888 2022. 6
[51] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3101-3109,
2021. 1,3
[52] Zhiqiang Yan, Yuankai Lin, Kun Wang, Yupeng Zheng, Yufei Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Tri-perspective view decomposition for geometry-aware depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4874-4884, 2024. 3
[53] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Rignet: Repetitive image guided network for depth completion. In Proceedings of the European Conference on Computer Vision, pages 244-230. 2022. 3
[54] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones bird's s-yee-view recognition via perspective supervision. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17830-17839, 2023. 1.3
[55] Jiawei Yao and Jusheng Zhang. Depthssc: Depth-spatial alignment and dynamic voxel resolution 2023. 7, 16
for monocular 3d semantic scene completion. arXiv preprint
12

---

[56]
Zhu Yu, Zehua Sheng, Zili Zhou, Lun Luo, Si- Yuan Cao, Hong Gu, Huaqi Zhang. and Hui-Liang
Shen. Aggregating feature point cloud for depth completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8732- 8743, 2023. 3
[57] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang. Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection.
arXiv preprint arXiv:2302. /4325, 2022. 3
[58] Jiahui Zhang. Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, and Hongen Liao. Efficient semantic scene completion network with spatial group convolution. In Proceedings of the European Conference on Computer Vision, pages 733-749, 2018.
(59] Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision- based 3d semantic occupancy prediction. Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pages 9433-944 9943,9443, 2023. 7, 15, 16, 17, 18
[60] Yupeng Zheng, Xiang Li, Pengfei Li, Yuhang Zheng, Bu Jin, Chengliang Zhong, Xiaoxiao Long, Hao Zhao, and Qichao Zhang. Monooce: Digging into monocular semantic occupancy
prediction. arXiv preprint arXiv:2403,088 08766, 2024. 2. 3, 4, 7, 16
[61] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,
2020. 2,3
[62] Sicheng Zuo, Wenzhao Zheng, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Pointoce: Cylin- drical tri- tr-ppesspective view for point-based 3d semantic occupancy prediction. arXiv preprint arXiv:2308. 16896, 2023. 3, 6
13

---

A Appendix Supplemental Material
In the appendix, we mainly provide implementation details and more experiment results.
A.1 Datasets and Metrics
Datasets. We evaluate our CGFormer on two datasets: SemanticKITTI [1] and BSC--ench-KIYH 360 [22]. These datasets are derived from the KITTI Odometry [6] and KITTI-360 [26] Benchmarks, respectively. The evaluation focuses on specific spatial volume: 51. 11.2mm in front of the car, 25.6m to the left and right sides, and 6.4m above the car. Voxelization of this volume results in set of 3D voxel grids with a resolution of 256 x 256 x 32, where each voxel measures 0.2m x 0.2m x 0.2m. SemanticKITTI provides RGB images with dimensions of 1 1226 x 370 as inputs, encompassing 20 unique semantic classes (19 semantic classes and free class). The dataset includes 10 sequences for training, I sequence for validation, and 11 sequences for testing. SSC-Bench-EF BCCCBBnch-KKTTI--66 [22] offers 7 sequences for training, sequence for validation, and sequence for testing. contains 19 unique semantic classes (18 semantic classes and I free class), with input RGB images having resolution of 1408 x 376.
Metrics. Following previous methods [3. 23, 13], we report the intersection over union (IoU) and mean IoU (mloU) metries for occupied voxel grids and voxel-wise semantic predictions, respectively, The interplay between loU and mIoU offers a comprehensive perspective the model's effectiveness in capturing both geometry and semantic aspects of the scene.
A.2 Implementation Details
Network Structures. Consistent with previous researches [13, 3,,,,777 47], we utilize 2D UNet based on pretrained EfficientNetBB7 [41] as the image backbone. The CGVT generates 3D feature volume with dimensions of 128 x 128 x 16 and 128 channels. The numbers of deformable attention layers 3 and 2 respectively. We use 8 sampling points around each
for cross-attention and self-attention reference point for the cross and self-ateention head. The voxel-based branch of the LGE comprises 3 stages with 2 residual blocks [9] each. SwinT [30] is employed as the 2D backbone in the TPV-based branch. Both are followed by feature pyramid networks (FPNs) [27] to aggregate multi-scale features for dynamic fusion. The final prediction has dimensions of 128 x 128 x 16 and is upsampled to 256 x 256 x 32 through trilinear interpolation to align the resolution with the ground truth.
Training Setup. We train CGFormer for 25 epochs 4 NVIDIA 4090 GPUs, with batch size of 4. It approximately consumes 19 GB of GPU memory on each GPU during the training phase. We employ the AdamW [32] optimizer with
0.9, 83 0.99 and set the maximum learning rate 3 x 10-1 The cosine annealing learning rate strategy is adopted for the learning rate decay, where the cosine warmup strategy applied for the first 5% iterations.
A.3 Results Using Monocular Inputs
In alignment with previous methods [23, 14], we evaluate the performance our CGFormer using only monocular RGB image input. We replace the depth estimation network with AdaBins [2] and present the results on the semantickitti validation set in the table 7. To better demonstrate the advantage of CGFormer, OccFormer. Compared to the stereo-based methods when using only a monocular image (VoxFormer, Symphonize), CGFormer achieves superior performance in terms of both IoU and mloU, Furthermore, our method also surpasses OccFormer, the state-of of-the-ar monocular method.
also include the results of VoxFormer, Symphonize, and
Table 6: The performance of the CGFormer with more lightweight backbone networks.
Backbone Networks
loU
mioU
Parameters
Training Memory
EfficientNeBB7. Swin Block
45.99
16.87
122.42
19330
ResNet50, Swin Block
45.99
16.79
80.46
19558
ResNet50, ResBlock
45.86
16.85
54.8
18726
14

---

Table 7: Comparison of the performance using monocular inputs. For stereo-based methods, we
replace the MobileStereoNet [39] with Adabins [2].
Method
laU
mloU
VoxFormer-S [23]
38.68
10.67
VoxFormer-T [23]
38.08
11.27
Symphonize [14]
38.37
12.20
OceFormer [59]
36.50
13.46
CGFomer (ours)
41.82
14.06
Table 8: Comparison of training memory and inference time with SOTA methods on the and SemanticKITTI test set. These metrics were measured on the NVIDIA 4090 GPU.
Method
TPVTormer [13]
Oceluormer [39]
VoxFormer (23)
Symphonize [14]
StereoScene [16]
CGlvormer (ours)
Training Meomry (M)
18564
18080
18725
17757
19000
19330
Inference Time (ms)
207
199
204
216
258
205
loU
34.25
34,53
42.95
42.19
43,34
44.41
mlnU
12:20
16.63
A.4 Reults with More Lightweight Backbone Networks
We reanalyze the components of CGFormer, finding that replacing EfficientNe(B7, used the image backbone, and the Swin blocks, used in the TPV branch backbone, with more lightweight ResNet50 and residual blocks, respectively, can significantly reduce the number of parameters of network. Besides, we also remove the predefined parameters as we find it doesn't 't influence the final performance. The results on the semantickitii validation set are presented in the Table 6. Compared to the original architecture, CGFormer maintains stable performance regardless of the backbone networks used for the image encoder and TPV branch encoder, underscoring its effectiveness, robustness, and potential.
A.5 Additional Quantitative Results
For more comprehensive comparison, we list the results with input modality and image backbones in Table 9 and Table 10. Table 11 presents the comparison results of CGFormer with the state-of-the -fth--oftthe-art methods on the SemanticKITTI validation set. CGFormer outperforms all other methods in terms of both IoU and mloU, Additionally, it ranks either first or second on most of the classes, demonstrating consistent performance across various semantic categories, as indicated previous tables.
A.6
Computational Cost
In Table 8, we display the training memory and inference time of CGFormer, along with those of the comparison methods. Additionally, the table includes the corresponding IoU and mloU metrics for comprehensive comparison. As shown in the table, CGFormer achieves the best performance in terms of both IoU and mIoU, with comparable training memory and inference time.
A.7 Additional Qualitative Results
We offer additional visualization results in Fig.6 and Fig.7. These examples are randomly selected from the SemanticKITTI [1] validation set.
A.8
Failure Cases
We provide two failure cases
Fig.
A.9
Limitations
While CGFormer exhibits strong performance on benchmarks. but the accuracy on most of the categories (e.g., person, bicyelist, other vehicle) unsatisfactory. Improving the performance on these instances could be beneficial for the downstream application tasks. Furthermore, there is a need to explore designing depth estimation networks under multi-view scenarios extend the geometry-ane view transformation to these scenes. Despite these limitations, we are confident that CGFormer will contribute to advancing the field of 3D perception.
15

---

Table 9: Quatitative results on SemanticKITTI [1] test set. represents the reproduced results in [13, 59]. The best and the second best results in bold and underlined, respectively. Our CGFormer outperforms temporal stereo-based (Stereo-T) methods or those methods with larger image backbones in terms of IoU and mloU.
liper
Minskooe" 11
Mex
11.08
56.00
21 10
24.93
5.1
18.01
15:01
1.50
0.33
11:0
14.93
2.40
18.00
1.00
1-00
BA0
11.00
5:01
2.00
SumaaDo: (97)
Meas
36.12
11.06
56.09
18.30
33.30
8.83
15.3
1.40
1.43
1.30
4.40
14.05
1.40
18.30
1.80
2.0
010
11.33
3.8
240
JAMSSU
Meno
Recke00
12.31
38.0
23:30
19.23
1.00
3.50
11.45
620
4.10
Wathom--T [14]
$anae]
45:31
13:41
36.00
28.00
25.0
190
54.45
8.35
34.50
610
13.09
6.00
3:20
Depetse: [-1]
06.58
1651
13:1
6.3
MaaRINO
15.00
1.50
1000
3.3
Sreee
Reskurt
5190
5-6
Sesciaae (10]
Saxe
48.34
13.36
01.0
3130
31.00
1670
54.3
330
140
240
6.00
3333
840
2100
230
230
650
1633
730
720
#icRivees [16]
Trews
Reckedo
33.53
14:90
17:9
18400
11.0
35.38
10.30
10
610
8.50
2.90
6.6
Meechhec po)
Since
Inkrshrage-00
15.5
26.0
1846
13B
468
170
11.00
6.90
1-30
5.48
cofferer (oan)
Stane
44.41
1683
94.30
34.0
3438
4.30
335
270
3635
11.30
28.8
1639
K70
930
Table 10: Quantitative results JSCBench-KITTTES test set. The results for counterparts provided in [22]. The best and the second best results for all camera-based methods are in bold and underlined, respectively. The best results from the LiDAR-basee methods are in red.
STARE
App
45
Meteal
Iaage Baclhood
19535
11:0
113
311
10:00
1110
1177
13.10
RTT
Macadcene
Mins
17.90
13:31
0:33
0.03
3.05
0185
15
14:15
16.25
OxcReeee 19]
Mons
40.77
13.81
32.58
0.66
0.36
9.89
3.83
2.31
$5.30
13.44
11.5%
1.55
96.03
4.8
10
19:51
2.31
8.51
6.95
4.99
JAMBSCC
Mose
RadNart
41:91
12.57
245
8.2
1.8
39.36
51.53
33.17
13:24
DepesS0C 13]
Stzoo
40:53
13.18
21.90
2.36
1.00
11.11
435
2.9
12.50
53.35
5.2
21.61
11.30
597
3.11
114p
CCRana (st))
Stexo
48307
34.65
16.35
341
3.66
13:00
4.38
63.5
17.16
46.75
5.55
43.75
38.56
14.96
16:34
17.45
10.0
Table 11: Quatitative results on SemanticKITTI [1] validation set. represents the reproduced results
in [13, 59, 55]. The best and the second best results are in bold and underlined. respectively.
Moaa
14.27
DUG
11.65
061
0.45
25.7
23.01
3.34
59.36
2.51
1.89
5.00
1.50
Mows
36.90
13.65
76.55
1961
0.31
1140
3.99
2.45
500
1AMB5C (9]
36.36
1.33
1.56
Nathuss: (200
Sizze
Sccha3
4410
12.39
34.19
23.33
13.90
107
13.0
23.79
2,00
0331
1.7
34.99
3.88
20106
1.18
3.82
500
1.4
7.11
4.5
Depa50c (23)
Sasao
03.6
13.38
13.35
21.4
15:76
100
11:2
23.06
135
1.35
1.55
36.57
4.5
30119
225
432
500
7,4
40
BASSCS [43]
44.0
13.48
57.05
38355
1530
106
15.00
27.25
990
100
0.5
25.5
52
285
471
530
5.5
720
410
Reche53
000
19
19.00
100
0.07
1.39
2625
134
3.6
510
131
Spmma1
3331
3333
005
031
120
238
0.0
010
221
RGB
(b) Ground Truth
(c) CGFormer (ours)
Figure 5: Failure cases.
16

---

Coca
KEKKE
(a) MonoScene [3]
(b) VoxFormer [23]
(c) OccFormer [59]
(d) CGFormer (ours)
(e) Ground Truth
Figure 6: More qualitative comparison results on the SemanticKITTI [1] validation set.
17

---

(a) MonoScene [3]
(b) VoxFormer [23] (c) OccFormer [59] (d) CGFormer (ours)
(e) Ground Truth
Figure 7: More qualitative comparison results on the SemanticKITTI [1] validation set.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We describe the motivation of work at the beginning of the abstract. Then we describe the methodology we employed to tackle the presented issues, Our contributions are summarized the last of introduction. Refer to the introduction validate it.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a subsection of limitation in the supplement material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
We provide formulas for better understanding our job, but we don't propose new theorems.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed implementation the supplement material. Code is also provided to validate it.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code as an additional zip file. However, as the checkpoint of model is too large to upload, we provide anonymous link for downloading the pretrained checkpoints and scripts for retraining the model. Code will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//niss.cc///// httpsscccppblccccccccceeeccccscc htw//guidssss Cooeeubmiiss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details of datasets and metries, which includes the data splits and evaluation metrics. Hyper- parameters and optimizer can be found in the implementation details in the supplement material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following previous methods, we obtain the results by submitting results to the leaderboard. And all of our experiments conducted using the same seed 7240.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our model trained on NVIDIA 4090 GPUs, with a batch size of approximately consumes 19GB of GPU memory each GPU during the training phase. Refer to the implementation details for detailed information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /nerris.....p// cc/ppbliceieccc c/Ethi csGuidelinee ines?
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms with the NeurIPS Code of Ethics. The provided code and link are both anonymous.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We mentioned that we are confident that CGFormer will contribute to advanc- ing the field of 3D perception and autonomous driving at the end of the conclusion, while we also pointed that the accuracy should be further improved for actual application.
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No, our paper did not utilize any high-risk data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We will thank all the assets we refer to when the code is publicly available. We respect these methods and cite them in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "fpIqntdeCX", "reviewer": "Reviewer_jg8M", "summary": "This paper proposes a state-of-the-art Semantic Scene Completion method called CGFormer.  It introduces a Context and Geometry Aware Voxel Transformer that dynamically generates queries tailored to individual input images, addressing depth ambiguity through a 3D deformable cross-attention mechanism. The network leverages a multi-scale 3D representation by combining voxel and tri-perspective view to enhance both local semantic and global geometric information. State-of-the-art results are achieved in key benchmark tests.", "questions": "Refer to the weaknesses.", "limitations": "The limitations of this paper are primarily in the accuracy of certain categories (e.g., pedestrians and bicyclists), which suggests that there is room for improvement in these areas, as outlined by the authors in Section 5.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on limitations of existing methods that utilizes a combination of voxel and tri-perspective view representations to capture both local details and global structures.\n2. The idea that different input images have unique contextual features is very interesting.\n3. The experiments are adequate with complete qualitative analysis, ablation experiments and visualization results.\n4. In the SemanticKITTI test, the proposed method improves most of the metrics.", "weaknesses": "1. In the SSCBench-KITTI360 test set, CGFormer does not improve much compared to Symphonie. Symphonie has almost half of the metrics higher than CGFormer, and Symphonie seems has fewer parameters.\n2. Context and Geometry Aware Voxel Transformer seems to be a redesign of Symphonie and adds Deformable Self-Attention after Deformable Cross-Attention, which should be added with more explanation as to why it's done.\n3. Context net is not described in the main text or in the supplementary material.\n4. The table font is too small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Parameter and performance comparison with Symphonize.\n\n| Model | IoU&uarr; | mIoU&uarr; | Parameters (M) &darr; | Training Memory (M) &darr; |\n| --- | --- | --- | --- | --- |\n| EfficientNetB7, Swin Block | **45.99** | **16.87** | 122.42 | 19330 |\n| ResNet50, Swin Block | **45.99** | 16.79 | 80.46 | 19558 |\n| ResNet50, ResBlock | 45.86 | 16.85 | **54.8** | 18726 |\n| Symphonize | 41.92 | 14.89 | 59.31 | **17757** |\n\nThanks for your valuable suggestion. We revisit the architecture of CGFormer and discover some interesting new results. To compare its performance with Symphonize with a comparable number of parameters, we analyze the components of CGFormer, finding that replacing EfficientNetB7, used as the image backbone, and the Swin blocks, used in the TPV branch backbone, with more lightweight ResNet50 and residual blocks, respectively, can significantly reduce the number of parameters of our network. The results on the semantickitti validation set are presented in the table above. Compared to the original architecture, CGFormer maintains stable performance regardless of the backbone networks used for the image encoder and TPV branch encoder, underscoring its effectiveness, robustness, and potential.\n\nCompared to Symphonize, lightweight CGFormer achieves an IoU of 45.86 and mIoU of 16.85, significantly surpassing Symphonize's IoU of 41.92 and mIoU of 14.89 on the SemanticKITTI validation set with a comparable number of parameters.\n\nWe retrain this lightweight model on the KITTI-360 dataset, with the detailed results of each class in Table R1 of the uploaded PDF. The lightweight version of CGFormer achieves an IoU of 47.78 and mIoU of 20.03, demonstrating a substantial improvement of 1.45 mIoU and 3.5 IoU over Symphonize. For specific categories, the original CGFormer architecture outperforms Symphonize in 9 out of 18 classes, and with the lighter backbones, it surpasses Symphonize in 10 out of 18 classes. These results further highlight the superiority of our approach.\n\n> W2. More explanation of context and geometry aware voxel transformer.\n\nWe apologize for any misunderstanding. Context and geometry aware voxel transformer is not the redesign of Symphonize. We will provide a more detailed explanation here. Existing coarse-to-fine (sparse-to-dense) methods, such as VoxFormer and MonoOcc, generally follow a pipeline that first aggregates 3D information for visible voxels using depth-based queries. These queries are defined as a set of learnable parameters, which are the same for all the input images. Subsequently, these methods complete the 3D information for non-visible regions using the reconstructed visible areas as starting points. The aggregation of information for visible voxels is accomplished through deformable cross-attention, while the completion of information in non-visible regions is handled by deformable self-attention, similar to the MAE [1].\n\nFor the proposed context and geometry-aware voxel transformer, we take into account the context of different images and introduce context-dependent queries. Instead of solely predefining a set of learnable parameters that primarily capture the overall distribution of the dataset, the context-dependent queries are related to the image content, allowing them to aggregate information from points within contextually relevant regions. Additionally, we extend deformable cross-attention from the 2D to 3D pixel space, enabling the differentiation of points with similar image coordinates based on their depth coordinates.\n\nIn contrast, Symphonize employs context-independent voxel queries for all the input images. It first condenses the image into a set of tokens with higher-level instance semantics, then the voxel queries serve as the query of the deformable cross attention while the set of tokens serve as the key and value. The original manuscript of Symphonize referred to this set of tokens as \"queries\", which may have led to some misunderstanding.\n\n> W3. Details of context net.\n\nSorry for the oversight of this module. The image encoder consists of a backbone network for extracting multi-scale features and a feature pyramid network to fuse them. We employ SECONDFPN [2] network here. It fuses the multiscale image features and output a feature map $\\mathbf{F}^{2D}\\in\\mathbb{R}^{H\\times W\\times C}$, with $C$ set to $512$. The context net consists of several convolution blocks that take $\\mathbf{F}^{2D}$ as input and reduce the number of channels. The generated context feature $\\mathbf{C}$ has a shape of $H\\times W\\times 128$. We will include these details in the revised manuscript.\n\n> W4. The table font is too small to read.\n\nThanks for your suggestion. We will fix it in the revised manuscript. Examples of tables with larger font are provided in the uploaded PDF.\n\n[1] Masked autoencoders are scalable vision learners, CVPR 2022\n\n[2] SECOND: Sparsely Embedded Convolutional Detection, Sensors 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5gAzFEu0GA", "reviewer": "Reviewer_wDiN", "summary": "This manuscript studies the problem of road scene semantic scene completion from RGB images. The architecture and benchmarking frameworks follow a widely accepted literature. The innovation proposed here is better queries that are informed of the geometry and semantics of the input scene, a cross-attention variant that leverages rich information from different depth planes in the cost volume, and better information fusion from different represetnations. The whole architecture achieves state-of-the-art performance on public benchmarks.", "questions": "My questions are incorporated into the weakness section.", "limitations": "The manuscript is limited in scholarship and can be improved by incorporating semantic scene completion references like [A] and [B].\n\n[A] Efficient semantic scene completion network with spatial group convolution, ECCV 2018\n\n[B] Lode: Locally conditioned eikonal implicit scene completion from sparse lidar, ICRA 2023", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1+) The paper aims to improve the query quality and solve the deep ambiguity and context-independence problems, which are important and motivated for the SSC task.\n\n(2+) The paper proposes a novel method, the CGFormer, which introduce a context aware query generator to capture context -dependent queries and a novel Depth Net utilizing stereo depth and monocular depth for effective refinement.\n\n(3+) Experiments on SemanticKITTI (table 1.) and SSCBench-KITTI360 (table 2.) show CGFormer outperforms prior methods. The motivation and the proposed components are  validated by detailed ablation studies. (Table 3.,4.,5.)\n\n(4+) The paper is basically clearly written and model architecture is easy to follow.", "weaknesses": "(1-) The paper notes that context-dependent query tends to aggregate information from the points within the region of interest, Ablation on the the number of the cross-attention and self-attention layers should be provided.\n\n(2-) In alignment with previous methods, it would be better to demonstrate the performance when using only a monocular image as input (See Table 3. in VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion).\n\n(3-) The visualization (figure 4. and figure 5.) shows some common scenes (It would be even better if the corresponding image views could be added). Could you show more performance improvements for objects like the bicycle in Figure 3 (due to capturing the regions of interest)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Ablation on the number of the cross-attention and self-attention layers.\n\nWe present the results of different configurations of cross-attention and self-attention layers on the semantickitti validation set in the table below. As shown in this table, the performance improves gradually with the increase of the number of the attention layers. However, once the number exceeds a certain threshold, the performance tends to stabilize. In alignment with previous methods, we set the number of cross-attention layers to $3$ and the number of self-attention layers to $2$ in the manuscript. This table also includes the results of the model without the CAQG module. Notably, using just one self-attention and one cross-attention layer with the CAQG module yields better performance than using two self-attention and three cross-attention layers without the CAQG module. This demonstrates the effectiveness of our proposed context and geometry-aware voxel transformer.\n\n| Model | IoU&uarr; | mIoU&uarr; | Training Memory (M)&darr; |\n| --- | --- | --- | --- |\n| 1 self, 1 cross | 44.95 | 16.24 | **16602** |\n| 1 self, 2 cross | 45.12 | 16.26 | 16836 |\n| 1 self, 3 cross | 45.22 | 16.40 | 16920 |\n| 2 self, 3 cross | **45.99** | **16.87** | 19930 |\n| 2 self, 4 cross | 45.86 | 16.74 | 19601 |\n| 3 self, 4 cross | 45.76 | 16.77 | 22364 |\n| w/o CAQG      | 44.88 | 15.84 | 18959 |\n\n> W2. Performance using only monocular image.\n\nFollowing VoxFormer and Symphonize, we replace the depth estimation network with AdaBins and present the results on the semantickitti validation set in the table below. To better evaluate the performance of our CGFormer, we also include the results of VoxFormer, Symphonize, and OccFormer. Compared to the stereo-based methods when using only a monocular image (VoxFormer, Symphonize), CGFormer achieves superior performance in terms of both IoU and mIoU. Furthermore, our method also surpasses OccFormer, the state-of-the-art monocular method.\n\n| Model | IoU&uarr; | mIoU&uarr; |\n| ---       | ---   | ---      |\n| CGFormer (AdaBins) | **41.82** | **14.06** |\n| VoxFormer-S (AdaBins) | 38.68 | 10.67 |\n| VoxFormer-T (AdaBins) | 38.08 | 11.27 |\n| Symphonize (AdaBins)  | 38.37 | 12.20 |\n| OccFormer                    | 36.50 | 13.46 |\n\n> W3. More visualizations.\n\nFigure R1 in the uploaded PDF displays two visualization examples for objects with finer details. As shown in the RGB image, the sampling points of the context-dependent queries are typically situated within the region of interest. This allows CGFormer to capture much more clear details compared to other methods, highlighting the effectiveness of our proposed module. We will also provide the image view in the revised manuscript as done in this figure.\n\n> W4. The manuscript is limited in scholarship and can be improved by incorporating semantic scene completion references.\n\nThanks for your suggestion. We will incorporate these semantic scene completion references in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzc6XROcJi", "reviewer": "Reviewer_ryRH", "summary": "The authors present Context and Geometry Aware Voxel Transformer (CGFormer) for the semantic scene completion task. Their method extends the baseline VoxFormer with a Context-Aware Query Generator (CAQG), 3D deformable attention layers, a depth refinement block, and a dynamic fusion of voxel and TPV features. Experiments on two datasets demonstrated that CGFormer can achieve state-of-the-art mIoU performance.", "questions": "1. details of the CAQG should be put in Section 3 since it is a key component of the proposed method.\n2. Table 3 shows that directly adding TPV slightly increases mIoU while hurting the IoU. This is interesting and can the authors provide their thoughts and analysis?\n2. In Table 3, LB-only, DF, and adding each of the TPV planes are ablated. Is it possible to also ablate the TPV branch only? Conceptually this is the only missing setting.", "limitations": "The authors have discussed the limitations in their appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. solid experiments. Two datasets are experimented with and results tables are presented clearly along with many baseline methods.\n2. detailed ablation study for every designed module.\n3. The figures are well-organized and easy to follow, which is greatly appreciated.", "weaknesses": "1. The biggest concern to the reviewer is that the empirical results are not convincing enough to support the main claims of the proposed method. Specifically,   \n\n    a) The authors claim that context-aware queries are a major novelty that helps the model perform better in its region of interest. However, the empirical results show that the performance is superior mainly in categories with larger areas such as the roads and sidewalks, while not as good in categories with finer details such as trucks, persons, and bicycles. The fact that CGFormer is comparable or worse compared to the baselines with context-independent queries on 50% of the categories makes the claims less compelling.\n\n    b) Besides a), the qualitative difference in Figure 3 is not illustrative. According to the first two columns, the context-dependent query should sample more at locations with details such as cars and bicycles, therefore, in the 3rd column, shouldn’t the context-aware queries be focused more on the white car? It is not clear why the points in (b) are irrelevant. \n\n   c) The model is named CGFormer, while how context-aware plays a part has been discussed in the paper, how the performance reflects geometry-aware is barely touched on.\n\n2. Some minor issues with the writing. The proposed CAQG seems to be a key component in the model. However, its details are glossed over in Section 3.2 lines 137--146. Please consider elaborating on this part to emphasize the novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1 (a). Performance gain of the context-aware queries.\n\nCompared to methods that do not use temporal inputs or rely on much larger image backbone networks, CGFormer achieves the highest performance in 12 out of 19 categories on both the test and validation sets, as shown in Tables R2 and R3 of the uploaded PDF, encompassing both large-area and fine-detail objects, while the other methods excel in at most 3 to 4 categories.  Although CGFormer shows limitations in several categories such as person and truck, it excels in most of the categories, including car, traffic sign, trunk, and pole. Indeed, CGFormer attains the top performance on the bicycle under the same condition.\n\nTo further validate the effectiveness of the proposed context-aware queries, we remove the CAQG module while keeping all other components unchanged. We then compare the performance of the two models across various categories. The results are presented in Table R3 of the uploaded PDF, with categories where performance improves after integrating the CAQG module highlighted in red. The CAQG module enhances performance in most categories (11 out of 19), including both large-area objects (sidewalk, terrain) and small objects (trucks, bicycles, person), while having minor effects on others. The above results demonstrate the efficacy of the proposed module.\n\n> W1 (b). More explanation of Figure 3 in the manuscript.\n\nThanks for your suggestion to help illustate our motivation. We will replace Figure 3 with more clear examples in the revised manuscript, as shown in the Figure R1 in the uploaded PDF. Here's a more detailed explanation for this figure. The yellow point represents the projected location of the visible voxel queries of the deformable cross-attention from 3D space onto the 2D image plane using the camera's intrinsic and extrinsic matrices. This projection aggregates information from the image features.  For a voxel projected onto the image plane, the query should tend to aggregate information relevant to the semantics of the position where it lands. As shown in the left and middle columns, the yellow reference points fall onto the car and bicycle, and the sampling points of the context-aware query are mainly distributed within the regions of these two objects. In the right column, the query point projects onto the building, and the context-aware query points are primarily located on the building. In contrast, the context-independent query points are scattered in the region of the car.\n\n> W1 (c). More explanation for \"geometry-aware\".\n\nThe modules derived from depth information are referred to as \"geometry-aware\" in the title. We provide a more detailed discussion as follows.\n\n(1) As mentioned in the manuscript (line 49-51), visible queries are projected onto the image plane to aggregate information by deformable cross-attention. However, when projecting the 3D points onto the image plane, many points may end with the same 2D position with similar sampling points on the 2D feature map, causing a crucial depth ambiguity problem. To address this issue, we extend deformable cross-attention from 2D to 3D pixel space, which allows to differentiate points with similar image coordinates by their depth coordinates, as illustrated in Figure 1 of the manuscript. \n\n(2) Additionally, we introduce depth refinement to improve the accuracy of the estimated depth probability. \n\nModel (a) in Table 3 provides ablation results for the 3D deformable cross-attention, and Table 5 includes ablation results for depth refinement. \n\n> W2 & Q1. More explanation for the CAQG module.\n\nThanks for pointing out this point, here is a revised version (line141-line144) of the manuscript.\n\nTo elaborate, the context feature $\\mathbf{C}\\in\\mathbb{R}^{H\\times W\\times C}$ and depth probability $\\mathbf{D}\\in\\mathbb{R}^{H\\times W\\times D}$ are first derived from the 2D image feature $\\mathbf{F}^{2D}$. Taking $\\mathbf{C}$ and $\\mathbf{D}$ as inputs, the query generator $f$ maps them from 2D image space to 3D space to generate context-aware voxel queries $\\mathbf{V_{Q}}\\in\\mathbb{R}^{X\\times Y\\times Z}$, where $(X,Y,Z)$ denotes the spatial resolution of the 3D volume. The query generator can be any explicit view transformation approach (e.g., voxel pooling, FLoSP, CaDDN). Table 4 provides ablation experiments for different methods.\n\n> Q2. More analyses for the TPV branch.\n\nThanks for your constructive question. We find additional insights regarding the TPV branch. We reanalyze the ablation experiments (models d, e, f, g in Table 4 of the manuscript) and discover that the performances were quite similar. To determine whether these performance gains are statistically significant across multiple training seeds, we conduct further experiments. As shown in Table R5 of the uploaded PDF, combining the outputs from both branches does enhance performance, but with some variability. We speculate that the TPV branch primarily focuses on global information, making it challenging to capture fine-grained voxel details. In contrast, the local branch enhances the fine-grained structural details of the 3D voxels. Simple addition treats all features as equally important, which may introduce side effects. Instead, weighting the more important features from each branch can enhance overall performance by leveraging their distinct strengths.\n\n> Q3. Ablation experiments for TPV branch.\n\nThanks for your suggestion to complete our experiments. Taking the model (b) of the table 3 in the manuscript as baseline, we present the ablation results for the TPV branch. As shown in the Table R4 in the uploaded PDF, enhancing the features with TPV branch boost the performance in terms of IoU, while minor improvement in terms of the mIoU, following the assumption that TPV focuses more on the global information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0mgClmIoUU", "reviewer": "Reviewer_Lm1y", "summary": "This paper proposes CGFormer for semantic scene completion. It generates distinct voxel queries for different input images, instead of simply predefining a set of trainable parameters. The deformable cross attention is extend to 3D pixel space, avoiding sampling the same features for different projecting points. This method further enhances the 3D volume from both local and global perspectives. This paper incorporates stereo depth to improve the accuracy of depth probability by a depth refinement strategy.", "questions": "Please see the weaknesses above.", "limitations": "In this paper, the limitations have been well discussed. The accuracy on most of the categories is unsatisfactory. Furthermore, there is a need to explore designing depth estimation networks under multi-view scenarios to extend the geometry-aware view transformation to these scenes. The method is worth further exploration.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The code is submitted in the supplement material, whose guideline is detailed and easy to follow.\n\n2. The proposed blocks bring notable performance gain, with elaborate experiments.\n\n3. The CGFormer surpasses previous methods on both the SemanticKITTI and SSC-Bench-KITTI360 benchmarks.\n\n4. The paper is well-organized the motivation is clear.", "weaknesses": "1. EfficientNetB7 contains much more parameters than the ResNet50 employed in the previous coarse-to-fine methods (e.g., VoxFormer, Symphonize). Comparing the performance under the same setting will be fairer.\n\n2. Text error in Figure 1.\n\n3. Although the paper presents both superior qualitative and visualization results, it would be better to provide several failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Parameters and performance comparison between CGFormer and Symphonize.\n\nTo compare the performance of CGFormer with Symphonize with a comparable number of parameters, we revisited the design of our CGFormer and replaced the EfficientNetB7 with ResNet50 and the Swin blocks in the TPV branch with more lightweight residual blocks. The results on the semantickitti validation set are summarized in the table below. As shown in this table, our CGFormer is robust across different backbone networks and still outperforms Symphonize with a comparable number of parameters. This highlights CGFormer's potential and robustness.\n\n| Model | IoU&uarr; | mIoU&uarr; | Parameters (M)&darr; | Training Memory (M)&darr; |\n| --- | --- | --- | --- | --- |\n| CGFormer (EfficientNetB7, Swin Block) | **45.99** | **16.87** | 122.42 | 19330 |\n| CGFormer (ResNet50, Swin Block) | **45.99** | 16.79 | 80.46 | 19558 |\n| CGFormer (ResNet50, ResBlock) | 45.86 | 16.85 | **54.8** | 18726 |\n| Symphonize | 41.92 | 14.89 | 59.31 | **17757** |\n\n> Q2. Text error in Figure 1.\n\nThanks for pointing out this mistake. We will correct it in the revised manuscript.\n\n> Q3. Failure cases.\n\nFigure 2 in the uploaded PDF shows two examples of failure cases, where it is difficult to distinguish between adjacent cars. As seen in the RGB images, these objects are located in distant regions, making them challenging to differentiate. In the future, we plan to explore ways to improve the performance of our method in these far regions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable comments of reviewers, which have greatly contributed to enhancing the quality of our paper. We are glad that the reviewers recognized various strengths of our work, including the clear motivation [wDiN] [Lm1y], interesting idea [jg8M], comprehensive experiments [ryRH, jg8M, Lm1y], good performance [jg8M, wDiN, Lm1y], clear writing [wDiN] [Lm1y], and easy-to-follow figures [ryRH, wDiN]. Additionally, Reviewer [Lm1y] highlighted that the provided guideline for the submitted code is detailed and easy to follow. We will answer the specific questions from each reviewer. We upload a PDF file with figures and tables which we will present in our rebuttal.

Meta-review: The reviewers unanimously agree to accept the paper despite different degrees of excitement. The AC also agrees the merits warrant accepting the paper but also encourages the authors to revise the paper according to the review comments for the camera-ready revision.
Decision: Accept (spotlight)