SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving
Authors: Haiming Zhang
Yiyao Zhu
Wending Zhou
Xu Yan
Yingjie CAI
Bingbing Liu
Shuguang Cui
Zhen Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Autonomous Driving
Pre-training
3D Gaussian Splatting
Query-based Perception
Primary area: deep_learning
TL;DR: SQS introduces a novel self-supervised pre-training framework that enhances Sparse Perception Models by leveraging query-based 3D Gaussian splatting.

Abstract: Sparse Perception Models (SPMs) adopt a query-driven paradigm that forgoes explicit dense BEV or volumetric construction, enabling highly efficient computation and accelerated inference. In this paper, we introduce SQS, a novel query-based splatting pre-training specifically designed to advance SPMs in autonomous driving. SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. During fine-tuning, the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction mechanisms that explicitly connect pre-trained queries with task-specific queries, effectively accommodating the diverse requirements of occupancy prediction and 3D object detection. Extensive experiments on autonomous driving benchmarks demonstrate that SQS delivers considerable performance gains across multiple query-based 3D perception tasks, notably in occupancy prediction and 3D object detection, outperforming prior state-of-the-art pre-training approaches by a significant margin (i.e., +1.3 mIoU on occupancy prediction and +1.0 NDS on 3D detection).

SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving
Haiming Zhang 22 Yiyao Zhu Wending Zhou 1.2 Xu Yan Yingjie Cai Bingbing Liu", Shuguang Cui².1, Zhen L
FNii, Shenzhen SSE, CUHK-Shenzhen HKUST Huawei Noah's Ark Lab (haimingzhang@linkk. ,arpiiinnk,iiaiiion lizhen@) cuhk. edu, cn yzhucp&connect, ust.hk yanxu440huawei.co com
Abstract
Sparse Perception Models (SPMs) adopt query-driven paradigm that forgoes ex- plicit dense BEV or volumetric construction, enabling highly efficient computation and accelerated inference. In this paper, we introduce SQS, novel query-based splatting pre-training specifically designed to advance SPMs in autonomous driv- ing. SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. During fine-tuning, the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction mechanisms that explicitly connect pre- pre-trained queries with task-specific queries, effectively accommodating the diverse requirements of occupancy prediction and 3D object detection. Extensive experiments on autonomous driving benchmarks demonstrate that SQS delivers considerable performance gains across multiple query-basee -bsed 3D perception tasks, notably in occupancy prediction and 3D object detection, outperforming prior state-of-the-art pre-training approaches by significant margin (i.e., +13 mloU on occupancy prediction and +1.0 NDS on 3D detection).
Introduction
Recent advances vision-centric autonomous driving have driven significant progress in the field 7 60). From representation standpoint, existing approaches can be broadly categorized into dense BEV-
centric and sparse query-centric paradigms. Dense BEV-centric methods (2 [3 extract Bird's Eye View (BEV) features from multi-view images for downstream tasks, while Sparse Perception Models (SPMs) [55 33 31 bypass explicit dense representations and directly aggregate features from images using implicit queries, enabling faster inference. Sparse query-centrie methods have garnered increasing attention within the community due to their practical advantages for real-world deployment. Despite the dominance of supervised methods, their reliance precise ground-trute annotations presents a substantial challenge, as acquiring such labels is both costly and labor- intensive. Conversely, the abundance of unlabeled data offers promising avenue further enhance model performance. Nevertheless, effectively leveraging this data remains a non-trivial challenge.
To mitigate these challenges, various pre-training strategies have been proposed for autonomous driving. Earlier works leverage contrastive learning [45] and Masked Autoencoders (MAE) (38 for pre-training. However, the coarse supervision they provide limits their capacity to fully capture spatial- temporal geometry. In contrast, --asdd approaches such as UniPAD [62] and ViDAR 64 utilize
'Equal contribution. Work done during internship at Huawei.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pre-training
Inage
Prgacke
Buseline
SQS (Ours)
Fine-tuning
Multi-viow Inages
(a) Existing dense HEViOce-Centric pre-training and fine-tuning paradigm
60.2
PT and FT
Caussian Queries
59.2.
FT
20.4
Gaussian Decoder
Pre-training
19.1
Task Task-Specific Decoder
Fine-tuning
Occ.
Dct.
Multi-view Images
Object DO0 (b) Sparse query-based pre-training and fine-tuning paradigm
Queries
(c) The performance improvements after pre-training by our proposed approach
Figure I: The comparison of pre-training and fine-tuning paradigms. (a) Existing pre-training ap- proaches operate on dense BEV or Occupancy representations, which are subsequently shared during fine-tuning. (b) The proposed SQS can be integrated into any sparse query-based perception model, accepting Gaussian queries for pre-training and utilizing them for prediction. (c) We demonstrate the effectiveness of SQS on query-based 3D semantic occupancy prediction (Occ.) and 3D object detection (Det.) tasks. PT and FT denote pre-training and fine-tuning, respectively.
3D volumetric differentiable rendering to reconstruct and predict 3D scene structures. To reduce memory overhead and improve rendering efficiency, a separate line of research [58] 66) introduce 3D Gaussian Splatting (3DGS) [19] for explicit scene representation. By predicting Gaussian parameters for feedforward reconstruction, GaussianPretrain (58) achieves comprehensive scene understanding
by integrating geometric and texture representations. Moreover, VisionPAD [69] projects neighboring frames onto the current frame using rendered depths and relative poses, relying solely on RGB image supervision rather than explicit depth annotations. Although existing pre-training paradigms have substantially enhanced the performance of downstream applications, their reliance on dense BEV representations limits their applicability to SPMs (Fig. [a).
In this paper, we present the first attempt pretrain Sparse Perception Models (SPMs) unlabeled data to enhance their downstream performance, as shown in Fig. [T)). Unlike dense BEV-centric perception models, we find that the latent sparse queries in SPMs lack explicit spatial positions and semantic meanings, making it challenging to directly apply existing rendering-ban -based pre-training methods, which often fail to preserve informative representations during training. address this challenge, we propose SQS, novel pre-training framework for SPMs based on query-based splatting, Unlike previous approaches, SQS introduces small set of adaptive Gaussian queries during pre- training. These queries dynamically predict 3D Gaussians and reconstruct both depth and RGB images via splatting mechanism, enabling the model learn fine-grained representations from unlabeled data in a self-supervised manner. After pre-training, the learned Gaussian queries are used for fine-tuning, where they interact and fuse with task-specific queries, resulting in improved downstream performance. We evaluate our approach tasks such object detection and 3D occupancy prediction. Experimental results demonstrate that SQS consistently achieves significant performance improvements over state-of-the -f-th-artt methods without pre-training.
To this end, our contributions can be summarized as follows:
We propose SQS, the first query-based splatting pre-training technique specifically designed advance Sparse Perception Models (SPMs).
We introduce plug-and-play Gaussian queries, which learns fine-grained features in self-supervised manner during pre-training, and further enhances downstream tasks via interactive feature fusion during fine-tuning.
SQS significantly enhances performance in both occupancy prediction and 3D object detec- tion, surpassing previous state-of-the-e results on multiple autonomous driving benchmarks (i.e., +1.3 mloU on occupancy prediction and +1.0 NDS on 3D detection as Fig. [c)).

---

2 Related Work
Pre-training in autonomous driving. Pre-training has gained remarkable progress in recent years for autonomous driving. Conventional approaches cover supervised [42
54
50 59 con-
categories. With advances becomes an alternative by rendering images from dense BEV or Volume representation. For example, UniPAD [62 volumetric differentiable rendering to reconstruct 3D shape structures and appearance characteristics. Meanwhile, ViDAR (64) predicts the future point cloud using a Latent Rendering operator based on historical embeddings. To achieve effective and efficient rendering. 3D Gaussian Splatting [19] is introduced in some recent works. GaussianPretrain [58] considers 3D Gaussian anchors as volumetric LiDAR points for umified geometric and texture representations. Without explicit depth supervision, VisionPAD [69] reconstructs images employing both voxel velocity estimation and multi-frame pho- tometric consistency. These pre- pre-training pipelines successfully model apatial-tempone representation for dense BEV features.
trastive 23 41
45 651
and masked signal modeling 33
38
2
20
in neural rendering 37 rendering-basee
pre-training
62
64
66
utilizes 3D
52 Bi are not compatible with the paradigms above. Recently, the query-based pre-training in 2D image has been developed. Frozen-DETR (10] utilizes frozen foundation models with class token and patch token, which provide compact context and semantic details, respectively. GLID 32 models pre-training pretext task and other downstream tasks as gvury-tto-aswey problems, Since the sparse pre- vre-training in autonomous driving requires an accurate 3D geometric representation extracted from multi-view images, the existing methods for 2D are inapplicable for Sparse Perception Models (SPMs).
However, the emerging perception methods with sparse route [55 33 34
Sparse Perception Models for 3D Detection and Occupancy Prediction. For the sparse 3D detection, motivated by DETR [5], DETR3D (55) utilizes a sparse set of 3D object queries to sample the 2DD features from images by 3D point projection. To avoid the complex 2D-to-3D projection 18
30 67 directly interact with 3D position-aware features by encoding the 3D position into 2D image features. Without relying dense view transformation nor global attention, Sparse4D (27) iteratively refines anchor boxes via sparsely sampling and fuses spatial-tenn ppatial-temporal features. In the SparseBEV (31). to adapt the detector in both BEV and image space, set of sparse pillar queries initialized in BEV space are applied to interact with the image features.
and feature sampling, PETR series (33 34
52
Regarding occupancy prediction, the -uery approaches [5T] 48 47 are proposed to reduce computational cost. OPUS (51) formulates the task as streamlined set prediction paradigm. SparseOce [48) proposes an efficient occupancy network with 3D sparse diffuser and convolu- tional kernels while OSP 47 presents the Points of Interest (PoIs) represent the scene. Recently, 3DGS has demonstrated the capacity to adapt flexibly to varying object scales and regional complexi- ties in a deformable manner, thereby enhancing resource allocation and overall efficiency. Based on the afforementioned ad advantages, another line of works utilize 3DGS for supervised [16] [1, 75] or self-supervised [I 68 occupancy prediction. In conclusion, compared to the BEV based methods, the sparse algorithms reduce computational cost and broaden the perception range. This distinctive advantage makes the development of a sparse pre-training algorithm for them particularly imperative.
3D Gaussian Splatting in Autonomous Driving. 3D Gaussian Splatting (3DGS) 19 uses multiple 3D Gaussian primitives for fast radiance field rendering, enabling explicit representation with fewer parameters. For reconstructing driving scenes, several approaches are carefully designed for 3D) static 772 6 56] and 4D dynamic [63] scenes. More recently, 3DGS based perception models have been proposed, including occ prediction [16] 14 77).. bev segmentaiong [6] (36] and end-to- end tasks [T]. Alternatively, some recent works apply 3DGS for self-supervised oce prediction
[ 68 GaussianFlowOce [I] and TT-GaussOce [68] model scene dynamics by predicting the temporal flow for each Gaussian throughout the training procedure. Without requiring explicit annotations, GaussTR [17] splats the Gaussians onto 2D perspectives and aligns the extracted features with foundation models. Furthermore, regarding to the self-supervised pre-training, sevaral methods (58 69 adopt 3DGS for explicit geometry representation in the Dense BEV or Volume feature to improve the performance of downstream tasks. Nevertheless, up to now, there still no pre-training scheme that can effectively adapt to Sparse Perception Models (SPMs).
3

---

Proposed Method
In this section, introduce query-based splatting pre-training approach for autonomous driving. The overall architecture of the proposed SQS framework is depicted in Fig. 2
We first provide the necessary preliminaries on 3D Gaussian Splatting, which enables the rendering of both RGB images and depth maps from predicted 3D Gaussians. Subsequently, we briefty outline the image encoder employed to extract multi-scale features from multi-view input images. We then detail the query-based Gaussian transformer decoder, which utilizes Gaussian queries to predict 3D Gaussians and facilitates the learning of fine-grate fine-grained information via self-superrvser splatting. Finally, by incorporating the pre-trained Gaussian queries through a query interaction module during fine- tuning, approach effectively transfers knowledge from the pre-training stage, thereby enhancing downstream query-based learning performance.
3.1 Preliminaries
3D Gaussian Splatting (3DGS) 19 represents 3D scenes through collections of K Gaussians. Each primitive Sk contains 3D position (s E RR3, covariance Σg. opacity og E [0, I]. and spherical harmonics coefficients c g
For differentiable optimization, the covariance matrix is parameterized using scaling S E R and rotation R. E R matrices:
Σ RSS T R
Projection to image coordinates employs view transformation W and Jacobian J:
Σ' JWEW"J"
Rendering combines ordered Gaussians using an alpha-blend rendering proceduure [37, and color at pixel is computed as:
C(p) = Σ C
(1 az).
(3)
EK
1=1
To introduce geometric representation
8
depth rendering is computed as:
D(p) Σ d:o; KEK
II - (1)),
where d; represents the distance from i-th Gaussian to the camera. Unlike volume rendering [37
3DGS uses efficient splat-based rasterization that projects 3D Gaussians 2D image patches.
3.2 Image Encoder
Given a set of multi images I = (I, E 3 = 1,...,NN, corresponding intrinsics K. (K, R333 1,..., N} and extrinsics T (T, E 1,..; N) inputs, where N is the number of cameras. We first need to extract multi-scale multi-view image features for the feed multi-view images to the backbone network (e.g., ResNet- 101 .. and obtain the intermediate multi-level feature F'. To further enhance and aggregate these features across different spatial resolutions, we utilize a Feature Pyramid Network (FPN). The FPN processes the multi-level features and produces captures both high-level semantic information and fine- graigraind spatial details.
subsequent decoder. Specifically,
image features F, which effectively
3.3 Gaussian Transformer Decoder and Gaussian Queries
As illustrated at the top of Fig. SQS employs Gaussian Transformer Decoder to process 2D image features and reconstruct multi-view RGB and depth images. This reconstruction enables the model to capture the underlying geometry and appearance of Gaussian attributes, providing strong feature prior. As a result, SQS enhances downstream sparse perception tasks by supplying a pretrained image backbone and enriched Gaussian query representations.
Fach Gaussian query is initialized as learnable anchors gk Kx paired with queries @ R using zero vectors in high- disenseinnii space, where K is the number of Gaussians, C' and D are

---

Pre-training Stage
Gouasian Queries
Gaussian Transformer Decoder
Image Encoder
Rendered Images
Mulli-ciow
Images
Predicted 3D
Gaussians
Predicted Grassin Queries
Queries
Rendered
Fine-tuning Stage
Weights
Updated Queries
Transformer Decoder (Query Interaction)
Image Encoder
Task Pask-Specifiic Head
Occupancy
Multi-viww
Images
Task Queries
3D Objeet Detection
Figure 2: The pipeline of our proposed SQS. In order to adapt the sparse query-based downstream tasks, we design sparse query-based 3D Gaussian Splatting pre-training paradigm with RGB image and depth supervision. The pre-trained image encoder can be leveraged during the fine-tuning stage, and we also propose a query interaction module to fully exploit the knowledge encapsulated in the pre-trained queries. Our proposed light-weight pre-training paradigm be plugged into any sparse query-based downstream tasks to enhance their performance.
the dimension of Gaussian primitives and query features respectively. During pre-training, guided by the initialized vectors with learnable Gaussian primitives, these query features interact through self-encoding and deformable tross-attention with image features to predict the Gaussian attributes. enabling the retention of rich and detailed geometric information.
To capture the representation across the entire scene, 3D sparse convolution across Gaussian queries is employed to reduce memory cost with linear computational complexity. Here, the 3D position "* E R3 in Gaussian anchor is used voxelize each Gaussian and the sparse convolution leveraged on the voxel grid.
Then the multi-level image feature F is aggregated with deformable cross attention. Concretely, for each Gaussian query, multiple 3D reference points are calculated with offsets added to the 3D position y from the Gaussian anchor. With the intrinsics K. and extrinsics T, these 3D points projected onto 2D image planes to facilitate feature sampling. The resulting set of sampled features serves keys and values in the subsequent attention mechanism.
Finally, to enable the prediction of Gaussian properties, dedicated Gaussian head comprising multi- layer perceptrons (MLPs) is applied to each Gaussian query. To constrain the predicted parameters within appropriate value ranges, sigmoid activations are applied to the position, seale, and opacity, while the rotation is normalized to unit length. The Gaussian parameters μ4, S and R are iteratively refined across decoder layers, where only the position ş is predicted the form delta, while the remaining parameters directly replaced at each refinement step.
3.4 Reconstruction Loss for Pre-training
We employ an L1 loss function for both RGB and depth reconstruction. LiDAR points serve as the ground truth (GT) for depth, and the depth loss is supervised exclusively at pixels corresponding to valid LiDAR measurements. The overall loss formulated as follows:
C Wi Crgb + Wy Cyepthhh
(5)
where wi and we are set to 1.0 and 0.05, respectively, to weight the corresponding terms.
5

---

3.5 Query Interaction for Fine-tuning
Through query-based pre-training, both the image backbone and Gaussian queries acquire rich geometric feature. For fine-tuning downstream SPMs, loading the pre-trained image backbone straightforward; however, reusing pre-trained Gaussian queries remains challenging. Unlike Dense BEV-Centric perception frameworks, which leverage dense BEV representations as a unified intermediate feature, SPMs typically lack such a common structural basis. Instead, these methods often employ task-specific queries and are paired with dedicated decoders designed for different perception tasks. The decoder architectures across various sparse perception algorithms can differ significantly. For instance, SparseBEV [31] initializes queries in 2D BEV space, whereas PETR (33 conducts initialization in 3D space.
To enable consistent reuse of pre- pre-trained queries across diverse sparse perception tasks, shown in the bottom of Fig. 2 we propose plug-in framework based on Query Interaction, facilitating greater flexibility and generalization across diverse tasks. This module explicitly bridges pre-trained Gaussian queries with task-specific queries, facilitating effective transfer and adaptation within heterogeneous decoder frameworks.
Specifically, downstream SPMs initially load the weights of image backbone from sparse pre-trained model. Regarding the reuse of pre-trained Gaussian queries, we fix the parameters of the lightweight pre-trained model. For each perception case, the pre-trained model infers set of corresponding Gaussian anchors paired with query features. We set opacity threshold Othresh to filter out anchors with low opacity. To leverage pre-trained queries efficiently, spatial-awar local attention [46] is applied. To elaborate, given 3D position Hl of task query qe and fll from pre-trained anchors gk. we apply k-nearest neighbor algorithm find & closest 3D Gaussians for each task query. To this end, q: only aggregates features from nearest lo Gaussian queries, finally the local query interaction is formulated as follows:
= LocalAttn(q + MLP(()), 9u + MLP(g))).
Experiments
4.1 Experimental Settings
Dataset and Metrics. We conduct experiments on the nuScenes dataset [3) large-scale benchmark specifically curated for autonomous driving research. The dataset comprises 700 training scenes, 150 validation scenes, and 150 test scenes. Each scene includes synchronized sensor data from six surround-vies cameras and LiDAR, enabling comprehensive 3D perception across diverse urban environments. Comprehensive annotations are provided to support multiple tasks, including 3D object detection, LiDAR semantic segmentation, and 3D map segmentation. Building upon the nuScenes dataset, SurroundOcc (57] provides the dense 3D semantic occupancy annotation tailored for the occupancy prediction task. The annotated voxel grid spans [-50m. 50m] along both X and Y axes, and [-5m, 3m] along the axis with a resolution of 200 x 200 x 16. Each voxel is assigned one of 18 classes, comprising 16 semantic categories, empty class, and an unknown class.
The quality of semantic occupancy prediction evaluated using the mean Intersection Union (mloU) and Intersection-onn ceceiiion-vve--eee Union (IoU) metrics [49]. For 3D object detection, we adopt the standard nuScenes Detection Score (NDS) and mean Average Precision (mAP) metries contain five true positive (TP) metrics, including ATE, ASE, AOE, AVE, and AAE for measuring translation, scale, orientation, velocity, and attribute errors, respectively.
[ We also
backbone initialized from an FCOS3D (54) checkpoint for the occupancy prediction task, while ResNet50 and ResNet101 backbones that are pre-trained with nulmages [3] for the 3D object detection task. The feature extraction employs a feature pyramid network [25] (FPN), producing multi-scale image representations at downsampling factors of 4, 8, 16, and 32. We configure the Gaussian counts to 25,600, and apply two transformer layers to enhance Gaussian attributes. Model training utilizes the AdamW [35] optimizer, with a 0.01 weight decay. The learning rate linearly warms up over the initial 500 steps to 2e-4 and then follows a cosine decay schedule. Pre-training conducted for 20 epochs using batch size of 8. Only random horizontal flipping data augmentation is included. Our implementation is based on MMDetection3D (9. Fine-tuning follows the official downstream model configurations without modification. All experiments are conducted on server with 8 GPUs.
Implementation Details. During the pre-training stage, we adopt a ResNet101-DCN

---

Table 1: 3D semantic occupancy prediction results the SurroundOee val set. While the original TPVFormrr [13] is trained with LiDAR segmentation labels, TPVFormer* is supervised by dense occupancy annotations.
veh.
motorcycle
pelesrian
tralic cone
drive. suf.
sidewalk
vopoctationn
hurrirr
bieycle
mmller
SSC
mnloU
MonoSoone
0.35
4.01
Adaa 40
28.66 15.00
10.64
5:68
19:66
24.94
8.90
8.84
6.47
3.28
10.42
16.21
34.86
15.46
21.89
20.95
11.21
20.54
BEVFomer
30,50 16.75
14.22
6.58
23.46
28.28
8.66
10.77
6.64
4.05
11.20
17.78
37.28
18.00
22,88
22.17
13.50
22.21
TPVTomer 15
11.51 11.66
16.14
7.17
22.63
17.13
8.83
11.39
10:46
8.23
9.43
17.02
8.07
13.64
13.85
10.34
4,90
7.37
TPVTomer
17.10
5.31
27.32
38.87
OodPormer
31.39 19.03
18.65
10.41
20.29
14.19
10.13
12.49
20.77
38.78
24.19
22.21
13.48
21.35
SumoandCeee 37
31.49 20
30.59
11 68
28.06
90.86
10.70
15.14
14.09
12.06
14.38
22.26
37.29
23.70
24.49
22.77
14.89
21.86
GaussianFomer 16
29.83 19:10
19.52
11
26
11
29.78
10.47
13.83
12.58
8.67
12.74
21
39.63
23.28
24.46
22,99
9.59
19.12
GausianFormee SQS (Ours)
31.52 20.40
19.98
11.86
28.21
30.68
10.87
15.03
14.28
9.57
14.74
22.98 39.82
23.88
25.46
23.09
14.56
21.31
Performance IoU
Performannc mioU
25.0
22.5
30
+1.00
30
+2.42
24
Bacck
Bascliae
10
Propertion of Data (36)
Propaction of Data (%)
Figure 3: Data efficiency analysis. To assess data efficiency under limited annotation scenarios, reduce the amount of labeled data used for downstream fine-tuning in the 3D semantic occupancy prediction task. The outcomes demonstrate that our pre-training method significantly enhances performance, even when only a small portion of annotations available.
4.2
Main Results
We evaluate the effectiveness of SQS on two challenging downstream perception tasks: semantic occupancy prediction and 3D object detection.
Semantic Occupancy Prediction. In Tab. M we present comprehensive quantitative comparison of various methods for multi-view 3D semantic occupancy prediction on the SurroundOce validation
novel query- based occupancy prediction set. Among these methods, GaussianFormer [6]
method, performing by our method, the GaussianFormer obtains 1.69 IoU and 1.30 mloU improvements, achieving 20.40% mloU, when compared to the 19.10% mloU for Gaussian effectiveness of SQS for the query-based semantic occupancy prediction task.
par with OccFormer (70] and SurroundOcc (57]. After being pre- pre-trained
These results highlight the
3D Object Detection. We also have conducted experiments in the 3D object detection task, the results illustrated in Tab. 2 To validate the generality of SQS, we have performed two different sparse object detection methods, c.g., Sp [31 and Sparse4Dv3 [29] on the nuScenes validation set. When leveraging the ResNet50 as the image backbone, and the input image size 704 256, SparseBEV achieves 55.8 NDS performance, and an impressive 44.8 mAP metric. After being pre-trained by SQS, we reach the 56.6 NDS and 45.2 mAP performance. Meanwhile, we set the new performance record, that is 56.9 NDS and 47.4 mAP for the Sparse4Dv3 being pre-trained by SQS. Then, we upgrade the backbone ResNet101 and scale the input size to 1408 x 512. Under this setting, the SparseBEV also benefits from our pre-training paradigm with 0.8 mAP and 1.0 NDS improvements. Likewise, Sparse4Dv3 obtains corresponding improvements
0.7 mAP and 0.8 NDS. The results also validate the effectiveness and generality of our pre- pre-training paradigm.
7

---

Table 2: 3D object detection results the nuScenes val split. benefits from perspective
pre- pre-training [1. I indicates methods with CBGS 73 which will elongate epoch into 4.5 epochs.
Method
Backhone
Input Size
Epochs
NDS
mAP
mATE
mASE
mAOE
mAVE
mAAF
PETRv2 [4
ResNet50
704 256
60
45.6
34.9
0.700
0.275
0.580
0.437
0.187
BEVStereo
ResNet50
704 256
50.0
37.2
0.598
0.270
0.438
0.367
0.190
BEVPoolv2 12
ResNer50
704 256
52.6
40.6
0.572
0.275
0.463
0.275
0.188
SOL.OFusion [4
ResNet50
704 2.56
90 I
53.4
42.7
0.567
0.274
0.511
0.252
0.181
Sparse4Dv2 28
ResNer50
704
43.9
0.270
0,282
0.179
StreamPE:TR [33
ResNet50
704 2.66
55.0
45.0
0.613
0.267
0.413
0.265
0.196
SpanseBEV
ResNet50
704 256
54.5
43.2
0.606
0.274
0.387
0.231
0.186
SparseBEV
ResNet50
704
SparseBEV
SQS (Ours)
ResNet50
704 256
36
56.6
45.2
0,564
0.263
0.362
0.232
0.182
Sparse4Dv3
29
ResNet50
704 256
56.1
46.9
0.553
0.274
0.476
0.227
0.200
Sparse4Dv3 SQS (Ours)
ResNet50
704 256
100
56.9
47.4
0.542
0.266
0.458
0.218
0.191
DETR3D f 65
ResNet101-DCN
1600 x 900
24
43.4
34,9
0.716
0.268
0.379
0.842
0.200
BEVFomer
24
ResNet101-DCNN
1600 x 900
51.7
41.6
0.673
0.274
0.372
0.394
0.198
BEVDepth 2
ResNet101
1408
1
41.2
0.565
0.266
0.358
0.331
0.190
Sparse4D f 26
ResNer1O1-DCCN
1600 x 900
48
55.0
44.4
0.603
0.276
0.360
0.309
0.178
SOLOFusion 43
ResNet101
1408 x
:
48.3
0.503
0,264
0.381
0.246
0.207
SparseBEV
31
ResNet101
1408 x 512
59.2
50.1
0.562
0.265
0.321
0.243
0.195
SparseBEV
SQS (Ours)
ResNet101
1408 x 512
60.2
50.9
0.531
0.251
0.318
0.241
0.185
Sparse4Dv3
29
ResNet101
Sparse4Ds3
SQS (Ours)
ResNet101
1408 x 512
100
63.1
54.4
0.498
0.241
0.298
0,187
0.188
Ground Truth
Render Results
Figure 4: Rendering results visualization. Leveraging multi-view images and depth maps projected by the sparse point cloud supervision, SQS demonstrates compelling depth and image reconstruc- tion after pre-training.
Data Efficiency. One of the main advantages of pre-training lies in its ability improve data efficiency for downstream tasks, especially when annotated data is limited. To further demonstrate the effectiveness of our pre- pre-training strategy under conditions where there is plenty of pre-training data but restricted access to labeled downstream samples, we fine-tune our model-initially pre-trained on the full dataset-using different fractions (10% 25%. 50%, and 100%) of the SurroundOcc training set.
Fig. showcases how SQS improves data efficiency. With full fine-tuning data, SQS yields im- provements of +1.69 IoU and +1.3 mloU over the bascline. Notably, this benefit is amplified as less fine-tuning data is used: for instance, fine-tuning with only 10% of the data results in gain of about +3.7 mloU. These findings highlight the strength of SQS achieving notable performance improvements through query- based splatting pre- pre-training, particularly when downstream annotated data scarce.
Visualization of Renderings. As shown in Fig. employing 25,600 queries for 3DGS reconstruction through the multi-view RGB images and depth maps as supervision, SQS could predict promising depth and RGB images during the pre-training stage.

---

Table 3: Ablation studies. We report the IoU and mloU metrics on the SurroundOce val set for the 3D semantic occupancy prediction task. "Rend.", "B.b." and "Inter." denote rendering, image backbone, and query interaction, respectively.
Methods
Rend, RGB
Rend. Depth
Load B.h.
Query Inter.
loU
mloU
Baseline 16
25.8
15.2
Model
23.8 12.0
12.2 10.0
Model B
27.9 12.1
17.3 12.1
Model
28.2 12.4
17.5
Model D
26.3 10.5
15.9 10.7
Model E
25.7
15.3
SQS (Ours)
28.5 12:7
18.0 72.8
4.3
Ablation Studies
In this section, conduct ablations the semantic occupancy prediction task the validation split of the SurroundOce dataset. In order to reduce the training time, we utilize the quarter of training data during the pre-training and fine-tune stage for all experiments. The results are demonstrated in Tab. 3
Rendering Objectives. We first investigate the impact of various rendering objectives during the pre- training stage. Specifically, in ModelA, ModelB, and ModelC of Tab. 3 we employ RGB rendering only, depth rendering only, and combination of both RGB and depth rendering the pre-training objectives, respectively. The results reveal that utilizing only RGB rendering during pre-training impairs fine-tuning performance, resulting reduction 2.0 in IoU and 3.0 in mloU. In contrast, in- corporating depth rendering alone leads to improvements of 2.1 in both IoU and mloU metrics. These findings suggest that rendered depth supervision enhances the geometric representation capability of the pre-trained model, thereby facilitating improved fine-tuning performance. Furthermore, when both RGB and depth renderings are jointly applied, we observe marginal additional improvement. This indicates that rendered RGB supervision provides supplementary benefits in the presence of rendered depth supervision.
Effects of Query Interaction. To further assess the impact of query interaction during the fine-tuning stage, develop Model D, which exclusively incorporates the query interaction mechanism during fine-tuning. As presented in Tab. 3 the pre-trained model is capable of generating meaningful queries for reconstruction, which through the query interaction module during fine-tuning. This results in improvements of 0.5 loU and 0.7 mloU. To eliminate the influence of extra query interaction during the fine-tuning stage, we additionally design Model E to exclusively adapt the query interaction module without pre-training. Its performance remains nearly identical to that of the baseline, indicating that the additional query interaction module offers negligible benefit during fine-tuning, Finally, by initializing with the pre-trained image backbone and FPN neck, we obtain optimal fine-tuning performance, reaching 28.5% IoU and 18 18.0% mloU. These results demonstrate the superiority of the query interaction design within our proposed SQS paradigm.
be further leveraged enhance the query learning process
4.4 Limitations and Future Work
While SQS has achieved further improvements across various downstream tasks, becoming plug- and-play general pre-training paradigm for sparse perception models, it still faces several limitations. One limitation is the extra computation burden and memory consumption incurred by the plug-in pre- training model. Another limitation is the insufficient utilization of pre- pre-training queries for different downstream tasks.
In the future, we will explore how to introduce the semantic information during the pre-training stage and then use the semantic information to distinguish the pre-trained queries for various downstream tasks. We will also try to apply the SQS to query-based end-to-end autonomous driving approaches such Ap [67] and GaussianAD 7

---

5 Conclusion
In this paper, we introduced SQS, a novel query-based splatting pre-training paradigm tailored for autonomous driving SPMs. SQS overcomes the limitations of previous pre-training methods by enabling image backbone and Gaussian queries to learn rich 3D representations through 3D Gaussian prediction and the reconstruction of both images and depth maps. The plug-in design and query interaction strategy further allow seamless transfer and adaptation of the pre-trained model to diverse downstream tasks. Extensive experiments on benchmark datasets validate the effectiveness of SQS, showing promising improvements over various SOTA SPMs.
Acknowledgements
This work was supported by NSFC with Grant No. 62573371, by the Basic Research Project No. HZQB-KCCYZZ2001066 of Hetao Shenzhen-HK S&T Cooperation Zone, by Guangdong S&T Programme with Grant No. 202480101030002, by the Shenzhen General Program No. by the Shenzhen Outstanding Talents Training Fund 202002, by the Guangdong Research Project No.20172T07XX52 and No. 2019CX01X 104, by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B 221 2010000,, by the Guangdong Provincial Key Laboratory of Big Computing Shenzhhenzhen, by the NSFC 61931024& 223666 10662293482, by the Key Area R&D Program of Guangdong Province with grant No. 201 8B030338001, by the Shenzhen Key Laboratory of Big Data and Artificial Intelligence (Grant No. DSSPG20241211173850000 Care Program, by the Shenzhen- Hong Kong Joint Funding No. SGDX2021 1 1231 12401 002, and by Tencent Huawei Open Fund.
JCYJ202205301336000
by China Association for Science and Technology Youth
References
[1] Simon Boeder, Fabian Gigengack, and Benjamin Risse. Gaussianflowoce: Sparse and weakly supervised occupancy estimation using gaussian splatting and temporal flow. arXiv preprint arXi::2502.172 17288, 2025.
Alexandre Boulch, Corentin Sautier, Bjöm Michele, Gilles Puy, and Renaud Marlet. ALSO: automotive lidar self-supervision by occupancy estimation. In CVPR, 2023.
[3] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[4] Anh Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3991-4001, 2022.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213 219.229. Springer, 2020.
[6] Florian Chabot, Nicolas Granger, and Guillaume Lapouge. Gaussianbev: 3d gaussian representation meets perception models for bev segmentation. arXiv preprint arXiv: 2407. 14108, 2024.
[7] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[8] Kai Cheng, Xiaoxiao Long. Kaizhi Yang. Yao Yao, Wei Yin, Yuexin Ma, Weaping Wang. and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. In Forty-first International Conference on Machine Learning, 2024.
MMDetectionSD Contributors. MMDetection Open next-generation platform for general 3D object detection, htttps://githur com/ http:///w...pa..aaaaaa/aaataaaaa 2020.
[10] Shenghao Fu, Junkai Yan, Qize Yang. Xihan Wei, Xiaohua Xie, and Wei-Shi Shi Zheng. Frozen-detr: Enhancing detr with image understanding from frozen foundation models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
10

---

[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. 2016.
[12] Junjie Huang and Guan Huang. Bevpoolv2: A cutting-edge implementation of hevdet toward deployment. arXiv preprint arXiv:2211.17111 2022.
[13] Junjie Huang. Guan Huang. Zheng Zhu, and Dalong Du Bevdet: High-performance calei-camera 3d object detection in bird-eye-view. CoRR, abs/2112.11177 2021.
[14] Yuanhui Huang, Amonnut Thammatadatrakoon, Wenzhao Zheng, Yunpeng Zhang, Dalong Du, and Jiwen Lu. Probabilistic gaussian superposition for efficient 3d occupancy prediction. arXiv preprint arXiv:2412.04388 2024.
[15] Yuanhui Huang. Wenzhao Zheng. Yunpeng Zhang, Zhou, and Jiwen Lu. Tri-perspective view for viseon-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[16] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Gaussianformer. Scene as gaussians for vision-based 3d semantic occupancy prediction. In European Conference on Computer Vision, pages 376-393, Springer, 2024.
[17] Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, and Xinggang Wang. Gausstr: Foundation model- gaussian transformer for self-supervised 3d spatial understanding. arXiv preprint PrXXi:2412.131999 2024.
[18] Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xiangyu Zhang. Far3d: Expanding the horizon for surround-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2561-2569, 2024.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
[20] Georg Krispel, David Schinagl, Christian Fruhwirth-Reisinger, Horst Possegger, and Horst Bischof. Maeli: Masked autocncoder for large-scale lidar point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3383-3392 2024.
[21] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth es- timation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprimt arXiv:2209.1024 2022.
[22] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for nueti-view 3d object detection. In AAAI, 2023.
[23] Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, and Hang Zhao. Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for spatial-aware visual representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1500-1508, 2022.
[24] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird' sedd''----viiw representation from multi-camera images via spatiotemporal trans- formers. In ECCV, 2022.
[25] Tsung-Y Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
[26] Xuewu Lin, Tianwei Lin, Zixiang Pei. Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse ppatial-tempont fusion. arXiv preprint arXiv:2211.1053 2022.
[27] Xuewu Lin, Tianwi Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d v2: Recurrent temporal fusion with sparse model. arXiv preprint arXiv:2005. 14018. 2023.
[28] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d v2: Recurrent temporal fusion with sparse model. arXiv preprint arXiv:2305.14015 14018, 2023.
[29] Xuewu Lin, Zixiang Pei, Tianwei Lin, Lichao Huang, and Zhizhong Su. Sparse4d v3: Advancing end-to-end 3d detection and tracking. arXiv preprint 1...::2311.11772 2023.
[30] Feng Liu, Tengteng Huang, Qianjing Zhang. Haotian Yao, Chi Zhang, Fang Wan, Qixiang Ye, and Yanzhao Zhou. Ray denoising: Depth-aware hard negative sampling for multi-view 3d object detection. In European Conference on Computer Vision, pages 100-217. Springer, 2024.

---

[31] Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-perfornance sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18580-18590, 2023.
[32] Jihao Liu, Jinliang Zheng, Yu Liu, and Hongsheng Li. Glid: Pre-training a generalist encoder-decoder vision model. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition, pages 22851-22860, 2024.
[33] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European conference on computer vixion, pages 531-548. Springer, 2022.
[34] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Peirv2:
A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3262-3272, 2023.
[35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[36] Shu- Hei Lu, Yi- Hsuan Tsai, and Yi- Yi-Ting Chen. Gaussianlss toward real-world bev perception: Depth uncertainty estimation via gaussian splatting. arXiv preprint arXiv:2504.0195: 2025.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[38] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai. Occupancy-mac: Self-supervised pre- training large-scale lidar point clouds with masked occupancy autoencoders. IEEE Transactions on Intelligent Vehicles, 2023.
[39] Chen Min, Dawei Zhao, Liang Xiao. Yiming Nie, and Dai. Voxel-mae Masked autoencoders for pre pre-training large-scale point clouds. CoRR, 01s/22006.99000 2022.
[40] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Ra- binovich. Atlas: End-to-ent 3d seene reconstruction from posed images. In European conference on computer vision, pages 414-431. Springer, 2020.
[41] Lucas Nunes, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Segeontrast: point cloud feature representation learning through self-supervised segment discrimination. IEEE Robotics and Automation Letters, 7(2):21 1623, 2022.
[42] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3142 3152. 2021.
[43] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a bascline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.0 0244343, 2022.
[44] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting 3d. In ECCV, 2020.
[45] Corentin Sautier, Gilles Puy, Alexandre Boulch, Renaud Marlet, and Vincent Lepetit. BEVContrast: Self-supervision in bev space for automotive lidar point clouds. In International Conference on 3D Vision (3DV). 2024.
[46] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances Neural Information Processing Systems, 35:6531- 6543, 2022.
[47] Yiang Shi, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Xinggang Wang. Occupancy as set of points. In European Conference on Computer Vision, pages 72-87. 87. Springer, 2024.
[48] Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, and Chao Ma. Sparsenec: Rethinking sparse latent representation for vision-based semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15035-
15044, 2024.
[49] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang. Yilun Wang. and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[50] Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu. Lewei Lu, Ping Luo, Dahua Lin, al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8405-8415, 2023.
[51] JiaBao Wang. Zhaojiang Liu, Qiang Meng, Liujann Yan, Ke Wang, JIE YANG, Wei Liu, Qibin Hou, and Ming Cheng. Opus: Occupancy prediction using sparse set. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[52] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3621-3631, 2023..
[53] Shihao Wang. Yingfei Liu, Tiancai Wang. Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. arXiv preprint arXiv:2303. 11966, 2023.
[54] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913 923.922.222 2021.
[55] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. Conference on Robot Learning. pages 180-191. PMLR, 2022.
[56] Dongxu Wei, Zhiqi Li, and Peidong Liu. Omni-scene: scene: Omni gaussian representation for ego-centric sparse-view scene reconstruction. arXiv preprint 1rX22:241..06273, 2024.
2024.
[57] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera occupancy prediction for autonomous driving. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 21729-21740, 2023.
[58] Shaoqing Xu, Fang Li, Shengyin Jiang. Ziying Song, Li Liu, and Zhi-xin Yang. Gaussianpretrain: A simple unified 3d gaussian representation for visual pre-training in autonomous driving, arXiv preprint 1xXiv:2411112552 2024.
[59] Xiangchao Yan, Runjian Chen, Bo Zhang, Jiakang Yuan, Xinyu Cai, Botian Shi, Wenqi Shao, Junchi Yan, Ping Luo, and Yu Qiao. Spot: Scalable 3d pre-training via occupancy prediction for autonomous driving. arXiv preprint arXiv:2309.1055 2022.
2023.
[60] Xu Yan, Haiming Zhang, Yingjie Cai, Jingming Guo, Weichao Qiu, Bin Gao, Kaiqiang Zhou, Yue Zhao, Huan Jin, Jiantao Gao, et al. Forging vision foundation models for autonomous driving: Challenges, methodologies, and opportunities. arXiv preprint arXiv:2401 100005,,..00.....0000 2024.
[61] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang. Haiyang Sun, Kun Zhan, Xianpeng Lang, Xi- aowei Zhou, and Sida Peng- Street gaussians for modeling dynamic urban scenes. arXiv preprint vrXiv:2401.01332 2024.
[62] Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang. Hengshuang Zhao, Qibo Qiu, Binbin Lin. et al. Unipad: A universal pre-training paradigm for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15238- 15250, 2024.
[63] Jiawei Yang, Jiahui Huang, Boris Ivanovic, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl. Peter Karkus, et al. Storm: spatioltemporal reconstruction model for large-scale outdoor scenes. In The Thirteenth International Conference on Learning Representations, 2025.
[64] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[65] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. Advances in Neural Information Processing Systems, 36, 2024.
[66] Zhang Yumeng, Gong Shi, Xiong Kaixin, Ye Xiaoqing, Tan Xiao, Wang Fan, Huang Jizhou, Wu Hua, and Wang Haifeng. Bevworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05679, 2024. 2024.
13

---

[67] Diankun Zhang, Guoan Wang, Runwen Zhu, Jianbo Zhao, Xiwu Chen, Siyu Zhang, Jiahao Gong, Qibin Zhou, Wenyuan Zhang, Ningzi Wang, al. Sparsead: Sparse query-centric paradigm for efficient end-to-end autonomous driving. arXiv preprint wrXiv:2404.06892 2024.
[68] Fengyi Zhang. Huitong Yang. Zheng Zhang. Zi Huang. and Yadan Luo. Tt-gausooce: Test-time com- pute for self supervsie occupancy prediction via spatio-temporal gaussian splatting. arXiv preprint arXiv:2503. 084855 2025.
[69] Haiming Zhang, Wending Zhou, Yiyao Zhu, Xu Yan, Jiantao Gao, Dongfeng Bai, Yingjie Cai, Bingbing Liu, Shuguang Cui, and Zhen Li. Visionpad: vision-centi pre- tre-trannig paradigm for autonomous driving. arXiv preprint arXiv:2411.14716, 2024.
[70] Yunpeng Zhang, Zheng Zhu, and Dalong Du Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9433-9443 2023.
[71] Wenzhao Zheng, Junjie Wu, Yao Zheng, Sicheng Zuo, Zixun Xie, Longchao Yang. Yong Pan, Zhihui Hao, Peng Jia, Xianpeng Lang, al. Gaussianad: calssiaan-centric end-to-end autonomous driving. arXiv preprint arXiv:2412 10371, 2024.
[72] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaus- sian: Composite gaussian splatting surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21634-21643, 2024.
[73] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. sampling for point cloud 3d object detection. arXiv preprint arviv: 1908, 09492, 2019.
grouping and
[74] Jialv Zou, Bencheng Liao. Qian Zhang, Wenyu Liu, and Xinggang Wang. Mim4d: Masked modeling with multi-view video for autonomous driving representation learning. arXiv preprint arXiv:2403.0876/ 2024.
[75] Sicheng Zuo, Wenzhao Zheng, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Gaussianworkd: Gaussian world model for streaming 3d occupancy prediction. arXiv preprint 2rX1v:2412.10373, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We have claimed the contributions and results in the abstract and introduction
parts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The limitations and future work are illustrated in Sec. 4.4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
We do not contain the corresponding theoretical assumptions and proof in paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have elaborated the implementation details, experimental settings and training inference in our main paper.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We plan to release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic//giides/CodSSuunn
wwt.:/swbisiiiooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ctiips.cc/public//uue
Codeeuumiiii wettsbbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the training and test details are included in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The error bars are not necessary to report in our settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included this part in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and conform them accurately.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts included in the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pre-trained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper and stated the terms of use accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy //ap:::///eurrt cc/ //oofffrennces/2255 for what should or should be described.
21


Reviews:
[{"review_id": "8XkiUE1DAU", "reviewer": "Reviewer_DKKC", "summary": "The paper titled \"SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving\" introduces a novel pre-training technique called SQS (Sparse Query-based Splatting). It is designed to improve the performance of SPMs (Sparse Perception Models) in autonomous driving. These models are often used for their efficiency and inference speed, brought by the sparse nature of their representations, unlike BEVs or volumetric representations. The SQS method uses a module which predicts 3D Gaussian representations from sparse queries during pre-training, relying on self-supervised splatting to learn contextual features through reconstruction. During fine-tuning, pre-trained Gaussian queries are used in the downstream networks through queries. This connects pre-trained queries with task-specific queries. This makes the method usable for different tasks. \nEvaluations show significant improvements over the state of the art on both 3D object detection and occupancy prediction.  Authors discuss limitations of SQS such as the additional computational and memory loads due to the plug-in pre-training model. Authors also propose paths to explore for future works. The paper also discusses the limitations of SQS, including the additional computational burden.", "questions": "Authors mention potential computational overhead. Could they discuss potential solutions to mitigate this issue and make their contribution even more valuable?\n    \nSince authors mention that pre-training queries are not fully utilized for some downstream tasks, could they discuss which challenges this raises and if they envision any approach which could improve this issue?\n\n\nThe paper mentions the introduction of semantic information during pre-training as a future direction. Could the authors elaborate on how this could be done and how this would impact thbe pre-training?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\nThe proposed approach SQS is a novel contribution which can be very valuable to the field of autonomous driving,. The approach addresses limitations of existing pre-training methods and offers performance improvements. The self-superviseed nature of the splatting to learn contextual features is also very interesting as it reduces reliance on labeled data, which is sometimes scarce. The integration of the pre-trained Gaussian queries into downstream networks is also a strong idea.\nThe paper is clear, well-structured, and readable. Figures help to understand the overall architecture and method, and are clear and of good quality. Supplementary materials provide more experimental results and a discussion on metrics, which are both very welcome.\n\n## Weaknesses\nTable 1 is a bit small which makes it less readable, especially considering the number of figures. In Figure 4, the top right illustration is difficult to see. Maybe try a different background or color mapping. Authors acknowledge that the pre-training queries are not fully utilized for different downstream tasks. This means there might be some room for improvement in the design of the query interaction mechanisms. Authors could discuss potential improvements.", "was_revised": true, "final_justification": "This work is a solid contribution and authors provided clear and very detailed answers to my remarks, notably on computational overhead. Other reviewers also seem to agree that this work could be suitable for this conference. Therefore I will not update my rating and leave it at 5.", "rebuttal": "We sincerely thank you for your careful evaluation and constructive feedback. We address each specific concern below:\n\n**1. Table and Figure Readability (Table 1, Figure 4):**\n\nThank you for highlighting the readability issues. In the final version, we will enlarge Table 1 and reformat it to improve its clarity and readability. For Figure 4, we will adjust the color mapping and background of the top-right figure, to enhance visual contrast and ensure details are more easily discernible.\n\n**2. Fully Utilization of Pre-training Queries and Challenge Discussion:**\n\nWe mentioned that the current design does not fully exploit the pre-training queries for different downstream tasks in the limitation section. As described in Sec. 3.3 of our main paper, SQS leverages a self-supervised splatting module to extract contextual 3D Gaussian representations from sparse queries for reconstructing the RGB images and depth maps of the **whole scenario** in the pre-training stage. However, even though our SQS is a generic plug-in method for sparse perception models (SPMs), the role of these queries diverges depending on the downstream applications. For example, in occupancy prediction, queries represent both foreground and background, while in object detection, only foreground queries are learned. While in SQS, the query interaction module is utilized to learn from pre-trained queries **adaptively** without considering this discrepancy, achieving promising performance improvements.\n\nA potential direction to address this is to introduce explicit semantic information (e.g., pseudo-labels or clustering-based semantic grouping) during pre-training to annotate queries according to their semantic categories. This would enable more adaptive and targeted query interaction in downstream tasks, allowing the selection or weighting of queries based on semantic relevance. Such approaches could facilitate more effective knowledge transfer and potentially enhance performance, particularly in tasks requiring fine-grained discrimination.\n\n**3. Incorporating Semantic Information in Pre-training:**\n\nFollowing the above, incorporating semantic information during pre-training can enhance the query representation’s semantic awareness. This could be achieved by associating each query with semantic pseudo-labels, derived from clustering on features or through auxiliary losses related to semantic segmentation or object proposals. Introducing this semantic distinction during pre-training would allow queries to encode more discriminative and task-relevant features, boosting their transferability and downstream task adaptability. For instance, in the object detection head, only queries pre-trained with ‘foreground’ semantics would participate in the detection decoder, leading to improved sample efficiency and interpretability.\n\n**4. Computational Overhead and Possible Solutions:**\n\nWe acknowledge the concern about additional computational overhead and memory consumption, mainly due to the plug-in mechanism introduced by SQS, as illustrated in the table below for the occupancy prediction task with the quarter of training data during the pre-training and fine-tune stage. To address this, we propose several feasible strategies:\n\n- **Tight Computation Application:** In scenarios with tight computational constraints, users can opt to load only the pre-trained image encoder (backbone and neck) and omit the query interaction module during fine-tuning (SQS* in table below). This setup incurs no extra overhead compared to conventional models while still benefiting from the representational improvements gained through pre-training.\n  \n- **Partial Module Sharing:** It is also feasible to share the image encoder between the Gaussian queries (for query interaction) and downstream task queries, only enabling the Gaussian Transformer Decoder during fine-tuning. As detailed in the table below for SQS**, this results in a modest overhead but provides a favorable accuracy-resource trade-off.\n\n- **Future Optimization:** We are actively exploring model compression and distillation techniques that can further reduce memory consumption and computation, without sacrificing performance.\n\n| Methods                | IoU  | mIoU | Latency | Memory  |\n|------------------------|------|------|---------|---------|\n| GaussianFormer         | 25.8 | 15.2 | 350ms   | 6100 M  |\n| GaussianFormer + SQS   | 28.5 | 18.0 | 560ms (↑60%)   | 10880 M (↑78%) |\n| GaussianFormer + SQS*  | 28.2 | 17.5 | 350ms (↑0%)   | 6100 M (↑0%) |\n| GaussianFormer + SQS** | 28.3 | 17.8 | 452ms (↑30%)   | 8250 M (↑35%) |\n\nWe hope these clarifications directly address your questions and demonstrate our commitment to maximizing both scientific contribution and practical accessibility."}, {"review_id": "mfe0nYoEGe", "reviewer": "Reviewer_uHvD", "summary": "This paper presents SQS, a pretraining framework for sparse 3D perception models targeted towards improving downstream performance in perception tasks for autonomous driving vehicles. Unlike existing dense BeV-centric pretraining methods, SQS introduces adaptive Gaussian queries and a query splatting mechanism that uses a self-supervised learning approach to learn meaningful representations. \n\nIn this pretraining, SQS reconstructs RGB and depth from multi-view inputs and produces 3D Gaussian queries. These learned queries can then be used in downstream tasks (occupancy prediction, 3D object detection) using a query interaction module.", "questions": "Q1. I would like the authors to explain why Sparse4Dv3 is not included as a baseline in Table 2 (3D detection on nuScenes). In lieu of such an explanation, can the authors explain why they believe it is not necessary to achieve SOTA performance for this proposed method to be valuable, at least in relation to 3D detection? Perhaps there are other advantages, e.g., latency, memory, etc.? While not achieving SOTA in and of itself is not *necessarily* a weakness, the method must then be motivated in other ways. If this point cannot be justified, we are left with SQS allowing for SOTA performance in just one downstream perception task.\n\nQ2. Let's ignore W1 about not beating Sparse4Dv3 for now, and this is not necessarily a weakness -- Experiments are performed on two different downstream perception tasks. Showing that this pretraining framework improves performance in even more tasks would show that this approach could in fact be scaled up to production-level autonomous driving stacks. Did the authors experiment with any other tasks? If so, what did the results look like? If not, are there barriers to applying this to other tasks? Or are there plans for future work? Again, two downstream tasks is sufficient in my view for a single conference paper, and this is a question moreso out of curiosity.\n\nQ3. See W3 -- Can the authors provide analysis on how query reuse transpires when deployed? Either quantitatively (how often are queries reused, when are they reused, are there certain factors with which reuse is a covariate?) or with visualizations (is there a clever way we can visualize what the queries are actually learning, and how they may be reused?)\n\nQ4. Can the authors provide more analysis about the real-world applicability (i.e., latency and memory) of this pretraining method, as compared to existing works?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S1. The introduction of a novel sparse query-based splatting pretraining framework seems to be novel and is well-motivated by and aligned with the sparse query paradigm in recent perception systems. Ablation studies show that removing components of the method design does indeed degrade performance.\n\nS2. This pertraining method can be seen as a versatile plug-and-play framework for different downstream perception tasks, demonstrated by competent performance in two different such tasks: occupancy prediction and 3D object detection. This suggests that this framework *could* generalize to training larger more complete production-level autonomous driving systems.\n\nS3. The presentation is coherent, the method pipeline figure is simple and clean, and the visualizations are supportive of the claims made in the paper.\n\nW1. Lack of comparison with Sparse4Dv3 (2023). This is a significant omission in Table 2 (3D object detection on nuScenes), as it seems to me that Sparse4Dv3 is a relevant baseline that outperforms the presented method across many metrics, including NDS and mAP. For that matter, Sparse4Dv2 seems to also beat the presented method in some metrics. I would like the authors to argue why beating Sparse4Dv3 on this task is not necessary. Without a clear explanation, this reduces the number of tasks that this pretraining framework achieves SOTA performance in to just one. This may undercut the Significance of the proposed method.\n\nW2. Latency/memory comparisons are not made against baseline methods. Including this and an analysis would greatly improve the real-world relevance of this proposed pretraining framework.\n\nW3. Lack of in-depth study about success/failure modes. The paper discusses reuse of queries across tasks, but a study of this is not shown. Is there any way to quantify or visualize this supposed reuse of queries, and how it impacts performance in specific cases, whether in a positive or negative manner?", "was_revised": true, "final_justification": "The rebuttal has addressed my few concerns about the paper, so I am raising my score to an acceptance.", "rebuttal": "We sincerely thank the reviewer for the detailed feedback and the opportunity to clarify and strengthen our work. We address each point below:\n\n**1. Comparison with Sparse4Dv3 Performance**\n\nWe appreciate the reviewer’s concern about the lack of comparison to Sparse4Dv3. Our initial choice of SparseBEV as the primary baseline was motivated by its balanced design in terms of detection accuracy and model complexity, and because sparse detection frameworks generally share similar architectural paradigms. While the Sparse4D series has indeed introduced innovations in temporal fusion, many core mechanisms of sparse detection remain comparable across these approaches, making SparseBEV a widely recognized benchmark.\n\nTo further enhance the rigor of our work and address this omission, we have conducted additional experiments incorporating Sparse4Dv3. On the nuScenes validation set, under limited training (using only 1/4 of the training data), Sparse4Dv3 achieves **29.5** mAP and **36.1** NDS, while incorporating our SQS pre-training module leads to improvements, reaching **30.4** mAP and **37.3** NDS. These results demonstrate that our pre-training framework also benefits more advanced architectures like Sparse4Dv3, and thus our findings are not specific to a single backbone or detection framework. We will update the final version of our paper with full-scale results.\n\n**2. Latency and Memory Analysis**\n\nThank you for drawing attention to the question of real-world applicability. We recognize the importance of benchmarking computation and memory efficiency. Our SQS framework is designed with plug-and-play considerations: the extra computational overhead is predominantly introduced during pre-training via the query interaction module. During downstream fine-tuning and inference, it is possible to selectively activate or entirely omit the query interaction module. For most deployment scenarios, loading only the pre-trained image encoder introduces negligible additional latency or memory consumption compared to vanilla baselines, while still reaping the representational benefits of pretraining.\n\nWe have conducted preliminary latency and memory profiling with different variants of the SQS modules across the occupancy prediction baseline (i.e., GaussianFormer) as responded to Reviewer DKKC, and observed that:\nThe SQS-pretrained image encoder adds no extra inference latency when the query interaction module is omitted at deployment.\nWhen using the partial module sharing strategy, the latency increases by approximately **30\\%**, and GPU memory consumption increases by **35\\%** compared to baseline, with a notable performance boost on downstream tasks.\nThese findings suggest that our method is not only flexible, but also practical for scalable real-world applications. We will provide detailed results and hardware configurations in the final version of our paper.\n\n**3. Study and Visualization of Query Reuse Across Tasks**\n\nWe appreciate the reviewer’s insightful question regarding the quantitative analysis and visualization of query reuse across tasks, as well as how such reuse impacts downstream performance.\n\nIn our current implementation, the query interaction mechanism adopts a purely adaptive self-attention framework, where all pre-trained queries are dynamically weighted and selected according to the learned attention scores. This design enables flexible transfer of information but does not explicitly label or fix certain queries for reuse in particular downstream tasks. As such, it is inherently challenging to unambiguously trace and quantify “reused queries” or directly visualize their task-level roles, since query participation is emergent and distributed rather than discrete or statically assigned.\n\nWe acknowledge this limitation in our analysis. At the same time, we believe this adaptivity is a strength: self-attention enables each downstream task module to discover and leverage the most relevant subset of features, rather than hard-coding query-task assignments during pre-training. Consequently, improvements in downstream performance—in both detection and occupancy prediction—serve as indirect evidence that beneficial query reuse and knowledge transfer are indeed taking place.\n\nFor future work, we plan to conduct more comprehensive interpretability studies. For instance, techniques such as analyzing the learned attention distributions, performing query ablation, and visualizing per-query activation maps in downstream heads will likely yield more direct insights into how queries are reused or specialized for specific tasks. We see this as a promising direction for ongoing and follow-up research, and we are grateful to the reviewer for this suggestion."}, {"review_id": "eD498l9N6Q", "reviewer": "Reviewer_aTUF", "summary": "This paper proposes a method to a plug-in module that predicts 3D Gaussian representations\nfrom sparse queries during pre-training and uses the pretrained 3D Gaussian queries to fien-tune the downstream tasks.", "questions": "The reviewer acknowledges that the proposed method do have improvement (vital or subtle across different object categories) with utilization of GaussianFormer + the proposed un-supervised (semi-supervised method with the depth image utlized?). This experiment has demonstrated that SQS can improve the performance of GaussianFormer. However, the paper has claimed that the SQS is somehow model agnostic. Therefore, the reviewer do recommend the paper to add experiments that combine SQS to other models (maybe not the SOTA one, not necessary) to see if SQS has a stable improvement across different backbone models. By adding such an experiment, it can demonstrate the generality of SQS, possibly increase the solidity.", "limitations": "It includes the limitation of resource costing of the proposed method.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1. It proposes a method which leverage un-annotated data, which is promissing.\n\nS2. The Sparse Query utilization seems novel.\n\nS3. The improvement of object detection seems works on all object categories.\n\nW1. The clarity of the presentation should be revised. The reviewer has difficulty on understanding the training of sparse queries and how they are used in the fine-tuning procedure (even though there is Figure 2).\n\nW2. Experiment can be insufficient.", "was_revised": true, "final_justification": "The main concern of the reviewer is solved. Please update the content in rebuttal to the paper.", "rebuttal": "We thank you for the careful evaluation of our work and the valuable feedback. We address each comment in detail below:\n\n**1. Clarity on Query Training and Fine-tuning Procedure with Figure 2**\n\nWe appreciate the reviewer’s concern regarding the clarity of our description about sparse query training and their role during fine-tuning. We recognize that additional textual explanation could complement Figure 2 and improve understanding.\n\nTo clarify, our proposed SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries via a Gaussian Transformer Decoder in the top row of Figure 2 during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. In the downstream fine-tuning stage, we could share the pre-trained image encoder weights for downstream model initialization, and the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction in the Transformer Decoder of sparse perception models (SPMs) to enhance the performance.\n\nIn the revised version, we will further elaborate on the query interaction pipeline, supplement Figure 2 with additional explanatory notes, and provide a step-by-step procedure in the main text for greater clarity.\n\n**2. Generality of SQS Across Backbone Models**\n\nThank you for pointing out the importance of thoroughly demonstrating the model-agnostic claim of SQS. As you observed, our current experiments focus on integrating SQS with GaussianFormer for occupancy prediction and SparseBEV for the 3D object detection task. And in Table 2, we have conducted experiments on two different image backbones of SparseBEV to validate the effectiveness of our method.\n\nTo further evaluate the generality of SQS, we have performed additional experiments, integrating SQS into another sparse perception model e.g., Sparse4Dv3, as pointed by Reviewer uHvD. On the nuScenes validation set, under limited training (using only 1/4 of the training data to accelerate training), Sparse4Dv3 achieves **29.5** mAP and **36.1** NDS. Incorporating our SQS pre-training module leads to further improvements, reaching **30.4** mAP and **37.3** NDS. These new results combined with the results of SparseBEV could further validate the generality of SQS across different backbone models.\nThe final full dataset results will be detailed in our final version.\n\nThese results validate the model-agnostic property of the SQS framework and demonstrate its stable performance enhancement across architectures. We greatly appreciate the reviewer’s suggestion and agree that this substantially increases the solidity and generalizability of our method."}, {"review_id": "67UCsM1UYZ", "reviewer": "Reviewer_zzoE", "summary": "This paper proposes SQS, a pre-training framework for Sparse Perception Models (SPMs) in autonomous driving. SQS introduces query-based 3D Gaussian splatting to enable efficient and effective scene reconstruction from multi-view images and depth maps. The method leverages self-supervised learning during pre-training to capture fine-grained contextual features, which are then transferred to downstream tasks via an interactive query fusion mechanism during fine-tuning. Experimental results demonstrate that SQS improves performance on 3D semantic occupancy prediction and object detection tasks compared to previous state-of-the-art approaches.", "questions": "1) What is the core technical novelty of SQS beyond combining existing components (3DGS and query-based architectures) into a self-supervised pre-training framework?\n\n2) Can the authors provide evidence of cross-dataset generalization of the proposed pre-training method?  \n\n3) Can the authors provide SOTA methods in the comparison?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The experimental evaluation includes two key tasks: semantic occupancy prediction and 3D object detection. Comparisons with state-of-the-art methods demonstrate consistent performance improvements.\n\n2) The paper is well organized and easy to follow.\n\nWeaknesses:\n1) The overall novelty of the paper is somewhat limited. While the proposed method, SQS, introduces a query-based pre-training framework for Sparse Perception Models (SPMs) using 3D Gaussian Splatting (3DGS), both 3DGS and query-based architectures are well-established techniques that have been explored in prior works. The core contribution of the paper lies more in extending these existing components into the domain of self-supervised pre-training for autonomous driving, rather than proposing fundamentally new technical innovations.\n\n2) the evaluation of the pre-training effectiveness is primarily conducted on a single dataset (nuScenes), albeit across multiple downstream tasks. This approach may not fully demonstrate the generalization capability of the proposed pre-training paradigm. Given that different datasets can vary significantly in terms of LiDAR configuration, camera ISP characteristics, and the accuracy of pose annotations—factors that can greatly influence the quality of rendered RGB and depth images—the performance of a pre-training method should ideally be validated across multiple diverse datasets. A more comprehensive cross-dataset evaluation would strengthen the claim of the method’s robustness and broader applicability.\n\n3) The paper does not compare against all possible recent baselines (both in 3D det and 3D Occ), which slightly weakens the strength of the comparison.\n\n4) The description of the query interaction mechanism during fine-tuning (Section 3.5) is somewhat dense and could be improved with a diagram or step-by-step explanation to make it more accessible.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I believe the paper meets the acceptance bar; however, I hope the revised version includes the promised discussions and experiments to further strengthen the work.", "rebuttal": "We thank you for the careful evaluation of our work and the valuable feedback. We address each comment in detail below:\n\n**1. The Core Technical Novelty**\n\nThank you for your feedback and for highlighting the scope of our contributions. We acknowledge that 3D Gaussian Splatting and query-based architectures are established techniques. However, as the reviewers **DKKC** and **uHvD** concluded, our primary focus is on the novel integration of these components within a self-supervised pre-training framework specifically tailored for Sparse Perception Models (SPMs) in autonomous driving. The plug-and-play Gaussian queries can be conveniently integrated within SPMs using the proposed Query Interaction mechanism, leading to improved downstream performance without reliance on extensive labeled data.\n\nWe hope to clarify the novelty of our approach in the revised manuscript and appreciate your constructive comments that help us improve the presentation of our work.\n\n\n**2. Cross-dataset Generalization Analysis**\n\nWe appreciate your insightful comment regarding the cross-dataset evaluation of our method. While our current study primarily focuses on the nuScenes dataset, we selected it due to its comprehensive sensor setup and rich annotations. We agree that evaluating across datasets would show the generalization capabilities of our approach.\n\nTo further evaluate, we conduct experiments on the KITTI-360[1] dataset for 3D semantic occupancy prediction with monocular camera, which differs significantly in sensor setup compared with nuScenes. GaussianFormer is selected as the Sparse Perception Model (SPM) to be improved with our SQS. We use the dense semantic occupancy annotations from SSCBench-KITTI-360[2] for supervision and evaluation. We utilize all training data for pre-training and only **half** data for fine-tuning due to the time limitation. \n\nAfter being integrated with our method, the GaussianFormer obtains **0.48** mIoU and **1.74** IoU improvements, achieving **7.70\\%** mIoU and **29.70\\%** IoU, when compared to the **7.22\\%** mIoU and **27.96\\%**  IoU without SQS. The results successfully demonstrate the generalization of our SQS. Please note that the results could be further improved through hyperparameter tuning, provided that sufficient time is available for experimentation. We will update these results in our revised version of the paper.\n\n**3. Compare with SOTA Methods**\n\nThanks for your insightful comment. We agree that comprehensive comparisons are important. For the 3D semantic occupancy prediction, we evaluate SQS using the SurroundOcc benchmark, and all possible recent baselines in this dataset are included for comparison. For the 3D object detection, we primarily choose SparseBEV as the baseline, given its balanced performance and model complexity. In the revised version, we plan to include results from Sparse4Dv3 and other more recent 3D detection methods in the table.\n\nImportantly, through comparison, we aim to demonstrate that using SQS can significantly improve the performance of current popular SOTA SPMs, rather than solely striving for SOTA metrics in 3D Occupancy and Detection tasks.\n\n**4. More Accessible Explanation for The Query Interaction Mechanism**\n\nThank you for your valuable feedback. We acknowledge that the description of the query interaction mechanism is complex and may benefit from additional clarification. To address this, we will include a diagram illustrating the process flow and step-by-step explanations to enhance clarity in the revised version. The step-by-step explanations are listed as follows:\n\n1. Infer the pre-trained Gaussian queries paired with anchors using the fixed pre-trained model;\n\n2. Obtain Gaussian queries from step 1 by filtering out corresponding anchors with low opacity;\n\n3. In the downstream SPMs, each task queries only aggregate features from $k$ closest 3D Gaussians selected from step 2 by applying $k$-nearest neighbor algorithm.\n\n[1] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292–3310, 2022.\n\n[2] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, et al. Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13333–13340. IEEE, 2024."}]

Author rebuttal:

Meta-review: This paper introduces a novel pre-training approach for sparse perception models in self-driving. The proposed approach is based on predicting Gaussian-splatting representations. Results show improved accuracy and greater data efficiency.
Reviewers all agree on acceptance, pointing to the novelty of the ideas as well as the versatility of the approach in terms of the downstream tasks it improves.
Decision: Accept (spotlight)