Title: SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous Driving

Pre-training

3D Gaussian Splatting

Query-based Perception
Abstract: Sparse Perception Models (SPMs) adopt a query-driven paradigm that forgoes explicit dense BEV or volumetric construction, enabling highly efficient computation and accelerated inference. In this paper, we introduce SQS, a novel query-based splatting pre-training specifically designed to advance SPMs in autonomous driving. SQS introduces a plug-in module that predicts 3D Gaussian representations from sparse queries during pre-training, leveraging self-supervised splatting to learn fine-grained contextual features through the reconstruction of multi-view images and depth maps. During fine-tuning, the pre-trained Gaussian queries are seamlessly integrated into downstream networks via query interaction mechanisms that explicitly connect pre-trained queries with task-specific queries, effectively accommodating the diverse requirements of occupancy prediction and 3D object detection. Extensive experiments on autonomous driving benchmarks demonstrate that SQS delivers considerable performance gains across multiple query-based 3D perception tasks, notably in occupancy prediction and 3D object detection, outperforming prior state-of-the-art pre-training approaches by a significant margin (i.e., +1.3 mIoU on occupancy prediction and +1.0 NDS on 3D detection).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SQS, a pre-training framework for Sparse Perception Models (SPMs) in autonomous driving. SQS introduces query-based 3D Gaussian splatting to enable efficient and effective scene reconstruction from multi-view images and depth maps. The method leverages self-supervised learning during pre-training to capture fine-grained contextual features, which are then transferred to downstream tasks via an interactive query fusion mechanism during fine-tuning. Experimental results demonstrate that SQS improves performance on 3D semantic occupancy prediction and object detection tasks compared to previous state-of-the-art approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The experimental evaluation includes two key tasks: semantic occupancy prediction and 3D object detection. Comparisons with state-of-the-art methods demonstrate consistent performance improvements.\n\n2) The paper is well organized and easy to follow.\n\nWeaknesses:\n1) The overall novelty of the paper is somewhat limited. While the proposed method, SQS, introduces a query-based pre-training framework for Sparse Perception Models (SPMs) using 3D Gaussian Splatting (3DGS), both 3DGS and query-based architectures are well-established techniques that have been explored in prior works. The core contribution of the paper lies more in extending these existing components into the domain of self-supervised pre-training for autonomous driving, rather than proposing fundamentally new technical innovations.\n\n2) the evaluation of the pre-training effectiveness is primarily conducted on a single dataset (nuScenes), albeit across multiple downstream tasks. This approach may not fully demonstrate the generalization capability of the proposed pre-training paradigm. Given that different datasets can vary significantly in terms of LiDAR configuration, camera ISP characteristics, and the accuracy of pose annotations—factors that can greatly influence the quality of rendered RGB and depth images—the performance of a pre-training method should ideally be validated across multiple diverse datasets. A more comprehensive cross-dataset evaluation would strengthen the claim of the method’s robustness and broader applicability.\n\n3) The paper does not compare against all possible recent baselines (both in 3D det and 3D Occ), which slightly weakens the strength of the comparison.\n\n4) The description of the query interaction mechanism during fine-tuning (Section 3.5) is somewhat dense and could be improved with a diagram or step-by-step explanation to make it more accessible.\n\nQUESTIONS:\n1) What is the core technical novelty of SQS beyond combining existing components (3DGS and query-based architectures) into a self-supervised pre-training framework?\n\n2) Can the authors provide evidence of cross-dataset generalization of the proposed pre-training method?  \n\n3) Can the authors provide SOTA methods in the comparison?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper proposes a method to a plug-in module that predicts 3D Gaussian representations
from sparse queries during pre-training and uses the pretrained 3D Gaussian queries to fien-tune the downstream tasks.

STRENGTHS AND WEAKNESSES:
S1. It proposes a method which leverage un-annotated data, which is promissing.

S2. The Sparse Query utilization seems novel.

S3. The improvement of object detection seems works on all object categories.

W1. The clarity of the presentation should be revised. The reviewer has difficulty on understanding the training of sparse queries and how they are used in the fine-tuning procedure (even though there is Figure 2).

W2. Experiment can be insufficient.

QUESTIONS:
The reviewer acknowledges that the proposed method do have improvement (vital or subtle across different object categories) with utilization of GaussianFormer + the proposed un-supervised (semi-supervised method with the depth image utlized?). This experiment has demonstrated that SQS can improve the performance of GaussianFormer. However, the paper has claimed that the SQS is somehow model agnostic. Therefore, the reviewer do recommend the paper to add experiments that combine SQS to other models (maybe not the SOTA one, not necessary) to see if SQS has a stable improvement across different backbone models. By adding such an experiment, it can demonstrate the generality of SQS, possibly increase the solidity.

LIMITATIONS:
It includes the limitation of resource costing of the proposed method.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SQS, a pretraining framework for sparse 3D perception models targeted towards improving downstream performance in perception tasks for autonomous driving vehicles. Unlike existing dense BeV-centric pretraining methods, SQS introduces adaptive Gaussian queries and a query splatting mechanism that uses a self-supervised learning approach to learn meaningful representations. \n\nIn this pretraining, SQS reconstructs RGB and depth from multi-view inputs and produces 3D Gaussian queries. These learned queries can then be used in downstream tasks (occupancy prediction, 3D object detection) using a query interaction module.\n\nSTRENGTHS AND WEAKNESSES:\nS1. The introduction of a novel sparse query-based splatting pretraining framework seems to be novel and is well-motivated by and aligned with the sparse query paradigm in recent perception systems. Ablation studies show that removing components of the method design does indeed degrade performance.\n\nS2. This pertraining method can be seen as a versatile plug-and-play framework for different downstream perception tasks, demonstrated by competent performance in two different such tasks: occupancy prediction and 3D object detection. This suggests that this framework *could* generalize to training larger more complete production-level autonomous driving systems.\n\nS3. The presentation is coherent, the method pipeline figure is simple and clean, and the visualizations are supportive of the claims made in the paper.\n\nW1. Lack of comparison with Sparse4Dv3 (2023). This is a significant omission in Table 2 (3D object detection on nuScenes), as it seems to me that Sparse4Dv3 is a relevant baseline that outperforms the presented method across many metrics, including NDS and mAP. For that matter, Sparse4Dv2 seems to also beat the presented method in some metrics. I would like the authors to argue why beating Sparse4Dv3 on this task is not necessary. Without a clear explanation, this reduces the number of tasks that this pretraining framework achieves SOTA performance in to just one. This may undercut the Significance of the proposed method.\n\nW2. Latency/memory comparisons are not made against baseline methods. Including this and an analysis would greatly improve the real-world relevance of this proposed pretraining framework.\n\nW3. Lack of in-depth study about success/failure modes. The paper discusses reuse of queries across tasks, but a study of this is not shown. Is there any way to quantify or visualize this supposed reuse of queries, and how it impacts performance in specific cases, whether in a positive or negative manner?\n\nQUESTIONS:\nQ1. I would like the authors to explain why Sparse4Dv3 is not included as a baseline in Table 2 (3D detection on nuScenes). In lieu of such an explanation, can the authors explain why they believe it is not necessary to achieve SOTA performance for this proposed method to be valuable, at least in relation to 3D detection? Perhaps there are other advantages, e.g., latency, memory, etc.? While not achieving SOTA in and of itself is not *necessarily* a weakness, the method must then be motivated in other ways. If this point cannot be justified, we are left with SQS allowing for SOTA performance in just one downstream perception task.\n\nQ2. Let's ignore W1 about not beating Sparse4Dv3 for now, and this is not necessarily a weakness -- Experiments are performed on two different downstream perception tasks. Showing that this pretraining framework improves performance in even more tasks would show that this approach could in fact be scaled up to production-level autonomous driving stacks. Did the authors experiment with any other tasks? If so, what did the results look like? If not, are there barriers to applying this to other tasks? Or are there plans for future work? Again, two downstream tasks is sufficient in my view for a single conference paper, and this is a question moreso out of curiosity.\n\nQ3. See W3 -- Can the authors provide analysis on how query reuse transpires when deployed? Either quantitatively (how often are queries reused, when are they reused, are there certain factors with which reuse is a covariate?) or with visualizations (is there a clever way we can visualize what the queries are actually learning, and how they may be reused?)\n\nQ4. Can the authors provide more analysis about the real-world applicability (i.e., latency and memory) of this pretraining method, as compared to existing works?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Driving\" introduces a novel pre-training technique called SQS (Sparse Query-based Splatting). It is designed to improve the performance of SPMs (Sparse Perception Models) in autonomous driving. These models are often used for their efficiency and inference speed, brought by the sparse nature of their representations, unlike BEVs or volumetric representations. The SQS method uses a module which predicts 3D Gaussian representations from sparse queries during pre-training, relying on self-supervised splatting to learn contextual features through reconstruction. During fine-tuning, pre-trained Gaussian queries are used in the downstream networks through queries. This connects pre-trained queries with task-specific queries. This makes the method usable for different tasks. \nEvaluations show significant improvements over the state of the art on both 3D object detection and occupancy prediction.  Authors discuss limitations of SQS such as the additional computational and memory loads due to the plug-in pre-training model. Authors also propose paths to explore for future works. The paper also discusses the limitations of SQS, including the additional computational burden.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe proposed approach SQS is a novel contribution which can be very valuable to the field of autonomous driving,. The approach addresses limitations of existing pre-training methods and offers performance improvements. The self-superviseed nature of the splatting to learn contextual features is also very interesting as it reduces reliance on labeled data, which is sometimes scarce. The integration of the pre-trained Gaussian queries into downstream networks is also a strong idea.\nThe paper is clear, well-structured, and readable. Figures help to understand the overall architecture and method, and are clear and of good quality. Supplementary materials provide more experimental results and a discussion on metrics, which are both very welcome.\n\n## Weaknesses\nTable 1 is a bit small which makes it less readable, especially considering the number of figures. In Figure 4, the top right illustration is difficult to see. Maybe try a different background or color mapping. Authors acknowledge that the pre-training queries are not fully utilized for different downstream tasks. This means there might be some room for improvement in the design of the query interaction mechanisms. Authors could discuss potential improvements.\n\nQUESTIONS:\nAuthors mention potential computational overhead. Could they discuss potential solutions to mitigate this issue and make their contribution even more valuable?\n    \nSince authors mention that pre-training queries are not fully utilized for some downstream tasks, could they discuss which challenges this raises and if they envision any approach which could improve this issue?\n\n\nThe paper mentions the introduction of semantic information during pre-training as a future direction. Could the authors elaborate on how this could be done and how this would impact thbe pre-training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}