Title: VoxDet: Rethinking 3D Semantic Scene Completion as Dense Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Semantic Scene Completion

Free-lunch

Voxel-to-Instance (VoxNT) Trick

Dense Object Detection
Abstract: Semantic Scene Completion (SSC) aims to reconstruct the 3D geometry and semantics of the surrounding environment. With dense voxel labels, prior works typically formulate SSC as a *dense segmentation task*, independently classifying each voxel. However, this paradigm neglects critical instance-centric discriminability, leading to instance-level incompleteness and adjacent ambiguities. To address this, we highlight a "free lunch" of SSC labels: the voxel-level class label has implicitly told the instance-level insight, which is ever-overlooked by the community. Motivated by this observation, we first introduce a training-free **Voxel-to-Instance (VoxNT) trick**: a simple yet effective method that freely converts voxel-level class labels into instance-level offset labels. Building on this, we further propose **VoxDet**, an instance-centric framework that reformulates the voxel-level SSC as *dense object detection* by decoupling it into two sub-tasks: offset regression and semantic prediction. Specifically, based on the lifted 3D volume, VoxDet first uses (a) Spatially-decoupled Voxel Encoder to generate disentangled feature volumes for the two sub-tasks, which learn task-specific spatial deformation in the densely projected tri-perceptive space. Then, we deploy (b) Task-decoupled Dense Predictor to address SSC via dense detection. Here, we first regress a 4D offset field to estimate distances (6 directions) between voxels and the corresponding object boundaries in the voxel space. The regressed offsets are then used to guide the instance-level aggregation in the classification branch, achieving instance-aware scene completion. VoxDet can be deployed on both camera and LiDAR input and jointly achieves state-of-the-art results on both benchmarks, which gives 63.0 IoU on the SemanticKITTI test set, **ranking 1$^{st}$** on the online leaderboard.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to tackle semantic scene completion (SSC), which reconstructs 3D semantic occupancy from camera input. Starting from an interesting observation that the object instance can be regressed in the voxel space, the authors first propose a simple yet effective VoxNT trick to freely convert the voxel-level class labels to the instance-level offsets. Leveraging these free offset labels, they further propose VoxDet, an instance-centric framework that reformulates SSC as a dense object detection task with two subtasks: semantic prediction and offset regression. VoxDet first employs a spatially decoupled voxel encoder to learn task-specific volumetric representations via distinct spatial deformations. Then it introduces a task-decoupled dense predictor to regress instance borders and aggregate instance semantics. Extensive experiments on both camera and LiDAR benchmarks show that VoxDet achieves new state-of-the-art results jointly, which also gives the highest 63.0 IoU on the SemanticKITTI hidden test set (online leaderboard). Overall, this work is both exciting and inspiring for the community.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is clearly organized and well-motivated with good technical flow.\n- Section 3 offers practical, interesting observations that clearly motivate the core method.\n- The proposed VoxNT trick cleverly extracts instance-level scale information from voxel-level class labels without extra supervision and training.\n- In VoxDet, the reformulation of SSC as dense object detection is both novel and effective, providing new insights and perceptive. The detailed algorithm designs are also tailor-designed and technically sound.\n- The approach is computationally efficient and delivers state-of-the-art results on both camera and LiDAR benchmarks, with the highest IoU on the online leaderboard. The multi‐run experiment in appendix demonstrates good robustness.\n- A comprehensive experimental suite, such as feature visualizations and quantitative studies, supports the method design and yields additional insights.\n\n\nAfter a thorough review of the main paper and appendix, I found no major weaknesses and have only the following minor question:\n\n- In the appendix, the method achieves remarkable performance on LiDAR benchmark, which is suggested to reported in the main paper. Accordingly, given the strong LiDAR-based results, I recommend reframing the paper’s focus from camera-based SSC to the more general SCC task across sensing modalities.\n- Can the proposed method be further improved by using both camera and LiDAR inputs together?\n\nQUESTIONS:\nOverall, this is an interesting paper. All questions are listed above, I would appreciate if those issues can be addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a semantic scene completion algorithm based on voxel-to-instance class label offset prediction. After 2D to 3D lifting, the framework conducts two separate spatial decoupling modules, where one branch predicts voxel semantic labels while the other aims to capture instance-level information of each voxel. The learned instance-level voxels are integrated into the semantic-level voxels for scene completion. The authors have conducted exhaustive experiments across multiple benchmarks and comparisons with various baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The idea is simple yet effective.\n- The experiments are quite strong. The authors have conducted extensive ablation studies and analysis of different designs of different modules.\n- The paper is well-written, with proper content for high-level motivations, low-level technical details, and clear figures.\n\n**Weaknesses**\n- The algorithm relies on additional instance-level supervision, which is not always available in the training datasets. Is it possible to leverage existing instance segmentation algorithms (SAM) to create such labels? How large a negative effect would it have on the system?\n- The authors are expected to discuss some existing instance detection algorithms [1,2] in the related works, as they do complementary research (integrating 3D knowledge for 2D tasks).\n- Though this algorithm is designed for outdoor scene completion. I am wondering if it is also applicable to indoor scenarios [3,4]? And if it does not, what would be the key bottleneck / potential ways to address it?\n\n[1] Li, Bowen, et al. \"VoxDet: voxel learning for novel instance detection.\" Advances in Neural Information Processing Systems 36 (2023): 10604-10621.\n[2] Shen, Qianqian, et al. \"Solving instance detection from an open-world perspective.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n[3] Straub, Julian, et al. \"The replica dataset: A digital replica of indoor spaces.\" arXiv preprint arXiv:1906.05797 (2019).\n[4] Dai, Angela, et al. \"Scannet: Richly-annotated 3d reconstructions of indoor scenes.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for semantic scene completion. It formulates the task as object detection. Given that there are no instance-level labels in the data, the authors propose a heuristic for extracting object instances from semantic labels. It is based on the assumption that individual objects are separable in 3D so there is a change in semantic label at the object boundaries. The method achieves state-of-the-art results on two semantic scene completion benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nI acknowledge that the paper achieves state-of-the-art results on two semantic scene completion benchmarks. However, I believe the fundamental assumption behind the method design is wrong and can lead to dangerous consequences on other datasets.\n\nThe authors suggest heuristically extracting instance-level labels from semantic labels based on the fact that there is a semantic label change on the object boundaries. This is, however, not always correct. In L.127 they even acknowledge that sometimes \"minor ambiguities may arise in cases such as the densely overlapping foliage of adjacent trees\". I would argue this is not a minor ambiguity but rather a fundamentally flawed assumption that can be harmful in many real-world situations.\n\nFor example if there is a row of cars parked on the side of the road close to each other, they might end up as a single semantic blob when discretized into a voxel grid. The community often works with voxel sizes between 20 and 40 cm so situations like this might not be rare at all. This in turn would lead to the proposed method learning wrong geometric priors. In the context of semantic scene completion this can have dire consequences: the model would to predict a wrong geometry because of the wrong supervision used during training. Automated driving is a safety-critical application so I believe introducing wrong assumptions like this is extremely dangerous.\n\nThe fact that this assumption happens to improve the results on some datasets does not justify its use in general. First, because on other datasets those cases where it does not hold true might be statistically more significant. Second, because even a small fraction of safety-critical mistakes introduced by this assumption is enough to refrain from using it.\n\nQUESTIONS:\nDo the authors believe that the situations where their assumption is not true are not significant?\n\nOr did I oversee something in the method that makes it robust against wrong supervision due to the proposed heuristic?\n\nAnswers to those are extremely important for the method's applicability.\n\nLIMITATIONS:\nNo, see the strengths and weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates semantic scene completion (SSC) as a dense object detection task, incorporating offset regression and semantic prediction to address 3D semantic scene completion. The spatially-decoupled voxel encoder and task-decoupled dense predictor are introduced to effectively model task-specific spatial deformation and estimate the 4D offset field. Experimental comparisons demonstrate that the proposed method achieves superior performance compared to other recent SSC methods.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths:\n\nThe main insight of this work is the reformulation of SSC. The authors utilize two modules (e.g., spatially-decoupled voxel encoder and task-decoupled dense predictor) to achieve effective semantic scene completion, leading to improved SSC performance.\n\n\nPaper Weaknesses:\n\n1.\tMy first concern is the motivation and high-level summary of the reformulation of SSC. The authors reformulate the SSC as a dense object detection, which is different from previous works. However, as demonstrated in Figure 1(b) and technical details (e.g., classification loss) in Section 4.3 Task-decoupled Dense Predictor, the proposed method appears to be similar to previous segmentation-based methods. The authors should provide more analysis to differentiate between these two types of approaches.\n\n2.\tMy second concern is the technical details. Conv_{(T)}^{(P)} is a 2D convolution module for place (P) and task (T). How many convolutional layers are involved in Conv_{(T)}^{(P)}? Do the different convolution modules share weights or not? It is confusing how the authors utilize these convolution modules to enable effective decoupling. Additionally, the authors utilize FPN to generate V_{cls} and V_{reg}, do the FPNs for these two different tasks have the same network structure and share weights?\n\nQUESTIONS:\nTiny weaknesses:\n\n1.\tThe authors should provide more references to support the assumption “Although prior works, such as Symphonize [16]”\n2.\tUsing (a), (b), (c), and (d) to distinguish the four examples in Figure 2, rather than simply 'left' or 'right,' may be better.\n3.\tConfusing underlines. For example, “Voxel-level SSC as instance-centric object Detection” in Lines 62-63 of the main manuscript.\n4.\tThe authors should increase the font size of the text in the figures.\n\n\nThe paper reformulates the SSC as a dense object detection task, without sufficient analyses and explanations on the motivations. The missing technical details make the main manuscript hard to follow. Additionally, the authors provide more comparisons and deeper analyses to differentiate their proposed method from previous approaches, which formulate SSC as a dense segmentation task. Therefore, I vote for borderline reject in this review round. I would like to raise the rating if the authors can address my concerns.\n\nLIMITATIONS:\nYes. The authors have discussed limitations in the supplemental material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}