Title: IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
Venue: NeurIPS 2025
Accepted: yes
Keywords: Segmentation

Panoptic Scene Completion

Semantic Scene Completion

Scene Understanding

3D Computer Vision
Abstract: Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Extensive experimental results show that our approach achieves state-of-the-art in-domain performance, exhibits superior zero-shot generalization on out-of-domain data, and achieves a runtime reduction exceeding 14$\times$. These results highlight our introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion. Code available at https://github.com/markus-42/ipformer.

IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals

Abstract

Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Extensive experimental results show that our approach achieves state-of-the-art in-domain performance, exhibits superior zeroshot generalization on out-of-domain data, and achieves a runtime reduction exceeding 14 × . These results highlight our introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion. Code available at https://github.com/markus-42/ipformer .

1 Introduction

Panoptic Scene Completion provides a holistic scene understanding that can serve applications like autonomous driving and robotics by reconstructing volumetric geometry from sparse sensor data and assigning meaning to objects in the scene. Such holistic scene understanding requires both accurate geometry and semantic information about a scene. Historically, these tasks have been treated separately and evolved independently due to the distinct nature of geometric reconstruction and semantic interpretation, each requiring specialized algorithms and data representations. Despite the inherent interconnection between geometry and semantics, their separation hindered unified scene understanding. Classical structure-from-motion evolved into simultaneous localization and mapping (SLAM) systems for accurate reconstructions [38, 12, 25], while deep learning facilitated monocular depth estimation (MDE) methods [14, 15, 53], which are restricted to visible surfaces. In contrast,

Symphonies

IPFormer

+DBSCAN

(ours)

Semantic

Scene

三

三

Panoptic

Completion

Scene

InstanceQueries

Instance

Completion

(context-aware)

Clustering

Proposals

I(context-adaptive)

StaticatTestTime

Adaptive atTest Time

一一

一一一一一一

Scene Completion refers to the process of inferring the complete 3D structure of a scene, including both visible and occluded regions, from partial observations [52, 16].

Apart from geometry, semantic image segmentation divides an image into semantically meaningful regions and labels them [58, 7]. Panoptic segmentation [23, 22, 41] extends this approach by segmenting instances [17] for things (countable objects like person or car) while maintaining semantic segmentation for stuff (amorphous regions like road or vegetation). This unified approach is vital for comprehensive scene understanding, as it distinguishes individual object instances while preserving semantic context for background elements, enabling applications like autonomous navigation. The recent trend in scene understanding has been to unify these tasks into Semantic Scene Completion (SSC) [19] and Panoptic Scene Completion (PSC) [3]. SSC integrates geometric completion with semantic labeling, predicting both the 3D structure and semantic categories, represented as a 3D grid composed of voxels. PSC builds on this by adding instance-level segmentation for each voxel, thus distinguishing between individual objects. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene.

To address these limitations, we introduce IPFormer, a 3D Vision Transformer designed to address Panoptic Scene Completion. IPFormer lifts contextual 2D image features to 3D and samples them based on visibility to initialize instance proposals, each implicitly representing semantic instances within the camera view. Based on this context-adaptive initialization, we establish a robust reconstruction signal that enhances subsequent encoding and decoding stages to reconstruct a complete panoptic scene. Sampling instance proposals solely from visible surfaces is grounded in the principle that all potentially identifiable objects within the observed scene must exhibit a visual cue in the camera image to facilitate completion. This approach of directed feature aggregation at train and test time significantly improves instance identification, semantic classification, and geometric completion.

Our contributions can be summarized as follows:

We present IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion.

We introduce a visibility-based sampling strategy, which initializes instance proposals that dynamically adapt to scene characteristics, improving PQ-All by 3 . 62 % and Thing-metrics on average by 18 . 65 % , compared to non-adaptive initialization.

Our experimental results demonstrate that IPFormer achieves state-of-the-art performance on in-domain data, exhibits superior zero-shot generalization on out-of-domain data, and offers a runtime reduction of over 14 × , from 4 . 51 seconds to 0 . 33 seconds.

Comprehensive ablation studies reveal that employing a dual-head architecture combined with a two-stage training strategy, in which SSC and PSC tasks are trained independently and sequentially, significantly improves performance and effectively balances metrics.

2 Related work

Semantic Scene Completion. The task of Semantic Scene Completion has seen a large amount of interest after it was first introduced in [44]. The initial work focused on solving the task for indoor scenes from camera data [60, 61, 35, 29, 28, 27, 8, 2] and used point clouds in outdoor scenarios with LiDAR-based methods [45, 9, 57, 30, 43]. In the following, we will discuss the most relevant contributions for Semantic Scene Completion in the context of autonomous driving. For a more thorough overview, we refer the reader to the surveys of [56] and [62].

Recent progress in perception for autonomous driving has increasingly adopted Vision Transformer architectures [11], notably DETR [5]. DETR introduced learnable object queries, which are trainable embeddings that allow the model to directly predict object sets in parallel, eliminating hand-crafted components like anchor boxes. In SSC, learnable queries can be adapted to predict semantic labels for 3D scene elements ( e.g. , voxels), streamlining the process and enhancing efficiency. Architectures like OccFormer [63] and VoxFormer [32] exemplify this trend, with OccFormer using multiscale voxel representations with query features to decode 3D semantic occupancy, and VoxFormer employing depth-based queries fused with image features, refined via deformable attention. Similarly, Symphonies [21] refines instance queries iteratively using cross-attention between image features and 3D representations, while TPVFormer [20] leverages a tri-perspective view (TPV) to query scene features efficiently. HASSC [50] and CGFormer [59] further refine this general approach by focusing on challenging voxels and contextual queries, respectively. Alternative approaches, such as implicit scenes via self-supervised learning in S4C, [19] and CoHFF [48], which leverages both voxel and TPV representations to fuse features from multiple vehicles, have also emerged, though they do not utilize learnable queries.

It is worth noting that existing literature presents ambiguity regarding context-awareness of queries for SSC. Although both Symphonies [21] and CGFormer [59] incorporate contextual cues during training, their approaches diverge at inference. In particular, Symphonies' instance queries remain static and are shared across all input images, whereas CGFormer's voxel queries preserve contextawareness by dynamically adapting to each input. To clarify this distinction, we differentiate between instance queries and instance proposals as context-aware and context-adaptive, respectively. Our introduced instance proposals effectively adapt scene characteristics to guide feature aggregation during both train and test time.

Panoptic Scene Completion. While panoptic LiDAR segmentation, which predicts panoptics for point clouds directly, has been widely studied [6, 55, 54], voxel-based PSC approaches have only recently been introduced by PaSCo [3]. This method employs a hybrid convolutional neural network (CNN) and Transformer architecture with static instance queries to address LiDAR-based PSC, enhanced by an uncertainty-aware framework. In parallel, camera-based PSC from multi-frame and multi-view (surround-view) images is addressed by PanoOcc [51], which utilizes static voxel queries to aggregate spatiotemporal information, while SparseOcc [34] introduces a sparse voxel decoder and sparse static instance queries to predict semantic and instance occupancy from up to 96 temporal frames. Concurrently, PanoSSC [47] introduces a camera-based PSC method using forward-facing, non-temporal imagery. Its dual-head TPV-based architecture separates semantic occupancy and instance segmentation tasks to infer a 3D panoptic voxel grid. Likewise, our method targets PSC using forward-facing, non-temporal imagery. However, it overcomes a key limitation of prior approaches by avoiding reliance on static queries and instead introduces context-adaptive instance proposals that dynamically adapt to the scene at both train and test time. Notably, PanoSSC [47] is trained on a non-public post-processed version of the SemanticKITTI dataset [1], which, in addition to the absence of released source code, challenges direct comparisons.

3 Methodology

3.1 Overview

Given an input image X ∈ R U × V × 3 (Fig. 2), the backbone predicts image features F ∈ R H × W × F , and the depth net predicts a depth map D ∈ R U × V × 1 , following VoxFormer [32]. Further refining the image features and the depth map allows for generating context-aware 3D representations, denoted as 3D context features F 3D and initial voxels V . Subsequently, visible and invisible voxels, sampled from V using the depth map, attend to the 3D image features F 3D via a series of 3D deformable cross

3DContextGeneration

ProposalInitialization

X

Image

F

Context

3DDeformable

Proposal

Backbone

Net

Self-Attention

Generator

口

Depth

F2D

F3D

Vvis

Estimation

Q

K,V

Vvis

Depth

2D-to-3D

Depth-Based

3DDeformable

3DDeformable

D

Refinement

Liting

VoxelSampling

Vinvis

Cross-Attention

Self-Attention

Encoding

Decoding

3DLocal

Attention

Cross-

Attention

Self-

MLP

PanopticHead

Aggregation

PSC

PSC

Learnable

andGlobal

K,V↑

Stage1

Encoder

Non-

Instance-Voxel

SSC

Learnable

Alignment

Head

SSC

and self-attention mechanisms. This process initializes instance proposals I P and voxel proposals V P . In this setup, every instance proposal can eventually correlate with a single instance within the observed scene.

The voxel proposals V P are encoded by 3D local and global encoding, in line with CGFormer [59], resulting in voxel features V F . We follow a two-stage training strategy in which the first stage uses these voxel features V F to address SSC exclusively. This approach effectively guides the latent space towards semantics and geometry. The second training stage targets overall PSC, thereby detecting individual instances. This is achieved by attending the instance proposals I P over the pre-trained voxel features from the first stage, resulting in instance features I F . These features are used to (i) predict semantics, and (ii) align instances and voxels to aggregate a complete panoptic scene.

3.2 3D Context Generation

Following CGFormer [59], we refine the depth map D and the feature map F ∈ R H × W × F into per-pixel, frustum-shaped depth probabilities D R ∈ R H × W × D , effectively resulting in | D | depth bins per pixel. This is achieved by applying Φ D as a mixture of convolution and neighborhood cross-attention layers [18], formulated as D R = Φ D ( D , F 2D ) . Moreover, the generic image features F are projected into contextual features F 2D ∈ R H × W × C through the context net Φ C , denoted as F 2D = Φ C ( F ) .

The next step is to lift context features F 2D to 3D, following the design of [40]. To elaborate, we lift them to (i) 3D context features F 3D and (ii) 3D initial voxels V . To achieve this, we distribute F 2D along rays defined by the camera intrinsics and weighted by the depth probability distribution D R . This is formulated as

where ( u, v ) are pixel coordinates, d indexes the depth bin, and c is the feature channel dimension. This approach distributes feature vectors across depth bins according to their corresponding probability distributions and thus enables effective lifting of 2D features into a probabilistic 3D volume, while preserving both contextual and spatial information. Note that Eq. (1) does not include learnable parameters.

To additionally create discretized initial voxels V ∈ R X × Y × Z × C , we voxelize F 3D to a regular grid. Let S = { ( u, v, d ) |Q (( u, v, d, c )) = ( x, y, z ) } , then

where Q quantizes continuous coordinates to discrete voxel indices, and C represents the feature dimension. The summation aggregates all features that map to the same voxel, effectively accumulating

IPFormerwithInstanceQueries

IPFormerwithInstanceProposals

Ground-Truth

(context-aware)

(context-adaptive)

II

II

×

Person

==========

Bicyclist

evidence for stronger feature representations. Building upon the concept of [59], we add learnable embeddings to V that enhance its representational capacity.

3.3 Proposal Initialization

Voxel Visibility. The initial voxels V are categorized into visible voxels V vis and invisible voxels V invis, following established SSC works such as [32]. This is achieved by first applying inverse projection Π -1 on the depth map D with camera intrinsics K ∈ R 4 × 4 . This process generates a pseudo point cloud by unprojecting a pixel ( u, v ) to a corresponding 3D point ( x, y, z ) . Furthermore, let M ∈ R X × Y × Z be a binary mask derived from this point cloud that filters the voxel features based on occupancy. Thus, the visible voxel features are defined as V vis = V ⊙ M , where M ( x, y, z ) = 1 if ( x, y, z ) corresponds to an unprojected point, and 0 otherwise. The operator ⊙ denotes the Schur product. Consequently, the invisible voxel features are given by V invis = V ⊙ ( 1 -M ) , with all-ones matrix 1 ∈ R X × Y × Z . Note that this sampling does not include learnable parameters.

To further adapt V vis with image context, we apply a tailored series of 3D deformable attention mechanisms, inspired by [21, 59]. Specifically, we apply 3D deformable cross-attention (DCA) to V vis (as queries) and the 3D context features F 3D (as keys and values), resulting in updated visible voxel features ˜ V x vis . More precisely, an updated voxel query ˜ V x vis at 3D location x computes

where Π( x ) obtains the reference points, ∆ x ∈ R 3 denotes the estimated displacement from the reference point x , and ψ ( · ) refers to trilinear interpolation applied to sample from the 3D context features F 3D. The index n loops through the sampled points out of a total of N points, A n ∈ [0 , 1] represents the attention weights, and W signifies the transformation weight. We present only the formulation of single-head attention and we utilize multiple layers of deformable cross-attention.

Instance Proposals. The visible voxels ˜ V vis are further processed to initialize context-adapted instance proposals I P . Recall from Sec. 1 that instance proposals are initialized at this stage of the architecture, because in principle, all potentially detectable instances within the observed scene must exhibit a visual cue in the camera image to facilitate their completion, thereby defining panoptic scene completion . Following this line of thought, we first apply 3D deformable self-attention (DSA) on ˜ V vis to foster global context-aware attention over visible voxels. This operation, for a query located at x , is expressed as

Inspired by CGFormer ´ s [59] query generator, our instance generator transforms the DSA output to initialize instance proposals via

where W I represents learnable embeddings to improve the representational capacity, and K denotes the maximum number of detectable instances. This initialization typically directs feature aggregation to highly relevant, instance-dependent image regions, as shown in Fig. 3.

Voxel Proposals. To initialize voxel proposals V P , we first merge the visible and invisible voxels through element-wise addition along dimension C , denoted as ˜ V vis ⊕ V invis . We then apply 3D deformable self-attention to distribute updated context information over the entire scene volume, formulated as V P = DSA ( ˜ V vis ⊕ V invis ) , which is in line with [32, 21, 59]:

3.4 Encoding

The purpose of this part of our architecture is to (i) transform voxel proposals V P into voxel features V F that encode semantic information and (ii) project this semantic information onto the instance proposals I P , thus prompting semantic instance-voxel relationships.

To encode semantic information, we propagate the voxel proposals V P through a 3D Local and Global Encoder (LGE) based on CGFormer [59]. This encoder enhances the semantic and geometric discriminability of the scene volume by integrating TPV [20] and voxel representations, effectively capturing global and local contextual features, respectively. For an in-depth discussion, we direct readers to [59]. The 3D LGE Φ LGE transforms voxel proposals V P into semantic voxel features V F ∈ R X × Y × Z × C , formulated as V F = Φ LGE ( V P ) . Building on these semantically encoded voxel features, we apply cross-attention (CA) to instance proposals I P ∈ R K × C (as queries) and semantic voxel features V F ∈ R X × Y × Z × C (as keys and values), producing updated instance features ˜ I CA P . Specifically, the updated instance features are computed as:

where N = X · Y · Z denotes the total number of voxels, p n represents the n -th voxel position, A n ∈ [0 , 1] is the attention weight, and W is the learnable projection weight.

Subsequently, we apply self-attention (SA) to the updated instance features ˜ I CA P to capture global dependencies among the K instances, yielding further refined instances ˜ I SA P . The self-attention mechanism is defined as:

where ˜ I CA P ( n ) denotes the feature vector of the n -th instance from ˜ I CA P . To obtain the final encoded semantic instance features, we further process the self-attention output ˜ I SA P through an MLP Encoder Φ E , yielding I F ∈ R K × C . This step refines the instance representations for downstream tasks. The encoding is computed as I F = Φ E ( ˜ I SA P ) ∈ R K × C .

3.5 Decoding

Since we employ a two-stage training scheme (see Sec. 3.6), the goal of decoding is twofold. In the first stage, a lightweight MLP head ϕ SSC predicts the semantic scene via ϕ SSC ( V F ) .

In the second training stage, a lightweight panoptic head decodes instance features I F to semantic class logits I L ∈ R K × L , where K is the number of instances and L is the number of semantic classes. These logits are further processed with voxel features V F to perform instance-voxel alignment that eventually enables panoptic scene aggregation, inspired by PaSCo [3]. To elaborate, we align the instance features I F with the voxel features V F ∈ R X × Y × Z × C to compute affinity scores H ∈ R X × Y × Z × K by applying the dot product, resulting in H = V F · I ⊤ F . These scores are converted to predicted probabilities via H P = sigmoid ( H ) , where H P denotes the model's confidence in each voxel belonging to each instance. For panoptic aggregation [3], voxel-wise logits

V L ∈ R X × Y × Z × L are computed as V L = H P · I L . It should be emphasized that voxel-wise logits do not represent the probability of belonging to a certain semantic class, but the confidence of belonging to one of the instances. Furthermore, semantic class IDs are assigned via argmax over V L along L , and instance IDs via argmax over H ⊤ P along K , yielding a complete panoptic scene. Note that the alignment process and the panoptic aggregation do not include learnable parameters.

3.6 Training Objective

We follow a two-stage training strategy in which we formulate the learning objectives for SSC and PSC tasks sequentially. Technically, the first stage optimizes

where the weights for the cross-entropy loss, semantic scene-class affinity loss (SCAL), and geometric SCAL [4] are set to λ ce = λ sem = λ geo = 1 , respectively, and the weight for the depth loss [59] is set to λ depth = 10 -4 .

The second stage calculates losses mask-wise by comparing each ground-truth mask with the best matching predicted mask. Consistent with previous research [3, 47], we find the best match by applying the Hungarian method [26] to perform bipartite matching [5], using an IoU threshold of > 50 %. This threshold follows the standard convention established in prior work on panoptic segmentation [23, 3], and the Hungarian algorithm ensures a globally optimal, permutation-invariant one-to-one assignment. After matching, the second stage optimizes

where L mask ce represents mask-wise cross-entropy loss, L dice is the dice loss of [37] and L focal is the focal loss based on [33]. In practice, we set the weights of these losses to λ mask = λ dice = 1 and λ focal = 40 , respectively, following PaSco [3]. Moreover, L depth is defined identically to that in L SSC.

4 Experiments

4.1 Quantitative Results

Dataset and Baselines. We conduct our experiments by (i) in-domain training and testing on the SemanticKITTI SSC dataset [1], and (ii) out-of-domain zero-shot generalization on the distinct SSCBench-KITTI360 [31]. The instance ground-truths for both datasets are provided by PaSCo [3]. Given the novelty of vision-based PSC, no established baselines currently exist for direct comparison. Furthermore, recall from Sec. 2 that PanoSSC [47] represents an existing vision-based PSC method using forward-facing imagery. This would make it a suitable baseline for comparison. However, this approach is trained on a non-public post-processed version of the SemanticKITTI dataset [1], which, in addition to the absence of released source code, challenges direct comparison with our method (see Sec. A.6 of the technical appendix for further details). Consequently, we adapt the latest state-of-the-art vision-based SSC methods [4, 21, 63, 59] and generate instance predictions by clustering Thing-classes from their outputs, in line with PaSCo [3], which introduced the task of 3D PSC. In particular, to ensure fairness and methodological rigor, we apply DBSCAN [13], the same Euclidean clustering algorithm used to construct the PSC ground truth in PaSCo. Furthermore, we utilize pre-trained checkpoints for the baselines, acquired from the official publicly available implementations. We provide further details in Sec. A.1 of the technical appendix.

In-Domain Performance. In summary, IPFormer exceeds all baselines in overall panoptic metrics PQ and PQ † , and achieves best or second-best results on individual metrics, as shown in Tab. 1. These results showcase the significant advancements IPFormer brings to Panoptic Scene Completion, highlighting its robustness and efficiency. Moreover, IPFormer attains superior performance on RQ-All, while securing the second-best position in SQ-All and PQ-Thing. Furthermore, although our approach places second in SQ-Stuff, it surpasses existing methods in RQ-Stuff and notably PQ-Stuff, indicating superior capability in recognizing Stuff-classes accurately. Notably, despite our method achieving state-of-the-art performance in PSC, it exhibits moderate SSC performance on in-domain data. These results are evaluated on the PSC output of our two-staged, fully-trained model. For SSC metrics, the instance IDs of all voxels are disregarded, effectively reducing PSC to SSC. Additionally, our method directly predicts a full panoptic scene, resulting in a significantly superior runtime of

0 . 33 s , thus providing a runtime reduction of over 14 × . In the technical appendix, we further report class-wise quantitative results and a detailed runtime analysis, including memory utilization during training.

Out-of-Domain Zero-Shot Generalization Performance. Trained on SemanticKITTI [1], we cross-validate our method on the distinct SSCBench-KITTI360 dataset [31]. Results in Tab. 2 show that IPFormer demonstrates superior zero-shot generalization capability. Especially in comparison to its closest baseline, CGFormer+DBSCAN, which it outperforms across PSC and SSC metrics. These results highlight the generalization capability of IPFormer by leveraging out-of-domain image context to robustly initialize context-adaptive instance proposals, resulting in superior SSC and PSC performance.

4.2 Ablation Study

Context-Adaptive Initialization. We compare our introduced context-adaptive instance proposals with context-aware, but non-adaptive instance queries (Tab. 3). While initializing proposals adaptively slightly decreases SQ-Stuff, it remarkably increases all Thing-related metrics by 18 . 65 % on average, with RQ-Things showing the best performance gain of 21 . 98 % . This insight demonstrates that adaptively initializing instance proposals from image context not only enables us to drastically recognize more objects, but also enables Thing segmentation with substantially higher quality.

Visibility-Based Sampling Strategy. We examine our proposed visible-only approach for proposal initialization to a method that employs both visible and invisible voxels (Tab. 4). Formally, these proposals are initialized similar to Eq. (5), specifically as DSA ( ˜ V x vis ⊕ V x invis , ˜ V vis ⊕ V invis , x ) + W I . Under our visible-only approach, almost all metrics increase significantly, most notably RQ-Thing and SQ-Thing by 48 . 00 % and 93 . 95 % , respectively, thus evidently enhancing recognition and segmentation of objects from the Thing-category. These findings confirm that visible voxels carry a robust reconstruction signal, which can be leveraged to initialize instance proposals and eventually improve PSC performance.

Deep Supervision. Inspired by PanoSSC [47], we investigate deep supervision in the form of auxiliary losses, essentially supervising the attention maps of each layer during the encoding of the instance proposals (Tab. 5). Our findings show that guiding intermediate layers in such way degrades all metrics. Most notably, when deep supervision is not applied, SQ-Thing registers a substantial improvement of 102 . 61 % .

Dual-head and Two-Stage Training. The baseline (a) in Tab. 6 uses a single head and single-stage training, performing well on Stuff but poorly in SQ-Thing. Introducing separate heads for SSC and PSC in (b) slightly reduces Stuff performance but improves SQ-Thing to nearly match the baseline. Replacing the dual-head with a purely two-stage approach in (c) severely degrades SQ and yields the worst RQ scores. In contrast, adopting both the dual-head architecture and two-stage training in (j) achieves the best overall results. Alternatively, freezing the first stage during the second-stage training (d) proves detrimental, notably reducing RQ-Thing. Furthermore, methods (e)-(i) evaluate the integration of the SSC objective into the second-stage training, by adding L SSC to L PSC via different factors of λ SSC. Overall, these configurations fall short of the superior performance achieved by our final method (j), which applies both a dual-head architecture and a two-stage training strategy that separates SSC and PSC objectives. Note that these findings are underscored by additional ablation experiments presented in Sec. A.7 of the technical appendix.

4.3 Qualitative Results

Presented in Tab. 4, IPFormer surpasses existing approaches by excelling at identifying individual instances, inferring their semantics, and reconstructing geometry with exceptional fidelity. Even for extremely low-frequency categories such as the person category ( 0 . 07 % ) under adverse lighting conditions, and in the presence of trace-artifacts from dynamic objects in the ground-truth data, our method proves visually superior. These advancements stem from IPFormer's instance proposals, which dynamically adapt to scene characteristics, thus preserving high precision in instance identification, semantic segmentation, and geometric completion. Conversely, other models tend to encounter challenges in identifying semantic instances effectively while simultaneously retaining geometric integrity. Moreover, our instance-specific saliency analysis in Fig. 3 underscores these findings.

Monoscene+DBSCAN

Symphonies+DBSCAN

OccFormer+DBSCAN

CGFormer+DBSCAN

IPFormer(ours)

Ground-Truth

Semantic Color Map:

CarBicycleMotorcycle

TruckOther-Veh.

■PersonBiycyclistMotorcyclist

RoadParkingSidewalkOther-GrndBuilding

FenceVegetationTrunk

TerrainPoleTraf.-Sign

5 Conclusion

IPFormer advances the field of 3D Panoptic Scene Completion by leveraging context-adaptive instance proposals derived from camera images at both train and test time. Its contributions are reflected in achieving state-of-the-art in-domain performance, exhibiting superior zero-shot generalization on out-of-domain data, and achieving a runtime reduction exceeding 14 × . Ablation studies confirm the critical role of visibility-based proposal initialization, the dual-head architecture and the two-stage training strategy, while qualitative results underscore the method's ability to reconstitute true scene geometry despite incomplete or imperfect ground truth. Taken together, these findings serve as a promising foundation for downstream applications like autonomous driving and future research in holistic 3D scene understanding.

References

J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV) , 2019.

Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 324-333, 2021.

Anh-Quan Cao, Angela Dai, and Raoul de Charette. Pasco: Urban 3D panoptic scene completion with uncertainty awareness. In CVPR , 2024.

Anh-Quan Cao and Raoul de Charette. MonoScene: Monocular 3D semantic scene completion. In CVPR , 2022.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 213-229. Springer, 2020.

Gang Chen, Zhaoying Wang, Wei Dong, and Javier Alonso-Mora. Particle-based instanceaware semantic occupancy mapping in dynamic environments. IEEE Transactions on Robotics , 41:1155-1171, 2025.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE TPAMI , 2017.

Xiaokang Chen, Kwan-Yee Lin, Chen Qian, Gang Zeng, and Hongsheng Li. 3D sketch-aware semantic scene completion via semi-supervised structure prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4193-4202, 2020.

Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Liu Bingbing. S3CNet: A sparse semantic scene completion network for lidar point clouds. In Conference on Robot Learning , pages 2148-2161. PMLR, 2021.

Peter Christen, David J Hand, and Nishadi Kirielle. A review of the F-measure: Its history, properties, criticism, and alternatives. ACM Computing Surveys , 56(3):1-24, March 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Large-scale direct monocular SLAM. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 834-849. Springer, 2014.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , KDD'96, page 226-231. AAAI Press, 1996.

Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In ICCV , pages 270-279, 2017.

Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In ICCV , pages 3828-3838, 2019.

Keonhee Han, Dominik Muhle, Felix Wimbauer, and Daniel Cremers. Boosting self-supervision for single-view scene completion via knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9837-9847, 2024.

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV , 2014.

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, June 2023.

Adrian Hayler, Felix Wimbauer, Dominik Muhle, Christian Rupprecht, and Daniel Cremers. S4c: Self-supervised semantic scene completion with neural fields. In 2024 International Conference on 3D Vision (3DV) , pages 409-420. IEEE, 2024.

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9223-9232, 2023.

Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, and Xinggang Wang. Symphonize 3d semantic scene completion with contextual instance queries. CVPR , 2024.

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR , 2019.

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR , pages 9404-9413, 2019.

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. Tandem: Tracking and dense mapping in real-time using deep multi-view stereo. In Conference on Robot Learning , pages 34-45. PMLR, 2022.

Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly , 2(1-2):83-97, March 1955.

Jie Li, Kai Han, Peng Wang, Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3351-3359, 2020.

Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid. Rgbd based dimensional decomposition residual network for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7693-7702, 2019.

Jie Li, Yu Liu, Xia Yuan, Chunxia Zhao, Roland Siegwart, Ian Reid, and Cesar Cadena. Depth based semantic scene completion with position importance aware loss. IEEE Robotics and Automation Letters , 5(1):219-226, 2019.

Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Semisupervised implicit scene completion from sparse lidar, 2021.

Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, Yue Wang, Hang Zhao, Zhiding Yu, and Chen Feng. Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2024.

Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9087-9098, 2023.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision (ICCV) , pages 2999-3007, 2017.

Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang, Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. arXiv preprint arXiv:2312.17118 , 2023.

Shice Liu, Yu Hu, Yiming Zeng, Qiankun Tang, Beibei Jin, Yinhe Han, and Xiaowei Li. See and think: Disentangling semantic scene completion. In Advances in Neural Information Processing Systems , volume 31, 2018.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV) , pages 565-571, 2016.

Raul Mur-Artal and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147-1163, 2015.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , 2019.

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision , 2020.

Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In CVPR , 2019.

Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8269-8278, 2019.

Christoph B Rist, David Emmerichs, Markus Enzweiler, and Dariu M Gavrila. Semantic scene completion using local deep implicit functions on lidar data. IEEE transactions on pattern analysis and machine intelligence , 44(10):7205-7218, 2021.

Luis Roldão, Raoul de Charette, and Anne Verroust-Blondet. Lmscnet: Lightweight multiscale 3d semantic completion. In International Conference on 3D Vision (3DV) , 2020.

Luis Roldao, Raoul De Charette, and Anne Verroust-Blondet. 3d semantic scene completion: A survey. IJCV , 2022.

Faranak Shamsafar, Samuel Woerz, Rafia Rahim, and Andreas Zell. Mobilestereonet: Towards lightweight deep networks for stereo matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2417-2426, 2022.

Yining Shi, Jiusi Li, Kun Jiang, Ke Wang, Yunlong Wang, Mengmeng Yang, and Diange Yang. Panossc: Exploring monocular panoptic 3d scene reconstruction for autonomous driving. In 2024 International Conference on 3D Vision (3DV) , pages 1219-1228. IEEE, 2024.

Rui Song, Chenwei Liang, Hu Cao, Zhiran Yan, Walter Zimmer, Markus Gross, Andreas Festag, and Alois Knoll. Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17996-18006, 2024.

Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6105-6114. PMLR, 2019.

Song Wang, Jiawei Yu, Wentong Li, Wenyu Liu, Xiaolu Liu, Junbo Chen, and Jianke Zhu. Not all voxels are equal: Hardness-aware semantic scene completion with self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14792-14801, 2024.

Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 17158-17168, 2024.

Felix Wimbauer, Nan Yang, Christian Rupprecht, and Daniel Cremers. Behind the scenes: Density fields for single view reconstruction. In CVPR , 2023.

Felix Wimbauer, Nan Yang, Lukas Von Stumberg, Niclas Zeller, and Daniel Cremers. Monorec: Semi-supervised dense reconstruction in dynamic environments from a single moving camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6112-6122, 2021.

Binbin Xiang, Yuanwen Yue, Torben Peters, and Konrad Schindler. A review of panoptic segmentation for mobile mapping point clouds. ISPRS Journal of Photogrammetry and Remote Sensing , 203:373-391, 2023.

Zihao Xiao, Longlong Jing, Shangxuan Wu, Alex Zihao Zhu, Jingwei Ji, Chiyu Max Jiang, Wei-Chih Hung, Thomas Funkhouser, Weicheng Kuo, Anelia Angelova, Yin Zhou, and Shiwei Sheng. 3d open-vocabulary panoptic segmentation with 2d-3d vision-language distillation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision - ECCV 2024 , volume 15098 of Lecture Notes in Computer Science , pages 23-41. Springer, Cham, 2025.

Huaiyuan Xu, Junliang Chen, Shiyu Meng, Yi Wang, and Lap-Pui Chau. A survey on occupancy perception for autonomous driving: The information fusion perspective. Information Fusion , 114:102671, 2025.

Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3101-3109, 2021.

Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 , 2015.

Zhu Yu, Runmin Zhang, Jiacheng Ying, Junchen Yu, Xiaohai Hu, Lun Luo, Si-Yuan Cao, and Hui-Liang Shen. Context and geometry aware voxel transformer for semantic scene completion. In Advances in Neural Information Processing Systems , 2024.

Jiahui Zhang, Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, and Hongen Liao. Efficient semantic scene completion network with spatial group convolution. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 733-749, 2018.

Pingping Zhang, Wei Liu, Yinjie Lei, Huchuan Lu, and Xiaoyun Yang. Cascaded context pyramid for full-resolution 3d semantic scene completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7801-7810, 2019.

Yanan Zhang, Jinqing Zhang, Zengran Wang, Junhao Xu, and Di Huang. Vision-based 3d occupancy prediction in autonomous driving: a review and outlook. arXiv preprint arXiv:2405.02595 , 2024.

Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for visionbased 3d semantic occupancy prediction. arXiv preprint arXiv:2304.05316 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are supported by experimental results presented in Sec 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in the experimental setup and the challenges in comparing with existing methods due to the lack of publicly available datasets, as mentioned in Sec. 4.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: We do not include theoretical results that require assumptions or proofs. Our work focuses on experimental validation of the proposed method.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed information on the experimental setup, including datasets, metrics, and baselines (Sec. 4 and technical appendix) and we will release the source code upon publication, which allows for reproducibility of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our source code will be released upon publication, facilitating reproducibility and further research.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify training and test details, including data splits, hyperparameters, and optimizer settings in Sec. 4 and the technical appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report error bars or statistical significance tests for the experiments, focusing instead on established performance metrics and runtime analysis in Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on the compute resources, including GPU and CPU type, memory utilization during training, as well as training and inference time, found in the technical appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms to the NeurIPS Code of Ethics, with no special circumstances requiring deviation.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss potential positive impacts in applications like autonomous driving and robotics, but do not explicitly address negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We not describe safeguards for data or models, as our research does not pose high risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credit the datasets and models used, such as SemanticKITTI [1] and PaSCo [3], and respect their terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets that require documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development does not involve LLMs as important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Technical Appendix

A.1 Experimental setup

Datasets. We utilize the SemanticKITTI dataset, a large-scale urban dataset designed for Semantic Scene Completion. SemanticKITTI provides 64-layer LiDAR scans voxelized into grids of 256 × 256 × 32 with 0.2m voxel resolution, alongside RGB images of 1226 × 370 pixel resolution, covering 20 distinct semantic classes (19 labeled classes plus 1 free class). The dataset comprises 10 training sequences, 1 validation sequence, and 11 test sequences, with our experiments adhering to the standard split [44] of 3834 training and 815 validation grids.

To enable panoptic evaluation , we adopt the PaSCo dataset [3], which extends by generating pseudo panoptic labels. PaSCo employs DBSCAN [13] to cluster voxels of 'thing' classes into distinct instance IDs, using a distance threshold of ϵ = 1 and a minimum group size of MinPts = 8 . The authors of PaSCo ensure that the pseudo labels are valid by comparing them against the available LiDAR single-scan point-wise panoptic ground truth from the validation set. Both the pseudo labels (generated by DBSCAN) and the ground truth are voxelized, and their quality is assessed in regions where both are defined. For more details, including quantitative and qualitative evaluation, see PaSco [3, supplementary material Sec. 8.2]. Since instance labels cannot be derived for SemanticKITTI's hidden test set, we perform evaluations on the validation set.

Metrics. To evaluate our Panoptic Scene Completion approach, we employ the Panoptic Quality (PQ) metric [24], which combines Segmentation Quality (SQ) and Recognition Quality (RQ). PQ is defined as:

where SQ = ∑ ( p,g ) ∈ TP IoU ( p,g ) | TP | and RQ = | TP | | TP | + 1 2 | FP | + 1 2 | FN | . TP , FP , and FN represent true positives, false positives, and false negatives, respectively, and IoU is the intersection-over-union. SQ measures segmentation fidelity via the average IoU of matched segments, while RQ assesses recognition accuracy as the F1-score [10]. We compute PQ, SQ, and RQ across all classes, as well as separately for 'stuff' (amorphous regions) and 'things' (countable objects), to analyze category-specific performance.

The standard PQ metric requires a predicted segment to match a ground-truth segment with IoU > 0 . 5 . However, this strict threshold can be overly conservative for Stuff classes, which typically lack welldefined boundaries. The metric PQ † [42] relaxes the matching criterion specifically for Stuff classes. Formally, PQ † retains the same formulation as PQ:

but relaxes the matching condition used to define true positives (TP † ), and thus false positives (FP † ) and false negatives (FN † ). Specifically, for Thing classes, predicted and ground-truth segment pairs ( p, g ) are matched if IoU ( p, g ) > 0 . 5 , identical to the original PQ definition. For Stuff classes, matches are accepted if IoU ( p, g ) > 0 , thereby allowing any overlapping prediction to contribute to the metric. This relaxation acknowledges the inherent ambiguity in delineating stuff regions and reduces penalties for minor misalignments. As with PQ, we compute PQ † jointly across all classes and separately for stuff and thing categories to enable detailed performance analysis.

A.2 Implementation Details

Training and Architecture. In accordance with [4, 20, 32, 21], we train for 25 epochs in the first stage and 30 epochs in the second stage, using AdamW [36] optimizer with standard hyperparameters β 1 = 0 . 9 , β 2 = 0 . 99 , and a batch size of 1 . We utilize a single NVIDIA A100 80GB GPU, adopt a maximum learning rate of 1 × 10 -4 , and implement a cosine adaptive learning rate schedule decay, with a cosine warmup applied over the initial 2 epochs. Our implementation is based on PyTorch [39]

Method

■

Car (3.92%)

■

Bicycle (0.03%)

■

Motorcycle (0.03%)

■

Truck (0.16%)

■

Other-Vehicle

(0.20%)

■

Person (0.07%)

■

Bicyclist (0.07%)

■

■

Road (15.30%)

■

Parking (1.12%)

■

Sidewalk (11.13%)

■

Other-Ground (0.56%)

■

Building (14.10%)

■

Fence (3.90%)

■

Vegetation (39.30%)

■

Trunk (0.51%)

■

Terrain (9.17%)

■

Pole (0.29%)

■

Traffic-Sign (0.08%)

Mean

PQ

MonoScene [4]*

4

.

10

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00 0

.

00

53

.

92

1

.

27

1

.

01

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

4

.

95

0

.

00

0

.

00

3

.

43

Symphonies [21]*

7

.

80

0

.

00

0

.

00

0

.

14

0

.

37

0

.

22

0

.

00 0

.

00

54

.

79

0

.

46

1

.

92

0

.

00

0

.

00

0

.

00

0

.

63

0

.

00

4

.

88

0

.

00

0

.

00

3

.

75

OccFormer [63]*

3

.

51

0

.

00

0

.

00

1

.

60

0

.

33

0

.

00

0

.

00 0

.

00

59

.

52

3

.

49

6

.

79

0

.

00

0

.

00

0

.

00

0

.

51

0

.

00

6

.

30

0

.

00

0

.

00

4

.

32

CGFormer [59]*

14

.

14

0

.

58

1

.

14

0

.

99

0

.

51

0

.

25

0

.

00 0

.

00

66

.

78

3

.

43

11

.

08

0

.

00

0

.

48

0

.

09

1

.

14

0

.

10

15

.

35

0

.

63

0

.

28

6

.

16

IPFormer (ours)

12

.

83

0

.

45

0

.

56

1

.

69

0

.

92

0

.

27

0

.

00 0

.

00

66

.

28

6

.

16

14

.

52

0

.

00

0

.

13

0

.

00

2

.

28

0

.

00

13

.

25

0

.

12

0

.

15

6

.

30

SQ

Monoscene [4]*

58

.

89

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00 0

.

00

66

.

55

55

.

27

52

.

11

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

55

.

00

0

.

00

0

.

00

15

.

15

Symphonies [21]*

61

.

45

0

.

00

0

.

00

51

.

35

52

.

88

55

.

47

0

.

00 0

.

00

65

.

11

50

.

71

53

.

09

0

.

00

0

.

00

0

.

00

51

.

64

0

.

00

53

.

94

0

.

00

0

.

00

26

.

09

OccFormer [63]*

58

.

62

0

.

00

0

.

00

61

.

53

51

.

59

0

.

00

0

.

00 0

.

00

68

.

26

56

.

29

54

.

69

0

.

00

0

.

00

0

.

00

52

.

22

0

.

00

56

.

33

0

.

00

0

.

00

24

.

19

CGFormer [59]*

65

.

59

52

.

01

54

.

16

57

.

16

59

.

22 67

.

53

0

.

00 0

.

00

70

.

37

54

.

51

55

.

15

0

.

00

50

.

91

52

.

86

52

.

41

51

.

43

58

.

94

58

.

72 53

.

72 48

.

14

IPFormer (ours)

65

.

84

51

.

25

58

.

60

52

.

34

57

.

34

56

.

02

0

.

00 0

.

00

70

.

40 56

.

44 55

.

36

0

.

00

52

.

93

0

.

00

52

.

68

0

.

00

59

.

37

57

.

59

50

.

91

41

.

95

RQ

MonoScene[4]*

6

.

97

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00 0

.

00

81

.

02

2

.

30

1

.

94

0

.

00

0

.

00

0

.

00

0

.

00

0

.

00

9

.

01

0

.

00

0

.

00

5

.

33

Symphonies[21]*

12

.

69

0

.

00

0

.

00

0

.

27

0

.

70

0

.

40

0

.

00 0

.

00

84

.

15

0

.

91

3

.

61

0

.

00

0

.

00

0

.

00

1

.

22

0

.

00

9

.

04

0

.

00

0

.

00

5

.

95

OccFormer[63]*

5

.

98

0

.

00

0

.

00

2

.

60

0

.

64

0

.

00

0

.

00 0

.

00

87

.

20

6

.

20

12

.

41

0

.

00

0

.

00

0

.

00

0

.

98

0

.

00

11

.

19

0

.

00

0

.

00

6

.

69

CGFormer[59]*

21

.

56

1

.

11

2

.

11

1

.

73

0

.

87

0

.

37

0

.

00 0

.

00

94

.

91

6

.

30

20

.

09

0

.

00

0

.

94

0

.

17

2

.

18

0

.

20

26

.

05

1

.

06

0

.

51

9

.

48

IPFormer (ours)

19

.

49

0

.

88

0

.

95

3

.

23

1

.

60

0

.

49

0

.

00 0

.

00

94

.

16

10

.

91 26

.

23

0

.

00

0

.

24

0

.

00

4

.

32

0

.

00

22

.

31

0

.

22

0

.

30

9

.

75

with an fp32 backend. Moreover, we operate on a 50 % voxel grid resolution of X = 128 , Y = 128 , Z = 16 and finally upsample to the ground-truth grid resolution of 256 × 256 × 32 via trilinear interpolation. The feature dimension is set to C = 128 . Training IPFormer takes approximately 3 . 5 days for each of the two stages. The second stage training is initialized with the final model state of the first stage, and we eventually present results for the best checkpoint based on PQ † . Aligning with [32, 21, 59], we adopt a pretrained MobileStereoNet [46] to estimate depth maps, and employ EfficientNetB7 [49] as our image backbone, consistent with [63, 59]. Moreover, the context net consists of a lightweight CNN, while the panoptic head represents a single linear layer for projection to class logits. The deformable cross and self attention blocks during proposal initialization consist of three layers and two layers, respectively, while 8 points are sampled for each reference point. Finally, the cross and self-attention blocks during decoding each consist of three layers.

Clustering. To cluster the predictions of SSC baselines and retrieve individual instances, we apply DBSCAN [13] with parameters ϵ = 1 and MinPts=8, in line with the work of PaSCo [3], which provides ground-truth instances for the SemanticKITTI dataset [1]. The clustering is performed on an AMD EPYC 7713 CPU (allocating 16 cores) with 64GB memory.

A.3 Class-Wise Quantitative Results

In addition to the overall performance on Panoptic Scene Completion in Tab. 1, we report class-wise results in Tab. 7. IPFormer consistently ranks first or second, thereby demonstrating state-of-the-art performance in vision-based PSC, aligning with our primary findings.

As also shown in Tab. 1, all methods showcase suboptimal performance on Thing classes, which arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4 . 53 % of all voxels. To address this, we employ the Sigmoid Focal Loss (Eq. 12), which down-weights easier examples and focuses on harder-to-classify ones, particularly rare Thing classes. Sec. A.7 presents and discusses additional ablation results on the Focal Loss to demonstrate its effectiveness. Our proposed method balances performance between Thing and Stuff classes, achieving state-of-the-art results by prioritizing equitable performance across both categories rather than solely optimizing for Thing classes.

A.4 Additional Qualitative Results.

We provide further qualitative results in Fig. 5, aligning with the primary results in that IPFormer reconstructs and identifies diverse objects of various sizes.

Symphonies+DBSCAN

OccFormer+DBSCAN

CGFormer+DBSCAN

IPFormer(ours)

Ground-Truth

Semantic Color Map:

CarBicycle

Motorcycle

Truck Other-Veh.

Person

RoadParking

Sidewalk

Other-GrndBuilding

Fence

VegetationTrunk

Terrain Pole Traf.-Sign

A.5 Compute Resources and Runtime

In Tab. 8, we provide memory utilization during training and a detailed runtime analysis, in combination with the resulting performance for PQ † . Our method has the highest memory utilization with 52 . 8 GB , while OccFormer has the lowest with 17 . 3 GB . As elaborated in Sec. 4.1, we retrieve PSC predictions for the baselines by clustering their SSC predictions. Consequently, the total runtime for all baselines consists of inference time in addition to clustering time, with the latter being a constant of 4 . 43 s seconds for all baselines. Since our method directly predicts a panoptic scene, the total runtime is equal to the inference time. Thus, IPFormer exhibits a significantly superior runtime of 0 . 33 s compared to the second-best method in terms of PQ † , CGFormer, which has a total runtime of 4 . 70 s . IPFormer therefore provides a runtime reduction of over 14 × .

A.6 Limitations

Experimental Results. Experimental quantitative and qualitative results show IPFormer's state-ofthe-art performance in vision-based PSC. However, there are remaining Thing-classes ( e.g. Motorcyclist) and Stuff-classes ( e.g. Other-Ground) which have not been recognized, due to their low class frequency or geometric fidelity. Despite these challenges, we believe that IPFormer's introduction of context-aware instance proposals will play a significant role in progressing 3D computer vision and specifically Panoptic Scene Completion.

Comparison with PanoSSC. As described in Sec. 4.1, we are not able to compare the performance of IPFormer with the vision-based PSC approach of PanoSSC [47], as this method is trained on a post-processed version of SemanticKITTI that is not publicly available. We aim to collaborate with the authors of PanoSSC to train IPformer on this dataset and evaluate it on PanoSSC ´ s relaxed

20 % IoU threshold for matching of ground-truth and predicted instances, to present representative evaluation results. However, in Tab. 9, we provide evaluation results of IPFormer under PanoSSC's relaxed 20 % IoU threshold. Expectedly, RQ metrics increase substantially, since more instances are recognized, while these are segmented with less fidelity. Thus, SQ metrics decrease.

A.7 Additional Ablation Experiments

Tab. 10 presents extensive additional ablation experiments on the two-stage training, the dual-head architecture, the interplay between SSC and PSC, and sensitivity to hyperparameters. All findings are in line and underscore our findings discussed in Sec. 4.

Two-Stage Training and Dual-Head Architecture. Single-head methods (p, q) struggle to balance SSC and PSC, with single-stage configuration (p) performing the worst due to its inability to separate semantic and instance-level learning. Single-stage, dual-head methods (a, b) also fall short, as the lack of stage-wise optimization hinders instance registration. Two-stage methods highlight the advantages of stage-wise training: SSC-focused approaches (e, f, k) excel in SSC but underperform in PSC due to limited adaptation, while PSC-prioritized methods (j, n) improve instance registration but compromise semantic consistency or balance. IPFormer, method (r), decouples SSC and PSC optimization, achieving strong instance registration and balanced performance across both tasks, with a moderate SSC trade-off.

Interplay between SSC and PSC. Across the design space, methods that emphasize SSC (e, f, k) achieve strong semantic scores but degrade PSC performance, especially PQ-Thing. Conversely, approaches prioritizing PSC (j, n, q) boost PQ-Thing or PQ-Stuff at the cost of SSC quality or overall balance. Joint or single-head variants (a, b, p, q) further struggle with instance registration or overall consistency. In contrast, our dual-head, two-stage method (r) yields the best PQ-All and strong performance across PQ-Thing and PQ-Stuff, with only a minor SSC trade-off.

Sensitivity to Hyperparameters. We train the SSC task in Stage 1 (Eq. 8) using established hyperparameters for the cross-entropy, Semantic-SCAL, and Geometric-SCAL cost functions (Sec. 3.6). All weights associated with these functions are set to 1 , consistent with state-of-the-art and established SSC works, such as CGFormer [59], OccFormer [63], and MonoScene [4]. Nevertheless, we investigate the effect of removing the Depth loss, as its impact has not been extensively studied. Furthermore, we analyze the effect of varying hyperparameters of the Sigmoid Focal Loss [33], specifically designed to down-weight easier examples and focus the training process on harder-toclassify examples:

where p t is the predicted probability for the true class after applying the sigmoid function, α t is the class-balancing weight, and γ controls the down-weighting of well-classified examples.

While methods (e) and (f) in Tab. 10 achieve the highest SSC performance in terms of mIoU and IoU, respectively, their PSC performance deteriorates significantly. A similar trend is observed for method (k), which attains the second-best results for both mIoU and IoU. In contrast, the best PSC performance, measured by PQ-Thing, is achieved by method (j). Although this method demonstrates satisfactory SSC performance, it suffers from reduced PQ-Stuff and, consequently, lower PQ-All performance. Moreover, the highest PQ † and PQ-Stuff performance is achieved by methods (n) and (q), respectively. However, both methods exhibit a notable decline in PQ-Thing. Specifically, method (n) experiences a substantial reduction in SSC performance, whereas method (q) maintains adequate SSC scores.

Head(s)

Stage 1

Stage 2

SSC Metrics

PSC Metrics

SSC Losses

SSC Losses

PSC Losses

Depth

CE

Sem

Geo

CE

Sem

Geo

Depth

CE

DICE

Focal

IoU

mIoU

PQ

†

PQ-All

PQ-Thing

PQ-Stuff

λ

depth

λ

ce

λ

sem

scal

λ

geo

scal

λ

ce

λ

sem

scal

λ

geo

scal

λ

depth

λ

ce

λ

dice

λ

focal

α

γ

(a)

Dual

-

-

-

-

1

.

00

1

.

00

1

.

00

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

41

.

30

15

.

26

14

.

21

5

.

57

1

.

06

8

.

84

(b)

Dual

-

-

-

-

1

.

00

1

.

00

1

.

00

0

.

0001

1

.

00

1

.

00

50

.

00

0

.

21

2

.

30

39

.

45

13

.

06

14

.

45

5

.

03

0

.

99

7

.

79

(c)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

1

.

00

1

.

00

1

.

00

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

43

.

44

15

.

79

14

.

35

6

.

06

1

.

66

9

.

27

(d)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

50

0

.

50

0

.

50

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

43

.

62

15

.

30

14

.

42

6

.

27

1

.

88

9

.

46

(e)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

20

0

.

20

0

.

20

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

43

.

99

16

.

29

14

.

46

6

.

05

1

.

60

9

.

29

(f)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

10

0

.

10

0

.

10

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

44

.

13

16

.

11

14

.

41

6

.

08

1

.

72

9

.

26

(g)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

01

0

.

01

0

.

01

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

43

.

94

15

.

04

14

.

67

5

.

16

0

.

90

8

.

27

(h)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

1

.

00

1

.

00

1

.

00

0

.

0001

1

.

00

0

.

00

40

.

00

0

.

25

2

.

00

43

.

34

15

.

83

13

.

89

5

.

33

0

.

49

8

.

86

(i)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

10

0

.

10

0

.

10

0

.

0001

1

.

00

0

.

10

40

.

00

0

.

25

2

.

00

42

.

97

15

.

64

14

.

11

5

.

59

1

.

27

8

.

74

(j)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

03

0

.

03

0

.

03

0

.

0001

1

.

00

1

.

00

50

.

00

0

.

21

2

.

30

43

.

53

15

.

65

14

.

39

6

.

19

2

.

38

8

.

96

(k)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

0

.

50

0

.

50

0

.

50

0

.

0001

1

.

00

1

.

00

50

.

00

0

.

21

2

.

30

44

.

07

16

.

25

14

.

23

5

.

97

1

.

67

9

.

11

(l)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

-

-

-

0

.

0001

1

.

00

2

.

00

40

.

00

0

.

25

2

.

00

32

.

45

7

.

28

13

.

92

5

.

92

1

.

58

9

.

08

(m)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

-

-

-

0

.

0001

1

.

00

1

.

00

50

.

00

0

.

21

2

.

30

40

.

45

14

.

74

14

.

59

5

.

74

1

.

53

8

.

80

(n)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

-

-

-

0

.

0001

1

.

00

1

.

00

45

.

00

0

.

21

2

.

30

36

.

73

11

.

69

15

.

14

4

.

43

0

.

78

7

.

08

(o)

Dual

-

1

.

00

1

.

00

1

.

00

-

-

-

-

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

41

.

16

14

.

90

14

.

42

6

.

15

1

.

68

9

.

40

(p)

Single

0

.

0001

1

.

00

1

.

00

1

.

00

-

-

-

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

4

.

90

2

.

53

10

.

66

0

.

42

0

.

13

0

.

63

(q)

Single

-

-

-

-

-

-

-

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

42

.

31

14

.

80

14

.

64

6

.

23

1

.

75

9.49

(r)

Dual

0

.

0001

1

.

00

1

.

00

1

.

00

-

-

-

0

.

0001

1

.

00

1

.

00

40

.

00

0

.

25

2

.

00

40

.

90

15

.

33

14

.

45

6.30

2.09

9

.

35

Finally, our proposed IPFormer configuration, method (r), achieves a balanced PSC performance by obtaining the best score for PQ-All and the second-best results for both PQ-Thing and PQ-Stuff, with a moderate trade-off in SSC performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives the instance queries from proposal generator with semantic scene completion. With known initialization, the proposed methods out perform without any clustering.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths : Better reconstruction quality for the thing-category instances.\n\nWeaknesses : \n1. Inefficient pipeline - The proposed network are working with two stages, one is for learning semantic scene completion to initialize the  instances, and with this IPFormer compute instance segmentation. \n\n2. The problem is from the SemanticKITTI dataset not from the method.  As the ground truth gives misclassification and geometric ambiguity, instance queries might not aligned well as shown in Figure 3. If the method itself is matter, then the author should demonstrate that instance are not distinguished in the other clean dataset( such as Occ3D or nuCraft)\n\n3. Not the first time for the Image only Panoptic Scene Completion. \nSparseOcc* already proposed the Panoptic Scene Completion in Occ3D based on the nuScene dataset.  \nSparseOcc* : Fully Sparse 3D Panoptic Occupancy Prediction, ECCV'24\n\nQUESTIONS:\n1. What is the main contribution to reduce the runtime? No additional DBSCAN computation? \n2. Should we consider the invisible voxel features? This might be occurs another inefficient computation and might mislead at instance-voxel alignment.  What about just drop the invisible voxel features? This might increase the efficiency of whole pipeline. \n3. Could this also comparable in the Occ3d nuScene dataset? Especially for the SparseOcc that I've method the weakness.\n\nLIMITATIONS:\nThanks for the contribution in autonomous vision community. \nAs the SemanticKITTI dataset is more difficult setup to improve the task. Author should formulate the problem with this hard condition(such as sparse, single view) or tackling the weaknesses like inefficiency pipeline (compare to the Mask2Former based architecture). But still I concern the problem itself is from the SemanticKITTI dataset not the architecture. So I suggest to demonstrate on the other dataset, and prove the problem is also valid.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose IPFormer, a monocular image-based 3D panoptic scene completion method which jointly recovers the complete 3D geometry of a scene while identifying individual objects and their semantics from a single RGB image. To improve upon previous works which learn query proposals during training and fix them at test time, they propose to generate instance proposals from an input image's context at both training and testing, enabling better identification and localization of object instances. Furthermore, a two stage training pipeline is proposed where the semantic scene completion task is first optimized for to learn semantics and complete geometry and then the panoptic scene completion task is optimized to identify individual object instances.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- DETR-style queries, which are learned during training and fixed at test time, have limited representational capacity and ignore input image's context which may hurt generalization. Therefore the proposed dynamically adaptive instance proposals which are generated from the input image context seems like a sound improvement over DETR-style instance queries. This improvement is backed by the ablation in Table 2 which shows consistent improvement in almost all metrics when switching from learned but fixed instance queries to their proposed context adaptive instance proposals.\n- As the proposed method directly predicts instance segmentations, it offers a significant speed up (14x) over the monocular semantic scene completion works which require running DBScan for identifying object instances.\n\n**Weaknesses:**\n- Instance proposal generation is the main contribution of this work yet there is no description of what $g$ is in Eq. 5. It mentions inspiration from CGFormer's query generator, but from my understanding CGFormer’s query generator generates a dense voxel grid of queries and then does a depth-based query proposal. When comparing architectures to CGFormer, these two steps are $V$ from Eq. 2 and $\\tilde{V_{vis}}$ from Eq. 3 in this work, so it's not clear how CGFormer's query generator is adapted to generate $I_{p}$. The technical details of the contribution needed to be more clearly stated.\n- Despite being trained for the panoptic scene completion task, IPFormer barely outperforms CGFormer + DBScan which is not trained for panoptic scene completion. Looking at Table 1 in the main paper and Table 6 in the appendix, this boost in performance really only comes from better RQ in *stuff* categories (e.g., sidewalks, vegetation). Alternatively, CGFormer actually outperforms the proposed IPFormer on segmenting individual object instances of the *thing* categories. Moreover, this small boost in performance comes at a cost of a large drop in performance on SSC metrics as shown in Table 8 of the appendix.\n- Since the work is heavily based off of other works (CGFormer, PaSCo), the methodology section could be written a little more clearly to identify what is from those works and what is unique or a contribution from the proposed approach. For example, the \"voxel visibility\" and \"voxel proposals\" from Section 3.2 are essentially straight from CGFormer but there is no mention of this. Section 3.4 could use a better transition between what part of the encoder is from CGFormer and what is not. For Section 3.5, it is unclear whether this panoptic aggregation is just directly from PaSCo or if there are differences.\n\nQUESTIONS:\n**Questions:**\n- The definition of $S$ in line 151 is confusing to me. Why is the quantization of the 3D feature $F_{3D}$ a point coordinate $(x, y, z)$? The point coordinate should only be dependent on $(u, v, d)$ and has nothing to do with the feature $F_{3D}$.\n\n**Suggestions:**\n- In Figure 2, it would be better to have an additional arrow going from the RGB image to depth map to show that the depth is estimated. The current figure makes it seem as if the method is an RGBD method instead of a RGB method.\n- The description of 3D deformable cross attention in Eq. 3 and lines 172-175 could be improved. There is no mention of what *reference* points and *sampled* points are. Furthermore, $\\Pi$ is defined as the projection from 3D to 2D in line 160 but then seems to be redefined to mean something else in line 172.\n- In line 265, I think it was meant to say “Stuff-instances” not “Thing-instances”\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n​This paper proposes IPFormer, a novel approach to ​​vision-based 3D Panoptic Scene Completion (PSC)​​ by leveraging ​​context-adaptive instance proposals​​ dynamically derived from image context. The contributions of this paper can be summarized as follows.\n\n- Visibility-based sampling​​ (Sec. 3.3): Initializes instance proposals from visible voxels using camera depth and image features, ensuring dynamic adaptation to scene content at test time.\n\n- Dual-head architecture​​ (Sec. 3.1, Table 5): Separates semantic scene completion (SSC) and PSC tasks via a two-stage training strategy, guiding latent space toward geometry/semantics before instance registration.\n\n- Efficient PSC aggregation​​ (Sec. 3.5): Uses instance-voxel affinity scores to align semantics and instances without learnable parameters.\n\n- State-of-the-art results​​: Outperforms SSC baselines + clustering (e.g., CGFormer + DBSCAN) in PQ (14.45) and PQ-All (6.30) on SemanticKITTI, with ​​14× faster inference​​ (0.33s vs. 4.51s).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- To my knowledge (which might be limited), this is the first method to use ​​test-time adaptive proposals​​ for vision-based PSC, addressing limitations of static transformer queries\n\n- Adaptive proposals boost Thing-metrics by ​​18.65% avg​​ and PQ-All by ​​3.62%​​ versus random initialization\n\n- Clear exposition, but the writing flow could be improved for easier understanding.\n\n**Weaknesses**\n\n- Struggles with rare classes (e.g., Bicyclist: 0.95% in Table 6) due to training data bias?\n- High memory usage (52.8GB vs. 17.3GB for OccFormer, Table 7) limits accessibility. The proposed method is not that easy to train, advanced GPU such as A100/H100 is required.\n- ​​Dependency on 2D models​​: Depth estimation (MobileStereoNet) and features (EfficientNet) may propagate errors to 3D. I am not sure if the depth estimation is overfit to KITTI datset, and wonder the performance on real-world data (zero-shot generalization?)\n\nQUESTIONS:\n- How would IPFormer handle ​​outdoor scenes​​ with varying lighting/weather? Could visibility sampling fail for heavily occluded objects? How about the zero-shot generalization ability on in-the-wild data?\n\n- Why use ​​dual-head training​​ instead of end-to-end joint optimization? Does stage-wise training limit cross-task synergy?\n\n- No comparison with ​​PanoSSC​​ (vision-based PSC SOTA) due to inaccessible dataset (Sec 4.1). How would IPFormer fare under their relaxed IoU threshold? How's the progress now as the authors claim that this will be added to the final version of the paper?\n\n- Can the approach scale to ​​higher resolutions​​ (e.g., 512^3 voxels) without prohibitive memory costs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IPFormer, a 3D vision transformer that utilizes context-adaptive instance proposals to address 3D panoptic scene completion. The two-stage training strategy is utilized to further improve the completion performance. Experimental results on the SemanticKITTI dataset with other state-of-the-art approaches showcase the superiority of their proposed model.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths:\n\nThe main insight of this work is the usage of context-adaptive instance proposals. The proposed IPFormer can effectively address the vision-based 3D panoptic scene completion task, mitigating limitations in dynamically adapting specifically to the observed scene.\n\nPaper Weaknesses:\n\n1.\tThe insufficient experimental details bring confusion. The detailed calculation of PQ is provided in the supplemental material. However, there is no definition for PQ^{\\dag}. The authors should provide detailed descriptions for each evaluation metric and clarify the differences between them. Additionally, the usage of GPU memory during the training and the total training time of IPFormer are unclear.\n2.\tMore ablation experiments should be conducted to examine the sensitivity of semantic scene completion performance to hyperparameters. For example, the weights of different losses in Eq.(8) and Eq. (9).\n3.\tIn Table 1, quantitative comparisons are reported in terms of PQ^{\\dag, PQ, SQ, and RQ on the SemanticKITTI validation set. How about the performance on the SemanticKITTI test set? The quantitative results of each approach regarding Thing are poor and lack sufficient descriptions.\n\nQUESTIONS:\nTiny Weakness:\n\n1. The authors should enlarge the text in the figures since the current font size is too small.\n\nThe paper utilizes context-adaptive instance proposals to achieve high-quality semantic scene completion. Nevertheless, insufficient experiment details, ablation experiments, and visual comparisons weaken its effectiveness and superiority. More visualizations should be included for a comprehensive comparison. Hence, I vote for borderline reject in this review round. I would like to raise the rating if my concerns are adequately addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}