IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
Authors: Markus Gross
Aya Fahmy
Danit Niwattananan
Dominik Muhle
Rui Song
Daniel Cremers
Henri Meeß
Venue: NeurIPS 2025 poster ()
Keywords: Segmentation
Panoptic Scene Completion
Semantic Scene Completion
Scene Understanding
3D Computer Vision
Primary area: applications
TL;DR: IPFormer introduces context-adaptive instance proposals for vision-based 3D Panoptic Scene Completion, outperforming the state-of-the-art and reducing runtime by over 14x.

Abstract: Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Extensive experimental results show that our approach achieves state-of-the-art in-domain performance, exhibits superior zero-shot generalization on out-of-domain data, and achieves a runtime reduction exceeding 14$\times$. These results highlight our introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion. Code available at https://github.com/markus-42/ipformer.

IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals
Markus Gross 123,'
Aya Fahmy'
Danit Niwattananan'
Dominik Muhle
Rui Song'
Daniel Cremers
Henri MeeB'
Fraunhofer Institute IVI
Technical University of Munich
Munich Center for Machine Learning University California, Los Angeles
Abstract
Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Com- pletion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based camera images remain largely unexplored. Moreover, recent Transformer- based approaches utilize a fixed set learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability dynamically adapt specifically the observed scene. To overcome these limitations, we propose IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively ini- tializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-vort voxel relationships. Extensive experimental results show that our approach achieves state-of-the the-art in-domain performance, exhibits superior zero- shot generalization on oft-of-domain data, and achieves runtime reduction ex- ceeding 14x. These results highlight our introduction of context-adaptive instance proposals as a pioneering effort addressing vision-based 3D Panoptic Scene Completion. Code available at https: // htt//gitttt /gitttu.. com/ hommmarkus- 42/ //fffffaa ormer.
Introduction
Panoptic Scene Completion provides holistic scene understanding that can serve applications like autonomous driving and roboties by reconstructing volumetric geometry from sparse sensor data and assigning meaning objects in the scene. Such holistic scene understanding requires both accurate geometry and semantic information about a scene. Historically, these tasks have been treated separately and evolved independently due to the distinct nature of geometric reconstruction and semantic interpretation, each requiring specialized algorithms and data representations. Despite the inherent interconnection between geometry and semantics, their separation hindered unified scene understanding. Classical structure- from motion evolved into simultancous localization and mapping (SLAM) systems for accurate reconstructions [38, 12, 25], while deep learning facilitated monocular depth estimation (MDE) methods [14, 15, 53], which are restricted to visible surfaces. In contrast,
*Corresponiigg author: markus,
markus.grossed gross@ tum.de
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Symphonies
iPFrmrr
DBSGAN
(ours)
Scmartic
Scanne
Panoplic
Completon
Scene
Instance Queries
Instance
Completion
Qustering
Proposals
(contbat adaptiv)
Time
A/r Adaptive Test
Figure 1: Comparison of query initialization for Panoptic Scene Completion (PSC). Existing methods, e.g. Symphonies [21], randomly initialize instance queries and incorporate context-awareness during training. However, these queries retain their static nature test time, as they are shared across all inputs. Our method IPFormer initializes them as instance proposals, which preserve context- adaptivity at test time, effectively aggregating directed features for improved PSC performance. Due to the the novelty of vision-based PSC and the absence of established baselines, we apply DBSCAN [13] clustering to Symphonies' SSC output to retrieve its individual instances.
Scene Completion refers the process of inferring the complete 3D structure of a scene, including both visible and occluded regions, from partial observations [52, 16].
Apart from geometry, semantic image segmentation divides image into semantically meaningful regions and labels them [58, 7]. Panoptic segmentation [23, 22, 41] extends this approach by segmenting instances [17] for things (countable objects like person car) while maintaining semantic segmentation for stuff (amorphous regions like road or vegetation). This unified approach is vital for comprehensive scene understanding, it distinguishes individual object instances while preserving semantic context for background elements, enabling applications like autonomous navigation. The recent trend in scene understanding has been to unify these tasks into Semantic Scene Completion (SSC) [19] and Panoptic Scene Completion (PSC) [3]. SSC integrates geometric completion with semantic labeling, predicting both the 3D structure and semantic categories, represented grid composed of voxels. PSC builds on this by adding instance-level segmentation for each voxel, thus distinguishing between individual objects. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer based approaches utilize fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to observed scene.
3D
To address these limitations, we introduce IPFormer, a 3D Vision Transformer designed to address Panoptic Scene Completion. IPFormer lifts contextual 2D image features to 3D and samples them based on visibility to initialize instance proposals, each implicitly representing semantic instances within the camera view. Based on this confext-adaptive initialization, we establish a robust recon- struction signal that enhances subsequent encoding and decoding stages to reconstruct a complete panoptic scene. Sampling instance proposals solely from visible surfaces is grounded in the principle that all potentially identifiable objects within the observed scene must exhibit a visual cue in the camera image to facilitate completion. This approach directed feature aggregation at train and test time significantly improves instance identification, semantic classification, and geometric completion.
Our contributions can be summarized follows:
We present IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion.
We introduce a visibility-based sampling strategy, which initializes instance proposals that dynamically adapt scene characteristics, improving PQ-All by 3.62 % and Thing- metrics on average by 18.65 %, compared to non-adaptive initialization.
Our experimental results demonstrate that IPFormer achieves state-of- of- ff--hhe-n performance on in-domain data, exhibits superior zero-shot generalization on out- oft-of-domain data, and offers a runtime reduction of over 14x, from 4.51 seconds to 0.33 seconds.
Comprehensive ablation studies reveal that employing dual-head architecture combined with a two-stage training strategy, in which SSC and PSC tasks are trained independently and sequentially, significantly improves performance and effectively balances metrics.

---

2
Related work
Semantic Scene Completion. The task of Semantic Scene Completion has seen a large amount of interest after it was first introduced in [44]. The initial work focused solving the task for indoor scenes from camera data [60, 61, 35, 29, 28, 27, 8, 2] and used point clouds in outdoor scenarios with LiDAR-bbaed methods [45, 9, 30, 43]. In the following,
will discuss the most relevant contributions for Semantic Scene Completion in the context of autonomous driving. For a more thorough overview, we refer the reader to the surveys of [56] and [62].
Recent progress in perception for autonomous driving has increasingly adopted Vision Transformer architectures [11], notably DETR [5]. DETR introduced learnable object queries, which are trainable embeddings that allow the model to directly predict object sets in parallel, eliminating hand-crafted components like anchor boxes. In SSC, learnable queries can be adapted to predict semantic labels for 3D scene elements (e.g., voxels), streamlining the process and enhancing efficiency. Architectures like OccFormer [63] and VoxFormer [32] exemplify this trend, with OccFormer using multiscale voxel representations with query features to decode 3D semantic occupancy. and VoxFormer employing depth-based queries fused with image features, refined via deformable attention. Similarly, Symphonies [21] refines instance queries iteratively using cross-attention between image features and 3D representations, while TPVFormer [20] leverages rri-peespective view (TPV) to query scene features efficiently. HASSC [50] and CGFormer [59] further refine this general approach by focusing on challenging voxels and contextual queries, respectively. Alternative approaches, such as implicit scenes via self-supervised learning in S4C, [19] and CoHFF [48], which leverages both voxel and TPV representations to fuse features from multiple vehicles, have also emerged, though they do not utilize learnable queries.
It is worth noting that existing literature presents ambiguity regarding context- awarnness of queries for SSC. Although both Symphonies [2i] and CGFormer [59] incorporate contextual cues during training, their approaches diverge at inference. In particular, Symphonies' instance queries remain static and are shared across all input images, whereas CGFormer's voxel queries preserve context- awareness by dynamically adapting to each input. To clarify this distinction, we differentiate between instance queries and instance proposals as context-aware and context-adaptive, respectively. Our introduced instance proposals effectively adapt scene characteristies guide feature aggregation during both train and test time.
Panoptic Scene Completion. While panoptic LiDAR segmentation, which predicts panoptics for point clouds directly, has been widely studied [6, 55, 54], voxel-based PSC approaches have only recently been introduced by PaSCo [3]. This method employs hybrid convolutional neural network (CNN) and Transformer architecture with static instance queries to address LiDAR-basee PSC, enhanced by an uncertainty- framework. In parallel, camera-based PSC from multi-frame and multi-view (surround-view) images is addressed by PanoOcc [51], which utilizes static voxel queries to aggregate spatiotemporal information, while SparseOcc [34] introduces sparse voxel decoder and sparse static instance queries to predict semantic and instance occupancy from up to 96 temporal frames. Concurrently, PanoSSC [47] introduces a camera-based PSC method using forward-facing, non-temporal imagery. Its dual- head TPV-based architecture separates semantic occupancy and instance segmentation tasks to infer a 3D panoptic voxel grid. Likewise, our method targets PSC using neward-facing non-temporal imagery. However, it overcomes key limitation of prior approaches by avoiding reliance on static queries and instead introduces context-adaptive instance proposals that dynamically adapt to the scene at both train and test time. Notably, PanoSSC [47] is trained on a non-public post-processed version of the SemanticKITTI dataset [1], which, in addition to the absence of released source code, challenges direct comparisons.
3 Methodology
3.1
Overview
Given an input image X E RU and the depth net predicts a depth map D E RUXVx1 following VoxFormer [32]. Further refining the image features and the depth map allows for generating context-aware 3D representations, denoted as 3D context features F 3D and initial voxels V. Subsequently, visible and invisible voxels, sampled from V using the depth map, attend to the 3D image features FaD via series of 3D deformable cross
(Fig. 2), the backbone predicts image features F E R
3

---

Context Generation
Proposal Initialization
X
Contest Net
3D Deformable Self Alff-Allenion
Proposal Generrtoo
Backbone
Death Extimations
F
F
$30
ViS
20
Deformabie
Defo Self -atenit
Retinement
Ltting
Vosel Sampling
Attention
Vincts
Encoding
Decoding
3D ozai and Grobal Encoder
Cross- Altention
Setf. Attertion
MLL
Panoptic Head
PSC Aggregation
PBC
Leamaale
v
NoT
Lenrratie
Alggmmet
Haad
Figure 2: Detailed architecture IPFormer. Our method refines image features and a depth map to produce 3D context features, which sampled based on visibility to generate context-adaptive instance and voxel proposals. In a two-stage training strategy, voxel proposals first handle Semantic Scene Completion, guiding the latent space toward detailed geometry and semantics. The second stage attends instance proposals over the pretrained voxel features to register individual instances. This dual-head design aligns semanties, instances and voxels, enabling robust Panoptic Scene Completion.
and self-attention mechanisms. This process initializes instance proposals Ip and voxel proposals Vp. In this setup, every instance proposal can eventually correlate with a single instance within the observed scene.
The voxel proposals Vp p are encoded by 3D local and global encoding, in line with CGFormer [59], resulting in voxel features Vy. We follow two-stage training strategy in which the first stage uses these voxel features F to address SSC exclusively. This approach effectively guides the latent space towards semantics and geometry. The second training stage targets overall PSC, thereby detecting individual instances. This achieved by attending the instance proposals I over the pre-trained voxel features from the first stage, resulting in instance features Ip. These features are used to (i) predict semantics, and (ii) align instances and voxels to aggregate a complete panoptic scene.
3.2 3D Context Generation
Following CGFormer [59], we refine the depth map D and the feature map F RxxxF into per-pixel, frustum-shaped depth probabilities DR E IR HxW xD effectively resulting |D] depth bins per pixel. This is achieved by applying &D a mixture of convolution and neighborhood cross-attention layers [18], formulated as DR @ Fzu). Moreover, the generic image features F are projected into contextual features F2D R @xxxxxx through the context net cc. denoted as F2D = (F).
The next step is to lift context features F2D to 3D, following the design of [40]. To elaborate, we lift them to (i) 3D context features FSD and (ii) 3D initial voxels V. To achieve this, we distribute FND along rays defined by the camera intrinsics and weighted by the depth probability distribution D This is formulated as
(u,
FIo(u, v, Du(u, v, v,
(1)
where (u, v) are pixel coordinates, d indexes the depth bin, and is the feature channel dimension. This approach distributes feature vectors across depth bins according to their corresponding probability distributions and thus enables effective lifting of 2D features into probabilistic 3D volume, while preserving both contextual and spatial information. Note that Eq. (1) does not include learnable parameters.
To additionally create discretized initial voxels V RXXXXXxC we voxelize Fap to a regular grid. Let S = {(u, dd)QQ(u, v. d)) = ()), then
V(I
Σ Fx)(u,v,d)
(u.E,)ES
where quantizes continuous coordinates to discrete voxel indices, and C represents the feature di- mension. The summation aggregates all features that map to the same voxel, effectively accumulating

---

iPFormer with Instance Queries (context-owsre)
iPFormer with Instance Proposals
Ground-Truth
(cortext-adaplvv)
Person
Bicycist
Figure 3: Instance-specifie saliency. Through gradient- based attribution, we derive saliency maps that highlight image regions green, where an individual instance mainly retrieves context from. Our introduced instance proposals effectively adapt to scene characteristics by guiding feature aggregation, substantially improving identification, classification, and completion. In contrast, instance queries sample context in an undirected manner, causing misclassification and geometric ambiguity.
evidence for stronger feature representations. Building upon the concept of [59], add learnable embeddings to that enhance its representational capacity.
3.3 Proposal Initialization
Voxel Visibilityy The initial voxels V are categorized into visible voxels Vvis and invisible voxels V 'invis. following established SSC works such as [32]. This achieved by first applying inverse projection II=1 on the depth map D with camera intrinsics K IR This process generates pseudo point cloud by unprojecting a pixel (u, v) to corresponding 3D point (z, y, 2). Furthermore, let M IR Xxxx be binary mask derived from this point cloud that filters the voxel features based on occupancy. Thus, the visible voxel features are defined as Vis = V @ M, where M(x,, y, 2) = 1 if (x, y, 2) corresponds to unprojected point, and 0 otherwise. The operator @ denotes the Schur product. Consequently, the invisible voxel features are given by Vinvs = V (1-M) with all-ones matrix X Note that this sampling does not include learnable parameters.
To further adapt Vvis with image context, we apply a tailored series 3D deformable attention mechanisms, inspired by [21, Specifically, apply 3D deformable cross-attention (DCA) to Vvi (as queries) and the 3D context features F3D (as keys and values), resulting updated visible voxel features Vx
More precisely, an updated voxel query Vx vis at 3D location x computes
vis
V DCA(V vis: F3D, x) = Σ OF T(x) + Ax),
where II(x) obtains the reference points, Ax R denotes the estimated displacement from the reference point and v-(-) refers to trilinear interpolation applied to sample from the 3D context features F3p. The index loops through the sampled points out of total of N points, A, E [0. 1] represents the attention weights, and W signifies the transformation weight. We present only the formulation of single-head attention and we utilize multiple layers of deformable cross-attention.
Instance Proposals. The visible voxels V vis further processed to initialize context-adapted instance proposals Ip. Recall from Sec. that instance proposals are initialized at this stage of the architecture, because in principle, all potentially detectable instances within the observed scene must exhibit visual cue in the camera image to facilitate their completion, thereby defining panoptic scene completion. Following this line of thought, we first apply 3D deformable self-attention (DSA) on V to foster global context-aware attention over visible voxels. This operation, for query located at x, is expressed as
N
DSA(V% Viiss x)
Σ
An W WV vis; Ax)
(4)
n=1
5

---

Inspired by CGFormer's 's [59] query generator, our instance generator transforms the DSA output to initialize instance proposals via
1 DSA(V Vist Vis: x) - W E E
(5)
where Wi represents learnable embeddings to improve the representational capacity, and K denotes the maximum number of detectable instances. This initialization typically directs feature aggregation to highly relevant, instance-dependen image regions, shown in Fig. 3.
Voxel Proposals. To initialize voxel proposals Vp, we first merge the visible and invisible voxels through element wise addition along dimension C, denoted as Vis @ Vinvis We then apply 3D deformable atffntttentoon to distribute updated context information over the entire scene volume, formulated as Vp = DSA(V + iinis), which is in line with [32, 21,591 59]:
3.4 Encoding
The purpose of this part of our architecture is to (i) transform voxel proposals Vp into voxel features Vr F that encode semantic information and (ii) project this semantic information onto the instance proposals Ip, thus prompting semantic instance-voxel relationships.
To encode semantic information, we propagate the voxel proposals Vp through 3D Local and Global Encoder (LGE) based on CGFormer [59]. This encoder enhances the semantic and geometric discriminability of the scene volume by integrating TPV [20] and voxel representations, effectively capturing global and local contextual features, respectively. For in-depth discussion, direct readers to [59]. The 3D LGE & GG transforms voxel proposals Vp into semantic voxel features Vr X x2 formulated Vp (V.)r(Pp). Building on these semantically encoded voxel features, we apply cross-attention (CA) to instance proposals Ip E R XxC (as queries) and semantic voxel features Vr E xxxxxxxXC (as keys and values), producing updated instance features iÇA. Specifically, the updated instance features are computed as:
IS CA(IP, VF) Σ AVWWv((P))
(6)
where N X Y .Z denotes the total number of voxels, Pn represents the n-th voxel position, A, E [0, is the attention weight, and W is the learnable projection weight.
Subsequently, we apply self-attention (SA) to the updated instance features IÇA to capture global dependencies among the K instances, yielding further refined instances ĪŞA The self-attention mechanism is defined as:
K Σ A, WiCA(A))).
Ī = SA(TAA
where (n) denotes the feature vector of the n-th instance from I To obtain the final encoded semantic instance features, further process the self-attention output Īę. through an MLP Encoder & Eg yielding I TRKxx This step refines the instance representations for downstream tasks. The encoding is computed as Ip (ITA)) Kxx
3.5 Decoding
Since we employ a two-stage training scheme (see Sec. 3.6), the goal of decoding is twofold. In the first stage, a lightweight MLP head éssc predicts the semantic scene via dssc FF)).
In the second training stage, lightweight panoptic head decodes instance features 1p semantic class logits I E Io X where K is the number of instances and L is the number of semantic classes. These logits are further processed with voxel features V F to perform instance-voxel alignment that eventually enables panoptic scene aggregation, inspired by PaSCo [3]. To elaborate, we align the instance features I, with the voxel features Vp TRXxY'XZxx to compute affinity
scores H E X-XY x by applying the dot product, resulting in H V II. These scores are converted to predicted probabilities via Hp sigmoid(H), where Hp denotes the model's confidence in each voxel belonging each instance. For panoptic aggregation [3], voxel-wise logits

---

VL XXXxXZZ are computed as VL = Hp Iz. It should be emphasized that voxel-wise logits do not represent the probability of belonging to certain semantic class, but the confidence of belonging to one of the instances. Furthermore, semantic class IDss are assigned via argmax over V.. along T, and instance IDs via argmax over Hp along K, yielding a complete panoptic scene. Note that the alignment process and the panoptic aggregation do not include learnable parameters.
3.6 Training Objective
We follow two-stage training strategy in which we formulate the learning objectives for SSC and PSC tasks sequentially. Technically, the first stage optimizes
Lssc = Ace + Ateeth Seeph + Avem csem
Ageo C
(8)
where the weights for the cross-entropy loss, semantic scene- class affinity loss (SCAL), and geometric SCAL [4] are set to Ace = Asem= Ageo- = 1, respectively, and the weight for the depth loss [59] is set
to Adopth = 10-4.
The second stage calculates losses mask-wiss by comparing each ground-truth mask with the best matching predicted mask. Consistent with previous research [3, 47], we find the best match by applying the Hungarian method [26] to perform bipartite matching [5], using IoU threshold of > 50%. This threshold follows the standard convention established in prior work on panoptic segmentation [23, 3], and the Hungarian algorithm ensures globally optimal, permutation-innaaiaan one-to-one assignment. After matching, the second stage optimizes
Lpsc = Amask Cmakk
+ Adice + Afocal Crocal + Adepth Edopth
where Cmusk represents mask-wise cross-entropy loss, Laice is the dice loss of [37] and f is the focal loss based on [33]. In practice, we set the weights of these losses to Amask = Adice = 1 and
Arocal = 40, respectively, following PaSco [3]. Moreover, Ldepth is defined identically to that in Lssc.
4
Experiments
4.1
Quantitative Results
Dataset and Baselines. We conduct our experiments by (i) in-domain training and testing the SemanticKITTI SSC dataset [1], and (ii) out-of-domaan zero-shot generalization on the distinct goound-truthy for both datasets are provided by PaSCo [3]. Given the novelty of vision-based PSC, no established baselines currently exist for direct comparison. Furthermore, recall from Sec. 2 that PanoSSC [47] represents an existing vision-based PSC method using forward-facing imagery. This would make it a suitable baseline for comparison. However, this approach is trained on a non-public post-processed version of the SemanticKITTI dataset [1], which, in addition to the absence of released source code, challenges direct comparison with our method (see See. A.6 of the technical appendix for further details). Consequently, we adapt the latest state-of-the-art vision-based SSC methods [4, 21, 63, 59] and generate instance predictions by clustering Thing-classes from their outputs, in line with PaSCo [3], which introduced the task of 3D PSC. In particular, to ensure fairness and methodological rigor, we apply DBSCAN [13], the same Euclidean clustering algorithm used to construct the PSC ground truth PaSCo. Furthermore, we utilize pre-trained checkpoints for the baselines, acquired from the official publicly available implementations. We provide further details Sec. A. of the technical appendix.
SSCBench- -SCBench-KTTTT360 [31]. The instance
In-Domain Performance. In summary, IPFormer exceeds all baselines in overall panoptic metrics PQ and PQ', and achieves best or second-best results on individual metries, as shown in Tab. 1. These results showcase the significant advancements IPFormer brings to Panoptic Scene Completion, highlighting its robustness and efficiency. Moreover, IPFormer attains superior performance on RQ-AII, while securing the second-best position in SQ-All and PQ-Thing. Furthermore, although our approach places second in SQ-Stuff, surpasses existing methods in RQ-Stuff and notably PQ-Stufff indicating superior capability in recognizing Stuff- Stuff-classes accurately. Notably, despite our method achieving state-of-the-ar performance in PSC, it exhibits moderate SSC performance on in-domain data. These results are evaluated on the PSC output of our two-staged, fully-trained model. For SSC metrics, the instance IDs of all voxels are disregarded, effectively reducing PSC to SSC. Additionally, our method directly predicts full panoptic scene, resulting in significantly superior runtime of
7

---

Table 1: In-domain performance on SemanticKITTI val. set [1]. Best and second-best results are bold and underlined, respectively. Due to the absence of established baselines for vision- based PSC (see Sec. 4.1), we infer state-of tfrtt--rrtt SSC methods and apply DBSCAN [13] to retrieve instances.
PSC Metries
SSC Metrics
Thing SQF
Stiff SQI
Method
PQ T
PQT
SQT
RQI
PQ1
RQI
PQ
RQ1
loUt
mloU
Runtime [4]
MonoScene [4] DBSCAN
10.12
3.43
15.15
5.33
0.51
7.36
0.87
5.56
20.91
8.57
36,50
11.31
4.51
Symphonies [2]] DBSCAN
11.69
3.75
26.09
5.95
1.07
27.65
1.76
5.70
24.95
8.98
41.92
15.02
4.54
Ocdfoemee [63] DBSCAN
1.15
4.70
OGFommer [59] DBSCAN
14.39
6.16
48. 14
9.18
2.20
44.46
3.47
9.03
50.82 82
13.86
45.98
4.70
iPFomer (ours)
14.45
6.30
41.95
9.75
2,09
42.67
3.33
9.35
41.48
14.43
40.90
16:33
0.33
Table 2: Out-of-doman zero-shot generalization performance of IPFormer and the closest base- line DGFNANer-BBSAAN by training on SemanticKITTI [1] and gross-validating on SSCBench- KITTI360 test set [31]. IPFormer demonstrates superior absolute and relative generalization perfor- mance across PSC and SSC metries.
PSC Metrics
SSC Metries
All
Thing
StulT SQ1
SQ"
RQT
SQT
RQ1
PQ
RQ1
mlol11
SemandicIITI
CGRemer [59] DRSCAN
14:39
0.48
9.47
50.82
13.56
IPFomme (ouES)
14.45
6.30
41.95
9.75
2.09
42.67
3.53
9.35
41.43
14:43
40.90
15:33
KITT1.360
CGFomee [59]
DBSCAN
8.46
1.08
17.52
1.87
0.53
20.06
0.96
1.48
16.
2.54
28. 11
0.04
IPFonter (0m23)
9.41
1.23
24.08
2.16
0.52
22.76
0.95
1.68
25. 89
2.93
28.74 74
9.58
Relative
CGFammr [59] DBSCAN
41.37%
52.47%
62.96%
80. 251265
54.81%
72.349
81.67%
44.00%
IPFomee (OUES)
34.8%%
77.85%
46.64%
71.53%
82.03%
37.52%
79.69%
37.81%
0,33 thus providing a runtime reduction of over 14x. In the technical appendix, we further report class-wise quantitative results and a detailed runtime analysis, including memory utilization during training.
Out-of-Domaile Zero-Shot Generalization Performance. Trained on SemanticKITTI [1], we cross-validate our method on the distinct SSCBench-KKTTISSS dataset [31]. Results in Tab. 2 show that IPFormer demonstrates superior zero-shot generalization capability. Especially in comparison to its closest baseline, CGFommer+DBCAAN which it outperforms across PSC and SSC metrics. These results highlight the generalization capability of IPFormer by leveraging out-of-domain image context to robustly initialize context- -dattivv instance proposals, resulting superior SSC and PSC performance.
4.2 Ablation Study
Context-Adaptive Initialization. We compare our introduced context-adaptive instance proposals with context-aware, but non- adaptive instance queries (Tab. 3). While initializing proposals adaptively slightly decreases SQ-Stuff, remarkably increases all Thing-related metrics by 18.65 % on average, with RQ-Things showing the best performance gain of 21.98 %. This insight demonstrates that adaptively initializing instance proposals from image context not only enables us to drastically recognize more objects, but also enables Thing segmentation with substantially higher quality.
Visibility Based Sampling Strategy. We examine our proposed visible-only approach for proposal initialization to a method that employs both visible and invisible voxels (Tab. 4). Formally, these proposals are initialized similar to Eq. (5), specifically as DSA(V* vis 4 V* invis V @ Vinvis, x) + Wr Under our visible-only approach, almost all metrics increase significantly, most notably RQ-Thing and SQ-Thing by 48.00 % and 93.95 %. respectively, thus evidently enhancing recognition and segmentation of objects from the Thing- These findings confirm that visible voxels carry a robust reconstruction signal, which can be leveraged to initialize instance proposals and eventually improve PSC performance.
Deep Supervision. Inspired by PanoSSC [47]. we investigate deep supervision in the form of auxiliary losses, essentially supervising the attention maps of each layer during the encoding the instance proposals (Tab. 5). Our findings show that guiding intermediate layers in such way degrades all metrics. Most notably, when deep supervision not applied, SQ-Thing registers a substantial improvement of 102.61 %.
8

---

Table Ablation on the initial-
Table 4: Ablation on the
Table 5: Ablation on deep supervi- sion for instance encoding during second-stage training.
ization of instance queries vs. instance proposals.
visibility-baseer sampling strat- egy for proposal initialization.
instance queries
visible invisible
visible
deep supervision
w/o
Method
proposals
Method
only
Method
supervision
All PQ"T
All
All
14.80
(- %)
PQ"1
14.85
14.45
(-2.09
%
PQ"1
14.36
(- %)
PQ1
6.08
6.301
3.62 %
PQT
5.86
6.30
7.51
%)
PQ
5.77
6,30
9.19 %)
SQT
41,85 (-1.6
95
-37.33 %)
SQ
++29.04 %c
RQ*
9.36
9.75 4.17 %)
RQI
9.01
9.75
8:21 %%)
RQ
9.06
9.75
7.62 %)
Thing
Thing
Thing
PQ
1.76
2.09
(+1.77
PQT
1.40
2.00
28
PQ*
1.26
2.09
(+665
%6)
SQ1
37.07
42.67(
15.23 %%
sQT
22.00
42 67
993.55 %%
SQ*
21 .6.
42.67
102.61 61 1%)
RQ"
2.73
3:33
(+211.98 96)
RQT
3.33
2.00
3.33
(4-66.80 50 %6
Stuff
Stuff
Stuff
PQT
9:22
9.35 (-1.41 %)
PQT
9.11
9.35
(2.03 %)
PQ
9:06
9.35 (+3.20 [6)
SQT
46.71
41,43
SQT
36.74
41.43 (-12.77 %6
SQ
50.83
(-144441.47
RQ
14.18
14:43 (+1.76 96)
RQT
13.93
14:43
(+3.59 56)
RQT
14.20
14.43 (-1.62 ())
Table 6: Ablation on the dual-head design and the training strategy. Methods (a)-(d) evaluate combinations of single/dual-head and one/two-stage trainings, where denotes frozen weights of the first stage during second-stage training. Methods (c)-(i) examine the incorporation of the first-stage
loss Lssc by different factors of Assc the second stage. (j) represents the final IPFormer config.
Dual
Two
Thing
Stuff
Method
Head
Stage
Assc
PQ"
PQ1
SQT
RQT
PQ
SQt
RQ1
SQT
RQ1
(a)
14.64
6.23
33.37
9.54
1.75
15.40
2.66
9.49
46.44 66.44
14.55
(b)
14.21
5.57
36.09
8.58
1.06
22.29
1.73
8.84
16.12
13.57
10.66
0.42
14.31
0.76
0.13
7.07
0.22
0.63
19.57
1.15
13.75
5.06
38.15
7.98
0.27
27.31
0.49
8.55
46.00
13.43
1.00
14.35
6.06
38.74
9.30
1.66
35.52
2.63
9.27
41.08
14.16
(0)
V
0.50
14.42
6.27
38.72
9.69
1.88
42.01
3.13
9.46
36.33
14.45
(g)
v
0.20
14.46
6.05
33.06
9.37
1.60
28.79
2.54
9.29
36.18
14.34
(h)
0.10
14.41
6.08
33.50
9.40
1.72
29.93
2.73
9.26
36.27
14.26
(1)
0.01
14.67
5.16
24.33
7.97
0.90
7.83
1.43
8.27
36.32
12.73
(
14.45
6.30
41.95
9.75
2.09
42.67
3.33
9.35
41.43
14.43
Dual-head and Two-Stage Training. The baseline (a) in Tab. 6 uses a single head and single-stage training, performing well Stuff but poorly in SQ-Thing. Introducing separate heads for SSC and PSC in (b) slightly reduces Stuff performance but improves SQ-Thing to nearly match the baseline. Replacing the dual-head with purely two-stage approach in (c) severely degrades SQ and yields the worst RQ scores. In contrast, adopting both the dual-head architecture and two-stage training in (i) achieves the best overall results. Alternatively, freezing the first stage during the second-stage training (d) proves detrimental, notably reducing RQ-Thing. Furthermore, methods (c)-(i) evaluate the integration of the SSC objective into the second-stage training, by adding Lssc Lysc via different factors of Assc. Overall, these configurations fall short of the superior performance achieved by our final method (j), which applies both dual-head architecture and two-stage training strategy that separates SSC and PSC objectives. Note that these findings are underscored by additional ablation experiments presented in Sec. A.7 of the technical appendix.
4.3 Qualitative Results
Presented in Tab, 4, IPFormer surpasses existing approaches by excelling at identifying individual instances, inferring their semantics, and reconstructing geometry with exceptional fidelity. Even for extremely low-frequency categories such as the person category (0.07 %) under adverse lighting conditions, and in the presence of arteffatsfacts from dynamic objects in the ground-truth data, our method proves visually superior. These advancements stem from IPFormer's 's instance proposals, which dynamically adapt to scene characteristics, thus preserving high precision in instance identifica- tion, semantic segmentation, and geometric completion. Conversely, other models tend to encounter challenges in identifying semantic instances effectively while simultaneously retaining geometric integrity. Moreover, our instance-specific saliency analysis in Fig. 3 underscores these findings.

---

Symphonies
(aun))
Byoccat
Road
lakiin
Vigeatoo
Figure 4: Qualitative results on the SemanticKITTI val. set [1]. Each top row illustrates purely semantic information, following the SSC color map. Each bottom row displays individual instances, with randomly assigned colors to facilitate differentiation. Note that we specifically show instances
of the Thing-category for clarity.
Conclusion
IPFormer advances the field of 3D Panoptic Scene Completion by leveraging context-adaptive instance proposals derived from camera images at both train and test time. Its contributions are reflected in achieving state-of -frt--rrttt-ttttt- in-domain performance, exhibiting superior zero-shot generalization out-of-domain data, and achieving a runtime reduction exceeding 14x. Ablation studies confirm the critical role of visibility-bon -based proposal initialization, the dual-head architecture and the two-stage training strategy, while qualitative results underscore the method's ' ability to reconstitute true scene geometry despite incomplete imperfect ground truth. Taken together, these findings serve promising foundation for downstream applications like autonomous driving and future research in holistic 3D scene understanding.
References
[1] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and Gall. Se- ManticKITTI: A dataset for semantic scene understanding of LiDAR sequences. In Proc. of the JEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
[2] Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yas Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 324-333, 2021.
[3] Anh-Quan Cao, Angela Dai, and Raoul Charette. Pasco: Urban 3D panoptic scene completion with uncertainty awareness. In CVPR, 2024.
[4] Anh-Quan Cao and Raoul de Charette, MonoScene: Monocular 3D semantic scene completion.
In CVPR, 2022.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 213-2299 Springer, 2020.
[6] Gang Chen, Zhaoying Wang. Wei Dong, and Javier Alonso-Mora. Particle-based instance- aware semantic occupancy mapping in dynamic environments. IEEE Transactions on Robotics, 11:1155-1171, 2025.
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplabb Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE TPAMI, 2017.
10

---

[8] Xiaokang Chen, Kwan-Yee Lin, Chen Qian, Gang Zeng, and Hongsheng Li. 3D sketch-ane aware semantic scene completion via semi superviie structure prior. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4193 4202, 2020.
[9] Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Liu Bingbing. S3CNet: A sparse semantic scene completion network for lidar point clouds. In Conference on Robot Learning, pages
PMLR, 2021.
[10] Peter Christen, David Hand, and Nishadi Kirielle. review of the F-measure: Its history, properties, criticism, and alternatives. ACM Computing Surveys, 56(3):1-24, March 2024.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[12] Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Large-scale direct monocular SLAM. In Proceedings of the European Conference on Computer Vision (ECCV), pages Springer, 2014.
[13] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page 226-231. AAAI Press, 1996.
[14] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In ICCV, pages 270-279, 2017.
[15] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In ICCV, pages 3828-3838, 2019.
[16] Keonhee Han, Dominik Muhle, Felix Wimbauer, and Daniel Cremers. Boosting self superviiiii for single-view scene completion via knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9837-9847, 2024.
[17] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014.
[18] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023.
[19] Adrian Hayler, Felix Wimbauer, Dominik Muhle, Christian Rupprecht, and Daniel Cremers. S4c: Self-supervised semantic scene completion with neural fields. In 2024 International Conference on 3D Vision (3DV), pages 409 420. IEEE, 2024.
[20] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9223-9232, 2023.
[21] Haoyï Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, and Xinggang Wang. Symphonize 3d semantic scene completion with contextual instance queries. CVPR, 2024.
[22] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
[23] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In CVPR, pages 9404-9413 2019.
[24] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.

---

[25] Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. Tandem: Tracking and dense mapping in real-time using deep multi-view stereo. In Conference on Robot Learning, pages 34 45. PMLR, 2022.
[26] Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logisties Quarterly, 2(1-2):83-97, March 1955.
[27] Jie Li, Kai Han, Peng Wang, Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3351-3359, 2020.
[28] Jie Li, Yu Liu, Dong Gong, Qinfeng Shi, Xia Yuan, Chunxia Zhao, and Ian Reid. Rghd based dimensional decomposition residual network for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7693-7702. 2019.
[29] Jie Li, Yu Liu, Xia Yuan, Chunxia Zhao, Roland Siegwart, Ian Reid, and Cesar Cadena. Depth based semantic scene completion with position importance aware loss. IEEE Robotics and Automation Letters, 5(1):219-228) 2019.
[30] Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Semi- supervised implicit scene completion from sparse lidar, 2021.
[31] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang. Zhiheng Li, Tao Jiang, Fisher Yu, Yue Wang, Hang Zhao, Zhiding Yu, and Chen Feng. Sscbench: A lcrge-scale 3d semantic scene completion benchmark for autonomous driving. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
[32] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera- based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9087-9098, 2023.
[33] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In JEEE International Conference on Computer Vision (ICCV), pages 2999-3007, 2017.
[34] Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang, Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. arXiv preprint arXiv:2312.17118, 2023.
[35] Shice Liu, Yu Hu, Yiming Zeng. Qiankun Tang, Beibei Jin, Yinhe Han, and Xiaowei Li. See and think: Disentangling semantic scene completion. In Advances in Neural Information Processing Systems, volume 31, 2018.
[36] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[37] Fausto Milletari, Nassir Navab, and Seyed- Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), pages 565-571, 2016.
[38] Raul Mur- Artal and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147-116; 2015.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances Neural Information Processing Systems 32, 2019.
[40] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.
12

---

[41] Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In CVPR, 2019.
[42] Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 8269-88688888 9269-8278, 2019.
[43] Christoph B Rist, David Emmerichs, Markus Enzweiler, and Dariu M Gavrila. Semantic scene completion using local deep implicit functions on lidar data. IEEE transactions on pattern analysis and machine intelligence, 44110)7205--2188 2021.
[44]
Luis Roldão, Raoul de Charette, and Anne Verroust-Blondet. Lmsenet: Lightweight multiscale 3d semantic completion. International Conference on 3D Vision (3DV), 2020.
[45] Luis Roldao, Raoul De Charette, and Anne Verroust-Blondet. 3d semantic scene completion: A survey. IJCV, 2022.
[46] Faranak Shamsafar, Samuel Woerz, Rafia Rahim, and Andreas Zell. Mobilestereonet: Towards lightweight deep networks for stereo matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2417-2426, 2022.
[47] Yining Shi, Jiusi Li, Kun Jiang, Ke Wang, Yunlong Wang. Mengmeng Yang, and Diange Yang. Panosse: Exploring monocular panoptic 3d scene reconstruction for autonomous driving. In 2024 International Conference on 31D Vision (3DV), pages 1219-1228. IEEE, 2024.
[48] Rui Song, Chenwei Liang, Hu Cao, Zhiran Yan, Walter Zimmer, Markus Gross, Andreas Festag, and Alois Knoll. Collaborative semantic occupancy prediction with hybrid feature fusion in connected automated vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18996-18006, 2024.
[49] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105 14. PMLR, 2019.
[50] Song Wang, Jiawei Yu, Wentong Li, Wenyu Liu, Xiaolu Liu, Junbo Chen, and Jianke Zhu. Not all voxels are equal: Hardness-aware semantic scene completion with self-distillation In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14801, 2024.
[51] Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. Panooce: Unified occupancy representation for sumera-based 3d panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17158-17168 2024.
[52] Felix Wimbauer, Nan Yang, Christian Rupprecht, and Daniel Cremers. Behind the scenes: Density fields for single view reconstruction. In CVPR, 2023.
[53] Felix Wimbauer, Nan Yang, Lukas Von Stumberg, Niclas Zeller, and Daniel Cremers. Monorec: Semi-supervised dense reconstruction in dynamic environments from single moving camera. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6112-6122, 2021.
[54] Binbin Xiang, Yuanwen Yue, Torben Peters, and Konrad Schindler. A review of panoptic segmentation for mobile mapping point clouds. ISPRS Journal of Photogrammetry and Remote Sensing, 203:373-391, 2023.
[55] Zihao Xiao, Longlong Jing, Shangxuan Wu, Alex Zihao Zhu, Jingwei Ji, Chiyu Max Jiang. Wei-Chih Hung. Thomas Funkhouser, Weicheng Kuo, Anelia Angelova, Yin Zhou, and Shiwei Sheng. 3d open-vocabulary panoptic segmentation with 2d-3d vasion-language distillation. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gil Varol, editors, Computer Vision - ECCV 2024, volume 5598 of Lecture Notes Computer Science, pages 23-41. Springer, Cham, 2025.
13

---

[56] Huaiyuan Xu, Junliang Chen, Shiyu Meng, Yi Wang, and Lap-Pui Chau. A survey on occupancy perception for autonomous driving: The information fusion perspective. Information Fusion, 114:102671, 2025.
[57] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3101-3109, 2021.
[58] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 2015.
[59] Zhu Yu, Runmin Zhang, Jiacheng Ying, Junchen Yu, Xiaohai Hu, Lun Luo, Si- Yuan Cao, and Hui-Liang Shen. Context and geometry aware voxel transformer for semantic scene completion. In Advances Neural Information Processing Systems, 2024.
[60] Jiahui Zhang, Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, and Hongen Liao. Efficient semantic scene completion network with spatial group convolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 733-749, 2018.
[61] Pingping Zhang, Wei Liu, Yinjie Lei, Huchuan Lu, and Xiaoyun Yang. Cascaded context pyramid for full-resolution 3d semantic scene completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7801-7810 10, 2019.
[62] Yanan Zhang, Jinqing Zhang, Zengran Wang, Junhao Junhao Xu, and Di Huang. Vision-based 3d occu- pancy prediction in autonomous driving: review and outlook. arXiv preprint arXi::440..0259; 2024.
[63] Yunpeng Zhang. Zheng Zhu, and Dalong Du. Oceformer: Dual-path transformer for vision- based 3d semantic occupancy prediction. arXiv preprint arXiv:230.005316, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims supported by experimental results presented in Sec
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations the experimental setup and the challenges in compar- ing with existing methods due to the lack of publicly available datasets, as mentioned in Sec. 4. 1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfficatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [No]
Justification: We do not include theoretical results that require assumptions or proofs. Our work focuses experimental validation of the proposed method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We provide detailed information on the experimental setup, including datasets, metrics, and baselines (Sec. and technical appendix) and we will release the source code upon publication, which allows for reproducibility of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Our source code will be released upon publication, facilitating reproducibility and further research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
gww.//gu///e//eeeeeeeeeee /aaeee///ooebbiiiiecc asssonooiicyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify training and test details, including data splits, hyperparameters, and optimizer settings in Sec. 4 and the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars or statistical significance tests for the experiments, focusing instead established performance metrics and runtime analysis in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide information on the compute resources, including GPU and CPU type, memory utilization during training, well training and inference time, found in the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttu//eurip.../ cttps.ccp/puee httpppullltiiis/eeeeeieeppp
Answer: [Yes]
Justification: Our research conforms to the NeurlPS Code Ethies, with no special circum- stances requiring deviation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential positive impacts in applications like autonomous driving and robotics, but do not explicitly address negative societal impacts.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We not describe safeguards for data models, as research does not pose high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the datasets and models used, such as SemanticKITTI and PaSCo [3], and respect their terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets that require documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development does not involve LLMs as important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //nnruurpp.... for what should or should not be described.
21

---

A Technical Appendix
A.1 Experimental setup
Datasets. We utilize the SemanticKITTI dataset, a large-scale urban dataset designed for Se- mantic Scene Completion. SemanticKITTI provides 64-layer LiDAR scans voxelized into grids of 256x256x with 0.2m voxel resolution, alongside RGB images of 1226x370 pixel resolution, covering 20 distinct semantic classes (19 labeled classes plus free class). The dataset comprises 10 training sequences, I validation sequence, and 11 test sequences, with our experiments adhering to the standard split [44] of 3834 training and 815 validation grids.
To enable panoptic evaluation, we adopt the PaSCo dataset [3], which extends by generating pseudo panoptic labels. PaSCo employs DBSCAN [13] to cluster voxels of "thing" classes into distinct instance IDs, using distance threshold of e = 1 and minimum group size of MinPts 8. The authors of PaSCo ensure that the pseudo labels are valid by comparing them against the available LiDAR single-scan point- point-wise panoptic ground truth from the validation set. Both the pscudo labels (generated by DBSCAN) and the ground truth are voxelized, and their quality is assessed in regions where both are defined. For more details, including quantitative and qualitative evaluation, see PaSco [3, supplementary material Sec. 8.2]. Since instance labels cannot be derived for SemanticKITTT's hidden test set, perform evaluations on the validation set.
Metries. To evaluate our Panoptic Scene Completion approach, we employ the Panoptic Quality (PQ) metric [24], which combines Segmentation Quality (SQ) and Recognition Quality (RQ). PQ is defined as:
Σ (paeTTP IoU(p. g)
PQ = SQ x RQ
(10)
[TP] + FPP + [|
where SQ Elewierp loU(p.9) and RQ ITP T TP, FP, and FN represent true [TP] positives, false positives, and false negatives, respectively, and IoU is the
TP
SQ measures segmentation fidelity via the average IoU of matched segments, while RQ assesses recognition accuracy as the Fl-score [10]. We compute PQ. SQ. and RQ across all classes, as well as separately for "stuff" (amorphous regions) and "things" (countable objects), to analyze category-specific performance.
The standard PQ metric requires predicted segment match ground-truth segment with IoU > 0.5. However, this strict threshold can be overly conservative for Stuff classes, which typically lack well- defined boundaries. The metric PQ [42] relaxes the matching criterion specifically for Stuff classes. Formally, PQ retains the same formulation as PQ:
Σ E(P loU(p.g) TPP
PQ
(1I)
FP
+ 2 FINN
but relaxes the matching condition used to define true positives (TP'), and thus false positives (FP1) and false negatives (FN'). ). Specifically, for Thing classes, predicted and ground-truth segment pairs (P. g) are matched if IoU(p. (p.g) g) > 0.5, identical to the original PQ definition. For Stuff classes, matches are accepted if loU(p.g) > O, thereby allowing any overlapping prediction to contribute to the metric. This relaxation acknowledges the inherent ambiguity delineating stuff regions and reduces penalties for minor misalignments. As with PQ, we compute PQ jointly across all classes and separately for stuff and thing categorics to enable detailed performance analysis.
A.2 Implementation Details
Training and Architecture. In accordance with [4, 20, 32, 21], we train for 25 epochs in the first stage and 30 epochs the second stage, using AdamW [36] optimizer with standard hyperparameters 31 = 0.9, .2 0.99, and a batch size of 1. We utilize single NVIDIA A100 80GB GPU, adopt maximum learning rate of 10-1, and implement cosine adaptive learning rate schedule decay, with cosine warmup applied over the initial 2 epochs. Our implementation is based on Py PyTorc [39]
22

---

Table 7: Class-wiss quantitative results on SemanticKITTI val. set [1] (best, second-best) with corresponding class frequencies. The asterisk (+) indicates SSC methods, for which the outputs are clustered to identify their instances, as described in Sec. 4.1.
OhhesCCous (AA0
Motoggetee (0 (000))
Silooak (1,339
Builing (nine))
Vegrnation (0L000)
Tratllc Sign (0083)
Car (MES)
Bleyle RLADS
Trick Trukk (039)
Nchick
Pasus dumsy
Bicyelas
Moseeyyllee
Road (3339))
Foue (390))
Trunk Ttark s1818)
Purkigg
Mon
Method
MonoScese [d]*
4.30
0.00
0.00
0.00
0.00
3.00
1.01
0.30
0.00
0.00
0.00
4.35
0.00
0.00
3.43
Symphonies
0.00
0.00
8.00
0.00
0.63
4.88
0.00
3.75
Ocailormer [6.3*
3.51
0.00
0.00
1.00
9.00
58.50
0.00
0.31
0.00
4.32
Culeemer pSp
14.14
0.58
1.14
7M
11
11.0%
9.48
1.14
0.30
35
0
0.63
0.
6:11
[PPemmer (oana
12:43
0.45
0.58
0. 27
1.00
06.25 28
14
0.00
2.28
0.00
13.25
0.15
1.50
H+]
Symphonies
0.00
0.00
63.0
0.00
61.64
25.09
OceFormer
58
0..00
0.00
64
0.00
56.35
0.00
0.00
24.19
CGFemner pa9
68
54.16
57.16
G0
67
8.00
70.37
54.51
05.
62.
43
63.73
66
00
57.34
56.02
4.
046
56
52.
59
37
57.50
50.91
MonoSoene--j
0.00
0.00
0.00
0,00
0.00
5:33
Xymphomes:211
0.00
1:22
5.05
1.11
2.11
9.48
0.85
0.30
0.76
with fp32 backend. Moreover, we operate on 50 % voxel grid resolution of X 128, Y = 128,
Z = 16 and finally upsample to the ground-truth grid resolution of 256 x 256 x 32 via trilinear interpolation. The feature dimension is set to C = 128. Training IPFormer takes approximately 3.5 days for each of the two stages. The second stage training is initialized with the final model state of the first stage, and we eventually present results for the best checkpoint based on PQ Aligning with [32, 21, 59], adopt a pretrained MobileStereoNet [46] to estimate depth maps, and employ EfficientNetB7 [49] as our image backbone, consistent with [63, 59]. Moreover, the context net consists of lightweight CNN, while the panoptic head represents a single linear layer for projection to class logits. The deformable cross and self attention blocks during proposal initialization consist of three layers and two layers, respectively, while 8 points are sampled for each reference point. Finally, the cross and self-attention blocks during decoding each consist of three layers.
To cluster the predictions of SSC baselines and retrieve individual instances, we apply DBSCAN [13] with parameters c = 1 and MinPts=8, in line with the work of PaSCo [3], which provides ground- instances for the SemanticKITTI dataset [1]. The clustering is performed on an AMD BPYC 7713 CPU (allocating 16 cores) with 64GB memory.
Clustering.
A.3 Class- Wise Quantitative Results
In addition to the overall performance on Panoptic Scene Completion in Tab. 1, we report class-wise results in Tab. IPFormer consistently ranks first or second, thereby demonstrating state-of-the-ar performance in vision- based PSC, aligning with our primary findings.
As also shown in Tab. 1, all methods showcase suboptimal performance Thing classes, which arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4.53 % of all voxels. To address this, we employ the Sigmoid Focal Loss (Eq. 12), which down-weights easier examples and focuses on harder-to-llassify ones, particularly rare Thing classes. Sec. A.7 presents and discusses additional ablation results on the Focal Loss to demonstrate its effectiveness. Our proposed method balances performance between Thing and Stuff classes, achieving state-of-the-as art results by prioritizing equitable performance across both categories rather than solely optimizing for Thing classes.
A.4 Additional Qualitative Results.
We provide further qualitative results in Fig. 5, aligning with the primary results in that IPFormer reconstructs and identifies diverse objects of various sizes.
23

---

Monoscene
Symphonies USEEAN
DECONN
(cum)
Dement
Figure 5: Additional qualitative results the SemanticKITTI val. set [I] Each top row illustrates purely semantic information, following the SSC color map. Each bottom row displays individual instances, with randomly assigned colors to facilitate differentiation. Note that we specifically show instances of the Thing category for clarity.
Table 8: Comparison of compute resources during training, and detailed runtime analysis. Addition- ally, we show the PQ metric on SemanticKITTI val. set [1] (best, second- best). Operations are performed on a single NVIDIA A100 GPU with 80 GB memory and an AMD EPYC 7713 CPU (allocating 16 cores) with 64 GB memory. The asterisk (*) indicates SSC methods, for which the outputs are clustered to identify their instances, as described in See. 4.1.
Method
MonoScene [4]"
Symphonies [21]*
OceFormer [63]*
CGForner [59]+
IPFormer (ours)
Inference Time [s]]
0.08
0.11
0.27
0.27
0.33
Clustering Time [s]]
4.43
4.43
4.43
4.43
0.00
Total Runtime [s]]
4.51
4.54
4.70
4.70
0.33
Training Memory [GB].
18 20.00
20.50
17.30
19.10
52,80
PQ"1
10.12
11.69
11.25
14.39
14.45
A.5
Compute Resources and Runtime
In Tab. 8, we provide memory utilization during training and a detailed runtime analysis, in combina- tion with the resulting performance for PQ Our method has the highest memory utilization with 52.8 GB, while OccFormer has the lowest with 17.3 GB. As elaborated in Sec. 4.1, we retrieve PSC predictions for the baselines by clustering their SSC predictions. Consequently, the total runtime for all baselines consists of inference time in addition to clustering time, with the latter being a constant of 4.43 s seconds for all baselines. Since method directly predicts panoptic scene, the total runtime equal to the inference time. Thus, IPFormer exhibits a significantly superior runtime of 0.33 compared the second-hest method in terms of PQ CGFormer, which has total runtime of s. IPFormer therefore provides a runtime reduction of over 14x.
A.6 Limitations
Experimental Results. Experimental quantitative and qualitative results show IPFormer's 's state-of- the-art performance in vision-based PSC. However, there are remaining Thing-classes (e.g. Motorcy- clist) and Stuff-classes (e.g. (ther-Ground] which have not been recognized, due their low class frequency or geometric fidelity. Despite these challenges, we believe that IPFormer's 's introduction of context-aware instance proposals will play a significant role progressing 3D computer vision and specifically Panoptic Scene Completion.
Comparison with PanoSSC. As described Sec. 4.1, we are not able to compare the performance of IPFormer with the vision-based PSC approach of PanoSSC [47], as this method is trained on a post-processed version of SemanticKITTI that not publicly available. We aim collaborate with the authors of PanoSSC to train IPformer on this dataset and evaluate on PanoSSC's 's relaxed
24

---

Table 9: Ablation on IPFormer's PSC and SSC performance under PanoSSC's relaxed 20% IoU matching threshold on the SemanticKITTI dataset.
IoUt Threshold
PQ
PQ-AIl
NQ-AII
RQ-AII
PQ-Thing
SQ-Thing
RQ-Thing
PQ-SSuuf
SQU-iuff
RQ-Stuff
[oll
mloU
20%
15.38
12.74
32.76
30.85
4.31
32,80
9.92
18.88
32.74
46.08
40.90
15.33
50%
14:45
41.95
9.75
2.09
42.67
3.53
9.35
41.43
14:43
40.90
15.33
20 % IoU threshold for matching of ground truth and predicted instances, to present representative evaluation results. However, in Tab. 9, we provide evaluation results of IPFormer under PanoSSC's relaxed 20 % loU threshold. Expectedly, RQ metrics increase substantially, since more instances are recognized, while these are segmented with less fidelity. Thus, SQ metrics decrease.
A.7 Additional Ablation Experiments
Tab. 10 presents extensive additional ablation experiments on the two-stage training, the dual-head architecture, the interplay between SSC and PSC, and sensitivity to hyperparameters. All findings in line and underscore our findings discussed in Sec. 4.
Single-head methods (p, struggle to balance SSC and PSC, with single-stage configuration (p) performing the worst due to its inability to separate semantic and instance-level learning. Single-stage, dual-head methods (a, b) also fall short, as the lack of stage-wise optimization hinders instance registration. Two-stage methods highlight the advantages of stage-wise training: SSCufoeused approaches (e, f, k) excel in SSC but underperform in PSC due to limited adaptation, while PSC- -Si-priori methods (j. n) improve instance registration but compromise semantic consistency balance. IPFormer, method (r), decouples SSC and PSC optimization, achieving strong instance registration and balanced performance across both tasks, with moderate SSC trade-off.
Two-Stage Training and Dual-Head Architecture.
Interplay between SSC and PSC. Across the design space, methods that emphasize SSC (e, f, k) achieve strong semantic scores but degrade PSC performance. especially PQ-Thing, Conversely, approaches prioritizing PSC (j. n, q) boost PQ-Thing or PQ-Stuff at the cost of SSC quality or overall balance. Joint single-head variants (a, b, p. q) further struggle with instance registration overall consistency. In contrast, our dual-head, two-stage method (r) yields the best PQ-All and strong performance across PQ-Thing and PQ-Stuff, with only minor SSC trade-off.
Sensitivity Hyperparameters. We train the SSC task in Stage (Eq. 8) using established hyperparameters for the cross-entropy, Semantic-SCAL, and GeoMetric-SCAL cost functions (Sec. 3.6). All weights associated with these functions are set to 1, consistent with state-of-the-art and established SSC works, such as CGFormer [59], OccFormer [63], and MonoScene [4]. Nevertheless, we investigate the effect of removing the Depth loss, as its impact has not been extensively studied. Furthermore, analyze the effeect of varying hyperparameters of the Sigmoid Focal Loss [33], specifically designed to down-weight easier examples and focus the training process on harder-to- classify examples:
(L = a1 - log(pt)
(12)
where pt is the predicted probability for the true class after applying the sigmoid function, az is the class-balancing weight, and y controls the woightiig of well-classified examples.
While methods (c) and (f) in Tab. 10 achieve the highest SSC performance in terms of mloU and IoU, respectively, their PSC performance deteriorates significantly. A similar trend is observed for method (k), which attains the second-best results for both mloU and IoU. In contrast, the best PSC performance, measured by PQ-Thing, is achieved by method (j). Although this method demonstrates satisfactory SSC performance, it suffers from reduced PQ-Stuff and, consequently, lower PQ-All performance. Moreover, the highest PQ and PQ-Stuff performance achieved by methods (n) and (q). respectively. However, both methods exhibit notable decline in PQ-Thing. Specifically, method (n) experiences substantial reduction SSC performance, whereas method (q) maintains adequate SSC scores.
25

---

Table 10: Further ablation experiments analyzing the sensitivity SSC and PSC performance with respect to the hyperparameters of the objective functions, the two-stage training strategy. For hyperparameters, blue values indicate a difference from our proposed IPFormer configuration (r). For SSC and PSC metrics, bold and underlined values represent best and second-best results, respectively.
architecture, and the
Mage
Stge
NSC Merries
PSC Metries
Deyth
Scm
Goo
CE
Sect
Goo
Desth
CE
DICE
Focal
1-01
PQ'
FQ-AII
PQ-Thiu
PQ-Suuff
Ape
Azd
Mec
Axe
Adust
Aue
Adkc
*bccl
0
Dual
1,00
0.0001
Dusl
000000
1.00
1.000
10,0001
0135
14:35
6.00
Dual
0.0021
1.00
1.00
1.00
0.50
0..00
0.50
0.0001
4.0
1.02
05.00
0.25
2.00
01.62
16.30
14.42
6.27
1.58
3.46
101
10:00011
1:00
1.00
1:10
00.91
18:30
01:00
01:0001
1.00
1.00
03.00
03.33
2.09
13:50
16:20
18:00
6.03
1.101
9:20
10
Dual
0100031
1.00
1.00
1,80
0.31
11.00
0.30
0.00031
-00
1.00
+0.00
0.23
2.00
64.15
16.11
14.41
6.02
1.72
9.20
10
1001
0000001
0..25
11.02
Dual
0.00
0.0003
01.55
14:23
Dual
000001
1.00
1.00
0.50
0.:00
0.50
0000001
2.00
1.00
0.21
2.50
03.07
18.25
14.22
5.50
1.67
9:11
10
Dusl
01:00-1
1.00
110
1.10
0.0001
1.00
2.05
03.18
0..55
2.00
32. 15
7.39
13.10
5.97
1.58
9:18
190
Dual
000003
1,00
1.00
1.00
000003
2.00
1.00
000.00
0.23
2.20
00.45
18.74
14:00
0.74
1.58
5,50
0.0001
0,0001
10.00
Sangle
0.0001
3.00
1.00
33.00
0.25
2.00
12.31
18.80
14.00
6.53
1.75
9.49
Doal
0.0001
1.00
1.00
1,00
0,0001
3.00
1.00
30.01
0.50
2.00
00.30
15.35
14.45
6.30
3.00
9.35
Finally, our proposed IPFormer configuration, method (r), achieves a balanced PSC performance by obtaining the best score for PQ-All and the second-best results for both PQ-Thing and PQ-Stuff, with a moderate trade-off SSC performance.
26


Reviews:
[{"review_id": "kqCfTvv2Ga", "reviewer": "Reviewer_z9JQ", "summary": "This paper introduces IPFormer, a 3D vision transformer that utilizes context-adaptive instance proposals to address 3D panoptic scene completion. The two-stage training strategy is utilized to further improve the completion performance. Experimental results on the SemanticKITTI dataset with other state-of-the-art approaches showcase the superiority of their proposed model.", "questions": "Tiny Weakness:\n\n1. The authors should enlarge the text in the figures since the current font size is too small.\n\nThe paper utilizes context-adaptive instance proposals to achieve high-quality semantic scene completion. Nevertheless, insufficient experiment details, ablation experiments, and visual comparisons weaken its effectiveness and superiority. More visualizations should be included for a comprehensive comparison. Hence, I vote for borderline reject in this review round. I would like to raise the rating if my concerns are adequately addressed.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Paper Strengths:\n\nThe main insight of this work is the usage of context-adaptive instance proposals. The proposed IPFormer can effectively address the vision-based 3D panoptic scene completion task, mitigating limitations in dynamically adapting specifically to the observed scene.\n\nPaper Weaknesses:\n\n1.\tThe insufficient experimental details bring confusion. The detailed calculation of PQ is provided in the supplemental material. However, there is no definition for PQ^{\\dag}. The authors should provide detailed descriptions for each evaluation metric and clarify the differences between them. Additionally, the usage of GPU memory during the training and the total training time of IPFormer are unclear.\n2.\tMore ablation experiments should be conducted to examine the sensitivity of semantic scene completion performance to hyperparameters. For example, the weights of different losses in Eq.(8) and Eq. (9).\n3.\tIn Table 1, quantitative comparisons are reported in terms of PQ^{\\dag, PQ, SQ, and RQ on the SemanticKITTI validation set. How about the performance on the SemanticKITTI test set? The quantitative results of each approach regarding Thing are poor and lack sufficient descriptions.", "was_revised": true, "final_justification": "Thanks very much to the authors' response. Although the experiments of the paper is still limited, which are not easy to address through rebuttal, yet the authors have explained some of the equation and experimental setting details. I have updated the score.", "rebuttal": "We sincerely thank thank you for the insightful comments and constructive suggestions. We provide detailed responses to all the questions raised and aim to address the concerns adequately.\n\n> `W1-1:` Clarify differences between PQ and PQ$^{\\dagger}$.\n\nWe appreciate the effort in pointing this out.\n\nThe standard PQ metric requires a predicted segment to match a ground-truth segment with an $\\textbf{IoU} >0.5$. However, this strict threshold can be overly conservative for Stuff classes, which typically lack well-defined boundaries. The metric PQ$^\\dagger$ relaxes the matching criterion specifically for Stuff classes.\n\nFormally, PQ$^\\dagger$ retains the same formulation as PQ:\n\n\\begin{equation}\n\\text{PQ}^\\dagger = \\frac{\\sum_{(p,g) \\in \\text{TP}^\\dagger} \\text{IoU}(p,g)}{|\\text{TP}^\\dagger| + \\frac{1}{2}|\\text{FP}^\\dagger| + \\frac{1}{2}|\\text{FN}^\\dagger|}  ,\n\\end{equation}\n\nbut relaxes the matching condition used to define true positives ($\\text{TP}^\\dagger$), and thus false positives ($\\text{FP}^\\dagger$) and false negatives ($\\text{FN}^\\dagger$). Specifically,\n\n- for Thing-classes, predicted and ground-truth segment pairs $(p,g)$ are matched if $\\text{IoU}(p,g) > 0.5$, identical to the original PQ definition;\n- for Stuff-classes, matches are accepted if $\\text{IoU}(p,g) > 0$, thereby allowing any overlapping prediction to contribute to the metric.\n\nThis relaxation acknowledges the inherent ambiguity in delineating stuff regions and reduces penalties for minor misalignments.\n\n> `W1-2:` Unclear GPU memory and training time of IPFormer.\n\nWe apologize for any misunderstanding and are pleased to provide a summary of Tab. 7 and lines 521–522 of our manuscript:\n\nIPFormer requires $52.80$ GB of GPU memory during training and approximately $3.5$ days to train each of its two stages. Note that we further elaborate on this topic in Sec. `W2` of the rebuttal to Reviewer [rTop.]\n\n> ` W2 & Q2-2:` More ablation experiments regarding SSC performance and hyperparameters. \n\nWe are thankful for this suggestion and are glad to provide an extensive set of additional ablation experiments in Tab. 1 of the rebuttal to Reviewer [Kgxr]. We kindly direct you to this rebuttal, as further ablation experiments are also of interest to other Reviewers. \n\nSince our primary objective is to address the PSC task, we mainly focus on analyzing the influence of Stage 2 hyperparameters in the context of both the SSC and PSC objectives. To this end, we primarily train the SSC task in Stage 1 using established hyperparameters for the cross-entropy, Semantic-SCAL, and Geometric-SCAL cost functions. All weights $\\lambda$ associated with these functions are set to $1$, consistent with state-of-the-art and established SSC works, such as CGFormer (NeurIPS 2024 Spotlight), OccFormer (ICCV 2023), and MonoScene (CVPR 2022). Nevertheless, we investigate the effect of removing the Depth loss, as its impact has not been extensively studied. The corresponding weight $\\lambda_{\\text{depth}}$ was mistakenly reported as $1$ in our manuscript, but it is actually $0.0001$. We apologize for this and will revise it.\n\nFurthermore, we analyze the effect of varying hyperparameters of the Sigmoid Focal Loss [2], specifically designed to down-weight easier examples and focus the training process on harder-to-classify examples:\n\n\\begin{equation}\n\\text{FL}(p) = -\\alpha_t (1 - p_t)^{\\gamma} \\log(p_t)  ,\n\\end{equation}\n\nwhere $p_t$ is predicted probability for the true class after applying the sigmoid function, $\\alpha_t$ is the class-balancing weight, and $\\gamma$ controls the down-weighting of well-classified examples.\n\nReducing the weights of second-stage SSC losses improves SSC performance but leads to a significant drop in PSC metrics, particularly PQ-Thing. Conversely, increasing the Sigmoid Focal Loss weight enhances PSC performance, especially PQ-Thing, but negatively impacts PQ-Stuff and overall SSC metrics. Methods that balance these weights achieve more consistent PSC results but may sacrifice SSC performance.\n\nFinally, our proposed IPFormer configuration, method (r), achieves a balanced PSC performance by obtaining the best score for PQ-All and the second-best results for both PQ-Thing and PQ-Stuff, despite a reduction in SSC performance. This observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with the findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors of PaSCo note that SSC and PSC metrics are not directly correlated, and the findings of our ablation experiments further support and validate this result. Moreover, since PaSCo is LiDAR-based, our method offers the insight that this phenomenon appears to extend across distinct data modalities.\n\nWe hope this analysis addresses your concerns by demonstrating that our method effectively balances SSC and PSC objectives. The observed trade-offs do not undermine the method's effectiveness or superiority but rather highlight its flexibility in addressing the distinct challenges. If further clarification is needed, we are happy to provide additional details.\n\n> `W3-1:` Quantitative PSC performance on the SemanticKITTI test set.\n\nThanks for the opportunity to clarify on this topic. Since the SemanticKITTI test set is hidden to ensure fair evaluation, prevent overfitting, and maintain leaderboard integrity, instance labels cannot be derived for panoptic scene completion (lines 502-503 in the manuscript). Therefore, it is not possible for us or other researchers to evaluate PSC performance on the test set.\n\n> ` W3-2:` Lack of description on why Thing-classes showcase suboptimal performance.\n\nWe appreciate the detailed observation regarding the suboptimal performance on Thing classes. As shown in Tab. 6 of our manuscript, this issue affects all methods and arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4.53% of all voxels. To address this, we employ the Sigmoid Focal Loss (see Sec. ` W2`), which down-weights easier examples and focuses on harder-to-classify ones, particularly rare Thing classes. We kindly direct the Reviewer to Tab. 1 of the rebuttal to Reviewer [Kgxr], where extensive additional ablation results demonstrate its effectiveness. Our approach balances performance between Thing and Stuff classes, achieving state-of-the-art results by prioritizing equitable performance across both categories rather than solely optimizing for Thing classes.\n\nWe hope this explanation clarifies the concern and encourage asking for further clarification if needed.\n\n> `Q1:` Enlarge text in figures.\n\nThank you for the suggestion. We will enlarge the text in the figures to improve readability in the revised version of our manuscript.\n\n> `Q2-1:` Insufficient experiment details.\n\nWe appreciate the Reviewer's concern regarding the experimental details. We provide experimental details in the main manuscript and the technical appendix. This includes \n\n- Datasets: SemanticKITTI and PaSCo ground-truth, voxel resolution, splits, and DBScan clustering parameters.\n- Metrics: PQ, SQ, RQ definitions and formulations, as well as category-specific evaluations.\n- Model and subnets: Overall architecture and structure of individual sub-networks.\n- Training setup: Hyperparameters, batch size, number of epochs, optimizer, learning rate schedule, and runtime analysis.\n- Hardware requirements: For GPU and CPU\n\nMoreover, we are pleased to provide the following details:\n\n- Software Framework: PyTorch with an fp32 backend.  \n- Baseline Comparisons: We utilize pre-trained checkpoints from the official publicly available implementations.\n\nWe will add these details to the revised manuscript. If there are specific details that the Reviewer feels are unclear, we kindly ask for clarification so that we can address them.\n\n> `Q2-3:` More visualizations should be included for a comprehensive comparison.\n\nWe appreciate the Reviewer's suggestion to include more visual comparisons for a comprehensive evaluation. Unfortunately, as per the rebuttal guidelines outlined in the email from the NeurIPS Program Chairs (dated July 27), it is not possible to provide a global rebuttal, including a single-page PDF containing additional figures. We sincerely apologize for this limitation. \n\nHowever, we kindly direct you to the video provided in the supplementary material of the initial submission, which contains extensive visualizations (starting from 3:52 min.), including more than $75$ camera frames and their corresponding predictions (incl. the full panoptic scene, and instances-only) for all baselines and IPFormer, as well as the ground-truths. Furthermore, starting from 5:36 min., we provide an additional $20$ camera frames and corresponding predictions of IPFormer. \n\nWe value your suggestion and will gladly incorporate additional visual comparisons in the revised version of our manuscript.\n\n> Closing Statement.\n\nWe sincerely appreciate your effort and constructive suggestions, which we will thoughtfully incorporate into the revised version of our manuscript. In light of the new evidence regarding **(1) comprehensive clarifications** on experimental details, evaluation metrics, and performance evaluations, **(2) extensive additional ablation experiments** on two-stage training, the dual-head architecture, and sensitivity to hyperparameters, as well as **(3) additional results on zero-shot generalization** (see Tab. 1 in the rebuttal to Reviewer [rTop]), we are confident that our contributions represent a key advancement in 3D vision research. \n\nWe hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating, as you have mentioned in the review. Should there be any remaining questions, we would be happy to provide further clarification.\n\n---\n[1] Porzi et al., Seamless Scene Segmentation, CVPR 2019\n\n[2] Lin et al., Focal Loss for Dense Object Detection, ICCV 2017"}, {"review_id": "YS6aHfDKKn", "reviewer": "Reviewer_rTop", "summary": "​This paper proposes IPFormer, a novel approach to ​​vision-based 3D Panoptic Scene Completion (PSC)​​ by leveraging ​​context-adaptive instance proposals​​ dynamically derived from image context. The contributions of this paper can be summarized as follows.\n\n- Visibility-based sampling​​ (Sec. 3.3): Initializes instance proposals from visible voxels using camera depth and image features, ensuring dynamic adaptation to scene content at test time.\n\n- Dual-head architecture​​ (Sec. 3.1, Table 5): Separates semantic scene completion (SSC) and PSC tasks via a two-stage training strategy, guiding latent space toward geometry/semantics before instance registration.\n\n- Efficient PSC aggregation​​ (Sec. 3.5): Uses instance-voxel affinity scores to align semantics and instances without learnable parameters.\n\n- State-of-the-art results​​: Outperforms SSC baselines + clustering (e.g., CGFormer + DBSCAN) in PQ (14.45) and PQ-All (6.30) on SemanticKITTI, with ​​14× faster inference​​ (0.33s vs. 4.51s).", "questions": "- How would IPFormer handle ​​outdoor scenes​​ with varying lighting/weather? Could visibility sampling fail for heavily occluded objects? How about the zero-shot generalization ability on in-the-wild data?\n\n- Why use ​​dual-head training​​ instead of end-to-end joint optimization? Does stage-wise training limit cross-task synergy?\n\n- No comparison with ​​PanoSSC​​ (vision-based PSC SOTA) due to inaccessible dataset (Sec 4.1). How would IPFormer fare under their relaxed IoU threshold? How's the progress now as the authors claim that this will be added to the final version of the paper?\n\n- Can the approach scale to ​​higher resolutions​​ (e.g., 512^3 voxels) without prohibitive memory costs?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- To my knowledge (which might be limited), this is the first method to use ​​test-time adaptive proposals​​ for vision-based PSC, addressing limitations of static transformer queries\n\n- Adaptive proposals boost Thing-metrics by ​​18.65% avg​​ and PQ-All by ​​3.62%​​ versus random initialization\n\n- Clear exposition, but the writing flow could be improved for easier understanding.\n\n**Weaknesses**\n\n- Struggles with rare classes (e.g., Bicyclist: 0.95% in Table 6) due to training data bias?\n- High memory usage (52.8GB vs. 17.3GB for OccFormer, Table 7) limits accessibility. The proposed method is not that easy to train, advanced GPU such as A100/H100 is required.\n- ​​Dependency on 2D models​​: Depth estimation (MobileStereoNet) and features (EfficientNet) may propagate errors to 3D. I am not sure if the depth estimation is overfit to KITTI datset, and wonder the performance on real-world data (zero-shot generalization?)", "was_revised": true, "final_justification": "I keep my original rating.", "rebuttal": "We sincerely appreciate your insightful feedback and constructive suggestions. We address the concerns thoroughly.\n\n> ` W1`: Struggles with rare classes.\n\nWe thank you for your detailed observation. As shown in Tab. 6 of our manuscript, this issue affects all methods and arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4.53% of all voxels.\n\nWe kindly direct you to Sec. `W2` of the rebuttal to Reviewer [z9JQ], where we elaborate on countermeasures.\n\n> `W2 & Q4:` High memory usage limits accessibility & scalability to higher voxel grid resolutions.\n\nWe value your concern regarding the high memory usage and its implications on increased voxel grid resolution. We agree that this can be a limitation.\n\nSystem tools report $52.8$ GB memory usage due to CUDA overhead and preallocation, while detailed profiling shows the model itself uses $35$ GB. Unlike IPFormer, baselines like OccFormer do not perform PSC directly and require post-hoc clustering, adding extra computation. Thus, comparing GPU memory across SSC and PSC methods is not an entirely equitable assessment. \n\nNoteably, reliance on advanced hardware such as A100 GPUs has become increasingly common in recent 3D vision research. For instance, FB-OCC (CVPR 2023) [1] and VGGT (CVPR 2025) [2] were trained on 32 and 64 A100s, respectively. In comparison, IPFormer operates on a significantly lower computational budget, positioning it on the more efficient end of the spectrum while still achieving competitive performance.\n\n> `W3:` Dependency on 2D models, error propagation to 3D, and generalization beyond KITTI.\n\nWe appreciate raising this concern. MobileStereoNet3D undergoes robust training to enhance generalizability. The model is pre-trained on the large synthetic SceneFlow dataset [3] (approx. 40,000 samples). Subsequently, the model is fine-tuned on two real-world datasets: DrivingStereo [4] (approx. 181,000 samples) and KITTI Scene Flow 2015 (KITTI2015) [5] (~400 samples). Note that SemanticKITTI is distinct from KITTI2015.\n\nAdditionally, the official MobileStereoNet GitHub repository highlights strong zero-shot cross-dataset generalization capabilities. Moreover, the significant difference in the number of samples among the training datasets argues against overfitting.\n\nFinally, to further mitigate potential 2D-to-3D error propagation, we incorporate a depth refinement network, which enhances the quality of depth predictions before they are used for 3D scene completion. \n\n> `Q1:` Robustness to varying lighting, weather, heavy occlusion, and zero-shot generalization.\n\nWe appreciate the Reviewer’s insightful questions regarding the performance for certain scenarios and its zero-shot generalization ability.\n\n1. **Zero-shot Generalization:** We cross-validate IPFormer on the SSCBench-KITTI-360 dataset [6], which is distinct from the SemanticKITTI dataset it has been trained on. Results in Tab. 1 show that IPFormer demonstrates superior zero-shot generalization capability. Especially in comparison to its closest competitor, CGFormer+DBScan, which it outperforms across all PSC and SSC metrics.\n\n2. **Weather/Lighting Variations:** While KITTI-360 is not intentionally diverse, its larger sample count (12,764 images) introduces more scene variations. As per the NeurIPS Program Chairs’ rebuttal guidelines, we cannot submit a global rebuttal or an additional PDF with figures to visually support our findings. However, this quantitative results on zero-shot generalization demonstrate robustness under varying conditions.\n\n3. **Visibility Sampling:** The visualizations in the manuscript and the supplementary video provide compelling evidence of IPFormer’s ability to handle heavily occluded objects. For instance, starting at 4:30 min. in the video, distant occluded cars with minimal visual cues are precisely recognized and segmented, especially when compared to the baselines.\n\nTable 1: Zero-shot generalization for CGFormer+DBScan and IPFormer, by training on SemanticKITTI and cross-validating on SSCBench-KITTI-360.\n|                 | Method            | PQ$^\\dagger$ | PQ-All | SQ-All | RQ-All | PQ-Thing | SQ-Thing | RQ-Thing | PQ-Stuff | SQ-Stuff | RQ-Stuff | IoU   | mIoU  |\n|-----------------|-------------------|--------------|--------|--------|--------|----------|----------|----------|----------|-----------|-----------|-------|-------|\n| SemanticKITTI ↑  | CGFormer+DBScan  | 14.39        | 6.16   | 48.14  | 9.48   | 2.2      | 44.46    | 3.47     | 9.03     | 50.82     | 13.86     | 45.98 | 16.89 |\n|                 | IPFormer          | 14.45    | 6.3    | 41.95  | 9.75   | 2.09     | 42.67    | 3.33     | 9.35     | 41.43     | 14.43     | 40.9  | 15.33 |\n| KITTI360 ↑      | CGFormer+DBScan  | 8.44         | 1.08   | 17.82  | 1.87   | 0.53     | 20.06    | 0.96     | 1.48     | 16.19     | 2.54      | 28.11 | 9.44  |\n|                 | IPFormer          | 9.41     | 1.23   | 24.68  | 2.16   | 0.52     | 22.76    | 0.95     | 1.68     | 25.89     | 2.93      | 28.74| 9.53  |\n| Relative Gap ↓  | CGFormer+DBScan  | 41.37%       | 82.47% | 62.98% | 80.28% | 75.91%   | 54.89%   | 72.34%   | 83.61%   | 68.15%    | 81.67%    | 38.88%| 44.09% |\n|                 | IPFormer          | **34.88%**   | **80.48%** | **41.19%** | **77.85%** | **75.12%**   | **46.64%**   | **71.53%**   | **82.03%**   | **37.52%**    | **79.69%**    | **29.73%**| **37.81%** |\n\n> `Q2:` Rationale for dual-head training and its impact on cross-task synergy.\n\nTo adequately address these questions, we are pleased to provide an extensive set of additional ablation experiments, which are presented in Tab. 1 in the rebuttal to Reviewer [Kgxr]. Because the ablation results are relevant to other Reviewers as well, we kindly direct you to this rebuttal, where we discuss the ablation setup in detail.\n\nAcross the design space, methods that emphasize SSC (e, f, k) achieve strong semantic scores but degrade PSC performance, especially PQ-Thing. Conversely, approaches prioritizing PSC (j, n, q) boost PQ-Thing or PQ-Stuff at the cost of SSC quality or overall balance. Joint or single-head variants (a, b, p, q) further struggle with instance registration or overall consistency. In contrast, our dual-head, two-stage method (r) effectively decouples objectives, yielding the best PQ-All and strong performance across PQ-Thing and PQ-Stuff, with only a minor SSC trade-off.\n\nThe interplay between PSC and SSC tasks primarily driven by their inherent objectives rather than being significantly influenced by stage-wise training, though slight deviations exist. Notably, this observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with the findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors note that SSC and PSC metrics are not directly correlated, a conclusion further reinforced by the results of our experiments. Moreover, since PaSCo is LiDAR-based, our method offers the insight that this phenomenon appears to extend across distinct data modalities.\n\n> `Q3:` Progress on PanoSSC’s dataset and evaluation on its relaxed IoU threshold.\n\nDespite our efforts to gain access to this dataset, it remains unavailable. Consequently, we have not been able to train IPFormer on this post-processed dataset or evaluate it under PanoSSC’s relaxed $20$% IoU matching threshold. We remain committed to presenting representative evaluation results should access to the dataset become possible in the future.\n\nHowever, in Tab. 2, we provide evaluation results of IPFormer under PanoSSC’s relaxed $20$% IoU threshold. Expectedly, RQ metrics increase substantially, since more instances are recognized, while these are segmented with less fidelity. Thus, SQ metrics decrease.\n\nTable 2: Ablation on IPFormer’s PSC and SSC performance under PanoSSC’s relaxed $20$% IoU matching threshold on the SemanticKITTI dataset.\n| IoU Threshold | PQ$^\\dagger$ | PQ-All | SQ-All | RQ-All | PQ-Thing | SQ-Thing | RQ-Thing | PQ-Stuff | SQ-Stuff | RQ-Stuff | IoU   | mIoU  |\n|---------------|--------------|--------|--------|--------|----------|----------|----------|----------|----------|----------|-------|-------|\n| 20%          | **15.38**    | **12.74** | 32.76  | **30.85** | **4.31**   | 32.80    | **9.92**   | **18.88** | 32.74    | **46.08** | **40.90** | **15.33** |\n| 50%          | 14.45        | 6.30   | **41.95** | 9.75   | 2.09     | **42.67** | 3.33     | 9.35     | **41.43** | 14.43    | **40.90** | **15.33** |\n\n\n> Closing Statement.\n\nWe sincerely appreciate your effort and thoughtful suggestions, which we will carefully incorporate into the revised version of our manuscript. In light of the new evidence regarding **(1) comprehensive clarifications** on the robustness to heavy occlusion, performance on rare classes, and dependency on 2D models, **(2) extensive additional ablation experiments** on cross-task-synergy, stage-wise training, and joint optimization, as well as **(3) additional results on zero-shot generalization**, we are confident that our contributions represent a key advancement in 3D vision research.\n\nWe hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification.\n\n---\n[1] Li et al., FB-OCC: 3D Occupancy Prediction based on Forward-Backward\nView Transformation, CVPR 2023 3D Occupancy Prediction Challenge\n\n[2] Wang et al., VGGT: Visual Geometry Grounded Transformer, CVPR 2025\n\n[3] Mayer et al., SceneFlow: A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation, CVPR 2016\n\n[4] Yang et al., DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios, CVPR2019\n\n[5] Menze et al., Object Scene Flow for Autonomous Vehicles, CVPR 2015\n\n[6] Li et al., SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views, ICLR 2024"}, {"review_id": "MTB8YvDRRE", "reviewer": "Reviewer_8FSR", "summary": "The authors propose IPFormer, a monocular image-based 3D panoptic scene completion method which jointly recovers the complete 3D geometry of a scene while identifying individual objects and their semantics from a single RGB image. To improve upon previous works which learn query proposals during training and fix them at test time, they propose to generate instance proposals from an input image's context at both training and testing, enabling better identification and localization of object instances. Furthermore, a two stage training pipeline is proposed where the semantic scene completion task is first optimized for to learn semantics and complete geometry and then the panoptic scene completion task is optimized to identify individual object instances.", "questions": "**Questions:**\n- The definition of $S$ in line 151 is confusing to me. Why is the quantization of the 3D feature $F_{3D}$ a point coordinate $(x, y, z)$? The point coordinate should only be dependent on $(u, v, d)$ and has nothing to do with the feature $F_{3D}$.\n\n**Suggestions:**\n- In Figure 2, it would be better to have an additional arrow going from the RGB image to depth map to show that the depth is estimated. The current figure makes it seem as if the method is an RGBD method instead of a RGB method.\n- The description of 3D deformable cross attention in Eq. 3 and lines 172-175 could be improved. There is no mention of what *reference* points and *sampled* points are. Furthermore, $\\Pi$ is defined as the projection from 3D to 2D in line 160 but then seems to be redefined to mean something else in line 172.\n- In line 265, I think it was meant to say “Stuff-instances” not “Thing-instances”", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- DETR-style queries, which are learned during training and fixed at test time, have limited representational capacity and ignore input image's context which may hurt generalization. Therefore the proposed dynamically adaptive instance proposals which are generated from the input image context seems like a sound improvement over DETR-style instance queries. This improvement is backed by the ablation in Table 2 which shows consistent improvement in almost all metrics when switching from learned but fixed instance queries to their proposed context adaptive instance proposals.\n- As the proposed method directly predicts instance segmentations, it offers a significant speed up (14x) over the monocular semantic scene completion works which require running DBScan for identifying object instances.\n\n**Weaknesses:**\n- Instance proposal generation is the main contribution of this work yet there is no description of what $g$ is in Eq. 5. It mentions inspiration from CGFormer's query generator, but from my understanding CGFormer’s query generator generates a dense voxel grid of queries and then does a depth-based query proposal. When comparing architectures to CGFormer, these two steps are $V$ from Eq. 2 and $\\tilde{V_{vis}}$ from Eq. 3 in this work, so it's not clear how CGFormer's query generator is adapted to generate $I_{p}$. The technical details of the contribution needed to be more clearly stated.\n- Despite being trained for the panoptic scene completion task, IPFormer barely outperforms CGFormer + DBScan which is not trained for panoptic scene completion. Looking at Table 1 in the main paper and Table 6 in the appendix, this boost in performance really only comes from better RQ in *stuff* categories (e.g., sidewalks, vegetation). Alternatively, CGFormer actually outperforms the proposed IPFormer on segmenting individual object instances of the *thing* categories. Moreover, this small boost in performance comes at a cost of a large drop in performance on SSC metrics as shown in Table 8 of the appendix.\n- Since the work is heavily based off of other works (CGFormer, PaSCo), the methodology section could be written a little more clearly to identify what is from those works and what is unique or a contribution from the proposed approach. For example, the \"voxel visibility\" and \"voxel proposals\" from Section 3.2 are essentially straight from CGFormer but there is no mention of this. Section 3.4 could use a better transition between what part of the encoder is from CGFormer and what is not. For Section 3.5, it is unclear whether this panoptic aggregation is just directly from PaSCo or if there are differences.", "was_revised": true, "final_justification": "After looking over the rebuttals and further discussion with the authors, most of the concerns I had have been addressed by the authors. Therefore, I have chosen to raise my score to borderline accept. While the work has limited novelty, the proposed method boasts a large speed up improvement alongside a moderate performance improvement over existing solutions to camera-based panoptic scene completion, making it overall an interesting work.", "rebuttal": "We are sincerely thankful for your valuable feedback and thoughtful suggestions, as well as for taking the time and effort to thoroughly understand the detailed technical intricacies of our work – we deeply appreciate it.\n\n> `W1:` Clarification on instance proposal generation and its adaptation from CGFormer.\n\nWe thank you for pointing out the need for additional clarity regarding the technical details of our instance proposal generation. We acknowledge that the formulation in our manuscript could have been more explicit and are pleased to provide further elaboration. \n\nCGFormer’s query generator performs two key steps: (1) it generates a dense voxel grid of context-dependent queries, and (2) enhances their representational capacity by adding learnable embeddings. This understanding is derived from CGFormer’s official implementation on GitHub, as the details are not fully described in the paper. In our work, we extend these ideas to a novel domain by introducing the first context-adaptive instance proposal mechanism for PSC. Specifically, we enhance our proposal generator by integrating learnable embeddings to improve its representational capacity. This approach represents a significant advancement in 3D vision research, as no prior work has explored context-adaptive instance proposals for PSC. Technically, our proposal generator $g$ (Eq. 5 in the manuscript) can be more precisely expressed as:  \n\n\\begin{equation}\n\\mathbf{I}_P = g(\\text{DSA}(\\tilde{\\mathbf{V}}\\_\\text{vis}\\^\\mathbf{x},\\tilde{\\mathbf{V}}\\_\\text{vis},\\mathbf{x})) = \\text{DSA}(\\tilde{\\mathbf{V}}\\_\\text{vis}\\^\\mathbf{x},\\tilde{\\mathbf{V}}\\_\\text{vis},\\mathbf{x}) + \\mathbf{W}\\_\\mathbf{I} \n\\end{equation}\n\nTo validate the effectiveness of this approach, we conduct an additional ablation experiment to analyze the performance of context-adaptive instance proposals with and without learnable embeddings. These results, presented in Tab. 1, demonstrate that the inclusion of learnable embeddings consistently improves SSC and PSC metrics, further supporting the value of this design choice.\n\nTable 1: Ablation on IPFormer’s instance proposals wiht and without the addition of learnable embeddings, on the SemanticKITTI dataset.\n| Method          | PQ$^\\dagger$ | PQ-All | SQ-All       | RQ-All | PQ-Thing | SQ-Thing     | RQ-Thing | PQ-Stuff | SQ-Stuff     | RQ-Stuff | IoU   | mIoU        |\n|-----------------|-------------|--------|--------------|--------|----------|--------------|----------|----------|--------------|----------|-------|-------------|\n| w/o embeddings | **14.53**       | 6.04   | 33.26        | 9.43   | 1.73     | 22.37        | 2.68     | 9.18     | 41.19        | 14.33    | 39.9  | 14.8        |\n| w/ embeddings  | 14.45       | **6.30** | **41.95**   | **9.75** | **2.09** | **42.67**   | **3.33** | **9.35** | **41.43**   | **14.43** | **40.90** | **15.33** |\n\n> `W2:` Performance trade-offs between IPFormer and CGFormer across PSC and SSC metrics.\n\nWe appreciate the insightful comments on the performance comparison between IPFormer and CGFormer+DBScan. To address this concern, we want to emphasize the following points:\n\nCGFormer itself does not outperform IPFormer. Its performance gains stem from the addition of DBScan clustering, which enables instance registration within the SSC prediction. For end-to-end solutions like IPFormer, the difference in SSC and PSC performance is primarily influenced by their distinct objectives. This observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors of PaSCo note that SSC and PSC metrics are not directly correlated, a conclusion further supported and validated by our findings. Additionally, since PaSCo is LiDAR-based, our method provides the insight that this phenomenon appears to extend across distinct data modalities.\n\nA more nuanced interpretation of the performance comparison is that CGFormer dedicates its capacity to optimizing SSC, while DBScan’s exhaustive deterministic clustering facilitates instance registration. In contrast, IPFormer addresses PSC in an end-to-end fashion, by dedicating its capacity to both the SSC and PSC objectives. While this leads to performance results that appear less favorable for SSC, IPFormer surpasses CGFormer+DBScan in primary panoptic metrics PQ$^\\dagger$ and PQ-All, performs on par in individual metrics, and achieves a significant runtime reduction of over 14$\\times$.\n\n> `W3:` Clarification of contributions and distinctions from CGFormer and PaSCo in the methodology.\n\nWe are thankful for the careful reading of our manuscript and acknowledge the opportunity to clarify the distinctions between our contributions and prior works.\n\nRegarding the voxel visibility and voxel proposals in Section 3.3, we agree that voxel visibility could be cited more precisely, as it is a well-established process in existing works on voxel-based scene understanding, including CGFormer, OccFormer, Symphonies, and VoxFormer. In contrast, voxel proposals are explicitly stated in line 191 of our manuscript to align with the works referenced therein.\n\nFor Section 3.4, we note in line 197 that the 3D Local and Global Encoder is based on CGFormer. This is the only component of the IPFormer encoder based on prior works, which is why no additional references are mentioned in this section. In the context of Section 3.5, we indicate in line 224 that the panoptic aggregation is inspired by PaSCo. However, our implementation extends this approach to operate on dense voxels without incorporating uncertainty calculations, which represents a key difference from PaSCo.\n\nWe are pleased to provide these clarifications and will incorporate the details into the revised version of our manuscript to improve clarity and attribution.\n\n> `Q1:` Clarification of the definition and role of $S$ in relation to $\\mathbf{F}_{3D}$ and point coordinates.\n\nWe appreaciate your effort to point out the confusion regarding the definition of $S$ in line 151 and are pleased to clarify this aspect of our methodology.\n\nWe confirm that the point coordinates $(x, y, z)$ are determined by the pixel coordinates $(u, v)$ and the depth bin $d$, independent of the feature $\\mathbf{F}_{\\text{3D}}$. The correct notation is therefore given by $S = ${$(u,v,d) \\mid \\mathcal{Q}((u,v,d)) = (x,y,z)$}.\n\nWe will revise the manuscript to ensure that the distinction between spatial quantization and feature aggregation is more explicit.\n\n> `S1:` Suggestion to update Fig. 2 to clarify depth estimation from RGB input.\n\nThanks for this suggestion. We will update Fig. 2 in the revised manuscript to include an arrow from the RGB image to the depth map, clarifying that the depth is estimated, and ensuring to properly convey that our method is RGB-based, not RGBD-based.\n\n>`S2:` Improvement of the description of 3D deformable cross-attention and clarification of reference points, sampled points, and the definition of $\\mathbf{\\Pi}$.\n\nWe thank you for pointing out this potential ambiguity and are glad to clarify.  \n\nIn deformable cross-attention, \"reference points\" are predefined query locations in the target space ($\\mathbf{V}\\_\\text{vis}$), and \"sampled points\" are specific locations in the source space ($\\mathbf{F}_{\\text{3D}}$) around the reference points, selected based on learned offsets to aggregate relevant information.\n\nFurthermore, in our manuscript, $\\Pi$ consistently refers to the projection function, mapping 3D query positions $\\mathbf{x}$ to their corresponding 2D reference points. This projected reference point $\\Pi(\\mathbf{x})$ is a 2D location on the feature map $F_{\\text{3D}}$, which corresponds to the $(u, v)$ coordinates in the image plane. The offsets $\\Delta \\mathbf{p}$ are continuous displacement vectors calculated relative to the projected reference point $\\Pi(\\mathbf{x})$, enabling the model to sample features flexibly from locations in the 3D feature map using trilinear interpolation, which is denoted as $\\psi(\\cdot)$ in Eq. 5 of the manuscript. We will revise the manuscript to clarify this and ensure consistency in the explanation of $\\Pi$.\n\n> `S3:` In line 265, I think it was meant to say “Stuff-instances” not “Thing-instances”\n\nThank you for the careful reading. We agree that the term should indeed be \"Stuff-instances\" rather than \"Thing-instances\". We will correct this in the revised manuscript and appreciate the Reviewer’s attention to detail.\n\n> Closing Statement.\n\nWe deeply appreciate the Reviewer's thorough analysis and exceptional attention to technical detail, as well as the constructive suggestions, which we will thoughtfully incorporate into the revised version of our manuscript. In light of the new evidence regarding **(1) comprehensive clarifications** on our contributions, technical details, and performance trade-offs, **(2) extensive additional ablation experiments** (see Tab. 1 in the rebuttal to Reviewer [Kgxr]), and **(3) additional results on zero-shot generalization** (see Tab. 1 in the rebuttal to Reviewer [rTop]), we are confident that our work represents a significant advancement in 3D vision research. \n\nWe hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification."}, {"review_id": "sv9ZSaK2k3", "reviewer": "Reviewer_Kgxr", "summary": "This paper derives the instance queries from proposal generator with semantic scene completion. With known initialization, the proposed methods out perform without any clustering.", "questions": "1. What is the main contribution to reduce the runtime? No additional DBSCAN computation? \n2. Should we consider the invisible voxel features? This might be occurs another inefficient computation and might mislead at instance-voxel alignment.  What about just drop the invisible voxel features? This might increase the efficiency of whole pipeline. \n3. Could this also comparable in the Occ3d nuScene dataset? Especially for the SparseOcc that I've method the weakness.", "limitations": "Thanks for the contribution in autonomous vision community. \nAs the SemanticKITTI dataset is more difficult setup to improve the task. Author should formulate the problem with this hard condition(such as sparse, single view) or tackling the weaknesses like inefficiency pipeline (compare to the Mask2Former based architecture). But still I concern the problem itself is from the SemanticKITTI dataset not the architecture. So I suggest to demonstrate on the other dataset, and prove the problem is also valid.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths : Better reconstruction quality for the thing-category instances.\n\nWeaknesses : \n1. Inefficient pipeline - The proposed network are working with two stages, one is for learning semantic scene completion to initialize the  instances, and with this IPFormer compute instance segmentation. \n\n2. The problem is from the SemanticKITTI dataset not from the method.  As the ground truth gives misclassification and geometric ambiguity, instance queries might not aligned well as shown in Figure 3. If the method itself is matter, then the author should demonstrate that instance are not distinguished in the other clean dataset( such as Occ3D or nuCraft)\n\n3. Not the first time for the Image only Panoptic Scene Completion. \nSparseOcc* already proposed the Panoptic Scene Completion in Occ3D based on the nuScene dataset.  \nSparseOcc* : Fully Sparse 3D Panoptic Occupancy Prediction, ECCV'24", "was_revised": true, "final_justification": "As authors resolve my concerns and justify motivation especially for the instance proposals, I incline the rate as borderline accept.", "rebuttal": "> `W1:` Inefficient pipeline: Two-stage process for SSC and PSC.\n\nWe appreciate your insightful feedback. To address your concerns, we conduct additional ablation experiments, shown in Tab. 1. We apologize for the limited formatting due to Markdown.\n\nSingle-stage methods (p, q) struggle to balance SSC and PSC, with single-head configuration (p) performing the worst due to its inability to separate semantic and instance-level learning. Single-stage, dual-head methods (a, b) also fall short, as the lack of stage-wise optimization hinders instance registration. Two-stage methods highlight the advantages of stage-wise training: SSC-focused approaches (e, f, k) excel in SSC but underperform in PSC due to limited adaptation, while PSC-prioritized methods (j, n) improve instance registration but compromise semantic consistency or balance. IPFormer, method (r), decouples SSC and PSC optimization, achieving strong instance registration and balanced performance across both tasks, with a moderate SSC trade-off.\n\nWe also improve IPFormer’s efficiency by integrating non-learnable operations, such as panoptic aggregation, noted as efficient by Reviewer [rTop]. Additionally, we use a single intermediate grid resolution, unlike methods like OccFormer and PaSCo, which rely on multi-scale voxel features.\n\nTable 1: Additional ablation experiments. Note that $\\lambda_{\\text{depth}}$ was mistakenly reported as $1$ in our manuscript, but is actually $0.0001$. We apologize.\n|     |Head(s)| 1st Stage |     |     |     | 2nd Stage |      |      |            |     |      |       |      |     |**Results**|       |               |        |          |          |\n|:----|:----- |:-----------|:----|:----|:----|:-----------|:-----|:-----|:-----------|:----|:-----|:------|:-----|:----|:--------------|:------|:--------------|:-------|:---------|:---------|\n|     |       | SSC Losses|     |     |     | SSC Losses |      |      | PSC Losses |     |      |       |      |     |SSC Metrics    |       |PSC Metrics    |        |          |\n|     |       | Depth      | CE  | Sem | Geo | CE         | Sem  | Geo  | Depth      | CE  | DICE | Focal |      |     | IoU           | mIoU  | PQ$^\\dagger$| PQ-All | PQ-Thing | PQ-Stuff |\n|     |       | $λ$        | $λ$ | $λ$ | $λ$ | $λ$        | $λ$  | $λ$  | $λ$        | $λ$ | $λ$  | $λ$   | $α$  | $γ$ |               |       |               |        |          |          |\n| (a) | Dual  | **-**      |**-**|**-**|**-**| 1.00       | 1.00     | 1.00     | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 41.3          | 15.26 | 14.21         | 5.57   | 1.06     | 8.84     |\n| (b) | Dual  | **-**      |**-**|**-**|**-**| 1.00       |1.00      |1.00      | 0.0001 | 1   | 1    | **50.00**| **0.21**| **2.30** | 39.45         | 13.06 | 14.45         | 5.03   | 0.99     | 7.97     |\n| (c) | Dual  | 0.0001     | 1   | 1   | 1   | **1.00**   | **1.00** | **1.00** | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 43.44         | 15.79 | 14.35         | 6.06   | 1.66     | 9.27     |\n| (d) | Dual  | 0.0001     | 1   | 1   | 1   | **0.50**   | **0.50** | **0.50** | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 43.62         | 15.3  | 14.42         | **6.27** | 1.88     | _**9.46**_ |\n| (e) | Dual  | 0.0001     | 1   | 1   | 1   | **0.20**   | **0.20** | **0.20** | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 43.99         | **16.29** | 14.46         | 6.05   | 1.6      | 9.29     |\n| (f) | Dual  | 0.0001     | 1   | 1   | 1   | **0.10**   | **0.10** | **0.10** | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | **44.13**     | 16.11 | 14.41         | 6.08   | 1.72     | 9.26     |\n| (g) | Dual  | 0.0001     | 1   | 1   | 1   | **0.01**   | **0.01** | **0.01** | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 43.94         | 15.04 | _**14.67**_     | 5.16   | 0.9      | 8.27     |\n| (h) | Dual  | 0.0001     | 1   | 1   | 1   | **1.00**   | **1.00** | **1.00** | 0.0001     | 1   | **0.00** | 40.00   | 0.25   | 2.00   | 43.34         | 15.83 | 13.89         | 5.33   | 0.49     | 8.86     |\n| (i) | Dual  | 0.0001     | 1   | 1   | 1   | **0.10**   | **0.10** | **0.10** | 0.0001     | 1   | **0.10** | 40.00   | 0.25   | 2.00   | 42.97         | 15.64 | 14.11         | 5.59   | 1.27     | 8.74     |\n| (j) | Dual  | 0.0001     | 1   | 1   | 1   | **0.03**   | **0.03** | **0.03** | 0.0001     | 1   | 1    | **50.00** | **0.21** | **2.30** | 43.53         | 15.65 | 14.39         | 6.19   | **2.38** | 8.96     |\n| (k) | Dual  | 0.0001     | 1   | 1   | 1   | **0.50**   | **0.50** | **0.50** | 0.0001     | 1   | 1    | **50.00** | **0.21** | **2.30** | _**44.07**_     | _**16.25**_ | 14.23         | 5.97   | 1.67     | 9.11     |\n| (l) | Dual  | 0.0001     | 1   | 1   | 1   | -          | -     | -     | 0.0001     | 1   | **2.00** | 40.00   | 0.25   | 2.00   | 32.45         | 7.28  | 13.92         | 5.92   | 1.58     | 9.08     |\n| (m) | Dual  | 0.0001     | 1   | 1   | 1   | -          | -     | -     | 0.0001     | 1   | 1    | **50.00** | **0.21** | **2.30** | 40.45         | 14.74 | 14.59         | 5.74   | 1.53     | 8.8      |\n| (n) | Dual  | 0.0001     | 1   | 1   | 1   | -          | -     | -     | 0.0001     | 1   | 1    | **45.00** | **0.21** | **2.30** | 36.73         | 11.69 | **15.14**     | 4.43   | 0.78     | 7.08     |\n| (o) | Dual  | **0.0000** | 1   | 1   | 1   | -          | -     | -     | **0.0000** | 1   | 1    | 40.00   | 0.25   | 2.00   |      41.16 | 14.90  |   14.41    |  6.15   |   1.68      |  9.40      |\n| (p) |**Single** |0.0001     | 1   | 1   | 1   | -          | -     | -     | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 4.90          | 2.53 | 10.66         | 0.42 | 0.13 | 0.63 |\n| (q) |**Single** | **-**      |**-**|**-**|**-**| -       | -     | -     | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 42.31          | 14.80 | 14.64         | 6.23   | 1.75     | **9.49**     |\n| (r) | Dual | 0.0001     | 1   | 1   | 1   | -          | -     | -     | 0.0001     | 1   | 1    | 40.00   | 0.25   | 2.00   | 40.9          | 15.33 | 14.45         | **6.30** | _**2.09**_ | 9.35 |\n\n> `W2:` Ground truth artifacts in SemanticKITTI and the need for evaluation on cleaner datasets.\n\nWe thank you for noting that SemanticKITTI contains trace artifacts from aggregated LiDAR scans. To address this, we cross-validate IPFormer on the SSCBench-KITTI-360 dataset, a clean dataset without trace artifacts, similar to Occ3D and nuCraft. As shown in Tab. 1 of the rebuttal to Reviewer [rTop], IPFormer consistently outperforms its closest competitor, CGFormer+DBScan, across all PSC and SSC metrics, demonstrating that instance delineation in our method is not influenced by ground-truth artifacts.\n\n> `W3 & Q3:` Comparison with SparseOcc for image-only PSC on the Occ3D-nuScenes dataset.\n\nWe appreciate your effort in highlighting the camera-based occupancy method SparseOcc. However, we argue that a comparison with IPFormer is not equitable. SparseOcc uses (i) $6$ surround-view cameras ($6$ multi-view images) and (ii) up to $96$ images per forward pass (8 or 16 frames, each with 6 images) to reconstruct a panoptic grid. Additionally, SparseOcc evaluates performance using non-standard metrics, differing from classical panoptic metrics established in the field.\n\nRegarding the Occ3D-nuScenes dataset, we kindly refer to Tab. 1 of the rebuttal to Reviewer [rTop], where we cross-validate IPFormer on the SSCBench-KITTI-360 dataset, a clean dataset without trace artifacts. These results on zero-shot generalization demonstrate that IPFormer consistently outperforms its closest competitor, CGFormer+DBScan, across all PSC and SSC metrics.\n\n> `Q1:` What is the main contribution to reduce the runtime? No additional DBSCAN computation?\n\nWe thank you for raising these important questions. The main contribution of our work is not the removal of DBSCAN to reduce runtime, as this advantage is a byproduct of addressing PSC in an end-to-end fashion. Instead, our primary contribution lies in the introduction of context-adaptive instance proposals, which extend state-of-the-art instance queries that are static at test time. \n\nAs shown in Tab. 2 of our manuscript, this approach improves Thing-related metrics by an average of $18.65$%. This highlights the effectiveness of context-adaptive proposals and marks a key advancement in 3D vision research.\n\n> `Q2:` Consideration of invisible voxel features and their impact on efficiency and instance-voxel alignment.\n\nWe appreciate these insightful questions and provide additional ablation results in Tab. 2.\n\nIncluding invisible voxels demonstrates superior performance across all PSC metrics. For SSC metrics, the configuration without invisible voxels achieves slightly better IoU, while our method attains marginally higher mIoU. Importantly, GPU utilization remains unchanged, as IPFormer operates on a dense voxel grid.\n\nTable 2: Ablation experiments on the influence of invisible voxels, on the SemanticKITTI dataset.\n| Method          | PQ$^\\dagger$ | PQ-All | SQ-All       | RQ-All | PQ-Thing | SQ-Thing     | RQ-Thing | PQ-Stuff | SQ-Stuff     | RQ-Stuff | IoU   | mIoU        |\n|-----------------|-------------|--------|--------------|--------|----------|--------------|----------|----------|--------------|----------|-------|-------------|\n| w/o invisible voxels | 14.28       | 6.00   | 33.20        | 9.33   | 1.69     | 28.77        | 2.66     | 9.13     | 36.42        | 14.18    | **41.38**  | 15.05        |\n| w/ invisible voxels  | **14.45**       | **6.30** | **41.95**   | **9.75** | **2.09** | **42.67**   | **3.33** | **9.35** | **41.43**   | **14.43** | 40.90 | **15.33** |\n\n> Closing Remark.\n\nThe Reviewer's effort and constructive suggestions are sincerely appreciated. We hope our efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification."}]

Author rebuttal:

Meta-review: This paper addresses the Panoptic Scene Completion (PSC) task. The key idea is to initialize queries as panoptic instance proposals derived from image context, which are then refined through attention-based encoding and decoding to reason about semantic instance–voxel relationships.

The paper initially received mixed scores (1× borderline accept, 3× borderline reject). Reviewers noted several strengths:
- The idea of test-time adaptive proposals for vision-based PSC is novel.
- The method is effective in boosting performance.
- It achieves a significant speedup (14×).

Concerns were raised regarding novelty, insufficient experimental evaluation and ablation, limited improvement over CGFormer+DBScan, and the trade-off between SSC and PSC metrics. However, the authors provided a strong rebuttal, and through intensive reviewer–author discussions, these concerns were addressed. After rebuttal, all reviewers suggested borderline accept. The AC has read the reviews, discussions, and the paper, and concurs with this recommendation, considering the paper worthy of acceptance.
Decision: Accept (poster)