<figure><img src="image_1.png" /></figure>

HAL
open science

# PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

Anh-Quan Cao, Angela Dai, Raoul de Charette

► To cite this version:

Anh-Quan Cao, Angela Dai, Raoul de Charette. PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness. Computer Vision and Pattern Recognition Conference (CVPR), Jun 2024, Seattle, United States. hal-04324930

HAL Id: hal-04324930

https://hal.science/hal-04324930v1

Submitted on 5 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

Anh-Quan Cao¹

Angela Dai²

Raoul de Charette¹

¹Inria

²Technical University of Munich

https://astra-vision.github.io/PaSCo

## Abstract

We propose the task of Panoptic Scene Completion (PSC) which extends the recently popular Semantic Scene Completion (SSC) task with instance-level information to produce a richer understanding of the 3D scene. Our PSC proposal utilizes a hybrid mask-based technique on the non-empty voxels from sparse multi-scale completions. Whereas the SSC literature overlooks uncertainty which is critical for robotics applications, we instead propose an efficient ensembling to estimate both voxel-wise and instance-wise uncertainties along PSC. This is achieved by building on a multi-input multi-output (MIMO) strategy, while improving performance and yielding better uncertainty for little additional compute. Additionally, we introduce a technique to aggregate permutation-invariant mask predictions. Our experiments demonstrate that our method surpasses all baselines in both Panoptic Scene Completion and uncertainty estimation on three large-scale autonomous driving datasets. Our code and data will be made open-source.

## 1. Introduction

Understanding scenes holistically plays a vital role in various fields, including robotics, VR/AR, and autonomous driving. A fundamental challenge in this domain is the simultaneous estimation of complete scene geometry, semantics, and instances from incomplete 3D input data, which is often sparse, noisy, and ambiguous due to occlusions and the inherent complexity of the real scenes. Despite these challenges, achieving this level of understanding is crucial to enable machines to interact with their environment in a smart and safe manner.

Semantic Scene Completion (SSC) tackles 3D scene understanding by inferring the full scene geometry and semantics from a sparse observation. There have been significant advancements in SSC which has gained in popularity. Initial methods [7, 9, 16, 44, 64] focused on indoor scenes characterized by dense, regular, and small-scale input point clouds. The recent release of the Semantic KITTI

<figure><img src="image_3.png" /><figcaption>Figure 1. PaSCo output. Our method infers Panoptic Scene Completion (PSC) from a sparse input point cloud while concurrently assessing uncertainty at both the voxel and instance levels.</figcaption></figure>

dataset has ignited interest for SSC in outdoor driving scenarios [13, 61, 69, 70], which present unique challenges due to the sparsity, large scale, and varying densities of input point clouds [62].

Despite its remarkable performance, current SSC techniques overlook instance-level information and uncertainty prediction. The absence of instance-level prediction hinders their utility in applications that require identification and tracking of individual objects while the lack of uncertainty estimation limits their deployment in real-world safety-critical applications.

To address these challenges, we propose the novel task of Panoptic Scene Completion (PSC), which aims to historically predict the geometry, semantics, and instances of a scene from a sparse observation. We present the first method for this task, named PaSCo, which is a MIMO-inspired [30] ensemble approach boosting PSC performance and uncertainty estimation at minimal computational cost. It combines multi-scale generative sparse networks with a transformer decoder, implementing a mask-centric strategy for instance prediction [11, 12]. Consequently, we introduce a novel ensembling technique for combining unordered mask sets. Through extensive evaluations, our method demonstrates superior performance in PSC and provides valuable insights into the predictive uncertainty. Our contributions can be summarized as follows:

* We formulate the new task of Panoptic Scene Completion (PSC), extending beyond Semantic Scene Completion to reason about instances.
* Our proposed method, PaSCo, utilizes a sparse CNN-Transformer architecture with a multi-scale sparse generative decoder and transformer prediction, optimized for efficient PSC in extensive point cloud scenes.
* By adapting to the MIMO setting and introducing a novel ensembling strategy for unordered sets, our method boosts PSC performance and enhances uncertainty awareness, outperforming all baselines across three datasets.

## 2. Related works

**Semantic Scene Completion (SSC).** SSC was first proposed by SSCNet [64], and recently surveyed in [62]. Prior works mainly focus on indoor scenes [7, 9, 10, 15, 16, 18, 24, 33, 44, 45, 50, 66, 71, 72] with dense, uniform and small-scale point clouds. Semantic KITTI [3] sparked interest in SSC for urban scenes, which pose new challenges due to LiDAR sparsity, large scale, and varying density. To address this, a number of works rely on added modalities [13, 44, 50], while JS3CNet [70] improves by jointly training on semantic segmentation. Strategies for efficient SSC include 2D convolutions on BEV representation in LMSCNet [61] or group convolution in [71]. S3CNet [13] enhances SSC with spatial feature engineering, multi-view fusion, spatial propagation and geometric-aware loss. SCP-Net [69] proposes a novel completion sub-network and distills knowledge from multi-frames model. Still, despite impressive results, none of these works predict instance-level information or consider uncertainty estimation.

**LiDAR Panoptic Segmentation** Panoptic segmentation was initially introduced in [37] for images. Since then it was extended to 3D point clouds, first using range-based representations [35, 42, 57, 63] with 2D convolutions for efficiency which sacrifice spatial detail. Consequently, some leverage sparse convolutions [26, 32, 49, 60] for efficient 3D processing. Panoptic can be structured as a two-stage method, comprising a non-differentiable clustering followed by semantic segmentation [32, 42, 46, 57, 60], or as a proposal-based approach [35, 63], building on Mask R-CNN [31] with an added semantic head. CPSeg [43] and CenterLPS [56] were the first to propose proposal- and clustering-free end-to-end methods relying on pillarized point features [43] or center-based instance encoding and decoding [56]. MaskPLS [55] offers an end-to-end, mask-based architecture. While these methods exhibit strong performance, they only label the input points. Our work goes a step further by predicting a complete panoptic scene with incorporated uncertainty information, thereby facilitating a more comprehensive understanding of the scene.

**Uncertainty Estimation with Efficient Ensemble.** Early Bayesian Neural Networks (BNNs) [53] quantified uncertainty in shallow networks, but remain limited in scale [20], despite recent advances in variational inference techniques [5, 36]. Instead, Deep Ensemble [40] offers

a practical approach to approximate BNNs' posterior weight distribution [68] and is acknowledged as the leading technique for uncertainty estimation and predictive performance [29, 41, 58]. Yet, its computational demand spurred alternatives like deep sparse networks [51] or BatchEnsemble [67] using partially shared weights. Multi-input multi-output (MIMO) [30] offers a lightweight alternative with diversified outputs, training independent subnetworks within a larger network. Techniques also involve selective dropping of neural weights [19, 23] or multiple model checkpoints of a training session [25, 34] but require multiple inferences. Alternatives also approximate the weight posterior during training to sample ensemble members [22, 54], or use grouped convolution [41]. Our work builds on the simplicity and single-inference MIMO [30], which we complement with a novel permutation-invariant mask ensembling.

## 3. Method

We introduce the task of Panoptic Scene Completion (PSC), taking an incomplete point cloud $X$ as input and producing a denser output $Y=f(X)$ as $K$ voxels masks each with semantic class, i.e. $\{(m_k, c_k)\}_{k=1}^K$. Inspired by Semantic Scene Completion [62] (SSC), we build a more holistic understanding by reasoning jointly about geometry, semantics and instances. Like panoptic segmentation [37] for semantic segmentation, PSC is a strict generalization of SSC.

To address PSC, we propose PaSCo, which leverages a multiscale sparse generative architecture and proxy completion in a mask-centric architecture [11, 12, 55]. As model calibration is critical for real-world applications like autonomous driving, we also seek to estimate uncertainty. This is crucial as generative tasks hallucinate part of the occluded scenery. Yet, to the best of our knowledge, uncertainty is overlooked in the SSC literature. To boost uncertainty awareness, we employ a multi-input multi-output strategy [30] with a *constant computational budget*, which outputs multiple PSC variations from augmentations of a single input point cloud. To then infer a unique PSC output, we introduce a custom permutation-invariant ensembling.

The schematic view of our method is in Fig. 2, highlighting how PaSCo enables panoptic scene completion with both semantic and instance-wise uncertainty. For simplicity, we first describe the architecture for panoptic scene completion in Sec. 3.1 and then extend to multi-input/output in Sec. 3.2 for uncertainty awareness. Finally, we detail the training strategy in Sec. 3.3.

### 3.1. Panoptic Scene Completion

Fig. 3 describes our PSC architecture which employs a mask-centric backbone [11, 12, 55]. We rely on multiscale geometric completion (Sec. 3.1.1) followed by a
<figure><img src="image_4.png" /><figcaption>Figure 2. PaSCo overview. Our method aims to predict multiple variations of Panoptic Scene Completion (PSC) given an incomplete 3D point cloud, while allowing uncertainty estimation through mask ensembling. For PSC we employ a sparse 3D generative U-Net with a transformer decoder (Sec. 3.1). The uncertainty awareness is enabled using multiple subnets each operating on a different augmented version of an input data source (Sec. 3.2). PaSCo allows the first Panoptic Scene Completion while providing a robust method for uncertainty estimation. Instance-wise uncertainty shows only “things” classes for clarity.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3. Architecture for PSC. Our architecture builds on a sparse generative U-Net coupled with a transformer decoder applied on pruned non-empty voxels to predict PSC.</figcaption></figure>

transformer decoder for mask predictions of both stuff and things (Sec. 3.1.2) to produce panoptic scene completion.

### 3.1.1 Multiscale Geometric Guidance

We first extract multiscale semantic completion to serve as geometric guidance for PSC. For computational efficiency, we rely on sparse generative 3D U-Net as in [13, 17].

Technically, as detailed in Fig. 3, we process unstructured input point cloud X with an MLP and pass the voxelized features through a light-weight encoder $\mathcal{E}$ to produce 1:8 resolution features. To generate geometry beyond input manifold, we then employ a dense CNN, resulting in denominated 1:8 features $\mathbf{f}^{1:8}$ which are decoded with sparse generative decoders $\{\mathcal{D}^{1:\ell}\}, \forall \ell \in \{4, 2, 1\}$ producing features, written as $\mathbf{f}^{1:\ell}$. At each scale, a lightweight segmentation head $\mathcal{H}^{1:\ell}$ extracts the proxy SSC, i.e., $S^{1:\ell} = \mathcal{H}^{1:\ell}(\mathbf{f}^{1:\ell})$.

Importantly, we prune features after each decoder to preserve sparsity and thus computational efficiency:

$$ \mathbf{f}^{1:\ell} = \mathcal{D}^{1:\ell}(\text{prune}(\mathbf{f}^{1:2\ell})), \forall \ell \in \{4, 2, 1\}. \quad (1) $$

Note that contrary to the literature [13, 17] using binary occupancy maps, we use semantic predictions for $\text{prune}(\cdot)$.

We advocate that semantics better balance performance across small classes, while binary occupancy is dominated by large structural classes (road, building, etc.). As such, PSC can better inherit geometric guidance.

### 3.1.2 Semantic and Instance Prediction as Masks

We now estimate the panoptic completion $\{(m_k, c_k)\}_{k=1}^K$, with $m_k$ being a voxel mask and $c_k$ its corresponding semantic class, for both stuff and things. To do so, we follow the latest mask-centric transformer models [11, 12, 55] predicting PSC from the multiscale features of Sec. 3.1.1.

The transformer takes input queries as mask proposals and use the multi-scale features $\{\mathbf{f}^{1:4}, \mathbf{f}^{1:2}, \mathbf{f}^{1:1}\}$ to predict the final queries for mask prediction. We use the multi-scale decoder layer of Mask2Former [12, 55] with masked attention to foster spatial relationships, improving efficiency and training. Hence, each transformer decoder layer $\mathcal{T}^{1:\ell}, \forall \ell \in \{4, 2, 1\}$ is a tailored mix of masked cross attention, self-attention and feed forward network which ultimately produces a set of queries $Q^{1:\ell} \in \mathbb{R}^{K \times D}$ where K is the number of queries and D is feature dimension.

In practice, since empty voxels dominate 3D scenes [62] and contribute little to semantic understanding, we apply mask prediction only on non-empty voxels. In fine, transformer $\mathcal{T}^{1:\ell}$ takes in query embeddings of the lower scale $Q^{1:2\ell}$, and sparse query features $\hat{\mathbf{f}}^{1:\ell}$ from non-empty voxels of the same scale decoder $\mathcal{D}^{1:\ell}$, i.e., $\hat{\mathbf{f}}^{1:\ell} = \text{nonempty}(\mathbf{f}^{1:\ell})$. Notably, at the lowest resolution (1:4) the input query embedding $Q^{1:8}$ is initialized and optimized during training.

For each query embedding $Q^{1:\ell}$, semantic probabilities $p \in \mathbb{R}^{K \times C}$ and mask scores $m \in \mathbb{R}^{K \times N}$ are extracted, where C is the number of classes and N the number of voxels. Probability p is derived by applying a linear layer
to $Q^{1:\ell}$. The mask score $m$ is computed from the dot product of $Q^{1:\ell}$ and the full scale non-empty voxel features $\hat{\mathbf{f}}^{1:\ell}$: $m^{1:\ell} = \text{sigmoid}(\hat{\mathbf{f}}^{1:\ell} \cdot Q^{1:\ell})$. The resulting masks are obtained with argmax over $p$ and $m$.

Similar to previous works, small masks occluded by others are filtered out to minimize false positives [11, 12, 55]. In fine, the PSC output is the 1:1 panoptic prediction, so $Y = \{(m, c)\}_{k=1}^K = \{(m^{1:1}, p^{1:1})\}_{k=1}^K$, while predictions from query embeddings at other scales (i.e., $\ell \neq 1$) serve for additional guidance with multiscale supervision.

## 3.2. Uncertainty awareness

We equip PaSCo with uncertainty awareness for both efficient and robust panoptic scene completion. Inspired by MIMO [30] doing image classification, we employ a subnetworks formulation to estimate uncertainty on the much more complex task of panoptic scene completion.

Hence, we adjust our PSC architecture (Sec. 3.1) to predict $M$ variations of PSC outputs in a single inference, as seen in Fig. 2, and at a similar computation cost. Intuitively, having several PSC outputs yield better predictive uncertainty estimation and robustness to out-of-distribution [30, 40, 67]. At inference, we use a permutation-invariant mask ensembling strategy to obtain a final unique PSC output.

### 3.2.1 MIMO Panoptic Scene Completion

In the general case of $M$ subnets, PaSCo infers $M$ outputs<fnref n="1" /> given inputs $\{X_i\}_{i=1}^M$. Crucially, at training $\{X_i\}$ are distinct point clouds while, in inference, they are augmentations of the same point cloud. As in MIMO [30], only heads are duplicated and subnets are in fact trained concurrently in our architecture with minimal but effective adjustments, thus keeping the parameter number roughly constant irrespective of the subnets used. E.g., a subnet of PaSCo($M=3$) has 3 times less capacity than that of PaSCo($M=1$).

Specifically, referring to Sec. 3.1.1 we share the MLP among subnets and then concat the voxelized representations along the features dimension before passing it to the encoder and decoders. A major difference, is that each subnet has its own semantic heads leading to $\{\mathcal{H}_i^{1:\ell}\}_{i=1}^M$ so that they infer distinct semantic outputs $\{S_i^{1:\ell}\}_{i=1}^M$. Notably also, the prune(·) operation of Eq. (1) prune only voxels predicted empty by all subnets.

To decode per-subnet panoptic output, we follow Sec. 3.1.2, using a dedicated set of query embeddings per subnet, with a shared transformer decoder to increase diversity of the masks predictions at little cost. Interestingly, we note that this also introduces more diversity into the mask predictions, as each query represents one mask.

Finally, PaSCo output is the combination of all subnets outputs, so $\{Y_i\}_{i=1}^M$ with $Y_i = \{(m_k, c_k)\}_{k=1}^K$.

### 3.2.2 Mask ensembling

Unlike classification in MIMO [30], ensembling several PSCs is complex since each subnet infers a set of masks that are permutation invariant. To ensemble these sets, we introduce a pair-wise alignment strategy.

Given two sets of $K$ masks $Y=\{(m_k, p_k)\}_{k=1}^K$ and $\hat{Y}=\{(\hat{m}_k, \hat{p}_k)\}_{k=1}^K$. We densify the voxel grid, setting empty voxels to 0, such that both mask sets have the same dimension. As they are permutation invariant, we map the two sets using Hungarian matching [39] with the assignment cost matrix $C(\cdot, \cdot) \in \mathbb{R}^{K \times K}$ where

$$C(Y, \hat{Y})_{lk} = - \frac{m_l \hat{m}_k^{\top}}{|m_l| + |\hat{m}_k| - m_l \hat{m}_k^{\top}}, \quad (2)$$

$l$ and $k$ iterate over all mask indices. Rather than matching binary masks, we find that using “soft matching” with sigmoid probabilities improves results (see Sec. 4.3). Once mapped together, the ensemble output is obtained by averaging the semantic probability $p$ and binary mask probability $m$ of these mapped queries.

With more than two sets, we arbitrarily use the first set of masks and iteratively align with the remaining sets.

## 3.3. Training

We train PaSCo end-to-end from scratch with pairs of input point cloud and semi-dense panoptic/semantic labeled voxels, applying losses only on voxels with ground truth labels as in [61, 69, 70].

**Voxel-query semantic loss.** For subnet $i$ predicting binary mask $m_i \in \mathbb{R}^{N \times K}$ and mask softmax probability $p_i \in \mathbb{R}^{K \times C}$, we estimate a subsidiary per-voxel semantic prediction: $S_i' = m_i p_i$, $S_i' \in \mathbb{R}^{N \times C}$. As masks are predicted at full scale, $S_i'$ is optimized with:

$$\mathcal{L}'_{\text{sem}} = \sum_{i=1}^{M} (\text{CE}(S_k'^{1:1}, \bar{S}^{1:1}) + \lambda_1 \text{lovasz}(S'^{1:1}, \bar{S}^{1:1})), \quad (3)$$

being $\bar{S}_i'^{1:1}$ the labels, and $\lambda_1 = 0.3$ empirically fixed [4].

**Semantic loss.** For each scale $1:\ell$ and subnet $i$, we optimize the semantic output $S_i^{1:\ell}$ against the ground truth $\bar{S}_i'^{1:\ell}$ (majority pooled to scale $1:\ell$), using a similar loss as Eq. (3), applied across all scales $\ell \in \{1, 2, 4\}$.

**Masks matching loss.** For each subnet $i$, we match the output masks $Y_i = \{(m_k, c_k)\}_{k=1}^K$ to the ground truth masks $\bar{Y}_i = \{(\bar{m}_k, \bar{c}_k)\}_{k=1}^K$, using the Hungarian matching as in [11, 55] to learn an optimal mapping $\bar{\sigma}$ by minimizing the assignment map $C(\cdot, \cdot) \in \mathbb{R}^{K \times K}$. The latter is defined as $C_{\bar{k}, \bar{k}} = -p_k(\bar{c}_{\bar{k}}) + \mathcal{L}_{\text{mask}}$ with

$$\mathcal{L}_{\text{mask}} = \lambda_{\text{dice}} \text{dice}(m_k, \bar{m}_{\bar{k}}) + \lambda_{\text{bce}} \text{BCE}(m_k, \bar{m}_{\bar{k}}). \quad (4)$$

<footnote n="1">¹In this new light, our PSC architecture in Sec. 3.1 is equal to the special case of $M=1$ subnetwork. i.e., PaSCo($M=1$).</footnote>
<table><caption>Table 1. **Panoptic Scene Completion.** On both Semantic KITTI [3] (val) and SSCBench-KITTI360 [47] (test), our method PaSCo outperforms all baselines across almost all metrics, in particular, All PQ†. * denotes our own re-implementation of SCPNet.</caption><thead><tr><th rowspan="2">method</th><th colspan="10">Semantic KITTI (val set)</th><th colspan="10">SSCBench-KITTI360 (test set)</th></tr><tr><th>PQ<sup>†</sup></th><th>All PQ</th><th>All SQ</th><th>All RQ</th><th>Thing PQ</th><th>Thing SQ</th><th>Thing RQ</th><th>Stuff PQ</th><th>Stuff SQ</th><th>Stuff RQ</th><th>mIoU</th><th>PQ<sup>†</sup></th><th>All PQ</th><th>All SQ</th><th>All RQ</th><th>Thing PQ</th><th>Thing SQ</th><th>Thing RQ</th><th>Stuff PQ</th><th>Stuff SQ</th><th>Stuff RQ</th><th>mIoU</th></tr></thead><tbody><tr><td>LMSCNet [61] + MaskPLS [55]</td><td>13.81</td><td>4.17</td><td>36.13</td><td>6.82</td><td>1.62</td><td>29.87</td><td>2.68</td><td>6.02</td><td>40.69</td><td>9.82</td><td>17.02</td><td>12.76</td><td>4.14</td><td>26.52</td><td>6.45</td><td>0.88</td><td>20.41</td><td>1.58</td><td>5.78</td><td>29.58</td><td>8.88</td><td>15.10</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>18.41</td><td>6.85</td><td>41.90</td><td>11.34</td><td>4.18</td><td>43.10</td><td>7.22</td><td>8.79</td><td>41.03</td><td>14.34</td><td>22.70</td><td>16.42</td><td>6.79</td><td>51.16</td><td>10.71</td><td>3.36</td><td>48.41</td><td>5.83</td><td>8.51</td><td>52.54</td><td>13.15</td><td>21.31</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td>19.39</td><td>8.59</td><td>49.49</td><td>13.69</td><td>4.88</td><td>46.41</td><td>7.70</td><td>11.30</td><td>51.73</td><td>18.04</td><td>22.44</td><td>16.54</td><td>6.14</td><td>51.18</td><td>10.15</td><td>4.23</td><td>48.46</td><td>7.05</td><td>7.09</td><td>52.55</td><td>11.70</td><td>21.47</td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>23.21</td><td>10.89</td><td>48.29</td><td>17.80</td><td>7.35</td><td>42.98</td><td>12.75</td><td>13.46</td><td>52.15</td><td>21.46</td><td>27.89</td><td>18.20</td><td>7.47</td><td>50.67</td><td>11.92</td><td>3.98</td><td>48.13</td><td>6.80</td><td>9.21</td><td>51.94</td><td>14.48</td><td><b>22.66</b></td></tr><tr><td>PaSCo (Ours)</td><td><b>31.42</b></td><td><b>16.51</b></td><td><b>54.25</b></td><td><b>25.13</b></td><td><b>13.71</b></td><td><b>48.07</b></td><td><b>20.68</b></td><td><b>18.54</b></td><td><b>58.74</b></td><td><b>28.38</b></td><td><b>30.11</b></td><td><b>26.29</b></td><td><b>10.92</b></td><td><b>56.10</b></td><td><b>17.09</b></td><td><b>4.88</b></td><td><b>57.53</b></td><td><b>8.48</b></td><td><b>13.94</b></td><td><b>55.39</b></td><td><b>21.39</b></td><td><b>22.39</b></td></tr></tbody></table>

We set $K$ always greater than $\bar{K}$ the number of ground truth masks. Predicted masks without ground truth are mapped to a generic $\emptyset$ class. For the $k$-th mask of $Y_i$ matched to the $\bar{\sigma}(k)$-th ground truth mask, the loss is

$$ \mathcal{L}_{\text{matched}} = \sum_{k=1}^{\bar{K}} \lambda_{\text{CE}} \text{CE}(c_k, \bar{c}_{\bar{\sigma}(k)}) + \mathcal{L}_{\text{mask}}. \quad (5) $$

The $K-\bar{K}$ unmatched predicted masks are optimized to predict $\emptyset$ class $\mathcal{L}_{\text{unmatched}} = \sum_{k=\bar{K}+1}^{K} \lambda_{\emptyset} \text{CE}(c_k, \emptyset)$, where $\lambda_{\emptyset}$ is set to 0.1 as in [11]. $\lambda_{\text{dice}}$ and $\lambda_{\text{bce}}$ are empirically set to 1 and 40. We further apply auxiliary mask matching losses and $\mathcal{L}'_{\text{sem}}$ on the PSC outputs of intermediate scales, i.e., $\{(m^{1:\ell}, p^{1:\ell})\}, \ell \neq 1$.

## 4. Experiments

We evaluate PaSCo on both panoptic scene completion and uncertainty estimation, while also reporting the subsidiary SSC metrics. As there are no urban PSC datasets and baselines, we produce our best effort to extend existing SSC datasets and baselines for fair evaluation.

**Datasets.** To evaluate PSC, we extend three large-scale urban LiDAR SSC datasets: Semantic KITTI, SSCBench-KITTI360 and Robo3D. **Semantic KITTI** [3] has 64-layer LiDAR scans voxelized into 256x256x32 grids of 0.2m voxels. We follow the standard train/val split [61, 69], leading to 3834/815 grids. **SSCBench-KITTI360** [47] is a very recent SSC benchmark derived from KITTI-360 [48] with urban scans encoded as in Semantic KITTI. We follow the standard train/val/test splits of 8487/1812/2566 grids. **Robo3D** [38] is a new robustness benchmark, extending popular urban datasets [3, 6, 27, 65] by modifying point cloud inputs with various type and intensity of corruptions (e.g., fog, motion blur, etc.). We use corrupted input point clouds from the SemanticKITTI-C set of Robo3D to evaluate robustness to Out Of Distribution (OOD) effects.

To extract pseudo panoptic labels from semantic grids, we cluster things instances from ad-hoc classes using DB-SCAN [8, 21] with a distance of $\epsilon = 1$ and groups with MinPts = 8. Following the original panoptic segmentation formulation [37], stuff masks are made of voxels with ad-hoc classes. For Semantic KITTI, labels cannot be generated on the hidden test set, so we evaluate on val. set only.

**PSC/SSC metrics.** We evaluate panoptic quality (PQ), segmentation quality (SQ) and recognition quality (RQ) following [37] on the complete scene. Due to the difficulty of the PSC task, most masks have low IoU w.r.t. ground truth. Hence, we note that the over-penalization effect of stuff classes described in [59] is amplified for PSC. Thus, we also evaluate PQ†, as in [59], removing the >0.5-IoU rule for stuff classes. We also complement our PSC study with subsidiary SSC metrics, i.e., mean IoU (mIoU).

**Uncertainty metrics.** Following [41], we employ the maximum softmax probability as a measure of model confidence. Consistent with the established practices [30, 40, 41], we assess the model predictive uncertainty by evaluating its calibration [28] using the Expected Calibration Error (ECE) and Negative Log Likelihood (NLL). Notably, we distinguish between two forms of uncertainty: *voxel uncertainty* and *instance uncertainty*. The former is derived voxel-wise from semantic completion outputs, and the latter mask-wise from class probability predictions. To account for the dominance of empty voxels within 3D scenes, we calculate voxel uncertainties by averaging the uncertainties for empty and non-empty voxels. In line with [37], the label of predicted masks are assigned by finding the matched ground truth masks with >0.5-IoU rule. Unmatched masks are classified under a 'dustbin' category.

**Training details.** We train PaSCo for 30 epochs on Semantic KITTI and 20 epochs on SSCBench-KITTI360, both using AdamW [52] optimizer and batch size of 2. The learning rate is 1e-4, unchanged for Semantic KITTI but divided by 10 at epoch 15 on SSCBench-KITTI360. We apply random rotations in [-30°, 30°] on Semantic KITTI and in [-10°, 10°] on SSCBench-KITTI360, random crop to reduce the scene size to 80% along both the x and y axes, and random translations of ±0.6m on x/y axes and ±0.4m on z axis. Unless otherwise mentioned, PaSCo refers to the optimal number of subnets, which is $M = 3$ for Semantic KITTI and Robo3D, and $M = 2$ for SSCBench-KITTI360. This choice of subnets is justified in Tab. 4.

### 4.1. Panoptic Scene Completion

To evaluate PSC, we first establish baselines for this new task, and then report results on the aforementioned datasets.
<figure><img src="image_6.png" /><figcaption>Figure 4. **Qualitative Panoptic Scene Completion.** We report PSC outputs for all baselines of Tab. 1. PaSCo shows better instance separation, with stronger instance shapes and scene structure, with fewer holes.</figcaption></figure>

**Baselines.** We combine existing SSC methods with 3D panoptic segmentation. We select three SSC open-source methods: LMSCNet [61], JS3CNet [70], SCPNet [69], and also add SCPNet* — our own reimplementation with much stronger performance. For 3D panoptic segmentation, we use MaskPLS [55], well-suited for dense voxelized scene and the best open-source 3D panoptic segmentation to date. All baselines are retrained with their reported parameters.

We train the four PSC baselines using the SSC method to predict the complete semantic scene followed by the 3D panoptic segmentation method.

**Performance.** Tab. 1 compares PaSCo with the 4 baselines on Semantic KITTI and SSCBench-KITTI360. Our method is superior across all panoptic metrics (All, Things, Stuff) on both datasets. We see a major boost in All-PQ†/PQ of +8.21/+5.62 on Semantic KITTI and +8.09/+3.45 on SSCBench-KITTI360, due to our effective ensembling approach for PSC. Additionally, PaSCo outperforms baselines in individual metrics for both ‘things’/‘stuff’ categories, showing significant improvements in PQ with +6.36/+5.08 and +0.9/+4.73 on each dataset. On the subsidiary mIoU metric we perform on-par, being first on Semantic KITTI (+2.22) and 2nd in SSCBench-KITTI360 (−0.27). Incidentally, we note that PSC and SSC metrics are not directly correlated since we improve the former drastically.

Fig. 4 shows that our qualitative PSC results similarly show visual superiority. Overall, we observe that instances are much better separated by PaSCo compared to SCPNet*

(our best competitor), with less holes in the geometry.

## 4.2. Uncertainty estimation

We further evaluate uncertainty as it correlates with model calibration [1], and is crucial for many applications.

**Baselines.** Using our architecture, we design three uncertainty estimation baselines based on state-of-the-art uncertainty literature. Each baseline provides multiple outputs, enabling similar computation of uncertainty to ours from the maximum softmax probability across inferences. Test-Time Augmentation (TTA) is a classical strategy [2] to improve robustness using multiple inferences of a unique network with input augmentations. MC Dropout [23] provides a bayesian approximation of the model uncertainty by randomly dropping activations (i.e., setting to 0) of neurons, applied with multiple inferences. Finally, we report Deep Ensemble [40], where duplicate networks solving the same task are trained independently and ensembled at test time for better predictive uncertainty than a single network.

For fair comparison, all baselines use our architecture. However, we note that in contrast to our approach, these baselines require more than one pass, either using multiple inferences for TTA and MC Dropout or multiple networks for Deep Ensemble which translate in more parameters.

**Uncertainty estimation.** Tab. 2 reports uncertainties for all baselines using our architecture, as well as for PaSCo and PaSCo($M=1$) which uses a single subnet. To ensure comparable performance, we set the number of inferences
<table><thead><tr><th rowspan="2">method</th><th colspan="6">Semantic KITTI (val set)</th></tr><tr><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th><th>All PQ↑</th><th>All PQ↑</th><th>mloU↑</th><th>Params↓</th><th>Passes↓</th></tr></thead><tbody><tr><td>TTA</td><td>-</td><td>-</td><td>0.0456</td><td>0.7224</td><td>-</td><td>-</td><td>28.84</td><td><b>111M</b></td><td><b>3</b></td></tr><tr><td>MC Dropout [23]</td><td>-</td><td>-</td><td>0.0472</td><td>0.7437</td><td>-</td><td>-</td><td>28.82</td><td><b>111M</b></td><td><b>3</b></td></tr><tr><td>Deep Ensemble [40]</td><td>-</td><td>-</td><td>0.0428</td><td>0.6993</td><td>-</td><td>-</td><td>30.10</td><td><b>333M</b></td><td><b>3</b></td></tr><tr><td>PaSCo(M=1)</td><td>0.6181</td><td>4.6559</td><td>0.0610</td><td>0.8250</td><td>26.49</td><td>15.36</td><td>28.22</td><td><b>111M</b></td><td><b>1</b></td></tr><tr><td>PaSCo (ours)</td><td><b>0.4922</b></td><td><b>3.9155</b></td><td><b>0.0426</b></td><td><b>0.5835</b></td><td><b>31.42</b></td><td><b>16.51</b></td><td><b>30.11</b></td><td><b>120M</b></td><td><b>1</b></td></tr></tbody></table>

<table><caption>**Table 2. Uncertainty evaluation.** We evaluate uncertainty on Semantic KITTI (top) and SSCBench-KITTI360 (bottom). Baselines only produce voxel uncertainty ('voxel ece,' 'voxel nll') which we outperform while also estimating PSC uncertainty (All PQ/PQ†).</caption><thead><tr><th rowspan="2">method</th><th colspan="6">SSCBench-KITTI360 (test set)</th></tr><tr><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th><th>All PQ↑</th><th>All PQ↑</th><th>mloU↑</th><th>Params↓</th><th>Passes↓</th></tr></thead><tbody><tr><td>TTA</td><td>-</td><td>-</td><td>0.1580</td><td>2.1282</td><td>-</td><td>-</td><td>21.78</td><td><b>111M</b></td><td><b>2</b></td></tr><tr><td>MC Dropout [23]</td><td>-</td><td>-</td><td>0.1548</td><td>2.0737</td><td>-</td><td>-</td><td>21.73</td><td><b>111M</b></td><td><b>2</b></td></tr><tr><td>Deep Ensemble [40]</td><td>-</td><td>-</td><td>0.1540</td><td>2.0653</td><td>-</td><td>-</td><td><b>22.51</b></td><td>222M</td><td><b>2</b></td></tr><tr><td>PaSCo(M=1)</td><td>0.7899</td><td>5.4405</td><td>0.1749</td><td>2.3556</td><td>19.53</td><td>9.91</td><td>21.17</td><td><b>111M</b></td><td><b>1</b></td></tr><tr><td>PaSCo (ours)</td><td><b>0.6015</b></td><td><b>4.1454</b></td><td><b>0.1348</b></td><td><b>1.6112</b></td><td><b>26.29</b></td><td><b>10.92</b></td><td><b>22.39</b></td><td><b>115M</b></td><td><b>1</b></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>**Figure 5. Qualitative uncertainty comparison on SSCBench-KITTI360 and Semantic KITTI.** Note that "ins. unc." only shows examples from the "thing" class for clearer visualization. PaSCo(M=1) tends towards overconfidence in both voxel and ins. unc. In contrast, PaSCo gives more intuitive uncertainty estimates, e.g., at segment boundaries, in areas with hallucinated scenery, and in regions with low input point density.</figcaption></figure>

(for TTA and MC Dropout) and number networks (for Deep Ensemble) equal to the number of subnets in PaSCo — i.e., 3 on Semantic KITTI and 2 on SSCBench-KITTI360. Notably, baselines can only estimate voxel-wise uncertainties, for which we outperform by a large margin. Only the voxel ece of Deep Ensemble for Semantic KITTI is a close second (0.0428 vs 0.0426), though at the cost of ≈ 3 times our number of parameters and 3 passes. Comparing PaSCo(M=1) and PaSCo highlights that our ensemble approach brings a clear boost on all metrics at a minor increase of number of parameters (111M vs 115M).

Fig. 5 visualizes uncertainty estimation from PaSCo(M=1) and PaSCo on Semantic KITTI

<table><caption>**Table 3. Effect of our ensembling.** We apply our permutation-invariant ensembling strategy (Sec. 3.2.2) to all baselines to enable PSC uncertainty estimation. Even when using our technique, we note PaSCo remains the best performing.</caption><thead><tr><th rowspan="2">method</th><th colspan="6">Semantic KITTI (val set)</th></tr><tr><th>All PQ↑</th><th>All PQ↑</th><th>ins ece↓</th><th>ins nll↓</th><th>Passes↓</th><th>All PQ↑</th><th>All PQ↑</th><th>ins ece↓</th><th>ins nll↓</th><th>Passes↓</th></tr></thead><tbody><tr><td>TTA</td><td>28.16</td><td>15.95</td><td>0.5295</td><td>4.3804</td><td>3</td><td>23.31</td><td>9.76</td><td>0.6953</td><td>4.8958</td><td>2</td></tr><tr><td>MC Dropout</td><td>29.62</td><td>16.11</td><td>0.5684</td><td>4.8617</td><td>3</td><td>23.73</td><td>9.95</td><td>0.6804</td><td>4.6174</td><td>2</td></tr><tr><td>Deep Ensemble</td><td>30.71</td><td>16.41</td><td>0.5008</td><td>3.9181</td><td>3</td><td>23.85</td><td>9.88</td><td>0.6673</td><td>4.7809</td><td>2</td></tr><tr><td>PaSCo (Ours)</td><td><b>31.42</b></td><td><b>16.51</b></td><td><b>0.4922</b></td><td><b>3.9155</b></td><td><b>1</b></td><td><b>26.29</b></td><td><b>10.92</b></td><td><b>0.6015</b></td><td><b>4.1454</b></td><td><b>1</b></td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>**Figure 6. Effects of Out Of Distribution.** We evaluate uncertainties on corruptions of the Robo3D [38], shown in the x axis. Each bar reports the metric average per corruption while its error bar indicates the per-intensity minimum and maximum metric. PaSCo outperforms all methods by a large margin on all corrections for instance-wise uncertainty (left) and better on 7 of 8 conditions (except 'cross-talk') on voxel-wise uncertainty (right).</figcaption></figure>

and SSCBench-KITTI360. For clarity, instance-wise uncertainty shows only “thing” categories. PaSCo(M=1) often shows high confidence, likely due to deep networks’ tendency for overconfidence [28]. For voxel-wise uncertainty, PaSCo exhibits increased uncertainty at segment boundaries (e.g., roads, sidewalks), low point density areas, and large missing regions. Instance-wise, PaSCo indicates more uncertainty in regions with ambiguous predictions, like sparse input points or close object proximity.

**Mask ensembling.** As the uncertainty-aware baselines do not estimate instance uncertainties, we apply our permutation-invariant ensembling (Sec. 3.2.2) to all baselines, in order to enable instance-wise uncertainty estimation for all. Tab. 3 shows that our MIMO-strategy performs better than the baselines on all metrics, using a single pass.

**Effects of Out Of Distribution.** In the literature, uncertainty is classically used as a proxy of robustness to Out Of Distribution (OOD). To complement our study, we evaluate on the Robo3D [38], which provides point cloud under eight types of corruptions (e.g., fog, beam missing, cross-sensor, wet ground, etc.), each with three level of intensities (light, moderate, heavy). We evaluate on the complete set of 24 corruptions and plot instance and voxel uncertainties in Fig. 6, showing that PaSCo demonstrates consistent improvement over baselines. Each bar shows the mean uncertainty of a method on a given corruption, while the error bar shows the per-level minimum and maximum uncertainties. Interestingly, we note that instance (Fig. 6, left) and voxel
<table><caption>Table 4. Performance when varying number of subnets on Semantic KITTI [3] and SSCBench-KITTI360 [47] validation sets. PSC performance improves as the number of *M* increases, peaking at *M*=3 for Semantic KITTI and at *M*=2 for SSCBench-KITTI360. Further increasing the subnets can also help with uncertainty estimates. We choose *M*=3 for Semantic KITTI and *M* = 2 for SSCBench-KITTI360 to balance high PSC performance and uncertainty estimation.</caption><thead><tr><th rowspan="2"># subnets</th><th colspan="8">Semantic KITTI (val. set)</th><th colspan="8">SSCBench-KITTI360 (val. set)</th><th rowspan="2"># Params↓</th></tr><tr><th>All PQ<sup>†</sup>↑</th><th>All PQ↑</th><th>mIoU↑</th><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th><th>All PQ<sup>†</sup>↑</th><th>All PQ↑</th><th>mIoU↑</th><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th></tr></thead><tbody><tr><td>1</td><td>26.49</td><td>15.36</td><td>28.22</td><td>0.6181</td><td>4.6559</td><td>0.0610</td><td>0.8250</td><td>18.87</td><td>7.77</td><td>20.59</td><td>0.8355</td><td>6.2581</td><td>0.1744</td><td>2.5785</td><td><b>111M</b></td></tr><tr><td>2</td><td>30.34</td><td><b>17.23</b></td><td><u>30.04</u></td><td>0.5535</td><td><u>4.1474</u></td><td>0.0530</td><td>0.6449</td><td><b>27.20</b></td><td><b>8.36</b></td><td><b>21.63</b></td><td>0.6022</td><td>4.4120</td><td>0.1285</td><td>1.8063</td><td><b>115M</b></td></tr><tr><td>3</td><td><b>31.42</b></td><td><b>16.51</b></td><td><b>30.11</b></td><td><b>0.4922</b></td><td><b>3.9155</b></td><td><u>0.0426</u></td><td><u>0.5835</u></td><td>22.31</td><td>6.88</td><td><u>20.60</u></td><td><u>0.5293</u></td><td><u>3.4189</u></td><td><u>0.1233</u></td><td><u>1.6188</u></td><td><b>120M</b></td></tr><tr><td>4</td><td><u>31.20</u></td><td>16.33</td><td>29.41</td><td><u>0.5304</u></td><td>4.2681</td><td><b>0.0349</b></td><td><b>0.5572</b></td><td><u>23.23</u></td><td>6.49</td><td>20.34</td><td><b>0.4098</b></td><td><b>2.4370</b></td><td><b>0.1011</b></td><td><b>1.4814</b></td><td><b>125M</b></td></tr></tbody></table>

<table><caption>Table 5. Method ablation. We ablate inference (top) and training (bottom) components of our method, showing that each contributes to the best performance.</caption><thead><tr><th></th><th>All PQ<sup>†</sup>↑</th><th>All PQ↑</th><th>mIoU↑</th><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th></tr></thead><tbody><tr><td>w/o augmentation</td><td>27.89</td><td>14.07</td><td>28.30</td><td><u>0.5031</u></td><td>4.4245</td><td>0.0442</td><td>0.6713</td></tr><tr><td>w/o rotation augmentation</td><td>28.84</td><td>14.95</td><td>28.95</td><td>0.5074</td><td>4.2987</td><td>0.0432</td><td>0.6309</td></tr><tr><td>w/o voxel-query sem. loss</td><td>28.82</td><td><u>15.55</u></td><td><u>29.87</u></td><td>0.5205</td><td>4.2909</td><td>0.0437</td><td><u>0.5878</u></td></tr><tr><td>w/o sem. pruning</td><td><u>30.12</u></td><td>15.04</td><td>29.04</td><td>0.5380</td><td>4.1814</td><td><u>0.0440</u></td><td><u>0.5980</u></td></tr><tr><td>PaSCo (Ours)</td><td><b>31.42</b></td><td><b>16.51</b></td><td><b>30.11</b></td><td><b>0.4922</b></td><td><b>3.9155</b></td><td><u>0.0426</u></td><td><b>0.5835</b></td></tr></tbody></table>

(Fig. 6, right) uncertainties are not strongly correlated, although methods' rankings remain rather stable across conditions. For instance-wise uncertainty ('ins ece'), PaSCo is significantly better than *all* baselines on *all* corruptions, improving in 7 out of 8 on voxel-wise uncertainty ('ssc ece').

## 4.3. Ablation Studies

**Method ablation.** We ablate our method on SemanticKITTI [3] in Tab. 5 and report SSCBench-KITTI360 in Tab. 6. The upper table ablates our inference augmentations (i.e., rotation+translation), which benefit overall performance, especially All-PQ†/PQ. We attribute this to the increased variance profitable to the subnetworks as in MIMO [30]. In the lower table, we retrain PaSCo while removing some components. We show that removing our voxel-query semantic loss $L'_{sem}$ (Eq. (3)) harms training; such proxy supervision boosts performance at no additional cost. Finally, 'w/o sem. pruning' replaces our semantic pruning with binary occupancy pruning [13, 17], resulting in degraded performance due to loss of smaller classes.

**Subnets ablation.** Tab. 4 ablates different numbers of subnets $M \in \{1, 2, 3, 4\}$ on the validation sets of our main datasets. Our main PQ metrics increase significantly with more subnets, though plateauing at $M=3$ for SemanticKITTI and $M=2$ for SSCBench-KITTI360. This is due to the preserved constant computational cost implying that more subnets mean less per-subnet capacity, leading to more noise in the ensembling. Our finding confirms that of MIMO [30] in the classification setting, though we argue our plateau is reached before since PSC being is a much more complex task than classification.

<figure><img src="image_9.png" /><figcaption>Figure 7. Ensemble vs subnets. We compare our ensemble method with averaging individual subnets, across varying # subnets (x-axis). Error bars show standard deviation across subnets. Peak performance is at *M*=3, where our ensembling compensates for reduced per-subnet capacity with more subnets.</figcaption></figure>

**Mask matching.** We ablate our mask matching, substituting our 'soft matching' (sigmoid probabilities) with 'hard matching' with binary mask IoU for assignment cost matrix calculation (Sec. 3.2.2). This results in a large drop in All-PQ†/PQ of -3.75/-1.12. Entirely removing mask matching severely impacts mask quality, dropping to 0.02/0.02.

**Ensemble vs. Subnets.** To further shed light on subnets performance, Fig. 7 displays metrics of SemanticKITTI as a function of number of subnets *M* for our ensembling (Sec. 3.2.2) or the averaging of the individual subnet performance. When averaging subnets, optimal performance is reached at *M* = 2, with larger *M* unable to solve PSC efficiently. However, ensembling reaches improved performance at *M* = 3, showing that our ensembling effectively leverages weaker subnets.

**Limitations.** Like MIMO [30], our method can accommodate a limited number of subnets, depending on the task nature and the network's capacity. Our approach may overlook objects or mix up nearby objects, particularly when they are small and exhibit semantic resemblance. Our method does not distinguish between types of uncertainty, such as epistemic or aleatoric. Exploring this aspect could be a valuable direction for future research.

## 5. Conclusion

We first address Panoptic Scene Completion (PSC) which aims to complete scene geometry, semantics, and instance-level information from a sparse observation. To address this task, we introduce an efficient ensembling method complemented by a novel technique designed to combine predictions of unordered sets, enhancing the overall prediction ac-
accuracy and reliability in terms of uncertainty.

**Acknowledgment.** The research was supported by the French project SIGHT (ANR-20-CE23-0016), the ERC Starting Grant SpatialSem (101076253), and the SAMBA collaborative project co-funded by BpiFrance in the Investissement d'Avenir Program. Computation was performed using HPC resources from GENCI-IDRIS (2023-AD011014102, AD011012808R2). We thank all Astra-Vision members for their valuable feedbacks, including Andrei Bursuc for his excellent suggestions and Tetiana Martyniuk for her kind proofreading.
# PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

## Supplementary Material

We begin by reporting implementation details of PaSCo and baselines in Sec. 6, and present additional ablations in Sec. 7. Finally, we provide additional experiments on uncertainty estimation and panoptic scene completion in Sec. 8, showcasing PaSCo performance.

We refer to the **supplementary video** for better qualitative judgment of PaSCo performance.

## 6. Implementation details

**PaSCo.** Our network employs sparse convolution from the MinkowskiEngine library [14]. The architecture of our Dense 3D CNN is similar to the 3D Completion Subnetwork of SCPNet [69]. Additionally, the implementation of the MLP and voxelization is based on Cylinder3D [73].

Training PaSCo required three days for the Semantic KITTI dataset and five days for the SSCBench-KITTI360 dataset using 2 A100 GPUs (1 item per GPU).

**Baselines.** We employed the official implementations of LMSCNet<fnref n="2" />, JS3CNet<fnref n="3" />, SCPNet<fnref n="4" />, and MaskPLS<fnref n="5" /> with their provided parameters.

For SCPNet, despite many email exchanges with the authors we were unable to reproduce their reported performance using their official code as also mentioned by other users <fnref n="6" />. Hence, we put extra effort to reimplement their method following authors' recommendation, which resulted in SCPNet*. Note that the latter is several points better than the official implementation.

## 7. Additional ablations

**Method ablation on SSCBench-KITTI360.** We provide additional ablations of our method on SSCBench-KITTI360 validation set in Tab. 6, which align with the results reported in Tab. 5. The first two rows of Tab. 6 ablate the augmentations used during inference (i.e. rotation + translation). The subsequent rows present the performance of PaSCo, retrained w/o our proposed components.

Notably, substituting semantic pruning with binary occupancy pruning [13, 17] leads to a substantial decline in performance, particularly in PSC metrics with -4.82/-1.86 in All PQ†/All PQ. This result is expected as semantic pruning not only balances supervision across classes, particularly

<table><caption>Table 6. Method ablation on SSCBench-KITTI360 (val. set.)
We ablate different components of our method during inference (top) and training (bottom), demonstrating that each plays a significant role in achieving the best performance.</caption><thead><tr><th></th><th>All PQ†↑</th><th>All PQ↑</th><th>mlou↑</th><th>ins ece↓</th><th>ins nll↓</th><th>voxel ece↓</th><th>voxel nll↓</th></tr></thead><tbody><tr><td>w/o augmentation</td><td>26.42</td><td>8.09</td><td>20.40</td><td>0.6114</td><td>4.7203</td><td>0.1296</td><td>2.0639</td></tr><tr><td>w/o rotation augmentation</td><td>26.48</td><td>7.90</td><td>21.20</td><td>0.6028</td><td>4.6321</td><td>0.1293</td><td>1.9261</td></tr><tr><td>w/o voxel-query sem. loss</td><td>24.96</td><td>7.59</td><td>20.90</td><td>0.6381</td><td>4.4526</td><td>0.1307</td><td>1.8124</td></tr><tr><td>w/o sem. pruning</td><td>22.38</td><td>6.50</td><td>19.59</td><td>0.6244</td><td>4.5468</td><td>0.1364</td><td>2.0997</td></tr><tr><td>PaSCo (Ours)</td><td>27.20</td><td>8.36</td><td>21.63</td><td>0.6022</td><td>4.4119</td><td>0.1285</td><td>1.8063</td></tr></tbody></table>

smaller ones, but also provides additional information to the network. Removing our voxel-query semantic loss (Eq. (3)) also leads to a remarkable decrease in performance, with a -2.24/-0.77 drop in All PQ†/All PQ, demonstrating its effectiveness without incurring additional computational costs. Lastly, the augmentations applied during inference (top rows) contribute to increased variation among subnetworks, thereby enhancing overall performance.

**Semantic pruning.** We further ablate the use of our semantic pruning in Tab. 7, by replacing it with binary pruning as employed in [13, 17]. From the table, semantic pruning significantly enhances performance across most classes, notably for rare classes such as truck (+33.72/+2.67 PQ on Semantic KITTI/SSCBench-KITTI360), other-vehicle (+4.21/3.22 PQ on Semantic KITTI/SSCBench-KITTI360), and pole (+1.48/+2.22 PQ on Semantic KITTI/SSCBench-KITTI360). These improvements can be attributed to a more balanced supervision among classes and the inclusion of additional semantic information.

## 8. Additional experiments

We provide further experiments for uncertainty estimation in Sec. 8.1 and panoptic scene completion in Sec. 8.2.

### 8.1. Uncertainty estimation

**Robustness to Out Of Distribution (OOD).** We extend our evaluation for Out Of Distribution, initially reported in Sec. 4.2 and Fig. 6. In Fig. 8, we again evaluate baselines MC Dropout [23], TTA [2], Deep Ensemble [40] along with PaSCo($M=1$) and PaSCo, on the Robo3D dataset [38] which contains corrupted version of SemanticKITTI. All methods are trained on the clean version of SemanticKITTI.

Different from the main paper, we report performance on the 'clean' set (i.e., the original SemanticKITTI) to better assess the effect of OOD. It's also important to note that better calibration may come at the cost of worse performance.

<footnote n="2">²https://github.com/cv-rits/LMSCNet</footnote>

<footnote n="3">³https://github.com/yanx27/JS3C-Net</footnote>

<footnote n="4">⁴https://github.com/SCPNet/Codes-for-SCPNet</footnote>

<footnote n="5">⁵https://github.com/PRBonn/MaskPLS</footnote>

<footnote n="6">⁶https://github.com/SCPNet/Codes-for-SCPNet/issues/8</footnote>
<figure><img src="image_10.png" /><figcaption>Table 7. Ablation of sem. pruning on (a) Semantic KITTI (val. set) and (b) SSCBench-KITTI360 (val. set) for Panoptic Scene Completion class-wise performance. Semantic pruning improves the performance of the majority of classes on both datasets.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 8. Impact of Out Of Distribution data on PSC and uncertainty performance. We evaluate PaSCo and baselines, trained on 'clean' Semantic KITTI on corrupted versions of the same set from the Robo3D [38] dataset. Top part reports, uncertainty ece metrics (lower is better), while bottom part reports performance metrics (higher is better). The types of corruptions are shown on the x-axis. Each bar represents the average metric for each type of corruption, with the error bars showing the minimum and maximum metric across intensities. PaSCo surpasses all baselines in terms of uncertainty measurement (i.e., lower ece) and demonstrates comparable or better performance metrics (higher All PQ† and mIoU).</figcaption></figure>

Hence, Fig. 8 presents not only uncertainty metrics (instance ece, SSC ece – lower is better) but also two performance metrics (All PQ† and mIoU – higher is better). The latter demonstrates that our better calibration (Fig. 8, top) comes along with better performance (Fig. 8, bottom) in almost all corruptions. This further demonstrates the superiority of PaSCo compared to both its one-subnet variation, PaSCo(M=1), and all other baselines.

**Qualitative results.** Fig. 9a and Fig. 9b present additional qualitative results of uncertainty estimation on Semantic KITTI and SSCBench-KITTI360 validation sets. We also illustrate the uncertainty of “stuff” class which we omitted for brevity in the main paper.

Overall, PaSCo(M=1) exhibits higher level of confidence across both voxel and instance uncertainties. Instances of small classes such as person, motorcycle, pole, and traffic light show higher uncertainty levels compared to larger objects like buildings, road and sidewalk. Furthermore, uncertainty tends to increase around the edges of instances, in areas with occluded views, and in regions with lower density of input points.

## 8.2. Panoptic Scene Completion (PSC)

**Class-wise performance.** Tab. 8 presents a class-wise comparison of our method, PaSCo, against baselines on (a) the Semantic KITTI (val. set) and (b) the SSCBench-KITTI360 (test set). For the important PQ metric, PaSCo outperforms all baselines in most classes on Semantic KITTI, with the exception of the fence category. Note that no method successfully predicted the bicyclist and motorcyclist classes. The SSCBench-KITTI360 dataset demonstrates its greater complexity with generally lower performance metrics (All PQ† and mIoU – higher is better).
formance across all classes when compared to Semantic KITTI. Nonetheless, PaSCo still demonstrates strong performance, ranking first in 10 out of 18 classes and second in 3 out of 18. The effectiveness of PaSCo is further illustrated in its superior ability to detect masks and produce high-quality masks, as reflected by its first or second highest performance in most classes based on SQ and RQ metrics on both datasets.

**Qualitative results.** Fig. 10 presents further qualitative results on Semantic KITTI and SSCBench-KITTI360. PaSCo predicts a more complete scene geometry, as illustrated in rows 1, 3, 5 and 6. It also infers better instance quality, illustrated by the increased accuracy in instance shape in rows 2, 4, 5 and 6, and by the clearer separation observable in rows 2, 3, 4 and 5.
<table><thead><tr><th></th><th>Method</th><th>car (3.92%)</th><th>bicycle (0.03%)</th><th>motorcycle (0.05%)</th><th>truck (0.16%)</th><th>other-veh. (0.20%)</th><th>person (0.07%)</th><th>bicyclist (0.07%)</th><th>motorcyclist. (0.05%)</th><th>road (15.30%)</th><th>parking (1.2%)</th><th>sidewalk (11.13%)</th><th>other-grnd (0.56%)</th><th>building (14.10%)</th><th>fence (3.90%)</th><th>vegetation (39.30%)</th><th>trunk (0.51%)</th><th>terrain (9.17%)</th><th>pole (0.29%)</th><th>traf.-sign (0.08%)</th><th>mean</th></tr></thead><tbody><tr><td rowspan="4">PQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>9.43</td><td>0.00</td><td>0.76</td><td>2.32</td><td>0.00</td><td>0.47</td><td>0.00</td><td>0.00</td><td>53.53</td><td>1.82</td><td>5.63</td><td><u>0.00</u></td><td>0.26</td><td>0.19</td><td>0.00</td><td>0.27</td><td>3.52</td><td>1.00</td><td>0.00</td><td>4.17</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>9.57</td><td>1.07</td><td>4.19</td><td>17.54</td><td>0.91</td><td>0.12</td><td>0.00</td><td>0.00</td><td>58.45</td><td>5.32</td><td>15.89</td><td><u>0.00</u></td><td>1.02</td><td>1.33</td><td>0.00</td><td>0.76</td><td>13.63</td><td>0.28</td><td>0.00</td><td>6.85</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td>18.44</td><td><u>4.84</u></td><td>6.72</td><td>4.42</td><td>2.79</td><td><u>1.81</u></td><td>0.00</td><td>0.00</td><td>63.89</td><td>7.92</td><td>19.92</td><td><u>0.00</u></td><td><u>3.11</u></td><td><u>3.28</u></td><td>0.13</td><td>2.29</td><td>21.55</td><td>1.99</td><td>0.17</td><td>8.59</td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>11.72</td><td>1.80</td><td>14.70</td><td>26.44</td><td>3.83</td><td>0.33</td><td>0.00</td><td>0.00</td><td>66.44</td><td>18.71</td><td><u>25.29</u></td><td><u>0.00</u></td><td>2.06</td><td><u>4.12</u></td><td>0.39</td><td>3.11</td><td>22.24</td><td>3.97</td><td>1.72</td><td>10.89</td></tr><tr><td></td><td>PaSCo (Ours)</td><td><u>24.55</u></td><td><u>7.82</u></td><td><u>18.09</u></td><td><u>44.89</u></td><td><u>11.32</u></td><td><u>3.00</u></td><td>0.00</td><td>0.00</td><td><u>76.22</u></td><td><u>28.12</u></td><td><u>30.42</u></td><td><u>1.33</u></td><td><u>4.85</u></td><td>0.27</td><td><u>12.97</u></td><td><u>4.22</u></td><td><u>32.61</u></td><td><u>9.69</u></td><td><u>3.26</u></td><td><u>16.51</u></td></tr><tr><td rowspan="4">SQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>62.65</td><td>0.00</td><td>53.44</td><td>53.87</td><td>0.00</td><td><u>69.00</u></td><td>0.00</td><td>0.00</td><td>63.30</td><td>57.83</td><td>52.70</td><td><u>0.00</u></td><td>53.93</td><td>52.58</td><td>0.00</td><td><u>59.76</u></td><td>54.12</td><td>53.37</td><td>0.00</td><td>36.13</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>59.88</td><td>53.79</td><td>55.17</td><td><u>57.73</u></td><td>55.70</td><td><u>62.50</u></td><td>0.00</td><td>0.00</td><td>65.98</td><td>55.70</td><td>54.53</td><td><u>0.00</u></td><td>52.62</td><td>53.41</td><td>0.00</td><td>55.01</td><td>56.38</td><td><u>57.68</u></td><td>0.00</td><td>41.90</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td><u>66.69</u></td><td><u>57.78</u></td><td><u>65.30</u></td><td>55.30</td><td><u>65.15</u></td><td>61.01</td><td>0.00</td><td>0.00</td><td>68.56</td><td>58.72</td><td>55.81</td><td><u>0.00</u></td><td><u>54.94</u></td><td><u>54.45</u></td><td>51.04</td><td>55.58</td><td>59.86</td><td><u>57.14</u></td><td>49.49</td><td></td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>63.98</td><td>54.46</td><td>60.54</td><td>54.03</td><td>58.55</td><td>52.31</td><td>0.00</td><td>0.00</td><td>70.61</td><td>59.25</td><td>56.69</td><td><u>0.00</u></td><td>53.61</td><td><u>55.70</u></td><td>52.84</td><td>56.05</td><td>58.76</td><td>53.55</td><td><u>56.62</u></td><td>48.29</td><td></td></tr><tr><td></td><td>PaSCo (Ours)</td><td><u>70.10</u></td><td><u>57.84</u></td><td><u>67.00</u></td><td><u>67.33</u></td><td><u>62.15</u></td><td>60.14</td><td>0.00</td><td>0.00</td><td><u>77.52</u></td><td><u>62.62</u></td><td><u>59.95</u></td><td><u>54.71</u></td><td><u>55.87</u></td><td>51.29</td><td><u>52.85</u></td><td><u>57.50</u></td><td><u>63.88</u></td><td>54.78</td><td>55.17</td><td><u>54.25</u></td></tr><tr><td rowspan="4">RQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>15.05</td><td>0.00</td><td>1.42</td><td>4.30</td><td>0.00</td><td>0.67</td><td>0.00</td><td>0.00</td><td>84.56</td><td>3.15</td><td>10.69</td><td><u>0.00</u></td><td>0.48</td><td>0.37</td><td>0.00</td><td>0.45</td><td>6.50</td><td>1.87</td><td>0.00</td><td>6.82</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>15.98</td><td>2.00</td><td>7.59</td><td>30.38</td><td>1.63</td><td>0.20</td><td>0.00</td><td>0.00</td><td>88.59</td><td>9.55</td><td>29.14</td><td><u>0.00</u></td><td>1.93</td><td>2.49</td><td>0.00</td><td>1.39</td><td>24.17</td><td>0.48</td><td>0.00</td><td>11.34</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td><u>27.65</u></td><td><u>8.38</u></td><td>10.29</td><td>8.00</td><td>4.28</td><td><u>2.96</u></td><td>0.00</td><td>0.00</td><td>93.18</td><td>13.50</td><td>35.69</td><td><u>0.00</u></td><td><u>5.66</u></td><td><u>6.03</u></td><td>0.25</td><td>4.12</td><td>36.00</td><td>3.76</td><td>0.30</td><td>13.69</td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>18.32</td><td>3.31</td><td>24.29</td><td>48.94</td><td><u>6.54</u></td><td>0.63</td><td>0.00</td><td>0.00</td><td>94.09</td><td>31.58</td><td><u>44.61</u></td><td><u>0.00</u></td><td>3.84</td><td><u>7.39</u></td><td>0.73</td><td>5.56</td><td>37.84</td><td>7.42</td><td>3.04</td><td>17.80</td></tr><tr><td></td><td>PaSCo (Ours)</td><td><u>35.03</u></td><td><u>13.51</u></td><td><u>27.00</u></td><td><u>66.67</u></td><td><u>18.21</u></td><td><u>4.98</u></td><td>0.00</td><td>0.00</td><td><u>98.32</u></td><td><u>44.91</u></td><td><u>50.73</u></td><td><u>2.44</u></td><td><u>8.69</u></td><td>0.52</td><td><u>24.54</u></td><td><u>7.33</u></td><td><u>51.05</u></td><td><u>17.70</u></td><td><u>5.91</u></td><td><u>25.13</u></td></tr></tbody></table>

(a) Semantic KITTI (val. set)

<table><thead><tr><th></th><th>Method</th><th>car (2.85%)</th><th>bicycle (0.02%)</th><th>motorcycle (0.01%)</th><th>truck (0.16%)</th><th>other-veh. (0.58%)</th><th>person (0.02%)</th><th>road (14.98%)</th><th>parking (2.31%)</th><th>sidewalk (6.43%)</th><th>other-grnd (2.05%)</th><th>building (15.67%)</th><th>fence (0.96%)</th><th>vegetation (41.99%)</th><th>terrain (7.10%)</th><th>pole (0.22%)</th><th>traf.-sign (0.06%)</th><th>other-structure (4.33%)</th><th>other-object (0.28%)</th><th>mean</th></tr></thead><tbody><tr><td rowspan="4">PQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>4.64</td><td><u>0.00</u></td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.67</td><td>49.87</td><td>0.31</td><td>5.75</td><td>0.00</td><td>0.00</td><td>0.00</td><td><u>0.00</u></td><td>1.59</td><td>0.13</td><td>4.24</td><td>0.00</td><td>0.00</td><td>4.14</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>13.77</td><td><u>0.00</u></td><td>0.81</td><td>3.58</td><td>0.48</td><td>1.50</td><td>63.13</td><td>1.63</td><td>23.99</td><td>0.12</td><td>0.14</td><td>0.19</td><td><u>0.00</u></td><td>4.36</td><td>1.51</td><td>6.55</td><td>0.09</td><td>0.44</td><td>6.79</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td><u>17.77</u></td><td><u>0.00</u></td><td><u>2.56</u></td><td>1.45</td><td><u>1.69</u></td><td><u>1.91</u></td><td>46.77</td><td>0.54</td><td>22.06</td><td>0.04</td><td><u>0.14</u></td><td>0.37</td><td><u>0.00</u></td><td><u>5.28</u></td><td>2.02</td><td>7.48</td><td><u>0.14</u></td><td>0.29</td><td>6.14</td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td><u>15.87</u></td><td><u>0.00</u></td><td>1.53</td><td>3.80</td><td>0.97</td><td>1.72</td><td>57.55</td><td><u>3.34</u></td><td><u>30.65</u></td><td><u>0.16</u></td><td>0.04</td><td><u>0.68</u></td><td><u>0.00</u></td><td><u>5.66</u></td><td><u>2.46</u></td><td>9.62</td><td>0.08</td><td>0.26</td><td>7.47</td></tr><tr><td></td><td>PaSCo (Ours)</td><td>14.85</td><td><u>0.11</u></td><td><u>1.73</u></td><td><u>9.01</u></td><td><u>1.97</u></td><td>1.60</td><td><u>72.05</u></td><td>1.47</td><td><u>35.82</u></td><td>0.00</td><td><u>24.29</u></td><td><u>0.53</u></td><td><u>8.96</u></td><td>5.07</td><td><u>3.15</u></td><td><u>14.47</u></td><td><u>0.11</u></td><td><u>1.38</u></td><td><u>10.92</u></td></tr><tr><td rowspan="4">SQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>54.47</td><td><u>0.00</u></td><td>0.00</td><td>0.00</td><td>0.00</td><td><u>67.99</u></td><td>66.89</td><td>58.36</td><td>53.46</td><td>0.00</td><td>0.00</td><td>0.00</td><td><u>0.00</u></td><td>54.17</td><td><u>56.27</u></td><td><u>65.93</u></td><td>0.00</td><td>0.00</td><td>26.52</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>58.34</td><td><u>0.00</u></td><td><u>59.77</u></td><td>52.01</td><td>52.67</td><td>67.64</td><td>69.77</td><td>56.09</td><td>57.48</td><td><u>53.95</u></td><td>52.22</td><td>53.28</td><td><u>0.00</u></td><td>54.62</td><td>54.94</td><td>64.73</td><td><u>55.20</u></td><td>58.21</td><td>51.16</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td><u>60.89</u></td><td><u>0.00</u></td><td><u>61.42</u></td><td>52.88</td><td><u>57.20</u></td><td>58.39</td><td>63.06</td><td>55.22</td><td>56.97</td><td><u>55.56</u></td><td><u>52.45</u></td><td>55.16</td><td><u>0.00</u></td><td>54.96</td><td>54.54</td><td>65.31</td><td><u>61.72</u></td><td>55.60</td><td><u>51.18</u></td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>59.32</td><td><u>0.00</u></td><td>54.73</td><td><u>55.96</u></td><td>53.73</td><td>65.01</td><td>69.17</td><td><u>59.13</u></td><td>58.09</td><td>52.13</td><td>50.13</td><td>55.66</td><td><u>0.00</u></td><td>55.33</td><td>55.10</td><td>63.70</td><td>52.62</td><td>52.18</td><td>50.67</td></tr><tr><td></td><td>PaSCo (Ours)</td><td>58.22</td><td><u>53.03</u></td><td>57.14</td><td><u>55.89</u></td><td><u>55.53</u></td><td>65.39</td><td><u>76.15</u></td><td>58.04</td><td><u>58.51</u></td><td>0.00</td><td><u>56.18</u></td><td><u>61.65</u></td><td><u>55.28</u></td><td><u>57.99</u></td><td><u>56.83</u></td><td><u>67.85</u></td><td>54.74</td><td><u>61.42</u></td><td><u>56.10</u></td></tr><tr><td rowspan="4">RQ</td><td>LMSCNet [61] + MaskPLS [55]</td><td>8.51</td><td><u>0.00</u></td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.99</td><td>74.56</td><td>0.53</td><td>10.75</td><td>0.00</td><td>0.00</td><td>0.00</td><td><u>0.00</u></td><td>2.94</td><td>0.24</td><td>6.43</td><td>0.00</td><td>0.00</td><td>6.45</td></tr><tr><td>JS3CNet [70] + MaskPLS [55]</td><td>23.61</td><td><u>0.00</u></td><td>1.36</td><td>6.88</td><td>0.91</td><td>2.22</td><td>90.49</td><td><u>2.91</u></td><td>41.72</td><td><u>0.22</u></td><td><u>0.28</u></td><td>0.36</td><td><u>0.00</u></td><td>7.98</td><td>2.75</td><td>10.11</td><td>0.15</td><td>0.76</td><td>10.71</td></tr><tr><td>SCPNet [69] + MaskPLS [55]</td><td><u>29.18</u></td><td><u>0.00</u></td><td><u>4.17</u></td><td>2.75</td><td><u>2.95</u></td><td><u>3.27</u></td><td>74.16</td><td>0.98</td><td>38.72</td><td>0.08</td><td><u>0.28</u></td><td>0.61</td><td><u>0.00</u></td><td>9.60</td><td>3.71</td><td>11.45</td><td><u>0.23</u></td><td>0.53</td><td>10.15</td></tr><tr><td>SCPNet* [69] + MaskPLS [55]</td><td>26.76</td><td><u>0.00</u></td><td>2.79</td><td>6.79</td><td>1.81</td><td><u>2.65</u></td><td>83.20</td><td><u>5.65</u></td><td><u>52.77</u></td><td><u>0.30</u></td><td>0.09</td><td><u>1.23</u></td><td><u>0.00</u></td><td><u>10.24</u></td><td><u>4.47</u></td><td><u>15.10</u></td><td>0.16</td><td>0.50</td><td><u>11.92</u></td></tr><tr><td></td><td>PaSCo (Ours)</td><td>25.51</td><td><u>0.21</u></td><td><u>3.03</u></td><td><u>16.13</u></td><td><u>3.55</u></td><td>2.45</td><td><u>94.62</u></td><td>2.53</td><td><u>61.22</u></td><td>0.00</td><td><u>43.24</u></td><td><u>0.86</u></td><td><u>16.21</u></td><td>8.75</td><td><u>5.54</u></td><td><u>21.33</u></td><td><u>0.19</u></td><td><u>2.24</u></td><td><u>17.09</u></td></tr></tbody></table>

(b) SSCBench-KITTI360 (test set)

<caption>Table 8. Class-wise performance on (a) Semantic KITTI (val. set) and (b) SSCBench-KITTI360 (test set) for Panoptic Scene Completion. We report the performance of our method and baselines for each class across the two datasets. Our approach exceeds the performance of baseline methods in most classes, particularly in the crucial PQ metric. The SSCBench-KITTI360 dataset exhibits a higher level of complexity compared to Semantic KITTI, as evidenced by its overall lower performance metrics. PaSCo also shows superior performance in SQ and RQ by being either first or second in most classes.</caption>
<figure><img src="image_12.png" /><figcaption>Figure 9. **Additional qualitative uncertainty comparison on Semantic KITTI and SSCBench-KITTI360.** PaSCo offers more insightful estimates of uncertainty, particularly in smaller instances with incomplete geometry, along the boundaries of segments, in areas with sparse input points, and in extrapolated regions.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Figure 10. **Additional qualitative results on Panoptic Scene Completion.** PaSCo demonstrates improved instance quality, evident from superior geometry (rows 1, 3 to 6), and better separation (rows 2 to 4). Additionally, it predicts more accurate scene structure, with less missing geometry (rows 1, 3, 5, and 6).</figcaption></figure>
## References

[1] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. *Information Fusion*, 2021. 6

[2] Murat Seckin Ayhan and Philipp Berens. Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. In *MIDL*, 2018. 6, 1

[3] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In *ICCV*, 2019. 2, 5, 8

[4] Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In *CVPR*, 2018. 4

[5] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In *ICML*, 2015. 2

[6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multi-modal dataset for autonomous driving. In *CVPR*, 2020. 5

[7] Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. In *CVPR*, 2021. 1, 2

[8] Hui Chen, Man Liang, Wanquan Liu, Weina Wang, and Peter Xiaoping Liu. An approach to boundary detection for 3d point clouds based on dbscan clustering. *Pattern Recognition*, 2022. 5

[9] Xiaokang Chen, Kwan-Yee Lin, Chen Qian, Gang Zeng, and Hongsheng Li. 3d sketch-aware semantic scene completion via semi-supervised structure prior. In *CVPR*, 2020. 1, 2

[10] Yueh-Tung Chen, Martin Garbade, and Juergen Gall. 3d semantic scene completion from a single depth image using adversarial training. In *ICIP*, 2019. 2

[11] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In *NeurIPS*, 2021. 1, 2, 3, 4, 5

[12] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In *CVPR*, 2022. 1, 2, 3, 4

[13] Ran Cheng, Christopher Agia, Yuan Ren, Xinhai Li, and Bingbing Liu. S3cnet: A sparse semantic scene completion network for lidar point clouds. In *CoRL*, 2020. 1, 2, 3, 8

[14] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In *CVPR*, 2019. 1

[15] Manuel Dahnert, Ji Hou, Matthias Nießner, and Angela Dai. Panoptic 3d scene reconstruction from a single rgb image. *Advances in Neural Information Processing Systems*, 34: 8282–8293, 2021. 2

[16] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4578–4587, 2018. 1, 2

[17] Angela Dai, Christian Diller, and Matthias Nießner. Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. In *CVPR*, 2020. 3, 8, 1

[18] Aloisio Dourado, Teofilo E. de Campos, Hansung Kim, and Adrian Hilton. EdgeNet: Semantic scene completion from a single RGB-D image. In *ICPR*, 2020. 2

[19] Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Masksembles for uncertainty estimation. In *CVPR*, 2021. 2

[20] Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-An Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In *ICML*, 2020. 2

[21] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In *KDD*, 1996. 5

[22] Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Severine Dubuisson, and Isabelle Bloch. Tradi: Tracking deep neural network weight distributions for uncertainty estimation. In *ECCV*, 2020. 2

[23] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In *ICML*, 2016. 2, 6, 7, 1

[24] Martin Garbade, Johann Sawatzky, Alexander Richard, and Juergen Gall. Two stream 3d semantic scene completion. In *CVPRW*, 2019. 2

[25] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnn's. In *NeurIPS*, 2018. 2

[26] Stefano Gasperini, Mohammad-Ali Nikouei Mahani, Alvaro Marcos-Ramiro, Nassir Navab, and Federico Tombari. Panoster: End-to-end panoptic segmentation of LiDAR point clouds. *RA-L*, 2021. 2

[27] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. *International Journal of Robotics Research*, 2013. 5

[28] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In *ICML*. PMLR, 2017. 5, 7

[29] Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. Evaluating scalable bayesian deep learning methods for robust computer vision. In *CVPRW*, 2020. 2

[30] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M. Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In *ICLR*, 2021. 1, 2, 4, 5, 8

[31] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In *ICCV*, 2017. 2
[32] Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Lidar-based panoptic segmentation via dynamic shifting network. In CVPR, 2021. 2

[33] Ji Hou, Angela Dai, and Matthias Nießner. Revealnet: Seeing behind objects in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2098–2107, 2020. 2

[34] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get m for free. In ICLR, 2017. 2

[35] Juana Valeria Hurtado, Rohit Mohan, Wolfram Burgard, and Abhinav Valada. Mopt: Multi-object panoptic tracking. In CVPR, 2020. 2

[36] Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An Introduction to Variational Methods for Graphical Models, page 105–161. MIT Press, 1999. 2

[37] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019. 2, 5

[38] Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In ICCV, 2023. 5, 7, 1, 2

[39] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955. 4

[40] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017. 2, 4, 5, 6, 7, 1

[41] Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Marc Martinez, Andrei Bursuc, and Gianni Franchi. Packed-ensembles for efficient uncertainty estimation. In ICLR, 2023. 2, 5

[42] Enxu Li, Ryan Razani, Yixuan Xu, and Bingbing Liu. Smac-seg: Lidar panoptic segmentation via sparse multi-directional attention clustering. In ICRA, 2022. 2

[43] Enxu Li, Ryan Razani, Yixuan Xu, and Bingbing Liu. Cpseg: Cluster-free panoptic segmentation of 3d lidar point clouds. In ICRA, 2023. 2

[44] Jie Li, Kai Han, Peng Wang, Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In CVPR, 2020. 1, 2

[45] Jie Li, Laiyan Ding, and Rui Huang. Imenet: Joint 3d semantic scene completion and 2d semantic segmentation through iterative mutual enhancement. In IJCAI, 2021. 2

[46] Jinke Li, Xiao He, Yang Wen, Yuan Gao, Xiaoqiang Cheng, and Dan Zhang. Panoptic-phnet: Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. In CVPR, 2022. 2

[47] Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, Yue Wang, Hang Zhao, Zhiding Yu, and Chen Feng. Sscbench: Monocular 3d semantic scene completion benchmark in street views. arXiv, 2023. 5, 8

[48] Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. TPAMI, 2022. 5

[49] Minzhe Liu, Qiang Zhou, Hengshuang Zhao, Jianing Li, Yuan Du, Kurt Keutzer, Li Du, and Shanghang Zhang. Prototype-voxel contrastive learning for lidar point cloud panoptic segmentation. In ICRA, 2022. 2

[50] Shice Liu, Yu Hu, Yiming Zeng, Qiankun Tang, Beibei Jin, Yinhe Han, and Xiaowei Li. See and Think: Disentangling Semantic Scene Completion. In NeurIPS, 2018. 2

[51] Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. In 2022, 2022. 2

[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5

[53] David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Computation, 1992. 2

[54] Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In NeurIPS, 2019. 2

[55] R. Marcuzzi, L. Nunes, L. Wiesmann, J. Behley, and C. Stachniss. Mask-Based Panoptic LiDAR Segmentation for Autonomous Driving. RA-L, 2023. 2, 3, 4, 5, 6

[56] Jianbiao Mei, Yu Yang, Mengmeng Wang, Zizhang Li, Xiaojun Hou, Jongwon Ra, Laijian Li, and Yong Liu. Centerlps: Segment instances by centers for lidar panoptic segmentation. In ACM MM, 2023. 2

[57] Andres Milioto, Jens Behley, Chris McCool, and Cyrill Stachniss. Lidar panoptic segmentation for autonomous driving. In IROS, 2020. 2

[58] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In NeurIPS, 2019. 2

[59] Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In CVPR, 2019. 5

[60] Ryan Razani, Ran Cheng, Enxu Li, Ehsan Taghavi, Yuan Ren, and Liu Bingbing. Gp-s3net: Graph-based panoptic sparse semantic segmentation network. In ICCV, 2021. 2

[61] Luis Roldão, Raoul de Charette, and Anne Verroust-Blondet. Lms網: Lightweight multiscale 3d semantic completion. In 3DV, 2020. 1, 2, 4, 5, 6

[62] Luis Roldão, Raoul De Charette, and Anne Verroust-Blondet. 3D Semantic Scene Completion: a Survey. IJCV, 2021. 1, 2, 3

[63] Kshitij Sirohi, Rohit Mohan, Daniel Büscher, Wolfram Burgard, and Abhinav Valada. Efficientlps: Efficient lidar panoptic segmentation. T-RO, 2021. 2

[64] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In CVPR, 2017. 1, 2

[65] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 5
[66] Yida Wang, David Joseph Tan, Nassir Navab, and Federico Tombari. ForkNet: Multi-branch volumetric semantic completion from a single depth image. In *ICCV*, 2019. 2

[67] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. In *ICLR*, 2020. 2, 4

[68] Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In *NeurIPS*, 2020. 2

[69] Zhaoyang Xia, Youquan Liu, Xin Li, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, and Yu Qiao. Scpnet: Semantic scene completion on point cloud. In *CVPR*, 2023. 1, 2, 4, 5, 6

[70] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In *AAAI*, 2021. 1, 2, 4, 5, 6

[71] Jiahui Zhang, Hao Zhao, Anbang Yao, Yurong Chen, Li Zhang, and Hongen Liao. Efficient semantic scene completion network with spatial group convolution. In *ECCV*, 2018. 2

[72] Pingping Zhang, Wei Liu, Yinjie Lei, Huchuan Lu, and Xi-aoyun Yang. Cascaded context pyramid for full-resolution 3d semantic scene completion. In *ICCV*, 2019. 2

[73] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In *CVPR*, 2021. 1