$\alpha$-OCC: Uncertainty-Aware Camera-based 3D Semantic Occupancy Prediction

## Abstract

Abstract In the realm of autonomous vehicle (AV) perception, comprehending 3D scenes is paramount for tasks such as planning and mapping. Camera-based 3D Semantic Occupancy Prediction (OCC) aims to infer scene geometry and semantics from limited observations. While it has gained popularity due to affordability and rich visual cues, existing methods often neglect the inherent uncertainty in models. To address this, we propose an uncertainty-aware camera-based 3D semantic occupancy prediction method ( α 𝛼 \alpha italic_α -OCC). Our approach includes an uncertainty propagation framework (Depth-UP) from depth models to enhance geometry completion (up to 11.58% improvement) and semantic segmentation (up to 12.95% improvement) for a variety of OCC models. Additionally, we propose a hierarchical conformal prediction (HCP) method to quantify OCC uncertainty, effectively addressing the high-level class imbalance in OCC datasets. On the geometry level, we present a novel KL-based score function that significantly improves the occupied recall of safety-critical classes (45% improvement) with minimal performance overhead (3.4% reduction). For uncertainty quantification, we demonstrate the ability to achieve smaller prediction set sizes while maintaining a defined coverage guarantee. Compared with baselines, it reduces up to 92% set size. Our contributions represent significant advancements in OCC accuracy and robustness, marking a noteworthy step forward in autonomous perception systems.

## 1 Introduction

Achieving a comprehensive understanding of 3D scenes is crucial for downstream tasks such as planning and map construction in autonomous vehicles (AVs) and robotics (Wang & Huang, ). 3D Semantic Occupancy Prediction (OCC) emerges as a solution that jointly infers the geometry completion and semantic segmentation from limited observations (Song et al., ; Hu et al., ), which is also known as 3D semantic scene completion. OCC approaches typically fall into two categories based on the sensors they use: LiDAR-based OCC and camera-based OCC. While LiDAR sensors offer precise depth information (Roldao et al., ; Cheng et al., ), they are costly and less portable. Conversely, cameras, with their affordability and ability to capture rich visual cues of driving scenes, have gained significant attention (Cao & De Charette, ; Li et al., ; Tian et al., ; Zhang et al., ). For camera-based OCC, depth prediction is essential for the accurate 3D reconstruction of scenes. However, existing methodologies often ignore errors inherited from depth models in real-world scenarios (Poggi et al., ). Moreover, how to utilize the propagated depth uncertainty information and rigorously quantify the uncertainty of the final OCC outputs, especially when a high-level class imbalance exists in OCC datasets, remains challenging and unexplored. In the rest of this paper, OCC is referred to as camera-based OCC unless otherwise specified, which is the focus of our work.

Figure: Figure 1: (a): Influence of depth estimation uncertainty on the accuracy of OCC (mIoU$\uparrow$). As the percentage of depth uncertaintiy increases, the accuracy of OCC decreases significantly. (b): Example: the influence of high class imbalance on OCC. The percentage next to each class is its percentage in the SemanticKITTI dataset. Since the safety-critical class “bicyclist” only occupied 0.01%, the trained OCC model fails to detect the bicyclist in front, leading to a crash. However, after quantifying the uncertainty and post-processing using our HCP, the crash is avoided. This is because our HCP improves the occupied recall of rare classes. Due to visualization constraints, each occupied voxel is represented by the nonempty class with the highest probability in our HCP results.
Refer to caption: x1.png

We explain the importance of considering depth uncertainty propagation and OCC uncertainty quantification in Fig. 1. The influence of depth estimation uncertainty on OCC accuracy is shown in Fig. 1(a). We introduced perturbations to the ground-truth depth values by multiplying them by a factor of $(1+\beta\%),\ \forall\beta\in\{0\%,2\%,4\%,6\%,8\%,10\%,20\%\}$, simulating real-world depth estimation uncertainties. Uncertainties of depth estimation significantly reduce the performance of OCCs, which should be considered in OCCs.
In this paper, we propose a flexible uncertainty propagation framework (Depth-UP) from depth models to improve the performance of a variety of OCC models.

The datasets utilized in OCC tasks often exhibit a high class imbalance, with empty voxels comprising a significant proportion (92.91% for the widely used SemanticKITTI (Behley et al., ) dataset), as illustrated in the dotted box of Fig. 1(b). Bicyclist voxels and person voxels, crucial for safety, only occupy 0.01% and 0.007%. Consequently, neural networks trained on such imbalanced data, coupled with the maximum posterior classification, may inadvertently disregard infrequent classes within the dataset (Tian et al., ). This leads to reduced accuracy and recall for rare classes. However, for safety-critical systems such as autonomous vehicles (AV), ensuring occupied recall for rare classes is important for preventing potential collisions and accidents (Chan et al., ). As shown in Fig. 1(b), the basic OCC model fails to detect the bicyclist in front and causes a crash for the bicyclist class is very rare in the dataset. To address this problem, we propose a hierarchical conformal prediction (HCP) method that improves the occupied recall of rare classes for geometry completion and generates prediction sets for predicted occupied voxels with class coverage guarantees for semantic segmentation. So after quantifying the uncertainty and post-processing using our HCP, the OCC model detects the voxels of the rare bicyclist class and avoids the crash.

Through extensive experiments on two OCC models (VoxFormer Li et al. () and OccFormer Zhang et al. ()) and two datasets (SemanticKITTI Behley et al. () and KITTI360 Li et al. ()), we show that our Depth-UP achieves up to 11.58% increase in geometry completion and 12.95% increase in semantic segmentation. Our HCP achieves 45% increase in the geometry prediction for the person class, with only 3.4% IoU overhead. This improves the prediction of rare safety-critical classes, such as persons and bicyclists, thereby reducing potential risks for AVs. Compared with baselines, our HCP reduces up to 92% set size and up to 84% coverage gap. These results highlight the significant improvements in both accuracy and uncertainty quantification offered by our $\alpha$-OCC approach.

Our contributions can be summarized as follows:

- 1.
To address the challenging OCC problem for autonomous driving, we recognize the problem from a fresh uncertainty quantification (UQ) perspective. More specifically, we propose the uncertainty-aware camera-based 3D semantic occupancy prediction method ($\alpha$-OCC), which contains the uncertainty propagation (Depth-UP) from depth models to improve OCC performance and the novel hierarchical conformal prediction (HCP) method to quantify the uncertainty of OCC.
- 2.
To the best of our knowledge, we are the first attempt to propose the uncertainty propagation framework Depth-UP to improve the OCC performance, where the uncertainty quantified by the direct modeling is utilized on both geometry completion and semantic segmentation. This leads to a solid improvement in common OCC models.
- 3.
To solve the high-level class imbalance challenge on OCC, which results in biased prediction and low recall for rare classes, we propose the HCP. On geometry completion, a novel KL-based score function is proposed to improve the occupied recall of safety-critical classes with little performance overhead. For uncertainty quantification, we achieve a smaller prediction set size under the defined class coverage guarantee. Overall, the proposed $\alpha$-OCC, combined with Depth-UP and HCP, has shown that UQ is an integral and vital part of OCC tasks, with an extendability over to a broader set of 3D scene understanding tasks that go beyond the AV perception.

## 2 Related Work

Semantic Occupancy Prediction. The concept of 3D Semantic Occupancy Prediction (OCC), which is also known as 3D semantic scene completion, was first introduced by SSCNet (Song et al., ), integrating both geometric and semantic reasoning. Since its inception, numerous studies have emerged, categorized into two streams: LiDAR-based OCC (Roldao et al., ; Cheng et al., ; Yan et al., ) and camera-based OCC (Cao & De Charette, ; Li et al., ; Tian et al., ; Zhang et al., ; Huang et al., ; Tang et al., ; Vobecky et al., ). Recently, camera-based OCC has gained increasing attention owing to cameras’ advantages in visual recognition and cost-effectiveness (Ma et al., ). Depth predictions are instrumental in projecting 2D information into 3D space for camera-based OCC tasks. Existing approaches generate query proposals using depth estimation and leverage them to extract rich visual features from the 3D scene. However, they overlook depth estimation uncertainty. In this work, we propose an uncertainty propagation framework from depth models to enhance the performance of OCC models.

Uncertainty Quantification and Propagation. Uncertainty quantification (UQ) holds paramount importance in ensuring the safety and reliability of autonomous systems such as robots (Jasour & Williams, ) and AVs (Meyer & Thakurdesai, ). Moreover, UQ for perception tasks can significantly enhance the planning and control processes for safety-critical autonomous systems (Xu et al., ; He et al., ). Different types of UQ methods have been proposed. Monte-Carlo dropout (Miller et al., ) and deep ensemble (Lakshminarayanan et al., ) methods require multiple runs of inference, which makes them infeasible for real-time UQ tasks. In contrast, direct modeling methods (Feng et al., ) can estimate uncertainty in a single inference pass in real-time perception, which is used to estimate the uncertainty of depth in our work.

Several studies have integrated uncertainty into 3D tasks, but their objectives differ from ours. Eldesokey et al. () improves 3D depth completion with uncertainty by normalized convolutional neural networks. Cao et al. () used a deep ensemble method to manage uncertainty for LiDAR-based OCC, which increases computational complexity. While uncertainty propagation (UP) frameworks from depth to 3D object detection have demonstrated efficacy in enhancing accuracy (Lu et al., ; Wang et al., ), no prior works have addressed UP from depth to OCCs for improving the performance of OCCs. This paper aims to bridge this gap by proposing a novel approach to UP. We design a depth UP module called Depth-UP based on direct modeling.

Conformal prediction (CP) can construct statistically guaranteed uncertainty sets for model predictions (Angelopoulos & Bates, ; Su et al., ; Manokhin, ), however, there is limited CP literature for highly class-imbalanced tasks. Rare and safety-critical classes (e.g., person) remain challenging for OCC models. Hence, we develop a hierarchical conformal prediction method to quantify uncertainties of OCC characterized by highly imbalanced classes. More related works are introduced in Appendix A.1 and A.4.

## 3 Method

We design a novel uncertainty-aware camera-based 3D semantic occupancy prediction method ($\alpha$-OCC), which contains the uncertainty propagation (Depth-UP) from depth models to improve the performance of different OCC models and the hierarchical conformal prediction (HCP) to quantify the uncertainty of OCC. Figure 2 presents the whole methodology overview and the structure of our Depth-UP. Figure 3 presents the structure of our HCP. The major novelties are: (1) Depth-UP quantifies the uncertainty of depth estimation by direct modeling (DM) and then propagates it through probabilistic geometry projection (for geometry completion) and depth feature extraction (for semantic segmentation). (2) HCP calibrates the probability outputs of the OCC model. First, it predicts the voxels’ occupied state by the quantile on the novel KL-based score function as Eq. 4, which can improve the occupied recall of rare safety-critical classes. Then it generates prediction sets for predicted occupied voxels, achieving a better coverage guarantee and smaller sizes of prediction sets.

Figure: Figure 2: Overview of our $\alpha$-OCC method. The non-black colors highlight the novelties and important techniques in our method. In the Depth-UP part, we calculate the uncertainty of depth estimation through direct modeling. Then we propagate it through depth feature extraction (for semantic segmentation) and building a probabilistic voxel grid map $M_{p}$ by probabilistic geometry projection (for geometry completion). Each element of $M_{p}$ is the occupied probability of the corresponding voxel, computed by considering the depth distribution of all rays across the voxel.
Refer to caption: x2.png

### 3.1 Preliminary

OCC predicts a dense semantic scene within a defined volume in front of the vehicle solely from RGB images (Cao & De Charette, ) as shown in Figure 2. Specifically, with an input image denoted by $\mathbf{X}\in\mathbb{R}^{3\times H\times W}$, one OCC model first extracts 2D image features $\mathbf{F}_{I}$ using backbone networks like ResNet (He et al., ) and estimates the depth value for each pixel, denoted by $\hat{\mathbf{D}}\in\mathbb{R}^{H\times W}$, employing depth models such as monocular depth estimation (Bhat et al., ) or stereo depth estimation (Shamsafar et al., ). Subsequently, the model generates a probability voxel grid $\hat{\mathbf{Y}}\in[0,1]^{M\times U\times V\times D}$ based on $\mathbf{F}_{I}$ and $\hat{\mathbf{D}}$, assigning each voxel to the class with the highest probability. Each voxel within the grid is categorized as either empty or occupied by a specific semantic class. The ground truth voxel grid is denoted as $\mathbf{Y}$. Here, $H$ and $W$ signify the height and width of the input image, while $U$, $V$ and $D$ represent the height, width, and length of the voxel grid, $M$ denotes the total number of relevant classes (including the empty class), respectively.

### 3.2 Uncertainty Propagation Framework (Depth-UP)

In contemporary OCC methods, depth models facilitate the projection from 2D to 3D space, primarily focusing on geometric aspects. Nonetheless, these approaches often overlook the inherent uncertainty associated with depth prediction. Recognizing the potential to enhance OCC performance by harnessing this uncertainty, we introduce a novel framework (Depth-UP) centered on uncertainty propagation from depth models to OCC models. Our Depth-UP is a flexible framework applicable to a variety of OCC models. It involves quantifying the uncertainty inherent in depth models through a direct modeling (DM) method and integrating this uncertainty information into both geometry completion and semantic segmentation of OCC to improve the final performance.

Direct Modeling (DM).
Depth-UP includes a DM technique (Su et al., ; Feng et al., ) to infer the standard deviation associated with the estimated depth value of each pixel in the image, with little time overhead. An additional regression header, with a comparable structure as the original regression header for $\hat{\mathbf{D}}$, is tailored to predict the standard deviation $\hat{\mathbf{\Sigma}}$. Subsequently, this header is retrained based on the pre-trained depth model. We assume that the estimated depth value is represented as a single-variate Gaussian distribution, and the ground truth depth follows a Dirac delta function (Arfken et al., ). For the retraining process, we define the regression loss function as the Kullback-Leibler (KL) divergence between the estimated distribution and the ground truth distribution, where $\textbf{D}\in\mathbb{R}^{H\times W}$ is the ground truth depth matrix for the image:
$\mathcal{L}_{KL}(\mathbf{D},\hat{\mathbf{D}},\hat{\mathbf{\Sigma}})=\frac{1}{
HW}\sum_{h=1}^{H}\sum_{w=1}^{W}\frac{(d_{hw}-\hat{d}_{hw})^{2}}{2\hat{\sigma}_
{hw}^{2}}+\log|\hat{\sigma}_{hw}|.$

Propagation on Geometry Completion.
Depth information is used to generate the 3D voxels on geometry in OCC. There are two key challenges: lens distortion during geometric transformations and occupied probability estimation for each voxel. Lens distortion is a deviation from the ideal image formation by a lens, resulting in a distorted image (Zhang, ). Existing OCC models, such as VoxFormer (Li et al., ), solve the lens distortion by projecting depth into a 3D point cloud, and then generating the binary voxel grid map $\mathbf{M}_{b}\in\{0,1\}^{U\times V\times D}$, where each voxel is marked as 1 if occupied by at least one point. However, they ignore the uncertainty of depth. Here we propagate the depth uncertainty into the geometry of OCC to solve the above two challenges.

Our Depth-UP generates a probabilistic voxel grid map $\mathbf{M}_{p}\in[0,1]^{U\times V\times D}$ that considers lens distortion and depth uncertainty, with $\{\hat{\mathbf{D}},\hat{\mathbf{\Sigma}}\}$ from DM. For pixel $(h,w)$ with estimated depth mean $\hat{d}_{hw}$, we project it into point $(x,y,z)$ in 3D space:
$x=\frac{(h-c_{h})\times z}{f_{u}},y=\frac{(w-c_{w})\times z}{f_{v}},z=\hat{d}_
{hw}$, where $(c_{u},c_{v})$ is the camera center and $f_{u}$ and $f_{v}$ are the horizontal and vertical focal length.

When the estimated depth follows a single-variate Gaussian distribution, the location of the point may be on any position along a ray starting from the camera. It is difficult to get the exact location of the point, but we can estimate the probability of one voxel $(u,v,d)$ being occupied by points. Due to the density of visual information, a single voxel may correspond to multiple pixels, which means a voxel can be passed by multiple rays. We denote this set of rays as $\Psi_{uvd}$, and a single ray within this set as $\rho_{hw}$, corresponding to pixel $(h,w)$. When a ray $\rho_{hw}$ passes through a voxel, it has two crosspoints: $z_{s}$ where the ray enters the voxel, and $z_{e}$ where the ray exits the voxel. By cumulating the probability of the ray inside the voxel using the probability density function, we obtain the probability of voxel $(u,v,d)$ being occupied by points:

$$ $\displaystyle\vspace*{-8pt}\mathbf{M}_{p}(u,v,d)=\min\left(1,\sum_{\rho_{hw} \in\Psi_{uvd}}\int_{z_{s}}^{z_{e}}\mathcal{N}(z|\hat{d}_{hw},\hat{\sigma}_{hw} ^{2})dz\right).$ (1) $$

The original binary voxel grid map is replaced by the probabilistic voxel grid map $\mathbf{M}_{p}\in[0,1]^{U\times V\times D}$ to propagate the depth uncertainty into the geometry completion of OCC.

Propagation on Semantic Segmentation. The extraction of 2D features $\mathbf{F}_{I}$ from the input image has been a cornerstone for OCC to encapsulate semantic information. However, harnessing the depth uncertainty information on the semantic features is ignored. Here by augmenting the architecture with an additional lightweight backbone, such as ResNet-18 backbone (He et al., ), we extract depth features $\mathbf{F}_{D}$ from the concatenated depth mean and standard deviation $\{\hat{\mathbf{D}},\hat{\mathbf{\Sigma}}\}$. These newly acquired depth features are then seamlessly integrated with the original 2D image features, constituting a novel set of input features $\{\textbf{F}_{I},\textbf{F}_{D}\}$ as shown in Figure 2. This integration strategy capitalizes on the extensive insights gained from prior depth predictions, enhancing the OCC performance with enhanced semantic understanding.

### 3.3 Hierarchical Conformal Prediction (HCP)

#### 3.3.1 Preliminary

Standard Conformal Prediction. For classification, conformal prediction (CP) (Angelopoulos & Bates, ; Ding et al., ) is a statistical method to post-process any models by producing the set of predictions with theoretically guaranteed marginal coverage of the correct class. With $M$ classes, consider the calibration data $(\mathbf{X}_{1},\mathbf{Y}_{1}),...,(\mathbf{X}_{N},\mathbf{Y}_{N})$ with N data points that are never seen during training, the standard CP (SCP) includes the following steps: (1) Define the score function $s(\mathbf{X},y)\in\mathbb{R}$. (Smaller scores indicate better agreement between $\mathbf{X}$ and $y$). The score function is a vital component of CP. A typical score function of a classifier $f$ is $s(\mathbf{X},y)=1-f(\mathbf{X})_{y}$, where $f(\mathbf{X})_{y}$ represents the $y^{th}$ softmax output of $f(\mathbf{X})$. (2) Compute $q$ as the $\frac{\lceil(N+1)(1-\alpha)\rceil}{N}$ quantile of the calibration scores, where $\alpha\in[0,1]$ is a user-chosen error rate. (3) Use this quantile to form the prediction set $\mathcal{C}(\mathbf{X}_{test})\subset\{1,...,M\}$ for one new example $\mathbf{X}_{test}$ (from the same distribution of the calibration data): $\mathcal{C}(\mathbf{X}_{test})=\{y:s(\mathbf{X}_{test},y)\leq q\}$.
The SCP provides a coverage guarantee that $\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test}))\geq 1-\alpha$ which has been proved in Angelopoulos & Bates ().

Class-Conditional Conformal Prediction. The SCP achieves the marginal guarantee but may neglect the coverage of some classes, especially on class-imbalanced datasets (Angelopoulos & Bates, ). Class-Conditional Conformal Prediction (CCCP) targets class-balanced coverage under the user-chosen class error rate $\alpha^{y}$:

$$ $\displaystyle\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y)\geq 1-\alpha^{y},\ \forall y\in\{1,...,M\}.$ (2) $$

Every class $y$ has at least $1-\alpha^{y}$ probability of being included in the prediction set when the label is $y$. Hence, the prediction sets satisfying Eq. 2 are effectively fair to all classes, even the rare ones.

#### 3.3.2 Our Hierarchical Conformal Prediction

Figure: Figure 3: Overview of our Hierarchical Conformal Prediction (HCP) module. We predict voxels’ occupied state by the quantile on the novel KL-based score as Eq. 4, which can improve occupied recall of rare classes, and then only generate prediction sets for these predicted occupied voxels. The occupied quantile $q_{o}^{y}$ and semantic quantile $q_{s}^{y}$ are computed during the calibration step of HCP.
Refer to caption: x3.png

Current CP does not consider the hierarchical structure of classification, such as the geometry completion and semantic segmentation in OCCs. And it cannot achieve good coverage for very rare and safety-critical classes. Here we propose a novel hierarchical conformal prediction (HCP) to address these challenges, which is shown in Figure 3. The detailed algorithm is shown in Appendix A.3.

Geometric Level. On the geometric level, it is important and safety-critical to guarantee the occupied recall of some sensitive classes, such as the person and bicyclist for AVs. Hence, we define the occupied coverage for the specific safety-critical class $y$ as:

$$ $\displaystyle\mathbb{P}(o=T|\mathbf{Y}_{test}=y)\geq 1-\alpha_{o}^{y},$ (3) $$

where $o=T$ means the occupancy state is true. The probability of the voxels with label $y$ are predicted as occupied is guaranteed to be no smaller than $1-\alpha_{o}^{y}$. The empty class is $y=1$ and occupied classes are $y\in\{2,...,M\}$. To achieve the above guarantee under the high class-imbalanced dataset, we propose a novel score function based on the KL divergence. Here we define the ground-truth distribution for occupancy as $\mathbf{O}=\{\varepsilon,1,...,1\}^{M}$, where $\varepsilon$ is the minimum value for the empty class to avoid the divide-by-zero problem. With the output softmax probability $f(\mathbf{X})=\{p_{1},p_{2},...,p_{M}\}$ from the model $f$, we define the KL-based score function for $y\in\mathcal{Y}_{r}$:

$$ $\displaystyle s_{kl}(\mathbf{X},y)=\mathrm{D}_{kl}(f(\mathbf{X})||\mathbf{O})= p_{1}\log(\frac{p_{1}}{\varepsilon})+\sum_{i=2}^{M}p_{i}\log(p_{i}),$ (4) $$

where $\mathcal{Y}_{r}$ is the considered rare class set. The quantile $q^{y}_{o}$ for class $y$ is computed as the $\frac{\lceil(N_{y}+1)(1-\alpha_{o}^{y})\rceil}{N_{y}}$ quantile of the score $s_{kl}(\mathbf{X},y)$ on $\Upsilon^{y}$, where $\Upsilon^{y}$ is the subset of the calibration dataset with $\mathbf{Y}=y$ and $N_{y}=|\Upsilon^{y}|$. Then we predict the voxel $\mathbf{X}_{test}$ as occupied if $\exists y\in\mathcal{Y}_{r},s_{kl}(\mathbf{X}_{test},y)\leq q^{y}_{o}$.

Semantic Level. On the semantic level, we need to achieve the same class-balanced coverage as Eq. 2, under the geometric level coverage guarantee. For all voxels that are predicted as occupied in the previous step, we generate the prediction set $\mathcal{C}(\mathbf{X}_{test})\subset\{2,...,M\}$ to satisfy the guarantee:

$$ $\displaystyle\mathbb{P}(\mathbf{Y}_{text}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{text}=y,o=T)\geq 1-\alpha_{s}^{y}.$ (5) $$

The score function here is $s(\mathbf{X},y)=1-f(\mathbf{X})_{y}$. We compute the quantile $q^{y}_{s}$ for class $y$ as the $\frac{\lceil(N_{yo}+1)(1-\alpha)\rceil}{N_{yo}}$ quantile of the score on $\Upsilon^{y}_{o}$, where $\Upsilon^{y}_{o}$ is the subset of the calibration dataset that has label $y$ and are predicted as occupied on the geometric level of our HCP. $N_{yo}=|\Upsilon^{y}_{o}|$.

The prediction set is generated as:

$$ $\displaystyle\mathcal{C}(\mathbf{X}_{test})=\{y:s_{kl}(\mathbf{X},y)\leq q^{y} _{o}\wedge s(\mathbf{X},y)\leq q^{y}_{s}\}$ (6) $$

###### Proposition 1 .

For a desired $\alpha^{y}$ value, we select $\alpha_{o}^{y}$ and $\alpha_{s}^{y}$ as $1-\alpha^{y}=(1-\alpha_{s}^{y})(1-\alpha_{o}^{y})$, then the prediction set generated as Eq. 6 satisfies $\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})|\mathbf{Y}_{test
}=y)\geq 1-\alpha^{y}$.

The proof is in Appendix A.2.

###### Proposition 1 .

## 4 Experiments

OCC Model. We assess the effectiveness of our approach through comprehensive experiments on two different OCC models VoxFormer (Li et al., ) and OccFormer (Zhang et al., ). A detailed introduction to these two models is in Appendix A.4.

Dataset. The datasets we used are SemanticKITTI (Behley et al. (), with 20 classes) and KITTI360 (Li et al. (), with 19 classes). More details on these two datasets are in Appendix A.5 and detailed experiment settings are in Appendix A.6.

### 4.1 Uncertainty Propagation Performance

**Table 1: Performance evaluation of our Depth-UP on two OCC models. Values in parentheses indicate the improvement of our Depth-UP compared with the baseline.**
| Dataset | Basic OCC | Method | IoU $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | mIoU $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SemanticKITTI | VoxFormer | Base | 44.02 | 62.32 | 59.99 | 12.35 |
| Our | 45.85 (+1.83) | 63.10 (+0.78) | 62.64 (+2.65) | 13.36 (+1.01) |  |  |
| OccFormer | Base*^1 | 36.50 | - | - | 13.46 |  |
| Base | 37.48 | 48.71 | 61.92 | 12.83 |  |  |
| Our | 41.64 (+4.16) | 53.99 (+5.28) | 64.54 (+2.62) | 14.56 (+1.73) |  |  |
| KITTI360 | VoxFormer | Base | 38.76 | 57.67 | 54.18 | 11.91 |
| Our | 43.25 (+4.49) | 65.81 (+7.29) | 55.78 (+2.34) | 13.55 (+1.64) |  |  |

Metric. For OCC performance, we employ the intersection over union (IoU) to evaluate the geometric completion, regardless of the allocated semantic labels. This is very crucial for obstacle avoidance for AVs. We use the mean IoU (mIoU) of all semantic classes to assess the performance of semantic segmentation of OCC. Since there is a strong negative correlation between IoU and mIoU (Li et al., ), the model should achieve excellent performance in both of them.

The experimental results of our Depth-UP on VoxFormer and OccFormer are presented in Table 1. Since the existing OccFormer is not implemented on the KITTI360 dataset (Zhang et al., ), we only evaluate the OccFormer with our Depth-UP on the SemanticKITTI dataset. These results demonstrate that Depth-UP effectively leverages quantified uncertainty from the depth model to enhance OCC model performance, achieving up to a 4.49 (11.58%) improvement in IoU and up to a 1.73 (12.95%) improvement in mIoU, while also significantly improving both precision and recall in the geometry completion aspect of OCC. When assessing the performance of OCC models, even slight improvements in IoU and mIoU mean good progress (Zhang et al., ; Huang et al., ). The detailed mIoU results of each class are presented in Appendix A.7.

Figure 4 presents visualizations of the VoxFormer with and without our Depth-UP on SemanticKITTI. In this figure, we can also see that our Depth-UP can help OCC models predict rare classes, such as persons and bicyclists, as highlighted with the orange dashed boxes. Especially for the third row, our Depth-UP predicts the person crossing the road in the corner, while the baseline ignores him. Our Depth-UP can significantly reduce the risk of hurting humans for AVs and improve safety. More visualization results are in Appendix A.7.

Figure: Figure 4: Qualitative results of the base VoxFormer model and that with our Depth-UP.
Refer to caption: x4.png

### 4.2 Uncertainty Quantification Performance

We evaluate our HCP on the geometric level and the final uncertainty quantification. Since we do not have the labeled test part of SemanticKITTI, we randomly split the original validation part of SemanticKITTI into the calibration dataset (take up 30%) and the test dataset (take up 70%). For KITTI360, we use the validation part as the calibration dataset and the test part as the test dataset.

Geometric Level. For the geometric level, the target of methods is to achieve the best trade-off between IoU performance and the occupied recall of rare classes. To show the effectiveness of our novel KL-based score function on the geometric level, we compare it with two common score functions in Angelopoulos & Bates (): class score ($1-f(\textbf{X})_{y}$) and occupied score ($1-\sum_{y=2}^{M}f(\textbf{X})_{y}$). Figure 5(a) shows the IoU results across different occupied recalls of the rare class person for different datasets. Figure 5(b) shows the IoU results across different occupied recalls of the rare class bicyclist for different basic OCC models. Here “Our Depth-UP” means the basic OCC model with our Depth-UP method. We can see that our KL-based score function always achieves the best geometry performance for the same occupied recall, compared with two baselines.

Our HCP significantly outperforms baselines because it not only considers the occupied probability across all nonempty classes but also leverages the entire probability distribution. Compared with the class score, which only considers individual class probabilities, our score function accounts for all nonempty classes. Predicting rare classes is challenging for models, but they tend to identify these as occupied, assigning lower probabilities to the empty class and higher probabilities to all nonempty classes. Therefore, it’s crucial to consider the probability of all nonempty classes. Although the occupied score addresses this by summing probabilities of all nonempty classes, it loses sensitivity to the distribution. When facing difficult classifications (such as rare classes), deep learning models tend to produce output probabilities that are more evenly distributed across the possible classes (Guo et al., ). The Kullback-Leibler (KL) divergence measures how one probability distribution diverges from a reference distribution, considering the entire shape of the probability distribution (Raiber & Kurland, ). This sensitivity to distribution shape enables our KL-based score function to identify rare classes more effectively.

To achieve the optimal balance between IoU and occupied recall, we can adjust the desired occupied recall. For instance, in the top right subfigure of Figure 5(a), the OCC model without HCP shows an IoU of 45.85 and an occupied recall for the person class of 20.69. By setting the occupied recall to 21.75, the IoU improves to 45.94. Increasing the occupied recall beyond 30 ($45.0\%$ improvement) results in a decrease in IoU to 44.38 (3.4% reduction). This demonstrates that our HCP method can substantially boost the occupied recall of rare classes with a minor reduction in IoU.

Figure: Figure 5: Compare our KL-based score function with the class score function and the occupied score function. Evaluate OCC’s geometry performance across different occupied recalls of the rare class (person or bicyclist). The red dotted line shows the IoU of the OCC model without CP. (a): Results on basic VoxFormer across different datasets for the considered class person. (b): Results on SemanticKITTI across different basic OCC models for the considered class bicyclist.
Refer to caption: x5.png

Uncertainty Quantification. To measure the quantified uncertainty of different CP methods, we usually use the average class coverage gap (CovGap) and average set size (AvgSize) of the prediction sets (Ding et al., ) as metrics. For a given class $y\in\mathcal{Y}\backslash\{1\}$ with the defined error rate $\alpha^{y}$, the empirical class-conditional coverage of class $y$ is $c_{y}=\frac{1}{|\Upsilon^{y}|}\sum_{i\in\Upsilon^{y}}\mathbb{I}\{\mathbf{Y}_{i
}\in\mathcal{C}(\textbf{X}_{i})\}$. The CovGap is defined as $\frac{1}{|\mathcal{Y}|-1}\sum_{y\in\mathcal{Y}\backslash\{1\}}|c_{y}-(1-\alpha
^{y})|$. This measures how far the class-conditional coverage is from the desired coverage $1-\alpha^{y}$. The AvgSize is defined as $\frac{1}{T}\sum_{i=1}^{T}|\mathcal{C}(\mathbf{X}_{i})|$, where $T$ is the number of samples in the test dataset and $\mathcal{C}(\mathbf{X}_{i})$ does not contain the empty class. A good UQ method should achieve both small CovGap and AvgSize.

Table 2 compares our HCP method with standard conformal prediction (SCP) and class-conditional conformal prediction (CCCP), as introduced in Subsection 3.3.1. Our results demonstrate that HCP consistently achieves robust empirical class-conditional coverage and produces smaller prediction sets. In contrast, the performance of SCP and CCCP varies across different OCC models. Specifically, for our Depth-UP based on VoxFormer and KITTI360, HCP reduces the set size by 92% and the coverage gap by 84%, compared to SCP. For our Depth-UP based on VoxFormer and SemanticKITTI, HCP reduces the set size by 79% and the coverage gap by 64%, compared to CCCP. As noted in Subsection 3.3.1, SCP consistently fails to provide conditional coverage, although sometimes it provides a very small set size. Both SCP and CCCP tend to generate nonempty $C(\mathbf{X})$ for most voxels, potentially obstructing AVs. In contrast, HCP only generates nonempty $C(\mathbf{X})$ for these selected occupied voxels, thereby minimizing prediction set sizes while maintaining reliable class-conditional coverage.

**Table 2: Compare our HCP (referred to as “Ours”) with the standard conformal prediction (SCP) and class-conditional conformal prediction (CCCP) on CovGap and AvgSize.**
| Dataset | SemanticKITTI | KITTI360 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Basic OCC | VoxFormer | OccFormer | VoxFormer |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Method | Base | Our Depth-UP | Base | Our Depth-UP | Base | Our Depth-UP |  |  |  |  |  |  |  |  |  |  |  |  |
| CP | SCP | CCCP | Ours | SCP | CCCP | Ours | SCP | CCCP | Ours | SCP | CCCP | Ours | SCP | CCCP | Ours | SCP | CCCP | Ours |
| CovGap $\downarrow$ | 0.22 | 0.03 | 0.04 | 0.26 | 0.11 | 0.04 | 0.26 | 0.03 | 0.04 | 0.31 | 0.04 | 0.03 | 0.64 | 0.26 | 0.10 | 0.62 | 0.25 | 0.10 |
| AvgSize $\downarrow$ | 1.53 | 1.71 | 1.13 | 0.97 | 6.43 | 1.36 | 0.10 | 3.42 | 0.94 | 0.10 | 2.96 | 1.24 | 6.30 | 1.03 | 0.56 | 13.24 | 1.51 | 1.12 |

### 4.3 Ablation Study

**Table 3: Ablation study on our Depth-UP framework with VoxFormer and SemanticKITTI.**
| PGC | PSS | IoU $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | mIoU $\uparrow$ | FPS $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | 44.02 | 62.32 | 59.99 | 12.35 | 8.85 |
| ✓ |  | 44.91 | 63.76 | 60.30 | 12.58 | 7.14 |
|  | ✓ | 44.40 | 62.69 | 60.35 | 12.77 | 8.76 |
| ✓ | ✓ | 45.85 | 63.10 | 62.64 | 13.36 | 7.08 |

Uncertainty Propagation. We conducted an ablation study to assess the contributions of each technique proposed in our Depth-UP, as detailed in Table 3. The best results are shown in bold. The results indicate that Propagation on Geometry Completion (PGC) significantly enhances IoU, precision, and recall, which are key metrics for geometry. Additionally, Propagation on Semantic Segmentation (PSS) markedly improves mIoU, a crucial metric for semantic accuracy. Notably, the combined application of both techniques yields performance improvements that surpass the sum of their individual contributions.

Figure: Figure 6: Compare our HCP with SCP and CCCP on CovGap and AvgSize based on VoxFormer. Each point represents one desired class error rate setting. Lower values indicate better performance for both CovGap and AvgSize. (a): The results of CovGap vs. AvgSize on different settings across different datasets. (b): The results of CovGap vs. scale and AvgSize vs. scale on the SemanticKITTI dataset where the scale represents the desired class error rate.
Refer to caption: x6.png

Uncertainty Quantification. We compare our HCP with SCP and CCCP under different desired class-specific error rate $\alpha^{y}$ settings with the basic model VoxFormer, as shown in Figure 6. For each class, the desired error rate is set by multiplying the original error rate of OCC models with the scale $\lambda<1$, which raises the coverage requirement. We consider five settings with $\lambda\in\{0.86,0.89,0.92,0.95,0.98\}$. The points of our HCP are always located in the left bottom corner of subfigures in Figure 6(a) which means our HCP achieves the best performance on set size and coverage gap under all error rate settings. In Figure 6(b), our HCP always achieves low CovGap indicating it can always satisfy the coverage guarantee even under high requirements. For all CP approaches, as the desired error rate becomes smaller, the set size tends to be larger. CPs increase the set size to satisfy the coverage guarantee. The results on other OCC models are shown in Appendix A.8, where our HCP is applied to one LiDAR-based OCC to show its scalability.

Limitation. Regarding frames per second (FPS), our Depth-UP results in a 20% decrease. However, this reduction does not significantly impact the overall efficiency of OCC models. It is important to note that we have not implemented any specific code optimization strategies to enhance runtime. Consequently, the computational overhead introduced by our framework remains acceptable.

## 5 Conclusion

This paper introduces a novel approach to enhancing camera-based 3D Semantic Occupancy Prediction (OCC) for AVs by incorporating uncertainty inherent in models. Our proposed framework, $\alpha$-OCC, integrates the uncertainty propagation (Depth-UP) from depth models to improve OCC performance in both geometry completion and semantic segmentation. A novel hierarchical conformal prediction (HCP) method is designed to quantify OCC uncertainty effectively under high-level class imbalance. Our extensive experiments demonstrate the effectiveness of our $\alpha$-OCC. The Depth-UP significantly improves prediction accuracy, achieving up to 11.58% increase in IoU and up to 12.95% increase in mIoU. The HCP further enhances performance by achieving robust class-conditional coverage and small prediction set sizes. Compared to baselines, it reduces up to 92% set size and up to 84% coverage gap. These results highlight the significant improvements in both accuracy and uncertainty quantification offered by our approach, especially for rare safety-critical classes, such as persons and bicyclists, thereby reducing potential risks for AVs. In the future, we will extend HCP to other highly imbalanced classification tasks.

## Appendix A Appendix

### A.1 More Related Work

Class Imbalance.
In real-world applications like robotics and autonomous vehicles (AVs), datasets often face the challenge of class imbalance (Chen et al., ). Rare classes, typically encompassing high safety-critical entities such as persons, are significantly outnumbered by lower safety-critical classes like trees and buildings. Various strategies have been proposed to tackle class imbalance. Data-level methods involve random under-sampling of majority classes and over-sampling of minority classes during training (Van Hulse et al., ). However, they struggle to address the pronounced class imbalance encountered in OCC (Megahed et al., ), as shown in Section 1. Algorithm-level methods employ cost-sensitive losses to adjust the training process for different tasks, such as depth estimation (Eigen & Fergus, ) and 2D segmentation (Badrinarayanan et al., ). While algorithm-level methods have been widely implemented in current OCC models (Voxformer (Li et al., ) utilizes Focal Loss (Lin et al., ) as the loss function), they still fall short in accurately predicting minority classes. In contrast, classifier-level methods postprocess output class probabilities during the testing phase through posterior calibration (Buda et al., ; Tian et al., ). In this paper, we propose a hierarchical conformal prediction method falling within this category, aimed at enhancing the recall of rare safety-critical classes in the OCC task.

### A.2 Proof of Proposition 1

Proposition 1. For a desired $\alpha^{y}$ value, we select $\alpha_{o}^{y}$ and $\alpha_{s}^{y}$ as $1-\alpha^{y}=(1-\alpha_{s}^{y})(1-\alpha_{o}^{y})$, then the prediction set generated as Eq. 6 satisfies that $\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})|\mathbf{Y}_{test
}=y)\geq 1-\alpha^{y}$.

###### Proof.

$$ $\displaystyle\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y)=\sum_{o}\mathbb{P}(\mathbf{Y}\in\mathcal{C}(\mathbf{X})_{ test})|\mathbf{Y}_{test}=y,o)\mathbb{P}(o|\mathbf{Y}_{test}=y)$ $\displaystyle=$ $\displaystyle\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y,o=T)\mathbb{P}(o=T|\mathbf{Y}_{test}=y)$ $\displaystyle+\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y,o=F)\mathbb{P}(o=F|\mathbf{Y}_{test}=y)$ $\displaystyle=$ $\displaystyle\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y,o=T)\mathbb{P}(o=T|\mathbf{Y}_{test}=y)\geq(1-\alpha_{s}^{ y})(1-\alpha_{o}^{y})$ $\displaystyle\Rightarrow$ $\displaystyle\mathbb{P}(\mathbf{Y}_{test}\in\mathcal{C}(\mathbf{X}_{test})| \mathbf{Y}_{test}=y)\geq 1-\alpha^{y},\texttt{when}\ 1-\alpha^{y}=(1-\alpha_{s }^{y})(1-\alpha_{o}^{y})$ $$

∎

###### Proof.

### A.3 Algorithm of HCP

Algo. 1 shows the detailed algorithm of our hierarchical conformal prediction (HCP).

Figure: Algorithm 1 Our Hierarchical Conformal Prediction (HCP)

### A.4 Introduction on OCC Models

Camera-based OCC has garnered increasing attention owing to cameras’ advantages in visual recognition and cost-effectiveness. Depth predictions from depth models are instrumental in projecting 2D information into 3D space for OCC tasks. Existing methodologies can be classified into two paradigms based on their utilization of depth information: querying 2D from 3D and lifting 2D to 3D. The former (Li et al., ; ) generates query proposals using depth estimation and leverages them to extract rich visual features from the 3D scene. The latter (Tian et al., ; Zhang et al., ), meanwhile, projects multi-view 2D image features into depth-aware frustums, as proposed by LSS (Philion & Fidler, ). However, these methods overlook depth estimation uncertainty. Despite leveraging latent depth distribution, lifting 2D to 3D technique sacrifices precise information and neglects lens distortion issues during geometry completion (Lucas et al., ). During the experiments, we used two OCC models: VoxFormer (Li et al., ) and OccFormer (Zhang et al., ). VoxFormer is the querying 2D from 3D approach and OccFormer is the lifting 2D to 3D approach. So we have considered both paradigms that utilize depth information on OCC models in our experiments.

### A.5 Introduction on Datasets

During the experiments, we use two datasets: SemanticKITTI (Behley et al., ) and KITTI360 (Li et al., ). SemanticKITTI provides dense semantic annotations for each LiDAR sweep composed of 22 outdoor driving scenarios based on the KITTI Odometry Benchmark (Geiger et al., ). Regarding the sparse input to an OCC model, it can be either a single voxelized LiDAR sweep or an RGB image. The voxel grids are labeled with 20 classes (19 semantics and 1 empty), with the size of 0.2m $\times$ 0.2m $\times$ 0.2m. We only used the train and validation parts of SemanticKITTI as the annotations of the test part are not available. SSCBench-KITTI-360 provides dense semantic annotations for each image based on KITTI360 (Liao et al., ), which is also called KITTI360 for simplification. The voxel grids are labeled with 19 classes (18 semantics and 1 empty), with the size of 0.2m $\times$ 0.2m $\times$ 0.2m. Both SemanticKITTI and KITTI360 are interested in a volume of 51.2m ahead of the car, 25.6m to left and right side, and 6.4m in height.

### A.6 Experimental Setting

We used two different servers to conduct experiments on the SemanticKITTI and KITTI360 datasets. For the SemanticKITTI dataset, we employed a system equipped with four NVIDIA Quadro RTX 8000 GPUs, each providing 48GB of VRAM. The system was configured with 128GB of system RAM. The training process required approximately 30 minutes per epoch, culminating in a total training duration of around 16 hours for 30 epochs. The software environment included the Linux operating system (version 18.04), Python 3.8.19, CUDA 11.1, PyTorch 1.9.1+cu111, and CuDNN 8.0.5.

For the KITTI360 dataset, we used a different system equipped with eight NVIDIA GeForce RTX 4090 GPUs, each providing 24GB of VRAM, with 720GB of system RAM. The training process required approximately 15 minutes per epoch, culminating in a total training duration of around 8 hours for 30 epochs. The software environment comprised the Linux operating system(version 18.04), Python 3.8.16, CUDA 11.1, PyTorch 1.9.1+cu111, and CuDNN 8.0.5. These settings ensure the reproducibility of our experiments on similar hardware configurations.

In our training, we used the AdamW optimizer with a learning rate of 2e-4 and a weight decay of 0.01. The learning rate schedule followed a Cosine Annealing policy with a linear warmup for the first 500 iterations, starting at a warmup ratio of $\frac{1}{3}$. The minimum learning rate ratio was set to 1e-3. We applied gradient clipping with a maximum norm of 35 to stabilize the training.

The user-defined target error rate $\alpha^{y}$ for each class $y$ is decided according to the prediction error rate of the original model. For each class, It is set by multiplying the original prediction error rate of OCC models with the scale $\lambda<1$, which raises the coverage requirement. For example, for the person class, if the original model has $90\%$ prediction error rate and we set the scale $\lambda=0.9$, the user-defined target error rate $\alpha^{person}$ of person is decided as $90\%*0.9=81\%$.

### A.7 More Results on Depth-UP

Table 4 presents a comparative analysis of our Depth-UP models against various OCC models, providing detailed mIoU results for different classes. Our Depth-UP demonstrates superior performance in geometry completion and semantic segmentation, outperforming all other OCC models and even surpassing LiDAR-based OCC models on the SemanticKITTI dataset. The VoxFormer with our Depth-UP achieves the best IoU on SemanticKITTI and the OccFormer with our Depth-UP achieves the best mIoU on SemanticKITTI. This improvement is attributed to the significant influence of depth estimation on geometry performance and depth feature extraction, which utilizes inherent uncertainty in depth.
Notably, on the KITTI360 dataset, our Depth-UP achieves the highest mIoU for bicycle, motorcycle, and person classes, which are crucial for safety.

Figure 7 provides additional visualizations of the OCC model’s performance with and without our Depth-UP on the SemanticKITTI dataset. These visualizations demonstrate that our Depth-UP enhances the model’s ability to predict rare classes, such as persons and bicyclists, which are highlighted with orange dashed boxes. Notably, in the fourth row, our Depth-UP successfully predicts the presence of a person far from the camera, whereas the baseline model fails to do so. This indicates that Depth-UP improves object prediction in distant regions. By enhancing the detection of such critical objects, our Depth-UP significantly reduces the risk of accidents, thereby improving the safety of autonomous vehicles.

**Table 4: Separate results on SemanticKITTI and KITTI360. We evaluate our Depth-UP models on two datasets. The default evaluation range is 51.2$\times$51.2$\times$6.4m^3. Due to the label differences between the two subsets, missing labels are replaced with “-”. “Depth-UP^∗” means the VoxFormer with our Depth-UP method. “Depth-UP^†” means the OccFormer with our Depth-UP method.**
| Dataset | Method | Input | IoU | mIoU | car | bicycle | motorcycle | truck | other-veh. | person | road | parking | sidewalk | other-grnd | building | fence | vegetation | terrain | pole | traf.-sign | bicyclist | trunk |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SemanticKITTI | LMSCNet | L | 38.36 | 9.94 | 23.62 | 0.00 | 0.00 | 1.69 | 0.00 | 0.00 | 54.9 | 9.89 | 25.43 | 0.00 | 14.55 | 3.27 | 20.19 | 32.3 | 2.04 | 0.00 | 0.00 | 1.06 |
| SSCNet | L | 40.93 | 10.27 | 22.32 | 0.00 | 0.00 | 4.69 | 2.43 | 0.00 | 51.28 | 9.07 | 22.38 | 0.02 | 15.2 | 3.57 | 22.24 | 31.21 | 4.83 | 1.49 | 0.01 | 4.33 |  |
| MonoScene | C | 36.80 | 11.30 | 23.29 | 0.28 | 0.59 | 9.29 | 2.63 | 2.00 | 55.89 | 14.75 | 26.50 | 1.63 | 13.55 | 6.60 | 17.98 | 29.84 | 3.91 | 2.43 | 1.07 | 2.44 |  |
| VoxFormer | C | 44.02 | 12.35 | 25.79 | 0.59 | 0.51 | 5.63 | 3.77 | 1.78 | 54.76 | 15.50 | 26.35 | 0.70 | 17.65 | 7.64 | 24.39 | 29.96 | 7.11 | 4.18 | 3.32 | 5.08 |  |
| TPVFormer | C | 35.61 | 11.36 | 23.81 | 0.36 | 0.05 | 8.08 | 4.35 | 0.51 | 56.50 | 20.60 | 25.87 | 0.85 | 13.88 | 5.94 | 16.92 | 30.38 | 3.14 | 1.52 | 0.89 | 2.26 |  |
|  | OccFormer | C | 36.50 | 13.46 | 25.09 | 0.81 | 1.19 | 25.53 | 8.52 | 2.78 | 58.85 | 19.61 | 26.88 | 0.31 | 14.40 | 5.61 | 19.63 | 32.62 | 4.26 | 2.86 | 2.82 | 3.93 |
|  | Depth-UP^∗ (ours) | C | 45.85 | 13.36 | 28.51 | 0.12 | 3.57 | 12.01 | 4.23 | 2.24 | 55.72 | 14.38 | 26.20 | 0.10 | 20.58 | 7.70 | 26.24 | 30.26 | 8.03 | 5.81 | 1.18 | 7.03 |
|  | Depth-UP^† (ours) | C | 41.97 | 14.56 | 26.53 | 1.12 | 1.54 | 10.64 | 9.37 | 2.63 | 62.38 | 21.58 | 29.79 | 1.97 | 18.85 | 7.69 | 24.68 | 34.09 | 7.86 | 5.82 | 1.61 | 7.40 |
| KITTI-360 | LMSCNet | L | 47.53 | 13.65 | 20.91 | 0 | 0 | 0.26 | 0 | 0 | 62.95 | 13.51 | 33.51 | 0.2 | 43.67 | 0.33 | 40.01 | 26.80 | 0 | 0 | - | - |
| SSCNet | L | 53.58 | 16.95 | 31.95 | 0 | 0.17 | 10.29 | 0.58 | 0.07 | 65.7 | 17.33 | 41.24 | 3.22 | 44.41 | 6.77 | 43.72 | 28.87 | 0.78 | 0.75 | - | - |  |
| MonoScene | C | 37.87 | 12.31 | 19.34 | 0.43 | 0.58 | 8.02 | 2.03 | 0.86 | 48.35 | 11.38 | 28.13 | 3.22 | 32.89 | 3.53 | 26.15 | 16.75 | 6.92 | 5.67 | - | - |  |
| VoxFormer | C | 38.76 | 11.91 | 17.84 | 1.16 | 0.89 | 4.56 | 2.06 | 1.63 | 47.01 | 9.67 | 27.21 | 2.89 | 31.18 | 4.97 | 28.99 | 14.69 | 6.51 | 6.92 | - | - |  |
| Depth-UP^∗ (ours) | C | 43.25 | 13.55 | 22.32 | 1.96 | 1.58 | 9.43 | 2.27 | 3.13 | 53.50 | 11.86 | 31.63 | 3.20 | 34.49 | 6.11 | 32.01 | 18.78 | 11.46 | 13.65 | - | - |  |

Figure: Figure 7: Qualitative results of the baseline OCC model and that with our Depth-UP method.
Refer to caption: x7.png

### A.8 More Results on HCP

Figure: Figure 8: The results of CovGap vs, AvgSize for our HCP, SCP and CCCP on SemanticKITTI. The considered OCC models are the basic OccFormer, the OccFormer with our Depth-UP, and the LiDAR-based OCC model LMSCNet.
Refer to caption: x8.png

Figure: Figure 9: The results of CovGap vs. scale and AvgSize vs. scale for our HCP, SCP and CCCP on SemanticKITTI. The considered OCC models are the basic OccFormer, the OccFormer with our Depth-UP, and the LiDAR-based OCC model LMSCNet. The scale represents the desired class error rate.
Refer to caption: x9.png

We compare our HCP with SCP and CCCP under different desired class-specific error rate settings on more OCC models: the basic OccFormer, the OccFormer with our Depth-UP, and the LiDAR-based OCC model LMSCNet (Roldao et al., ) to show the scalability of our HCP. The dataset used here is SemanticKITTI. For each class, the desired error rate is set by multiplying the original error rate of OCC models with the scale $\lambda$, $\lambda\in\{0.86,0.89,0.92,0.95,0.98\}$, which raises the coverage requirement. Figure 8 shows the CovGap vs, AvgSize results. We can see that our HCP always outperforms the two baselines for the points of our HCP are located in the left bottom corner, compared with points of SCP and CCCP. Figure 9 shows the detailed results of CovGap vs. scale and AvgSize vs. scale. For most cases, as the desired error rate becomes smaller, the set size tends to be larger in order to satisfy the coverage guarantee. The results on the LiDAR-based OCC model LMSCNet (Roldao et al., ) show that our HCP is effective in LiDAR-based OCCs, even though they are not the primary focus of our work.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Overall, I think this is an interesting paper. The uncertainty propagation improves the geometry completion performance and the HCP improves the accuracy of the safety-critic classes in class im-balanced datasets. As I am not an expert in this area, it would also be very important to see other reviewers’ comments before making the final decision.
soundness: 3
strengths: 1.	Interesting uncertainty propagation for geometric completion. The uncertainty exists in almost all computer vision tasks, and it is also very important to the OCC task. In this paper, the authors use the network to predict the uncertainty for each pixel is straightforward. The strategy of computing the uncertainty of each 3D voxel by accumulating all potential rays though the voxel makes sense. Experiments also show the improvements in the geometry completion task.

2.	Useful hierarchical conformal prediction (HCP). The proposed HCP, compared to SCP and CCP, are specially designed for vary and safety-critic classes from the perspective of both geometric and semantic levels. A KL-based score function enforces these classes to have a higher prediction during the training process. 

3.	Good performance. Experiments on SemanticKITTI and KITTI360 datasets show the proposed uncertainty propagation and uncertainty quantification strategies achieve significant improvements when they are applied to two basic models VoxFormer and OccFormer.

4.	The paper is well-organized and easy to read.
summary: In this paper, the authors focus on improving the Camera-based 3D semantic occupancy prediction (OCC) task by introducing the geometric the geometric uncertainty. 

Specifically, an uncertainty propagation network named Depth-UP is used to predict depth values and pixel-wise uncertainty as well. The predicted uncertainty, together with the depth map the image features are used for semantic occupancy prediction.

In order to effectively utilize the uncertainty, a hierarchical conformal prediction (HCP) method is proposed to quantity the OCC uncertainty, which effectively addresses the class im-balance in OCC datasets. 

Additionally, a KL-based score function is leveraged to improve the occupied recall of safety-critical classes.

Experiments on public SemanticKITTI and KITTI360 datasets demonstrate the effectiveness of the proposed uncertainty propagation and uncertainty quantification strategies.
weaknesses: The proposed method is simple and easy to follow, so I don’t have many concerns about this paper. 

In L209, the authors say that the depth prediction model is retrained to allow to regress both depth values and uncertainties. I am curious about the performance of the newly trained depth model and the basic depth model. If the joint training improves the depth prediction？

Also in the experiments, do the two base models (indicated as Base) use the original depth prediction networks or the newly trained networks with uncertainty branch removed?

Other minor mistakes

a.	L051, (1 + beta%) -> (1 + beta)?

b.	L356-357, in Table 1, 55.78 (+2.34) -> 55.78 (+1.60)

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: (1) The change in the number of parameters after adding the module should be provided in the ablation study section.

(2) The proposed Depth-Up and HCP modules appear to be relatively independent. I would like to understand the relationship between them.

(3) Could you provide an evaluation of how the estimated uncertainty varies with distance?

(4) I would like to know whether the uncertainty of the final predicted semantic occupancy has been improved, specifically with respect to ECE-related metrics, as provided in methods like PaSCo[1].

[1] Cao A Q, Dai A, de Charette R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14554-14564.
soundness: 3
strengths: (1) This paper leverages uncertainty to lift 2D features to 3D space and designs a post-processing module to improve the accuracy of the classification head. The motivation throughout the paper is strong, and the writing is relatively clear, with detailed explanations of the background knowledge and the proposed method.

(2) The paper presents experiments conducted on the SemanticKITTI and KITTI360 datasets, demonstrating improvements in both geometric and semantic predictions. Additionally, it provides relevant performance metrics for uncertainty quantification.
summary: This paper proposes to incorporate uncertainty inherent in models for the camera-based semantic occupancy prediction. The presented framework integrates the uncertainty propagation (Depth-UP) from depth models to improve occupancy prediction performance in both geometry and semantics. A hierarchical conformal prediction (HCP) method is designed to quantify occupancy uncertainty effectively under high-level class imbalance. The extensive experiments demonstrate the effectiveness of proposed α-OCC.
weaknesses: (1) The change in the number of parameters after adding the module should be provided in the ablation study section.

(2) The proposed Depth-Up and HCP modules appear to be relatively independent. I would like to understand the relationship between them.

(3) Could you provide an evaluation of how the estimated uncertainty varies with distance?

(4) I would like to know whether the uncertainty of the final predicted semantic occupancy has been improved, specifically with respect to ECE-related metrics, as provided in methods like PaSCo[1].

[1] Cao A Q, Dai A, de Charette R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14554-14564.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you please explain what uncertainty means in occupancy prediction, and why it is important, and how it could benefit occupancy prediction as raised in weakness 1.
2. Why do you not compare against SOTA methods on the two datasets?
soundness: 3
strengths: 1. The authors propose Depth-UP to improve geometry completion and semantic segmentation by propagating uncertainty from depth models to occupancy prediction models. This framework enhances the performance of various OCC models.
2. To tackle class imbalance in OCC datasets, the paper introduces HCP, which quantifies the uncertainty of OCC and improves the recall of safety-critical classes like pedestrians and bicyclists.
3. The paper demonstrates significant improvements in OCC accuracy and robustness. Depth-UP achieves up to 11.58% improvement in geometry completion and 12.95% in semantic segmentation. HCP improves the occupied recall of safety-critical classes by 45% with only a 3.4% reduction in performance overhead.
4. The paper shows the ability to achieve smaller prediction set sizes while maintaining a defined coverage guarantee, reducing set size by up to 92% compared to baselines.
5. Extensive experiments on two OCC models (VoxFormer and OccFormer) and two datasets (SemanticKITTI and KITTI360) validate the effectiveness of the proposed α-OCC approach.
summary: The paper introduces α-OCC, an uncertainty-aware camera-based 3D semantic occupancy prediction method designed for autonomous vehicle (AV) perception systems. The method aims to enhance the understanding of 3D scenes, which is crucial for tasks like planning and mapping. It addresses the limitations of existing methods that often overlook the inherent uncertainty in models. The paper achieves uncertainty modeling by incorporating a depth distribution in the initialization process of the ocupancy prediction process. The paper concludes that by integrating uncertainty quantification into OCC tasks, significant advancements in accuracy and robustness can be achieved, particularly for rare safety-critical classes, thus reducing potential risks for AVs.
weaknesses: 1. The paper incorporates uncertainty modeling into occupancy prediciton. Despite its good preformance, the paper does not provide an explicit illustration of what uncertainty means in occupancy prediction, and why it is important, and how it could benefit occupancy prediction.
2. The paper does not compare against state of the art methods on SemanticKITTI and KITTI360.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: This paper is well-written and presents the ignore of depth uncertainty hinder the performance of geometry and semantics in OCC models. The authors conduct extensive experiments to demonstrate the effectiveness of their uncertainty propagation and quantification.
summary: The paper presents an uncertainty-aware camera-based method for 3D semantic occupancy prediction called $\alpha$-OCC. This approach includes Depth-UP that propagates depth uncertainty, and HCP that quantifies the uncertainty of the OCC model, which enhance the performance of geometry prediction and semantic segmentation, while also improving the recall of rare safety-critical classes. The authors test their methods using the SemanticKITTI and KITTI360 datasets based on VoxFormer and OccFormer.
weaknesses: 1. Table 1 shows improved scores across each metric, while Figure 4 provides better visualizations for the safety-critical class. In Table 4, the mIoU for the "person" category decreases slightly, and the mIoU for "bicyclist" also declines a bit on SemanticKITTI datasets. Could the authors explain the relationship between the higher recall and some decreasing mIoU?
2. the authors take experiments mainly on SemanticKITTI and KITTI360, Could it keep the performance of the methods for propagate uncertainty in multi-cameras datasets, such as NuScenes or Waymo?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. It seems that the ground truth in the third row of Figure 4 appears to be incorrect. Why is the entire middle row filled with cars?
2. In the text, does "Hierarchical" in HCP refer to first distinguishing whether it is occupied and then identifying the specific category?
3. From Figure 2, it seems that the semantic prediction is directly based on the depth prediction as input, which appears somewhat odd.
soundness: 3
strengths: 1. The paper is well-structured, clearly identifying two key challenges and outlining corresponding solutions.
2. Although rare classes have little impact on the performance, they hold significant value for real-world safety.
3. This work makes the first attempt to propose the uncertainty propagation framework, Depth-UP, to improve OCC performance.
summary: This paper proposes an uncertainty-aware camera-based 3D semantic occupancy prediction method, named α-OCC. Specifically, it addresses two key challenges in occupancy prediction: depth estimation uncertainty and high-level class imbalance. To tackle these challenges, it introduces uncertainty propagation (Depth-UP) from depth models to enhance occupancy prediction performance and proposes a novel hierarchical conformal prediction (HCP) method to improve the recall of rare classes. The effectiveness of the proposed approach is validated through experiments on two methods (VoxFormer and OccFormer) and two datasets (SemanticKITTI and KITTI360).
weaknesses: 1. Considering that the challenges addressed in the paper are reflected across almost all datasets, the experiments conducted solely on the KITTI dataset lack some persuasiveness. In the camera-based occupancy prediction field, more recent methods tend to evaluate on the nuScenes occupancy dataset. Therefore, performance results on nuScenes would be more compelling.
2. How was the value of alpha determined in the experiments, and how should it be decided in practical applications?

Meta-review: This paper tackles semantic occupancy estimation from camera views. Given a camera view, the goal is to estimate the corresponding occupancy in a bounded, voxelized space. In this voxelized space, the network estimates the occupancy value for each voxel and a posterior over a (pre)fixed set of semantic classes. 

The core contribution of this work is an investigation of how we can improve the performance of camera-based semantic occupancy estimation models by accounting for uncertainty in model predictions. To this end, work addressed uncertainty estimation on two fronts:
* To improve occupancy estimation robustness, the paper proposes a Depth-UP (Depth Uncertainty Propagation) module that learns to account for uncertainties in monocular depth estimation modules within the voxelized volume. This approach alone improves occupancy estimation accuracy. 
* To improve semantic segmentation results (specifically, for rarer classes), the paper proposes HPC (Hierarchical Conformal Prediction). Accounting for uncertainty in semantic posterior estimation helps this approach with rarer classes.

Proposed modules are evaluated in conjunction with two state-of-the-art models (*VoxFormer*, Li et al. (2023b) and *Occ- Former*, Zhang et al. (2023)), and are evaluated on two canonical datasets for semantic occupancy estimation. 

The paper received mixed reviews (5, 6, 6, 6, avg. 5.8). 

Reviewers appreciate:
* That papers address uncertainty estimation to improve semantic occupancy prediction- occupancy estimation task is inherently uncertain (ill-posed), and accounting for uncertainty makes sense.
* Liked the approach to semantic uncertainty calibration, that utilizes hierarchical conformal prediction to obtain well-calibrated semantic scores on two levels (classes and super-classes).
* Acknowledge that papers shows improvements consistent improvements in both occupancy and semantic predictions (on two relevant datasets, SemanticKITTI and KITTI360).
* Agree that the paper reports good results, 
* Agree that improvements (compared to baselines) are indeed due to the two added modules, and,
* that the paper is well-written and organized.  

However, reviewers also point out several issues with this paper, among those are: 
* Compared to baselines, the number of network parameters changed, and this was not discussed nor ablated;
* There is a disconnect between the proposed depth uncertainty propagation module (Depth UP) and the semantic (HCP) module. While reading the paper, reviewers (as well as AC) did not see a clear connection. It is true that both proposed modules improve two different aspects of the end-task (depth uncertainty propagation helps with occupancy estimation, while conformal prediction is helpful for semantic calibration). Both can be understood as two independent contributions.
* As MRxz pointed out, the Expected Calibration Error (ECE) metric reveals that the improvement due to uncertainty estimation is quite limited compared to baselines and suggests that the improvements may originate from a higher number of parameters for depth estimation (+19.25%) compared to baselines. 

Overall, I agree with MRxz. It is not entirely clear whether improvements are due to uncertainty quantification (as revealed by the ECE metric). The paper should provide a more transparent discussion of the network changes / additional parameters and provide thorough empirical evidence to support the claims (i.e., that improvements are indeed due to improved uncertainty calibration). 

Finally, author responses solicit reviewers to change a score during the rebuttal or discussion phase (see "Additional Comments On Reviewer Discussion"), which violates the ethical principle of "Be Honest, Trustworthy and Transparent" outlined in the ICLR Code of Ethics, as it undermines the integrity and independence of the review process.

Decision: Reject