$\alpha$-OCC: Uncertainty-Aware Camera-based 3D Semantic Occupancy Prediction

Authors: Sanbao Su, Nuo Chen, Felix Juefei-Xu, Chen Feng, Fei Miao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Uncertainty Propagation, Semantic Occupancy Prediction, Conformal Prediction

Abstract:
In the realm of autonomous vehicle (AV) perception, comprehending 3D scenes is paramount for tasks such as planning and mapping. Camera-based 3D Semantic Occupancy Prediction (OCC) aims to infer scene geometry and semantics from limited observations. While it has gained popularity due to affordability and rich visual cues, existing methods often neglect the inherent uncertainty in models. To address this, we propose an uncertainty-aware camera-based 3D semantic occupancy prediction method ($\alpha$-OCC). Our approach includes an uncertainty propagation framework (Depth-UP) from depth models to enhance geometry completion (up to 11.58\% improvement) and semantic segmentation (up to 12.95\% improvement) for a variety of OCC models. Additionally, we propose a hierarchical conformal prediction (HCP) method to quantify OCC uncertainty, effectively addressing the high-level class imbalance in OCC datasets. On the geometry level, we present a novel KL-based score function that significantly improves the occupied recall of safety-critical classes (45\% improvement) with minimal performance overhead (3.4\% reduction). For uncertainty quantification, we demonstrate the ability to achieve smaller prediction set sizes while maintaining a defined coverage guarantee. Compared with baselines, it reduces up to 92\% set size. Our contributions represent significant advancements in OCC accuracy and robustness, marking a noteworthy step forward in autonomous perception systems.

Decision: reject
Reviewer scores: 6, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729596911811

thread_type:
official_review

<thread_object>
creator: reviewer_a2b2

<content>
summary:
This paper proposes an uncertainty-aware camera-based 3D semantic occupancy prediction method, named α-OCC. Specifically, it addresses two key challenges in occupancy prediction: depth estimation uncertainty and high-level class imbalance. To tackle these challenges, it introduces uncertainty propagation (Depth-UP) from depth models to enhance occupancy prediction performance and proposes a novel hierarchical conformal prediction (HCP) method to improve the recall of rare classes. The effectiveness of the proposed approach is validated through experiments on two methods (VoxFormer and OccFormer) and two datasets (SemanticKITTI and KITTI360).

strengths:
1. The paper is well-structured, clearly identifying two key challenges and outlining corresponding solutions.
2. Although rare classes have little impact on the performance, they hold significant value for real-world safety.
3. This work makes the first attempt to propose the uncertainty propagation framework, Depth-UP, to improve OCC performance.

weaknesses:
1. Considering that the challenges addressed in the paper are reflected across almost all datasets, the experiments conducted solely on the KITTI dataset lack some persuasiveness. In the camera-based occupancy prediction field, more recent methods tend to evaluate on the nuScenes occupancy dataset. Therefore, performance results on nuScenes would be more compelling.
2. How was the value of alpha determined in the experiments, and how should it be decided in practical applications?

questions:
1. It seems that the ground truth in the third row of Figure 4 appears to be incorrect. Why is the entire middle row filled with cars?
2. In the text, does "Hierarchical" in HCP refer to first distinguishing whether it is occupied and then identifying the specific category?
3. From Figure 2, it seems that the semantic prediction is directly based on the depth prediction as input, which appears somewhat odd.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730111783623

thread_type:
official_review

<thread_object>
creator: reviewer_l8q4

<content>
summary:
The paper presents an uncertainty-aware camera-based method for 3D semantic occupancy prediction called $\alpha$-OCC. This approach includes Depth-UP that propagates depth uncertainty, and HCP that quantifies the uncertainty of the OCC model, which enhance the performance of geometry prediction and semantic segmentation, while also improving the recall of rare safety-critical classes. The authors test their methods using the SemanticKITTI and KITTI360 datasets based on VoxFormer and OccFormer.

strengths:
This paper is well-written and presents the ignore of depth uncertainty hinder the performance of geometry and semantics in OCC models. The authors conduct extensive experiments to demonstrate the effectiveness of their uncertainty propagation and quantification.

weaknesses:
1. Table 1 shows improved scores across each metric, while Figure 4 provides better visualizations for the safety-critical class. In Table 4, the mIoU for the "person" category decreases slightly, and the mIoU for "bicyclist" also declines a bit on SemanticKITTI datasets. Could the authors explain the relationship between the higher recall and some decreasing mIoU?
2. the authors take experiments mainly on SemanticKITTI and KITTI360, Could it keep the performance of the methods for propagate uncertainty in multi-cameras datasets, such as NuScenes or Waymo?

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730291602149

thread_type:
official_review

<thread_object>
creator: reviewer_x1wm

<content>
summary:
The paper introduces α-OCC, an uncertainty-aware camera-based 3D semantic occupancy prediction method designed for autonomous vehicle (AV) perception systems. The method aims to enhance the understanding of 3D scenes, which is crucial for tasks like planning and mapping. It addresses the limitations of existing methods that often overlook the inherent uncertainty in models. The paper achieves uncertainty modeling by incorporating a depth distribution in the initialization process of the ocupancy prediction process. The paper concludes that by integrating uncertainty quantification into OCC tasks, significant advancements in accuracy and robustness can be achieved, particularly for rare safety-critical classes, thus reducing potential risks for AVs.

strengths:
1. The authors propose Depth-UP to improve geometry completion and semantic segmentation by propagating uncertainty from depth models to occupancy prediction models. This framework enhances the performance of various OCC models.
2. To tackle class imbalance in OCC datasets, the paper introduces HCP, which quantifies the uncertainty of OCC and improves the recall of safety-critical classes like pedestrians and bicyclists.
3. The paper demonstrates significant improvements in OCC accuracy and robustness. Depth-UP achieves up to 11.58% improvement in geometry completion and 12.95% in semantic segmentation. HCP improves the occupied recall of safety-critical classes by 45% with only a 3.4% reduction in performance overhead.
4. The paper shows the ability to achieve smaller prediction set sizes while maintaining a defined coverage guarantee, reducing set size by up to 92% compared to baselines.
5. Extensive experiments on two OCC models (VoxFormer and OccFormer) and two datasets (SemanticKITTI and KITTI360) validate the effectiveness of the proposed α-OCC approach.

weaknesses:
1. The paper incorporates uncertainty modeling into occupancy prediciton. Despite its good preformance, the paper does not provide an explicit illustration of what uncertainty means in occupancy prediction, and why it is important, and how it could benefit occupancy prediction.
2. The paper does not compare against state of the art methods on SemanticKITTI and KITTI360.

questions:
1. Could you please explain what uncertainty means in occupancy prediction, and why it is important, and how it could benefit occupancy prediction as raised in weakness 1.
2. Why do you not compare against SOTA methods on the two datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643085919

thread_type:
official_review

<thread_object>
creator: reviewer_mrxz

<content>
summary:
This paper proposes to incorporate uncertainty inherent in models for the camera-based semantic occupancy prediction. The presented framework integrates the uncertainty propagation (Depth-UP) from depth models to improve occupancy prediction performance in both geometry and semantics. A hierarchical conformal prediction (HCP) method is designed to quantify occupancy uncertainty effectively under high-level class imbalance. The extensive experiments demonstrate the effectiveness of proposed α-OCC.

strengths:
(1) This paper leverages uncertainty to lift 2D features to 3D space and designs a post-processing module to improve the accuracy of the classification head. The motivation throughout the paper is strong, and the writing is relatively clear, with detailed explanations of the background knowledge and the proposed method.

(2) The paper presents experiments conducted on the SemanticKITTI and KITTI360 datasets, demonstrating improvements in both geometric and semantic predictions. Additionally, it provides relevant performance metrics for uncertainty quantification.

weaknesses:
(1) The change in the number of parameters after adding the module should be provided in the ablation study section.

(2) The proposed Depth-Up and HCP modules appear to be relatively independent. I would like to understand the relationship between them.

(3) Could you provide an evaluation of how the estimated uncertainty varies with distance?

(4) I would like to know whether the uncertainty of the final predicted semantic occupancy has been improved, specifically with respect to ECE-related metrics, as provided in methods like PaSCo[1].

[1] Cao A Q, Dai A, de Charette R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14554-14564.

questions:
(1) The change in the number of parameters after adding the module should be provided in the ablation study section.

(2) The proposed Depth-Up and HCP modules appear to be relatively independent. I would like to understand the relationship between them.

(3) Could you provide an evaluation of how the estimated uncertainty varies with distance?

(4) I would like to know whether the uncertainty of the final predicted semantic occupancy has been improved, specifically with respect to ECE-related metrics, as provided in methods like PaSCo[1].

[1] Cao A Q, Dai A, de Charette R. Pasco: Urban 3d panoptic scene completion with uncertainty awareness[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14554-14564.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656393329

thread_type:
official_review

<thread_object>
creator: reviewer_mkcp

<content>
summary:
In this paper, the authors focus on improving the Camera-based 3D semantic occupancy prediction (OCC) task by introducing the geometric the geometric uncertainty. 

Specifically, an uncertainty propagation network named Depth-UP is used to predict depth values and pixel-wise uncertainty as well. The predicted uncertainty, together with the depth map the image features are used for semantic occupancy prediction.

In order to effectively utilize the uncertainty, a hierarchical conformal prediction (HCP) method is proposed to quantity the OCC uncertainty, which effectively addresses the class im-balance in OCC datasets. 

Additionally, a KL-based score function is leveraged to improve the occupied recall of safety-critical classes.

Experiments on public SemanticKITTI and KITTI360 datasets demonstrate the effectiveness of the proposed uncertainty propagation and uncertainty quantification strategies.

strengths:
1.	Interesting uncertainty propagation for geometric completion. The uncertainty exists in almost all computer vision tasks, and it is also very important to the OCC task. In this paper, the authors use the network to predict the uncertainty for each pixel is straightforward. The strategy of computing the uncertainty of each 3D voxel by accumulating all potential rays though the voxel makes sense. Experiments also show the improvements in the geometry completion task.

2.	Useful hierarchical conformal prediction (HCP). The proposed HCP, compared to SCP and CCP, are specially designed for vary and safety-critic classes from the perspective of both geometric and semantic levels. A KL-based score function enforces these classes to have a higher prediction during the training process. 

3.	Good performance. Experiments on SemanticKITTI and KITTI360 datasets show the proposed uncertainty propagation and uncertainty quantification strategies achieve significant improvements when they are applied to two basic models VoxFormer and OccFormer.

4.	The paper is well-organized and easy to read.

weaknesses:
The proposed method is simple and easy to follow, so I don’t have many concerns about this paper. 

In L209, the authors say that the depth prediction model is retrained to allow to regress both depth values and uncertainties. I am curious about the performance of the newly trained depth model and the basic depth model. If the joint training improves the depth prediction？

Also in the experiments, do the two base models (indicated as Base) use the original depth prediction networks or the newly trained networks with uncertainty branch removed?

Other minor mistakes

a.	L051, (1 + beta%) -> (1 + beta)?

b.	L356-357, in Table 1, 55.78 (+2.34) -> 55.78 (+1.60)

questions:
Overall, I think this is an interesting paper. The uncertainty propagation improves the geometry completion performance and the HCP improves the accuracy of the safety-critic classes in class im-balanced datasets. As I am not an expert in this area, it would also be very important to see other reviewers’ comments before making the final decision.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733097819772

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We noticed the feedback from Reviewer MRxz whose question (https://openreview.net/forum?id=sgaMYvGRG5&noteId=UxDtrSbfSl) about our new experiment results might be of interest to other reviewers, although we respectfully disagree with MRxz's speculation and comments. For the sake of constructive rebuttal discussion, we provide our response to all reviewers as follows:

1. **“the performance gains might primarily stem from better depth estimation.”**  We freeze all parameters of the original depth model in all our experiments (both baselines and our method)  and only retrain the additional head for estimating depth uncertainty, denoted as $\hat{\mathbf{\Sigma}}$. Consequently, the performance of depth estimation $\hat{\mathbf{D}}$ remains unchanged between the newly trained model and the original depth model. The number of parameters of  the depth estimation is not changed. Of the 11.54M additional parameters in our Depth-UP, 0.11M parameters are contributed to the additional head for the depth **uncertainty** estimation and 11.43M parameters are from the extra ResNet backbone in the OCC model.
2. **"However, in the response to Q4, they state that "Uncertainty quantification for OCC is not the primary focus of our Depth-UP method," which appears somewhat contradictory. "** Our Depth-UP method focuses on propagating the uncertainty of the depth model to the OCC model to enhance its performance (accuracy). Here, "UP" stands for uncertainty propagation, not uncertainty quantification. Therefore, Depth-UP is not an uncertainty quantification method for OCC models. The uncertainty quantification method we proposed for OCC models is HCP.

We appreciate constructive comments and feedback from all reviewers. Thank you for taking the time to review our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles semantic occupancy estimation from camera views. Given a camera view, the goal is to estimate the corresponding occupancy in a bounded, voxelized space. In this voxelized space, the network estimates the occupancy value for each voxel and a posterior over a (pre)fixed set of semantic classes. 

The core contribution of this work is an investigation of how we can improve the performance of camera-based semantic occupancy estimation models by accounting for uncertainty in model predictions. To this end, work addressed uncertainty estimation on two fronts:
* To improve occupancy estimation robustness, the paper proposes a Depth-UP (Depth Uncertainty Propagation) module that learns to account for uncertainties in monocular depth estimation modules within the voxelized volume. This approach alone improves occupancy estimation accuracy. 
* To improve semantic segmentation results (specifically, for rarer classes), the paper proposes HPC (Hierarchical Conformal Prediction). Accounting for uncertainty in semantic posterior estimation helps this approach with rarer classes.

Proposed modules are evaluated in conjunction with two state-of-the-art models (*VoxFormer*, Li et al. (2023b) and *Occ- Former*, Zhang et al. (2023)), and are evaluated on two canonical datasets for semantic occupancy estimation. 

The paper received mixed reviews (5, 6, 6, 6, avg. 5.8). 

Reviewers appreciate:
* That papers address uncertainty estimation to improve semantic occupancy prediction- occupancy estimation task is inherently uncertain (ill-posed), and accounting for uncertainty makes sense.
* Liked the approach to semantic uncertainty calibration, that utilizes hierarchical conformal prediction to obtain well-calibrated semantic scores on two levels (classes and super-classes).
* Acknowledge that papers shows improvements consistent improvements in both occupancy and semantic predictions (on two relevant datasets, SemanticKITTI and KITTI360).
* Agree that the paper reports good results, 
* Agree that improvements (compared to baselines) are indeed due to the two added modules, and,
* that the paper is well-written and organized.  

However, reviewers also point out several issues with this paper, among those are: 
* Compared to baselines, the number of network parameters changed, and this was not discussed nor ablated;
* There is a disconnect between the proposed depth uncertainty propagation module (Depth UP) and the semantic (HCP) module. While reading the paper, reviewers (as well as AC) did not see a clear connection. It is true that both proposed modules improve two different aspects of the end-task (depth uncertainty propagation helps with occupancy estimation, while conformal prediction is helpful for semantic calibration). Both can be understood as two independent contributions.
* As MRxz pointed out, the Expected Calibration Error (ECE) metric reveals that the improvement due to uncertainty estimation is quite limited compared to baselines and suggests that the improvements may originate from a higher number of parameters for depth estimation (+19.25%) compared to baselines. 

Overall, I agree with MRxz. It is not entirely clear whether improvements are due to uncertainty quantification (as revealed by the ECE metric). The paper should provide a more transparent discussion of the network changes / additional parameters and provide thorough empirical evidence to support the claims (i.e., that improvements are indeed due to improved uncertainty calibration). 

Finally, author responses solicit reviewers to change a score during the rebuttal or discussion phase (see "Additional Comments On Reviewer Discussion"), which violates the ethical principle of "Be Honest, Trustworthy and Transparent" outlined in the ICLR Code of Ethics, as it undermines the integrity and independence of the review process.