Laplace Approximation Based Epistemic Uncertainty Estimation in 3D Object Detection
Authors: Peng Yun
Ming Liu
Venue: CoRL 2022 Poster (2022)
Keywords: Laplace approximation
epistemic uncertainty
3D object detection

Abstract: Understanding the uncertainty of predictions is a desirable feature for perceptual modules in critical robotic applications. 3D object detectors are neural networks with high-dimensional output space. It suffers from poor calibration in classification and lacks reliable uncertainty estimation in regression. To provide a reliable epistemic uncertainty estimation, we tailor Laplace approximation for 3D object detectors, and propose an Uncertainty Separation and Aggregation pipeline for Bayesian inference. The proposed Laplace-approximation approach can easily convert a deterministic 3D object detector into a Bayesian neural network capable of estimating epistemic uncertainty. The experiment results on the KITTI dataset empirically validate the effectiveness of our proposed methods, and demonstrate that Laplace approximation performs better uncertainty quality than Monte-Carlo Dropout, DeepEnsembles, and deterministic models.

Laplace Approximation Based Epistemic Uncertainty Estimation in 3D Object Detection
Peng Yun
Ming Liu
Department of Computer Science and Engineering Hong Kong University of Science and Technology pyunOcse. ust .kk
Hong Kong University of Science and Technology (Guangzhou) eeli .hk
Abstract: Understanding the uncertainty of predictions is a desirable feature for perceptual modules in critical robotic applications. 3D object detectors neural networks with high-dimensional output space. It suffers from poor calibration classification and lacks reliable uncertainty estimation in regression. To provide reliable epistemic uncertainty estimation, we tailor Laplace approximation for 3D object detectors, and propose Uncertainty Separation and Aggregation pipeline for Bayesian inference. The proposed Laplace-approximation approach can easily convert a deterministic 3D object detector into Bayesian neural network capable of estimating epistemic uncertainty. The experiment results on the KITTI dataset empirically validate the effectiveness of our proposed methods, and demonstrate that Laplace approximation performs better uncertainty quality than Monte-Carde Dropout, DeepEnsembles, and deterministic models.
Keywords: Laplace approximation, epistemic uncertainty, 3D object detection
1 Introduction
Detecting key objects in 3D space is an important task in robotic applications. It provides both semantic and spatial information for decision-making Due the success of deep learning, existing detectors have got great performance on benchmarks [1-3]. However, robots directly interact with the real world, the data distribution of which may drift over time and space. is essential for robotic perceptual module to know when it is uncertain, since the consequences of mistakes can be fatal in critical applications, like autonomous driving.
Laplace approximation (LA) [4] provides principled uncertainty estimation as other Bayesian ap- proaches [5-10], and computes posterior weight distributions in terms of local curvatures. It can convert deterministic model, trained under the maximum-a-posterion framework, into a Bayesian model for uncertainty estimation, which is in a post-hoc way without changing the training proce- dure. Recent research reduces the computational costs on computing local curvatures with various approximations 1-15], and makes Laplace approximation applicable in modern neural networks. However, existing research is limited to end-to-end networks, and rare has been applied to networks with nigh-dimensional output space, like 3D object detection.
Neural networks with high- sigh-dimensional output space require Monte Carlo-based Bayesian inference (MCBI) to approximate the predictive distribution. The MCBI is not applicable in two-stage 3D de- tectors due to its non-end-to-en architecture. A two-stage detector contains two parts: a Region Proposal Network (RPN) generates 3D proposals, and RCNN selects and refines subset of pro- posals Sp to get final results. The different weight samples in MCBI have different selections Sp which are mis-matched in order shown in Figure (a). It causes overestimating the epistemic un- certainty in predictive distributions, as shown in Figure 1 (c). We propose an Uncertainty Separation and Aggregation (U-SPA) method apply Laplace approximation in two-stage detectors, as shown in Figure (b). It separates the uncertainty estimation in RPN and RCNN and aggregates them in the final predictive distribution (d).
Peng Yun also with Clear Water Bay Institute of Autonomous Driving, Nanshan, Shenzhen. 1 Minn Liu (correspouding author) also with Hong Kong University Science and Technology, Hong Kong SAR, China and HKUST Shenzhen Shenzhen-HongKong Collaborative Innovation Research Institute, Futian, Shenzhen.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

NMS-TopK
RPN
RCNN
MMTT-RST
NMS- Topk
x
(a) Monte-Carto Bayesian Inference
RPN
RCNN
MMTT-EST
NMS-TopX
MMTT-EST BEST-EST
XT
ym
uncertainity aggregaaion
Uncertainty Separation and Aggregation
uncerfaintty aguregation
(c) Monte-Cart Canne Bayesian Inference
(d) Uncertainty Separation and Aggregation
Bayesian inference pipeline for two-stage 3D detectors. (a) Monte-Carde Bayesian In- ference pipeline and (c) bird's-eye-view visualization of predictive distribution y. (b) Uncertainty Separation and Aggregation pipeline and (d) visualization of The contours reflect the objective- ness probability. The purple box denotes the ground truth.
Figure 1:
In this paper, we tailor Laplace approximation methods to estimate epistemic uncertainty for 3D object detection under the context of autonomous driving. (1) We propose an Uncertainty Sepa- ration and Aggregation (U-SPA) pipeline to solve the mismatching problem in Bayesian inference of two-stage 3D object detectors. (2) We apply Laplace approximation to state- of-the-art art 3D object detection networks, including both one-stage and two-stage detectors, and also provide a benchmark on the KITTI dataset to compare various Laplace approximation approaches, including LA (diagonal Fisher). LA (empirical diagonal Fisher), and sub-net LA. The proposed saplace-aprroximatio method can easily convert off-the-shelf 3D object detectors into a Bayesian neural network. The experiment results show that the Laplace approximation performs consistently better uncertainty quality than deterministic, MCDropout, and DeepEnsembles baselines.
2
Related Works
2.1 Laplace approximation
The major challenge in applying Laplace approximation [4] to large-scale neural networks is its expensive computation on the Hessian matrix. Even though the Fisher information matrix can be its equivalent, the exact Fisher requires storing and manipulating an ((P2) matrix, which is im- practical deploy in modern neural networks with the limited computational resources on mobile robots. Diagonal Fisher approximation is the lightest weight approach to approximate the Hes- sian. ignores the correlation among model parameters and largely reduces the Fisher matrix size into O(P), which has been well applied second-order optimization [16] well as incremental learning [17]. To balance the curvature information and computational efficiency, some researchers approximate the exact Fisher matrix with Kronecker Factorized Fisher [15] and low-rank approx- imations [14]. There also exists research applying Laplace approximation to a subset of model though empirical Fishers lack theoretical foundation [18], it was adopted low-cost surrogate of Fisher information matrix and got success in practical applications [19-21]. This paper considers the diagonal Fisher approximation, empirical Fisher implementation, and subnet Laplace approximations in 3D object detection.
parameters [1 [11-13], called SubNet-LA. Besides,
LiDAR-based 3D object detection
The geometrical information in LiDAR scans can be used to classify and locate key objects in space. 3D object detector methods can categorized into one-stage and two-stage methods. One-stage 3D detectors are end-to-end networks which consume LiDAR scans and estimate 2D

---

Figure 2: Illustra- tion of one-stage (a) and two-stage 3D detectors (b). The one-stage de- tector also acts as an RPN subnetwork in two-stage detec- tors. (c) shows the RCNN subnetwork architecture. The
One-stage 3D detector, Elissificaation (RPN)
RCNN
classification
predictions
predictions
fo 2D:3D CNN
regression predictions
x proposals
famm MLP
regression
clouds
predictions
[a) Two-stage 3D) detector
NMB-Topk
"MLP" is short for
RPN
RCNN
Multi-layer percep- trons.
X point clouds
noo
y
proposals
proposals
maps 3D volumes for classification and regression [22-25], as shown in Figure 2 (a). Its out- put y = [ydls, yrrg] contains classification predictions yol and regression predictions yreg, Yan et al. [24] exploited the sparsity of LiDAR scans and replaced the conventional dense convolution layers with sparse convolution operations. Lang et al. [25] extracted features from point clouds with PointNet [26] estimate classification and regression maps hird's-eye-virs view images.
Two-stage detectors [27-29] contains two subnetworks: Region Proposal Network (RPN) and RCNN, as shown in Figure 2 (b). RPN generates proposals from point clouds with the same structure as one-stage detectors. non-manne maximization--uppresssee
(NMS) module then selects top-K unover- lapped proposals in terms of their classification score from RPN proposals y*p". RCNN network refines these proposals with multiple linear layers, as shown in Figure 2 (c). The two-stage pipeline was proposed in [27]. where Shi et al. adopted PointNet-basee RPN and generated point-wise pro- posals, and further refined these proposals with an MLP-based RCNN sub-network. To balance ac- curacy and computational efficiency, their follow-up work [28, 30] adopt sparse-convolution-baas RPN to generate proposals in 3D volume. In this paper, we adopt one-stage 3D detectors, PointPil- lar [25] Second [24], and two-stage detectors PointRCNN [27] demonstrate the effectiveness of Laplace approximation in 3D object detection.
2.3 Uncertainty estimation in object detection
Some recent works estimate uncertainties in robotic perceptual modules. Hall et al. [31] extended object detection probabilistic object detection and quantified spatial and semantic uncertainties of detections with Monte-Caria Dropout. The idea of probabilistic object detection tended to 3D object detection [32-34]. Feng et al. [32, 33] made an one-stage detector PIXOR [23] to estimate aleatoric uncertainties by adding additional regression layer, as [35]. The estimated uncertainties were evaluated independently for each output dimension with Expected Calibration Erorrs (ECE) [36, 37]. To evaluate the general predictive distribution for LiDAR-based 3D object detection, Wang et al. [34] proposed mAP(JloU), which evaluates the differences between the esti- mated predictive distribution and ground-truth predictive distribution. They proposed generative model to obtain the ground-truth predictive distribution. In our work, we evaluate the uncertainty quality with ECE following the method in[32], and also report the negative log-likelihood (NLL) performance as well as mAP(JlUU [34].
further ex-
Methodology
3.1 Preliminaries
1 we have a neural network fo(x with its prior weight distribution p(0). We want to derive the posterior weight distribution X( Y), that we can conduct Bayesian inference and get the predictive distribution with
Given a dataset D (X
ply X X.Y
pxy"|x", (0)((0)), Y)d0,
(1)
, f) depicts the predictive model. In classification tasks, it is commonly defined as categorical distribution with p'y 1|x, 0) offtmax"(Jo where the superscript denotes the th element of vector. In regression tasks, it can be defined as p(y|x, 0) N(y; fx(x),==11 The can be hyper-parameter or an estimation from data. This Bayesian inference equation catches the epistemic uncertainty caused by the weight distribution p(0 X, Y).
where p(y*|x' [x",

---

Algorithm 1: Monte-Carlo Bayesian In- ference (MCBI)
Algorithm 2: Uncertainty Separation and Ag- gregation (U-SPA)
Input :
x: point cloud; ; X Y):
Input
x: point cloud; p(0[X, Y):
posterior weight distribution T': number of MC samples
posterior weight distribution.
T:: number of MC samples
Output : y: set of 3D bounding-bol predictive distributions
Output : y: a set of 3D bounding-box predictive distributions
9t + MCBI(X. ((0pp X, Y).
preds 0
for < 1 to 7 do
y is the mode of ynmn y=pn f y y²
0 sample from p(0[X, Y);
y = x(0,x)
Sp c NMS- (
append y into preds; end
genn + {y " E Spł:
yramm + MCBI(x renn p(0 X, Y), MM
- (preES)(predd):;
y t aggregate ymrn and yoonn with (9) (10);
return y
return y;
The log- log-posterior weight distribution log ( Y) can be unfolded with Bayes formula: lo] Y) = log (ogp(() 0) + logp(0) - log p ( Laplace approximation unfolds it at 0 (Vo log P(0g((0 Y) lo 0) with second-order Taylor expansion to construct
Laplace approximation
normal distribution approximating p(0) X, Y). i.e.
log p(0 X, Y) ~ log p(0 X Y)
-0 -00-0"1-
0-) Hog (ogg(6- ( - 0²),
(2)
log ( X Y) is the Hessian matrix of log p(0" X, Y). It is easy to get 8² H Poggp(00 = H Ho + 220 [log p(0)), where the second term can be evaluated easily if p(0) simple in closed form. The Hessian matrix in the first term can be approximated with the
/ where H pog (X,Y) b2
Fisher information matrix [15, 17, 38], which is defined as
F = E [EP V log (Y|,, 0) log ( Y 0)]
(3)
The approximate posterior weight distribution is p(0 X, Y) N(0;0", E), where = [F ---E we consider a Gaussian weight prior p(0) = (0: 0, Eg).
3.2 Bayesian inference detection
3D object detectors contain both classification and regression predictions. We conduct Bayesian inference and estimate p(y" [x", X, Y) in (I) with Monte-Carlo estimators as [39]. classification, (fo(xxx
p(y*' 1|x* X, Y) x + > LBEP(XXY softmax (ff(x*)), )), where p(y*|x*,X, xx,x,XX7 Y) is assumed cate- gorical distribution, and T denotes the number of Monte Carlo samples. In regression, we assume p(y" [x", X, Y) as a normal distribution. Its mean and covariance matrix can be estimated with
By-ly*1
Σ
fo(x*)
(4)
T
(EP(O)X,Y)
Covly"] Σ
+r Err'--- Ey" [y").
(5)
(EP(
For conciseness, we denote this moment estimation step with MMT-EST(-).
One-stage detectors are end-to-end neural networks, as shown in Figure 2 (a). It outputs two sibling predictions yole E MxC and yreg E MDD where M denotes the number of all 3D bounding- box proposals (anchors), C' is the number of classes, and 7) is the dimension of a parameterized 3D bounding box. Algorithm 1 shows the MCBI with moment estimation estimate predictive distributions y. It is note that the set of 3D bounding- predictive distributions y contains two components: set of 3D bounding-boo predictive classification distributions yo and its regression counterpart Te i.c. y Ly res].
Two-stage detectors are not end-to-end neural networks and contain two separate sub-networks: RPN and RCNN. RPN generates proposals yr Sepo (0rpn: x). non-maximizzaiinnsuppee suppression (NMS) module computes the overlapped area among proposals and selects the top-K unoverlapped proposals by their classification scores. We
Mis-matching in two-stage detectors

---

denote this step as Sp = (MS-TopK(y"mm))) where the Sp contains the indices of selected propos- als. Since the ranking results of NMS depending on the RPN output frym (0mp: x). different weight samples Ppmn drawn in Monte-Carlo sampling may generate different selections Sp. Formally,
Sp. (MMSTTop(.frpi (0/mm: x)). 6pp. ~ p(Orpn) rm X, Y).
(6)
Sp2 AMS-TTpK(() (Jrpu) (0 (0 (xpp::)).. Oppm ~ p(Orpp|XX Y):
(7)
P(Sp, Spz) if 01 + 02 Orgn-
(8)
These selected proposals across different weight samples are probably not matched. As result, the moment estimation process in RCNN cannot estimate the true moments, as shown in Figure 1 (a) (c).
Uncertainty Separation and Aggregation (U-SPA) To solve this problem, we separate the un- certainty estimation for RPN and R-CNN in two-stage detectors, and aggregate them in the final predictive distribution as shown in Figure (b). We conduct MCBI on RPN to obtain the predictive distributions of RPN results yrpn, We then extract the mode y*p" and conduct NMS get selected proposals Sp. In estimating uncertainties in RCNN, we adopt the selected modes fyp"" S as inputs and conduct MCBI. With the same input, the outputs of Monte-Cart Carlo samples will be matched in the order defined by Sp-
In aggregating the uncertainties from RPN and R-CNN, we keep the mode of yna and accumulate the covariance matrices of yrrn.reg and y yee,,.reg It can be considered adding up uncertainties from two independent sources. Formally, the final predictive distribution with aggregated uncertain- ties can be formulated as
[yd categorical (Iyronn 1i), E Sp:
y r N M(y rennnregg] E Syyro.eeu) + [ggennnren) ).), Sp,
(10)
where the [+] denotes the i-th element in a set, (a) extracts the mode of a predictive distribution a, and Σ, extracts its covariance matrix. Algorithm 2 summarizes the U-SPA pipeline.
3.3 Diagonal Fisher approximation
In 3D detectors, the number of parameters could be the order of million. is hard to compute the exact Fisher and its inversion directly. To make the Fisher calculation feasible, diagonal Fisher approximation assumes each single weight parameter as independent:
(11) where diag(-) converts matrix or vector its corresponding diagonal matrix. To evaluate stan- dard diagonal Fisher (11), we adopt the Monte-Carke Carlo method used in [38] to calculate the inner expectation over Prix.::
diag(F) = Ex |E [diag( VaaggVlogg logp(y)> 0))
diag(F) F) ~ E
E [diag( Ve log 0))
(12)
yεРrix.o
where T denotes the number of Monte-Carlo samples in caleulating the Fisher. It requires O(NT,) backward computations to calculate the diagonal standard Fisher and O(P) storage space.
Empirical diagonal Fisher The empirical Fisher was adopted as a low-cost surrogate of the stan- dard Fisher or Hessian in [19-21]. It is defined as
F=
Vogrryyxx 0)" V p 0)..
(13)
N
(KY)ED
we approximate the empirical Fisher with diagonal approximation, we have the following empir- ical diagonal Fisher formulation:
diag F)
Σ [diag V log (ogg((x),0)))
(14)
N
(N.Y)ED
Compared to the standard Fisher information matrix, it avoids the expectation over the predictive distribution and computes the expectation over the training distribution instead of the predictive distribution. Therefore, it only requires O(N) backward computations. Even though the empir-
We denote y!T (17rp, C (yrtprr E R RR). the [t]s returns the i-th element from its input.
5

---

PP
SC
PR regression
classification methods mean std. x10
regression mean std. =
classification regression mean std. x1o
classification mean std. x mean std. x10
mean std. x30
det. 0.115
0.100
00.113
dropout 0.112
1.4
0.763
0.7
0.126
2.1
0.594
1.5
0.104
10.7
0.746
2.6
deep-en. 0.106
0.0
0.664
0.0
0.096
0.0
0.640
0.0
0.608
0.0
0.767
0.0
LL 0.115
0.5
0.528
3.6
0.100
1.0
0.510
2.2
0.111
18.9
0.357
3.2
LM (E.)
0.064
16.9
0.316
2.5
full (E.) 0.097
10.2
0.418
2.3
0.087
5.2
0.395
5.1
0.061
11.9
0.576
9.6
LL (S.) 0.115
0.8
0.302
6.2
0.100
0.4
0.300
22.8
0.111
19.1
0.519
0.4
LM (S.)
0.095
15.0
0.467
1.7
full (S.) 0.111
1.6
0.338
7.7
0.089
6.5
0.311
14.3
0.070
11.8
0.481
34.8
Table 1: Expected calibration error (4). The "det", "LL", "LM", and "full" are short for "determin-
istic", "last-layer", "last-module", and "full-net". The results are computed from 5 random seeds.
PP
SC
PR
methods
mean
std.
mean
std.
mean
std.
dropout
3.055
0.604
0.185
0.011
1.435
0.175
deep-en.
0.104
0.000
0.043
0.000
1.546
0.000
LL (E.)
0.332
0.021
0.193
0.019
-0.599
0.007
LM (E.)
-0.690
0.007
full (E.)
-0.744
0.002
-0.773
0.006
-0.435
0.011
LL (S.)
-0.561
0.022
-0.603
0.079
-0.007
0.012
LM (S.)
-0.297
0.011
full (S.)
-0.666
0.005
-0.746
0.021
-0.574
0.041
Table Negative likeliiel (Q). The notations are the same to Table
ical Fisher has superior computation efficiency and practical success, it lacks a solid theoretical grounding to capture the local curvature information[ [8]]. In Section 4, will compare the Laplace approximation with standard diagonal Fisher and empirical diagonal Fisher in 3D object detection.
4
Experiments
In this section, we conduct experiments on the KITTI dataset [1] to explore the following ques- tions: (1) How is the quality of the estimated predictive distribution? (2) Does the accuracy of mode estimation get affected after applying Laplace approximation? We adopt well-trained 3D ob- ject detectors as deterministic models and convert them with the aaplace-approiimaioon-bees proaches in Section 3. The adopted 3D detectors contain one-stage detectors PoinPillar[25] ("PP") and [23] ("SC"), as well as a two-stage detector PointRCNNI27] ("PR"). In particular, we compute the Fisher on the training set and inference with it on the validation set. Ten Monte-carlo samples are adopted to approximate the integration in (1). We induce a as the scalar factor of the posterior weight covariance matrix, and set 0.003 for the standard Fisher, and 0.01 for the empirical Fisher by tuning it on the hand-hold set.
ap-
4.1 Quality of predictive distributions
In this section, we present the Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL) to demonstrate the predictive distribution quality, compare the following baselines and vari- ous
implementations:
deterministic model: we use the softmax outputs as classification confidences;
mc-dropout [39]: we add a dropout layer before the last convolution/linear layer generate the predictive distribution. To make fair comparison, we use the same model parameters as the others without retraining:
deep-ensembles [40]: we train the same network independently with different initialization T times, and ensemble them generate the predictive distribution. The deep-ensembles (PR) adopts the proposed U-SPA pipeline to make it work in practice;
full-net (E./S.): ): we implement Laplace approximation on the layers with the diagonal approx- imation of the diagonal empirical/standard Fisher;
layt- (F./S.): Laplace approximation applied on the last-layer only;
last-module (E./S.): Laplace approximation is applied on the R-CNN network.

---

(a)etteemiic uncertainly (Laplace approximation)
(b) ground-truth unceriainly (generative model [34])
Figure 3: Visualization of estimated epistemic uncertainty (left), and the ground-truth spatial un- certainty [34] used in evaluating mAP(IoU) (right). The top, middle, and bottom rows are front, hird's-eye, and zoom-in views. The contours reflect the objectiveness probability.
Table and Table 2 report the ECE and NLL results. The estimated predictive distributions are better calibrated than deterministic model outputs. Full-net Laplace approximation perform consis- tently better than MCDropout and DeepEnsembles baselines, if we compare the rows of full(E./S.) with dropout and deep-en. Table MCDropout models the posterior weight distribution of subset of parameters with Bernoulli distribution. It probably cannot characterize the real poste- rior. DeepEnsembles adopt I independently-trainn trainee optimal weight points to generate the predictive distribution. It can model the multi-modality in weight posterior, but each weight point is deter- ministic model and suffers from poor calibration.
We have two interesting findings. Firstly, the empirical Fisher results perform competitive results the standard Fisher counterparts Table 1 and 2, though it lacks theoretical foundations. We attribute this to the fact that we calculate the empirical Fisher at the optimal 0", which makes the predictions similar to the ground-trutt labels. Secondly, an increasing classification calibration qual- ity can be seen when Laplace approximation is applied to increasing number of layers, if we compare LL, LM, full in Table 1. The same trend can also be seen the NLL results (Table 2). It is oppo- site to the finding in [11], which claims the last-layer empirical Fisher owns the best performance and full-net Fisher suffers from an under-fitting problem. We attribute our result to the fact that we implement the Laplace approximation with diagonal approximation which contains much fewer parameters than the either full or K-FAC Fisher matrix used [11].
4.2 Accuracy of estimated mode
In this section, we explore the effect of Laplace approximation on detection accuracy by varying a, which the scalar factor of the posterior weight covariance matrix. When rior weight distribution will degenerate to deterministic model. When increases, the posterior weight distribution increasingly spans the weight space. We calculate the average mAP(IoU) over all classes and different levels denoted as Eelass,eeell [mAP(IoU) in Figure 4. It is found that as in- creases, Eclaes, Nevell (mA(((OU) decreases. It achieves its maximum at a = O, in which the model is deterministic and cannot estimate uncertainty. The mAP(IoU) degrades slightly when the a is small, while an extremely large value of a will greatly deteriorate the mAP(1oU). It suggests adopting a relatively small value of a to obtain the uncertainty and preserve the mAP(IoU) performance.
0, the poste-
4.3 Visualization and discussion on predictive distributions
To better understand the estimated epistemic uncertainty, we visualize our estimated predictive dis- tributions compared to those estimated with generative model in [34]. The generative model characterizes the generative process of key-object points within 3D bounding boxes. It generates the predictive distribution with ground-truth 3D bounding box annotations and LiDAR observations.

---

bev
3d
method class
easy
mod.
hard
easy mod.
hard
60
Edoos evelmAR/oU)
car
80.2
78.9 75.5 43.7
28.0
23.5
50
PP
ped.
19.2
20.8
20.7
29.9
29.8
27.6
cyc.
48.3
42.6
40.3
46.3
33.0
31.2
PP-standard
car
87.0
84.8
81.3
58.2
37.6
31.3
PR-empirical
SC
ped.
23.6
26.2
25.7
30.9
30.9
28.2
SC-standard
20
SC.empirical
cyc.
50.6
48.3
46.4
45.5
34.7
33.1
PR-standard
81.1
78.8
75.1
41.4
29.9
26.1
PR-empirical
PR
ped.
26.0
25.9
24.0
39.4
35.8
31.3
10
cyc.
62.7 51.9 49.4
61.3
43.1
40.7
Figure Relationship between a and lev (f) (full-net). The stars depict the results of the deterministic models. The results computed from random seeds.
Table 3: mAP(JIoU) (standard Fisher, full-net) re- sults on the KITTI vol set .The reported results are mean values from 5 random seeds.
We project the predictive distributions to the bird's-eye-view (BEV) image plane and get objective- ness probability for visualization. The objectiveness probability is the probability of BEV region subject to a key object, like cars, pedestrians, and cyclists. Figure 3(b) shows the objectiveness probability of the ground-truth generative model. The objectiveness probability of the ground-truth generative model is high around key objects. is relatively low in the region with sparse LiDAR ob- servations. We visualize estimated predictive distributions of Laplace approximation in Figure (a). shows similar behavior to the ground-trun generative model in Figure 3(b), and confident in dense-point regions and relatively uncertain when the LiDAR observation is sparse.
Table reports the mAP(JIoU) between predictive distributions estimated with Laplace approxima- tion and those with the generative model. It shows that the estimated epistemic uncertainty behaves like the generative model in relatively large objects, like cars and cyclists.
Limitations
The proposed method computes the Bayesian inference formula based Monte-Carlo, It requires Tx computational costs more than a single forward computation. The run-time performance can be improved if it parallels the Monte-Carle computations with batched inputs [39]. For the robotic application with strong constraints on computational resources, the subnet implementation that only applies Laplace approximation
the last layer also provides good-quality uncertainty estimations, as shown in Table 1. This paper assumes that both weight and predictive distributions conform to the normal distribution. is limited in characterizing the multi-modality. It can be partially alleviated by fitting Laplace approximation to each modality [41].
6 Conclusion
In this paper, we tailor Laplace approximation for 3D object detectors. An uncertainty-separatioo and-aggregation method is proposed to make it work in two-stage 3D object detectors. The experi- ment results on the KITTI dataset show that the estimated epistemic uncertainty is of better quality than deterministic models and Monte-Carie Dropout baselines. Even though it lacks theoretical foundations, empirical diagonal fisher shows competitive performance to standard diagonal fisher in Laplace approximation. Full-net diagonal Laplace approximation, which provides good uncertainty quality, is recommended in offline applications, like model diagnosis, active learning, and life-long learning. If the robotic applications have strong computational resource constraints, subnet Laplace approximation is a good balance between run- run-time performance and uncertainty quality. the fu- ture, it will be interesting to overcome the discussed limitations and explore the usefulness of the estimated epistemic uncertainties in downstream tasks, like path planning.

---

Acknowledgments
This work was supported by Guangdong Basic and Applied Basic Research Foundation, un- der project 202 2021B151512003,, Zhongshan Science and Technology Bureau Fundunder project 2020AG002, and Project of Hetao Shenzhen-Hong Kong Science and Technology Innovation Co- operation Zone(HZZB-KCZYD) 22220083,, awarded to Prof. Ming Liu.
References
[1] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and paltern recognition, pages 3354-3361. IEFF, 2012.
[2] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[3] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, Han, J. Ngiam, H. Zhao, A. Timofeev, Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] D. J. MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448 472, 1992.
[5] C. Blundell, Cornebise, Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622. PMLR, 2015.
[6] D. Kingma and M. Welling. Auto-encoding variational bayes. ICLR, 2014.
[7] D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameteriza- tion trick. Advances in neural information processing systems, 28, 2015.
[8] A. Wu, Nowozin, E. Meeds, R. E. Turner, M. Hernández-Lobato, and A. L. Gaunt. Deter- ministic variational inference for robust bayesian neural networks. In International Conference on Learning Representations, 2018.
[9] A. D. Cobb and B. Jalaian. Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting. In Uncertainty in Artificial Intelligence, pages 675-685. PMLR, 2021.
[10] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pages 681 1- 688. Citeseer, 2011.
[11] E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34, 2021.
[12] A. Kristiadi, M. Hein, and Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In International conference on machine learning, pages 5436-5446. PMIR, 2020.
[13] E. Daxberger, E. Nalisnick, J. U. Allingham, Antorán, and M. Hernández-Lobuee Bayesian deep learning via subnetwork inference. In International Conference Machine Learning, pages 2510-2521. PMLR, 2021.
[14] J. Lee, M. Humt, J. Feng, and R. Triebel. Estimating model uncertainty of neural networks in sparse information form. In International Conference on Machine Learning, pages 5702-5713. PMLR, 2020.

---

[15] H. Ritter, A. Botev, and D. Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations. ICLR Conferenee Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
[16] M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava. Fast and scalable bayesian deep learning by weight-perturaation in adam. In International Conference Ma- chine Learning, pages 2611 1-2620. PMIR, 2018.
[17] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, Quan, TT. Ramalho, A. Barnnkkk, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
[18] Kunstner, P. Hennig, and L. Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
[19] J. Martens and L. Sutskever. Training deep and recurrent networks with hessian-free optimiza- tion. In Neural networks: Tricks of the trade, pages 479-535. Springer, 2012.
[20] N. Roux, P.-A. Manzagol, and Y. Bengio. Topmoumoute online natural gradient algorithm. Advances in neural information processing systems, 20, 2007.
[21] J. Martens et al. Deep learning via hessian-free optimization. In ICML, volume 27, pages 735-742, 2010.
[22] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4490 4499, 2018.
[23] B. Yang, W. Luo, and R. Urtasun. Pixor:
Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7660, 2018.
[24] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10), 2018. ISSN 1424-8220. doi:10.3390/s18103333. URL https://www. mdpi. com/ 1424-8220/188 18/10/3337.
[25] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast en- coders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[26] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[27] S. Shi, X. Wang, and H. Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770-779, 2019.
[28] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi. X. Wang, and H. Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15529-10534 2020.
[29] C. R. Qi, O. Litany, K. He, and J. Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEF/CVF International Conference Computer Vision, pages 9277-9286, 2019.
[30] S. Shi, Z. Wang, X. Wang, and H. Li. Part-a 2 net: 3d part-aware and aggregation neural network for object detection from point cloud. arXiv preprint 1907:::197.00.5... 2(3). 2019.
[31] D. Hall, F. Dayoub, J. Skinner, H. Zhang, D. Miller, P. Corke, G. Camneio, A. Angelova, and N. Sünderhauf. Probabilistic object detection: Definition and evaluation. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1031-1040, 2020.
10

---

[32] D. Feng, L. Rosenbaum, C. Glaeser, F. Timm, and K. Dietmayer. Can we trust you? on calibration probabilistic object detector for autonomous driving. JEEE/RSJ International Conference Intelligent Robots and Systems (IROS) Workshop, 2019.
[33] D. Feng, L. Rosenbaum, F. Timm, and K. Dietmayer. Leveraging heteroscedastic aleatoric uncertainties for robust real-time lidar 3d object detection. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 1180-1288)) IEEE, 2019.
[34] Z. Wang, D. Feng, Y. Zhou, L. Rosenbaum, F. Timm, K. Dietmayer, M. Tomizuka, and W. Zhan. Inferring spatial uncertainty in object detection. In 2020 IEEE/RSJ Interna- lional Conference Intelligent Robots and Systems (IROS), pages 5792-5799, 2020. doi:
10. 10000/00004544...0 9340998.
[35] K. Alex, B. Vijay. and C. Roberto. Bayesian segnet: Model uncertainty in deep convolu- tional encoder-decoder architectures for scene understanding. In G. B. Tae-Kyun Kim, Ste- fanos Zafeiriou and K. Mikolajczyk., editors, Proceedings of the British Machine Vision Con- ference (BMVC), pages 12.1-57.12. BMVA Press, September 2017. ISBN 1-901725-60-X. URL https:: //xp http//xxoo.oooo///// wttpeddoo..o/g...../o wttpoo////...///.3333 1224//.31.55
[36] V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using cali- brated regression. International conference on machine learning, pagees 2796 2804. PMLR, 2018.
[37] M. P. Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[38] J. Martens and R. Grosse. Optimizing neural networks with kronecker- factoree approximate curvature. In International conference on machine learning, pages 2408-2417. PMLR, 2015.
[39] G. Yarin. Uncertainty deep learning. University of Cambridge, Cambridge, 2016.
[40] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[41] K. Kruschke. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, (5):658-676, 2010.


Reviews:
[{"review_id": "xyyb1H4jSpCr", "reviewer": "Reviewer_dWMY", "summary": "This paper describes an approach for estimating the epistemic uncertainty of DNN predictions for 3D object detection.  The main technical contribution is to propose a strategy for handling two properties of 3D object detectors that make it difficult to apply standard (Laplace approximation-based) uncertainty quantification methods: (i) the high dimensionality of the outputs (which makes calculating the required Hessian or Fisher Information Matrix prohibitively costly), and (ii) the fact that many of these models include nondifferentiable selection and ranking operations (which means that the overall predictors are not end-to-end differentiable, as required in order to evaluate the Hessian used in the Laplace approximation).  The paper proposes an approach to address these challenges using (i) Monte Carlo approximation and (ii) an uncertainty separation and aggregation method that splits over the nondifferentiable operations in two-stage predictors, respectively.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**  The paper addresses a problem that is clearly important (as it has safety implications for autonomous robots, among other applications) and proposes what appears to be a technically well-founded and clever approach to handling the nondifferentiability of two-stage 3D detectors by showing how to decompose the calculation of the required predictive distribution in a way that avoids the need to propagate derivatives through the (discrete) selection.  The experimental results also seem to show that the proposed method is practically effective, especially in comparison with prior alternatives.\n\n**Weaknesses:**  The paper’s most prominent weakness is that (in my opinion) it is very difficult to follow, primarily due to a lack of clarity and precision in the technical development.  \n\nFor example, the description of how to approximate the Fisher Information Matrix (presented in Section 3.1.1) is given primarily in the form of a (in my opinion quite vague) “natural language” description in a paragraph of text in lines 127 - 143; the only description approaching a precise mathematical or algorithmic formulation is Algorithm 1, which contains a joint distribution (in the calculation of the gradient) that i don’t think is actually defined in the paper.\n\nThe description of the uncertainty aggregation and separation approach is similarly opaque; once again this primarily takes the form of a long (similarly imprecise) “natural language” description in a pair of paragraphs in lines 144-170, which in turn reference a pictograph (Fig. 2) that I find to be basically incomprehensible.\n\n\nI highlight these two examples because these portions of the paper are supposed to be explaining its *two principal contributions*, yet they are so inscrutable that I am very skeptical a typical reader would actually be able to implement the approach they are meant to describe.   (Large portions of the experimental results section are similarly imprecise.)\n\nUnfortunately it is difficult for me to meaningfully comment on other aspects of the work because (due to this lack of clarity in the writing) I am not confident that I understand the proposed method well enough to offer an informed opinion.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend rejection on the basis that the paper as currently written does not effectively communicate the technical information that is supposed to be its principal contribution.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The clarity of the writing needs to be improved throughout, but especially in the presentation of the main technical results.  In general, “natural language” is a very poor medium for communicating algorithms or mathematics (precisely because it *is* so vague).  I would strongly encourage the authors to replace these narrative descriptions with precise mathematical statements.\n"}}, {"review_id": "RYFTDErjNz", "reviewer": "Reviewer_S5g8", "summary": "This paper presents an epistemic uncertainty estimation technique for 3D object detection. Specifically, a class of approximate Baysian inference method called Laplace Approximation is applied to uncertainty estimation in 3D object detection. The paper suggests three techniques for bringing Laplace Approximation to 3D object detection: (1) a technique of approximating fisher information matrix via monte carlo sampling from pre-specified normal and categorical distributions, (2) a technique of separating uncertainty estimates from region proposal network and RCNN, and (3) a heurstic based approach to weight prior determination. Experiments on KITTI dataset show that the Laplace Approximation performs better than widely used MC-dropout, and different versions of Laplace Approximation is ablated for further empirical insights.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Uncertainty estimation in deep learning models is an important and practical problem for robotics. This paper shows how to tailor Laplace Approximation to 3D object detection task by introducing several new techniques. As a result, the paper suggests an alternative method for episdemic uncertainty estimation over MC-dropout. The findings overall reflect the current state-of-the-art in Bayesian Deep Learning community, and can be a useful reference for their applications in 3D object detection tasks. There are also no notable grammar and spelling mistakes, and limitations are addressed in detail at the end of the paper. Code is also attached, and the video in the supplementary material helps in acquring the main points of the paper.\n\nHowever, I find that the paper requires significant revision in improving the clarity. Experiments require also significantly more evidence to strengthen the contributions of the paper. Concretely:\n\n1. Line 26-31/56-59: The claim that Laplace Approximation is limited to classification task requires some revision. Laplace Approximation is merely an inference tool to obtain the posterior distribution, and many of the existing works are not limited to classification. Some examples are given below, and I also note that many of the cited paper in that paragraph contains experiments on regression tasks, e.g.,\n[1] learning multiplicative interactions with bayesian neural networks for visual-inertial odometry, 2020.\n[2] sparse bayesian deep learning for dynamic system identification, 2020.\n[3] trust your robots! predictive uncertainty estimation of neural networks with sparse gaussian processes, 2021.\nIn my view, it would work better to argue and explain specific problems of directly applying laplace approximation to existing 3D object detection architectures, which will lead to three method contributions of this paper, instead of these lines.\nOn the same note, the argument on output dimenision can be misleading -- there are alternative formulation to get the dimensions to scale with weight dimensions.\n\n2. Line 90-126: some of these basic information can go to appendix, and rather use the space to describe the contributions of the paper. In my view, many details about the contribution is missing in later part of the text, whereas this basic information is rather explained in too much detail. Equally important, stating the challenges in line 123-126 without explanations on why, feels difficult to follow.\n\n3. Algorithm 1: there are so many unintroduced variables. I could not find what these symbols mean in the text as well. I guess that it means some prior parameters which is specified by the user. Howeover, having to guess makes it hard to read the paper. Moreover, the last equation  to compute the fisher, is not the formulation for diagonal approximation right? Computationally, that equation cannot be done due to its size.\n\n4. section 3.1.1/section 3.1.3: As a reader, I find these sections difficult to parse in depth, which is very unfortuante given that these texts are the main contributions of this paper. I do not get the following points:\n(a) why do you need to compute fisher as in equation 10, which poses the problem with outer dimensions. Just like cited laplace approximation methods, posterior distribution can be inferred in weight space, and predictive distribution can also be computed through marginalization of the posterior distribution\n(b) there exists libraries for laplace approximation and curvature computations. why do you need algorithm 1? \"because 3D object detection network is originally determinstic\" --> but all the pretrained networks are deterministic, if one does not consider the posterior distribution.\n(c) for prior weight determination, line 180, what do you mean by infinitely large value? why 30 percent quantile for each layer? (empirical rule of thumb is not convincing that it will work always). Figure 3 contains lagend that are not explained.\n(d) comparion with naive approach also do not reflect the state-of-the-art. How about using bayesian optimization or variational inference like existing literature? e.g., [4] Bayesian optimization meets laplace approximation for robotic introspection, 2020. [5] Sparse Uncertainty Representation in Deep Learning with Inducing Weights, 2021.\n\n5. Regarding experiments, deep ensembles must be added as additional baseline. Comparison only in KITTI is limited -- how about citiscape? This will show that the trend will follow also in other data-sets. Monte carlo run of 10 is limited, and I suggest to try 20-30.\n\n6. Run-time comparison is also needed. The authors state that \"The real-time performance can be improved if it parallels the M Monte-Carlo computations with batched input [32]\". Howeover, I cannot find the empirical evidence that real-time performance can be reached, or the stack is more efficient. It is highly recommended to implement such module, and report the run-time of all these systems, in order to show that contributions presented here is in principle, applicable to robotics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is a weak reject. I appreciated the contribution to adapt existing techniques from machine learning literature to practical robotic vision tasks. Reported problems and adjustments made to the pipeline were interesting (though I was not convinced apart from separation and aggregation of uncertainty for 3D object detection).\n\nHowever, I find that the presentation of the paper needs significant improvement, in order to comprehend the materials. Experiments require more work in terms of adding more compelling baselines, and evaluating run-time for examining its applicability to robotics. From methdological point of view, the prior weight determination part requries more motivations on design choices, and the fisher estimation using output samples with pre-specified distribution requires more clarifications -- all these points with respect to existing methods.\n\n----------------------------- Update after rebuttal --------------------------\n\nAfter the rebuttal, I have increased my recommendation to the weak accept. This was due to addition of another dataset for evaluation, and certain clarifications made in the revised paper, e.g., new figures that are helpful to pass the insights, clarifying the main contributions, etc.\n\nMaybe, though, further improving the writing could help. Specially, certain claims also need to be thoroughly checked, like output dimensions posing difficulties (which can be mitigated by working on a weight space formulation with certain approximations like KFAC), or clearer discussion about empirical fisher Vs gradient based ones, etc.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Other minor comments are:\n\n1. Section 2.3 should change the name to metrics and measures of uncertainty. Moreover, this part is not needed, since there is no contribution of the paper in that direction. I would use that space to discuss existing probabilistic object detection and what advancements the paper is making with respect to that literature.\n\n2. Figure 1: I would use KITTI and realistic object detection architecture to validate this point, rather than using mnist with some linear model. This is because, there are suddle difference between above two set ups, and mnist with linear model does not validate that the fisher matrix converges with enough samples.\n\n3. Line 187-193: what is exact difference between point 1 and point 3? are they both validated with empirical data?\n\n4. Error bars needed for the results, especially table 2 and figure 5."}}, {"review_id": "KWdkMaJUUx", "reviewer": "Reviewer_mzVN", "summary": "The authors present a method of applying Laplace approximation (LA) to one- and two-stage 3D object detectors for epistemic uncertainty estimation. In the two-stage case, LA cannot directly be applied as some steps in the pipeline are non-differentiable. The presented solution consists of individual application of LA to both stages and an aggregation step. Further, a heuristic for prior estimation, i.e. regularization of LA, is established through experimentation. In a comparison to a deterministic and a Monte Carlo Dropout baseline on the KITTI dataset, superiority in uncertainty quality, i.e. calibration, is established.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n* Practical approach to uncertainty estimation in 3D deep neural networks (DNNs).\n* Detailed and largely consistent exposition (e.g. LA background section).\n* Helpful algorithm summaries.\n* Comparison of empirical and standard Fisher performance.\n* Comparison of varying degrees of inclusion of layers into the LA computation.\n* Insightful discussion of limitations.\n\n## Weaknesses\n* Lack of originality. The application of LA to 3D object detection is rather straight forward and mostly combines existing algorithms and model architectures.\n* The Fisher information matrix is not merely a \"good substitute\" (line 119) to the Hessian but equivalent to it given piecewise linear activation functions and exponential family loss functions (this does not hold for the empirical Fisher).\n* Missing distinction between the Fisher and Generalized Gauss-Newton (GNN) matrix which are only equivalent in the special case of exponential family loss functions.\n* Ad-hoc method for posterior regularization.\n* ECE as sole metric for uncertainty quality quantification and missing discussion of its limitations. NLL might be a worthwhile inclusion in the regression case.\n* Uncertainty quality is reduced to calibration. Separation of out-of-distribution examples (KL-divergence) could be included.\n* Only diagonal LA.\n* Missing code.\n* No discussion of future work.\n* Slight language issues, e.g.:\n  * _admirable_ vs _desirable_ (line 1)\n  * _indifferentiable_ vs. _non-differentiable_ (line 4)\n  * _adorable_ vs _useful/desirable_ (line 62)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper presents a practical approach to uncertainty estimation in 3D object detection which is quite relevant for robotics applications, missing originality and lacking experimental evaluation and discussion hold it back.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Clear distinction between Hessian, Fisher and GNN.\n* Potential comparison to more expressive Posteriors than isotropic Gaussian like KFAC.\n* Influence of number of MC samples during inference.\n* Further evaluation of uncertainty quality going beyond calibration. Inclusion of additional metrics like NLL or KL-divergence.\n* Clearer discussion of benefits of uncertainty estimation for the 3D object detection case (autonomous driving and robotics). What is possible when using the proposed approach that was not possible before?\n* Inclusion of future work and code."}}, {"review_id": "3Eaqj_jfIw_", "reviewer": "Reviewer_z9CZ", "summary": "This work proposed to estimate the epistemic uncertainty of a deep 3D object detector by treating the deep neural net as a Bayesian Neural Network (BNN) whose weight posterior is inferred by Laplace approximation(LA).  In order to conquer different issues such as high-dimensional output space and in-differentiable operations during the adaptation, the authors employed the \"standard Fisher\" approximation along with the empirical one and introduced a moment matching processes to collect uncertainty from different modules. Besides, a heuristic method to select the hyper-parameters of the Covariance matrix has been adopted to boost the performance. In experimental validation, the proposed approach is demonstrated better performance over deterministic networks and Monte-Carlo dropout on KITTI dataset, with the metric of expected calibration error for both classification and regression. The performance of mAP (JIoU), another metric for predictive distribution, is also reported.   ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the reviewer appreciates the authors' work because it is a quite relevant and urgent topic for robotic perception in safety-critical applications. Though there has been a plethora of advances in uncertainty estimation within the field of machine learning, to adapt them on more practical tasks and validate their effectiveness is also imperative and of greater importance. Besides the topic relevance, the presentation is generally clear and well-organized, easy to follow though with some flaws mentioned later. Nevertheless there is room to improve in terms of presentation, algorithms and experiments from reviewer's perspective. The reviewer hopes that the following suggestions are helpful on improving the quality of the paper.\n\n#### Presentation\n- the main text spends much space on introducing the full Laplace Approximation with less focus on the diagonal variant which is the one used finally. This gives the reader a misleading impression. Though there are motivations such as memory issue for only using the diagonal variant but they have not been discussed thoroughly. \n-  it would be more clear to move the introduction of LA in section 3 into a separate section for preliminaries or background; Further this section would be better to include a brief description of the commonly used architecture for 3D object detectors. Currently, there is no definition of RPN and R-CNN except for their appearance in the related work section. \n- regarding two different ways of computing the Fisher (empirical and standard) in section 3.1.1, it would be more clear to put the equations of both ways in the main text since this is one of the relevant parts of the method.\n- it would be much more concise if the authors can use equations to describe the uncertainty separation and aggregation module instead of Fig.2 which has many undefined letters and such as N, W, L, M and so on. And this figure is not a bit too complex in communicating the main idea.\n- better to explain with equation for section 3.1.3;   \n- experiments:\n     - in Table 1 `std.` is ambiguous since it can also mean standard deviation. \n     - in Fig.3 the figure size is too small, while the width of the curves is large.\n     - in Fig.5 there is no `stars` in the figure but it is mentioned in the caption.\n\n#### Algorithms\n-  the foremost concern is that, the authors highlight the problem of high-dimensional output space when applying LA and suggest a Monte-Carlo approximation for standard Fisher (evaluation over the expectation of predictive distribution) to get around the intractable analytical form, which is fine. But this is not the problem of empirical Fisher (evaluation over the expectation of data distribution). And in the experiments, the results show that empirical can have better performance than the standard one. The question here is, why don't the authors just use the empirical version? What's the main motivation of employing standard Fisher? This is a significant but very interesting point. It would be great if the author can explain the main difference in details between these two, and try to validate the insights from the previous works in theory and verify if these insights are reflected in a more practical setting or if they are helpful in guiding the design for a more effective algorithm.  \n\n- The second concern is still around LA. It's understandable to use diagonal variant only with the motivation of resources-restricted requirements in robotics. But the problem is that,  what kind of role does the relation between weights play in tasks like 3D object detection, indispensable or ignorant? Since the run-time problem from sampling is anyway there, therefore there is no difference on this aspect for the variants with richer relation modeling between weights such as K-FAC [1]. The authors mentioned this in the related work but neglect it without a persuasive argumentation. \n\n- Regarding the uncertainty separation and aggregation, there are still in-differential operations such as ranking and selection in the proposed module, how does the information flow back and differ with the old model? Why to use addition for uncertainty aggregation instead of other operations such as multiplication or maximum and so on?\n\n- What's the main difference between the heuristic approach in section 3.1.3 and hyper-parameter tuning based on validation set?\n\n#### Experiments\n- As mentioned before in the algorithm part, the reviewer would like to see the comparison with some more generally accepted baseline such as deep Ensemble [3]; It would also be great to compare with K-FAC [1] or at least to provide more arguments why it's not considered in this task.\n\n- In table 4, it's a little unreasonable to have zero standard deviation in classification for PP and SC. Have the authors tried to vary different number of samples drawn from the weight posterior?\n\n- It would be better to have a ablation study on the uncertainty separation and aggregation module, which can provide more evidence that this module can work and the analysis of the mechanism behind is valuable as well.\n\n- It would be better to provide comparison against other baselines in experiment 4.3. \n\n#### Limitation\n\n- Besides the parallel computation for sampling, some approach on uncertainty propagation such as [2] can be used to mitigate the run-time inefficiency problem.\n \n\n[1] Martens, James, and Roger Grosse. \"Optimizing neural networks with kronecker-factored approximate curvature.\" International conference on machine learning. PMLR, 2015.\n\n[2] Postels, Janis, et al. \"Sampling-free epistemic uncertainty estimation using approximated variance propagation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n[3] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems 30 (2017).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "According to the strengths and weakness section, the reviewer would like to weakly reject this work at current status. Though the overall idea is relevant and interesting, the lack of clear motivation in technical details and evident experimental evaluations on sub-components, data sets and with other baselines constitutes the main concern for the reviewer when assessing this work. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As mentioned in the section of strengths and weakness, if the majority of the aforementioned points can be addressed, the reviewer would consider to raise the score of this work. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)