Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper describes an approach for estimating the epistemic uncertainty of DNN predictions for 3D object detection.  The main technical contribution is to propose a strategy for handling two properties of 3D object detectors that make it difficult to apply standard (Laplace approximation-based) uncertainty quantification methods: (i) the high dimensionality of the outputs (which makes calculating the required Hessian or Fisher Information Matrix prohibitively costly), and (ii) the fact that many of these models include nondifferentiable selection and ranking operations (which means that the overall predictors are not end-to-end differentiable, as required in order to evaluate the Hessian used in the Laplace approximation).  The paper proposes an approach to address these challenges using (i) Monte Carlo approximation and (ii) an uncertainty separation and aggregation method that splits over the nondifferentiable operations in two-stage predictors, respectively.

strengths_and_weaknesses: **Strengths:**  The paper addresses a problem that is clearly important (as it has safety implications for autonomous robots, among other applications) and proposes what appears to be a technically well-founded and clever approach to handling the nondifferentiability of two-stage 3D detectors by showing how to decompose the calculation of the required predictive distribution in a way that avoids the need to propagate derivatives through the (discrete) selection.  The experimental results also seem to show that the proposed method is practically effective, especially in comparison with prior alternatives.

**Weaknesses:**  The paper’s most prominent weakness is that (in my opinion) it is very difficult to follow, primarily due to a lack of clarity and precision in the technical development.  

For example, the description of how to approximate the Fisher Information Matrix (presented in Section 3.1.1) is given primarily in the form of a (in my opinion quite vague) “natural language” description in a paragraph of text in lines 127 - 143; the only description approaching a precise mathematical or algorithmic formulation is Algorithm 1, which contains a joint distribution (in the calculation of the gradient) that i don’t think is actually defined in the paper.

The description of the uncertainty aggregation and separation approach is similarly opaque; once again this primarily takes the form of a long (similarly imprecise) “natural language” description in a pair of paragraphs in lines 144-170, which in turn reference a pictograph (Fig. 2) that I find to be basically incomprehensible.


I highlight these two examples because these portions of the paper are supposed to be explaining its *two principal contributions*, yet they are so inscrutable that I am very skeptical a typical reader would actually be able to implement the approach they are meant to describe.   (Large portions of the experimental results section are similarly imprecise.)

Unfortunately it is difficult for me to meaningfully comment on other aspects of the work because (due to this lack of clarity in the writing) I am not confident that I understand the proposed method well enough to offer an informed opinion.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend rejection on the basis that the paper as currently written does not effectively communicate the technical information that is supposed to be its principal contribution.

issues: The clarity of the writing needs to be improved throughout, but especially in the presentation of the main technical results.  In general, “natural language” is a very poor medium for communicating algorithms or mathematics (precisely because it *is* so vague).  I would strongly encourage the authors to replace these narrative descriptions with precise mathematical statements.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an epistemic uncertainty estimation technique for 3D object detection. Specifically, a class of approximate Baysian inference method called Laplace Approximation is applied to uncertainty estimation in 3D object detection. The paper suggests three techniques for bringing Laplace Approximation to 3D object detection: (1) a technique of approximating fisher information matrix via monte carlo sampling from pre-specified normal and categorical distributions, (2) a technique of separating uncertainty estimates from region proposal network and RCNN, and (3) a heurstic based approach to weight prior determination. Experiments on KITTI dataset show that the Laplace Approximation performs better than widely used MC-dropout, and different versions of Laplace Approximation is ablated for further empirical insights.  
strengths_and_weaknesses: Uncertainty estimation in deep learning models is an important and practical problem for robotics. This paper shows how to tailor Laplace Approximation to 3D object detection task by introducing several new techniques. As a result, the paper suggests an alternative method for episdemic uncertainty estimation over MC-dropout. The findings overall reflect the current state-of-the-art in Bayesian Deep Learning community, and can be a useful reference for their applications in 3D object detection tasks. There are also no notable grammar and spelling mistakes, and limitations are addressed in detail at the end of the paper. Code is also attached, and the video in the supplementary material helps in acquring the main points of the paper.

However, I find that the paper requires significant revision in improving the clarity. Experiments require also significantly more evidence to strengthen the contributions of the paper. Concretely:

1. Line 26-31/56-59: The claim that Laplace Approximation is limited to classification task requires some revision. Laplace Approximation is merely an inference tool to obtain the posterior distribution, and many of the existing works are not limited to classification. Some examples are given below, and I also note that many of the cited paper in that paragraph contains experiments on regression tasks, e.g.,
[1] learning multiplicative interactions with bayesian neural networks for visual-inertial odometry, 2020.
[2] sparse bayesian deep learning for dynamic system identification, 2020.
[3] trust your robots! predictive uncertainty estimation of neural networks with sparse gaussian processes, 2021.
In my view, it would work better to argue and explain specific problems of directly applying laplace approximation to existing 3D object detection architectures, which will lead to three method contributions of this paper, instead of these lines.
On the same note, the argument on output dimenision can be misleading -- there are alternative formulation to get the dimensions to scale with weight dimensions.

2. Line 90-126: some of these basic information can go to appendix, and rather use the space to describe the contributions of the paper. In my view, many details about the contribution is missing in later part of the text, whereas this basic information is rather explained in too much detail. Equally important, stating the challenges in line 123-126 without explanations on why, feels difficult to follow.

3. Algorithm 1: there are so many unintroduced variables. I could not find what these symbols mean in the text as well. I guess that it means some prior parameters which is specified by the user. Howeover, having to guess makes it hard to read the paper. Moreover, the last equation  to compute the fisher, is not the formulation for diagonal approximation right? Computationally, that equation cannot be done due to its size.

4. section 3.1.1/section 3.1.3: As a reader, I find these sections difficult to parse in depth, which is very unfortuante given that these texts are the main contributions of this paper. I do not get the following points:
(a) why do you need to compute fisher as in equation 10, which poses the problem with outer dimensions. Just like cited laplace approximation methods, posterior distribution can be inferred in weight space, and predictive distribution can also be computed through marginalization of the posterior distribution
(b) there exists libraries for laplace approximation and curvature computations. why do you need algorithm 1? "because 3D object detection network is originally determinstic" --> but all the pretrained networks are deterministic, if one does not consider the posterior distribution.
(c) for prior weight determination, line 180, what do you mean by infinitely large value? why 30 percent quantile for each layer? (empirical rule of thumb is not convincing that it will work always). Figure 3 contains lagend that are not explained.
(d) comparion with naive approach also do not reflect the state-of-the-art. How about using bayesian optimization or variational inference like existing literature? e.g., [4] Bayesian optimization meets laplace approximation for robotic introspection, 2020. [5] Sparse Uncertainty Representation in Deep Learning with Inducing Weights, 2021.

5. Regarding experiments, deep ensembles must be added as additional baseline. Comparison only in KITTI is limited -- how about citiscape? This will show that the trend will follow also in other data-sets. Monte carlo run of 10 is limited, and I suggest to try 20-30.

6. Run-time comparison is also needed. The authors state that "The real-time performance can be improved if it parallels the M Monte-Carlo computations with batched input [32]". Howeover, I cannot find the empirical evidence that real-time performance can be reached, or the stack is more efficient. It is highly recommended to implement such module, and report the run-time of all these systems, in order to show that contributions presented here is in principle, applicable to robotics.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My recommendation is a weak reject. I appreciated the contribution to adapt existing techniques from machine learning literature to practical robotic vision tasks. Reported problems and adjustments made to the pipeline were interesting (though I was not convinced apart from separation and aggregation of uncertainty for 3D object detection).

However, I find that the presentation of the paper needs significant improvement, in order to comprehend the materials. Experiments require more work in terms of adding more compelling baselines, and evaluating run-time for examining its applicability to robotics. From methdological point of view, the prior weight determination part requries more motivations on design choices, and the fisher estimation using output samples with pre-specified distribution requires more clarifications -- all these points with respect to existing methods.

----------------------------- Update after rebuttal --------------------------

After the rebuttal, I have increased my recommendation to the weak accept. This was due to addition of another dataset for evaluation, and certain clarifications made in the revised paper, e.g., new figures that are helpful to pass the insights, clarifying the main contributions, etc.

Maybe, though, further improving the writing could help. Specially, certain claims also need to be thoroughly checked, like output dimensions posing difficulties (which can be mitigated by working on a weight space formulation with certain approximations like KFAC), or clearer discussion about empirical fisher Vs gradient based ones, etc.
issues: Other minor comments are:

1. Section 2.3 should change the name to metrics and measures of uncertainty. Moreover, this part is not needed, since there is no contribution of the paper in that direction. I would use that space to discuss existing probabilistic object detection and what advancements the paper is making with respect to that literature.

2. Figure 1: I would use KITTI and realistic object detection architecture to validate this point, rather than using mnist with some linear model. This is because, there are suddle difference between above two set ups, and mnist with linear model does not validate that the fisher matrix converges with enough samples.

3. Line 187-193: what is exact difference between point 1 and point 3? are they both validated with empirical data?

4. Error bars needed for the results, especially table 2 and figure 5.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposed to estimate the epistemic uncertainty of a deep 3D object detector by treating the deep neural net as a Bayesian Neural Network (BNN) whose weight posterior is inferred by Laplace approximation(LA).  In order to conquer different issues such as high-dimensional output space and in-differentiable operations during the adaptation, the authors employed the "standard Fisher" approximation along with the empirical one and introduced a moment matching processes to collect uncertainty from different modules. Besides, a heuristic method to select the hyper-parameters of the Covariance matrix has been adopted to boost the performance. In experimental validation, the proposed approach is demonstrated better performance over deterministic networks and Monte-Carlo dropout on KITTI dataset, with the metric of expected calibration error for both classification and regression. The performance of mAP (JIoU), another metric for predictive distribution, is also reported.   
strengths_and_weaknesses: Overall, the reviewer appreciates the authors' work because it is a quite relevant and urgent topic for robotic perception in safety-critical applications. Though there has been a plethora of advances in uncertainty estimation within the field of machine learning, to adapt them on more practical tasks and validate their effectiveness is also imperative and of greater importance. Besides the topic relevance, the presentation is generally clear and well-organized, easy to follow though with some flaws mentioned later. Nevertheless there is room to improve in terms of presentation, algorithms and experiments from reviewer's perspective. The reviewer hopes that the following suggestions are helpful on improving the quality of the paper.

#### Presentation
- the main text spends much space on introducing the full Laplace Approximation with less focus on the diagonal variant which is the one used finally. This gives the reader a misleading impression. Though there are motivations such as memory issue for only using the diagonal variant but they have not been discussed thoroughly. 
-  it would be more clear to move the introduction of LA in section 3 into a separate section for preliminaries or background; Further this section would be better to include a brief description of the commonly used architecture for 3D object detectors. Currently, there is no definition of RPN and R-CNN except for their appearance in the related work section. 
- regarding two different ways of computing the Fisher (empirical and standard) in section 3.1.1, it would be more clear to put the equations of both ways in the main text since this is one of the relevant parts of the method.
- it would be much more concise if the authors can use equations to describe the uncertainty separation and aggregation module instead of Fig.2 which has many undefined letters and such as N, W, L, M and so on. And this figure is not a bit too complex in communicating the main idea.
- better to explain with equation for section 3.1.3;   
- experiments:
     - in Table 1 `std.` is ambiguous since it can also mean standard deviation. 
     - in Fig.3 the figure size is too small, while the width of the curves is large.
     - in Fig.5 there is no `stars` in the figure but it is mentioned in the caption.

#### Algorithms
-  the foremost concern is that, the authors highlight the problem of high-dimensional output space when applying LA and suggest a Monte-Carlo approximation for standard Fisher (evaluation over the expectation of predictive distribution) to get around the intractable analytical form, which is fine. But this is not the problem of empirical Fisher (evaluation over the expectation of data distribution). And in the experiments, the results show that empirical can have better performance than the standard one. The question here is, why don't the authors just use the empirical version? What's the main motivation of employing standard Fisher? This is a significant but very interesting point. It would be great if the author can explain the main difference in details between these two, and try to validate the insights from the previous works in theory and verify if these insights are reflected in a more practical setting or if they are helpful in guiding the design for a more effective algorithm.  

- The second concern is still around LA. It's understandable to use diagonal variant only with the motivation of resources-restricted requirements in robotics. But the problem is that,  what kind of role does the relation between weights play in tasks like 3D object detection, indispensable or ignorant? Since the run-time problem from sampling is anyway there, therefore there is no difference on this aspect for the variants with richer relation modeling between weights such as K-FAC [1]. The authors mentioned this in the related work but neglect it without a persuasive argumentation. 

- Regarding the uncertainty separation and aggregation, there are still in-differential operations such as ranking and selection in the proposed module, how does the information flow back and differ with the old model? Why to use addition for uncertainty aggregation instead of other operations such as multiplication or maximum and so on?

- What's the main difference between the heuristic approach in section 3.1.3 and hyper-parameter tuning based on validation set?

#### Experiments
- As mentioned before in the algorithm part, the reviewer would like to see the comparison with some more generally accepted baseline such as deep Ensemble [3]; It would also be great to compare with K-FAC [1] or at least to provide more arguments why it's not considered in this task.

- In table 4, it's a little unreasonable to have zero standard deviation in classification for PP and SC. Have the authors tried to vary different number of samples drawn from the weight posterior?

- It would be better to have a ablation study on the uncertainty separation and aggregation module, which can provide more evidence that this module can work and the analysis of the mechanism behind is valuable as well.

- It would be better to provide comparison against other baselines in experiment 4.3. 

#### Limitation

- Besides the parallel computation for sampling, some approach on uncertainty propagation such as [2] can be used to mitigate the run-time inefficiency problem.
 

[1] Martens, James, and Roger Grosse. "Optimizing neural networks with kronecker-factored approximate curvature." International conference on machine learning. PMLR, 2015.

[2] Postels, Janis, et al. "Sampling-free epistemic uncertainty estimation using approximated variance propagation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

[3] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: According to the strengths and weakness section, the reviewer would like to weakly reject this work at current status. Though the overall idea is relevant and interesting, the lack of clear motivation in technical details and evident experimental evaluations on sub-components, data sets and with other baselines constitutes the main concern for the reviewer when assessing this work. 
issues: As mentioned in the section of strengths and weakness, if the majority of the aforementioned points can be addressed, the reviewer would consider to raise the score of this work.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a method of applying Laplace approximation (LA) to one- and two-stage 3D object detectors for epistemic uncertainty estimation. In the two-stage case, LA cannot directly be applied as some steps in the pipeline are non-differentiable. The presented solution consists of individual application of LA to both stages and an aggregation step. Further, a heuristic for prior estimation, i.e. regularization of LA, is established through experimentation. In a comparison to a deterministic and a Monte Carlo Dropout baseline on the KITTI dataset, superiority in uncertainty quality, i.e. calibration, is established.
strengths_and_weaknesses: ## Strength
* Practical approach to uncertainty estimation in 3D deep neural networks (DNNs).
* Detailed and largely consistent exposition (e.g. LA background section).
* Helpful algorithm summaries.
* Comparison of empirical and standard Fisher performance.
* Comparison of varying degrees of inclusion of layers into the LA computation.
* Insightful discussion of limitations.

## Weaknesses
* Lack of originality. The application of LA to 3D object detection is rather straight forward and mostly combines existing algorithms and model architectures.
* The Fisher information matrix is not merely a "good substitute" (line 119) to the Hessian but equivalent to it given piecewise linear activation functions and exponential family loss functions (this does not hold for the empirical Fisher).
* Missing distinction between the Fisher and Generalized Gauss-Newton (GNN) matrix which are only equivalent in the special case of exponential family loss functions.
* Ad-hoc method for posterior regularization.
* ECE as sole metric for uncertainty quality quantification and missing discussion of its limitations. NLL might be a worthwhile inclusion in the regression case.
* Uncertainty quality is reduced to calibration. Separation of out-of-distribution examples (KL-divergence) could be included.
* Only diagonal LA.
* Missing code.
* No discussion of future work.
* Slight language issues, e.g.:
  * _admirable_ vs _desirable_ (line 1)
  * _indifferentiable_ vs. _non-differentiable_ (line 4)
  * _adorable_ vs _useful/desirable_ (line 62)
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the paper presents a practical approach to uncertainty estimation in 3D object detection which is quite relevant for robotics applications, missing originality and lacking experimental evaluation and discussion hold it back.
issues: * Clear distinction between Hessian, Fisher and GNN.
* Potential comparison to more expressive Posteriors than isotropic Gaussian like KFAC.
* Influence of number of MC samples during inference.
* Further evaluation of uncertainty quality going beyond calibration. Inclusion of additional metrics like NLL or KL-divergence.
* Clearer discussion of benefits of uncertainty estimation for the 3D object detection case (autonomous driving and robotics). What is possible when using the proposed approach that was not possible before?
* Inclusion of future work and code.
review_title: null

Meta-review: metareview: All reveiwers appreciate the relevance of the uncertainty estimation problem to the deep learning domain and its applicability in robotics. In particular, the proposed approach for 3D object detection is interetsing and extends current LA based techniques to this important problem. However, there are also important concerns regarding the clarity of the descripton and the experimental evaluation. For the former, a more mathematically concise formulation is required with a better definition of symbols and less descriptive "natural" language. For the latter, experimental comarisons to state-of-the-art methods such as deep ensembles or K-FAC are necessary. 

Post-rebuttal:
The authors have done a good job addressing the points raised by the reviewers. They have shown additional experimental results in the revised version of the paper, which has improved significantly. As the main open point, it remains the question why the theoretically not fully motivated approach of the empirical Fisher often performs better that the theoretically more sound method using the standard Fisher. In the rebuttal, the authors claim that this is due to the diagonal approximation used in the standard Fisher, but this needs to be verified at least empirically. Still, as it is, the paper is a worthwile contribution to the field of uncertainty-aware robotic perception and an interesting uncertainty-based extension to the problem of 3D object detection. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)