Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of introspection, an intrinsic understanding of how confident the model is in its own predictions. The paper provides a theoretical connection between DNNs and MoE-GPs and finally provides a practical learning application. Experiments are carried out on data sets derived from robotic sampling and inverse dynamics problems. 

strengths_and_weaknesses: Strengths
- The paper addresses an important problem of uncertainty estimation in prediction. The work provides a theoretical foundation from a Bayesian DNN and GPs viewpoint which culminates in a practical learning algorithm. 
- The paper provides a scalable approach using a mixture of GPs approach and other techniques to reduce the online inference time making them suitable for robotics application. 
Results on real robotics data sets. 

Weaknesses
- The approach in the current form does not distinguish aleatoric and epistemic uncertainty in predictions and hence is likely to be less resilient to model uncertainty. 

summary_of_recommendation: This paper contributes a theoretical framework for uncertainty estimation as well as a practical learning algorithm. The result is likely to have impact on real robotics applications. 

issues: Issues
- I would recommend improvements in the technical exposition of the material. For example, Section 3 can provide a roadmap for how the proof will develop. 
- Does the current system has a way of forgetting points. In an online application, the incoming data may exceed available memory. Hence, is there a possibility for a principled approach for removing points as is done in Informative Vector Machines (IVMs)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach to equipping trained DNNs with a measure of epistemic uncertainty. Leveraging a connection between the local liberalization of a DNN and the resulting equivalent GP (using the neural tangent kernel), the authors propose to use this GP to characterize the uncertainty on test points. The approach approximates this equivalent GP using a mixture-of-experts GP model, wherein a hard gating function divides the input into disjoint sets, each of which correspond to a separate GP. These GPs have a kernel with the same features equal to the Jacobian of the DNN. The approach requires (1) performing approximate kernel k-means to design the gating function and split the training data into subsets; (2) fitting a NTK GP to each of the subsets, by (2a) taking an active learning approach to include only a limited number of informative datapoints in the gram matrix for each GP; and (2b) pruning features by removing elements of the Jacobian which have small values and thus are less relevant to the kernel. To ensure smoothness across the separate GPs, each GP also includes the data from some of its neighbors. At test time, one can compute an uncertainty estimate by constructing evaluating the gating function to determine the expert GP to to evaluate, and subsequently evaluating the kernel for that GP on the test input and the selected training data for that expert, and finally, using this information to compute the posterior covariance.

The authors provide theoretical motivation for their approach justifying why the MoE GP is a good approximation of the DNN uncertainty.
strengths_and_weaknesses: This paper presents an interesting approach exploiting a connection between local linearization of a DNN and GPs. The presented approach takes this theoretical idea, and presents an algorithm that yields tractable inference times at test time. While the method is complex, the paper included helpful figures which communicated the ideas well. I especially liked Figure 3 in this regard. I also appreciated the practical results, demonstrating that the approach can be deployed in hardware, and offers good OOD detection performance over using mc-dropout. 

However, there are several areas for improvement.

First, I found the paper to be occasionally hard to follow, especially in the theoretical derivations. The jump from DNN setup to the NLM was abrupt, without explanation providing intuition for the particular choice of covariances. The construction of the practical model also felt disconnected from the theoretical derivation, specifically, while the NLM treats the output as having additive gaussian noise with covariance equal to the inverse Hessian of the loss (eq. 1), a quantity that is input dependent, the predictions are made assuming equal noise for each sample of \sigma_0. Furthermore, eq. 3, connecting the GP regression uncertainty to a classification task, lacked theoretical justification beyond an intuitive notion that it would have the right effect on the predictive uncertainty. This felt at odds with the fully general derivation of the NLM. 

The theoretical results could be presented more clearly. While I appreciated the visual description of the theoretical results in Figure 2, I felt that the choice to start by viewing a single DNN as a mixture of expert DNNs with identical networks for each expert, and subsequently showing that each expert can be cast as a GP. In contrast the premise of the approach was to approximate the posterior predictive uncertainty of the full DNN, in which case it would have been clearer to consider the GP corresponding to the full DNN, and subsequently show how the MoE-GP approximates the full GP. Furthermore, the paper could be strengthened by including theoretical results bounding the errors made by using the various approximation techniques applied to make the approach practical: the active learning for data subsampling, and the feature pruning.

Second, the scalability of this approach is unclear. While the authors take several steps to make the approach more practical, including subsampling data and breaking the data into subsets, the kernel-GP approach taken requires storing significant amounts of data at test time (even if all of the data isn’t used in evaluation). In addition, the authors use separate GPs per output dimension, meaning the approach cannot capture correlations across output dimensions, or scale easily to higher-dimensional outputs (e.g. semantic segmentation tasks). Also, while the authors demonstrate the approach on a large dataset in simulation, the on-robot results use a much smaller dataset. Furthermore, the appendix suggests that the results in the paper also only consider the Jacobian of the last few layers of the network, rather than the whole DNN. This hinders the authors’ argument for the scalability and efficiency of the approach.

Finally, the authors should compare to more standard benchmarks, in addition to MC-dropout. Experiments comparing to Deep Ensembles are crucial, as they are a commonly used tool for DNN uncertainty, and can give good results even with a low number of ensemble members, allowing for efficient real-time application. Furthermore, there are recent works considering different Laplace approximations that are closely related to a weight-space view of the DNN GP, that are worth discussing if not comparing against [1,2].


[1] Madras et al., “Detecting Extrapolation with Local Ensembles,” ICLR 2019
[2] Sharma et al., “Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks,” UAI 2021

summary_of_recommendation: Overall, I think this paper presents some interesting, novel ideas, but the message is unclear, and the practical implementation of the idea has some shortcomings in its ability to scale. The paper would be improved with improved clarity in the presentation of the ideas and results, and a more thorough experimental comparisons highlighting the benefit of this approach over existing baselines for predictive uncertainty estimation. 

---
Having read the revisions, I believe the authors have addressed my issues and I have increased my score.
issues: - The authors should, if possible, refactor the theoretical results as suggested in the main review section, or provide more justification why the posterior predictive uncertainty of the MoE-DNN model is a useful analog to the posterior predictive uncertainty of a single GP.
- Address the mismatch between the NLM derivation and the GP expressions used in practice (specifically, the choice of observation noise)
- The main missing experiments are comparisons to Deep Ensembles.
- More details on the stopping criterion for the active learning and Jacobian pruning strategies should be given.
- The discussion should highlight the fact that this method requires access to training data during the inference procedure.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is motivated by predictive uncertainty that can be used on mobile robotic platforms (in this case MAVs). 
The proposed model in a mixture-of-experts of BNNs, specfically the neural tangent kernel. 
The model is evaluted on inverse dynamics learning and object detection. 
strengths_and_weaknesses: Fast predictive models with useful uncertainty quantification would be a useful tool in robotics.
Improving the uncertainty quantification of BNNs to be as good as GPs is an active topic of research.  

My central issue with this paper is that I couldn't tell if it was a BNN paper or robotics paper. 
The real robot experiment and motivation is cool and relevant to CoRL, while the actual contribution of the paper is as a BNN.
Moreover, w.r.t. to the robotic applications, it is not clear what uncertainty is needed for, i.e. what is the downstream task (active learning, Bayesian optimization, etc)?
As a BNN paper, the baselines and experiments were somewhat limited compared to standard studies.  

1) The neural linear model is actually an established class of Bayesian neural network (i.e. [1,2]), so you need a different term. This family should be included in the related work.

2) As I understand it, the NTK is more a tool for understanding the NN/GP connection rather than a practical model, since it comes with the kernel scaling issues and the need to compute Jacobians.

3) The core issue with this paper is that key design decsisions are centered on making BNNs like GPs, but GPs are never evaluated as a baseline. Since so much work is done to make the NTK scale to the small partioned small data, why aren't GP's evaluated for regression? The paper does not cite [3], which is a GP version of the idea presented here.

4) I was frequently confused by the memory requirements of your problem setting. Ensembles and GPs are described as being too memory intensive, and yet the MoE-GP model seems to cache an awful lot of memory (pre-computing various quanities and lots of models). I would like to see a comparison between the memory footprint of a MoE-GP vs ensembles and GPs to validate these claims. 
 
5). 'To our knowledge, we are the first to show that you can use GPs to estimate the uncertainty of DNNs on a Jetson TX2 of a MAV.' While this is probably true, this is not a contribution worth stating, as it is too specific to be noteworthy.   



[1] Benchmarking the Neural Linear Model, Ober et al AABI

[2] Neural Linear Models with Gaussian Process Priors, Watson et al, AABI

[3] Incremental Local Gaussian Regression, Meier et. al NeurIPS 2014


Minor

'eigen-vector' should just be eigenvector 

typo on theory in appedix line 194
summary_of_recommendation: This paper has lots of interesting ideas, but perhaps too many as the current draft does not carefully motivate and validate the many design choices that go into the model. 

I would be happy to increase the score if these design choices become more coherent. 
issues: See above
review_title: null

Meta-review: This paper proposes an approach to novelty detection for deep neural networks using Gaussian Processes (GPs). More specifically, the paper uses mixture of experts (MoE)-GPs with a neural network-based kernel to estimate predictive uncertainty associated with a neural network’s output. The resulting approach is demonstrated on two robotic tasks corresponding to learning the inverse dynamics of a manipulator and object detection on a micro-aerial vehicle. 

Strengths:
+ The reviewers are generally in agreement that the paper addresses an important problem (uncertainty estimation for DNNs) that is relevant to robotics.
+ The reviewers generally agree that there is novelty in the technical approach (using MoE-GPs to perform uncertainty estimation).
+ The reviewers appreciated the experimental results on real-world robotics datasets.

Weaknesses:
- The primary concern raised by the reviewers related to the technical presentation of the approach. In particular, the theoretical derivations in Section 3 are not clearly described; certain technical choices are not clearly motivated or explained (see the comments from Reviewers mLeS and hvPu for detailed comments and suggestions).
- There are also concerns regarding the scalability of the approach. The onboard robotic experiments use relatively small datasets and the Jacobian used for uncertainty estimation is only associated with the last few layers of the DNN. Reviewers also raised concerns regarding the memory footprint of the approach.
- Reviewers suggested additional comparisons with baselines, including Deep Ensembles, and GPs. 

Suggestions:
I urge the authors to consider the reviewers’ detailed feedback in order to improve the paper. The primary suggestions for improvement include:
- Improving the technical exposition and justifying theoretical derivations and choices in Section 3.
- Addressing the concerns regarding scalability of the approach.
- Providing additional comparisons with baselines (and including discussions of some of the papers that the reviewers mentioned; particularly the ones mentioned by Reviewer Q7n6).

----- Post rebuttal -----

The reviewers have made significant improvements to the paper in the rebuttal and revision period. In particular, they have addressed the primary concerns raised by reviewers by (i) adding an additional strong baseline (Deep Ensembles), (ii) addressing questions regarding the memory footprint, and (iii) improving the presentation of the work. Given the strengths of the work in terms of importance of the problem, novelty, and real-world robotics evaluations (and also taking into account the improvements made to the paper), I believe this work represents a strong contribution. 
Decision: Accept (Poster)