Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Pros and the Key Idea**
This paper studies uncertainty quantification (UQ) in model-based learning for control, which is a timely and important research direction. The proposed method (EpiOpt) trains a neural network to predict the epistemic uncertainty directly. The training data for epistemic uncertainty prediction is artificially generated based on a simple nearest neighbor principle. The key ideas are: given the labeled training dataset $X_{tr},Y_{tr}$ (the data size $|X_{tr}|=N_{tr}$), this paper first randomly samples $X_{epi}$ around $X_{tr}$, where $|X_{epi}|=N_{epi}=k\times N_{tr}$. Then this paper labels $x\in X_{epi}$ by $1$ if the minimum distance from $x$ to $X_{tr}$ is far, and by $0$ if the distance is short. Finally, a neural network is trained for this binary classification task. 

Finally, this paper uses this idea in online control: the learned epistemic uncertainty is for adaptive data collection, and the aleatoric uncertainty is for control gain tunning. The advantage of this framework is that it is very simple, and doesn't need sampling or test-time dropout at the inference time.

**Cons and Suggestions**
(1) Many related work is missing especially for domain shift and adaptive control. As mentioned in this paper, the epistemic uncertainty is mainly from the data distribution shift, but there is no discussion about domain shift in this paper. The main idea of domain shift in ML is to *quantify the "distance'' between the source and target domains*, which is similar to the epistemic uncertainty prediction $\eta(\cdot)$ in this paper. People also considers domain shift in control and learning (http://proceedings.mlr.press/v120/liu20a.html, https://arxiv.org/abs/2006.13916 and many others). 
Also, adaptive control can handles epistemic uncertainty in an online manner as well. It would be great to discuss the difference between adaptive control. 
The ensemble method (e.g., https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf) is also an active method for UQ.

(2) The key method, *EpiOpt*, is a bit too simple and there are no theoretical justifications or insights, such that it is hard to convince me about the generalirity and robustness of this method. As I mentioned above, the key idea of *EpiOpt* is to train a "distance function" $\eta(\cdot)$ to quantify the distance between the source and target data. This idea has been both theoretically and empiracally studied in domain shift and transfer learning. A few questions pop up from this paper: 
* Why do you need to sample $X_{epi}$ around $X_{tr}$? Since the goal is to get $\eta(\cdot)$, why don't you consider some analytical solutions such as KDE (kernel density estimation) to derive $\eta(\cdot)$? In other words, you could just use $X_{tr}$ to estimate a density function for the source data, and then evaluating this density function to get $\eta(\cdot)$. I didn't see a clear reason to *train* a neural network to estimation this distance.  
* In equation (5), this paper labels $X_{epi}$ either $1$ or $0$. Why isn't it a continuous value from $0$ to $1$? For example, you can rank $d_j$ to get this continuous value.

(3) The title and abstract of this paper emphasize a lot on *epistemic and aleatoric uncertainty decomposition*. However, the key method *EpiOpt* is only about the epistemic uncertainty, and how to deal with the aleatoric uncertainty only appears in the experimental section in equation (9). I highly recommend the authors discuss about these two types in the method section (Section 3) and give a general framework. The current aleatoric uncertainty is more like a gain tunning method, which is not related to *uncertainty decomposition* .

(4) The experimental section is a bit vague. I highly recoomend the authors present the concrete learning and control problem first, e.g., Which part in the dynamics is learned? How to collect data? How to decompose the epistemic and aleatoric parts? A good example is https://ieeexplore.ieee.org/abstract/document/8794351, where the task is very similar. 

**Code-of-Ethics**
I see no ethic issues in this paper.
review_title: An interesting paper about uncertainty quantification in dynamics learning for control. The proposed method EpiOut is sample-free at the inference time and it outperms some baselines. But there are no theoretical justification or insights, and the binary epistemic output is too simplified.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Epistemic uncertainty is a useful measure to have from a learned model. This paper proposes to output epistemic uncertainty from NN by training with samples outside the training set. My concerns are listed below:

1. The paper samples epi points around the training set. How does this perform when training set is very sparse or very spread out. 
2. Changing the yellow line to something darker and also making the thick black line to be a lighter color would improve clarity of the results.
review_title: Epistemic uncertainty is output from NN using data sampled outside training data

final_score: rating: 7: Good paper, accept
review_content: Summary
--------
The authors consider the problem of efficient modeling of epistemic uncertainty, separated from aleatoric uncertainty,  for neural networks. They propose a novel methodology, involving automatically constructing a epistemic uncertainty support data set used to extend a given NN with an epistemic uncertainty output. The method is compared with previous, less efficient, approaches and is applied to the important problem of data-efficient online learning of a controller for real-time use with convincing results.

Strong points
------------- 
1. The paper is well written, and the important problem considered is clearly presented and motivated.
2. The proposed approach with *epi points*, although seemingly simplistic at first glance due to its heavy dependence on a useful distance metric, is demonstrated to be quite effective for learning control in euclidean space. 
3. The proposed method is demonstrated empirically in an online learning setting using real disturbance data, making the applicability convincing. 


Weak points
-------------
1. While I recognizing that it is problematic to compare *bounded* (e.g probability value) and *unbounded* (e.g. regression target variance) epistemic uncertainty predictions, I wonder if isn't warranted to consider making the comparison fairer w.r.t. the baselines? Since the experimental maximum value for either baseline method depends on the choice of window size (i.e. $x\in [-4 , 4]$), a fairer comparison might be to find (optimize) a scaling parameter $\alpha \in [0,1]$ of the computed normalized epistemic uncertainty prediction, which minimizes (6). Basically a worst-case situation w.r.t. the proposed method, with the baselines performing the best they could have given any upper bound (larger than observed within the window) on their epistemic uncertainty prediction. I still expect that the proposed method will perform comparatively well based on for example the functional forms in Figure 1.
2. It is stated in the paper that "*... (Dropout and BNN) have a larger total discount on the training set than on the test set.*" (3.4, last bullet point). Can this be due to different maximum values between training and test when normalizing their epistemic uncertainty prediction?
3. I regard the inclusion of the Vanilla GP model baseline as important. A major concern, however, is about the treatment of this baseline in the experiment methodology:
   1. The epistemic uncertainty estimate for the GP is calculated based on the predictive variance. Is it not more appropriate to use the predictive standard deviation instead, such that the uncertainty has the same unit as the target? (The same applies to Dropout and BNN too?)
   2. The data in *1D center* and *1D split* is very well suited for GP regression with an SE ARD kernel and should report a well calibrated epistemic regression uncertainty in input ranges where training data is absent. I find the miss-match between the left figure (showing the the GP mean prediction deviation) and the right figure (showing the epistemic uncertainty estimate) in Figure 1 odd. Implementing 3.1. might alleviate this issue?
   3. I find it odd that EpiOut outperform the GP model on *1D center* and *1D split*, taking into consideration the performance metric (6). Maybe it is a consequence of issue 3.1.?
   4. Table 4 and Table 5 in the appendix: It is odd that the GP test discount is smaller than the GP training discount on 2D Gaussian. It would indicate, as mentioned in the paper, that the epistemic uncertainty estimate is off for the GP model. Figure 3 (appendix) does indeed seem odd (or unclear) for the GP model, as does Figure 2 (appendix). Maybe it is a consequence of issue 3.1.? 
4. I am missing details on the online learning for the experiment, e.g: How long is each NN training and how is it related to the real-time aspects? Will the quadcopter fly far into state for which the current NN designates as high epistemic uncertainty before the NN is updated with new data points and  retrained such that the epistemic uncertainty is reduced? Or is the simulation paused and waiting for the NN training to complete on every new data point? If so this should be made clear in the paper.

Reason for score
----------------
It is a good paper on an important topic, with supportive empirical evaluation of the applicability of the proposed novel approach. My major concern is with certain aspects of the comparative evaluation and about the clarity of the online learning experiment.

Minor comments
--------------
1. I suggest that references to material in the appendix are made in the paper. Technicalities such as the online learning algorithm and relevant results would be made clearer.
2. Figure 4: Left and middle figures are a bit hard to interpret. It seems like the figures show multiple runs, which makes them even more difficult. They might benefit from a thinner marker/stroke, transparency and/or a 3D effect such as a color gradient for view depth etc. They could also be changed to plotted against time instead of against space.
3. GP regression with a squared exponential (SE) automatic relevance determination (ARD) kernel has, for a given training data (X,Y), a maximum epistemic regression uncertainty, given by the signal variance parameter $\sigma_f$ of the kernel: As $d(x^*, x)$ grows the exponential in the kernel decrease towards 1 rapidly, where $x$ is a training data point and $x^*$ is an arbitrary data point. The epistemic uncertainty is consequently bounded for the selected choice of kernel family.

3.5, bullet list, first bullet: 
  "\textit{EpiOut}model" -> "\textit{EpiOut} model"
  "\textit{EpiOut}is" -> "\textit{EpiOut} is"
  
4.1, paragraph 1: 
  "\textit{EpiOut}approach" -> "\textit{EpiOut} approach"
  "\textit{EpiOut}\eta(\cdot)" -> "\textit{EpiOut} \eta(\cdot)"

Edit: Markdown problem with nested lists

Edit: Upgraded rating due paper improvements

------------
Thank you for addressing my concerns.

Figure 1 is not updated (it looks the same w.r.t. the GP, but the change from variance to std should be noticeable?).
review_title: Good paper, but questions are raised about the experiments.

final_score: rating: 3: Clear rejection
review_content: **Update**

My assessment remains unchanged. See the comments for details.
The proposed approach is circular---it tries to sample points at a "sufficient distance" from the data, then tries to learn this
"sufficient distance" by predicting the points. This circular dependency should be broken somehow. The current paper set this distance arbitrarily by sampling from a Gaussian with covariance $I$.

**Summary of work**

They propose an epistemic uncertainty estimation method (though it
is not really estimating the uncertainty, but just whether a datapoint
existed in the dataset or not). The proposed method samples new
points $x_{epi}$ in the vicinity of the training data points $x_{tr}$
from a Gaussian distribution (they set the covariance to $I$).
Then it creates a dataset where the target
is 1 for $x_{epi}$ and 0 for $x_{tr}$ (with some tweaks on removing N
datapoints from $x_{epi}$, which were the closest to training data points,
but the main principle is roughly the same). Then they train a network to
predict this target, and this estimate is called $\eta$.

They compare their new model against Gaussian processes, Bayesian neural
networks, Dropout uncertainty estimation on some datasets. To evaluate,
they propose a new weighted mean squared error metric, where the mean squared
error is weighted by $1-\eta$ and the mean is taken w.r.t. the sum of the
$1-\eta$ terms. For the other methods, $\eta$ is computed by dividing the
uncertainty for the prediction with the largest uncertainty in the data.
Their method outperformed the other methods using this metric, except for
GPs on some datasets, but they said that GPs do not scale, and are not
suitable for real-time applications.

They use these networks in a quadcopter simulation control task. Their 
epistemic uncertainty estimation was used to decide whether to add a
data point to their data set (if the epistemic uncertainty is high) or
to discard it. The performance of their method including a disturbance
estimation model that uses their EpiOut approach improved the tracking
performance compared to not using any disturbance estimation model at all.

**Strengths and weaknesses**

Strengths:
- The exposition was good.

Weaknesses:
- It is not clear to me how to pick the covariance for the sampling distribution
of $x_{epi}$.
- It is not clear to me that there is a good justification for the method.
- It is not clear to me that using the largest uncertainty in the dataset
for computing the $\eta$ values for the other methods is appropriate. What if
the other method just predicts an infinite uncertainty somewhere? It seems
arbitrary to use the maximum, and it may be biasing the results to favor
the newly proposed approach. I have not seen this metric used anywhere else.
- The implementations use different frameworks, e.g. the GPs are based on
GPy using numpy (where there are other frameworks e.g. GPyTorch, Pyro,
or GPFlow that allow using GPUs), the BNN, dropout and EpiOut use tensorflow.
- The use in the quadcopter task was unclear to me. Why not just store all
of the data, or use each data point for a gradient update of the model, then
discard it?
- As far as I understood, in the experiment on the quadcopter the comparison
was between not using a disturbance model at all, and using a disturbance
model with EpiOut. It would be better to remove only the EpiOut component
to show that the uncertainty estimation was necessary, rather than just
any arbitrary disturbance estimation model.
- It seemed the number of data points gathered was around 150, which should
be easily handled with GPs, so the task does not seem to suggest that GPs
will not scale to it.
- The paper made up a new set of benchmarks for everything, and never tested
on an existing benchmark to prior works.

**Recommendation**

I recommend rejecting the paper with the main reason that I did not see
any principle behind the newly proposed method, and how the $x_{epi}$ dataset
could be constructed in a sensible way. Currently the covariance of the
Gaussians seemed to be set arbitrarily, but there should be a principled
way to set this covariance. Unless I have greatly misunderstood something,
I do not see how the paper could be modified for me to recommend it for
acceptance. I was also not convinced by the proposed quadcopter application for
the binary uncertainty estimation, and the other weaknesses listed above
also make me tend towards suggesting to reject.


**Questions**

Please clarify if there were misunderstandings.

**Additional feedback**

Below are some raw notes and additional thoughts from when I was reviewing
the paper.

"However, sampling the network during inference is a high computational
burden and is therefore not suitable in real-time critical control
tasks."
Really? With parallel computing it may not be that heavy. Do you have any
evidence for your claims? I see that in the appendix you have some
computational times, but it would have been good to point to this in the
main text.

"An extension to heteroscedastic GP regression is presented in
(Lazaro-Gredilla & Titsias, 2011), how-ever, it is a variational
approximation and further increases the computational complexity of
GPs,which is prohibitive when employing them for large data sets."
As far as I know, it is basically training two GPs, and the computational
complexity is the same, and differs only by a constant factor. Do you
have more evidence for your claim?

"preformed" -> performed

"considering a new data point takes $O(N_{tr}^3)$ computations"
No, I believe it takes $O(n^2)$ computation to add one new data point
using block matrix inversion.

Why was GPy used, and not some other framework with GPU support such as
GPyTorch, Pyro or GPFlow.

The Bayesian neural network was not clear, because it was a link to a
blogpost with many codes including numpy and tensorflow. (the code
cleared this up though)

"The measure in equation 6 relies on epistemic uncertainty prediction
in the interval[0, 1].  This is only ensured for the
proposed EpiOut approach and therefore the uncertainty measures provided
bythe GP, Dropout and BNN are scaled to the unit interval based on the
evaluation on all test points."
Is this a fair evaluation method? If one of the other methods gives an
infinite uncertainty on some point, this will negatively affect its entire
score, even though the uncertainty predictions for the other points may
be fine. The evaluation method does not appear sound to me, and no
references were provided for other works using such an evaluation method.

Being only confident close to the data trajectory may not be the best
approach. The model should be confident when the prediction is likely to
be accurate. For example if the model is smooth, then it should be
possible to interpolate reasonably accurately, and it should be OK
for the model to be confident as long as it has been able to learn
the structure sufficiently well.

review_title: Unclear whether the method is principled. Update: review remains unchanged.

Meta-review: This paper aims to do efficient epistemic uncertainty quantification for model-based learning for control. It does so by augmenting the dataset with synthetic data around the true data points, and trying to classify whether a point is close to the training set or not. I agree with many of the criticisms that R3 and R5 brought fourth. Namely, it's not clear why a kernel density estimate couldn't be used instead (runtime complexity is cited as the reason, but could be addressed through approximations, inducing points etc). It is not clear how to set the sampling distribution for X_epi. Also, since efficiency is a motivation for the work, I suggest that the authors look at and cite:

https://arxiv.org/abs/2002.06715 

I think at the moment the paper is not ready for publication, but the idea is interesting. Aside from comparing with the work above, what would improve this paper is an automatic way to select the distribution, or at least the covariance, of X_epi.  
Decision: Reject