Real-time Uncertainty Decomposition for Online Learning Control
Authors: Jonas Umlauft
Armin Lederer
Thomas Beckers
Sandra Hirche
Venue:  (2021)
Keywords: uncertainty decomposition
epistemic uncertainty
online learning
real-time control

Abstract: Safety-critical decisions based on machine learning models require a clear understanding of the involved uncertainties to avoid hazardous or risky situations. While aleatoric uncertainty can be explicitly modeled given a parametric description, epistemic uncertainty rather describes the presents or absence of training data. This paper proposes a novel generic method for modeling 
epistemic uncertainty and shows its advantages over existing approaches for neural networks on various data sets. It can be directly combined with aleatoric uncertainty estimates and 
allows for prediction in real-time as the inference is sample-free. We exploit this property in a model-based quadcopter control setting and demonstrate how the controller benefits from a differentiation between aleatoric and epistemic uncertainty in online learning of thermal disturbances.

Under review as a conference paper at ICLR 2021
REAL-TIME UNCERTAINTY DECOMPOSITION FOR ONLINE LEARNING CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
Safety-critical decisions based on machine learning models require a clear un- derstanding of the involved uncertainties to avoid hazardous or risky situations. While aleatoric uncertainty can be explicitly modeled given a parametric descrip- tion, epistemic uncertainty rather describes the presence or absence of training data. This paper proposes a novel generic method for modeling epistemic un- certainty and shows its advantages over existing approaches for neural networks various data sets. It can be directly combined with aleatoric uncertainty esti- mates and allows for prediction real-time as the inference is sample-free We exploit this property in model-based quadcopter control setting and demonstrate how the controller benefits from a differentiation between aleatoric and epistemic uncertainty in online learning of thermal disturbances.
INTRODUCTION
With improved sensor quality and more powerful computational resources, data-driven models increasingly applied in safety-critical domains such as autonomous driving or human-robot interac- tion (Grigorescu et al., 2020). However, measurements usually suffer from noise and the available data is often scarce compared to all possible states of a complex environment. This requires con- trollers, which rely on supervised learning techniques, to properly react ignorance and imprecision in the model to avoid dangerous situations. In order to allow an implementation of risk-averse (for exploitation and safety improvements) or risk-secking (for exploration) behavior, the model must clearly disaggregate the information in the data into more than just the "best estimate" and differ- entiate between different sources of uncertainty. Besides the point estimate of model, one can distinguish aleatoric (uncertainty in the data) and epistemic (uncertainty in the model) uncertainty. The former is irreducible as it is inherent to the stochastic process the data is recorded from, while the latter origins from limited expressive power of the model or scarce training samples (Der Ki- ureghian & Ditlevsen 2009).
Gaussian processes (GPs) inherently provide measure for its fidelity with the posterior standard deviation prediction (Rasmussen & Williams 2006). It also allows to differentiate alcatoric uncer- tainty (typically considered observation noise) and epistemic uncertainty (modeled by the kernel). However, the former allows only homoscedastic (constant) estimates, while real-world applications typically require heteroscedastic uncertainty models. An extension to heteroscedastic GP regression is presented in (Lazaro-Gredile & Titsias 2011), however, it is a variational approximation and further increases the computational complexity and GPs generally suffer from poor scaling to large datasets Stunonero-Candela & Rasmussen 2005).
In deep learning. the modeling of uncertainties also gained increasing interest over the past years (Kendall & Gal 2017). Heteroscedastic alcatoric uncertainty can be captured well, if the output of the stochastic process can directly be observed and its parametric distribution is known. However, for more general cases, approximation techniques such as variational inference or sam- pling is required (Bishop) 2006). For epistemic uncertainty estimation with neural networks (NN), the key idea for most approaches can be summarized as follows. Randomness is introduced to the neural network through sampling during training and inference. While the training robustifies the network against the injected noise the training locations, it allows the noise pass to the output at input locations where no training data is available. For inference, multiple predictions the net- work are sampled for the same inputs, allowing to compute statistical measure for the uncertainty

---

Under review as a conference paper at ICLR 2021
at the output (Depeweg al., 2018: Depeweg 2019). However, sampling the network during infer- ence is a high computational burden. and is therefore not suitable in real-time critical control tasks. An ensemble based approach by (Lakshminarayaniss network, but does not differentiate between aleatoric and epistemic uncertainty explicitly.
al. 2017) works with far less instances of
Despite those drawbacks in the uncertainty representation of data-driven models, the control com- munity started to incorporate them increasingly in the decision making for various applications. For example [Faneer al. (2016) uses an epistemic uncertainty measure to dynamically assign leader order follower roles for cooperative robotic manipulation. The work by [Berkenkamp et al. (2016) ensures safe exploration of an unknown task space based on GP error bounds and a gain scheduling approach for computed torque control is presented in Beckers et al. (2019). The work by Liu et al. (2020) considers the epistemic uncertainty as an estimate the distance between source and target domains (known domain shift) to design robust controller. In [mlluftt & Hirchel (2020) and Chowdhary al. (2015) an online learning control approach for GPs models is considered, which approach the dual control problem (Wittenmark 1995) model-based adaptive control problem. The work by Yesildirak & Lewis (1995) uses neural network for adaptive control a continuous time fashion, which relies on triggere (periodic) update of the model rather than event- based adaptation as we propose in this work. More general, risk averse control strategies have been presented by Umlauft et al. (2018); Medina al. (2013); [Todorov & Li (2005). However, all of these approaches only consider the model fidelity in general and do not differentiate between aleatoric and epistemic uncertainty.
Therefore, the main contributions of this paper are the following. We propose deep learning frame- work with a real-time capable epistemic uncertainty prediction. The resulting online learning model is employed by a controller, which shows a distinct reaction to epistemic and aleatoric uncertainty. We evaluate the proposed methods on synthetic and real-world benchmark data sets, and simulate a quadcopter controller, which learns online the disturbances injected by thermals.
2 PROBLEM FORMULATION
Consider the discrete-time dynamical system with control u E U C Rds and state x E X C Rd
- g( uk) yk:
where g: X
x U + X is known, while y is a i.i.d. random vector sampled in every time step from
yv (af())),
where D--) denotes a known distribution over real vectors y Y C Rd and depends on the param- eters p E IP C Rd, These state-dependent parameters arise from an unknown mapping f: + IP. We denote the unknown component yı: of the dynamical system generally disturbance but it could also be the unmodeled part of the dynamics, such as friction or serve as black-box model for the dy- namies if no analytic description is available (g(s))
0). We assume measurements can be taken to obtain the data set Dr {( ! with inputs X,r = { and outputs Wr { such that a model f()) of f-) can be learned. Nú E N denotes the current number of training data points and is initially zero, i.e., the training set is empty.
The task is to choose a control input uk such that the system 1 follows a given reference des. Furthermore, the controller can take new measurements of y improve its model over time. We consider each measurement of y to be costly and therefore new training points should only be collected when necessary. Applications, where data collection is costly can be found in distributed systems, where multiple sensors share the same scarce communication channel, or in autonomous systems with limited data storage capacity.
The need for high data efficiency requires models, which judge upon their own fidelity in real-time to identify valuable measurements. As existing approaches for modeling epistemic uncertainty in deep learning suffer from a high computational complexity we first focus on developing novel method for epistemic uncertainty predictions before proposing an online learning control strategy which makes use of neural network model decomposing its uncertainties.
'Boldcappial symbols generally denote vectons/matrices, (())) general parametric/the uni- form/ Baee/GGaussinnnRen Bamm/Gaussiaan/Beoallii /erroulli distribution, respectively.
2

---

Under review as a conference paper at ICLR 2021
3 EPISTEMIC UNCERTAINTY ESTIMATION
3.1 RELATED WORK
Learning an epistemic uncertainty estimator is not straight forward as it measures the absence of training data. Most prominently Gaussian processes with stationary kernels offer such measure implicitly with their posterior variance prediction. However, GPs are known to scale poorly for large data sets: While regression and uncertainty predictions can be performed with ((N)) and O 9), respectively, considering a new data point takes O(VV') 2) computations (also without hyperparameter optimization, ONN,,) 2) for rank-1 update methods). While various methods have been proposed to make GP computationally more efficient, including sparse GPs (Quinonrr--andelll & Rasmussen 2003), distributed GPs (Deisenroth & Ng, 2013) and local GPs (Nguyen-Tuong et al.] [2099]]b) these approximations typically focus only on the precision of the point estimate and distort the uncertainty prediction. For estimating the "distance" to the training set, kernel density estimation (KDE) can also be used (Rosenblati [1966). however, the non-parametric nature implies that the inference time grows linearly with the number of considered data points, which we aim to avoid.
More recently, several different approaches for epistemic uncertainty estimates using deep learning frameworks have been proposed. Popular approaches rely on Bayesian approximations Depeweg al. (0166) or permanent dropouts (not only during training to avoid overfitting) (Gall 2016 Gal & Ghahramani 2016). Furthermore, latent inputs can also be used to achieve a decompositon into aleatoric and epistemic uncertainty as presented in (Depeweg al. 2017). However, in particular for Bayesian NNs, these approaches become computationally challenging. Firstly, they have larger number of parameters to tune than their deterministic counterparts and rely on variational inference methods (Koon et al. 2020)). Secondly, the prediction requires to sample the entire network before the statistics of the output can be computed. For the application in real-time critical control problems (e.g., robotics with sampling rate of 1 kHz), these computational burdens prohibit an employment of these techniques. A sampling-free estimation methods is proposed by PPostels et al. (2019), but suffers from quadratic space complexity in the number of weights in the network and relies on first- order Tay lor approximations in the propagation of the uncertainties, which might become inaccurate depending on the non-linearity of the activation functions
3.2 EpiOut- EXPLICITLY LEARNING EPISTEMIC UNCERTAINTY
In order to allow the estimation of epistemic uncertainty in real-time, we introduce the idea of ex- plicitly modeling it with a separate output of neural network, calling it EpiOur. Since the epistemic uncertainty expresses the absence of data, the original data set Dr does not contain data for training EpiOut. Therefore, we generate an epistemic uncertainty data set, with inputs Xepi = { and
No
outputs Yep = {ūr Naj concatenated in api = {(ē, 3,) ) Nai Napi E N.
Different variations for sampling the set Xepi can be chosen depending on the desired scalability properties. A naive approach is sample the entire input space uniformly, which suffers from the curse of dimensionality. Alternatively, we propose to sample around existing training points from a normal distribution
N Xepi -,, ~ N( L,..,, Nopi/Nu).
1=1
where we implicitly assume that Nepi is chosen such that 5 = Neps NN, is positive integer. Sup- posing that standardization of the input space to unity is performed based on X,, - can be chosen if no further knowledge on f(.) is available. Otherwise, scaling can be interpreted simi- larly to the lengthscale of GP as measure for how far away from training point the prediction is reliable: Larger will lead to further spread of Xepi and therefore low epistemic uncertainty in the neighborhood of the training data, which would be meaningful if the true function is known to have a low Lipschitz constant, and vice versa.
We propose to set 8 to multiple of 2dg + which corresponds the intuition to pad each training point in both directions of each dimension with a epi point ă. The reasoning behind the additional +1 point will become clear in the following. To define the set W Vepi we first compute the minimal
3

---

Under review as a conference paper at ICLR 2021
distance (according any distance metric d: X X - Ro,+) to the training data for each epi point
dj min d( m),, =
Nepi
keeping in mind that the closest training data point is not necessarily the one used to generate the sample. Let dx, be the Nu-th smallest element of all dj, we generate Y epi and update Xepi as follows
{
1, 0, arg mix x)
if d dN. if dj dx.
80 =
(5)
arg
Thus, the N₂ points in Xeps with the least distance to training point are replaced by the correspond- ing point in Xg- Now the choice of 2d + 1 epi points becomes clear as one of them will be turned into y = 0 corresponding to "low epistemic uncertainty", while 2dl points further away from the training point
indicate the opposite.
Given the data set Depi. the neural network is now equipped with one additional output, i.e., the
parameter layer is dp + dimensional with output [f(-) nn-)]"" The new output 17(-) is terminated with a neuron using a sigmoidal activation function, such that n7 X [0. This beneficial because it allows immediately to judge, whether the predicted uncertainty is high ( 1) or low (> 0) without any reference evaluation (see comparison to alteranative methods below).
Independently of the loss function for the original network, the augmented output, also considered as epistemic output is trained using a binary cross-entropy loss, which the natural choice for binary outputs. It quantifies the uncertainty in the prediction of the other outputs based on the distance to the training data measured by d). For the sake of focus, we will be using the Euclidean distance, however the method can be easily extended to other metrics and we leave it to future work to investigate alternatives.
3.3 COMPUTATIONAL COMPLEXITY
The analysis of the computational complexity shows that (3) is a ((Nn)) op- eration, whereas (4p is for a trivial implementation a O Nen) = operation. However, implementation based on kd-tree (Cormen log( (Nnd)))) time. Finding the Nr smallest distances from all Nepi
al. 2009) allows execution
in OpN = points in (5) can obtained in O(N + (Nepi - -Nr) log( (N.) = + (rr log(())) time. Hence, the overall complexity results in Od log(d=~Nu)), and it is straight forward to derive an overall space complexity of ) 3) for storing the set Xepi- The following should be con- sidered when comparing to classical deep learning frameworks which generally can be trained in linear time.
The training of neural network with fixed number of weights requires (
When used streaming data (as for online learning control), the set Depi can be constructed iteratively, reducing the complexity to O(log( (Nw))
The most time critical computation (4) can efficiently be parallelized on a GPU.
The method is designed for applications where measuring data considered costly and therefore sparse data can be expected.
3.4 EVALUATION
For evaluation we compare the following models. The implementation is available in the supple- mentary material.
vanilla GP model with squared exponential automatic relevance determination kernel based on the GPy implementation
BNN: Bayesian Neural Network with 2 fully connected hidden layers each with 50 hidden
units and normal distributions over their weights based on this implementation
httesss//hheffiiemmm...hhhhh /ote https::/maatthewmcaeeeee
attaaaa-uuiiiiiiiieeee wwwck-innro-tott. bayes neural -etwwrkss
4

---

Under review as a conference paper at ICLR 2021
Dropout: A neural network with 2 fully connected permanent layers each with 50 hidden units with dropout probability p = 0.05.
EpiOut: The proposed model with fully connected layers (50 neurons) and I = I, 6 2.
For the evaluation we utilize a weighted mean square error measures defined as follows
(y6 f( (1-
(6)
i... if the model decides that it is uncertain about the prediction a test point, the squared error for this prediction is discounted (weighted less). However, the model can only achieve low p p is also certain at some test points, because the denominator, goes to zero for many uncertain predictions. In consequence, p is only defined if v7()) < 1 holds for at least one test point. Furthermore, the total discount, defined as Σi=1 na::) can additionally be utilized for plausibility check of the epistemic uncertainty predictions since it should generally be larger on the test than on the training data set.
The measure in (6) relies on epistemic uncertainty prediction in the interval [0, 1]. This is only ensured for the proposed EpiOut approach and therefore the uncertainty measures, more specifically the predicted standard deviation, provided by the GP, Dropout and BNN are scaled the unit interval based on the evaluation on all test and training points.
The following data sets are utilized for evaluation.
ID Center: The nominal function is f(x) = sin (æT)), with training points Xr
{zz ~
1()1,1))i= and Nie 961 test points are placed on grid [-4, 4].
1D Split:
Same as ID Center, but Xr = (x; ~ ~ -1) U f2z (11, ((1,2))]amm
2D Gaussian: The nominal function (dz
= 2, dp = 1) is f(x)
sin(5.r1)
x3 with training points X
{ 0. ~ N (1
0.02
0.1
0
500
321
0
(1)
0.02
0 o.
1000
23 ~
and N 961 test points are uniformly placed
5=501
on a grid 2|,,
2D Square: Same as 2D Gaussian, but with with Nr 80 training points placed uniformly along the boundary of the square 1J
PMSM temperature is 2Hz recording (d, = 8, dg = of the temperature from perma- nent magnet synchronous motor To allow comparison with the GP within reasonable computational limits, N₂ = 5000 and Ni = 1000 points were randomly extracted from a total of 10yh samples.
Sarcos is a data set for learning the inverse dynamics of a seven degrees-of ofgrees-ooffreeddm freedom SAR- COS anthropomorphic robot arm d₂ 21, dp
1 8 N₂ = 10000 and Nic- 2000 points were randomly extracted from total of x 5 x 10 samples.
3.5 RESULTS & DISCUSSION
The numerical results are presented in Table | and an illustration for the data set ID Split for all models is shown in Fig. I Besides showing empirically an advantage over existing approaches we want to point out the following benefits.
The EpiOut model predicts the uncertainty measure sample free manner. This is crucial in data-cfficient online learning scenarios, where the epistemic uncertainty is used to evalu- ate the usefulness of an incoming data point to decide upon its rejection. Hence, it is called more frequently than the online training function and must be computationally efficient. The prediction time for EpiOut is typically an order of magnitude faster than Dropout and BNN.
htttps://gittb..cmmm/// htte:///ttta..coorarrarrrpottttt http:///ww..gossiirrrro.ooooooo
5

---

Under review as a conference paper at ICLR 2021
Table I: Weighted mean squared error p defined in (6) for the considered models on different data sets. The GP model is grayed out since it does not scale towards larger data sets. To prevent the scaling of the epistemic uncertainty to the unit interval from deteriorating the performance of GP model, BNN and Dropout we minimize for the evaluation over factor [0, 11, od) to
ID Center
ID Split
2D Square 2D Gaussian
PMSM temperature
Sarcos
GPmodel
0.1660
0.1207
0.0655
0.0342
0.0007
4,8150
BNN
1.4177
0.5889
1.1953
0.8830
0.0101
21,9015
Dropout EpiOut
0.7904
0.6181
1.0681
0.4468
0.1490
21.3842
0.0480
0.1125
0.1893
0.0056
0.0024
14.2537
13g ((r
Dropout
EpiOut
M
epistemic unc.
0.5
2
20 Figure I: The point estimate (mean prediction) for the different models along with the training data ID split and the underlying function (xx sin(Tx) are shown the left. The right plot shows the epistemic uncertainty estimate, where BNN and Dropout clearly miss to predict higher uncertainty between the data clusters.
A single evaluation of 17(-) sufficient for conclusion whether the uncertainty is high or low, since it is bounded to the interval [0, 1]. whereas alternative approaches based BNN and Dropoul provide return value [0, 2oj, which can be difficult to interpret without a maxi- mum value reference.
For more extensive results and exact computation times, we refer to the supplementary material.
4 ONLINE LEARNING CONTROL USING UNCERTAINTY DECOMPOSITION
4.1 EVALUATION A QUADCOPTER CONTROL TASK
As application of proposed approach, consider the task to control quadcopter which explores a given terrain with unknown thermal We assume that the quadcopter dynamics ::). in [ is known (compare (Shi al. 2019)). The thermals act the quadcopter the acceleration in 29-direction We model the disturbance as normal distribution (()()), o (xx)),
disturbance
leading to [n, o²² and the NN models the state dependency of these parameters f X - P. Initially, there is training data available and the desired trajectory des three rounds at constant height 2 = on a square in the x9 -plane with edge length 0.1 followed by three rounds with edge length 0.05.
The online learning control approach for achieving this task with a high tracking precision and data- efficiency is presented in the following.
4.2 LEARNING CONTROL DESIGN
The EpiOut approach sufficiently fast to implemented in a mobile real-time control loop and therefore serves as trigger when new data points must be measured. By sampling from Bernoulli
Thhe data of the thermals is taken from publicly available paragliding data ht ps: / /ttrrral kk ch "Superscripts 7 distinguish the quadcopter position x² "y"","" from the general inputs and outputs y.

---

Under review as a conference paper at ICLR 2021
distribution, whose parameter corresponds to n7(), measurements (z), y:) added to the training according to
Dir t Dr U { ( 3i) if =
where i B(a), a 772)))
(7)
if
0
This ensures a high accuracy of the disturbance model f()) as training data added when necessary. It implements a stochastic event-triggered adaptive control approach, which highly data-efficient and reduces the number of costly measurements. In particular for mobile platforms this cost comes terms of reduced battery life (processing data requires power for computations), shorter opera- tion time (local data storage quickly fills up if operating near kHz sampling rates) or increased communication efforts (in case data not process or stored onboard).
The system is inherently random due to the stochastic nature of the disturbance f()). Therefore, we combine feedback and feedforward control law
K (x dess) Wir:
where æz, des is the (desired) state of the quadcopter and uh is feedforward control term deter- mined based on the known model g(.,)) (the gravitational force on the quadcopter) and the learned disturbance model f f-).. more specifically the mean of the predicted disturbance p(-). The choice of the feedback gain K, which compensates for imprecision the model and the stochasticity of the disturbance, is difficult trade-off because high control gains lead to high tracking performance, but also consume more energy and reinforce measurement noise, which can lead to instability. There- fore, it is generally advisable let the feedforward term uh take over most of the control effort and keep the feedback term small when the model is reliable. Therefore, we increase the gains only if the aleatoric uncertainty inferred by our model (-)) is high. Since the disturbance acts only in 2 direction, only the component of the gain is affected i.e.,
k2 = kg fo(x))
where B E 18,++ is the sensitivity and k E I.+, defines the minimum control gain. This gain scheduling allows robustify the closed-loop against process noise and can even guarantee stabil- ity /Beckers et al.) (2019), while at the same time we can keep the energy consumption low. While previous works by Fanger al. (2016); Beckers al. (2019) take general uncertainty measure, we tune the feedback gains only based on the irreducible (aleatoric) uncertainty, while we combat the epistemic uncertainty with an increased data collection rate
A summary of the control strategy is given in Algorithm I and visualization of the neural network Fig. 2 The outputs (red nodes) are iteratively trained with the indicated loss functions and respec- tive data sets D, Depi for two epochs with learning rate of 0.01 with each incoming data point. The current implementation pauses during the training the disturbance model before the control loop continuous.
inpul layer
hiddlen relu layer(s)
ourput layer
observation
Algorithm 1 Online learning controller
layer meam disturbance
1:
initialize disturbance model
2: while control task is not completed do
3: get state
4: evaluate disturbance model
fit
5: update control gains (9)
y
N likelihood
6:
if measurement required 7 then
7:
measure disturbance y/ update data set DD + Du U (z), vi)
8:
alentoric uncertainty
9:
resample Dop
10:
retrain disturbance model
y
cross entropy
end if
12:
apply control
epistemic uncertainty
13:
i it
14:
end while
Figure 2: Schematic drawing of the NN to decom- pos the uncertaintics for the quadcopter controller.

---

Under review as a conference paper at ICLR 2021
disturbance model without EpiOut
(desired) position
0.1
des
des
0.05
10
14
18
disturbance model including EpiOut
(desired) position
0.1
0.05
10
12
16
time
epistemic uncertainty
X
n
400
0.1
Nr
200
without EpiOut
with EpiOut
10
14
18
0.1
time
r
Figure 3: The EpiOut prediction of the disturbance model allows smart selection of the measure- ments taken. While random selection constantly takes data points, out proposed approach takes more measurements when entering new areas (Os to 3s and 9s 12s) and less when repeating the pattern (bottom). This results in overall improved tracking performance of root mean squared error in z-direction of 0.00733 (middle) vs 0.00773 (top) while taking less data. The epistemic uncertainty estimate is low near training data and high in unobserved regions.
The tracking performance of the proposed quadcopter controller, the data collections rate and the epistemic uncertainty model is illustrated in Fig. The implementation and further results are provided in the supplementary material.
CONCLUSION
This paper presents a novel deep learning structure for decomposing epistemic and aleatoric un- certainty and proposes an advanced control framework making distinct use these uncertainty measures. As the prediction by the model are obtained sample-free, it allows for real-time critical online learning and outperforms existing methods on the proposed uncertainty-weighee precision measure. The proposed online learning control algorithm is inherently data-efficient by adding only required points to the data set.
For future work will consider alternative functions for sorting the epi points to encode prior knowl- edge, such as as periodicity (similar to a kernel GP) and investigate the effect continuous valued ÿ.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Thomas Beckers, Dana Kulić, and Sandra Hirche. Stable Gaussian process based tracking control of Euler-Lagrange systems. Automatica, 23(103))330-3393 -979 2019. doi: a..101//..uttmmaiiaa.ooo 01.023.
Felix Berkenkamp, Riccardo Moriconi, Angela Schoellig, and Andreas Krause. Safe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes. arXiv preprint ar3iv:1603.04915, 2016.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Girish Chowdhary, Hassan A. Kingravi, Jonathan. P. How, and Patricio A. Vela. Bayesian nonpara- metric adaptive control using Gaussian processes. IEEE Transactions Neural Networks and Learning Systems, 26(3):537-550, March 2015. ISSN 2162-237X. doi: 10.1109/TNNLS.0 2014
2319052.
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2009.
Marc Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In International Conference on Machine Learning (ICML), pp. 1481-1490, 2015.
Stefan Depeweg. Modeling Epistemic and Aleatoric Uncertainty with Bayesian Neural Networks and Latent Variables. PhD thesis, Technische Universität München, 2019.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi- Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with Bayesian neural networks. arXiv preprint arXiv: I00::0605.122 2016.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Boshi-Velez, Steffen Udluft. Un- certainty decomposition in Bayesian neural networks with latent variables. arXiv preprint arXiv:1706.08495, 2017.
Stefan Depeweg, Mose-Miguel Hernandez-Lobato. Finale Doshi Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference Machine Learning, pp. 1184-1193, PMLR, 2018.
Armen Der Kiureghian and Ove Ditlevsen. Alcatory or epistemic? does it matter? Structural safety, 31 2009.
Yunis Fanger, Jonas Umlauft, and Sandra Hirche. Gaussian processes for dynamic movement prim- itives with application in knowledge- based cooperation. In International Conference on Intelli- gent Robots and Systems (IROS), pp. 3913-3919, IEEE, October 2016. doi: 10. 10.1109//O00 7759576.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, September 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, June 2016. PMLR. URL
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3),62-336, 2020. doi: 1001/////0.2..9.. URL http://eeennea.....rpeeee///////
m prtps//48//gaaaaa/coe.p hmt.
wi wwwey. www.wwwwwo//o.////a///eeaee
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 5574-5584. Curran Associates, Inc., 2017.

---

Under review as a conference paper at ICLR 2021
Yongchan Kwon, foong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik. Uncertainty quan- tification using bayesian neural networks in classification: Application to biomedical image seg- mentation. Computational Statistics and Data Analysis, 142: 2020. ISSN 0167-9473.
doi: 2.1016/..csda.2019... 106816 URL http::tttp///e sci encedincdiiic rect. comm sc ltmm:sciince/////
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS). pp. 6402-6413, 2017.
Miguel Lazaro-Gredilla and Michalis Titsias Variational heteroscedastic gaussian process gression. In International Conference on Machine Learning (ICML), pp. 841-848, New York, NY, USA, June 2011. ACM. ISBN 978-1-4503-061995. URL https: http::::////mmmcc///..11 cc/2011/
papers/456 icml
Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Conference on Learning for Dynamics and Control (L4DC), volume 120 of Proceedings of Machine Learning Research, pp. 608-619, The Cloud, June 2020. PMLR. URL http:/ http:://proceedinggg press/ v120/1 1220a. html
Jose R. Medina, Dominik Sieber, and Sandra Hirche. Risk-sensitive interaction control in uncertain manipulation tasks. In International Conference on Robotics and Automation (ICRA), pp. 502- 507. IEEE, May 2013. doi: 10. 100//CRA.2013.666666666
Duy Nguyen-Tuone Jan R. Peters, and Matthias Seeger. Local Gaussian process regression for real time online model learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1193-1200. Curran Associates, Inc., 2009a.
Duy Nguyen-Tuong, Matthias Seeger, and Jan Peters. Model learning with local Gaussian process regression. Advanced Robotics, 23(15):2015-20344 2009b.
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2931 2931-2940, 2019.
Joaquin Cainonero-Candela and Carl E. Rasmussen. A unifying view of sparse approximate Gaus- sian process regression. The Journal of Machine Learning Research, 6:1939-1955 2005.
Carl E. Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, USA, January 2006.
Murray Rosenblatt. Remarks some nonparametric estimates of density function. Ann. Math. Statist., 27(3):832-837, 09 1956. doi: 10. 10.1214//0mm/1177772899999 httts:s/:/// /rr// 10. 2et/2////mmm/11772555555
G. Shi, X. Shi, M. O'Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S. Chung- Neural lander: Stable drone landing control using learned dynamics. In International Conference on Robotics and Automation (ICRA), pp. 9784-9790, 2019. doi: 10. 10.100//CRA.200 89949179...11...
Emanuel Todorov and Weiwei Li. A generalized iterative LQG method for control of constrained nonlinear stochastic systems. In American Control Conference (ACC), pp- 300-306. IEEE, 2005.
feedback
Jonas Umlauft and Sandra Hirche. Feedback linearization based Gaussian processes with event- triggered online learning. IEEE Transactions on Automatic Control (TAC), pp. 1-16, 2020. ISSN 1558-2523
doi: 10.1109/AA..2019.99938 Sccumentt/9932275
URL htttp:://ieeexp::ppp ore. feee. err/
Jonas Umlauft, Lukas Pöhler, and Sandra Hirche. An uncertainty- based control Lyapunov approach for control-affine systems modeled by Gaussian process. IEEE Control Systems Letters, 2(3):
483-488. July 2018 ISSN 2475- 1456. doi: 10. 10.100004 URL https:
7/1 7/ieeexplore. leee. org /eee.oor//cccmmmm///3333332
10

---

Under review as a conference paper at ICLR 2021
Bjöm Wittenmark.
Adaptive dual control methods: An overview. IFAC Proceed- ings Volumes, 28 28 72, June 1995. ISSN 1474-6670. doi: htpp://doi.og// 10.
com/ ccm/science/ / 7453277 5th IFAC Symposium on Adaptive Systems in Con- trol and Signal Processing 1995, Budapest, Hungary, 14-16 June, 1995.
(1161514774-66770114455577777777
URL http://ver http://wwwwwwee.ee.eeeeeo
Aydin Yesildirak and Frank Lewis. Feedback linearization using neural networks. Automatica, 31(11)115991164 1995.


Reviews:
[{"review_id": "Z7JHMAPSIa2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\n\nMy assessment remains unchanged. See the comments for details.\nThe proposed approach is circular---it tries to sample points at a \"sufficient distance\" from the data, then tries to learn this\n\"sufficient distance\" by predicting the points. This circular dependency should be broken somehow. The current paper set this distance arbitrarily by sampling from a Gaussian with covariance $I$.\n\n**Summary of work**\n\nThey propose an epistemic uncertainty estimation method (though it\nis not really estimating the uncertainty, but just whether a datapoint\nexisted in the dataset or not). The proposed method samples new\npoints $x_{epi}$ in the vicinity of the training data points $x_{tr}$\nfrom a Gaussian distribution (they set the covariance to $I$).\nThen it creates a dataset where the target\nis 1 for $x_{epi}$ and 0 for $x_{tr}$ (with some tweaks on removing N\ndatapoints from $x_{epi}$, which were the closest to training data points,\nbut the main principle is roughly the same). Then they train a network to\npredict this target, and this estimate is called $\\eta$.\n\nThey compare their new model against Gaussian processes, Bayesian neural\nnetworks, Dropout uncertainty estimation on some datasets. To evaluate,\nthey propose a new weighted mean squared error metric, where the mean squared\nerror is weighted by $1-\\eta$ and the mean is taken w.r.t. the sum of the\n$1-\\eta$ terms. For the other methods, $\\eta$ is computed by dividing the\nuncertainty for the prediction with the largest uncertainty in the data.\nTheir method outperformed the other methods using this metric, except for\nGPs on some datasets, but they said that GPs do not scale, and are not\nsuitable for real-time applications.\n\nThey use these networks in a quadcopter simulation control task. Their \nepistemic uncertainty estimation was used to decide whether to add a\ndata point to their data set (if the epistemic uncertainty is high) or\nto discard it. The performance of their method including a disturbance\nestimation model that uses their EpiOut approach improved the tracking\nperformance compared to not using any disturbance estimation model at all.\n\n**Strengths and weaknesses**\n\nStrengths:\n- The exposition was good.\n\nWeaknesses:\n- It is not clear to me how to pick the covariance for the sampling distribution\nof $x_{epi}$.\n- It is not clear to me that there is a good justification for the method.\n- It is not clear to me that using the largest uncertainty in the dataset\nfor computing the $\\eta$ values for the other methods is appropriate. What if\nthe other method just predicts an infinite uncertainty somewhere? It seems\narbitrary to use the maximum, and it may be biasing the results to favor\nthe newly proposed approach. I have not seen this metric used anywhere else.\n- The implementations use different frameworks, e.g. the GPs are based on\nGPy using numpy (where there are other frameworks e.g. GPyTorch, Pyro,\nor GPFlow that allow using GPUs), the BNN, dropout and EpiOut use tensorflow.\n- The use in the quadcopter task was unclear to me. Why not just store all\nof the data, or use each data point for a gradient update of the model, then\ndiscard it?\n- As far as I understood, in the experiment on the quadcopter the comparison\nwas between not using a disturbance model at all, and using a disturbance\nmodel with EpiOut. It would be better to remove only the EpiOut component\nto show that the uncertainty estimation was necessary, rather than just\nany arbitrary disturbance estimation model.\n- It seemed the number of data points gathered was around 150, which should\nbe easily handled with GPs, so the task does not seem to suggest that GPs\nwill not scale to it.\n- The paper made up a new set of benchmarks for everything, and never tested\non an existing benchmark to prior works.\n\n**Recommendation**\n\nI recommend rejecting the paper with the main reason that I did not see\nany principle behind the newly proposed method, and how the $x_{epi}$ dataset\ncould be constructed in a sensible way. Currently the covariance of the\nGaussians seemed to be set arbitrarily, but there should be a principled\nway to set this covariance. Unless I have greatly misunderstood something,\nI do not see how the paper could be modified for me to recommend it for\nacceptance. I was also not convinced by the proposed quadcopter application for\nthe binary uncertainty estimation, and the other weaknesses listed above\nalso make me tend towards suggesting to reject.\n\n\n**Questions**\n\nPlease clarify if there were misunderstandings.\n\n**Additional feedback**\n\nBelow are some raw notes and additional thoughts from when I was reviewing\nthe paper.\n\n\"However, sampling the network during inference is a high computational\nburden and is therefore not suitable in real-time critical control\ntasks.\"\nReally? With parallel computing it may not be that heavy. Do you have any\nevidence for your claims? I see that in the appendix you have some\ncomputational times, but it would have been good to point to this in the\nmain text.\n\n\"An extension to heteroscedastic GP regression is presented in\n(Lazaro-Gredilla & Titsias, 2011), how-ever, it is a variational\napproximation and further increases the computational complexity of\nGPs,which is prohibitive when employing them for large data sets.\"\nAs far as I know, it is basically training two GPs, and the computational\ncomplexity is the same, and differs only by a constant factor. Do you\nhave more evidence for your claim?\n\n\"preformed\" -> performed\n\n\"considering a new data point takes $O(N_{tr}^3)$ computations\"\nNo, I believe it takes $O(n^2)$ computation to add one new data point\nusing block matrix inversion.\n\nWhy was GPy used, and not some other framework with GPU support such as\nGPyTorch, Pyro or GPFlow.\n\nThe Bayesian neural network was not clear, because it was a link to a\nblogpost with many codes including numpy and tensorflow. (the code\ncleared this up though)\n\n\"The measure in equation 6 relies on epistemic uncertainty prediction\nin the interval[0, 1].  This is only ensured for the\nproposed EpiOut approach and therefore the uncertainty measures provided\nbythe GP, Dropout and BNN are scaled to the unit interval based on the\nevaluation on all test points.\"\nIs this a fair evaluation method? If one of the other methods gives an\ninfinite uncertainty on some point, this will negatively affect its entire\nscore, even though the uncertainty predictions for the other points may\nbe fine. The evaluation method does not appear sound to me, and no\nreferences were provided for other works using such an evaluation method.\n\nBeing only confident close to the data trajectory may not be the best\napproach. The model should be confident when the prediction is likely to\nbe accurate. For example if the model is smooth, then it should be\npossible to interpolate reasonably accurately, and it should be OK\nfor the model to be confident as long as it has been able to learn\nthe structure sufficiently well.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P6tt2yGZ1jM", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros and the Key Idea**\nThis paper studies uncertainty quantification (UQ) in model-based learning for control, which is a timely and important research direction. The proposed method (EpiOpt) trains a neural network to predict the epistemic uncertainty directly. The training data for epistemic uncertainty prediction is artificially generated based on a simple nearest neighbor principle. The key ideas are: given the labeled training dataset $X_{tr},Y_{tr}$ (the data size $|X_{tr}|=N_{tr}$), this paper first randomly samples $X_{epi}$ around $X_{tr}$, where $|X_{epi}|=N_{epi}=k\\times N_{tr}$. Then this paper labels $x\\in X_{epi}$ by $1$ if the minimum distance from $x$ to $X_{tr}$ is far, and by $0$ if the distance is short. Finally, a neural network is trained for this binary classification task. \n\nFinally, this paper uses this idea in online control: the learned epistemic uncertainty is for adaptive data collection, and the aleatoric uncertainty is for control gain tunning. The advantage of this framework is that it is very simple, and doesn't need sampling or test-time dropout at the inference time.\n\n**Cons and Suggestions**\n(1) Many related work is missing especially for domain shift and adaptive control. As mentioned in this paper, the epistemic uncertainty is mainly from the data distribution shift, but there is no discussion about domain shift in this paper. The main idea of domain shift in ML is to *quantify the \"distance'' between the source and target domains*, which is similar to the epistemic uncertainty prediction $\\eta(\\cdot)$ in this paper. People also considers domain shift in control and learning (http://proceedings.mlr.press/v120/liu20a.html, https://arxiv.org/abs/2006.13916 and many others). \nAlso, adaptive control can handles epistemic uncertainty in an online manner as well. It would be great to discuss the difference between adaptive control. \nThe ensemble method (e.g., https://proceedings.neurips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf) is also an active method for UQ.\n\n(2) The key method, *EpiOpt*, is a bit too simple and there are no theoretical justifications or insights, such that it is hard to convince me about the generalirity and robustness of this method. As I mentioned above, the key idea of *EpiOpt* is to train a \"distance function\" $\\eta(\\cdot)$ to quantify the distance between the source and target data. This idea has been both theoretically and empiracally studied in domain shift and transfer learning. A few questions pop up from this paper: \n* Why do you need to sample $X_{epi}$ around $X_{tr}$? Since the goal is to get $\\eta(\\cdot)$, why don't you consider some analytical solutions such as KDE (kernel density estimation) to derive $\\eta(\\cdot)$? In other words, you could just use $X_{tr}$ to estimate a density function for the source data, and then evaluating this density function to get $\\eta(\\cdot)$. I didn't see a clear reason to *train* a neural network to estimation this distance.  \n* In equation (5), this paper labels $X_{epi}$ either $1$ or $0$. Why isn't it a continuous value from $0$ to $1$? For example, you can rank $d_j$ to get this continuous value.\n\n(3) The title and abstract of this paper emphasize a lot on *epistemic and aleatoric uncertainty decomposition*. However, the key method *EpiOpt* is only about the epistemic uncertainty, and how to deal with the aleatoric uncertainty only appears in the experimental section in equation (9). I highly recommend the authors discuss about these two types in the method section (Section 3) and give a general framework. The current aleatoric uncertainty is more like a gain tunning method, which is not related to *uncertainty decomposition* .\n\n(4) The experimental section is a bit vague. I highly recoomend the authors present the concrete learning and control problem first, e.g., Which part in the dynamics is learned? How to collect data? How to decompose the epistemic and aleatoric parts? A good example is https://ieeexplore.ieee.org/abstract/document/8794351, where the task is very similar. \n\n**Code-of-Ethics**\nI see no ethic issues in this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GWdh4M-2w7G", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Epistemic uncertainty is a useful measure to have from a learned model. This paper proposes to output epistemic uncertainty from NN by training with samples outside the training set. My concerns are listed below:\n\n1. The paper samples epi points around the training set. How does this perform when training set is very sparse or very spread out. \n2. Changing the yellow line to something darker and also making the thick black line to be a lighter color would improve clarity of the results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4tHorJFHdk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------\nThe authors consider the problem of efficient modeling of epistemic uncertainty, separated from aleatoric uncertainty,  for neural networks. They propose a novel methodology, involving automatically constructing a epistemic uncertainty support data set used to extend a given NN with an epistemic uncertainty output. The method is compared with previous, less efficient, approaches and is applied to the important problem of data-efficient online learning of a controller for real-time use with convincing results.\n\nStrong points\n------------- \n1. The paper is well written, and the important problem considered is clearly presented and motivated.\n2. The proposed approach with *epi points*, although seemingly simplistic at first glance due to its heavy dependence on a useful distance metric, is demonstrated to be quite effective for learning control in euclidean space. \n3. The proposed method is demonstrated empirically in an online learning setting using real disturbance data, making the applicability convincing. \n\n\nWeak points\n-------------\n1. While I recognizing that it is problematic to compare *bounded* (e.g probability value) and *unbounded* (e.g. regression target variance) epistemic uncertainty predictions, I wonder if isn't warranted to consider making the comparison fairer w.r.t. the baselines? Since the experimental maximum value for either baseline method depends on the choice of window size (i.e. $x\\in [-4 , 4]$), a fairer comparison might be to find (optimize) a scaling parameter $\\alpha \\in [0,1]$ of the computed normalized epistemic uncertainty prediction, which minimizes (6). Basically a worst-case situation w.r.t. the proposed method, with the baselines performing the best they could have given any upper bound (larger than observed within the window) on their epistemic uncertainty prediction. I still expect that the proposed method will perform comparatively well based on for example the functional forms in Figure 1.\n2. It is stated in the paper that \"*... (Dropout and BNN) have a larger total discount on the training set than on the test set.*\" (3.4, last bullet point). Can this be due to different maximum values between training and test when normalizing their epistemic uncertainty prediction?\n3. I regard the inclusion of the Vanilla GP model baseline as important. A major concern, however, is about the treatment of this baseline in the experiment methodology:\n   1. The epistemic uncertainty estimate for the GP is calculated based on the predictive variance. Is it not more appropriate to use the predictive standard deviation instead, such that the uncertainty has the same unit as the target? (The same applies to Dropout and BNN too?)\n   2. The data in *1D center* and *1D split* is very well suited for GP regression with an SE ARD kernel and should report a well calibrated epistemic regression uncertainty in input ranges where training data is absent. I find the miss-match between the left figure (showing the the GP mean prediction deviation) and the right figure (showing the epistemic uncertainty estimate) in Figure 1 odd. Implementing 3.1. might alleviate this issue?\n   3. I find it odd that EpiOut outperform the GP model on *1D center* and *1D split*, taking into consideration the performance metric (6). Maybe it is a consequence of issue 3.1.?\n   4. Table 4 and Table 5 in the appendix: It is odd that the GP test discount is smaller than the GP training discount on 2D Gaussian. It would indicate, as mentioned in the paper, that the epistemic uncertainty estimate is off for the GP model. Figure 3 (appendix) does indeed seem odd (or unclear) for the GP model, as does Figure 2 (appendix). Maybe it is a consequence of issue 3.1.? \n4. I am missing details on the online learning for the experiment, e.g: How long is each NN training and how is it related to the real-time aspects? Will the quadcopter fly far into state for which the current NN designates as high epistemic uncertainty before the NN is updated with new data points and  retrained such that the epistemic uncertainty is reduced? Or is the simulation paused and waiting for the NN training to complete on every new data point? If so this should be made clear in the paper.\n\nReason for score\n----------------\nIt is a good paper on an important topic, with supportive empirical evaluation of the applicability of the proposed novel approach. My major concern is with certain aspects of the comparative evaluation and about the clarity of the online learning experiment.\n\nMinor comments\n--------------\n1. I suggest that references to material in the appendix are made in the paper. Technicalities such as the online learning algorithm and relevant results would be made clearer.\n2. Figure 4: Left and middle figures are a bit hard to interpret. It seems like the figures show multiple runs, which makes them even more difficult. They might benefit from a thinner marker/stroke, transparency and/or a 3D effect such as a color gradient for view depth etc. They could also be changed to plotted against time instead of against space.\n3. GP regression with a squared exponential (SE) automatic relevance determination (ARD) kernel has, for a given training data (X,Y), a maximum epistemic regression uncertainty, given by the signal variance parameter $\\sigma_f$ of the kernel: As $d(x^*, x)$ grows the exponential in the kernel decrease towards 1 rapidly, where $x$ is a training data point and $x^*$ is an arbitrary data point. The epistemic uncertainty is consequently bounded for the selected choice of kernel family.\n\n3.5, bullet list, first bullet: \n  \"\\textit{EpiOut}model\" -> \"\\textit{EpiOut} model\"\n  \"\\textit{EpiOut}is\" -> \"\\textit{EpiOut} is\"\n  \n4.1, paragraph 1: \n  \"\\textit{EpiOut}approach\" -> \"\\textit{EpiOut} approach\"\n  \"\\textit{EpiOut}\\eta(\\cdot)\" -> \"\\textit{EpiOut} \\eta(\\cdot)\"\n\nEdit: Markdown problem with nested lists\n\nEdit: Upgraded rating due paper improvements\n\n------------\nThank you for addressing my concerns.\n\nFigure 1 is not updated (it looks the same w.r.t. the GP, but the change from variance to std should be noticeable?).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to do efficient epistemic uncertainty quantification for model-based learning for control. It does so by augmenting the dataset with synthetic data around the true data points, and trying to classify whether a point is close to the training set or not. I agree with many of the criticisms that R3 and R5 brought fourth. Namely, it's not clear why a kernel density estimate couldn't be used instead (runtime complexity is cited as the reason, but could be addressed through approximations, inducing points etc). It is not clear how to set the sampling distribution for X_epi. Also, since efficiency is a motivation for the work, I suggest that the authors look at and cite:

https://arxiv.org/abs/2002.06715 

I think at the moment the paper is not ready for publication, but the idea is interesting. Aside from comparing with the work above, what would improve this paper is an automatic way to select the distribution, or at least the covariance, of X_epi.  
Decision: Reject