Year: 2020

Abstract:
Safety-critical decisions based on machine learning models require a clear understanding of the involved uncertainties to avoid hazardous or risky situations. While aleatoric uncertainty can be explicitly modeled given a parametric description, epistemic uncertainty rather describes the presence or absence of training data. This paper proposes a novel generic method for modeling epistemic uncertainty and shows its advantages over existing approaches for neural networks on various data sets. It can be directly combined with aleatoric uncertainty estimates and allows for prediction in real-time as the inference is sample-free. We exploit this property in a model-based quadcopter control setting and demonstrate how the controller benefits from a differentiation between aleatoric and epistemic uncertainty in online learning of thermal disturbances.

heading: 1 INTRODUCTION
text: With improved sensor quality and more powerful computational resources, data-driven models are increasingly applied in safety-critical domains such as autonomous driving or human-robot interaction (Grigorescu et al., 2020). However, measurements usually suffer from noise and the available data is often scarce compared to all possible states of a complex environment. This requires controllers, which rely on supervised learning techniques, to properly react to ignorance and imprecision in the model to avoid dangerous situations. In order to allow an implementation of risk-averse (for exploitation and safety improvements) or risk-seeking (for exploration) behavior, the model must clearly disaggregate the information in the data into more than just the “best estimate” and differentiate between different sources of uncertainty. Besides the point estimate of a model, one can distinguish aleatoric (uncertainty in the data) and epistemic (uncertainty in the model) uncertainty. The former is irreducible as it is inherent to the stochastic process the data is recorded from, while the latter origins from a limited expressive power of the model or scarce training samples (Der Kiureghian & Ditlevsen, 2009).
Gaussian processes (GPs) inherently provide a measure for its fidelity with the posterior standard deviation prediction (Rasmussen & Williams, 2006). It also allows to differentiate aleatoric uncertainty (typically considered as observation noise) and epistemic uncertainty (modeled by the kernel). However, the former allows only homoscedastic (constant) estimates, while real-world applications typically require heteroscedastic uncertainty models. An extension to heteroscedastic GP regression is presented in (Lazaro-Gredilla & Titsias, 2011), however, it is a variational approximation and further increases the computational complexity and GPs generally suffer from poor scaling to large datasets (Quinonero-Candela & Rasmussen, 2005).
In deep learning, the modeling of uncertainties also gained increasing interest over the past years (Kendall & Gal, 2017). Heteroscedastic aleatoric uncertainty can be captured well, if the output of the stochastic process can directly be observed and its parametric distribution is known. However, for more general cases, approximation techniques such as variational inference or sampling is required (Bishop, 2006). For epistemic uncertainty estimation with neural networks (NN), the key idea for most approaches can be summarized as follows. Randomness is introduced to the neural network through sampling during training and inference. While the training robustifies the network against the injected noise at the training locations, it allows the noise to pass to the output at input locations where no training data is available. For inference, multiple predictions of the network are sampled for the same inputs, allowing to compute a statistical measure for the uncertainty
at the output (Depeweg et al., 2018; Depeweg, 2019). However, sampling the network during inference is a high computational burden, and is therefore not suitable in real-time critical control tasks. An ensemble based approach by (Lakshminarayanan et al., 2017) works with far less instances of a network, but does not differentiate between aleatoric and epistemic uncertainty explicitly.
Despite those drawbacks in the uncertainty representation of data-driven models, the control community started to incorporate them increasingly in the decision making for various applications. For example Fanger et al. (2016) uses an epistemic uncertainty measure to dynamically assign leader order follower roles for cooperative robotic manipulation. The work by Berkenkamp et al. (2016) ensures a safe exploration of an unknown task space based on GP error bounds and a gain scheduling approach for computed torque control is presented in Beckers et al. (2019). The work by Liu et al. (2020) considers the epistemic uncertainty as an estimate of the distance between source and target domains (known as domain shift) to design a robust controller. In Umlauft & Hirche (2020) and Chowdhary et al. (2015) an online learning control approach for GPs models is considered, which approach the dual control problem (Wittenmark, 1995) as a model-based adaptive control problem. The work by Yesildirak & Lewis (1995) uses neural network for adaptive control in a continuous time fashion, which relies on a time-triggered (periodic) update of the model rather than a eventbased adaptation as we propose in this work. More general, risk averse control strategies have been presented by Umlauft et al. (2018); Medina et al. (2013); Todorov & Li (2005). However, all of these approaches only consider the model fidelity in general and do not differentiate between aleatoric and epistemic uncertainty.
Therefore, the main contributions of this paper are the following. We propose a deep learning framework with a real-time capable epistemic uncertainty prediction. The resulting online learning model is employed by a controller, which shows a distinct reaction to epistemic and aleatoric uncertainty. We evaluate the proposed methods on synthetic and real-world benchmark data sets, and simulate a quadcopter controller, which learns online the disturbances injected by thermals.

heading: 2 PROBLEM FORMULATION
text: Consider the discrete-time dynamical system1 with control u ∈ U ⊆ Rdu and state x ∈ X ⊆ Rdx
xk+1 = g(xk,uk) + yk, (1)
where g : X× U→ X is known, while y is a i.i.d. random vector sampled in every time step from yk ∼ D(f(xk)), (2)
whereD(·) denotes a known distribution over real vectors y ∈ Y ⊆ Rdx and depends on the parameters p ∈ P ⊆ Rdp . These state-dependent parameters arise from an unknown mapping f : X→ P. We denote the unknown component yk of the dynamical system generally as disturbance but it could also be the unmodeled part of the dynamics, such as friction or serve as black-box model for the dynamics if no analytic description is available (g(·, ·) = 0). We assume measurements can be taken to obtain the data set Dtr = {(xi,yi)}Ntri=1 with inputs Xtr = {xi} Ntr i=1 and outputs Ytr = {yi} Ntr i=1, such that a model f̂(·) of f(·) can be learned. Ntr ∈ N denotes the current number of training data points and is initially zero, i.e., the training set is empty.
The task is to choose a control input uk, such that the system (1) follows a given reference xdes. Furthermore, the controller can take new measurements of y to improve its model over time. We consider each measurement of y to be costly and therefore new training points should only be collected when necessary. Applications, where data collection is costly can be found in distributed systems, where multiple sensors share the same scarce communication channel, or in autonomous systems with limited data storage capacity.
The need for high data efficiency requires models, which judge upon their own fidelity in real-time to identify valuable measurements. As existing approaches for modeling epistemic uncertainty in deep learning suffer from a high computational complexity we first focus on developing a novel method for epistemic uncertainty predictions before proposing an online learning control strategy which makes use of a neural network model decomposing its uncertainties.
1Bold/capital symbols generally denote vectors/matrices,D(·)/U(·)/N (·)/B(·) a general parametric/the uniform/Gaussian/Bernoulli distribution, respectively.

heading: 3 EPISTEMIC UNCERTAINTY ESTIMATION

heading: 3.1 RELATED WORK
text: Learning an epistemic uncertainty estimator is not straight forward as it measures the absence of training data. Most prominently Gaussian processes with stationary kernels offer such a measure implicitly with their posterior variance prediction. However, GPs are known to scale poorly for large data sets: While regression and uncertainty predictions can be performed with O(Ntr) and O(Ntr2), respectively, considering a new data point takesO(Ntr3) computations (also without hyperparameter optimization, O(Ntr2) for rank-1 update methods). While various methods have been proposed to make GP computationally more efficient, including sparse GPs (Quinonero-Candela & Rasmussen, 2005), distributed GPs (Deisenroth & Ng, 2015) and local GPs (Nguyen-Tuong et al., 2009a;b), these approximations typically focus only on the precision of the point estimate and distort the uncertainty prediction. For estimating the ”distance” to the training set, kernel density estimation (KDE) can also be used (Rosenblatt, 1956), however, the non-parametric nature implies that the inference time grows linearly with the number of considered data points, which we aim to avoid.
More recently, several different approaches for epistemic uncertainty estimates using deep learning frameworks have been proposed. Popular approaches rely on Bayesian approximations(Depeweg et al., 2016) or permanent dropouts (not only during training to avoid overfitting) (Gal, 2016; Gal & Ghahramani, 2016). Furthermore, latent inputs can also be used to achieve a decomposition into aleatoric and epistemic uncertainty as presented in (Depeweg et al., 2017). However, in particular for Bayesian NNs, these approaches become computationally challenging. Firstly, they have a larger number of parameters to tune than their deterministic counterparts and rely on variational inference methods (Kwon et al., 2020). Secondly, the prediction requires to sample the entire network before the statistics of the output can be computed. For the application in real-time critical control problems (e.g., robotics with a sampling rate of 1 kHz), these computational burdens prohibit an employment of these techniques. A sampling-free estimation methods is proposed by Postels et al. (2019), but suffers from a quadratic space complexity in the number of weights in the network and relies on firstorder Taylor approximations in the propagation of the uncertainties, which might become inaccurate depending on the non-linearity of the activation functions
3.2 EpiOut - EXPLICITLY LEARNING EPISTEMIC UNCERTAINTY
In order to allow the estimation of epistemic uncertainty in real-time, we introduce the idea of explicitly modeling it with a separate output of a neural network, calling it EpiOut. Since the epistemic uncertainty expresses the absence of data, the original data set Dtr does not contain data for training EpiOut. Therefore, we generate an epistemic uncertainty data set, with inputs Xepi = {x̃j} Nepi j=1 and outputs Yepi = {ỹj} Nepi j=1 concatenated in Depi = {(x̃j , ỹj)} Nepi j=1, Nepi ∈ N.
Different variations for sampling the set Xepi can be chosen depending on the desired scalability properties. A naive approach is to sample the entire input space uniformly, which suffers from the curse of dimensionality. Alternatively, we propose to sample around existing training points from a normal distribution
Xepi = Ntr⋃ i=1 {x̃j ∼ N (xi, Γ), j = 1, . . . ,Nepi/Ntr} , (3)
where we implicitly assume that Nepi is chosen such that δ = Nepi/Ntr is a positive integer. Supposing that a standardization of the input space to unity is performed based on Xtr, Γ = I can be chosen if no further knowledge on f(·) is available. Otherwise, scaling Γ can be interpreted similarly to the lengthscale of a GP as a measure for how far away from a training point the prediction is reliable: Larger Γ will lead to further spread of Xepi and therefore low epistemic uncertainty in the neighborhood of the training data, which would be meaningful if the true function is known to have a low Lipschitz constant, and vice versa.
We propose to set δ to a multiple of 2dx + 1 which corresponds to the intuition to pad each training point in both directions of each dimension with a epi point x̃. The reasoning behind the additional +1 point will become clear in the following. To define the set Yepi, we first compute the minimal
distance (according to any distance metric d : X×X→ R0,+) to the training data for each epi point
dj = min x∈Xtr d(x̃j ,x), j = 1, . . . ,Nepi, (4)
keeping in mind that the closest training data point is not necessarily the one used to generate the sample. Let dNtr be theNtr-th smallest element of all dj , we generate Yepi and update Xepi as follows
ỹj = { 1, if dj > dNtr 0, x̃j ← arg minx∈Xtr d(x̃j ,x) if dj ≤ dNtr . (5)
Thus, theNtr points in Xepi with the least distance to a training point are replaced by the corresponding point in Xtr. Now the choice of 2dx + 1 epi points becomes clear as one of them will be turned into ỹ = 0 corresponding to “low epistemic uncertainty”, while 2dx points further away from the training point ỹ = 0 indicate the opposite.
Given the data set Depi, the neural network is now equipped with one additional output, i.e., the parameter layer is dp + 1 dimensional with output [f̂(·) η(·)]T . The new output η(·) is terminated with a neuron using a sigmoidal activation function, such that η : X → [0, 1]. This is beneficial because it allows immediately to judge, whether the predicted uncertainty is high (≈ 1) or low (≈ 0) without any reference evaluation (see comparison to alteranative methods below).
Independently of the loss function for the original network, the augmented output, also considered as epistemic output is trained using a binary cross-entropy loss, which is the natural choice for binary outputs. It quantifies the uncertainty in the prediction of the other outputs based on the distance to the training data measured by d(·, ·). For the sake of focus, we will be using the Euclidean distance, however the method can be easily extended to other metrics and we leave it to future work to investigate alternatives.

heading: 3.3 COMPUTATIONAL COMPLEXITY
text: The analysis of the computational complexity shows that (3) is a O(Nepi)=̂O(Ntrdx) operation, whereas (4) is for a trivial implementation a O(NtrNepi)=̂O(dxNtr2) operation. However, an implementation based on kd-tree (Cormen et al., 2009) allows an execution in O(Nepi log(Nepi))=̂O(dxNtr log(Ntrdx))) time. Finding the Ntr smallest distances from all Nepi points in (5) can obtained in O(Ntr + (Nepi −Ntr) log(Ntr))=̂O(Ntr +Ntr(dx − 1) log(Ntr)) time. The training of neural network with a fixed number of weights requiresO(Nepi)=̂O(Ntrdx). Hence, the overall complexity results in O(dxNtr log(dxNtr)), and it is straight forward to derive an overall space complexity of O(Nepidx)=̂O(Ntrdx2) for storing the set Xepi. The following should be considered when comparing to classical deep learning frameworks which generally can be trained in linear time.
• When used on streaming data (as for online learning control), the set Depi can be constructed iteratively, reducing the complexity to O(log(Ntr))
• The most time critical computation (4) can efficiently be parallelized on a GPU. • The method is designed for applications where measuring data is considered costly and
therefore sparse data can be expected.

heading: 3.4 EVALUATION
text: For evaluation we compare the following models. The implementation is available in the supplementary material.
• vanilla GP model with a squared exponential automatic relevance determination kernel based on the GPy implementation2
• BNN: Bayesian Neural Network with 2 fully connected hidden layers each with 50 hidden units and normal distributions over their weights based on this implementation.3
2https://sheffieldml.github.io/GPy/ 3https://matthewmcateer.me/blog/a-quick-intro-to-bayesian-neural-networks/
• Dropout: A neural network with 2 fully connected permanent layers each with 50 hidden units with dropout probability ρ = 0.05. 4.
• EpiOut: The proposed model with 2 fully connected layers (50 neurons) and Γ = I , δ = 2.
For the evaluation we utilize a weighted mean square error measures defined as follows
ρ = ∑Nte i=1(yi − f̂(xi))2(1− η(xi))∑Nte
i=1(1− η(xi)) , (6)
i.e., if the model decides that it is uncertain about the prediction at a test point, the squared error for this prediction is discounted (weighted less). However, the model can only achieve a low ρ if it is also certain at some test points, because the denominator, goes to zero for many uncertain predictions. In consequence, ρ is only defined if η(·) < 1 holds for at least one test point. Furthermore, the total discount, defined as ∑N i=1 η(xi) can additionally be utilized for a plausibility check of the epistemic uncertainty predictions since it should generally be larger on the test than on the training data set.
The measure in (6) relies on epistemic uncertainty prediction in the interval [0, 1]. This is only ensured for the proposed EpiOut approach and therefore the uncertainty measures, more specifically the predicted standard deviation, provided by the GP, Dropout and BNN are scaled to the unit interval based on the evaluation on all test and training points.
The following data sets are utilized for evaluation.
• 1D Center: The nominal function is f(x) = sin(xπ), with training points Xtr = {xi ∼ U(−1, 1)}100i=1 and Nte = 961 test points are placed on a grid [−4, 4]. • 1D Split: Same as 1D Center, but Xtr = {xi ∼ U(−2,−1)}100i=1 ∪ {xi ∼ U(1, 2)}200i=101. • 2D Gaussian: The nominal function (dx = 2, dp = 1) is f(x) =
sin(5x1) 5x1 + x22 with training points Xtr = { xi ∼ N ([ −1 0 ] , [ 0.02 0 0 0.1 ])}500 i=1
∪{ xi ∼ N ([ 1 0 ] , [ 0.02 0 0 0.1 ])}1000 i=501 and Nte = 961 test points are uniformly placed
on a grid [−2, 2]2. • 2D Square: Same as 2D Gaussian, but with withNtr = 80 training points placed uniformly
along the boundary of the square [−1, 1]2. • PMSM temperature is a 2Hz recording (dx = 8, dy = 1) of the temperature from a perma-
nent magnet synchronous motor.5. To allow a comparison with the GP within reasonable computational limits, Ntr = 5000 and Nte = 1000 points were randomly extracted from a total of ≈ 106 samples.
• Sarcos is a data set for learning the inverse dynamics of a seven degrees-of-freedom SARCOS anthropomorphic robot arm dx = 21, dp = 1.6. Ntr = 10000 and Nte = 2000 points were randomly extracted from a total of ≈ 5× 104 samples.

heading: 3.5 RESULTS & DISCUSSION
text: The numerical results are presented in Table 1 and an illustration for the data set 1D Split for all models is shown in Fig. 1. Besides showing empirically an advantage over existing approaches we want to point out the following benefits.
• The EpiOut model predicts the uncertainty measure in a sample free manner. This is crucial in data-efficient online learning scenarios, where the epistemic uncertainty is used to evaluate the usefulness of an incoming data point to decide upon its rejection. Hence, it is called more frequently than the online training function and must be computationally efficient. The prediction time for EpiOut is typically an order of magnitude faster than Dropout and BNN.
4https://github.com/yaringal/DropoutUncertaintyDemos 5https://www.kaggle.com/wkirgsn/electric-motor-temperature 6http://www.gaussianprocess.org/gpml/data/
For more extensive results and exact computation times, we refer to the supplementary material.

heading: 4 ONLINE LEARNING CONTROL USING UNCERTAINTY DECOMPOSITION

heading: 4.1 EVALUATION IN A QUADCOPTER CONTROL TASK
text: As an application of our proposed approach, we consider the task to control a quadcopter which explores a given terrain with unknown thermals7. We assume that the quadcopter dynamics i.e.g(·, ·) in 1 is known (compare (Shi et al., 2019)). The thermals act on the quadcopter as a disturbance on the acceleration in zq-direction8 We model the disturbance as normal distribution N (µ(x),σ2(x)), leading to p = [µ,σ]T and the NN models the state dependency of these parameters f̂ : X → P. Initially, there is no training data available and the desired trajectory xdes are three rounds at constant height zq = 0 on a square in the xq-yq-plane with edge length 0.1 followed by three rounds with edge length 0.05.
The online learning control approach for achieving this task with a high tracking precision and dataefficiency is presented in the following.

heading: 4.2 LEARNING CONTROL DESIGN
text: The EpiOut approach is sufficiently fast to be implemented in a mobile real-time control loop and therefore serves as trigger when new data points must be measured. By sampling from a Bernoulli
7The data of the thermals is taken from publicly available paragliding data https://thermal.kk7.ch. 8Superscripts q distinguish the quadcopter position xq ,yq ,zq from the general inputs x and outputs y.
distribution, whose parameter corresponds to η(·), new measurements (xi,yi) are added to the training according to
Dtr ← Dtr ∪ {
(xi,yi) if i = 1 ∅ if i = 0 where i ∼ B(α), α = η(xi). (7)
This ensures a high accuracy of the disturbance model f̂(·) as training data is added when necessary. It implements a stochastic event-triggered adaptive control approach, which is highly data-efficient and reduces the number of costly measurements. In particular for mobile platforms this cost comes in terms of reduced battery life (processing data requires power for computations), shorter operation time (local data storage quickly fills up if operating near 1 kHz sampling rates) or increased communication efforts (in case data is not process or stored onboard).
The system (1) is inherently random due to the stochastic nature of the disturbance f(·). Therefore, we combine feedback and feedforward control law
u = K ( x− xdes ) + uff, (8)
where x, xdes is the (desired) state of the quadcopter and uff is a feedforward control term determined based on the known model g(·, ·) (the gravitational force on the quadcopter) and the learned disturbance model f̂(·), more specifically the mean of the predicted disturbance µ(·). The choice of the feedback gain K, which compensates for imprecision in the model and the stochasticity of the disturbance, is a difficult trade-off because high control gains lead to high tracking performance, but also consume more energy and reinforce measurement noise, which can lead to instability. Therefore, it is generally advisable to let the feedforward term uff take over most of the control effort and keep the feedback term small when the model is reliable. Therefore, we increase the gains only if the aleatoric uncertainty inferred by our model as σ(·) is high. Since the disturbance acts only in zq direction, only the z component of the gain is affected i.e.,
kz = k̄z ( 1 + βσ(x)), (9)
where β ∈ R0,+ is the sensitivity and k̄ ∈ R+ defines the minimum control gain. This gain scheduling allows to robustify the closed-loop against process noise and can even guarantee stability Beckers et al. (2019), while at the same time we can keep the energy consumption low. While previous works by Fanger et al. (2016); Beckers et al. (2019) take a general uncertainty measure, we tune the feedback gains only based on the irreducible (aleatoric) uncertainty, while we combat the epistemic uncertainty with an increased data collection rate (7).
A summary of the control strategy is given in Algorithm 1 and a visualization of the neural network in Fig. 2. The outputs (red nodes) are iteratively trained with the indicated loss functions and respective data sets D, Depi for two epochs with a a learning rate of 0.01 with each incoming data point. The current implementation pauses during the training of the disturbance model before the control loop continuous.
Algorithm 1 Online learning controller 1: initialize disturbance model 2: while control task is not completed do 3: get state xi 4: evaluate disturbance model 5: update control gains (9) 6: if measurement is required (7) then 7: measure disturbance yi 8: update data set Dtr ← Dtr ∪ (xi,yi) 9: resample Depi (3) 10: retrain disturbance model 11: end if 12: apply control uk (8) 13: i← i + 1 14: end while
The tracking performance of the proposed quadcopter controller, the data collections rate and the epistemic uncertainty model is illustrated in Fig. 3. The implementation and further results are provided in the supplementary material.

heading: 5 CONCLUSION
text: This paper presents a novel deep learning structure for decomposing epistemic and aleatoric uncertainty and proposes an advanced control framework making distinct use of these uncertainty measures. As the prediction by the model are obtained sample-free, it allows for real-time critical online learning and outperforms existing methods on the proposed uncertainty-weighted precision measure. The proposed online learning control algorithm is inherently data-efficient by adding only required points to the data set.
For future work will consider alternative functions for sorting the epi points to encode prior knowledge, such as as periodicity (similar to a kernel of GP) and investigate the effect of a continuous valued ỹ.

Review summaries:
The authors consider the problem of efficient modeling of epistemic uncertainty, separated from aleatoric uncertainty,  for neural networks. They propose a novel methodology, involving automatically constructing a epistemic uncertainty support data set used to extend a given NN with an epistemic uncertainty output. The method is compared with previous, less efficient, approaches and is applied to the important problem of data-efficient online learning of a controller for real-time use with convincing results.