Real-time Uncertainty Decomposition for Online Learning Control

1 INTRODUCTION . With improved sensor quality and more powerful computational resources , data-driven models are increasingly applied in safety-critical domains such as autonomous driving or human-robot interaction ( Grigorescu et al. , 2020 ) . However , measurements usually suffer from noise and the available data is often scarce compared to all possible states of a complex environment . This requires controllers , which rely on supervised learning techniques , to properly react to ignorance and imprecision in the model to avoid dangerous situations . In order to allow an implementation of risk-averse ( for exploitation and safety improvements ) or risk-seeking ( for exploration ) behavior , the model must clearly disaggregate the information in the data into more than just the “ best estimate ” and differentiate between different sources of uncertainty . Besides the point estimate of a model , one can distinguish aleatoric ( uncertainty in the data ) and epistemic ( uncertainty in the model ) uncertainty . The former is irreducible as it is inherent to the stochastic process the data is recorded from , while the latter origins from a limited expressive power of the model or scarce training samples ( Der Kiureghian & Ditlevsen , 2009 ) . Gaussian processes ( GPs ) inherently provide a measure for its fidelity with the posterior standard deviation prediction ( Rasmussen & Williams , 2006 ) . It also allows to differentiate aleatoric uncertainty ( typically considered as observation noise ) and epistemic uncertainty ( modeled by the kernel ) . However , the former allows only homoscedastic ( constant ) estimates , while real-world applications typically require heteroscedastic uncertainty models . An extension to heteroscedastic GP regression is presented in ( Lazaro-Gredilla & Titsias , 2011 ) , however , it is a variational approximation and further increases the computational complexity and GPs generally suffer from poor scaling to large datasets ( Quinonero-Candela & Rasmussen , 2005 ) . In deep learning , the modeling of uncertainties also gained increasing interest over the past years ( Kendall & Gal , 2017 ) . Heteroscedastic aleatoric uncertainty can be captured well , if the output of the stochastic process can directly be observed and its parametric distribution is known . However , for more general cases , approximation techniques such as variational inference or sampling is required ( Bishop , 2006 ) . For epistemic uncertainty estimation with neural networks ( NN ) , the key idea for most approaches can be summarized as follows . Randomness is introduced to the neural network through sampling during training and inference . While the training robustifies the network against the injected noise at the training locations , it allows the noise to pass to the output at input locations where no training data is available . For inference , multiple predictions of the network are sampled for the same inputs , allowing to compute a statistical measure for the uncertainty at the output ( Depeweg et al. , 2018 ; Depeweg , 2019 ) . However , sampling the network during inference is a high computational burden , and is therefore not suitable in real-time critical control tasks . An ensemble based approach by ( Lakshminarayanan et al. , 2017 ) works with far less instances of a network , but does not differentiate between aleatoric and epistemic uncertainty explicitly . Despite those drawbacks in the uncertainty representation of data-driven models , the control community started to incorporate them increasingly in the decision making for various applications . For example Fanger et al . ( 2016 ) uses an epistemic uncertainty measure to dynamically assign leader order follower roles for cooperative robotic manipulation . The work by Berkenkamp et al . ( 2016 ) ensures a safe exploration of an unknown task space based on GP error bounds and a gain scheduling approach for computed torque control is presented in Beckers et al . ( 2019 ) . The work by Liu et al . ( 2020 ) considers the epistemic uncertainty as an estimate of the distance between source and target domains ( known as domain shift ) to design a robust controller . In Umlauft & Hirche ( 2020 ) and Chowdhary et al . ( 2015 ) an online learning control approach for GPs models is considered , which approach the dual control problem ( Wittenmark , 1995 ) as a model-based adaptive control problem . The work by Yesildirak & Lewis ( 1995 ) uses neural network for adaptive control in a continuous time fashion , which relies on a time-triggered ( periodic ) update of the model rather than a eventbased adaptation as we propose in this work . More general , risk averse control strategies have been presented by Umlauft et al . ( 2018 ) ; Medina et al . ( 2013 ) ; Todorov & Li ( 2005 ) . However , all of these approaches only consider the model fidelity in general and do not differentiate between aleatoric and epistemic uncertainty . Therefore , the main contributions of this paper are the following . We propose a deep learning framework with a real-time capable epistemic uncertainty prediction . The resulting online learning model is employed by a controller , which shows a distinct reaction to epistemic and aleatoric uncertainty . We evaluate the proposed methods on synthetic and real-world benchmark data sets , and simulate a quadcopter controller , which learns online the disturbances injected by thermals . 2 PROBLEM FORMULATION . Consider the discrete-time dynamical system1 with control u ∈ U ⊆ Rdu and state x ∈ X ⊆ Rdx xk+1 = g ( xk , uk ) + yk , ( 1 ) where g : X× U→ X is known , while y is a i.i.d . random vector sampled in every time step from yk ∼ D ( f ( xk ) ) , ( 2 ) whereD ( · ) denotes a known distribution over real vectors y ∈ Y ⊆ Rdx and depends on the parameters p ∈ P ⊆ Rdp . These state-dependent parameters arise from an unknown mapping f : X→ P. We denote the unknown component yk of the dynamical system generally as disturbance but it could also be the unmodeled part of the dynamics , such as friction or serve as black-box model for the dynamics if no analytic description is available ( g ( · , · ) = 0 ) . We assume measurements can be taken to obtain the data set Dtr = { ( xi , yi ) } Ntri=1 with inputs Xtr = { xi } Ntr i=1 and outputs Ytr = { yi } Ntr i=1 , such that a model f̂ ( · ) of f ( · ) can be learned . Ntr ∈ N denotes the current number of training data points and is initially zero , i.e. , the training set is empty . The task is to choose a control input uk , such that the system ( 1 ) follows a given reference xdes . Furthermore , the controller can take new measurements of y to improve its model over time . We consider each measurement of y to be costly and therefore new training points should only be collected when necessary . Applications , where data collection is costly can be found in distributed systems , where multiple sensors share the same scarce communication channel , or in autonomous systems with limited data storage capacity . The need for high data efficiency requires models , which judge upon their own fidelity in real-time to identify valuable measurements . As existing approaches for modeling epistemic uncertainty in deep learning suffer from a high computational complexity we first focus on developing a novel method for epistemic uncertainty predictions before proposing an online learning control strategy which makes use of a neural network model decomposing its uncertainties . 1Bold/capital symbols generally denote vectors/matrices , D ( · ) /U ( · ) /N ( · ) /B ( · ) a general parametric/the uniform/Gaussian/Bernoulli distribution , respectively . 3 EPISTEMIC UNCERTAINTY ESTIMATION . 3.1 RELATED WORK . Learning an epistemic uncertainty estimator is not straight forward as it measures the absence of training data . Most prominently Gaussian processes with stationary kernels offer such a measure implicitly with their posterior variance prediction . However , GPs are known to scale poorly for large data sets : While regression and uncertainty predictions can be performed with O ( Ntr ) and O ( Ntr2 ) , respectively , considering a new data point takesO ( Ntr3 ) computations ( also without hyperparameter optimization , O ( Ntr2 ) for rank-1 update methods ) . While various methods have been proposed to make GP computationally more efficient , including sparse GPs ( Quinonero-Candela & Rasmussen , 2005 ) , distributed GPs ( Deisenroth & Ng , 2015 ) and local GPs ( Nguyen-Tuong et al. , 2009a ; b ) , these approximations typically focus only on the precision of the point estimate and distort the uncertainty prediction . For estimating the ” distance ” to the training set , kernel density estimation ( KDE ) can also be used ( Rosenblatt , 1956 ) , however , the non-parametric nature implies that the inference time grows linearly with the number of considered data points , which we aim to avoid . More recently , several different approaches for epistemic uncertainty estimates using deep learning frameworks have been proposed . Popular approaches rely on Bayesian approximations ( Depeweg et al. , 2016 ) or permanent dropouts ( not only during training to avoid overfitting ) ( Gal , 2016 ; Gal & Ghahramani , 2016 ) . Furthermore , latent inputs can also be used to achieve a decomposition into aleatoric and epistemic uncertainty as presented in ( Depeweg et al. , 2017 ) . However , in particular for Bayesian NNs , these approaches become computationally challenging . Firstly , they have a larger number of parameters to tune than their deterministic counterparts and rely on variational inference methods ( Kwon et al. , 2020 ) . Secondly , the prediction requires to sample the entire network before the statistics of the output can be computed . For the application in real-time critical control problems ( e.g. , robotics with a sampling rate of 1 kHz ) , these computational burdens prohibit an employment of these techniques . A sampling-free estimation methods is proposed by Postels et al . ( 2019 ) , but suffers from a quadratic space complexity in the number of weights in the network and relies on firstorder Taylor approximations in the propagation of the uncertainties , which might become inaccurate depending on the non-linearity of the activation functions 3.2 EpiOut - EXPLICITLY LEARNING EPISTEMIC UNCERTAINTY In order to allow the estimation of epistemic uncertainty in real-time , we introduce the idea of explicitly modeling it with a separate output of a neural network , calling it EpiOut . Since the epistemic uncertainty expresses the absence of data , the original data set Dtr does not contain data for training EpiOut . Therefore , we generate an epistemic uncertainty data set , with inputs Xepi = { x̃j } Nepi j=1 and outputs Yepi = { ỹj } Nepi j=1 concatenated in Depi = { ( x̃j , ỹj ) } Nepi j=1 , Nepi ∈ N. Different variations for sampling the set Xepi can be chosen depending on the desired scalability properties . A naive approach is to sample the entire input space uniformly , which suffers from the curse of dimensionality . Alternatively , we propose to sample around existing training points from a normal distribution Xepi = Ntr⋃ i=1 { x̃j ∼ N ( xi , Γ ) , j = 1 , . . . , Nepi/Ntr } , ( 3 ) where we implicitly assume that Nepi is chosen such that δ = Nepi/Ntr is a positive integer . Supposing that a standardization of the input space to unity is performed based on Xtr , Γ = I can be chosen if no further knowledge on f ( · ) is available . Otherwise , scaling Γ can be interpreted similarly to the lengthscale of a GP as a measure for how far away from a training point the prediction is reliable : Larger Γ will lead to further spread of Xepi and therefore low epistemic uncertainty in the neighborhood of the training data , which would be meaningful if the true function is known to have a low Lipschitz constant , and vice versa . We propose to set δ to a multiple of 2dx + 1 which corresponds to the intuition to pad each training point in both directions of each dimension with a epi point x̃ . The reasoning behind the additional +1 point will become clear in the following . To define the set Yepi , we first compute the minimal distance ( according to any distance metric d : X×X→ R0 , + ) to the training data for each epi point dj = min x∈Xtr d ( x̃j , x ) , j = 1 , . . . , Nepi , ( 4 ) keeping in mind that the closest training data point is not necessarily the one used to generate the sample . Let dNtr be theNtr-th smallest element of all dj , we generate Yepi and update Xepi as follows ỹj = { 1 , if dj > dNtr 0 , x̃j ← arg minx∈Xtr d ( x̃j , x ) if dj ≤ dNtr . ( 5 ) Thus , theNtr points in Xepi with the least distance to a training point are replaced by the corresponding point in Xtr . Now the choice of 2dx + 1 epi points becomes clear as one of them will be turned into ỹ = 0 corresponding to “ low epistemic uncertainty ” , while 2dx points further away from the training point ỹ = 0 indicate the opposite . Given the data set Depi , the neural network is now equipped with one additional output , i.e. , the parameter layer is dp + 1 dimensional with output [ f̂ ( · ) η ( · ) ] T . The new output η ( · ) is terminated with a neuron using a sigmoidal activation function , such that η : X → [ 0 , 1 ] . This is beneficial because it allows immediately to judge , whether the predicted uncertainty is high ( ≈ 1 ) or low ( ≈ 0 ) without any reference evaluation ( see comparison to alteranative methods below ) . Independently of the loss function for the original network , the augmented output , also considered as epistemic output is trained using a binary cross-entropy loss , which is the natural choice for binary outputs . It quantifies the uncertainty in the prediction of the other outputs based on the distance to the training data measured by d ( · , · ) . For the sake of focus , we will be using the Euclidean distance , however the method can be easily extended to other metrics and we leave it to future work to investigate alternatives .

Summary: The authors consider the problem of efficient modeling of epistemic uncertainty, separated from aleatoric uncertainty,  for neural networks. They propose a novel methodology, involving automatically constructing a epistemic uncertainty support data set used to extend a given NN with an epistemic uncertainty output. The method is compared with previous, less efficient, approaches and is applied to the important problem of data-efficient online learning of a controller for real-time use with convincing results.