Uncertainty Prediction for Deep Sequential Regression Using Meta Models

Generating high quality uncertainty estimates for sequential regression , particularly deep recurrent networks , remains a challenging and open problem . Existing approaches often make restrictive assumptions ( such as stationarity ) yet still perform poorly in practice , particularly in presence of real world non-stationary signals and drift . This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates , makes no assumptions about stationarity , and outperforms competitive baselines on both drift and non drift scenarios . This work helps make sequential regression more effective and practical for use in real-world applications , and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general . 1 INTRODUCTION . The ability to quantify the uncertainty of a model is one of the fundamental requirements in trusted , safe , and actionable AI ( Arnold et al. , 2019 ; Jiang et al. , 2018 ; Begoli et al. , 2019 ) . This paper focuses on uncertainty quantification in regression tasks , particularly in the context of deep neural networks ( DNN ) . We define a sequential task as one involving an ordered series of input elements , represented by features , and an ordered series of outputs . In sequential regression tasks ( SRT ) , the output elements are ( possibly multivariate ) real-valued variables . SRT occur in numerous applications , among others , in weather modeling , environmental modeling , energy optimization , and medical applications . When the cost of making an incorrect prediction is particularly high , such as in human safety , models without a reliable uncertainty estimation are perceived high risk and may not be adopted . Uncertainty prediction in DNNs has been subject to active research , in particular , spurred by what has become known as the “ Overconfidence Problem ” of DNNs Guo et al . ( 2017 ) , and by their susceptibility to adversarial attacks Madry et al . ( 2017 ) . However , the bulk of work is concerned with non-sequential , classification tasks ( see Section 2 ) leaving a noticeable gap for SRT . In this paper we introduce a meta-modeling concept as an approach to achieving high-quality uncertainty quantification in DNNs for SRT . We demonstrate that it not only outperforms competitive baselines but also provides consistent results across a variety of drift scenarios . We believe the approach represents a new powerful addition to the modeling toolbox in general . The novel contributions of this paper are summarized as follows : ( 1 ) Application of the meta-modeling concept to SRT , ( 2 ) Developing a joint base-meta model along with a comparison to white- and black-box alternatives , ( 3 ) Generating asymmetric uncertainty bounds in DNNs , and ( 4 ) Proposing a new evaluation methodology for SRT . 2 RELATED WORK . Classical statistics on time series offers an abundance of work dealing with uncertainty quantification ( Papoulis & Saunders , 1989 ) . Most notably in econometrics , a variety of heteroskedastic variance models lead to highly successful application in financial market volatility analyses ( Engle , 1982 ; Bollerslev , 1986 ; Mills , 1991 ) . An Autoregressive Conditional Heteroskedastic , or ARCH , model ( Engle , 1982 ) , and its generalized version , GARCH , ( Bollerslev , 1986 ) are two such methods , the latter of which serves as one of our baselines . An illuminating study ( Kendall & Gal , 2017 ) describes an integration of two sources of uncertainty , namely the epistemic ( due to model ) and the aleatoric ( due to data ) . The authors propose a variational approximation of Bayesian Neural Networks and an implicit Gaussian model to quantify both types of variability in a non-sequential classification and regression task . Based on Nix & Weigend ( 1994 ) , Lakshminarayanan et al . ( 2017 ) also uses an implicit Gaussian model to improve the predictive performance of a base model , again in a non-sequential setting . Similar to Kendall & Gal ( 2017 ) , the study does not focus on comparing the quality of the uncertainty to one generated by other methods . We use the implicit variance model of Kendall & Gal ( 2017 ) ; Oh et al . ( 2020 ) ; Lakshminarayanan et al . ( 2017 ) , as well as the method of variational dropout of Gal & Ghahramani ( 2016 ) ; Kendall & Gal ( 2017 ) as baselines in our work . A meta-modeling approach was taken in Chen et al . ( 2019 ) aiming at the task of instance filtering using white-box models . The work relates to ours through the meta-modeling concept but concentrates on classification in a non-sequential setting . Besides its application in filtering , meta-modeling has been widely applied in the task of learning to learn and lifelong learning ( Schmidhuber , 1987 ; Finn et al. , 2019 ) . However , it should be pointed out that the two applications of meta-modeling are not comparable due to their different objectives . Uncertainty in data drift conditions was assessed in a recent study ( Snoek et al. , 2019 ) . The authors employ calibration-based metrics to examine various methods for uncertainty in classification tasks ( image and text data ) , and conclude , among others that most methods ’ quality degrades with drift . Acknowledging drift as an important experimental aspect , our study takes it into account by testing in matched and drifted scenarios . Finally , Shen et al . ( 2018 ) described a multi-objective training of a DNN in wind power prediction , minimizing two types of cost related to coverage and bandwidth . We expand on these metrics in Section 3.3 . 3 METHOD . 3.1 META MODELING APPROACH . The basic concept of Meta Modeling ( MM ) , depicted in Figure 1 , involves a combination of two models comprising a base model , performing the main task ( e.g. , regression ) , and a meta model , learning to predict the base model ’ s error behavior . Depending on the amount of information shared between these two , we distinguish several settings , namely ( 1 ) base model is a black-box ( BB ) , ( 2 ) base is a white-box ( WB , base parameters are accessible ) , and ( 3 ) base and meta components are trained jointly ( JM ) . The advantages of WB and JM are obvious : rich information is available for the meta model to capture salient patterns for it to generate accurate predictions . On the other hand , the BB setting often occurs in practice and is a given . We now formalize the MM concept as it applies to sequential regression . Let ŷ = Fφ ( x ) be the base model function parametrized by φ , where x = x1 , ... , xN and ŷ = ŷ1 , ... , ŷM represent sequences of N input feature vectors and M Ddimensional output vectors , with ŷ ∈ RD×M . Let ẑ = Gγ ( ŷ , x , φ ) denote the meta model , parameterized by γ , taking as input the pre- dictions , the original features , and the parameters of the base to produce a sequence of error predictions , ẑ ∈ RD×M . The parameters φ are obtained by solving an optimization problem , argminφ E [ lb ( ŷ , y ) ] , using a smooth loss function lb , e.g. , the Frobenius norm lb = ‖ŷ − y‖ 2 F . Similarly , the parameters γ are determined via argminγ E [ lm ( ẑ , z ) ] = argminγ E [ lm ( ẑ , lz ( ŷ , y ) ) ] involving a loss lz quantifying the target error ( residual ) from the base model , and lm quantifying the prediction error of the meta model . In general , lb , lz , and lm , may differ . The expectations are estimated using an available dataset . We used the L2F norm for lb and lm , and L1 for lz , as described in Section 4 . Given differentiable loss functions and the DNN setting , the base and the meta model can be integrated in a single network ( JM ) . In this case the parameters are estimated jointly via φ∗ , γ∗ = argmin φ , γ E [ βlb ( ŷ , y ) + ( 1− β ) lm ( ẑ , lz ( ŷ , y ) ) ] ( 1 ) whereby dedicated output nodes of the network generate ŷt and ẑt , and β is a hyper-parameter trading off the base with the meta loss . Thus , one part of the network tackles the base task , minimizing the base residual , while another models the residual as the eventual measure of uncertainty . As done in ( Kendall & Gal , 2017 ) , one can argue that the base objective minimizes the epistemic ( parametric ) uncertainty , while the meta objective captures the aleatoric uncertainty present in the data . Due to their interaction , the base loss is influenced by the estimated uncertainty encouraging it to focus on feature-space regions with lower aleatoric uncertainty . Moreover , we conjecture , the DNN base model is encouraged to encode the input in ways suitable for uncertainty quantification . Figure 2 shows an overview of a sequential DNN architecture applied throughout our study . It includes a base encoder-decoder pair and a meta decoder connected to them . Each of these contains a recurrent memory cell - the LSTM ( Hochreiter & Schmidhuber , 1997 ) . The role of the encoder is to process the sequential input , x , compress its information in a context vector and pass it to the base decoder . The recurrent decoder produces the regression output ŷ in M time steps feeding its predictions as input in the next time steps . Evolving in time , both base LSTMs update their internal states bt and ht , whereby the last state , bN , serves as the context vector for the decoder . This architecture has gained wide popularity in applications such as speech-to-text ( Chiu et al. , 2018 ; Tüske et al. , 2019 ) , text-to-speech ( Sotelo et al. , 2017 ) , machine translation ( Sutskever et al. , 2014 ) , and image captioning ( Rennie et al. , 2016 ) . Following the MM concept , we attach an additional decoder ( the meta decoder ) via connections to the encoder and decoder outputs . The context vector , bN , is transformed by a fully connected layer ( FCN in Figure 2 ) , and both the ŷt output as well as the internal state , ht , are fed into the meta component . As mentioned above , the meta decoder generates uncertainty estimates , ẑt . Given the architecture depicted in Figure 2 , we summarize the three settings as follows : ( 1 ) Joint Model ( JM ) : parameters are trained according to Eq . ( 1 ) with certain values of β . ( 2 ) White-Box model ( WB ) : base parameters φ are trained first , followed by parameters γ , also accessing φ . ( 3 ) Black-Box model ( BB ) : same as ( 2 ) without access to φ . Figure 2 : Encoder-Decoder architecture integrating a base and a meta model Figure 3 : Bandwidth , excess , and deficit costs . Generating Symmetric and Asymmetric Bounds The choice of loss function , lz , gives rise to two scenarios . If lz is an even function , e.g. , lz ( ŷ , y ) = ‖ŷ − y‖1 , the meta-model targets z capture base error equally in both directions : above and below the target . Hence , the uncertainty ẑ predicted at test time will represent a interval symmetric around ŷ . If , on the other hand , lz takes the sign in z into account , it is possible to dedicate separate network nodes γl , γu ∈ γ to capturing lower and upper band estimates , ẑl and ẑu , respectively , thus accomplishing asymmetric prediction . Let δ = ŷ − y . For the asymmetric scenario the meta objective is modified as follows : γ∗ = argmin γ E [ lm ( zl , max { δ , 0 } ) + lm ( zu , max { −δ , 0 } ) ] ( 2 )

Summary: In this paper, the authors propose a technique for uncertainty estimation in regression with neural networks. The basic idea is to use an auxiliary "meta model" that (in the authors' best performing setting) has access to the base model and is trained jointly with it. The purpose of the meta model is to predict the error characteristics of the base model, which of course naturally leads to error bars. In order to account for the fact that the models errors on the train set are unlikely to be representative, the authors make use of a validation set on which the meta model is trained further.