Uncertainty Prediction for Deep Sequential Regression Using Meta Models

Generating high quality uncertainty estimates for sequential regression , particularly deep recurrent networks , remains a challenging and open problem . Existing approaches often make restrictive assumptions ( such as stationarity ) yet still perform poorly in practice , particularly in presence of real world non-stationary signals and drift . This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates , makes no assumptions about stationarity , and outperforms competitive baselines on both drift and non drift scenarios . This work helps make sequential regression more effective and practical for use in real-world applications , and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general . 1 INTRODUCTION . The ability to quantify the uncertainty of a model is one of the fundamental requirements in trusted , safe , and actionable AI ( Arnold et al. , 2019 ; Jiang et al. , 2018 ; Begoli et al. , 2019 ) . This paper focuses on uncertainty quantification in regression tasks , particularly in the context of deep neural networks ( DNN ) . We define a sequential task as one involving an ordered series of input elements , represented by features , and an ordered series of outputs . In sequential regression tasks ( SRT ) , the output elements are ( possibly multivariate ) real-valued variables . SRT occur in numerous applications , among others , in weather modeling , environmental modeling , energy optimization , and medical applications . When the cost of making an incorrect prediction is particularly high , such as in human safety , models without a reliable uncertainty estimation are perceived high risk and may not be adopted . Uncertainty prediction in DNNs has been subject to active research , in particular , spurred by what has become known as the “ Overconfidence Problem ” of DNNs Guo et al . ( 2017 ) , and by their susceptibility to adversarial attacks Madry et al . ( 2017 ) . However , the bulk of work is concerned with non-sequential , classification tasks ( see Section 2 ) leaving a noticeable gap for SRT . In this paper we introduce a meta-modeling concept as an approach to achieving high-quality uncertainty quantification in DNNs for SRT . We demonstrate that it not only outperforms competitive baselines but also provides consistent results across a variety of drift scenarios . We believe the approach represents a new powerful addition to the modeling toolbox in general . The novel contributions of this paper are summarized as follows : ( 1 ) Application of the meta-modeling concept to SRT , ( 2 ) Developing a joint base-meta model along with a comparison to white- and black-box alternatives , ( 3 ) Generating asymmetric uncertainty bounds in DNNs , and ( 4 ) Proposing a new evaluation methodology for SRT . 2 RELATED WORK . Classical statistics on time series offers an abundance of work dealing with uncertainty quantification ( Papoulis & Saunders , 1989 ) . Most notably in econometrics , a variety of heteroskedastic variance models lead to highly successful application in financial market volatility analyses ( Engle , 1982 ; Bollerslev , 1986 ; Mills , 1991 ) . An Autoregressive Conditional Heteroskedastic , or ARCH , model ( Engle , 1982 ) , and its generalized version , GARCH , ( Bollerslev , 1986 ) are two such methods , the latter of which serves as one of our baselines . An illuminating study ( Kendall & Gal , 2017 ) describes an integration of two sources of uncertainty , namely the epistemic ( due to model ) and the aleatoric ( due to data ) . The authors propose a variational approximation of Bayesian Neural Networks and an implicit Gaussian model to quantify both types of variability in a non-sequential classification and regression task . Based on Nix & Weigend ( 1994 ) , Lakshminarayanan et al . ( 2017 ) also uses an implicit Gaussian model to improve the predictive performance of a base model , again in a non-sequential setting . Similar to Kendall & Gal ( 2017 ) , the study does not focus on comparing the quality of the uncertainty to one generated by other methods . We use the implicit variance model of Kendall & Gal ( 2017 ) ; Oh et al . ( 2020 ) ; Lakshminarayanan et al . ( 2017 ) , as well as the method of variational dropout of Gal & Ghahramani ( 2016 ) ; Kendall & Gal ( 2017 ) as baselines in our work . A meta-modeling approach was taken in Chen et al . ( 2019 ) aiming at the task of instance filtering using white-box models . The work relates to ours through the meta-modeling concept but concentrates on classification in a non-sequential setting . Besides its application in filtering , meta-modeling has been widely applied in the task of learning to learn and lifelong learning ( Schmidhuber , 1987 ; Finn et al. , 2019 ) . However , it should be pointed out that the two applications of meta-modeling are not comparable due to their different objectives . Uncertainty in data drift conditions was assessed in a recent study ( Snoek et al. , 2019 ) . The authors employ calibration-based metrics to examine various methods for uncertainty in classification tasks ( image and text data ) , and conclude , among others that most methods ’ quality degrades with drift . Acknowledging drift as an important experimental aspect , our study takes it into account by testing in matched and drifted scenarios . Finally , Shen et al . ( 2018 ) described a multi-objective training of a DNN in wind power prediction , minimizing two types of cost related to coverage and bandwidth . We expand on these metrics in Section 3.3 . 3 METHOD . 3.1 META MODELING APPROACH . The basic concept of Meta Modeling ( MM ) , depicted in Figure 1 , involves a combination of two models comprising a base model , performing the main task ( e.g. , regression ) , and a meta model , learning to predict the base model ’ s error behavior . Depending on the amount of information shared between these two , we distinguish several settings , namely ( 1 ) base model is a black-box ( BB ) , ( 2 ) base is a white-box ( WB , base parameters are accessible ) , and ( 3 ) base and meta components are trained jointly ( JM ) . The advantages of WB and JM are obvious : rich information is available for the meta model to capture salient patterns for it to generate accurate predictions . On the other hand , the BB setting often occurs in practice and is a given . We now formalize the MM concept as it applies to sequential regression . Let ŷ = Fφ ( x ) be the base model function parametrized by φ , where x = x1 , ... , xN and ŷ = ŷ1 , ... , ŷM represent sequences of N input feature vectors and M Ddimensional output vectors , with ŷ ∈ RD×M . Let ẑ = Gγ ( ŷ , x , φ ) denote the meta model , parameterized by γ , taking as input the pre- dictions , the original features , and the parameters of the base to produce a sequence of error predictions , ẑ ∈ RD×M . The parameters φ are obtained by solving an optimization problem , argminφ E [ lb ( ŷ , y ) ] , using a smooth loss function lb , e.g. , the Frobenius norm lb = ‖ŷ − y‖ 2 F . Similarly , the parameters γ are determined via argminγ E [ lm ( ẑ , z ) ] = argminγ E [ lm ( ẑ , lz ( ŷ , y ) ) ] involving a loss lz quantifying the target error ( residual ) from the base model , and lm quantifying the prediction error of the meta model . In general , lb , lz , and lm , may differ . The expectations are estimated using an available dataset . We used the L2F norm for lb and lm , and L1 for lz , as described in Section 4 . Given differentiable loss functions and the DNN setting , the base and the meta model can be integrated in a single network ( JM ) . In this case the parameters are estimated jointly via φ∗ , γ∗ = argmin φ , γ E [ βlb ( ŷ , y ) + ( 1− β ) lm ( ẑ , lz ( ŷ , y ) ) ] ( 1 ) whereby dedicated output nodes of the network generate ŷt and ẑt , and β is a hyper-parameter trading off the base with the meta loss . Thus , one part of the network tackles the base task , minimizing the base residual , while another models the residual as the eventual measure of uncertainty . As done in ( Kendall & Gal , 2017 ) , one can argue that the base objective minimizes the epistemic ( parametric ) uncertainty , while the meta objective captures the aleatoric uncertainty present in the data . Due to their interaction , the base loss is influenced by the estimated uncertainty encouraging it to focus on feature-space regions with lower aleatoric uncertainty . Moreover , we conjecture , the DNN base model is encouraged to encode the input in ways suitable for uncertainty quantification . Figure 2 shows an overview of a sequential DNN architecture applied throughout our study . It includes a base encoder-decoder pair and a meta decoder connected to them . Each of these contains a recurrent memory cell - the LSTM ( Hochreiter & Schmidhuber , 1997 ) . The role of the encoder is to process the sequential input , x , compress its information in a context vector and pass it to the base decoder . The recurrent decoder produces the regression output ŷ in M time steps feeding its predictions as input in the next time steps . Evolving in time , both base LSTMs update their internal states bt and ht , whereby the last state , bN , serves as the context vector for the decoder . This architecture has gained wide popularity in applications such as speech-to-text ( Chiu et al. , 2018 ; Tüske et al. , 2019 ) , text-to-speech ( Sotelo et al. , 2017 ) , machine translation ( Sutskever et al. , 2014 ) , and image captioning ( Rennie et al. , 2016 ) . Following the MM concept , we attach an additional decoder ( the meta decoder ) via connections to the encoder and decoder outputs . The context vector , bN , is transformed by a fully connected layer ( FCN in Figure 2 ) , and both the ŷt output as well as the internal state , ht , are fed into the meta component . As mentioned above , the meta decoder generates uncertainty estimates , ẑt . Given the architecture depicted in Figure 2 , we summarize the three settings as follows : ( 1 ) Joint Model ( JM ) : parameters are trained according to Eq . ( 1 ) with certain values of β . ( 2 ) White-Box model ( WB ) : base parameters φ are trained first , followed by parameters γ , also accessing φ . ( 3 ) Black-Box model ( BB ) : same as ( 2 ) without access to φ . Figure 2 : Encoder-Decoder architecture integrating a base and a meta model Figure 3 : Bandwidth , excess , and deficit costs . Generating Symmetric and Asymmetric Bounds The choice of loss function , lz , gives rise to two scenarios . If lz is an even function , e.g. , lz ( ŷ , y ) = ‖ŷ − y‖1 , the meta-model targets z capture base error equally in both directions : above and below the target . Hence , the uncertainty ẑ predicted at test time will represent a interval symmetric around ŷ . If , on the other hand , lz takes the sign in z into account , it is possible to dedicate separate network nodes γl , γu ∈ γ to capturing lower and upper band estimates , ẑl and ẑu , respectively , thus accomplishing asymmetric prediction . Let δ = ŷ − y . For the asymmetric scenario the meta objective is modified as follows : γ∗ = argmin γ E [ lm ( zl , max { δ , 0 } ) + lm ( zu , max { −δ , 0 } ) ] ( 2 )

Summary: In this work, the authors present a meta-modeling approach to provide predictions with uncertainty estimates in a sequential task. They develop a white box, black box, and joint modeling method that allows them to apply their method to a variety of scenarios. These methods differ based on the amount of information provided to a meta-learner which has the goal of predicting errors $\hat{z} \in \mathbb{R}^{D \times M} $. The authors also incorporate the ability to make asymmetric uncertainty bounds. They apply this method and many baeslines to two datasets: MITV and SPE9PR.