Uncertainty Prediction for Deep Sequential Regression Using Meta Models
Authors: Jiri Navratil
Matthew Arnold
Benjamin Elder
Venue:  (2021)
Keywords: Uncertainty Quantification
Uncertainty Prediction
Deep Learning
Regression
Meta Modeling

Abstract: Generating high quality uncertainty estimates for sequential regression, particularly deep recurrent networks, remains a challenging and open problem.
Existing approaches often make restrictive assumptions (such as stationarity) yet still perform poorly in practice, particularly in presence of real world non-stationary signals and drift. 
This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates,  makes no assumptions about stationarity, and outperforms competitive baselines on both drift and non drift scenarios.
This work helps make sequential regression more effective and practical for use in real-world applications, and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general.

Under review as a conference paper at ICLR 2021
UNCERTAINTY PREDICTION FOR DEEP SEQUENTIAL REGRESSION USING META MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Generating high quality uncertainty estimates for sequential regression, particularly deep recurrent networks, remains challenging and open problem. Existing ap- proaches often make restrictive assumptions (such as stationarity) yet still perform poorly in practice, particularly in presence of real world non-stationary signals and drift. This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates, makes assumptions about stationarity, and outperforms competitive baselines on both drift and non drift scenarios. This work helps make sequential regression more effective and practical for in real-world applications, and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general.
INTRODUCTION
The ability to quantify the uncertainty of model is one of the fundamental requirements in trusted, safe, and actionable (Arnold et al. 2019 Jiang et al., 2018 Begoli et al., 2019).
This paper focuses uncertainty quantification in regression tasks, particularly in the context of deep neural networks (DNN). We define a sequential task as one involving an ordered series of input elements, represented by features, and an ordered series of outputs. In sequential regression tasks (SRT), the output elements are (possibly multivariate) real-valued variables. SKT occur in numerous applications, among others, in weather modeling, environmental modeling, energy optimization, and medical applications. When the cost of making an incorrect prediction is particularly high, such as in human safety, models without reliable uncertainty estimation are perceived high risk and may be adopted.
Uncertainty prediction in DNNs has been subject to active research, in particular, spurred by what has become known as the "Overconfidence Problem" of DNNs Guo et al. (2017), and by their susceptibility adversarial attacks Madry et al. (2017). However, the bulk of work is concerned with non-sequential, classification tasks (see Section leaving noticeable gap for SRT.
In this paper we introduce meta-modeling concept as an approach to achieving high-quality uncertainty quantification in DNNs for SRT. We demonstrate that it not only outperforms competitive baselines but also provides consistent results across variety of drift scenarios. We believe the approach represents new powerful addition to the modeling toolbox in general.
The novel contributions of this paper are summarized as follows: (1) Application of the meta-modeling concept to SRT, (2) Developing joint base-meta model along with comparison to white- and black-box alternatives, (3) Generating asymmetric uncertainty bounds DNNs, and (4) Proposing a new evaluation methodology for SRT.
2 RELATED WORK
Classical statistics on time series offers an abundance of work dealing with uncertainty quantification (Papoulis & Saunders 1989). Most notably in econometrics, variety of heteroskedastic variance
models lead to highly successful application in financial market volatility analyses (Engle 1982
Bollerslev 1986. Mills 1991). An Autoregressive Conditional Heteroskedastic, ARCH, model

---

Under review as a conference paper at ICLR 2021
(Engle, 1982), and its generalized version, GARCH, (Bollerslev 1986) are two such methods, the
latter of which serves as one of our baselines.
An illuminating study (Kendall & Gal 2017) describes integration of two sources of uncertainty,
namely the epistemic (due to model) and the aleatoric (due to data). The authors propose a variational approximation of Bayesian Neural Networks and an implicit Gaussian model quantify both types of variability in a non-sequential classification and regression task. Based on Nix & Weigend (1994).
al. (2017) also uses an implicit Gaussian model to improve the predictive performance of a base model, again in non-sequential setting. Similar to [Kendall & Gal (2017), the study does not focus on comparing the quality of the uncertainty to one generated by other methods. We use the implicit variance model of Kendal & Gal (2017): Oh et al. (2020);; Lakshminarayaaa
Lakshminarayanan
al. (2017), as well as the method of variational dropout of Gal & Ghahramani (2016); Kendall & Gal (2017) as baselines in our work. A meta-modeling approach was taken in Chen et al. (2019) aiming at the task of instance filtering using white-box models. The work relates to ours through the meta-modeling concept but concentrates on classification in non-sequential setting. Besides its application in filtering, meta-modeling has been widely applied in the task of learning to learn and lifelong learning (Schmidhuber 1987; Finn et al.) 2019). However, it should be pointed out that the two applications of meta- modeling are not comparable due to their different objectives. Uncertainty in data drift conditions was assessed in a recent study (Snoek et al.) 2019). The authors employ calibration-bant based metrics to examine various methods for uncertainty in classification tasks (image and text data), and conclude, among others that most methods' quality degrades with drift. Acknowledging drift as an important experimental aspect, our study takes it into account by testing in matched and drifted scenarios. Finally, [Shen al. (2018) described a multi-objective training of DNN in wind power prediction, minimizing two types of cost related to coverage and bandwidth. We expand on these metrics in Section 3.3
METHOD
3.1
META MODELING APPROACI
The basic concept of Meta Modeling (MM), depicted in Figure ! involves a combination of two models comprising a base model, performing the main task (e.g., regression), and a mela model, learning to predict the base model's error behavior. Depending on the amount of in- formation shared between these two, we distinguish several settings. namely (I) base model is a black-box (BB). (2) base is a white-box (WB, base parameters are accessible), and (3) base and meta components are trained jointly (JM). The advantages of WB and JM are obvious: rich information is available for the meta model to capture salient patterns for it to generate accu- rate predictions. On the other hand, the BB setting often occurs in practice and is a given.
We now formalize the MM concept it applies to sequential regression. Let & = Fa (x) be the base model function parametrized by Ø, where x = "1.,,... TN and ÿ v1..... dM represent sequences of N input feature vectors and M D- dimensional output vectors, with & E RD Let à (ý, x, Φ) denote the meta model, parameterized by 7, taking as input the pre- dictions, the original features, and the parameters of the base to produce a sequence of error predictions, 2 E DxM. The parameters & are obtained by solving optimization problem, arg ming (3, y)]. using smooth loss function 4, c.g., the Frobenius norm 4, 1| yl? Similarly, the parameters y are determined via arg min, 18[m, (2, z)] arg min,, (2, involving a loss la quantifying the target error (residual) from the base model, and l, quantifying the prediction error of the meta model. In general, 4, L22 and lm. may differ. The expectations are estimated using an available dataset. We used the L3P norm for I, and lm and L1 for lg, as described in Section 4 Given differentiable loss functions and the DNN setting, the base and the meta model can be integrated in a single network (JM). In this case the parameters are estimated jointly via
Mata Model
Outout by...
Input Ky...
Sequertial Base Model
Outgut
to
Figure 1: The concept of meta modeling
6" 2,"" arg min E5 y) + (1 4m (2, ($,y)))
4,y
2

---

Under review as a conference paper at ICLR 2021
whereby dedicated output nodes of the network generate Ye and 2,, and 3 is hhper--praaeee trading off the base with the meta loss. Thus, one part of the network tackles the base task, minimizing the base residual, while another models the residual the eventual measure of uncertainty. As done in (Kendall & Gal 2017). one can argue that the base objective minimizes the epistemic (parametric) uncertainty, while the meta objective captures the aleatoric uncertainty present in the data. Due their interaction, the base loss is influenced by the estimated uncertainty encouraging it to focus on feature-space regions with lower aleatoric uncertainty. Moreover, we conjecture, the DNN base model is encouraged to encode the input in ways suitable for uncertainty quantification.
Figure 2 shows an overview of a sequential DNN architecture applied throughout our study. includes a base encoder-decoder pair and a meta decoder connected to them. Each of these contains recurrent memory cell the LSTTM (Hochreiter & Schmidhuber 1997). The role of the encoder is to process the sequential input, compress its information in context vector and pass it to the base decoder. The recurrent decoder produces the regression output y in M time steps feeding its predictions input in the next time steps. Evolving in time, both base LSTMs update their internal states bg and h4, whereby the last state, by. serves as the context vector for the decoder. This architecture has gained wide popularity in applications such speech-to-text (Chiu et al. 2018 Tüske et al.] (2019), textehh-speech (Sotelo et al.] (017)). machine translation (Sutskever et al. 2014), and image captioning (Rennie 2016). Following the MM concept, attach additional decoder (the meta decoder) via connections to the encoder and decoder outputs. The context vector, by. is transformed by fully connected layer (FCN in Figure and both the ýt output as well the internal state, hr, are fed into the meta component. As mentioned above, the meta decoder generates uncertainty estimates, 2..
Given the architecture depicted in Figure we summarize the three settings as follows:(1) Joint Model (JM): parameters are trained according to Eq. ( with certain values of 3. (2) White-Bor Box model (WB): base parameters & are trained first, followed by parameters 7, also accessing 6. (3) Black-Boo Box model (BB): same as (2) without access to Φ.
Uncertainty
Pa Par
Deficit
Bandwidth
Excess
Base Output
9t.... 9w
0
Race Decoder
Embedaings
(featuees)
optionally,
Figure 3: Bandwidth, excess, and deficit costs.
Figure 2: Encoder- Decoder architecture integrat- ing a base and a meta model
Generating Symmetric and Asymmetric Bounds The choice of loss function, 1., gives rise to two scenarios. If 12 is an even function, e.g., 15. y) = ll - yylı- the meta-model targets z capture base error equally in both directions: above and below the target. Hence, the uncertainty 2 predicted at test time will represent a interval symmetric around ý. If, on the other hand, le takes the sign in into account, it is possible dedicate separate network nodes n, Yu E y capturing lower and upper band estimates, 2, and ă., respectively, thus accomplishing asymmetric prediction. Let 5 For the asymmetric scenario the meta objective is modified follows:
7" arg min ElIm (zi, max 03) + Im (Zu, max - 0})]
3.2 BASELINES
Implicit Heteroskedastic Variance Lakshminarayanan et al. (2017); Kendall & Gal (2017); Oh al. (2020) applied Gaussian model. N N(H, 0²) to the output of neural network predictor, where l
3

---

Under review as a conference paper at ICLR 2021
represents the prediction and o² its uncertainty due to observational (aleatoric) noise. The model trained to minimize the negative log-likelihood (NLL), with the variance being an implicit un- certainty parameter (in that it is trained indirectly) which is allowed to vary across the feature space (heteroskedasticttty) We apply the Gaussian in the sequential setting by planting it onto the base decoder's output (replacing the meta decoder) and train the network using the NLL objec- tive: o² = arg ming E.
- Ed= d=1
(0..--v.a)) log o with D output nodes modeling the oi
regression variable, St, and separate D output nodes modeling the log of: time
Variational Dropout Gal & Ghahramani (016)) established a connection between dropout (Srivas tava et al. 2014), i.e., i.e.,
process of randomly omitting network connections, and an approximate Bayesian inference. We apply the variational dropout method to the base encoder and decoder. By performing multiple runs per test sequence, each with a different random dropout pattern, the base predictions calculated the mean and the base uncertainty the variance over such runs. This Bayesian method, along with the variational approximation, captures the parametric (epistemic) uncertainty of the model, hence it fundamentally differs from the Gaussian model well as proposed approach.
GARCH Variance Introduced (Bollerslev 1986 Engle 1982). the Generalized Autoregressive
Conditional Heteroskedastic (GARCH) variance model belongs among the most popular statistical methods. A GARCH(p.q) assumes the series to follow an autoregressive moving average model and estimates the variance at time as a linear combination of past q residual terms, e², and previous variances, o²: of = dry Ei=1 a;d;_ Ei= B:07-
The og term represents constant component of the variance. The parameters, a, 8 are estimated via Beaximuu-likellhhon on a training set. The GARCH process relates to the concept in Figure iin that it acts as the meta-model predicting the squared residual. We use the GARCH as baseline only one of the datasets for reasons discussed Section
Constant-Bann Baseline A consistent comparison of uncertainty methods is difficult due the fact that each generates an uncertainty around different base predictions. Therefore, as reference we also generate constant symmetrie band around each base predictor. Such bound represents a homoskedastic process sensible choice in many well-behaved sequential regression problems, corresponding to a GARCH(0,0) model. We will use this reference point to compute a relative gain of each method explained in Section 3.3
3.3 EVALUATION METHODDLLGGY
Core Metrics Unlike with classification tasks, where standard calibration based metrics apply (Snoek al., (209)), we need to consider two aspects arising regression, roughly speaking: (1) what is the extent of observations falling outside the uncertainty bounds (Type cost), and (2) how excessive are the bounds (Type cost). An optimal bound captures
the observation while being least excessive in terms of its bandwidth. [Shen et al. (2018), among others, defined two measures reflecting these aspects (miss rate and bandwidth) which we adopt below (Eqs. (3) and (4)) while adding refinements (Eqs. (5) and (6)). Let 5 ý â, and ÿ" ý â, denote the predicted lower and upper bound, respectively. Recall that ÿ E DDM We define the following metrics:
Missrate ,
Σ
MD
d,t.yar Elo.
D M
Bandwidth($? (8/, S0
y) y)= 2MD
Ddr The
(4)
Excess(y) "8",
Σ
min Yos Waan lār - yas
(5)
MD
d. d, El
Deficit($', ," v)
1
Σ
min {laae Hael Iuve - Dia
(6)
MD
d,t: of Flo..
4

---

Under review as a conference paper at ICLR 2021
Table 1: Overview statistics of the MITV and the SPE9PR datasets
Dataset
Total
Input Features
Output
Time
Partition Size
Samples
Dim
Resol.
TRAIN
DEV
DEV2
TEST
MITV
48204
8
1
I hr
33744
4820
4820
4820
SPE9PR
28000
248
4
90 days
24000
1000
1000
1000
x100
X100
X100
x100
X x000
Figure iillustrates these metrics. The relative proportion of observations lying outside the bounds (miss rate) ignores the extent of the bound's short fall. The Deficit, Eq. (G). captures this. The type 2 cost is captured by the Bandwidth, Eq. { However, its range is indirectly compounded by the underlying variation y and y. Therefore we propose the Excess measure, Eq. ( which also reflects the Type 2 cost, but just the portion above the minimum bandwidth necessary to include the observation.
Calibration In general, DNNs offer few guarantees about the behavior of their output. DNNs tend to produce miscalibrated classification probabilities (Guo al. 2017). In order to evaluate the uncertainty across models, necessary to establish a common operating point (OP). We achieve this via scaling calibration. For symmetric bounds, assume that y e E IR. DxM is a random i.i.d. matrix and & is the predicted non-negative uncertainty band. Let Using a held-out dataset, we can obtain an empirical distribution in each output
& O where
Zae
Vdi Vdi-Bas
Zdt
d = 1.... D. It is then possible to find the value (()) for desired quantile p. e.g., ed (0.95) and construct the prediction bound test time: [Aa E Ddt Eace Assuming Zu is stationary, in expectation, this bound will contain the desired proportion p of the observations, i.e., [MMissraee . .y)
dimension: adf([Zaiiiese
This scaling is applied in our evaluation to compare the excess, bandwidth and deficit at fixed miss rates as well as setting minimum cost OP (see Section (.3). An Algorithm to find a scale factor for a desired value of any of the four metrics in ((M operations is given in the Appendix.
Metrics Used in Reporting The following OP-based measures are used in reporting: (1) Excess, Deficit, Bandwidth fixed Missrate, averaged over Missrate = {0.1, 0.05, 0.01 ). and (2) Minimum Excess-Deficit cost, where cost (Excess + Deficit) with the minimum found over all calibrations (OPs).
For each system and measure, ma symmetric constant-band baseline, fijedd is also generated and a relative gain with respect this reference calculated: gain, 100 x m/iand %. Finally,
miiied
the error rate of the base predictor is calculated as Epase (8.y) = 1 Eb 1l.. Ball, where dds Md d-th vectors of ÿ,
D Malll
EXPERIMENTS
Two sequential regression datasets, namely the Metro Interstate Traffic Volume (MITV) datase and the SPE9 Reservoir Production Rales (SPE9PR) datase were experimented with. Both originate from real-world applications, involve sequential input/output variables, and provide for scenarios with varying degrees of difficulty.
Datasets
The MITV dataset is a collection of hourly weather features with the target of the regression being the hourly traffic volume, recorded by the Minnesota DoT continuously between 2012 and 2018. The SPE9PR, on the other hand, is a large collection of mathematical simulations of a reservoir field with varying input and output sequences with each simulation comprising sequence with 100 time steps. The regression targets in this case are multivariate and correspond to field production rates.

---

Under review as a conference paper at ICLR 2021
The SPE9PR also contains data partition collected under distributional drift. Full detail on both datasets including their preprocessing can be found in the Appendix.
Training Procedure Each dataset was partitioned into TRAIN, DEV. DEV2, and TEST sets (with SPE9PR also providing a TEST-drift set), as listed Table While the TRAIN/DEV partitions served basic training, the DEV2 was used in determining hyperparameters and operating points (calibration). The TEST sets were used to produce the reported metrics. While a single sample in the SPE9PR represents a complete sequence of 100 steps, the MITV data come as a single contiguous sequence. The partitioning of the MITV set is strictly ordered by time, whereby the DEV sequence follows TRAIN, DEV2 follows DEV, and TEST follows DEV2. After partitioning, each MITV sequence was processed by a sliding window of length 36 hours (in I-hour steps). This resulted in a series of (n- 35) x 36 subsequences (n denotes the partition size) to feed the encoder-decoder model. When testing on the MITV, DNN predictions from such sliding windows were recombined into a contiguous prediction sequence again.
The base encoder-decoder network (see Figure 2 is trained using the Adam optimizer (Kingma & Ba 2014) with a varying initial learning rate, 1r, in two stages: (1) Training of all parameters using TRAIN while providing the ground truth as the decoder input at each time step. (2) Building on the previous, the training continues, however, decoder predictions from step 1 are fed as decoder inputs at step (-a mode referred to as emulation by [Bengio et al. (2015). All hyperparameter values listed in the Appendix.
JOINT MODEL, SYMMETRIC (JMS), AND ASYMMETRIC (JMA): The common training steps are performed using the objective in Eq. ( with 3 = 1.0, first. Then, the joint training continues with 3 == 0.5 as long as the objective improves on DEV. In a final step. the model switches to using DEV as training with 3 = 0.0 until no improvement on TRAIN is seen. A similar procedure is followed for the JMA, except using Eq. (2).
WHITE-BOX MODEL, SYMMETRIC (WBMS): The basic training is performed with 3 1.0. Next, only meta model parameters are estimated using the DEV/TRAIN sets with B 0.0.
BLACK-BOX MODEL, SYMMETRIC (BBMS): Base training is performed. The base model processes the DEV set to generate residual z. separate encoder-decoder model is then trained using (x, 2).
JOINT MODEL WITH VARIANCE (JMV): The two common steps are performed using the NLL objective (see Section 3.2) with the variance-related parameters first fixed, and, in subsequent step, allowing the variance parameters to be adjusted, until convergence. This is to aid stability in training (Nix & Weigend 1994).
DROPOUT MODEL SYMMETRIC (DOMS): Dropout with a rate of 0.25 (inputs, outputs) and 0.1 (internal states), determined as best performing on the DEV2 set, were applied in both the base encoder and the decoder. The two common steps were performed. At test time, the model was run 10 times per each test sequence to obtain the mean prediction and standard deviation.
The GARCH model from Section 3.2 is used with = = 5. The lag value was determined using an autocorrelation chart showing attenuation at lags > 5. We only apply this baseline to the MITV dataset as it provides a contiguous time series. The model parameters were trained using the DEV2 partition.
GARCH:
Throughout the experiments, the size of each LSTM cell was kept fixed at 32 for the base en- coder/decoder, and at 16 for the meta decoder. The base sizing has been largely driven by our preliminary study showing it suffices in providing accurate base predictions.
Testing Procedure As mentioned above the SPE9PR dataset has two TEST partitions: one for matched and one for a drifted condition. While the MITV dataset does not provide an explicit source of drift, we induce drift by creating a discrepancy in the modeling procedure between training and test: In the non-drift condition, the DNN's decoder is given access to the past 12 hours worth of traffic observations to make forecast for the next 24 hours. This is achieved by spanning 36-hour window and feeding the decoder inputs the first 12 hours of ground truth, during training. Now to create the drift scenario we test the MITV model without providing those first 12 hours of observations and the model uses their own predictions for that period instead. This emulates a "model drift" condition in that the model, trained to rely on actual observations, is getting its own noisy predictions.

---

Under review as a conference paper at ICLR 2021
Table 2:
Relative optimum and cross-validated gains (G', G*) using the Excess--Defict metrics.
Ebase denotes the base predictor's 's error. Within each column, elements marked are in statistical tie, all other values are mutually significant at p < 0.01
MITV
SPE9PR
System
match
Model Drift
Match
Data Drift
Ebase
%G
%0
Ebase
%G
%G
Ebase
%G
%G
Ebase
%6
%0
JMS
.151
57.81
59.3
.171
52.5
55.6
.165
46.5
46.6
.291
56.6
50.5
WBMS
.159t
53,8
55.31
.190
39.4
42.3
.162
44.8
44.9
.313
54.5
43.8
JMV
.167
20.4
25.5
.179
20.01
17.5
.159
45.3
45.5
.334
-6.5
4.4
DOMS
.144
13.2
12.8
.155
16.6
17.3
.177
1.3
1.4
.279
-0.2
-5.2
BBMS
.1531
-0.4
1.2
.1881
-8.9
-3.0
.170
31.9
30.3
.326
11.6
12.8
GARCH
.1551
-1.8
3.1
.171
-14.7
-4.0
n/a
n/a
n/a
n/a
n/a
n/a
Table 3: Relative optimum and cross-validated gains (G" and G*) on Bandwidth and Excess-Deffs metries for the asymmetric JMA model.
MITV
SPE9PR
Evaluation
Match
Drift
Match
Drift
JMA Base Error
.158
2000
.168
.320
Ex Crt-DeeiiiiBBaiin Bandwidthhhh
JMA, %G"
44.1
37.0
35.5
54.3
JMA, %G
33.1
26.4
35.5
8.5
JMS. %6"
37.5
31.6
29.1
34.3
JMS, %6
35.2
27.2
28.7
-4.4
JMA, %6
45.3
29.0
30.9
50.8
JMA, %G=
40.4
28.6
30.8
24.2
JMS, %G*
57.8
52.5
46.5
56.6
JMS, %G
59.3
55.6
46.6
50.5
Results with Symmetric Bounds Table[comparee the proposed symmetric- bouiii systems (JMS, WBMS, BBMS) with the baselines (JMV, DOMS, GARCH). The relative error of the base predictor is given in the Ebase column. The uncertainty quality is reported in Table is the average gain in excess-deficit metrics, as defined in Section 3.3 Columns labeled as contain measurements made an operating point (OP) determined on the test set itself, while those labeled G use an OP from a held-out (DEV2) set. While G reflects generalization the calibration, G* values are interesting they reveal the potential of each method. Based paired permutation test (Dwass 1957) all but entries marked with are mutually significant at p < 0.01.
From Table we make the following observations: (1) the JMS model dominates all other models across all conditions. The fact that it outperforms the WBMS indicates there is a benefit to the joint training setup, conjectured earlier. (2) The WBMS dramatically outperforms the BBMS model, which remains only as good as constant band for MITV data, indicating it is hard to reliably predict residuals from only the input features. (3) The most competitive baseline is the JMV model. As discussed in Section 3.2 the JMV shares some similarity with the meta-modeling approach. (4) The JMS and WBMS models perform particularly well in the strong drift scenario (SPE9PR), suggesting that white-box features play an essential role in achieving generalization. Finally, (5) the DOMS model works well on MITV data but provides no benefit in the SPE9PR, which could due the aleatoric uncertainty playing dominant role in this dataset. In almost all cases, however, the averaging of base predictions in DOMS results in lowest error rates of the base predictor.
Representative samples of JMS and JMV uncertainty bounds are shown in Figure (MITV) and Figure (SPEPPR). They illustrate a clear trend the results, namely that the JMS (also seen with WBMS) model are better able to cover the actual observation, particularly when the base prediction tends to make large errors. Additional plots can be found the Appendix, and also a notebook to visualize all test samples is provided part of the Supplementary Material.
Results with Asymmetric Bounds Generating asymmetric bounds is a new intriguing aspect of DNN-based meta-models. Using the JMA model, first recorded the accuracy with which the
7

---

Under review as a conference paper at ICLR 2021
Predicted Observed
JMS
6000
JMV
Traffic Volume
6000
Constant Bounds
6000
325
350
375
4001
425
450
475
500
525
Hour
Figure 4: Sample of traffic volume predictions with uncertainty generated by the JMS and JMV models, along with constant bound (around JMV) (miss rate set to 0.1 on TEST).
JMS
Water Production Rates (Barrels)
Oil Production Rates (Barrels)
JMV
JMV
Padittc Ossrrvd
Reckial
Constant Bounds
Constant Bounds
0
Time Period (90 days)
100
Time Period (90 days)
100
Figure 5: SPE9PR samples of oil (left) and water (right) production rates ("drift" scenario, miss rate set to 0.1 TEST).
Oil Production
7000
M
0000
5000
10110
Trifk
300
360
280
m/o
120
450
days)
Figure 6: Samples of asymmetric bounds produced by the JMA. MITV sample (left) and SPE9PR (right) correspond to segments shown in Figure 4 and
8

---

Under review as a conference paper at ICLR 2021
asymmetric output agrees in sign (orientation) with the observed base discrepancy. Averaged over each of the two datasets, this accuracy is at 83.3%, and 91.1%. The promise of asymmetric bounds lies in its potential to reduce the bandwidth cost. Since the Excess and Deficit metrics ignore the absolute bandwidth, we also evaluate the JMA model using the Bandwidth metric (Eq. (4), averaged over the same OPs. The results are shown in Table comparing the JMA model to the best symmetric model, JMS. The JMS model outperforms JMA in all scenarios on Excess-Defices however, compared on the bandwidth metric, the JMA dominates benefiting from its orientation capability. Upon visual inspection the output of the JMA is appreciably better in bandwidth: Figure o shows samples on both datasets. In most instances the bounds behave as expected, expending the bulk of bandwidth in the correct direction. An interesting question arises whether it is possible to utilize the asymmetric output as a correction on the base predictor. Our preliminary investigation shows that a naive combination leads to degradation in the base error, however, this question remains of interest for future work.
CONCLUSIONS
In this work we demonstrated that meta meta-modeling (MM) provides a powerful new framework for uncertainty prediction. Through a systematic evaluation of the proposed MM variants we report considerable relative gains over a constant reference baseline and show that they not only outperform all competitive baselines but also show stability across drift scenarios. A jointly trained model integrating the base with a meta component fares best, followed by a white-box setup, indicating that trainable white-box features play essential role in the task. Besides symmetric uncertainty, also investigated generating asymmetric bounds using dedicated network nodes and showed their benefit in reducing the uncertainty bandwidth. We believe these results open exciting new research avenue for uncertainty quantification in sequential regression.
REFERENCES
M. Amold, R. K. E. Bellamy, M. Hind, S. Houde, S. Mehta, A. Mojsilović, R. Nair, K. N. Ra- mamurthy, A. Olteanu, D. Piorkowski, D. Reimer, J. Richards, J. Tsay, and K. R. Varshney. Factsheets: Increasing trust in ai services through supplier's declarations of conformity. IBM Journal of Research and Development, 63(4/5):6:1-6:133 July 2019. ISSN 0018-8646. doi: N0.1147//DD.2019294225
E. Begoli, T. Bhattacharya, and D Kusnezov. The need for uncertainty quantification in machine- assisted medical decision making. Nature Mach Intell, 1:20-23, 2019.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference Neural Information Processing Systems Volume 1, NIPS15, pp. 11711179, Cambridge, MA. USA, 2015. MIT Press.
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3)307-321) April 1986.
Tongfei Chen, Jifí Navrátil, Vijay Iyengar, and Karthikeyan Shanmugam. Confidence scoring using whitebox meta-models with linear classifier probes. In The 22nd International Conference on Artificial Intellignnce and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, pp. 1467-1475, 2019.
C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, Nguyen, Chen, Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani. with sequence-to-sequeece models. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4774 April 2018
speech recognition
Meyer Dwass. Modified randomization tests for nonparametric hypotheses. Ann. Math. Statist., 28 (1):181-187, 03 1957. doi:
Robert Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica, 50(4):987-1007, 1982. ISSN 001 14680262.

---

Under review as a conference paper at ICLR 2021
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta- meta-leaming. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
1920-1930, Long Beach, California, USA, 09-15 Jun 2019. PMIR.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. D. Lee, M. Sugiyama. U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1019-1027. Curran Associates, Inc., 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings the 34th International Conference on Machine Learning Volume 70, ICML17, pp. 13211330. JMLR.org, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):17351788 November 1997. ISSN 0899-7667. doi: 10.1 162/neco. 199..9881733.
Heinrich Jiang. Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 5541-5552. Curran Associates, Inc., 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5574-5584. Curran Associates, Inc., 2017.
J.B. Killough. Ninth SPE comparative solution project: A reexamination of black-oil simulation. In SPE Reservoir Simulation Symposium. Society of Petroleum Engineers, 1995. doi: 10.2118/ 29110-ms.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. cite arxiv:1412. 1eeee411..990commee 6980C0mmeet: Published as conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6402-6413, Curran Associates, Inc., 2017.
2017. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ArXiv,
Terence C. Mills. Time Series Techniques for Economists. Number 9780521405744 in Cambridge Books. Cambridge University Press, 1991.
D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN 94), volume 1, pp. 55 60 vol.1, June 1994. doi: 10.1109//CNN..1944.74444
Min-hwan Oh, Peder A. Olsen, and Karthikeyan Natesan Ramamurthy. Crowd counting with decompased uncertainty. In AAAI Conference on Artificial Intelligence, 2020.
A. Papoulis and H. Saunders. Probability, Random Variables and Stochastic Processes (2nd Edition). Journal of Vibration, Acoustics, Stress, and Reliability in Design, 111(1):123-225, 01 1989. ISSN 0739-3717. doi: 10.1115/1.3269811
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. CoRR, abs/16 2.000563,, 2016.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. learning to learn: The peeta-meta-meta...r Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL http: : wwwww ds ia ch/ / schh-juergen/diii p omaa html
10

---

Under review as a conference paper at ICLR 2021
Yanxia Shen, Xu Wang, and Jie Chen. Wind power forecasting using multi-objective evolutionary algorithms for wavelet neural network cetwork-optimized prediction intervals. Applied Sciences, 8(2):185, Jan 2018. ISSN 2076-3417. doi: 10. 10.3390/app8020118.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13969-13980, 2019.
Jose Sotelo, Soroush Mehri, Kundan Kumar, João Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. In 5th International Conference Learning Representations, ICLR 2017, Toulon. France, April 24-26, 2017, Workshop Track Proceedings, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky. Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958 1958, 2014.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS14, pp. 31043112, Cambridge, MA, USA, 2014. MIT Press.
Zoltán Tüske, Kartik Audhkhasi, and George Saon. Advancing sequence-to-sequeee based speech recognition. Proc. Interspeech 20/9, pp. 3780-3784, 2019.

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Find best scale for a given metric value
Input: Observation, base and meta predictions {úr. Mr. 2/, "t hisesMi metric function f; target
value p*
Output: Scale factor E for +- to M
6. <-)
Vt-
for de 2 0
Et +
-b otherwise
for k
< to M do
0k < jk 2
y6 yk E
end for
(1hh the the h
Pt t
end for
arg min, loe -p'l
+ Ex-
A ALGORITHM TO FIND A SCALING FACTOR
Section 3.3 discusses the scaling calibration in the context of the four metries: Missrate, Bandwidth, Excess, and Deficit. Algorithm | finds scale factor for desired value of any of these four metrics in (M2) operations.
B ADDITIONAL DATASET AND IMPLEMENTATION DETAILS
B. DATASETS
B.1.1 METRO INTERSTATE TRAFFIC VOLUME (MITV)
The dataset is a collection of hourly westbound traffic volume measurements on Interstate 94 reported by the Minnesota DoT ATR station 301 between the years 2012 and 2018. These measurements are aligned with hourly weather feature as well as holiday information, also part of the dataset. The target of regression is the hourly traffic volume. This dataset was released in May, 2019.
The MITV input features were preprocessed to convert all categorical features to trainable vector embeddings, as outlined Figure 2 All real-valued features as well as the regression output were standardized before modeling (with the test predictions restored to their original range before calculating final metrics). Overall dataset statistics are listed in Table I and further processing steps given in Section
1.1.2 MITV PREPROCESSING
As described Section I.I.I and Table [ the MITV dataset comes with 8 input features, among which 3 are categorical. Here we list the relevant parsing and encoding steps used in our setup. The raw time stamp information was parsed to extract additional features such as day of the week, day of the month, year-day fraction, etc. Table 4 shows the corresponding list. Standardization was performed on the input as well as output, as per Table whereby the model predictions were transformed to their original range before calculating final metrics.
B.1.3 SPE9 RESERVOIR PRODUCTION RATES (SPE9PR)
This dataset originates from an application oil reservoir modeling. reservoir model (RM) is space-discretized approximation of geological region subject to modeling. Given a sequence of drilling actions (input), a physics-basee based PDE-solver (simulator) applied the RM to generate
provided by Weatherth
12

---

Under review as a conference paper at ICLR 2021
Table 4: MITV Input and Output Specifications
Feature
Range
Categorical Embedding Standardized
Final Dimension
Name
Dimension
INPUT
day-of-month
integer E [0, 30
Y
3
N
3
day-of_week
integer E [0, 6]
3
N
3
month
integer E [0, 11]
Y
3
N
3
frac.yday
real E 3365 1]
N
Y
1
weather.type
integer E [0, 10]
Y
3
N
3
Y
toliday.type
integer E [0, 11]
3
N
3
temperature rain 1h
real E R
N
Y
real E [R
Y
snow_lh
real E [R
Y
clouds
real E [0, 100]
N
Y
Total
20
OUTPUT
traffic.volume
real E R.,)
N
Y
1
Total
sequences of future production rates (oil, gas, water productio)),,, typically over long horizons Killough (1995). The objective is to train DNN and accurately predict outputs unseen input sequences. We used the publicly available SPEC RM, considered a reference for benchmarking reservoir simulation in the industry, and open-source simulato to produce 28,000 simulations, each with 100 randomized actions (varying type, location, and control parameters of well) inducing production rate sequences over span of 25 years, in 90-day increments, i.c., 100 time steps. Furthermore, the RM was partitioned into two regions, A and B. While most of the actions are located in the region A, we also generated 1000 sequences with actions located in the region B thus creating a large degree of mismatch between training and test. The test condition in region B will be referred to as "drift" scenario.
B.1.4 SPE9PR PREPROCESSING
The Table lists details on the SPE9PR features (also refer Section B.1.3 and Table IT. The SPE9PR dataset contains input sequences of actions and output sequences of production rates. An action (feature type of well 1), at partieular time, represents decision whether drill, and if so, what type of well to drill (an injector or producer well), or not to drill (encoded by "0"), hence the cardinality is 3. case drill decision, further specifications apply, namely the x- and y-location on the surface of the reservoir, local geological features at the site, and well control parameters. There are 15 vertical cells the SPE9 each coming with geological features (rel. permeability, rel. porosity, rock type), thus the local geology is a 45-dimensional feature vector at particular (z. v) location. Finally, every well drilled so far may be controlled by parameter called "Bottom-Hole Pressure" (BHP). Since we provision up 100 wells of each of the two types, 200-dimensions vector arises containing BHP values for these wells at any given time. Standardization was performed on the input well as output specified in Table 5 whereby the model predictions were transformed to their original range before calculating and reporting final metrics.
hhtp://gjttihkkoom///mmmmmomaaas bhtps://oomm-mijeerror//////
13

---

Under review as a conference paper at ICLR 2021
Table 5: SPE9PR Input and Output Specifications
Feature
Range
Categorical Embedding Standardized
Final Dimension
Name
Dimension
INPUT
type.of.well location_x
integer E {0, 1,2)
Y
3
N
3
integer [0, 24
10
N
10
location y
integer [0. 25
Y
10
10
vertical.geology per.well.control
real E R45
N
Y
45
real E 12000
Y
200
Total
258
OUTPUT
oil.prod.fieldrate gas prod field rate water.prolfield.rate waterinnj.fieldratee
real E Rg
Y
VYNIA
real E Rg
real Rg
real E R
N
Y
Total
4
Table 6: Hyperparameter settings
Hyper-
Where used
Value
Comment
parameter
Learning rate
all
0.001
Stage I training
Learning rate
all
0.0002
Stage training, see Section 4
Batch size
all
100
La penalty coefficient all, except DOMS
0.0001
La penalty coefficient
DOMS
0.0
Dropout
DOMS all meta models 1. 1.0/0.5/0.0
0.25/0.1/0.25 LSTM Input/Statc/Output (encoder and decoder)
Base LSTM size
32
Meta LSTM size
16
B in Eq.
see Section
B.2 TRAINING SETUP
B.2.1 HYPERPARAMETERS
Hyperparameters have been determined in two ways: (1) learning rate, regularization, batch, and LSTM size were adopted from an unrelated experimental study performed on a modified reservoir SPE9 (Anonymized), (2) We used DEV2 to determine the dropout rates in the DOMS model. The value 3 = 0.5 was chosen ad-hoc (as midpoint between pure base and pure meta loss) without further optimization.
B.3 IMPLEMENTATION NOTES
All DNNs were implemented in Tensorflow 1.11. Training done Tesla K80 GPU, with total training time ranging between 3 (MITV) and 24 (SPE9PR) hours. The GARCH Python implementation provided in the arch library was used.
14

---

Under review as a conference paper at ICLR 2021
Table 7: Relative optimum and cross-validated gains (G', Gaval) for the MITV dataset, using the Excess-Deficit metrics. Ebase denotes the base predictor's 's error. Within each column, elements marked are in a statistical tie, all other values are mutually significant at p < 0.01
MITV
System
match
drift
Ebase
%G
%G
Ebase
%G
%Caval
JMS
.1551
$7.8
59.31
.187
52.5
55.6
WBMS
.1591
53.8
55.3
.190
39.4
42.3
JMV
.167
20.4
25.5
.179
20.0
17.5t
DOMS
.144
13.2
12.8
.155
16.6
17.3
BBMS
.1531
-0.4
1.2
.1881
-8.9
-3.0
GARCH
.1551
-1.8
3.1
.187'
-14.7
-4.0
Table 8: Relative optimum and cross-validated gains (G", Gaval) for the SPE9PR dataset, using the Excess-Deficts metrics. Ebase denotes the base predictor's error. Within each column, elements marked are in a statistical tie, all other values are mutually significant at p < 0.01
SPE9PR
System
match
drift
Ebase
%G*
%rrval
Ebase
%G
%Gavvll
JMS
.165
46.5
46.6
.291
56.6
50.5
WBMS
.122
44.8
44.9
.313
54.5
43.8
JMV
.159
45.3
45.5
.334
-6.5
4.4
DOMS
.177
1.3
1.4
.279
-0.2
-5.2
BBMS
.170
31.9
30.3
.326
11.6
12.8
C ADDITIONAL RESULTS
C.I INDIVIDUAL METRICS
For more detailed view of the averages in Table 2 we show split by the individual metrics in Table 2 and, for SPE9PR which has total of four output variables, split by the individual variables
Table 10
C.2 ADDITIONAL VISUALIZATIONS
In addition to the sample visualizations shown in Section for the JMS, JMV, and JMA systems, here we show same sections of the data and visualize output of all systems. Figures] and show the first simulation in the test set of the SPE9PR dataset for the drift and non-drift condition and all its output components, respectively. Figures and I0 show the output on the MITV drift and non-drift condition, respectively. For each model, a miss rate value of 0.1 across the entire test set was used in the visualizations.
and
C.2.1 INTERACTIVE NOTEBOOK
We also provide interactive notebook that allows for inspecting all system output on an arbitrary portion of the test data in both the non-drift and drift condition. Please refer to the README file within the zip-file uploaded as the Supplementary Material part of our submission.
C.2.2 JMV VARIANCE TUNING ON DEV DATA
Section lists individual training steps for each system. is noted that the meta- meta- modeling arrange- ments have used the DEV partition for tuning final step. The motivation for using partition not included in training the base model is the avoidance of meta-training on biased targets, i.c., targets generated by the base model its training data. In this context, question arises whether sim-
15

---

Under review as a conference paper at ICLR 2021
BAMS
50000
10000
Water Production Rate
DOMS
25000
Oil Production Rate
Predicten
-20000
JMA
250000
40000
10000
JMS
JMS
40000
10000
20000
JMV
JMV
50000
25000
WEMS
40000
30000
20
60
Pertod (90-day)
BBMS
100000
Gas Production Rate
Water Injection Rate
DONS
Prodis.
25000
Observed
50000
50000
25000
100000
40000
20000
WDMS
50000
40000
-50000
20
65
100
Perfod (30-day)
Figure 7: Sample from SPE9PR (simulation 0, drift condition), all components and systems shown.
16

---

Under review as a conference paper at ICLR 2021
BOMS
10000
V00AA AA
5000
Oil Production Rate
10000
DOMS
Oiservel
Water Production Rate
30000
Predicted
Am
JMs
10000
50000
3402
JNV
10000
AA
5002
10000
5002
26000
300
Pariod
IMC stey)
DONS
40000
20003
Gas Production Rate
DOMS
Water Injection Rate
DOWS
25003
Osserved
JA8
JMS
40000
50000
JAV
40000
20003
WHRSS
WIBWS
500.0
40000
Period
Period (90-499)
Figure 8: Sample from SPE9PR (simulation 0, non-drift condition), all components and all systems shown.
17

---

Under review as a conference paper at ICLR 2021
JMS
Traffic Volume
5000
WBMS
Traffic Volume
5000
MV
Traffic Volume
5000
DOMS
Traffic Volume
5000
BBMS
Traffic Volume
5000
0
JMA
Traffic Volume
5000
0-
500
525
Hour
Figure 9: Sample from MITV (drift condition), all systems shown.
18

---

Under review as a conference paper at ICLR 2021
JMS
Trattic Volume
5000
WBM4S
Tratfic Volume
JMV
Trattic Volume
5000
DOMS
Trattic Volume
BBMS
Trattic Volume
5000
0-
1MA
Traftic Volume
5000
400
450
500
Hluur
Figure 10: Sample from MITV (non-drift condition), all systems shown.
19

---

Under review as a conference paper at ICLR 2021
ilar tuning step could help the JMV model. We followed the training steps described in Section 4 and then updated the network nodes tied to the variance parameter while keeping the rest of the network fixed. The Table shows the results the MITV dataset. It seems the benefit of the tuning step does not materialize. In all but the cross-validated drift case the gain decreases (albeit insignificantly) when applying the DEV-only tuning. We conjecture that the benefit of the tuning step exists with the meta-model because of the direct supervision of the meta-model's prediction. In contrast, the variance in the JMV setting is learned implicitly and may suffer from the "biased-target" problem mentioned above.
20

---

Under review as a conference paper at ICLR 2021
Table Symmetric gains split by individual metric (compare Table
Model
0ef1cit@0.01 Deficit@0.05 Deficit@0.1 Excess@ 0.01 Excess@0.05 Excess@O, MinCost Average
%G*, MITV "Non-Drift" Scenario
JMS
61.3
74.8
75.6
64.5
54.6
40.1
33.9
57.8
WBMS
48.0
73.8
74.3
62.7
50.7
35.1
32.3
53.8
JMV
-33.9
31.9
32.5
35,8
31.4
28.2
16.8
20.4
DOMS
28.1
24.8
23.1
11.5
3.3
-1.6
2.8
13.1
BBMS
9.7
27.9
16.3
-18.8
-18.7
-18.7
-0.8
-0.4
%G MITV "Non-Drift" Scenario
JMS
75.9
78.1
77.4
58.0
51.9
39.6
33.9
59.3
WBMS
67.4
79.6
77.7
57.7
42.7
29.8
32.3
55.3
JMV
4.4
38.9
35.8
29.1
27.1
26.3
16.8
25.5
DOMS
78.2
63.9
62.3
-35.7
-32,4
-49.2
2.8
12.8
BBMS
61.6
21.5
15.4
-61.0
-12.5
-15.9
-0.9
1.2
%G",
MITV "Drift" Scenario
JMS
40.5
78.0
77.5
57.7
45.0
31.9
36.7
52.5
WBMS
18.4
68.8
70.8
43.6
32.8
15.6
25.8
39.4
UMV
-3.8
29.0
30.9
23.0
25.5
21.6
13.8
20.0
DOMS
23.4
23.7
24.7
11.6
14.2
11.4
7.3
16.6
BBMS
-38.2
5.7
8.7
-1.88
-12.2
-14.1
-0.2
-8.9
%Caval MITV "Drift" Scenario
JMS
71.7
81.6
80.1
53.2
39.1
27.1
36.7
55.6
WBMS
67.9
79.2
78.5
22.9
19.4
2.0
25.8
42.3
JMV
-27.2
28.5
32.7
29.1
25.9
19.6
13.8
17.5
DOMS
38.4
18.9
23.4
2.8
16.6
13.3
7.4
17.3
BBMS
6.2
13.2
13.1
-17.5
-17.4
-18.6
-0.3
-3.0
%G", SPE9PR "Non-Drift" Scenario
JMS
55.7
55.8
56.8
46.1
42.5
37.7
30.8
46.5
WBMS
50.9
54.5
55.6
47.4
41.5
35.4
28.5
44.8
UMV
50.5
54.2
53.7
51.8
43.3
37.0
26.4
45.3
DOMS
7.1
10.2
10.4
-6.9
-5.8
-6.8
1.0
1.3
BBMS
51.0
52.6
50.0
25.1
18.2
10.6
15.7
31.9
%..vvll SPE9PR "Non-Drift" Scenario
JMS
56.8
56.5
57.0
45.7
42.0
37.4
30.8
46.6
WBMS
50.7
54.9
56.2
47.5
41.2
34.9
28.5
44.8
JMV
51.4
56.0
55.2
51,6
42.3
35.5
26.4
45.5
DOMS
7.7
9.5
10.1
-7.0
-5.3
-6.4
1.0
1.4
BBMS
76.4
61.2
54,8
-11.5
10.6
5.0
15.7
30.3
%G*. SPE9PR "Drift" Scenario
JMS
65.4
69.0
68.4
52.6
50.0
46.3
44.5
56.6
WBMS
63.7
68.0
66.9
53.5
48.1
41.8
39.5
54.5
JMV
29.2
22.7
21.9
-33.9
-42,8
-52.9
10.3
-6.5
DOMS
-1.4
9.4
12.0
-10.7
-4.5
-8.5
2.7
-0.1
BBMS
19.4
33.8
31.0
-7.6
1.1
-3.0
6.7
11.6
%Gavol
SPE9PR "Drift" Scenario
JMS
95.0
85.5
78.7
22.8
18.8
14.4
38.5
50.5
WBMS
94.7
84.1
77.3
16.8
5.0
-5.9
34.9
43.8
JMV
-40.0
-25.2
-12.2
48.3
31.6
16.9
11.3
4.4
DOMS
16.4
10.1
9.1
-23.4
-23.5
-26.9
1.8
-5.2
BBMS
40.5
9.4
7.9
0.8
17.3
9.0
5.0
12.8
21

---

Under review as a conference paper at ICLR 2021
Table 10: Symmetric gains split by individual components SPE9PR only (compare to Table 2).
OPR=Oil Production Rate, WPR=Water Production Rate, GPR=Gas Production Rate, WIN=Water
Injection Rate.
Ebase GPR
Excess-Deficit
Model
OPR
WPR
WIN
Average
OPR
WPR
GPR
WIN
Average
%G*,
SPE9PR "Non-Drift" Scenario
JMS
0.12
0.28
0.17
0.09
0.17
18.25
66.45
53.60
47.61
46.48
WBMS
0.12
0.28
0.17
0.09
0.16
17.78
62.08
52.43
47.03
44.83
JMV
0.10
0.29
0.16
0.09
0.16
6.67
74.50
43.74
56.24
45.29
DOMS
0.12
0.31
0.18
0.11
0.18
-6.88
0.68
21.64
-10.16
1.32
BBMS
0.12
0.30
0.16
0.10
0.17
-3.85
74.44
16.89
40.08
31.89
%G", SPE9PR "Non- Drift" Scenario
JMS
0.12
0.28
0.17
0.09
0.17
19.13
66.98
52.18
48.19
46.62
WBMS
0.12
0.28
0.17
0.09
0.16
18.32
62.60
51.07
47.39
44.85
JMV
0.10
0.29
0.16
0.09
0.16
6.92
75.74
43.12
56.23
45.50
DOMS
0.12
0.31
0.18
0.11
0.18
-6.56
0.46
21.94
-10.27
1.39
BBMS
0.12
0.30
0.16
0.10
0.17
-3.84
76.80
7.71
40.63
30.33
%6"
SPE9PR "Drift" Scenario
JMS
0.28
0.30
0.45
0.13
0.29
49.97
24.13
119.73
32.54
56.59
WBMS
0.31
0.34
0.45
0.14
0.31
51.49
27.09
102.64
36.77
54.50
JMV
0.30
0.32
0.51
0.20
0.33
1.12
-6.18
26.99
-47.96
-6.51
DOMS
0.28
0.28
0.45
0.11
0.28
-20.24
-6.97
22.32
4.31
-0.15
BBMS
0.32
0.31
0.54
0.13
0.33
-4.18
-1.04
26.64
25.13
11.64
%.avvll
SPE9PR "Drift" Scenario
JMS
0.28
0.30
0.45
0.13
0.29
61.81
39.10
74.23
27.01
50.54
WBMS
0.31
0.34
0.45
0.14
0.31
59.11
34.21
60.51
21.51
43.84
JMV
0.30
0.32
0.51
0.20
0.33
2.15
9.15
19.51
-13.28
4.38
DOMS
0.28
0.28
0.45
0.11
0,28
0.81
-27.32
15.73
-10.09
-5.22
BBMS
0.32
0.31
0.54
0.13
0.33
2.55
4.67
20.74
23.36
12.83
Table 11: Relative optimum and cross-validated gains using Excess-deficit metrics for the JMV system without (JMV) and with (JMV-a) variance tuning on DEV. Elements marked within same column are in statistical tie.
Evaluation
%6*
%G
JMV (Drift)
20.0
17.5
JMV-a (Drift)
15.0
19.4
JMV (Match)
20.4
25.5
JMV-a (Match)
16.2
16.9.
22


Reviews:
[{"review_id": "x07MLtZyDbg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview**\n\nThis paper applies \"meta-modeling\" to the time sequence regression task.\nThe idea is to meta-learn a base model that not only fits the labels, but is also easier for a meta-model to learn uncertainty quantification. In practice, the paper proposes to jointly train two models: one predicts labels, and another predicts the residuals (can be interpreted as uncertainty measurement). The two loss terms are $\\beta$ weighted and summed together, as in traditional multi-task learning setup. Several variances are evaluated on two datasets: MITV and SPE9PR.\n\n**Pros**\n\nThe authors proposed and studied 3 variances: 1) jointly train base and meta model's weights $\\phi,\\gamma$ with the regression target and the residual term; 2, 3) sequentially train base and meta model's $\\phi,\\gamma$ with/without parameter access of the base model's $\\phi$.\nThe evaluation results on MITV and SPE9PR suggests that Jointly training approach out-performs other variances and is more robust in the presence of dataset and model drift.  The evaluation metrics consists of Missrate, Bandwidth, Excess and Deficit (Fig 3), which are well designed for the sequence regression and uncertainty prediction task.\n\n**Cons**\n\nThe main idea of the paper is to utilize meta-modeling to jointly train a meta model to estimate the error behavior of the base model.  However \"meta-modeling\" is not a new concept, and it is usually referred to as meta-learning and learning-to-learn in the literature.  The later normally requires compute the gradient of gradient (or with its first-order approximation, like MAML and Reptile) or directly predicting weights of a neural network. The proposed approach doesn't really fit into this category, as  the weight of base and meta model $\\phi,\\gamma$ are separately learnt in a multi-task fashion (e.g., there are 2 tasks in this setup: one is the regression task for the base model and the other is the residual prediction task for the meta model).\n\nI also think predicting the asymmetric residual bound loss term (eq 2) might confuse the meta-model. Would meta-model's predicted upper bound still be meaningful if it thinks the prediction is less than the regression target?\n\nThe two evaluation datasets are on limited domains. The second one is from a physics-based simulator instead of from real-world data collection. Have the author consider doing evaluation on a more broader domains to demonstrate the proposed approach can bring consistent gains? \n\nLastly, since the drift are of different nature for the two eval dataset (data drift vs model drift), perhaps worth using different names in Table 2 (data drift for SPE9PR and model drift for MITV) \n \n\n**Reason for the decision**\n\nThis paper attempts to introduce meta-learning to the sequence regression and uncertainty prediction task. The evaluation results are encouraging among a few variances and baselines, and the metrics are well thought of.  However the main reason for me to give marginally reject is that: 1. the approach doesn't really fit into the meta-learning category and 2. the two evaluation datasets are on somewhat limited domains. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "uIeQv8A4dHO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors present a meta-modeling approach to provide predictions with uncertainty estimates in a sequential task. They develop a white box, black box, and joint modeling method that allows them to apply their method to a variety of scenarios. These methods differ based on the amount of information provided to a meta-learner which has the goal of predicting errors $\\hat{z} \\in \\mathbb{R}^{D \\times M} $. The authors also incorporate the ability to make asymmetric uncertainty bounds. They apply this method and many baeslines to two datasets: MITV and SPE9PR.\n\nStrengths: \n* Strong presentation and emphasis of the different components of the white box, black box, and joint methods\n* Good spectrum of baselines \n* Comparison on drifting data\n\nWeaknesses: \n* Many of the figures seem compressed by the $\\LaTeX $ -- consider remaking your figures with the appropriate aspect ratio \n* The tables seem like they could refined a little more (i.e. Table 3) -- I think I may have missed the definition of \"xval\" somewhere in the work and \"match\" and \"drift\" could be capitalized. \n* Could you address how this work could be applied to other methods such as RNN's? \n\nUltimately, I think this is a strong work that establishes an exciting method for uncertainty estimation. Because of it's strong presentation, novelty, and experiments, I rate this as a clear accept. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "rm1bVAYFuPs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to generate symmetric and asymmetric uncertainty estimates. The method is proposed to work for the non-stationarity processes found in real-world applications. The paper introduces a meta-modelling concept as an approach to achieve high-quality uncertainty quantification in deep neural networks for sequential regression tasks. The paper also introduces metrics for evaluating the proposed approach. A proposed meta-modelling approach is related to the work of Chen et al. (2019) which is mainly used for classification task in a non-sequential setting, however, the proposed method is mainly for the sequential setting. \n\nThe paper has interesting explorations for handling uncertainty and its evaluation. However, the paper needs more work for clarity and rigorous analysis. For example, equation 2 is not clear to me and needs explanation and/or related citations. For instance, why even function reflects symmetric uncertainty and how separating network nodes produces lower and upper band estimates to accomplish asymmetric prediction.\n\nThe evaluation metrics were introduced (Eq: 3,4,5,6) to measure the effectiveness of the proposed method. Is there any existing evaluation metric for symmetric and asymmetric uncertainty estimates that can be used for measuring the effectiveness of the proposed method?\n\nThere are two proposals in this paper: one is a meta-modelling concept for uncertainty measurement and another is the evaluation metrics to evaluate the uncertainty. It will be better to separate them, the first part needs to be evaluated with exiting uncertainty evaluation metrics with some well-known benchmark datasets. This part is to measure the effectiveness of the meta-modelling concept using the existing metric. The second part can be for proposing new evaluation metric and justify the reason behind the proposed the new metrics.\n\nTwo real datasets was used for experimentation which is a good idea for evaluating on real applications. For the purpose of evaluating the proposed systems, it is also necessary to utilize some benchmark datasets from the literature.\n\nMinor comment: In figure 1, change the subscript N to M for consistency with the text.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nTy-ImkUITk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a technique for uncertainty estimation in regression with neural networks. The basic idea is to use an auxiliary \"meta model\" that (in the authors' best performing setting) has access to the base model and is trained jointly with it. The purpose of the meta model is to predict the error characteristics of the base model, which of course naturally leads to error bars. In order to account for the fact that the models errors on the train set are unlikely to be representative, the authors make use of a validation set on which the meta model is trained further. \n\nThe general idea of using a meta model to predict the error characteristics of a base model potentially has some advantages over existing approaches, including a fairly easy approach to modelling asymmetric error bars and only requiring one forward pass through the full model at test time. I do have some concerns about the experimental evaluation. \n\nA critical gap here is a lack of comparison to a relatively large literature on *explicit* ensembling and (recently) efficient explicit ensembling techniques (e.g., BatchEnsemble) for uncertainty calibration that have worked very well across a variety of complex domains including computer vision, natural language processing and sequence modelling, and model based reinforcement learning. For purely neural network based approaches, many would argue that these ensembling techniques represent the current state of the art.\n\nSome details of the experimental setup were a little unclear. Since the datasets hard a time component, how were DEV and DEV2 sets chosen relative to test? Can the authors confirm that the TRAIN set always occurred strictly before the DEV and TEST sets in time, and that the DEV sets occurred strictly before the TEST set in time? To be frank, some of the uncertainty intervals in 1D are shockingly good if this is pure extrapolation, with the uncertainty estimates seeming to always vary almost perfectly with the true error. If this were a consistent feature of the model, particularly with asymmetric uncertainty outputs, I would imagine it would be straightforward to dramatically improve mean predictive performance as well using information from the meta model, which seems surprising.\n\nHow much of the benefit is due to the additional tuning on a dev set? For example, suppose you tuned a standard heteroskedastic NN's variance outputs on a dev set? Why the decision to focus specifically on sequence modelling? I think the inclusion of WBMS and BBMS as training settings is largely unnecessary, to be frank, in particular the inclusion of BBMS. I think it would be very rare that we would need to augment a blackbox model with uncertainty estimates rather than having the ability to train from scratch to produce a well calibrated model.\n\nI am a little surprised to see the invention of new calibration metrics for regression but not the inclusion of existing metrics for this that have been used in statistics for a long time. Metrics like continuous ranked probability scores do not assume a particular distributional form of output and are well studied and likely to be highly correlated with some of these more slightly ad hoc metrics. The bandwidth \"metric\" is also a little strange as a metric, as it doesn't depend on the true label y and rather just encourages confidence generally.\n\nOn a minor note, most of the plots in the paper suffer from being scaled manually after being generated, which has lead to all of the text being unnaturally stretched.\n\n--------\nAfter author feedback, I feel that the authors did address a number of my points, including one or two that were indeed addressed in the text that I must have simply missed. Therefore, I am improving my score.\n\nThe one point if any that I continue to disagree with the authors on is that variational dropout is sufficient to cover the space of explicit ensembling methods. Many of these \"cost effective\" ensembling approaches still continue to compare to direct explicit ensembling with the very explicit goal of having a baseline that is known to work well but removed from the computational cost concerns. I feel that this comparison should be standard practice.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. Initially, it received  weaker reviews (than the current ones) and after rebuttal, one reviewer slightly increased the review rating.  Among 4 reviews, there is one clearly positive one.

Among negative reviews, there are repeated concerns about evaluation. While recognizing the difficulty of finding (large-scale) datasets for sequential  regression tasks,  reviewers suggest running the methods on benchmark datasets. One reviewer also points out the need to compare to efficient ensemble methods for uncertainty quantification.

The AC also read  the manuscript. The AC appreciates the strong performance of the proposed method. However, given the limited number of datasets this method is evaluated, and the lacking of analysis of understanding why the method is successful or could fail, the AC recommends Reject.  The author(s) could improve the manuscript by following the suggestion of the reviewers, as well as a more detailed analysis of how robust the proposed method is: there are many parameters tuned in the empirical results and it is not clear why sensitive those parameters are. 
Decision: Reject