Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel
Authors: Xin Qiu
Elliot Meyerson
Risto Miikkulainen
Venue:  (2020)
Keywords: Uncertainty Estimation
Neural Networks
Gaussian Process

Abstract: Neural Networks (NNs) have been extensively used for a wide spectrum of real-world regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, a new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with a Gaussian Process, whose kernel includes both the NN's input and its output. The framework is justified theoretically and evaluated in twelve real-world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied to any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.

Published as a conference paper at ICLR 2020
QUANTIFYING POINT-PREDICTION UNCERTAINTY IN NEURAL NETWORKS VIA RESIDUAL ESTIMATION WITH AN I/O KERNEL
Xin Qiu
Elliot Meyerson
Cognizant
Cognizant
qiuxin.njus mtw.:iinnju@mmiil1. gmail com
elliot. elliot. www..eeeeeesssoennieiaaooo
Risto Miikkulainen
Cognizant
The University of Texas at Austin Eisto@cognizant com
ABSTRACT
Neural Networks (NNs) have been extensively used for a wide spectrum of real- world regression tasks, where the goal is to predict numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with Gaussian Process, whose kernel includes both the NN's input and its output. The framework evaluated in twelve real- world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.
INTRODUCTION
Nowadays, Neural Networks (NNs) are arguably the most popular machine learning tool among Artificial Intelligence (AT) community. Researchers and practitioners have applied NNs wide variety of fields, including manufacturing (Bergmann et al. 22044), bioinformatics (LeCun et al. 2015), physics (Baldi et al. (2014). finance (Niaki & Hoseinzade 2013). chemistry (Anjos al. 2015). healthcare (Shahid et al. 2019). etc. Although standard NNs are good at making a point prediction (a single outcome) for supervised learning tasks, they are unable to provide uncertainty information about predictions. For real-world decision makers, representing prediction uncertainty is of crucial importance (Krzywinski Altman 2013; Ghahramami 2015). For example, in the case of regression, providing 95% confidence interval around the prediction allows the decision maker to anticipate possible outcomes with explicit probability. In contrast, simply returning a single point prediction imposes increased risks decision making, e.g., predictively good but actually risky medical treatment may be overconfidently interpreted without uncertainty information.
Conventional Bayesian models such Gaussian Processes (GP) (Rasmussen & Williams 2006 offer a mathematically grounded approach to reason about the predictive uncertainty, but often come with a prohibitive computational cost and lower prediction performance compared to NNs (Gal & Ghahramani 2016). As a potential solution, considerable research has been devoted to the combination of Bayesian models and NNs (see Section 2 for detailed review of such approaches),

---

Published as a conference paper at ICLR 2020
aiming overcome the downsides of each. However, from the classical Bayesian Neural Network (Neal 1996). in which a distribution of weights is learned, to the recent Neural Processes (Garnelo et al. 2018a b Kim et al. 2019). in which distribution over functions is defined, all such methods require significant modifications to the model infrastructure and training pipeline. Compared to standard Mion-Bayesian) NNs, these new models are often computationally slower train and harder to implement (Gal & Ghahramani] (0166; Lakshminarayanan et al. 2017), creating tremendous difficulty for practical uses. Gal & Ghahramani| (2016) derived a theoretical tool to extract uncertainty information from dropout training, however, the method can only be applied to dropout models, and requires changes to their internal inference pipeline. Quantifying
uncertainty in standard NNs, which are overwhelmingly popular in practical applications, still remains challenging problem with significant potential impact.
To circumvent the above issues, this paper presents a new framework that can quantitatively estimate the point-prediction uncertainty of standard NNs without any modifications to the model structure or training pipeline. The proposed approach works supporting tool that can augment any pretrained NN without retraining them. The idea is to capture the behavior of the NN by estimating its prediction residuals using a modified GP, which uses new composite (I/O) kernel that makes use of both inputs and outputs of the NNs. The framework is referred to as RIO (for Residual estimation with an 1/O kernel). In addition to providing valuable uncertainty estimation, RIO has an interesting side-effect: It provides a way to reduce the error of the NN predictions. Moreover, with the help of recent sparse GP models, RIO scales well to large datasets. Since classification problems can be treated as regression on class labels (Lee et al. 2018), this paper focuses on regression tasks. In this setting, empirical studies are conducted with twelve real-world datasets, and an initial theoretical investigation is presented that characterizes the benefits of RIO. The results show that RIO exhibits reliable uncertainty estimations, more accurate point predictions, and better scalability compared to alternative approaches.
2 RELATED WORK
There has been significant interest in combining NNs with probabilistic Bayesian models. An early approach was Bayesian Neural Networks, in which a prior distribution is defined on the weights and biases of NN, and posterior distribution is then inferred from the training data (MacKay) 1992]
Neal 1996). Notice that unlike these methods, RIO is concerned only with uncertainty over NN predictions, not over NN weights. Traditional variational inference techniques have been applied to the learning procedure of Bayesian NN, but with limited success (Hinton & van Camp 1993
Barber & Bishop 1998 Graves 201T). By using a more advanced variational inference method, new approximations for Bayesian NNs were achieved that provided similar prediction performance as dropout NNs (Blundell et al. 2015). However, the main drawbacks of Bayesian NNs remain: prohibitive computational cost and difficult implementation procedure compared to standard NNs.
Alternatives to Bayesian NNs have been developed recently. One such approach introduces a training pipeline that incorporates ensembles of NNs and adversarial training (Lakshminarayanaan et al. 2017).
Another approach, NNGP, considers a theoretical connection between NNs and GP to develop a model approximating the Bayesian inference process of wide deep neural networks (Lee et al. 2018). 1).. Deep kernel learning (DKL) combines NNs with GP by using a deep NN embedding as the input to the GP kernel (Wilson et al., 2016). In Iwata & Ghahramani (017)), NNs are used for the mean functions of GPs, and parameters of both NNs and GP kernels are simultaneously estimated by stochastic gradient descent methods. Conditional Neural Processes (CNPs) combine the benefits of NNs and GP, by defining conditional distributions over functions given data, and parameterizing this dependence with a NN (Garnelo et al. 2018a). Neural Processes (NPs) generalize deterministic CNPs by incorporating a latent variable, strengthening the connection to approximate Bayesian and latent variable approaches (Garnelo et al., 2018b). Attentive Neural Processes (ANPs) further extends NPs by incorporating attention to overcome underfitting issues (Kim et al.) 2019). The above models all require significant modifications to the original NN model and training pipeline. Compared to standard NNs, they are also less computationally efficient and more difficult for practitioners to implement. In the approach that shares the most motivation with RIO, Monte Carlo dropout was used to estimate the predictive uncertainty of dropout NNs GGal & Ghahramanil 2016). However, this method is restricted to dropout NNs, and also requires modifications to the NN inference process. Different from all above mentioned methods, which are independent alternatives to standard NNs,
2

---

Published as a conference paper at ICLR 2020
Data
Data Scientist
Uncertainty Estimation and Error Correction
Deployment
prediction
X'
NN
GP
uncertainty
Figure 1: Complete model-building process. Given dataset, first standard NN model is con- structed and trained by a data scientist. The RIO method takes this pretrained model and trains GP estimate the residuals of the NN using both the output of the NN and the original input. Blue pathways are only active during the training phase. In the deployment phase, the GP provides uncertainty estimates for the predictions, while calibrating them, i.c., making point predictions more accurate. Overall, RIO transforms the standard NN regression model into a more practical probabilistic estimator.
RIO is designed as a supporting tool that can be applied on top of any pretrained NN. RIO augments standard NNs without retraining or modifying any component them.
3 THE RIO FRAMEWORK
This section gives the general problem statement, develops the RIO framework, and discusses its scalability. For background introductions of NNs, GP, and its more efficient approximation, SVGP (Hensman et al. 2013 2015), see Appendix B
3.1 PROBLEM STATEMENT
Consider a training dataset D = x ,)) point prediction ú given x,. The problem two-fold: (1) Quantify the uncertainty the predictions of the NN, and (2) calibrate the point predictions of the NN (i.e. make them more accurate).
and pretrained standard NN that outputs
3.2 FRAMEWORK OVERVIEW
RIO solves this problem by modeling the residuals between observed outcomes y and NN predictions ý using GP with composite kernel. The framework can be divided into two phases: the training phase and the deployment phase.
In the training phase, the residuals between observed outcomes and NN predictions on the training dataset are calculated as
= 34 0c. for = 1,2......
(I)
Let r denote the vector of all residuals and y denote the vector of all NN predictions. A GP with com- posite kernel is trained assuming NO, ((((,, denotes X covariance matrix all pairs of training points based composite kernel
((((,))) $), (X, f)) + IfI), where K(((X, $). (X,$))
(xx;, (e), (x,, ú;)) kim (x,, xj) + Aout (out(Ue J5), for i,j 1,2,...
Suppose a radial basis function (RBF) kernel is used for both kin and kout. Then,
ke ūc), (x)) = - exp((
21 ||x
x1l13) + exp( 1li. 11113)
(3)
212 The training process of GP learns the hyperparameters of lins domr louts and o by maximizing the log marginal likelihood log p(rl2'')) given by
log 2m log (() = -r ((X, 9), (X. Dr log Ka((K, 8), (2,, (X, +
(4)
3

---

Published as a conference paper at ICLR 2020
Ruitaat
Figure 2: Capturing uncertainty of more and less accurate NNs. These figures illustrate the
behavior of RIO in cases: (left) The neural network has discovered true complex structure in the labels, so the residuals have low variance and are easy for the GP to fit with high confidence; (right) The ineffective neural network has introduced unnecessary complexity, the residuals modeled with high uncertainty. In both cases, RIO matches the intuition for how uncertain the NN really is.
In the deployment phase, test point is input to the NN to get output g.. The trained GP predicts the distribution of the residual as X 9, r, Xx, 9- ~ N (f.,, var(f-)), where
7. = k. ((K ()) -
-r
(5)
covariances (i.e., kio (xxx, 9-), (Xi, 0:))) between (x+, U-) var(f.) = ke (e(( i.); (x., ù.)) - k. (K (x) (2, + T
(6)
where k, denotes the vector of and the training points.
Interestingly, note that the predicted residuals can also be used to calibrate the point predictions of the NN, so that the final calibrated prediction with uncertainty information is given by
d' N(. + P., var(f.))
In other words, RIO not only adds uncertainty estimation standard NN-it also provides way to
calibrate NN predictions, without any modification to its architecture training. Figure [ showw the overall procedure when applying the proposed framework in real-world applications. shows example behavior of RIO that illustrates the intuition of the approach. An algorithmic description of RIO is provided in Appendix C
3.3 SCALABILITY
RIO is scalable large datasets (No. of data pointsx No. of features:> 1M) by applying sparse GP methods, e.g., SVGP (Hensman et al., 2013 2015). All the conclusions in previous sections still remain valid since sparse GP simply an approximation of the original GP. In the case of applying SVGP with a traditional optimizer, e.g., L-BFGS-B (Byrd et al. 1995 Zhu et al. 1997), the computational complexity Onm?) )) and space complexity is (mm) where n is the number of data points and m is the number of inducing variables, compared to (n²)) and ((n))
)) for traditional
GP. Experiments verify that the computational cost of RIO with SVGP significantly cheaper than other state-of-the -he----rtttttttt approaches.
4 EMPIRICAL EVALUATION
Experiments in this section compare nine algorithms on 12 real-world datasets. The algorithms include standard NN, the proposed RIO framework, four ablated variants RIO. and three state-of- the-art models that provide predictive uncertainty: SVGP (Hensman et al.] 2013). Neural Network Gaussian Process (NNGP) (Lee et al. 2018), and Attentive Neural Processes (ANP) (Km et al. 2019). In naming the RIO variants," "R" means estimating NN residuals then correcting NN outputs," "Y" "Y"" means directly estimating outcomes, "T" means only using input kernel, "O" means only using output kernel, and "IO" means using I/O kernel. For all RIO variants (including full RIO), SVGP is used as the GP component, but using the appropriate kernel and prediction target. Therefore. "Y+I" amounts to original SVGP, and it is denoted "SVGP" in all the experimental results. All 12 datasets
4

---

Published as a conference paper at ICLR 2020
Table 1: Summary of experimental results
Dutaset
Method
sean
RMSE
NLPD sod
Nuise Variance
(100)
Time
Duitact
Methol
mesn-tsd
RMSE
sen
NLPD
Varimee
Noise
Lime
yachr
RIO
NN
2
1.49
2.009:10.662
0.82
7.16
4.02
ENB/h
RIO
NN
0.70
1.03810,385
0.46
8.18
R+]
2.
1.5111
2.54
4.30
R-1
0.79:0.0611
1.347:0.0051
0.63
7.52
308
66111
1.3022061411
6.27
0.
1.16010 1581
X
Y+0
1.86
2.305
0.6391
1.89
9.93
Y+O
0.88:
1.248:10.405
0.75
11.06
Y-10
5.5
0,331
1.18
9.64
Y+10
0.76:00:11
1.124 368
0.56
10.64
SVGP
2.888200 10211
18.56
8.96
SVGP
2.13 1811
2.200:1:0
4,70
10.16
1..40:::1451
NNGP
0.27:
52,40:100660
ANP
7.591.3.2011
0.787:0088871
40.82
ANP
4.08:2.27:1
2.475 539
102.3
ENBI
NN
1.111:044411
0.38
1.816:-0.111
1:58
8:477
6.45
airfoil
NN
482200431
3077-..18
1.554
6.48
1.71 1.71404441
1999==0226f
2.22
5.02
R+1
3.16=001881
2.883+00051
19.07
768
R4D
1.75
0.2991
2.25
4.57
1,06
R4O
4.17-02611
2.849 06611
16.64
9.97
Y40
1.76204351
2
2.32
10.99
Y+0
4.24=02811
17.81
22.72
Y-10
1661100361
199620.21011
10.56
Y+10
364-00511
2.1210 1501
24.51
SVGP
2.634022111
2:40==007811
6.11
10.28
SVGP
3:59=02011
12,67
21,74
NNGP
4.91 4.111
30.14:088611
7704
NNGP
6.54202811
55600-020077
3355
ANP
4511
26982005-811
64.11
ANP
31
13902631611
231.7
OCS
RID
0.97
0.10
24.74
15:71
9,50
winetr
RIO
00
0.6724-0066
1094+00.100
0.25
9:25
1030
R+i
6.01
+0.501
0.112
25.40
9.52
1599
R+1
0.69-000311
1085 +00971
0.28
8.34
R+0
6.17110.5411
0.283
26.31
9.54
R+0
0.0551
1.09
0.29
5.02
Y1O
6.15+
3279-001177
26:53
21-35
YiO
0676=00044
1096+0.0
12.71
Y-10
6.06-009911
00.011
25.82
23.15
Y+10
0.72--00311
0.28
12.48
SVGP
0.391
1.336110.0481
44,55
19.85
SVGP
0.02-0000811
0.7411002111
0.40
12.17
NN
0.00111
7.17
4.96-0.3311
winetw
RID
0.704:20.018
1000
0.37
16.74
RIO
10.1
2818=1
16.30
42.65
1898
R+]
0.699
1
0.0371
0.38
13.5
9568
R-1
4.06 1.134
0:022100000
16.39
39.88
X
R+O
0.7101:00019:
0.3811
0.39
6.19
R+0
1.32=0.151
0.8831000511
18.50
18.48
0.719
0.962
18.30
0.37-0.2011
2.9420
Y-10
1
0.0001100000
0.38
20.06
Y+10
1.56:11000
31,06
46.8
SVGP
0
1 0.0222
0.50
18.18
SVGP
1.36
2:5011000111
19:04
46.43
protein
RID
NN
4.11::00011
0.08::0.06
2886-100004
15.71
151.
149.4
RIO
NN
02.25
3.53--0.422
105.83
53.39
45730
R+]
4.11 +:0001
0.047
15.99
141.2
21263
R+1
11.33-
107.35
47.72
R4O
1.14
0.151
16.18
115.1
R40
11 05.63=0521
5581
112.91
30.47
4.14
0.0151
16.17
138.4
YaO
04661
Y-10
2.826--0.014
15:22
155.5
Y+O
11.32-0.4511
3236--004111
106-93
57.74
SVGP
2967-20.0071
22.54
149.5
SVGP
0.251
4:1365:001111
239.28
50.89
NN
1.1120.3411
194.5
XN
12772:29715
1136
CT
0.88
122441-02119
516.4
MSD
9.26
3.639-0.022
53500
R+]
1.1710031411
1538:10028911
1.71
19.80
515345
R+1
10.92 13002
38111001281
135.34
282.0
X
R+O
0.88
1.02
159.4
R40
9.25
9.68:100021
84.05
1518
384
Y+0
0.9211
1.36
2.45
168.2
Y+O
10.00
1.118
169.90
1080
09011011611
127010.1771
0.62
578.6
9.43110521
36411002511
SVGP
52,00 +-0.191
5.372-00.00443
2712
27.56
SVGP
9,57=000015
3.677
92.21
2276
The symbols 1 and $ indicate that the difference between the marked entry and KIO statstically signdhcant at the 58 significance lovel using paired E-test and Wilcoson test, respoctivell. The hest entries that are significantly better than all the others under ot lenet one statistical test are mar marked in holdfacn (tics are allowwd).
are real-world regression problems (Dua& & Graff 2017), and cover a wide variety of dataset sizes and feature dimensionalities. Except for the "MSD" dataset, all other datasets are tested for 100 independent runs. During each run, the dataset is randomly split into training set, validation set, and test set, and all algorithms are trained on the same split. All RIO variants that involve output kernel or residual estimation are based on the trained NN in the same run, For "MSD", since the dataset split is strictly predefined by the provider, only 10 independent runs conducted. NNGP and ANP are only tested on the four smallest dataset (based on the product of dataset size and feature dimensionality) because they do not scale well to larger datasets. It is notable that for all the RIO variants, no extensive hyperparameter tuning is conducted; the same default setup is used for all experiments, i.e., standard RBF kernel and 50 inducing points. See Appendix D.I for additional details of experimental setups and a link to downloadable source code. Table I numerical results from these experiments. The main conclusions in terms of point-prediction error, uncertainty estimation, computational requirements, and ablation studies are summarized below.
summarizes the
Doint-Prediction Error The errors between point predictions of models and true outcomes of test points are measured using Root Mean Square Error (RMSE); the mean and standard deviation of RMSEs over multiple experimental runs are shown in Table For models that return a probabilistic distribution, the mean of the distribution is the point prediction. Although the main motivation of RIO is to enhance pretrained NN rather than construct a new state-of-the-a prediction model from scratch, RIO performs the best equals the best method (based on statistical tests) in 10 out of I2 datasets. RIO significantly outperforms original NN in all 12 datasets, while original SVGP performs significantly worse than NN in 7 datasets. For the CT dataset, which has 386 input features, SVGP fails severely since the input kernel cannot capture the implicit correlation information. ANP is unstable the airfoil dataset because it scales poorly with dataset size. Figure 3 compares NN RIO
5

---

Published as a conference paper at ICLR 2020
yacht
ENB/h
ENB/c
airfoll
CCS
wine/r
3.0
6.75
2.
5.70
2.
8.65
30
8:60
wine/w
CCPP
protein
SC
CT
MSD
RI00000 RMMS
9.776
9.0
0.73
9.
0.0
9.66
24 3 1-
NN RMSE
NN RMSE
NN RMSE
NN RMSE
NN RMSE
NN AMSE
Figure 3: Comparison among NN, RIO, and SVGP. The horizontal axis denotes the prediction
RMSE of the NN, and the vertical axis the prediction RMSE of RIO (blue dots) and SVGP (yellow dots). Each dot represents an independent experimental run. Since the scales are different, the solid blue line indicates where NN and RIO/SVGP have same prediction RMSE. Thus, a dot below the line means that the method (RIO or SVGP) performs better than the NN. and vice versa. Results of SVGP on the CT dataset are not plotted because its prediction RMSE exceeded the visible scale (i.e. they were 50). RIO consistently reduces the error of the NN, and outperforms SVGP in most cases.
and SVGP in terms of prediction RMSE. RIO is able to improve the NN predictions consistently regardless of how the dataset is split and how well the NN is trained. Even in the situations where original NN is much worse than SVGP, RIO still successfully improves the NN performance into a level that is comparable better than SVGP. Appendix DD. provides additional investigation into the behavior of RIO output corrections. RIO exhibits diverse behaviors that generally move the original NN predictions closer to the ground truth. To conclude, applying RIO to NNs not only provides additional uncertainty information, but also reliably reduces the point- point-prediction error.
Uncertainty Estimation Average negative log predictive density (NLPD) is used measure the quality of uncertainty estimation, which favours conservative models, but also effectively penalizes both over- and under-confident predictions Sarihonero-Candell et al.) 2006). The mean and standard deviation of NLPDs over multiple experimental runs are shown in Table (lower is better). RIO performs the best or equals the best method (based on statistical tests) in 8 out of 12 datasets. NNGP always yields high NLPD,; it returns over-confident predictions, because it does not include noise estimation in its original implementation. For the yacht dataset, ANP achieves the best NL.DD, but with high RMSE. This is because ANP is able to correctly return high predictive variance when its prediction error is high. For all other tested datasets, RIO variants consistently outperform ANP. Among all RIO variants, the full RIO provides the most reliable overall predictive uncertainty. The conclusion is that RIO successfully extracts useful uncertainty information from NN predictions. Appendix evaluuaes uncertainty estimates of estimated confidence intervals. Although RIO also provides reasonable estimates for this metric in most cases, comparison among algorithms using this metric is discouraged due its unreliability (see Appendix D.2 for examples).
additional metric: the true coverage probabilities
Computation Time Table [ the average wall clock time of each algorithm. All algorithms are implemented using Tensorflow under the same running environment (see Appendix D.I for implementation details). The RIO variants scale well to increasing dataset sizes and feature dimen- sionalities. L-BFGS-B B converges especially quickly for R+I on the three highest dimensional datasets, presumably because the residuals are very well-behaved compared to the raw targets or NN output. In contrast, ANP's computation time increases significantly with the scale of the dataset, and NNGP always needs very expensive computational budgets due to its costly grid search of hyperparameters.
Ablation Study The RIO variants with residual estimation generally perform better than its counter- parts in both point-prediction error and uncertainty estimation. This result confirms the effectiveness of residual estimation, suggested in Section 5.1 Another important result is that ++1 outperforms both Y+I (SVGP) and Y+O in most cases across all performance metrics, and RIO generally provides better performance than R+I and R+O in all respects. This result, in turn, confirms that the I/O kernel provides additional robustness, as suggested by the analysis in Section [52) In sum, both residual estimation and the I/O kernel contribute substantially to the performance of the framework.

---

Published as a conference paper at ICLR 2020
Table 2: Spearman's 's correlation between RMSE and of across RIO variants (including SVGP)
yacht
ENB/h
ENBle
airfoil
CCS
winehr
wine/w
CCPP
protein
SC
CT
MSD
correlation
0,943
0.943
1.0
1.0
0.943
0.099
0.886
1.0
0.943
1.0
0.771
0.943
p-value
0.005
0.005
0.0
0.072
Entries that are considered to indicate very strong posftiee monotonic correlation are marked in boldface.
Correlation between error and noise variance Table 2 shows the results of Spearman's ' Rank Correlation between RMSE and noise variance o² For each dataset, there are six pairs of data points, each of which contains the mean of RMSE and noise variance of RIO variant. A correlation value larger than 0.8 and p-value less than 0.05 indicate a very strong positive monotonic correlation, For 10 out of 12 datasets, very strong positive monotonic correlation between RMSE and noise variance was observed. This empirical result is in accordance with the theoretical prediction in Section 3.1
Application large-scale vision problem To show RIO's off-the-shelf applicability to modern deep convolutional architectures, it was applied to a recent pretrained NN for age estimation (Yang al. 2018). The pretrained NN and all data preprocessing were taken exactly from the official code release. The model is variant of BenseNet-121 (Huang et al., (017). and uses the IMDB age estimation dataset, which contains ~ 172K RGB images (Rothe et al. 2016). The goal is to predict the age of the individual in each image. The features for the GP input kernel were simply global max pool of the CNN's first stage output. From Table RIO substantially improves upon the mean absolute error (MAE) of the pretrained NN, outperforms SVGP in terms of both MAE and NLPD, and yields realistic confidence intervals (CIs). SVGP effectively learns nothing, so it estimates almost all variance noise, while RIO effectively augments the pretrained model.
Table 3: IMDB Age Estimation Results with DenseNet
Method Pretrained DenseNet Yang
MAE
NLPD
95% Coverage
90% Coverage
68% Coverage
2018
7.43
SVGP
36.45
5.06
0.99
0.96
0.62
RIO
3,59
0.94
0.91
0.75
Cl coverage means the percentage of testing outcomes that are within the estimated CL.
ANALYSIS OF RIO
This section presents an initial theoretical investigation into what makes RIO work, including why is useful to fit residuals, and why it is beneficial to use an I/O kernel.
5.1 BENEFITS OF FITTING NEURAL NETWORK RESIDUALS
Given pretrained NN, this section provides a theoretical perspective on how fitting its residuals with a GP can yield useful uncertainty information, while leading to prediction error lower than the NN or GP alone. The idea is that, due to its high expressivity, a pretrained NN may have learned complex structure that a GP would model poorly, when selecting a kernel that would capture this complexity is infeasible. Fitting GP to the residuals of the NN is easier, since this complex structure has been removed, and takes advantage of the predictive power of the NN, while providing useful uncertainty estimates.
Given a pretrained NN, when is fitting its residuals with a GP a good idea? To produce a model with uncertainty information, one could simply discard the NN and train GP directly on D. However, for problems with enough complexity, GP model selection (i.e., specifying an appropriate class of kernels) is challenging. Fortunately, the pretrained NN may have learned useful representation of this complexity, which can then be exploited. Suppose instead that a GP tries to model this complexity, and this attempt leads to worse generalization. This poor performance could come from many dimensions of the kernel selection process. As an initial theoretical investigation, an ideal case considered first: suppose the GP optimizer optimally avoids modeling this complexity incorrectly by estimating the variance of such complexity as noise. Such an estimate leads to the following decomposition:
y: h(xx)) + & = f(xs) g(xi) Eip
(8)
7

---

Published as a conference paper at ICLR 2020
where h(-) is the true signal, f()) is the apparent signal of the GP. g(.) is its apparent (spurious) noise, and & ~ (O, oa)) is real noise. Intuitively, due to its high expressivity. it is possible for a pretrained NN hav to correctly model part of g, along with part of f. resulting in the residuals
y/ NNN (x,) r(x.) + & = (x:))) + +x(()) + &,;
where (-) is now the apparent signal of the GP, and (.)+) is now its apparent noise. In such a case, it will be easier for the GP to learn the the residuals than to learn f directly. The uncertainty estimates of the GP will also be more precise, since the NN removes spurious noise. Thus, Eq. [8] limmediaelly predicts that the noise estimates for GP will be higher when fitting h directly than when fitting NN residuals. This prediction is confirmed in experiments (Table
To analyze these behaviors, the above decomposition is used to identify a class of scenarios for which fitting residuals leads to improvements (see Appendix for additional details and proofs of following theorems). For simplicity, suppose GP kernels are parameterized only by their signal variance (3, i.e. kernels are of the form (). for some kernel k. Then, the following two assumptions make Eq. concrete: (I) GP is well-suited for i.e., f(-) ~ GO :); (2) GP is ill-suited for g. i.e. independent of f and e-indistinguishable for GP on D (see Lemma A.2 for an example of such a function). Intuitively, this means the predictions of GP change by no more than in the presence of g.
Consider the residuals y/ (x,( = r(x,) + & = (x,) + (x,) + where ry is the remaining GP component, and rg is the remainder of g. The following assumption ensures that rg captures any spurious complexity added by an imperfectly trained NN: ry ~ GP(O, ol((,-)). for some a E (0, 1].
Let hcp be the GP predictor trained directly on D, and lIP+NNN = ĀNN + FGP be the final predictor after fitting residuals; let E. E and EN be the expected generalization errors of lIP,, hNN, and NGG+NNN, resp. The advantage of hGF+NN becomes clear as g becomes complex:
Theorem 5.1 (Advantage of fitting residuals).
lim Får FG 2 Bl))(x)) - (x)] and
lim (EEN F O. 5-40
The experimental results in Section $ support this result (Table T. First, in all experiments, GP+NN does indeed improve over the underlying NN. Second, notice that the improvement of GP+NN over GP is only guaranteed when IE[g² (x)] - [x)] is positive, i.c., when the NN successfully captures some underlying complexity. In experiments, when the NN is deficient, GP+NN often outperforms GP, but not always. For example, the NN is overfit to zero training loss, and uses the same training data as the GP, the GP cannot provide benefit. However, when the NN is well-trained, using proper overfitting prevention, the improvements are consistent and significant.
The prediction of Theorem 5.I that lower apparent noise corresponds to improved error is confirmed in experiments ( (Table 2 Note that an optimal model would learn h exactly, and have a predictive variance of of So, the improvement in error of hGr+NN over her also corresponds a predictive variance that is closer to that of the optimal model, since some spurious noise has been removed by fNN. Complementing this theoretical improvement uncertainty estimation, the above setting also leads to a key practical property, which is illustrated in Figure 2
Theorem 5.2. The uncertainty of For is positively correlated with the variance of NN residuals.
This property matches the intuition that the GP's variance should generally be higher for unstable NNs, i..., NNs with high residual variance, than for stable NNs. Such a property crucial to measuring the confidence of NN predictions in practice.
Overall, the initial theoretical results in this section (and the appendix) show that if problem is well-suited to be leamedd by an NN, and practitioner has access to pretrained NN for that problem, then training a GP to fit the residuals of the NN can provide potential benefits in uncertainty estimates, without sacrificing prediction performance. Further theoretical investigation regarding more complicated situations, c.g., misspecification of GP kernels, is considered as one interesting direction for future work.
5.2 ROBUSTNESS OF THE I/O KERNEL.
This section provides a justification for why a GP using the proposed I/O kernel is more robust than the standard GP, i.e., using the input kernel alone. The key assumption is that the output of NN can
8

---

Published as a conference paper at ICLR 2020
contain valuable information about its behavior, and, consequently, the structure of the target function. Consider the setup in Section 5.1 but now with yi =
+ E.. where fin ~ GP(O, kin) and fum GO,,, Ran), with non-trivial RBF kernels kim. haus (as in Equation 3). Let NNN be NN of sufficient complexity to be nontrivially non-affine, in that there exists positive-measure set of triples (x, x', x") s.t. ||x x'Il ||x x"II= but ||ăNN (x) (xx((x))l * ||ĀNN (x) ANN [x] Denote the generalization errors of the standard GP, GP with output kernel only, and GP with I/O kernel by E., E. and EY respectively. The expected result follows (proof in Appendix A.
Theorem 5.3 (Advantage of I/O kernel). F < EN and FN/o < F.
The optimizer associated with the GP simultancously optimizes the hyperparameters of both kernels, so the less useful kernel usually receives smaller signal variance. As a result, the I/O kernel is resilient to failures of either kernel. In particular, the GP using I/O kernel improves performance even in the case where the problem is complex that Euclidean distance in the input space provides no useful correlation information or when the input space contains some noisy features. Conversely, when the NN is a bad predictor, and ANN is better noise, the standard GP with input kernel alone is recovered. In other words, the I/O kernel is never worse than using the input kernel alone, and in practice it is often better. This conclusion is confirmed in the experiments in Section
DISCUSSION AND FUTURE DIRECTIONS
In addition to the reliable uncertainty estimation, accurate point prediction, and good scalability, demonstrated in Section 4 RIO provides other important benefits.
RIO can be directly applied to any standard NN without modification to the model architecture or training pipeline. Moreover, retraining of the NN or change of inference process are not required. The framework simply requires the outputs of NN; it does not need to access any internal structure. This feature makes the framework more accessible to practitioners in real-world applications, e.g.. data scientists can train NNs using traditional pipelines, then directly apply RIO the trained NNs.
RIO also provides robustness to a type of adversarial attack. Consider a worst-case scenario, in which an adversary can arbitrarily alter the output of the NN with minuscule changes to the input. It is well- that there are NNs for which this is possible (Goodfellow et al., (2015). In this case, with the help of the I/O kernel, the model becomes highly uncertain with respect the output kernel. A confident prediction then requires both input and output to be reasonable. In the real world, a high degree of uncertainty may meet a threshold for disqualifying the prediction as outside the scope of the model's 'ss ability..
There are several promising future directions for extending RIO:
First, applying RIO to reinforcement learning (RL) algorithms, which usually use standard NNs for reward predictions, would allow uncertainty estimation of the future rewards. Agents can then directly employ efficient exploration strategies, e.g., bandit algorithms [Thompson] 1933), rather than traditional stochastic approaches like s-greedy. Second, RIO applied to Bayesian optimization (BO) (Močkus) 1975) would make it possible to use standard NNs in surrogate modeling. This approach can potentially improve the expressivity of the surrogate model and the scalability of BO. Third, since RIO only requires access to the inputs and outputs of NNs, it could be directly applied to any existing prediction models, including hybrid and ensemble models. For example, experiments in Appendix [.6 show that RIO achieves good results when directly applied to random forests. This general applicability makes RIO a more general tool for real-world practitioners.
7 CONCLUSION
The RIO framework both provides estimates of predictive uncertainty of neural networks, and reduces their point-prediction errors. The approach captures NN behavior by estimating their residuals with an I/O kernel. RIO theoretically-gromddee using sparse GP approximation. scales well to large datasets. Remarkably, it can be applied directly to any pretrained NNs without modifications to model architecture or training pipeline. Thus, RIO can be used make NN regression practical and powerful in many real-world applications.
performs well on several real-world problems, and, by

---

Published as a conference paper at ICLR 2020
REFERENCES
Emmanuel Abbe and Colin Sandon. Provable limitations of deep learning. CoRR, abs/18 18t.08612. 2018.
C6369
Ofélia Anjos, Carla Iglesias, Fátima Peres, Javier Martínez, Ângela Gareía, and Javier Taboada.
Neural networks applied to discriminate botanical origin of honeys. Food Chemistry, 175:128 136, 2015. ISSN 0308-8144 146. doi: http:////...oo////..ooooooooooo.
URLhttp:
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5:4308, 07 2014. doi: 10. n0.103//commm5338.
D. Barber and Christopher Bishop. Ensemble learning in bayesian neural networks. In Gen- eralization Neural Networks and Machine Learning, pp. 215-237. Springer Verlag, Jan- wary 1998. URL htt.::
ensemble- - org.aaaeee....aaaaa..aaaaaaaaaar
S. Bergmann, S. Stelzer, and S. Strassburger. On the use of artificial neural networks in simulation- based manufacturing control. Journal of Simulation, 6(1):76-90) Feb 2014. ISSN 1747-7786. doi: 10.1057/jos. 2013.6. URL/tttss: //di: .rrg /10g//110/ 1057/jos 2013.6
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 1613-1622. JMLR.org, 2015. URL http://d]. acm. org/ ttttttiaaittnnnnn 3
Leo Breiman. Random forests. Machine Learning, (ate:miiiinn5551155550 45(1): Oct 2001. ISSN 1573-0565. doi: 10.1023/A:1010093444333 URL
R. Byrd, P. Lu, J. Nocedal, and C. Zhu. limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1100-1285 1995. doi: 10.1137/ 0916069.
Lehel Csató and Manfred Opper. Sparse on-line gaussian processes. Neural computation, 14:641-68. 04 2002. doi: (0.16622097760021225533
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017. URL http::/archivv
ics. www.cciccco u.s.ucciedd/// edu/mmedmm
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International
Conference on Machine Learning Volume 48, ICML' 16, pp. 1050-1059. JorR.org, 2016. URL d= 303333345999 3045502]
htttp://dl acm. org/ htgg/citai wwtpttii.tiiinn//ooo c fm
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713, PMLR, 10-15 Jul 2018a. URL http: pttt:://proceedingg mlr wwt.pppese///grpmpsss.
html
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. CoRR, abs/1807.01622,, 2018b. URL http:
01622
Zoubin Ghahramani. Probabilistic machliistttimmiieeeee machine learning and artificial intelligence. Nature, 521:452 EP 05 2015. URL bhtps:::: ://oi.////ooo///// /ttp://...../11100.000000001aaa
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. arxiv.org/abee 6572
10

---

Published as a conference paper at ICLR 2020
Alex Graves.
Practical variational inference for neural networks. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS' 11, pp. 2348 2356. USA. 2011. Curran Associates Inc. ISBN 178-1-61839-599-1 URL ci tat on cfm? i (12.:112999999998888
org/
James Hensman, Nicolò Fusi, and Neil Lawrence. Gaussian processes for big data. In Proceedings
of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI' 13, pp. 282-290, Ar- lington, Virginia, United States, 2013. AUAI Press. SKLhttp:: //11 acm. org/citation
C =fm?id=3023638 3023667
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan (eds.). Proceedings of the Eighteenth International Conference on Artificial Intelligence and Staristics, volume 38 of Pro- ceedings of Machine Learning Research, pp- 351-360) San Diego, California, USA, 09-12 May 2015. PMLR. URL http:::::p/oceeeeeiiii
mir press v 38/ htt.///enaaannnmmememmmeaamaaaaa html
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Anmual Conference on Computational Learning Theory, COLT '93, pp. 5-13, New York, NY, USA, 1993. ACM. ISBN 0-89791-611-5. doi: 10.1 10.1145/16834..1 168306 URL wtt.htt::/// acw.p: acm org/ 10 1141/168301111... 168306 wtt.://..........1.1111.1.......
Tin Kam Ho. Random decision forests. In Proceedings of the Third International Conference on Document Analysis and Recognition (Volume 1) Volume 1, ICDAR '95, pp. 278-, Washington, DC, USA, 1995. IEEE Computer Society. ISBN 0-8186-7128-9, URL http: : / //ll acm. org/
citation cf 844681
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. of CVPR, pp. 4700-4708, 2017.
Tomoharu Iwata and Zoubin Ghahramani. Improving Output Uncertainty Estimation and Gen- eralization in Deep Learning via Neural Network Gaussian Processes. arXiv e-prints, art. arXiv:1707.05999 Jul 2017.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee whye Teh. Attentive neural processes. CoRR, abs/1901.05 1057661,, 2019. URL
http::/arxizz org/ 2ttp://as/19110057766
Martin Krzywinski and Naomi Altman. Importance of being uncertain. Nature Methods, 10:809 EP -,08 08 2013. URL httpt:tt://// :/tp:::://o.... org/10 1038/ htt//teeh2....mmmmm 2613
Balaji Lakshminarayanann Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Curran Associates, Inc., 2017.
Systems 30, pp. 6402-6413.
Quoe Le. Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y. Ng. On optimization methods for deep learning. In Proceedings of the 28th International Conference International Conference on Machine Learning, ICML'11, pp. 265-272, USA, 2011 Omnipress. ISBN 978-1-4503-0619-5 URL http:////....ooooooc.ccaaaaaooot
3104516 6
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 EP 05 2015. URL
Jaehoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl- dickstein. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2018. URL
net/ www.ne//oooummmddiiii.....
David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Comput., 4(3):448 472, May 1992. ISSN 0899-7667. doi: 10. 19911629940000992244444444 URL
http:/ http://dxx.oom doi wrw. 016.0..66...60 neco. 1.....9999 1992. 1..2..43.44 4.3.448

---

Published as a conference paper at ICLR 2020
J. Močkus. On bayesian methods for seeking the extremum. In G. I. Marchuk (ed.), Optimiza- tion Techniques IFIP Technical Conference Novosibirsk, July 1-7. 1974, pp. 4004400044 Berlin, Heidelberg, 1975. Springer Berlin Heidelberg. ISBN 978-3-540-37497-8
Radford M. Neal. Bayesian Learning for Neural Networks. Seringer-Verlaee Berlin, Heidelberg, 1996. ISBN 0387947248.
Seyed Taghi Akhavan Niaki and Saeid Hoseinzade. Forecasting s&p 500 index using artificial neural networks and design of experiments. Journal of Industrial Engineering International, 9(1):1, Feb 2013. ISSN 2251-712X. doi: 10.1 10.1186/225--712X--111 URL 2251-7124 9-1
Manfred Opper and Francesco Vivarelli. General bounds bayes errors for regression with gaussian processes. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II. pp. 302-308, Cambridge, MA, USA, 1999. MIT Press. ISBN 0-262-11245-0) URL cfm? 340534 340656
http://dl acm. wttp://oocttitiio..oi/
Joaquin Quiñonero Candela and Carl Edward Rasmussen. unifying view of sparse approximate J. Mach. Learn. Res., 19199-19952 December 2005. ISSN 1532-4435,
gaussian process regression. processs
URL http: acm. org/ citatiiinnnn.. fww.immmim....... 6920900.6..... 111199990011999
Joaquin Quifionero-andella Carl Edward Rasmussen, Fabian Sinz, Olivier Bousquet, and Bernhard Schölkopf. Evaluating predictive uncertainty challenge. In Joaquin Satihonero-Candele Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 1-27, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 778-3-540-33428-6.
CE. Rasmussen and CKI. Williams. Gaussian Processes for Machine Learning. Adaptive Computa- tion and Machine Learning. MIT Press, January 2006.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision (IJCV), July 2016.
Matthias Seeger, Christopher K. Williams, and Neil Lawrence, Fast forward selection to speed up sparse gaussian process regression. In IN WORKSHOP ON Al AND STATISTICS 9, 2003.
Nida Shahid, Tim Rappon, and Whitney Berta. Applications of artificial neural networks in health care organizational decision-making makiing: A scoping review. PLOS ONE, 14(2):1-22 2019. doi: 13.1371/ 022.1..336.
pone. 0212356
Bemard W Silverman et al. Spline smoothing: the equivalent variable kemee method. The Annals of Statistics, 12(3):898-916, 1984.
Peter Sollich. Learning curves for gaussian processes. In Proceedings of the 1998 Conference Advances in Neural Information Processing Systems II, pp. 344-350, Cambridge, MA, USA, 1999. MIT Press. ISBN 0-262-11245-0. URL htttp:://lll acm. atgg/citatiion.cff..
346534 340667
Peter Sollich. Gaussian process regression with mismatched models. In Advances in Neural Information Processing Systems, pp. 519-526, 2002.
William R. Thompson. the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294 1933, ISSN 00063444. URL http: /twt:: jstor. org/ /tt/sa/eeee//322e////////
Michalis K. Titsias. Variational learning of inducing variables in sparse gaussian processes. In In Artificial Intelligence and Statistics 12, pp. 567-574, 2009.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016. PMLR. URL http::/proceedingg mlr
preas/ 251as/v1110 wttpwwwsoo.o 1son html
12

---

Published as a conference paper at ICLR 2020
Tsun-Yi Yi Yang, Yi-Hsuan Huang, Yen- Yu Lin, Pi- Pi-Cheng Hsiu, and Yung- Yu Chuang. Ssr-net: A compact soft stagewise regression network for age estimation. In Proc. of IJCAI, pp. 1078-1084, 2018.
Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23(4): 550-560, December 1997. ISSN 0098- 35000 doi: 10.1145/279233.2772266 URL htthttp: /doi
2on.ooog/10.1115/2222222..
A ADDITIONAL DETAILS AND PROOFS FOR SECTIONS 5.1 AND 5.2
provides theoretical perspective on how using GP to fit NN residuals can provide advantages over GP or NN alone. The idea is to decompose the target function into independent parts: one that is well-suited for GP to learn, and the other that is difficult for it to capture. The following definition provides formalization of this notion difficulty.
Section 5.1
Definition A.1 (e-indistinguihhable), Let fA be the predictor of a learner A trained on {(xi, Me) Mr) and A that of A trained on DD { [x,, Ss.Bi De + g(x:))} Then, if [fa(x) hx(x)) < e V x, we say
g is e-indistinguishable for A on D
In other words, the predictions of A change by no more than in the presence of g. If is small compared to [xg²(x)), then contains substantial complexity that cannot capture.
For notation, let VAA denote the predictor of a learner A trained on f(x,, (xx,) + (.) with ~ (0 a.) Let FX (v(()) 4 (x))² denote the expected generalization error of A on given training locations {x,) Now, consider a dataset = {(x,, = with
y: l(x,) & = f(x,) + 9(x)) Ei,
((0,0.). and Exg² og. For simplicity, assume the GP learner selects optimal hyperparameters for & and of. Suppose g independent of f and indistinguishable for GP on DD but neural network JNN may have successfully learned some of g's structure. These assumptions induce a concrete class examples of the decomposition in Eq. [ To demonstrate the existence of such decompositions, Lemma A.2 provides constructive example of such a function g.
where f() GP(O,k(,))) )),
Lemma A.2 (Construction of independent c-indistinguishable functions for GP). Given dataset ((x,, + pi for any o² and E > 0, there exists g with Elg² (x)) g is --indistinguishable on {(x;, fxx + g(x,) + )m for GP with continuous kernel k.
] = o and (x; 12n fimnt S.I.
Proof. Let X,+i = X; + v V smoother view of GP prediction Rasmussen & Williams 2006) fxp(x.) w w(x,) ) y, where w(x,) = (K + -)(x)_((x.)) is the weight function (Silverman et al. 1984), and y is the target vector. Since k and w(x,) are continuous around each X;,
I..... n. for a fixed vector v. Consider the linear
choose such that V x.
[w(x)); - w(x))n+sl
V vi=l,.... I,...,n.
no?
Let for be the predictor of GP trained on f(x;, fx,, + {m and her the predictor trained on [(x, fxx)) + g(x,) + 2m = D. Now, choose g so that
if S
g(x,))
-
if > n.
Then, hor(x.) = -w(xx g(x:) + Em w(xx)) ([fx)(((()))) + + + w(xx) (oxx.)((((xx))) (g(x;))
=fo((x.) + Σ o (x.))
Wx. = fxr(x.) + + (w(x.); w(Xx)n+s).
1=1
'Note that related information theoretic notion of indistinguisaabiliyy has recently been defined for analyzing the limits of deep learning (Abbe & Sandon 2018
13

---

Published as a conference paper at ICLR 2020
((xx); - w(x.)u+t)
< <
[for(x.) hcr(x.)] <8.
nog
Consider the residuals y; - hxx(xx) = + (r,(xx))) &.. Here, Ty is the remaining GP component, i.e., Ty GP(O, ak(:; )), for < < S 1. Similarly, Tg is the remainder of e-indistinguishabee for GP on {(x, r(x) +.)) -x)r = 6. These two assumptions about r's and r' ensure that any new spurious complexity added by the NN collected in rg. since the biggest advantage of the NN (its expressivity) is also its greatest source of risk. The final predictor after fitting residuals then hGP+NN = hNN + FCP. The following sequence of results captures the improvement due to residual estimation.
Lemma A.3 (Generalization of GP on h).
Ep + to - 2=(E6p) 2 < EOP < Epp tog +EE((E))) + 250g + E².
Proof. Let A (x) hxp(x) fcr denote the change in GP prediction due to g. Then, FOPP E[(h(xx) h [[((((()) + s(x)) (for(x) Afr((x)))
[[(([(x)) -x)P(x)) + (g(x) Ax)p(x)))
EP + 2x[([(x) - (co(())(((x) Afr (x)) B[(((x) Afpp(x)) E 2x(([(x) (xc(()) (x)) + [((((x) (x)()))
where the last line makes use of the fact that f and are independent. Now,
2B[([(x) - [cr(x))Af[rrxx)]] < 22 (F
and
[x(((x) - ffrr(x)) og ([2g((())AA(r(x) + E F())rr(x)
where
[2[g( < <200 and 0 E() < <
So, E + 0 - (3 220, < E < E + +og 2E + 25og + +
Lemma A.4 (Generalization of NN). E ax[ + + -.
Proof. Making use of the fact that E(ry] 0,
N = 2 (z)] = [[(r + ( +[r
Now, ry ~ GP ak) - [x) = f and 0f - [x) = = E[r
og - d. So, E
8.
Lemma A.5 (Generalization of GP fitting residuals).
to - - -2 8)$ < E - +2
Proof. Let Ar (Fr (x) For(x) Fycr(x) denote the change in GP prediction due to rg- Then,
Eh GGNNNN B[(((xx= TIGP+NN(x)) BX(()) iINN Fx](
[[(r)(x) + +r/(x)) - (F/GP(x) AF)GP(x)))
-E(((r(>) - (x)
Axfcr(x)))'
= =E' + 2B((r,(x) -
(or [[(r((x) AF)a)(())
= =E' (r) - (
- )x)c)(x))
Similar to the case of Lemma A.3
(x) (x)r)()))))) Arjor(x))] < 2
14

---

Published as a conference paper at ICLR 2020
and
[x(rg(x) (FrGp(x))9 = = - - + E[ArsGP(x)))
where
(x)r,(x) < <20(0) - 8) and 05 F(AF <
So,
20 8)3 < F GNNNNN oo Edr og - 28 F 2
(F
22
Lemma A.6 (Generalization of GP on and ry). From the classic result (Opper & Vivarelli 1999
Sollich 1999- Rasmussen & Williams 2006:: Consider the eigenfunction expansion k(x, x')
Ej X5oر (x)0) (x) and J k(x, x' (x)x(x))(()dd = {x),(x).. Let be the diagonal matrix of the eigenvalues Aj. and & be the design matrix, iee, D3 = =x(xx)) Then,
p = tr(( " and FGr tr + =- )-
F F(GHNNN) > 0. Theorem 5.1 lim +0 (F&r NN) 2 d and lim (FNN
Proof. From Lemmas A.3 A.4
and A.5 we have, resp.:
lim E = E + og -00
lim N = e00
oo - 5, and lim E GPPNNN = Eig + o7 - e-0
From Lemma A.6 we have
E
Edpp tr(A-1 -0, 25 T tr A-1 o, -20 1)-1 0,
and
(x) - EGG xx]] tr(a - A + on 222 1 - >0. 0.
So,
lim (FE FGPANN) 2 8 and lim E
E >0.
Theorem 5.2 The variance of NN residuals is positively correlated with the uncertainty of rGr-
Proof. Increases (x)) lead to increases as increases [xr"(x)) lead to decreases in and thus increases in the estimated noise level 6²,- So, an increase in either [x)] or (x)]
leads an increase in ak((x,; U.), (x.: ak, (aK((X 9), (X, aQT) -akk, which is the predictive variance of TGP.
Theorem
E < Ex and E E.g.
Proof.
x |x- x"1 km((x,x)) kin (x, x").
[\NN(X) NNN((")) (x'Il # [1 !xNx(x")] = hom (x, x²) of Kour (x, x")
These are true for all hyperparameter settings of kip or kout. since both are RBF kernels. So, there is no hyperparameter setting of kn that yields knn xX,xx) = kin (x,, x²) + kivul (x1, X2) V x1, X2. Similarly, there is no hyperparameter setting of hea that yields Kour X2) = km (x, X2) + (xm X2) V X2, Since, neither the input kernel over all sets of positive measure, their generalization error is greater than the Bayes error, which is achieved by the correct kerel (Sollich 2002), i.e., (x1, X2) + (xi,(x1,
output kernel alone
correctly specify the
B BACKGROUND
This section reviews notation for Neural Networks, Gaussian Process, and its more efficient approxi- mation, SVGP. The RIO method, introduced in OectionBoff the main paper, uses Gaussian Processes to estimate the uncertainty in neural network predictions and reduces their point-prediction enrors.
15

---

Published as a conference paper at ICLR 2020
B.1 NEURAL NETWORKS
Neural Networks (NNs) learn a nonlinear transformation from input to output space based on number of training examples. Let D C Rd
x Rdast denote the training dataset with size n, and A' =
{x; : (x) E x; x 2....,, 2 = ..2,...,77 n and y = {y: (x, y:) E D, lut, v.,..., (
1, ........ n denote the inputs and outputs (i.e., targets). A fully-connected feed-forward neural network with L hidden layers of width Ni (for layer / 2.2..... .......... L) performs the following computations: Let of denote the output value of jihh node in lth hidden layer given input 1
X3, then of <Ein with ki x. + l/),, for 1 = 1 and z/ Φ <
k= w/ " b/), for 1 = ...... L. where w denotes the weight on the connection from Athh node in previous layer to jth node in Ith hidden layer, by denotes the bias of jth node in ith hidden layer, and is a nonlinear activation function. The output value of jth node in output layer is then given by of Σn wout 2 + b where wout denotes the weight on the connection from kth node in last hidden layer to ,ith node in output layer, and llout denotes the bias of jth node in output layer.
gradient- based optimizer usually used to learn the weights and bias given a pre-defined loss function, e.g., a squared loss function L = Em (y; 8:)². For a standard NN, the learned parameters are fixed, so the NN output ý; is also fixed point. For Bayesian NN, distribution of the parameters is learned, so the NN output is distribution of y. However, pretrained standard NN needs to be augmented, e.g., with Gaussian Process, to achieve the same result.
B.2 GAUSSIAN PROCESS
Gaussian Process (GP) is collection of random variables, such that any finite collection of these variables follows a joint multivariate Gaussian distribution (Rasmussen & Williams 2006). Given a training dataset X' = {x, i 1.2..... n) and y = {8/ = f(x,) + 1, 1,2, ..... n2,, where e denotes additive independent identically distributed Gaussian noise, the first step for GP is to fit itself to these training data assuming y ~ (O, K(X, X)) + I), where N denotes a multivariate
Gaussian distribution with mean 0 and covariance matrix K(X, X) + I K(X, X') denotes the kernel-based covariance matrix at all pairs of training points with each entry ks.j = k(x,, x,), and of denotes the noise variance of observations. One commonly used kernel is the radial basis function (RBF) kernel, which is defined as k(x,, x,) = dfexp(- 27ς ||x; - x, 3) The signal variance oî
length scale ly and noise variance o² are trainable hyperparameters. The hyperparameters of the covariance function are optimized during the learning process maximize the log marginal likelihood log p(y)
After fitting phase, the GP is utilized to predict the distribution of label y, given a test point x.. This prediction is given by y, y.y.x. ~ N Na....ar(y.)) with V. k, (K(,, I 'y and var(u-) = kxx., x.) - k. (K(X, X') + I) where k, denotes the vector of kesnel-based covariances (i.e., k(x., x)) between x. and all the training points, and y denotes the vector of all training labels. Unlike with NN, the uncertainty of the prediction of a GP is therefore explicitly quantified.
B.3 SVGP
The main limitation of the standard GP, as defined above, is that it is excessively expensive in both computational and storage cost. For a dataset with n data points, the inference of standard GP has time complexity O(r²)) and space complexity (n²)) To circumvent this issue, sparse GP methods were developed to approximate the original GP by introducing inducing variables (Csató & Opper) 2002)
Seeger et al.] |2003] Quiñonero Candela & Rasmussen 2005 Titsias 2009). These approximation approaches lead to a computational complexity of (nm)² and space complexity of O(nm) where m is the number of inducing variables. Following this line of work, SVGP (Hensman al. 2013
2013) further improves the scalability of the approach by applying Stochastic Variational Inference (SVI) technique, as follows:
Consider the same training dataset and GP as in section B.2 and assume a set of inducing variables as 2 [2]] i 1,2,..., m. and 2/ (u,
f(: + ....,... m) (f(-) and are unknown).
1,2,...,
16

---

Published as a conference paper at ICLR 2020
Algorithm 1 Procedure of RIO
Require:
(X,y) = {(x,, yi)]i training data
ÿ = (ū: Jimii NN predictions on training data
x,: data to be predicted
g. NN prediction on x,
Ensure:
V. ~ NN. + var(f.)): a distribution of calibrated prediction
Training Phase:
1: calculate residuals r = {r. = yt 1.
2: for each optimizer step do
3: calculate covariance matrix K), (2, where each entry is given by
ke((x, (t), (x,, ú,]) = kin (x,; X2) hout (ūt, 0j), for i.j 1.2,...,,
4: optimize GP hyperparameters by maximizing log marginal likelihood logp(r[X,9) =
-3r" (K..((X, 9), 9)) + 1 glog (KX, ((),)) (x( + + log 2
Deployment Phase:
5: calculate residual mean R. k. (K. (K 8), (). + r and residual variance
var(f.) = (xx(.., (.), (x., .))) k. (K,( }), (2. + -
6: return distribution of calibrated prediction V4 ~ N(j. f., var(f.))
SVGP learns variational distribution q(4) by maximizing a lower bound of log p(y (X), where log p(y|X)) = log J P(y|4, (2))(()) and p(-) denotes the probability density under original GP. Trainable hyperparameters during the learning process include values of z; and hyperparameters of the covariance function of original GP. Given test point x., the predictive distribution is then given by p(y= x.) = p(y= [2, x. )q(L()d4, which still follows a Gaussian distribution. One advantage of SVGP is that minibatch training methods (Le et al. 2011) can be applied in case of very large dataset. Suppose the minibatch size is m and m & m/, then for each training step/iteration, the computational complexity is (m' mm)) and the space complexity is O(m'm m). For full details about SVGP, see Hensman et al. (2013). Since NNs typically are based on training with relatively large datasets, SVGP makes it practical to implement uncertainty estimates on NNs.
C PROCEDURE OF RIO
This section provides an algorithmic description of RIO (see Algorithm I
D EMPIRICAL STUDY
D. EXPERIMENTAL SETUPS
Dataset Description In total, 12 real-world regression datasets from UCI machine learning repos- itory (Dua & Graff] 2017) are tested. Table summarizes the basic information of these datasets. For all the datasets except MSD. 20% of the whole dataset is used as test dataset and 80% used as training dataset, and this split is randomly generated in each independent run. For MSD, the first 463715 samples are used as training dataset and the last 51630 samples are used testing dataset according to the provider's guideline. During the experiments, all the datasets except for MSD are tested for 100 independent runs, and MSD datasets are tested for 10 independent runs. For each independent run, the dataset is randomly split into training set, validation set, and test set (except for MSD, which the dataset split strictly predefined by the provider), and the same random dataset split are used by all the tested algorithms to ensure fair comparisons. All source codes for reproducing the experimental results are provided at: (https://github.comm wwwp://wccoo...aaaa//a.e -a/rri--apperr
Parametric Setup for Algorithms
NN: For SC dataset, fully connected forward NN with hidden layers, each with 128 hidden neurons, is used. For CT dataset, a fully connected forward NN with hidden layers, each with 256 hidden neurons, is used. For MSD dataset, fully connected
17

---

Published as a conference paper at ICLR 2020
Table 4: Summary of testing dataset
abbreviation
Full name UCI ML repository
dataset dimension
note
yacht
Yacht Hydrodynamiics Duta Set
308
6
ENB/h
Energy efficiency
768
Heating turget Cooling Load as target
ENB/e
Energy efficiency
768
airfoil
Airfoil Self-Noise
1505
CCS
Concrete Compressive Strength
1030
wine's
Wine Quality
1599
I1
only use winequality red data only winequality white data
wine/w
Wine Quality
4898
CCPP
Combined Cycle Power Plant
9568
CASP
Physicachemical Progerties Protein Teniary Stricture
SC
Superconduetivey Data
21263
81
CT
Relative location of CT slices on axial axis
53500
384
MSD
YeutPredictonMSDD
515345
90
train: lirst 463715, test: last 51630
feed-forward NN with 4 hidden layers, each with 64 hidden neurons, is used. For all the remaining datasets, fully connected feed-forward NN with 2 hidden layers, each with 64 hidden neurons, is used. The inputs to the NN are normalized to have mean and standard deviation The activation function is ReLU for all the hidden layers. The maximum number of epochs for training is 1000. 20% of the training data is used as validation data, and the split is random at each independent run. An early stop triggered if the loss on validation data has not be improved for 10 epochs. The optimizer is RMSprop with learning rate 0.001, and the loss function is mean squared error (MSE).
RIO, RIO variants and SVGP (Hensman et al. 2013): SVGP is used as an approxima- tor to original GP in RIO and all the RIO variants. For RIO, RIO variants and SVGP, the number of inducing points are 50 for all the experiments. RBF kernel is used for both input and output kernel. For RIO, RIO variants and SVGP, the signal variances and length scales of all the kernels plus the noise variance are the trainable hyperparameters. The optimizer is L-BFGS- B with default parameters as in ciipyooptimize documentation (https://docs.scit)) www.oo//o///ciiyr/ffeeeee http://///piiiie.mmiiiiieeeeeeee www.mmiiiieeiiiiee Ibf..b. htmml, and the max- imum number of iterations is set as 1000. The training process runs until the L-BFGS-B optimizer decides to stop.
NNGP (Lee et al. 2018): For NNGP kernel, the depth is 2, and the activation function is ReLUU "g 10T, no 151, and ne 131. Following the learning process in original paper, grid search is performed to search for the best values of a and oî. Same in the original paper, a grid of 30 points evenly spaced from 0.1 to 5.0 (for o²) and 30 points evenly spaced from 0 to 2.0 (for of was evaluated. The noise variance of is fixed 0.01. The grid search process stops when Cholesky decomposition fails or all the 900 points are evaluated. The best values found during the grid search will be used in the experiments. No pre-computed lookup tables are utilized.
ANP (Kim et al.) 2019): The parametric setups of ANP are following the recommendations in the original paper. The attention type is multihead. the hidden size is 64, the max number of context points is 50, the context ratio is 0.8. the random kernel hyperparameters option is on. The size of latent encoder is 64 x 64 x 64 x 64, the number of latents is 64, the size of deterministic encoder 64 x 64 x 64 x 64, the size of decoder is x x 64 64 2, and the deterministic path option is on. Adam optimizer with learning rate 10-4 4 is used, and the maximum number of training iterations is 2000.
Performance Metries
To measure the point predictioi error, the Root Mean Square Error (RMSE) between the method predictions and true outcomes test datasets calculated for each independent experimental run. After that, the mean and standard deviations of these RMSEs are used to measure the performance of the algorithms.
To quantitatively measure the quality of uncertainty estimation, average negative log pre- dietive density (NLPD) (Quiñonero- Candela al. 2006) is used measure the quality of
18

---

Published as a conference paper at ICLR 2020
uncertainty estimation. NLPD is given by
1 n Y2
L L=
log p(5: = yi(x,)
(10)
where y: indicates the prediction results, x, is the input with true associated outcome y:. p(-) is the probability density function (PDF) of the returned distribution based on input x;-
To investigate the behaviors of RIO variants during learning, the mean of estimated noise variance o, over all the independent runs are calculated.
To compare the computation time of the algorithms, the training time (wall clock time) of NN, RIO, all the ablated RIO variants, SVGP and ANP are averaged over all the independent runs as the computation time. It is notable that the computation time of all RIO variants does not include the training time of associated NN, because the NN is considered to be pretrained. For NNGP, the wall clock time for the grid search is used. In case that the grid search stops due Cholesky decomposition failures, the computation time of NNGP will be estimated as the average running time of all the successful evaluations x 900, which is the supposed number of evaluations. All the algorithms are implemented using Tensorflow, and tested in the exactly same python environment. All the experiments are running on machine with 16 Intel(R) Xeon(R) CPU E5- v4 @2.60GHz and 128GB memory.
D.2 RESULTS ON ESTIMATED CONFIDENCE INTERVALS
Confidence interval (CI) is useful tool to estimate the distribution of outcomes with explicit probabilities in real-world applications. In order to provide more insights for practitioners, the percentages of testing outcomes that within the 95%/90%%66% CIs as estimated by each algorithm are calculated. Ideally, these percentages should be as close to the estimated confidence levels as possible, e.g., a perfect uncertainty estimator would have exactly 95% of testing outcomes within its estimated 95% Cls. In practice, when two models have similar quality of CI estimations, the more conservative one favoured. Figure 4 and 5 show the distribution of the percentages that lesting outcomes are within the estimated 95%/90%/667 % CIs over all the independent runs for all the datasets and algorithms. Figure 6 7 and compare the distributions of coverage percentages for 1 1%-99% CIs for RIO and SVGP. The experimental data are extracted from the experiments that generate Table I in main paper. In Figure 4 5 6 / and the box extends from the lower to upper quartile values of the data (each data point represents an independent experimental run), with a line at the median. The whiskers extend from the box to show the range of the data. Flier points are those past the end of the whiskers, indicating the outliers.
According to the results, RIO is able to provide reasonable CI estimations in most cases. However, using this metric to compare algorithms regarding uncertainty estimation quality may be noisy and misleading. A method that simply learns the distribution of the labels would perform well in CI coverage metric, but it cannot make any meaningful point-wise prediction. More detailed examples are given as follows.
For "CT" dataset, SVGP has an extremely high RMSE of around 52 while RIO variants only have RMSEs of 1. However, SVGP still shows acceptable 95% and 90% CI coverage, and even has over-confident coverage for 68% CI. After investigation, it was found that this happened only by chance, and was not due to the accurate CI estimations of SVGP. Since SVGP is not able to extract any useful information from the high-dimensional input space, it simply treated all outcomes as noise. As a result, SVGP shows a very large RMSE compared to other algorithms, and the mean of its predicted outcome distribution is always around O. Since SVGP treats everything as noise, the estimated noise variance is very high, and the estimated 95% CI based on this noise variance is overly high and covers all the test outcomes in most cases. When the estimated 90% CI is evaluated, the large error in mean estimation and large error in noise variance estimation cancel mostly cancel each other out by chance, i.c., the estimated 90% CI is mistakenly shifted by erroncous mean then the overly wide noise variance fortunately covers slightly more than 90% test outcomes. The case of the estimated 68% CI is similar, but now the error in noise variance cannot fully cover the error in mean, so the coverage percentages are now below 68%, indicating over-confidence.
One interesting observation is that SVGP tends to be more "conservative" for high confidence levels (>90%), even in cases where they are "optimistic" for low confidence levels. After investigation,
19

---

Published as a conference paper at ICLR 2020
percondage
redic percartage of pa
yedii: parcortage 50%
EMdh zocerlage of paro sillen
193h pateraap: slick porto withr
910
APP
141
8-0
760
140
811
V.5
WO
D1B parstape poin withn
E1BC
persetge
2007
141
110
01181 percrriage of bol perto 5006150%.C
pursrrage blaud poblia 23%
055 pimpttgg bio aolrt sut
CCS panselage ofaat poblia
8.95
675
2907
300P
schol: purattguraatgggeeooooo beal poitoo withe 20%
smoll pwr.oraageegooooom tasl painta the
0.0
6378
Vio
140
1410
$750
Figure 4: Quality of estimated Cls. These figures show the distribution of the percentages that testing outcomes are within the estimated 95%/90%/66% Cls over all the independent runs.
20

---

Published as a conference paper at ICLR 2020
wwww penartage of pana
wwwi zecarnaage led wine
part within
1.72
1.00
2.70
2007
2907
901
CCPP potcesage of pombs within 50%
0CF7 paseniage ofint ports with
CC7 paverige aol wthn
1.00
1.5
00
ViB
0/0
710
Vie
1997
710
Via
perton: peroedage Test wine 95%
(e0/0) pew.eraee test parts win 90%6
300dhi aerooraaa pants wiree 68%
1.0
deneriage porto wizen
aeceniage Ind
aaceniage Indpptto
1.00
VID
Via
1977
Vib
160% percoltape COST peinss with
1507 porcedagge ontt peints 33%6.1
150h portetzge pents with 65SCC
1.00
A37
000
0.5
$10
710
YBC
615
110
80
Figure Quality of estimated Cls. These figures show the distribution of the percentages that testing outcomes are within the estimated 95%/99%9/68 Cls over all the independent runs.
21

---

Published as a conference paper at ICLR 2020
Distributions of coverage on yacht
1.0
Ri SVSP
0.6
0.2
0.0
Distributions coverage
1.0
Ri
SVEP
0.6
0.0
26
Confidence
Distributions of coverage
3.0
RIO
SVGP
0.8
Percettge
0.4
0.0
20
Confidence Level
Distributions
airfoil
SVGP
3.8
0.4
20
60
80
Confidance Levd (33)
Figure 6: Distribution of CI Coverage. These figures compare the distributions of empirical coverage percentages of estimated 9%-99% Cls over all the independent runs for RIO and SVGP.
22

---

Published as a conference paper at ICLR 2020
Distributions CI coverage on CCS
1.0
RD SVSP
0.6
0.0
coverage
1.0
RD
SVEP
0.8
0.4
0.0
Confidence
Distributions coverage
RIO
SVGP
0.8
0.4
0.0
20
Confidence Level
Distributions of
SVGP
3.8
Coveroge
3.4
20
60
80
Confidance Levd (33)
Figure Distribution of CI Coverage. These figures compare the distributions of empirical coverage percentages of estimated 99-99% Cls over all the independent runs for RIO and SVGP.
23

---

Published as a conference paper at ICLR 2020
Distributions coverage on protein
1.0
RD SVSP
0.0
coverage
Rd
SVEP
0.8
0.4
0.0
Distributions coverage
3.0
RIO
SVGP
0.8
Percedage
0.4
0.0
20
Confidence Level
Distributions
MSD
SVGP
2.6
Coveroge
3.4
20
60
80
Confidance Levd (33)
Figure 8: Distribution of CI Coverage. These figures compare the distributions of empirical coverage percentages of estimated 9%-99% Cls over all the independent runs for RIO and SVGP.
24

---

Published as a conference paper at ICLR 2020
this is because SVGP normally has an overly high noise variance estimation (also comes with higher prediction RMSE in most cases), so it has a higher chance of covering more points when the increase in CI width (influenced by noise variance) surpasses the erroneous shift of mean (depending on prediction errors).
D.3 ADDITIONAL RESULTS ON OUTPUT CORRECTION
This section shows additional demonstrations of the detailed behaviors of RIO output correction. The experimental data are extracted from the experiments that generate Table i main paper.
Figure 9 plots the distributions of ground truth labels (outcomes), original NN predictions and predictions corrected after RIO for randomly picked run for each dataset. Figure TO shows the point- wise comparisons between NN outputs and RIO-corrected outputs for the same experimental runs as in Figure Based on the results, it is clear that RIO tends to calibrate each NN prediction accordingly. The distribution of outputs after RIO calibration may be shift, or shrinkage, or expansion, or even more complex modifications of the original NN predictions, depending on how different are NN predictions from ground truths. As result, the distribution of RIO calibrated outputs are closer to the distribution of the ground truth. One interesting behavior can be observed in Figure 9 for "protein" dataset: after applying RIO, the range of whiskers shrunk and the outliers disappeared, but the box (indicating 25 to 75 percentile the data) expanded. This behavior shows that RIO can customize its calibration to each point. Another interesting behavior is that for "wine/r" dataset (see both Figure 9 and 10). RIO shifts all the original NN outputs to lower values, which are closer to the distribution of the ground truth.
To better study the behaviors of RIO, a new performance metric is defined, called improvement ratio (IR), which is the ratio between number of successful corrections (successfully reducing the prediction error) and total number of data points. For each run on each dataset, this IR value is calculated, and the distribution of IR values over 100 independent runs (random dataset split except for MSD, random NN initialization and training) on each dataset is plotted in Figure [T] According to the results, the IR values for RIO are above 0.5 in most cases. For datasets, IR values are above 0.5 in all 100 independent runs. For some runs in yacht, ENB/h, CT, and MSD, the IR values are above 0.8 or even above 0.9, All these observations show that RIO making meaningful corrections instead of random perturbations. Results in Figure 1T also provides useful information for practitioners: Although not all RIO calibrations improve the result, most of them do.
Same empirical analysis is also conducted for two RIO variants, namely R+I (predicting residuals with only input kernel) and R+O (predicting residuals with only output kernel). Figure 12 13 and [4 show results for R+I, and Figure 15
16 and 17 show results for R+O. From the results, the output kernel is helpful in problems where input kernel does not work well (CT and MSD), and it also shows more robust performance in terms of improvement ratio (IR) in most datasets. However it is still generally worse than full RIO. More specifically, R+I shows an extremely low IR in CT dataset (see Figure 14.. After investigation, this is because the input kernel itself is not able learn anything from the complex high-dimensional input space, so it treats everything as noise. As result, it keeps the NN output unchanged during correction in most cases. Applying output kernel instead solves the issue. After comparing Figure 10 13 and 16 it can be observed that the behaviors of RIO are either a mixture or selection between R+1 and R+O. This means RIO with I/O kernel is able choose the best kernel among these two or combines both if needed.
D.4 EXPERIMENTAL RESULTS WITH ARD
The experimental results shown in the main paper is based on the setup that all RIO variants are using standard RBF kernel without Automatic Relevance Determination (ARD). To investigate the performance of RIO under more sophisticated setups, same experiments are run for all RIO variants with ARD mechanism turned on (all other experimental setups are identical with section D.I except that the NN depth for MSD dataset is reduced from 4 hidden layers to 2, due to computation failure issue during Cholesky decomposition). Table 5 shows the summarized experimental results. From the results, RIO still performs the best equals the best method (based on statistical tests) in 9 out of 12 datasets for both RMSE and NLPD metrics. This clearly shows the effectiveness and robustness of RIO.
25

---

Published as a conference paper at ICLR 2020
Virr Dictmnatiin arsedtmeete and wediitions
ES84 DISIBASS grast tnsts axépreectioss
11a (822 Budio ground buticasd predetians
CCS Ostebaticn grand and proaccems
wmath Disotidtinn tryhh and cacttess
129
wititer: Dor Batlon greust prstitisss
16t1 grized outteand oreditioos
pround
215
315
swadiwhh
peadluih miriiierr Bam cndeciedilyt
Vilmiiiees pedilies seresiidii
Figure 9: Distribution of Ground Truths and NN/RIO-correcter predictions. The box extends from the lower to upper quartile values of the data points, with a line the median. The whiskers extend from the box to show the range of the data. Flier points those past the end of the whiskers, indicating the outliers.
26

---

Published as a conference paper at ICLR 2020
DME
6.5
135
100
125
50
CDP
luqus
100
120
109
1008 3393
7100
1200
7993 7156
Figure 10: Comparison between NN and RIO-corrected outputs. Each dot represents a data point. The horizontal axis denote the original NN predictions, and the vertical axis the corresponding predictions after RIO corrections. The solid line indicates where NN predictions are same as RIO- corrected predictions (i.e., no change in output). This figure shows that RIO exhibits diverse behavior how it calibrates output.
27

---

Published as a conference paper at ICLR 2020
Table 5: Summary of experimental results (with ARD)
Dataser
Method
RMSE
NLPD
Naise
Time
Dateset
Medhod
RMSE
NLPD
Noise
Time
(100)
3ach1
RIO
NN
11.611
1.22:20.4
1.31
0.46
3.31
7.75
ENR/h
RIO
NN
0.59:10.10
0.187
0.22
11.98
8,47
1.96+-0.391
0.83
6.89
0.61=0.121
2.208
0.77:1
0.380:1..2211
Y40
1.397
1.82
9.55
Y1O
0.33
1.226 34511
0.61
15.31
Y-10
1.09
1.391-004225 f
0.84
9.85
Y+10
0.11000081
0.74400.004
0.25
15.96
SVGP
0.911.0.2111
1.359.0..008
0.77
8.93
SVGP
0.77:
0.196
14.21
35.18-0053441
0.7-0.291
ANP
1.59--3.2011
0.793:00.8771
40.82
ANP
4.08 4.08-2.2111
102.3
ENB/
NN
1.881:004611
1.42:10.18
18825:10.155
11.61
nicfoil
NN
0,84
2.49:..15
1.349:10.048
8.83
145
1 1341
1.562011611
2.3762:00041
768
R+O
1.771:04611
2.021:0.2222 25221
2.34
9.73
1505
R+0
0.14
2541100711 0
16.31
10.97
Y40
1.77:04511
0.020::02311 231
2.56
20.54
Y+0
1.39
2,88910 196
20.08
22,90
2.111
1871
21.65
3.73
2.746-0
SVGP
1.731:2211
1.99:1001300
3.12
18.73
SVGP
3.36:10231
26321100551
11.11
21,80
NNGP
491200332
88.14:20.8611
7704
NNGP
6554202345
33 53:00:20000 420
3355
ANP
1..15.2.511
2.98:10
66.13
ANP
20,17
529911631612
231.7
CCS
RIO
6.29+005541
5..50
0.119
23.00
9.20
winelr
RIO
0.6:4400033
1091 +
0.28
7.55
1030
R41
0.80
0.106
23.19
8.80
1599
Ral
0670=00331
100441
0.25
9.52
0.121
0676=0.0541
1003
23:22
1.85+00811
Y-10
5:91 5.9111
3:22220.10 10811
24.30
12.09
Y+10
0672-0033
10052001171
0.29
14.28
SYGP
6.20:-0000
34.66
11.34
SVGP
0.020
0309100111
0.39
13.32
0.0200
RIO
0.707-10.017
1000110.005
0.38
11.8
RIO
3.99
1.796:10.026
15.82
26.27
4898
R+1
0.702210008 t
0.355
0.38
14.76
9368
R-I
3.99:..13
2.797
15.87
24.47
0.711 1
109R 0.057700577
0.39
6.46
R+O
0.33
2.879100227
9.99
0.71010.11911
1.05620.00771
8.911447811
2.974
Y-1O
0.708
1.033-000066
0.39
20.15
Y+10
0.77-
2568 23512
36.48
27.65
SVGP
0..14
1.74 0.022)
0.50
19.42
SVGP
8.80 44831
10.1811
1455
26.61
0.231
147.4
protein
4.08::005
2836:10.013
149.6
3.44
45730
R+1
4.11::00511 11
16.01
130.9
21263
Ral
11.27-00311
5.477 0291
105.5
95.33
R+O
4.152000111
184330001511
16.31
98.53
R+0
04661
1139
47.61
4.15200611
2.8420001511
16.11
120
11.68104211
3.82
114.2
4.08:0005
2.83:1000133
01.29-03381
5.58
SVGP
464-200111
2.05=20.00811
25.15
128.1
SVGP
14.12
400001001511
217.8
95.78
NI
1.22002911
196.6
NN
11.50:11011
777.8
1221:10.202
9.79.10.22
53500
R+l
1124+02991
1 0.005
1.55
20.03
515345
R+l
12.54 L16
156.6
91.01
R+O
0.16
1261 1.2200
0.95
159.4
R+O
9.82 0.19
3,701 0.002
95.11
2739
384
0.98-00777
1.303-1--336
2.02
180.
209.8-1996.8
7.00
2364
0.88--0.1311
SVGP
$2.09100.111
5.372.00.00441
2712
29.69
SVGP
10.97:00.01
3.901110001
199.7
2590
The symbols 1 and indicate that the differenes between the morked entry and R1D statistically significant at the 5% significance lovel using pairod E-test and Wilcoxon tett, respoctively. The best entries that are significantly better tian the others under lens one statocical test are marked in holdface (lies are allowed).
28

---

Published as a conference paper at ICLR 2020
Distribution of improvement ratio
0.9
percentage of successful improvement
0.8
0.5
yacht
ENB/h
ENB/c
airfoil
CCS
wine/r wine/w
CCPP
protein
SC
CT
MSD
dataset
Figure 11: Distribution of Impovement Ratio. The box extends from the lower to upper quartile values of the data (each data point represents an independent experimental run), with a line at the median. The whiskers extend from the box show the range of the data. Flier points are those past the end of the whiskers, indicating the outliers.
D.5 EXPERIMENTAL RESULTS WITH 10,000 MAXIMUM OPTIMIZER TERATIONS
The experimental results shown in the main paper is based on the setup that all RIO variants are using L-BFGS-B optimizer with maximun number of iterations as 1,000. To investigate the performance of RIO under larger computational budget. same experiments are run for all RIO variants with maximum number of optimizer iterations 10,000 (all other experimental setups are identical with section D.I). Table[]]shows the summarized experimental results for 8 smallest datasets. According to the results, the rankings of the methods are very similar to those in Table I (in the main paper), and RIO still performs best in all metrics.
D.6 RESULTS ON RANDOM FORESTS
In principle, RIO can be applied to any prediction model since it treats the pretrained model as black box. To demonstrate this extensibility, RIO applied to another classical prediction model
Random Forests (RF) (Ho) 1995 Breiman 2001). The same experimental setups as in section D.I are used. For RF, the number of estimators is as 100 for all datasets. To avoid overfitting, the minimum number of samples required to be at leaf node is 10 for all datasets, and the max depth of the tree is 7 for MSD and for other datasets. Table shows the experimental results. From Table RIO performs the best equals the best method (based on statistical tests) in out of 12 datasets in terms of both RMSE and NLPD. In addition, RIO significantly improves the performance original RF in 11 out of 12 datasets. These results demonstrates the robustness and broad applicability of RIO.
29

---

Published as a conference paper at ICLR 2020
19070 Cwwrossonn groud vvtra and pruddices
0184h Duoriation ground triba predetirn
240% Cltt bution of gound trstn and prediction
Rizeectionn
CCS Ostebaticn ground tvir and preccions
wmah Disttitiinn truits proddiees
554
Whanittion pobdees connded ty
witeee Ditrisstinnl gruus prstitloss
1601 groued ardictios
Cittibutionn
C.Drrribdden grundinuths
pround oudta arudation
215
315
swadiuhh
Figure 12: Distribution of Ground Truths and NN/R+I-corrected predictions. The box extends from the lower to upper quartile values of the data points, with a line the median. The whiskers extend from the box to show the range of the data. Flier points those past the end of the whiskers, indicating the outliers.
30

---

Published as a conference paper at ICLR 2020
CCS
125
314
135
120
130
14
81
50
wreah
COP
NSO
140
780
120
7008
7701
$1.00.35
1200
2553
7100
Figure 13: Comparison between NN and R+I-corrected outputs. Each dot represents data point. The horizontal axis denote the original NN predictions, and the vertical axis the corresponding predictions after R+I corrections. The solid line indicates where NN predictions same R+1- corrected predictions (i.e., no change in output).
31

---

Published as a conference paper at ICLR 2020
Distribution of improvement ratio for R+1
percentage successful improvement
0.6
0.4
0.2
yacht
ENB/h
ENB/C
airfoll
CCS
wine/r
wine/w
CCPP
protein
SC
CT
MSD
dataset
Figure 14: Distribution of Impovement Ratio. The box extends from the lower to upper quartile values of the data (each data point represents an independent experimental run), with a line at the median. The whiskers extend from the box show the range of the data. Flier points are those past the end of the whiskers, indicating the outliers.
Table 6: Summary of experimental results (with 10,000 maximum optimizer iterations)
Dataset
Method
meanshsad
RMSE
mewntsed
NLPD
Noise
Time
Dutasel
Methol
RMSE
NLPD meandt-id
Nosee Varinsee
Fime
2.001093111
0.9420.371
yacht
R1O
0.50
1883 08556
0.74
25.87
ENBh
RIO
0.64::0.36
0.968=0.277
0.31
45.05
R+]
1.951106511
2266:10.8411
2.19
5.59
R+1
0.70:10.3311
1.313:0.11711
0.41
22.22
1.7610557
2176200 $2511
1.30
768
0:72:003111
309
Y+0
1.75:005611
1,501:
1.60
23.99
X
Y+O
0.78:10.35
1,163 1.633-10.228--
0.55
Y-10
1.0110.44
1.919110.567
0.82
45.89
Y+10
0.66:10.330 261
1.013:10:2800
0.34
82.82
SVOP
36720.601
2689100 11
12.07
42.50
SVGP
2.112:1711
2,145::000711
4.24
79.46
12.40::1.4511
32.40:1:0:688
ANP
7.50:33.2011
1.79310.8871
40.82
ANP
4.08:12.7711
2.475
ENBA:
NK
1:87-04311
1.51::036
1882-10.198
11.70
sirfoil
NN
4842204711
3.06::0.00
1.551
9.44
3.96
1.70=0.411
L98-20.21
3.13-00211
9.91
768
R40
1.7504111
2011:021111
2.21
10.85
1505
R4O
4.16:202711
2.8481006811
16.55
10.92
Y40
1.7500411
20122021001
2:27
39.52
Y:O
421
2.802000111
17.89
38.69
16010:551
3:1910.3011
2.5811008711
SVOP
2:52-02111
23672000211
3.27:02011
2.08:00561
10.56
NNGP
43110:3311
30142088611
7704
NNGP
6.54:0.2311
51.00042007
3355
ANP
481221311
26882054811
64.11
ANP
21.17:107721
316911631611
231.7
6.25:0.991
06031000111
RIO
596:10477
3.230-10.108
25.31
14.31
wince
RIO
0.671 0000
1.88=00087
0.28
20.12
1030
Rel
0.991
3,235 0.107
26.04
4.91
1599
R+1
0.668
0.033tt
10000-008851
0.25
12.61
R+0
6.1910.911
12:110.
27.43
R+O
0.0331
009110088111
0.29
Y40
0674-00055f
1009+000611
Y+10
6.03--0.7773
321077
26.24
41,89
Y+10
0.671 +000
1087=00008
0.28
34.45
SVGP
0.210.3711
41.15
73.56
SVGP
8.642
0.973-1002211
0.39
70.53
NN
0,7230002211
NN
400:10.401
0:704110188
0.58400004
4.03-10.133
2.00=
1898
R+1
0.00010.017:1
10:9110033
0.38
26.43
9368
R+1
1.311101311
281011000011 1
16.28
116.0
R+0
0.71020003
0.39
8.87
R+0
4.33-20.144
2
18.56
19.02
0.71020.02011
1993
5.112
4161
Y+10
0,704:200011
1.088.10004
66.79
Y+10
1.71:1
2
33.58
267.1
SVGP
0.11320.01611
1076:20.022
0.5
158.
SVGP
4.2520.1311
2.850-10008
17.04
334.5
The symbbll and 5 indicate that the differencs berween the marked entry and KID staustically signdiicntt at the 3% significance lovel using pairod E-s5st and Wilcexon respectively. best that significanlly better than the under statacical marked boldfacc (ties allowed).
32

---

Published as a conference paper at ICLR 2020
men
CCS Dicritutio greand tues and prodiccens
whei Oittr ground truthe andicciin
geandioass
aivoe: Dintiuution www.deraateeee.......mm........e and prodetiss
Okriauian goand prodiciinn
didtebutioned creun triths preditiess
D-sreuted creurd trallo precitices
710
200
i000 100h
SPECCIINE
SnRiCtine ENOO corecctd
IMOINECTORE SPOCCIOA
Figure 15: Distribution Ground Truths and NNoR+O-coreetted predictions. The box extends from the lower to upper quartile values of the data points, with line the median. The whiskers extend from the box to show the range of the data. Flier points are those past the end of the whiskers, indicating the outliers.
33

---

Published as a conference paper at ICLR 2020
CCS
185
116
135
120
130
14
5.0
COP
15
NS0
180
120
2208 3701
1200
2953
7100
Comparison between NN and R+O-corrected outputs. Each dot represents data point. The horizontal axis denote the original NN predictions, and the vertical axis the corresponding predictions after R+O corrections. The solid line indicates where NN predictions same as R+O-corrected predictions (i.e., no change in output).
Figure 16:
34

---

Published as a conference paper at ICLR 2020
Distribution of improvement ratio for R+
0.9
percentage of successful improvement
0.8
0.6
0.5
0.3
ENB/h
ENB/C
airfoil
CCS
wine/r
wine/w
CCPP
protein
SC
CT
MSD
dataset
Figure 17: Distribution of Impovement Ratio. The box extends from the lower to upper quartile values of the data (each data point represents an independent experimental run), with a line at the median. The whiskers extend from the box show the range of the data. Flier points are those past the end of the whiskers, indicating the outliers.
Table 7: Summary of experimental results with Random Forests (RF)
Dataset
Methud
RMSE
NLPD meennisd
Nouse Variance
Time
RMSE
seanzzadd
(60c)
menndsstd
yacht
RIO
RF
135 1.5t
0.80--0.1
1,177 4- 0.187
0.49
14.93
0.55
ENB/h
KIO
RF
0.42 16f
0.79 0.13
1.100-0.13
0.61
16.06
0.35
1.22006711
1.83:20.5531
12.41
1.300
1.65==0:2677
X
Y+O
1434:0.3311
1881
2.29
16.43
Y+O
1.3-1
1.792
1.88
17.57
Y-10
20112022011
0.268220119
0.65
18.40
Y41
088-0.1111
1,280 11111
0.73
20.51
SVGP
4.41
1896120099611
15.44
SVGP
2.13
2.199:004473
12.40
55.18:00:3411
4.97-0.291
32.40
ANP
1.50
1.79520.88711
40.82
ANP
2.45
102.3
ENBY
RF
1991E011511
1884
0.31
sirfoal
RF
368201171
0.36
R+]
1.53:10.12
0.805
2.08
12:95
R+1 0
2.50110054
7.88
17.19
768
R+0
1.85 1411
0.89
2.03
10.62
1505
R40
0.54
2.60 0590
10.50
8.26
Y+0
1.1111
2.081:000011
3.20
19.82
Y+O
1.58
0.154
12.08
2.29:20.06711
3.31=0011
262110
SYGP
2.53 2.63-
2:00320.0/911
6.81
18.03
SVGP
3.39-020011
12.66
20.52
NNGP
19.320
50.14:20.8661
7704
NNGP
0,54
03.60-1--120-
3355
ANP
4.8112.1511
2698:20.54811
64.13
ANP
30,
539911631613
231.7
OCS
RID
7.17204111
5.84:10.34
0.160:10008
2488
17.72
winelr
RIO
0.625=00281
0.6422000 1.019200770
01.77
21.13
1030
R+1
8911035111
0.0991
25.34
14,62
1399
R-1
0625=
0
0.51
7.57
6:90:-11111
0627_00028
0.94410061
7.01-078811
0628=000281
Y-10
5.881003311
0.
25.78
22.01
Y+10
0.33
0.9001000443
0.28
24.88
SVGP
4011
1.3772000-81
44.62
18.00
SVGP
0264220008
0474
0140
20.41
4.11=01331
RIO
0.702-400015
1.088 4.025
0.43
36.60
K1O
4.22--..14
2801 0.002
15.18
44.81
1898
Rel
0.697 :: 0.016
1059
0.02511
0.44
33.17
9568
R-1
0.1
2,503 0.02
15:22
39,09
0.115
0.066t
4.1720.151
2339
13.26--6308
Y-10
0.710100117:1
1.000.0.02811
0.45
41.52
Y+10
4.631
2995
19331
3554
45.63
SYGP
0.71990000884
1081120.022
0.50
36,02
SNGP
4.36:01351
2087
185M
48.83
5.05100111
15.26202511
protein
RIO
4.55
2.95510.007
20.92
298.6
RIO
14.181075
4.072:0.0511
196.1
170.6
45730
R+1
4.5710.0411
0,0
21.06
278.1
21263
R-1
14 14.3811776
204.1
145.0
R+O
50022000441
0007:-0000111
24.79
342.9
R4O
1499804001
0.126:000411
213.
72.33
5.01110.111
14.00-0.3011
1
Y-10
4.561:00011
0.07710.00711
21 11:00
300.2
Y+10
14.3911.999
4.079::0.030
1969
234.4
SVGP
466--000411
0.062
22.54
281.1
SVGP
2466
4.1382001113
230.3
221.3
9.77:00.16f
312.0
9:3=000111
9.32-1..49
990+-001
53500
R+1
0.77 1611
0.0771
94.04
47.04
515345
R-l
9.0
37142000113
98.11
136.8
R4O
9.35
1654-100049
86.94
228.
R1O
9.92-00115
37144000172
97.77
1747
384
9.46-+-0.361
0.02211
448.7
9.91-00012
00011
Y-10
0.58-006611
367622.09411
9,94000011
SVGP
52.07-001771
0.37220
2712
61.70
SVGP
9.57=0.0011
3.007-4..0077
92.21
2276
The symbll i and 1 indicate that the difference bewwen the marked entry and RJO statistcallyy sigmificani at the 5% significunze level using paired t-iest and Wilcoson respectively. The best entrie that eignificantly better than the others under staticical test marked in holdface (iies allowed).
35


Reviews:
[{"review_id": "ryx5FWhcqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the model inference of neural networks (NN). The authors propose to use NN for the model, and fit the prediction residuals with a Gaussian process with input/output (IO) kernel. This kernel considers both input x and output y. The authors show that the NN+GP scheme has lower generalization error compared with solely using GP or NN to fit the model. Also, the IO kernel generalizes better than input kernel I, and output kernel O, in Gaussian process modeling. In experiments, the authors evaluate various methods in terms of several metrics to show that the proposed procedure gives better uncertainty estimation and more accurate point estimation. \n\nIn general it is a good paper, with good applications. The motivation is clear. The key idea of this paper is pretty common in statistical inference. \n1.\tIn more practical settings we cannot assume that NN is always trained well. In this case, does the proposed method perform much worse than GP? \n2.\tIs this the only proposal for fitting the residuals for uncertainty estimation? Is there any other similar approach? I would like to see more discussions on other related methods and how the idea is different. \n3.\tSummarizing the whole procedure in an algorithm could make things clearer.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Syge0X1BqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper solves an interesting scientific and applied problem: can we construct an algorithm to predict uncertainties without re-training/modifying existing neural network training algos? The authors propose a novel technique (called RIO) which leverages existing neural network but use both the input as well as the output of the neural net as an input to a GP which regresses on the residual error of the neural network. The authors describe the theoretical foundations as well as show empirical results on multiple datasets.\n\nMy thoughts on the paper:\n- The paper is well written and from section 2.1 it is clear how one could re-produce their method.\n- The theoretical section 2.2 feels a bit rushed, I think it would be worth sharing the high level intuition behind some of the theory first before going into the details.\n- Section 2.4 could be more explicit about what \"large scale\" means. I.o.w. from a practical point of view, the method is only limited by approximate inference for Gaussian processes. Anno 2019 this is ...\n- The empirical section is particularly strong and contains a wide variety of experiments with detailed analysis.\nAs a result, I think this is a good piece of scientific work that could be interesting to the wider community.\n\nAlthough I did not re-run the results, the authors do share full source code for their results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gTU-Ev5S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe authors propose a method for post-hoc correction and predictive variance estimation for neural network models. The method fits a GP to the model residuals, and learns a composite kernel that combines two kernels defined on the input space and the model’s output space (called RIO, R for residual, and IO for the input-output kernel). The authors suggest that residual means and variances at test points can then be calculated explicitly using predictive distributions from the GP. The authors run a large panel of experiments across a number of datasets, and compare to a number of methods that draw connections between neural networks and GP’s. In addition, the full method is compared to a number of methods that utilize only some components of the full RIO method. In these experiments, the RIO method generally shows strong performance in both RMSE and NLPD compared to these baselines.\n\n# Feedback\n\nOverall, this is a neat method. It has the flavor of a number of other composite ML methods that have worked well in the past---e.g., boosting and platt scaling---but is different enough to stand on its own. The experimental results are quite promising.\n\nHowever, I am torn about the paper, because the theoretical discussion of the method is quite convoluted and seems either irrelevant or incorrect. I wish that the authors had spent more time with small demonstrations of what the procedure does in some simple settings. This would give practitioners considering the method far more intuition about when they would expect it to work and fail than the current theoretical discussion.\n\n## Uncertainty Discussion is Lacking\n\nThe motivation and discussion sell this method as an uncertainty quantification method, but almost all of the theoretical development revolves around prediction correction. The methods properties as an uncertainty quantification tool are underdeveloped.\n\nThe only theoretical point made about uncertainty estimation is Theorem 2.7, which states that the scalar variance of the GP “nugget” is positively correlated with the variance of the NN’s residuals. Providing a scalar summary of noise is not particularly compelling for a method advertised as a point-prediction uncertainty quantification method. In addition, it is not clear what probability distribution the “correlation” is defined over. The argument made in the proof seems quite obvious: if a GP is used to model a noisier process (i.e., residuals with a larger variance), it will in some cases classify that variability as independent noise.\n\nIf the authors wanted to focus on the properties of their method as an uncertainty quantification tool, they could discuss the assumptions underlying the GP error estimates, and when they would be likely to diverge from practical properties like predictive interval coverage. For example, because the base NN predictor is treated as fixed, it seems that this method ignore uncertainty that stems from the NN fit due to random initialization. Likewise, it seems that this method would not quantify uncertainty from resampling the data and obtaining a new NN predictor. )The coverage experiments in the appendix seem to confirm this -- generally, the predictive intervals generally under-cover the predicted values.) It’s fine if the method doesn’t quantify these types of uncertainty, but discussion of these types of issues would be far more welcome than the current convoluted theory in Section 2. This discussion might not yield theorems, but it would give practitioners useful guidelines for deciding whether the particular scheme would likely work for this application.\n\n## Problems with the Error Correction Argument\n\nThe theory section, especially 2.2, was very difficult to parse. First, as a matter of style, a sequence of Lemma and Theorem statements are given without defining most of the notation used therein, and with almost no prose providing context or intuition. In the buildup to the theorems, it is also unclear which assertions about the decompositions of y_i are assumptions about the true data generating process, and which assertions are specifications of a particular GP model.\n\nThe substance also has some issues. I think the intention in this section is to get to a rather simple variance decomposition of the labels y. The question is how much variation in y or the residual is represented in the posterior predictive mean of a particular GP. It seems reasonable that in some cases, the structure in the residual may be more amenable to modeling with a stationary GP than the structure in the raw labels y. It is not clear that all of the theoretical complexity here is necessary to make this point.\n\nInstead, the authors make a convoluted argument that attempts to establish that the errors from the NN + GP approach will be smaller under very general circumstances. The argument is phrased somewhat ambiguously (it is not clear exactly what is being assumed, and what is corresponds to the specification of a working model), but depending on how one reads this section, the argument makes statements that are either too broad to be correct, or too narrow to be relevant.\n\nThe argument decomposes for the raw labels and the residuals into pieces that a GP can “capture” or “represent”, and parts that it cannot. The two equations are:\n\ny_i = f(x_i) + g(x_i) + \\xi_i\nR_i = (y_i - h_NN(x_i)) = r_f(x_i) + r_g(x_i) + \\xi_i\n\nf(.) and r_f(.) represent the portions of the label and residual processes, respectively, that the GP \"captures\". It is assumed that the GP will model this portion correctly, and leave the “epsilon-indistinguishable” portion g(.) or r_g(.) untouched. The argument then assumes that f(.) and r_f(.) will have proportional kernels, and so it is possible to show that the predictions of residuals based on r_f(.) will have smaller predictive variance than predictions based on f(.) as long as the variation represented by r_g(.) is smaller than the variation represented by g(.).\n\nOn its face, this argument raises some red flags. Because h_NN(.) is allowed to be an arbitrary function, the argument here should be symmetric. Why can’t we also get a guaranteed variance reduction by adding h_NN(.) to y rather than subtracting it? Perhaps some of this is captured in the parameter \\delta, which quantifies the reduction in variation represented in r_g(.) vs g(.), but the argument that the kernel of r_f(.) can be no larger than the kernel f(.) in terms of trace (that is, the proportionality constant \\alpha is not greater than 1) does not make sense. If h_NN(x_i) is simply -f(x_i), then these arguments would not go through. At the very least, conditions need to be articulated about the properties of h_NN(.).\n\nSome of the strangeness comes from the fact that this is a poor model of most prediction problems, where the main issue with fitting a GP is not “indistinguishability”, but misspecification. Consider a process y_i that is non-stationary; say g(.) has a linear trend in some component of x. A GP with a stationary covariance kernel fit to this process (such as RBF) will attempt to explain the variation due to the linear trend with a variance kernel that encodes long-range dependence. On the other hand, if this trend were removed by a base model like an NN, the residuals would have a very different structure (perhaps they would be stationary), and in this case, the GP would fit the data with a very different covariance kernel. \n\nUnfortunately, it does not seem like the formalism here can express a notion of misspecification at all. In the theory, it is assumed that the GP will only model the portion of the labels y_i for which it is property specified (in this case, f(.)). This generally does not occur in practice, as in the example above. It might be possible for this to apply in some circumstances, but the authors give no conditions (e.g., that the process y_i be stationary). Based on this assumption, the authors assert that the fitted GP to f(.) and r_f(.) will have the same covariance kernel parameters up to some proportionality constant \\alpha. Much of their theoretical argument depends on this proportionality. But this proportionality cannot apply in general, and again, no conditions are given for when we might expect this to hold.\n\nIt would be far more compelling if the authors proposed the very standard approach to modeling data via covariance kernels, where one first models non-stationary portions of the data with a base model, then models the correlation in the residuals with something like a GP. This is the bread-and-butter approach in, say, timeseries analysis (see, e.g., the Shumway and Stoffer textbook https://www.stat.pitt.edu/stoffer/tsa4/tsa4.htm), and the approach in this paper could be framed similarly.\n\n## Demos I Wish I Had Seen\n\nI wish the authors had presented some demonstrations of what the GP does to the fitted values of an NN. Giving a demonstration of how the output kernel modifies predicted values, for example, would give some nice intuition the value added by this portion. I suspect that this step essentially performs something like Platt scaling, but for continuous outcomes, by shrinking predictions together so that they better match the overall distribution of observed labels. Perhaps the mechanism is different. At any rate, it would be useful to understand where the information gain is coming from, and this would be far better expressed concretely in terms of a toy data example than the theoretical arguments that are given.\n\n## Coverage Experiments\n\nI wish the coverage experiments evaluating predictive intervals were included in the main text. As far as uncertainty quantification evaluations go, coverage is one of the few assessments that does not rely on the model itself (unlike NPLD, which uses the model’s own log-likelihood), and can be phrased as a concrete performance guarantee.\n\nHere, the goal for predictive intervals is to cover the true prediction value _at least as often_ as the nominal rate (95% intervals should cover the truth _at least_ 95% of the time), not merely that coverage be “close” to the nominal rate. This asymmetric evaluation gives you a concrete guarantee that the uncertainty estimate is conservative. The coverage experiments show that this method quite systematically under-covers compared to the end-to-end SVGP method, which generally satisfies this coverage property. I think this is important information to include about the model, and generally I think this behavior results from the fact that uncertainty is not propagated from the NN fit. This should be presented clearly in the main text.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx95fEwYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework (RIO) to estimate uncertainty in pretrained neural networks. For this purpose, RIO employs Gaussian Processes whose kernels are calculated by kernel functions of input and output samples and the corresponding target values.\n\n- The proposed approach is interesting and the initial results are promising. However, there are various major and minor problems with the paper:\n\n- The proposed method can be applied to any machine learning algorithm. It is not clear why you focus on employment of the proposed method for vanilla NNs. \n\n- Have you applied RIO for other learning algorithms as well? \n\n- Could you please explain more precisely, how you utilize which particular properties of NNs in RIO, and/or how RIO helps quantification and improvement of uncertainty of NNs particularly?\n\n- Following equation (7), you claim that “In other words, RIO not only adds uncertainty estimation to a standard NN—it also makes its predictions more accurate, without any modification to its architecture or training”. Could you please verify and justify how RIO makes predictions of NNs more accurate? In this statement, I guess that you consider the results given in Theorem 2.6. However, you should not that the error functions given in Theorem 2.6 are calculated in a cascaded manner, i.e., by applying a GP at the output of a NN.\n\n- The main proposal of the paper is that RIO makes it possible to estimate uncertainty in any pretrained standard NN. In order to verify that proposal, you should improve the experiments, esp. using larger datasets with larger neural networks, including deep neural networks.\n\nAfter Rebuttal:\n\nI read the comments of the other reviewers and response of the authors. Most of my questions were addressed in the rebuttal, and the paper was improved. However, there is still room to improve the paper with additional analysis using state-of-the-art algorithms on benchmark datasets, and to improve presentation of the work. Therefore, I improve my rating to Weak Accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to model uncertainty in deep learning regressors by applying a post-hoc procedure.  Specifically, the authors model the residuals of neural networks using Gaussian processes, which provide a principled Bayesian estimate of uncertainty.  The reviewers were initially mixed and a fourth reviewer was brought in for an additional perspective.  The reviewers found that the paper was well written, well motivated and found the methodology sensible and experiments compelling.  AnonReviewer4 raised issues with the theoretical exposition of the paper (going so far as to suggest that moving the theory into the supplementary and using the reclaimed space for additional clarifications would make the paper stronger).  The reviewers found the author response compelling and as a result the reviewers have come to a consensus to accept.  Thus the recommendation is to accept the paper.  

Please do take the reviewer feedback into account in preparing the camera ready version.  In particular, please do address the remaining concerns from AnonReviewer4 regarding the theoretical portion of the paper.  It seems that the methodological and empirical portions of the paper are strong enough to stand on their own (and therefore the recommendation for an accept).  Adding theory just for the sake of having theory seems to detract from the message (particularly if it is irrelevant or incorrect as initially pointed out by the reviewer).
Decision: Accept (Poster)