Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper focuses on the model inference of neural networks (NN). The authors propose to use NN for the model, and fit the prediction residuals with a Gaussian process with input/output (IO) kernel. This kernel considers both input x and output y. The authors show that the NN+GP scheme has lower generalization error compared with solely using GP or NN to fit the model. Also, the IO kernel generalizes better than input kernel I, and output kernel O, in Gaussian process modeling. In experiments, the authors evaluate various methods in terms of several metrics to show that the proposed procedure gives better uncertainty estimation and more accurate point estimation. 

In general it is a good paper, with good applications. The motivation is clear. The key idea of this paper is pretty common in statistical inference. 
1.	In more practical settings we cannot assume that NN is always trained well. In this case, does the proposed method perform much worse than GP? 
2.	Is this the only proposal for fitting the residuals for uncertainty estimation? Is there any other similar approach? I would like to see more discussions on other related methods and how the idea is different. 
3.	Summarizing the whole procedure in an algorithm could make things clearer.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: # Summary

The authors propose a method for post-hoc correction and predictive variance estimation for neural network models. The method fits a GP to the model residuals, and learns a composite kernel that combines two kernels defined on the input space and the model’s output space (called RIO, R for residual, and IO for the input-output kernel). The authors suggest that residual means and variances at test points can then be calculated explicitly using predictive distributions from the GP. The authors run a large panel of experiments across a number of datasets, and compare to a number of methods that draw connections between neural networks and GP’s. In addition, the full method is compared to a number of methods that utilize only some components of the full RIO method. In these experiments, the RIO method generally shows strong performance in both RMSE and NLPD compared to these baselines.

# Feedback

Overall, this is a neat method. It has the flavor of a number of other composite ML methods that have worked well in the past---e.g., boosting and platt scaling---but is different enough to stand on its own. The experimental results are quite promising.

However, I am torn about the paper, because the theoretical discussion of the method is quite convoluted and seems either irrelevant or incorrect. I wish that the authors had spent more time with small demonstrations of what the procedure does in some simple settings. This would give practitioners considering the method far more intuition about when they would expect it to work and fail than the current theoretical discussion.

## Uncertainty Discussion is Lacking

The motivation and discussion sell this method as an uncertainty quantification method, but almost all of the theoretical development revolves around prediction correction. The methods properties as an uncertainty quantification tool are underdeveloped.

The only theoretical point made about uncertainty estimation is Theorem 2.7, which states that the scalar variance of the GP “nugget” is positively correlated with the variance of the NN’s residuals. Providing a scalar summary of noise is not particularly compelling for a method advertised as a point-prediction uncertainty quantification method. In addition, it is not clear what probability distribution the “correlation” is defined over. The argument made in the proof seems quite obvious: if a GP is used to model a noisier process (i.e., residuals with a larger variance), it will in some cases classify that variability as independent noise.

If the authors wanted to focus on the properties of their method as an uncertainty quantification tool, they could discuss the assumptions underlying the GP error estimates, and when they would be likely to diverge from practical properties like predictive interval coverage. For example, because the base NN predictor is treated as fixed, it seems that this method ignore uncertainty that stems from the NN fit due to random initialization. Likewise, it seems that this method would not quantify uncertainty from resampling the data and obtaining a new NN predictor. )The coverage experiments in the appendix seem to confirm this -- generally, the predictive intervals generally under-cover the predicted values.) It’s fine if the method doesn’t quantify these types of uncertainty, but discussion of these types of issues would be far more welcome than the current convoluted theory in Section 2. This discussion might not yield theorems, but it would give practitioners useful guidelines for deciding whether the particular scheme would likely work for this application.

## Problems with the Error Correction Argument

The theory section, especially 2.2, was very difficult to parse. First, as a matter of style, a sequence of Lemma and Theorem statements are given without defining most of the notation used therein, and with almost no prose providing context or intuition. In the buildup to the theorems, it is also unclear which assertions about the decompositions of y_i are assumptions about the true data generating process, and which assertions are specifications of a particular GP model.

The substance also has some issues. I think the intention in this section is to get to a rather simple variance decomposition of the labels y. The question is how much variation in y or the residual is represented in the posterior predictive mean of a particular GP. It seems reasonable that in some cases, the structure in the residual may be more amenable to modeling with a stationary GP than the structure in the raw labels y. It is not clear that all of the theoretical complexity here is necessary to make this point.

Instead, the authors make a convoluted argument that attempts to establish that the errors from the NN + GP approach will be smaller under very general circumstances. The argument is phrased somewhat ambiguously (it is not clear exactly what is being assumed, and what is corresponds to the specification of a working model), but depending on how one reads this section, the argument makes statements that are either too broad to be correct, or too narrow to be relevant.

The argument decomposes for the raw labels and the residuals into pieces that a GP can “capture” or “represent”, and parts that it cannot. The two equations are:

y_i = f(x_i) + g(x_i) + \xi_i
R_i = (y_i - h_NN(x_i)) = r_f(x_i) + r_g(x_i) + \xi_i

f(.) and r_f(.) represent the portions of the label and residual processes, respectively, that the GP "captures". It is assumed that the GP will model this portion correctly, and leave the “epsilon-indistinguishable” portion g(.) or r_g(.) untouched. The argument then assumes that f(.) and r_f(.) will have proportional kernels, and so it is possible to show that the predictions of residuals based on r_f(.) will have smaller predictive variance than predictions based on f(.) as long as the variation represented by r_g(.) is smaller than the variation represented by g(.).

On its face, this argument raises some red flags. Because h_NN(.) is allowed to be an arbitrary function, the argument here should be symmetric. Why can’t we also get a guaranteed variance reduction by adding h_NN(.) to y rather than subtracting it? Perhaps some of this is captured in the parameter \delta, which quantifies the reduction in variation represented in r_g(.) vs g(.), but the argument that the kernel of r_f(.) can be no larger than the kernel f(.) in terms of trace (that is, the proportionality constant \alpha is not greater than 1) does not make sense. If h_NN(x_i) is simply -f(x_i), then these arguments would not go through. At the very least, conditions need to be articulated about the properties of h_NN(.).

Some of the strangeness comes from the fact that this is a poor model of most prediction problems, where the main issue with fitting a GP is not “indistinguishability”, but misspecification. Consider a process y_i that is non-stationary; say g(.) has a linear trend in some component of x. A GP with a stationary covariance kernel fit to this process (such as RBF) will attempt to explain the variation due to the linear trend with a variance kernel that encodes long-range dependence. On the other hand, if this trend were removed by a base model like an NN, the residuals would have a very different structure (perhaps they would be stationary), and in this case, the GP would fit the data with a very different covariance kernel. 

Unfortunately, it does not seem like the formalism here can express a notion of misspecification at all. In the theory, it is assumed that the GP will only model the portion of the labels y_i for which it is property specified (in this case, f(.)). This generally does not occur in practice, as in the example above. It might be possible for this to apply in some circumstances, but the authors give no conditions (e.g., that the process y_i be stationary). Based on this assumption, the authors assert that the fitted GP to f(.) and r_f(.) will have the same covariance kernel parameters up to some proportionality constant \alpha. Much of their theoretical argument depends on this proportionality. But this proportionality cannot apply in general, and again, no conditions are given for when we might expect this to hold.

It would be far more compelling if the authors proposed the very standard approach to modeling data via covariance kernels, where one first models non-stationary portions of the data with a base model, then models the correlation in the residuals with something like a GP. This is the bread-and-butter approach in, say, timeseries analysis (see, e.g., the Shumway and Stoffer textbook https://www.stat.pitt.edu/stoffer/tsa4/tsa4.htm), and the approach in this paper could be framed similarly.

## Demos I Wish I Had Seen

I wish the authors had presented some demonstrations of what the GP does to the fitted values of an NN. Giving a demonstration of how the output kernel modifies predicted values, for example, would give some nice intuition the value added by this portion. I suspect that this step essentially performs something like Platt scaling, but for continuous outcomes, by shrinking predictions together so that they better match the overall distribution of observed labels. Perhaps the mechanism is different. At any rate, it would be useful to understand where the information gain is coming from, and this would be far better expressed concretely in terms of a toy data example than the theoretical arguments that are given.

## Coverage Experiments

I wish the coverage experiments evaluating predictive intervals were included in the main text. As far as uncertainty quantification evaluations go, coverage is one of the few assessments that does not rely on the model itself (unlike NPLD, which uses the model’s own log-likelihood), and can be phrased as a concrete performance guarantee.

Here, the goal for predictive intervals is to cover the true prediction value _at least as often_ as the nominal rate (95% intervals should cover the truth _at least_ 95% of the time), not merely that coverage be “close” to the nominal rate. This asymmetric evaluation gives you a concrete guarantee that the uncertainty estimate is conservative. The coverage experiments show that this method quite systematically under-covers compared to the end-to-end SVGP method, which generally satisfies this coverage property. I think this is important information to include about the model, and generally I think this behavior results from the fact that uncertainty is not propagated from the NN fit. This should be presented clearly in the main text.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper solves an interesting scientific and applied problem: can we construct an algorithm to predict uncertainties without re-training/modifying existing neural network training algos? The authors propose a novel technique (called RIO) which leverages existing neural network but use both the input as well as the output of the neural net as an input to a GP which regresses on the residual error of the neural network. The authors describe the theoretical foundations as well as show empirical results on multiple datasets.

My thoughts on the paper:
- The paper is well written and from section 2.1 it is clear how one could re-produce their method.
- The theoretical section 2.2 feels a bit rushed, I think it would be worth sharing the high level intuition behind some of the theory first before going into the details.
- Section 2.4 could be more explicit about what "large scale" means. I.o.w. from a practical point of view, the method is only limited by approximate inference for Gaussian processes. Anno 2019 this is ...
- The empirical section is particularly strong and contains a wide variety of experiments with detailed analysis.
As a result, I think this is a good piece of scientific work that could be interesting to the wider community.

Although I did not re-run the results, the authors do share full source code for their results.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new framework (RIO) to estimate uncertainty in pretrained neural networks. For this purpose, RIO employs Gaussian Processes whose kernels are calculated by kernel functions of input and output samples and the corresponding target values.

- The proposed approach is interesting and the initial results are promising. However, there are various major and minor problems with the paper:

- The proposed method can be applied to any machine learning algorithm. It is not clear why you focus on employment of the proposed method for vanilla NNs. 

- Have you applied RIO for other learning algorithms as well? 

- Could you please explain more precisely, how you utilize which particular properties of NNs in RIO, and/or how RIO helps quantification and improvement of uncertainty of NNs particularly?

- Following equation (7), you claim that “In other words, RIO not only adds uncertainty estimation to a standard NN—it also makes its predictions more accurate, without any modification to its architecture or training”. Could you please verify and justify how RIO makes predictions of NNs more accurate? In this statement, I guess that you consider the results given in Theorem 2.6. However, you should not that the error functions given in Theorem 2.6 are calculated in a cascaded manner, i.e., by applying a GP at the output of a NN.

- The main proposal of the paper is that RIO makes it possible to estimate uncertainty in any pretrained standard NN. In order to verify that proposal, you should improve the experiments, esp. using larger datasets with larger neural networks, including deep neural networks.

After Rebuttal:

I read the comments of the other reviewers and response of the authors. Most of my questions were addressed in the rebuttal, and the paper was improved. However, there is still room to improve the paper with additional analysis using state-of-the-art algorithms on benchmark datasets, and to improve presentation of the work. Therefore, I improve my rating to Weak Accept.
review_title: Official Blind Review #2

Meta-review: This paper presents a method to model uncertainty in deep learning regressors by applying a post-hoc procedure.  Specifically, the authors model the residuals of neural networks using Gaussian processes, which provide a principled Bayesian estimate of uncertainty.  The reviewers were initially mixed and a fourth reviewer was brought in for an additional perspective.  The reviewers found that the paper was well written, well motivated and found the methodology sensible and experiments compelling.  AnonReviewer4 raised issues with the theoretical exposition of the paper (going so far as to suggest that moving the theory into the supplementary and using the reclaimed space for additional clarifications would make the paper stronger).  The reviewers found the author response compelling and as a result the reviewers have come to a consensus to accept.  Thus the recommendation is to accept the paper.  

Please do take the reviewer feedback into account in preparing the camera ready version.  In particular, please do address the remaining concerns from AnonReviewer4 regarding the theoretical portion of the paper.  It seems that the methodological and empirical portions of the paper are strong enough to stand on their own (and therefore the recommendation for an accept).  Adding theory just for the sake of having theory seems to detract from the message (particularly if it is irrelevant or incorrect as initially pointed out by the reviewer).
Decision: Accept (Poster)