Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach for computing epistemic uncertainty. The proposed approach, DEUP, builds a new model (in addition to the original model) which predicts epistemic uncertainty, defined as generalization error minus aleatory uncertainty. I highlight some features of DEUP below:
- DEUP works with a hold-out dataset that is used for training the error predictor. 
- In case there does not exist a hold-out set or in interactive settings (like RL or active learning), DEUP is extended to be used in a cross-validation setting and the features used to fit the error predictor is extended to include data density estimates and model variance.
DEUP is evaluated on different settings including OOD data, sequential model optimization and RL.
main_review: Pros:

1. The paper focuses on estimating epistemic uncertainty of machine learning models. Estimating predictive uncertainty of black box models is an important research area with promises on enabling risk aware decision making, which will enable safe deployment of such models to production.
2. The proposed approach is practical, which is important for its adoption by others. Interesting extensions are proposed (e.g. data density estimation and model variance features) that helps with improving DEUP's performance. It also does not affect the original model training, which is a positive.
3. Experiments are conducted in different settings with compelling performance compared to the state of the art methods like deep ensembles and MC Dropout.

Cons:

1. The paper compares DEUP's uncertainty estimates to variance estimates obtained from other techniques. Variance (and standard deviation) provides intuitive explanations on what they are capturing (i.e. the dispersion of predicted values). Furthermore, they have nice links to prediction intervals, which also provides nice intuitions on the estimated parameters with their probability. The other techniques (e.g. ensembles and MC Dropout) that are indicated as yielding variance estimates can also be directly used to compute such intervals from their empirical predictive distributions. I think the proposed uncertainty estimates would benefit a lot from such intuitions. For instance, by training the error predictor to predict the square error loss, my intuition is that DEUP is predicting extreme quantiles. I would love to hear authors' intuition as well as how the uncertainty estimates map to prediction intervals.
2. Since the approach relies on a predictive model to estimate the errors, do the authors believe that DEUP will be susceptible to issues predictive models face (e.g. how important is model fit, how to prevent overfitting)? For instance, in the case of fixed training set where the set used does not capture OOD dataset and density features are not used, is DEUP's error prediction able to generalize to OOD?
3. Questions on experiments:
    1. Section 5.1.2: I like the OOD analysis conducted. I have some minor comments:
        1. CIFAR-10 with SVHN provides a very strong OOD dataset, which is also evident from high AUROC values. In such a setting, I am interested to hear why the authors think the rank correlation metric between predicted uncertainty and OOD generalization error is a good metric. I actually would be interested to see an analysis where OOD definition is more subtle/gradual. For instance, Ovadia et al. 2019 provides such an analysis with increasing distribution shifts. I would be very interested to see how the method performs in such settings.
    2. Section 5.1.3: The metrics used here can be improved (except log likelihood). For instance, coverage probabilities and CI widths would be great metrics to use here, which will also help with the intuition point made above. I am not able to fully grasp what Upper Bound on Corr. w. res. metric is capturing (as well as the Ratio metric).

Minor comments:
- Algorithm 1 requires an estimator of aleatoric uncertainty. Does the authors use that in any of the experiments and how do they define it?
- Appendix pages 19, 20, 26 has missing links (indicated with ??'s).
- Appendix Table 6: I would be very interested to see the performance of features by themselves in this table.
summary_of_the_review: Overall I vote for accepting the paper. I like the approach of directly estimating uncertainty. I have some concerns on providing intuition on the estimated uncertainty, which I believe could be addressed by some additional discussion and some more analyses. Hopefully the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to estimate the epistemic uncertainty (uncertainty due to lack of knowledge/data) at a new model input. The paper takes an indirect approach towards this goal by a) first estimate the generalization error at the new input, b) next estimate the aleatoric error (inherent uncertainty in the data distribution / irreducible error), and c) subtract aleatoric error estimate from the total generalization error estimate to obtain the epistemic uncertainty estimate.

The authors claim that this method captures both the uncertainty due to lack of data, as well as model misspecification in the process - while other/existing methods focus mostly on the variance of the posterior distribution (or its approximation) as a measure of epistemic uncertainty (and thereby implicitly assume the model is well specified.)

Estimating the total generalization error itself is performed with a second neural network model, which uses residuals obtained from the primary model as its labels. Estimating aleatoric error either assumes the presence of an Oracle or 

The authors apply their technique on both static (fixed data set) as well as interactive (active learning) settings, though mostly focused on the mean squared error loss function.
main_review: Strengths of the paper:

- Attempts to estimate the epistemic uncertainty in a creative way (total minus aleatoric).
- Methods cover both static and interactive use cases
- 

Weaknesses of the paper:

- Dependence on the Oracle for estimating Aleatoric uncertainty makes its practical use very limited IMO. While the approach bypasses some of the limitations of existing epistemic uncertainty techniques (e.g vulnerability to misspecification), the dependence on having an aleatoric uncertainty estimator or Oracle can be a major limitation as well.

- The generalization of the secondary model (the one that predicts total generalization error) itself feels to be on shaky grounds. It appears that the main difference in the secondary model (compared to similar works that employ a secondary model) is that it uses labels from out-of-sample examples. However those examples are (by definition) something already available at training time, and it's not clear how or why we somehow ended up with the ability to predict total error in OOD settings?

- The overall narrative / presentation is hard to follow. There is too much of back and forth between the main paper and Appendix. Some of the key elements of the paper (e.g Algorithm 2, which is likely of high interest to a majority of readers) is down in the appendix.
summary_of_the_review: The main factors influencing the recommendation are the strengths and weaknesses highlighted above.

The paper can be improved significantly by:

a) Adding narrative to help justify why the secondary network generalized to OOD settings where the primary network couldn't. Perhaps present results that shine light on just this sub-aspect.

b) Improving the overall flow and presentation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Given some supervised task, this paper redefines the uncertainty of a solution as its generalisation risk. The authors then propose to learn a secondary function to estimate the generalisation risk of the first. This is done by using held-out points as training data for the secondary model. In turn, this held out used to estimate the primary model error comes from a k-fold split.  The inputs to the secondary model are the data points being evaluated, estimates of the predictive variance of the primary model, density estimates from a generative model and whether the point has been observed by the primary model.

The authors posit that the advantage of their method is that it can capture uncertainty due to model selection bias, something omitted by existing work, which focuses on the variance of the learnt estimator. The authors provide a diverse range of experiments: OOD rejection in image classification, active learning for drug discovery, function optimisation and exploration in reinforcement learning. 
main_review: 
This paper provides a relatively novel approach to uncertainty estimation, which explicitly includes estimation of model bias.  With a few exceptions, I found the paper to be well written. I found it to be an enjoyable read, as it made me think about and question some of the assumptions traditionally made in the deep learning uncertainty estimation literature. For this, I think that this work can present a relevant contribution to the ICLR community. Having said this, I have some doubts/concerns which I outline below:

**Major doubts**

1. The paper proposes to redefine uncertainty in terms of generalisation error instead of predictive variance. I am not sure how to feel about this:

Pros:

* The definition is clearly stated. The authors clearly put effort into avoiding confusion regarding this point. Additionally, the new definition clearly motivates the proposed method: a sensible alternative to existing methods.
*  The authors provide clear intuition for this definition in proposition 1, relating their definition of uncertainty to more traditional estimates of uncertainty like variance for specific choices of data generative process.
	
Cons:
  * This redefinition could cause confusion in the deep learning uncertainty quantification field, which is an already not very rigorous space. 
  * The new definition is not always intuitive or does it provide interpretable uncertainty estimates. I think this is due to there not being an explicit predictive distribution. Consider a binary disease diagnostic setting: The primary model outputs a probability of disease of 87%. The secondary model outputs a risk estimate of 4.7. What does this mean?
	
I look forward to discussing this point with the authors and other reviewers.


2.  The authors motivate their approach by contrasting the sub optimality of the Bayesian update under model misspecification with the asymptotic unbiasedness off large neural networks:

	* The GP shown in the first toy example of the paper clearly has a strong preference over some functions over others. However, the rest of the examples in the paper use NNs. As stated by the authors these are asymptotically unbiased. In part, this is what makes Bayesian deep learning so difficult. As a result, it is not clear to me why bias would be such a large concern in the deep learning setting. 
	* In section 3.1, the authors claim that DEUP has asymptotically optimal properties (for large enough secondary models and held out sets). It is not clear to me how this is a relevant property: under this setting the main model would have no uncertainty and we would not need an error predictor. In the finite data setting, if the error predictor incorporates less assumptions than Bayesian Methods it will have larger variance. I dont think it is clear enough from the paper that there is no free lunch in this regard. 

As a result, I think that the claim made at the end of section 2 is unfounded: “directly estimating the epistemic uncertainty as the reducible part of the loss is superior to existing variance-based predictions of epistemic uncertainty.” I think the claim in the introduction is much more reasonable ”direct estimation of epistemic uncertainty can lead to more precise estimates than variance-based ones.” I encourage the authors to explicitly discus the above and relax their claims. 

3. I found section 3.2 to not be very clear. The authors state that variance-based uncertainty estimators can not deal with non-stationarity in the observation distribution arising from an active learning setting. It is not clear to me why this is the case: as more data is observed, independently of what distribution it comes from, the range of possible functions that fit all the datapoints decreases and so does the variance. What am I missing here? In fact, I found this section to be written in a somewhat confusing manner overall. I am not sure I understood the problem. I encourage the authors to re-write this section.
	


**Minor concerns** 


1. At the end of the introduction, the authors mention dropout and ensembles as approaches to approximate the Bayesian predictive variance.  While the former is a rather crude approximation, the latter does not target the Bayesian posterior at all. Perhaps HMC is a better example. 

2. It took some effort to understand Fig 2 because it kind of looks like the grey area corresponds to posterior variance while orange is bias. In fact, the bias and variance labels are for the straight and dashed arrows. Does it make sense to have this diagram? Is it needed to understand the rest of the paper? Perhaps It would be more useful if it were made more relevant to the proposed method DEUP: for example, the authors could state how DEUP targets \epsilon instead of \Delta in the plot. 


**Experiments** 


1. The proposed method is somewhat elaborate: It combines a k-fold error estimation, with a NN variance estimator (DUQ specifically) and a deep generative model to produce uncertainty estimates. It is somewhat unsurprising that this combination is able to outperform a variance estimator on its own.

2. Apart from the toy GP experiment, I do not think that any of the experiments of the paper target the main claim of DEUP capturing error from bias in addition to variance.

3. On the GP experiment: As the authors state, the GP is not allowed access to the held out data, while DEUP is. I understand this choice was made to drive home a point. However, it would have been interesting to see the setting in which the GP does have access to this data, as this setting better matches that of the rest of the paper. The Bayesian marginal likelihood could also be seen as a proxy for generalisation error and it would be interesting to see it compared against DEUP.

4. In the image classification OOD rejection experiment, it is not obvious to me why we see such a difference in results when using Spearman-rank correlation and AUROC as metrics. Do the authors have any intuition for that?

5. The remaining are very cool but non-standard for this type of paper. It's hard to say much about them as I am not familiar with the settings. 
	* Their diversity is a definite plus
	* They seem to be relatively toy tasks however. I am somewhat surprised by this as I would have guessed the neural-network based approach to uncertainty estimation allows for good scalability. 

6.  Why is the vanilla DQN baseline better than uncertainty aware methods in Fig 5?
summary_of_the_review: This paper provides a novel view on uncertainty estimation and is an interesting read. I believe it is held back by some overly ambitions claims and a lack of depth in discussion at some points. Experimentally the paper is again interesting but fails to drive home its main points. 

I currently can not recommend acceptance. If the authors improve their discussion and claims I will raise my score to a weak accept. If the authors provide some experimental evidence for the existence of bias in a realistic setting and DEUP accurately estimating that bias while variance-based methods to fail, I will raise my score to a strong accept.

**update**

After a discussion with the authors I have raised my score from 5 to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper uses out-of-sample prediction error as a measurement of epistemic uncertainty.
Using this definition, it develops an estimator: direct epistemic uncertainty prediction. 
The idea is to have a main predictor to learn the task, and an error predictor to predict the generalization error.
Empirical studies show that their proposed estimator produces better estimation on downstream tasks such as sequential model optimization and reinforcement learning. 
main_review: It's possible that there are some interesting ideas in this paper, but at this stage, I find it difficult to understand this paper.

The paper argues a different definition of epistemic uncertainty for the interactive learning settings but did not provide a clear argument on why their definition is more compelling than the existing ones. I suggest discussing how interactive learning is different from the standard setting, then motivating their definition.

The authors also seem to use out-of-sample generalization and out-of-distribution generalization interchangeability. For example, on page 1, it says ``The total expected out-of-sample loss at a point can be decomposed into an epistemic (reducible) part and an aleatoric (irreducible) part". Then on page 2, it says ``DEUP error predictors can be explicitly trained to care about, and calibrate for estimating the generalization error for examples which may come from a distribution different from the distribution of most of the training examples, i.e., an out-of-distribution (OOD) setting". out of sample generalization is about generalizing to the same distribution. Out of distribution generalization is about how to generalize to a different one. These are two very different concepts.
 
It's unclear what contributions 3  and 4 are supposed to mean.

Proposition 1 is not a result. It should not be a proposition. Is it supposed to be an illustrative example? The paragraph after proposition 1 is half-page long. I am guessing the goal is to discuss some mathematical insights? But it is impossible to parse. 

The algorithm itself is also difficult to understand not at all self-contained. What is the stopping criterion? What does the ``optional (or every few interactions only)" statement supposed to mean? What is x_{acq}?  If a and e are estimators, what does a-e mean? 
This is one of the main results of the paper. Yet it is not at all self-contained. A reader should not have to go to the appendix to understand the algorithm.


summary_of_the_review: It is possible that there are some interesting ideas, but the writing is extremely unclear. I cannot understand this paper.
review_title: null

Meta-review: The manuscript develops a novel method for uncertainty prediction that can be used in the context of active or reinforcement learning problems. They consider experiments such as an OOD Detection task wherein a ResNet is trained on CIFAR10 and predictions must subsequently be made for in- versus out-of-distribution (SVHN) data. 
The work develops an approach based on directly estimating epistemic (as opposed to a aleatoric) uncertainty by learning to predict generalization error and then subtracting estimated aleatoric uncertainty.
Reviewers found the essential approach to be novel and creative. However, there were several issues raised by reviewers that are not well addressed by responses by the authors. For example, Zaec worries about the dependence of the approach on an oracle for estimating aleatoric uncertainty. Multiple reviewers were concerned that this would make the approach unsuitable for many situations and thus limit the applicability of the ideas. 
Multiple reviewers also found the manuscript to be difficult to understand. I agree with the sentiment. While there may indeed be an interesting and important idea here, the text and explication of the algorithm and approach are complicated and leave the reader unsure about the contribution. I would recommend that the authors invest time and effort in simplifying and streamlining the narrative and presenting the technical innovation so that it is easier to judge. In it's current form, the manuscript is premature for publication.
Decision: Reject