Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: I have read the authors' responses to all reviews and ultimately elected to leave my score as it is (weak accept). I think the empirical results are strong, and while I am not as troubled by the motivation and framing of the work as reviewers 3 and 4, I think their more conceptual and methodological critiques have merit, dampening my enthusiasm for the submission.

-----

This submission proposes a model-driven data augmentation strategy that aims to improve the calibration and reduce the over-confidence of a variety of Bayesian neural network architectures when dealing with out-of-distribution (OOD) samples. It involves adding a generator network that aims to generate plausible OOD samples during training, along with an objective term that tries to force the predictor network to make high entropy (low confidence) predictions for these samples. The paper does a fairly thorough empirical comparison with ten datasets (eight regression, two image classification) and half a dozen baselines, most of which can be combined with PAD. The results indicate that PAD usually improves both calibration and accuracy by at least a small amount.

This is a solid paper: the proposed method seems sensible (if pretty complex) and appears to be modestly effective in the included experimental results. The introduction summarizes the paper's contributions as:
1. It proposes a model-driven data augmentation technique aimed at improving calibration and reducing over-confidence for OOD samples.
2. It adapts and extends the technique to regression problems, which the paper argues is unprecedented.
3. It demonstrates empirically that the proposed approach improves the OOD accuracy and calibration of four different strong Bayesian neural net models.

I lack the broad familiarity with the data augmentation literature required to verify claim (2.). I suspect that if this simple claim is true, then it may be trivially so: it's hard to believe that _no one_ has applied data augmentation to regression tasks, so perhaps folks haven't bothered to publish it. The authors can always modify or remove this claim, if needed. The other two contributions seem supported, although the empirical improvements are for the most part small (and probably not statistically significant?). I lean weakly toward acceptance: I would not oppose its inclusion in the ICLR 2021 proceedings, but I wouldn't enthusiastically endorse it. I'll explain below.

The paper's motivation as laid out in Sections 1 and 2 is sound: calibration and proper quantification of uncertainty are increasingly important in a wide range of applications where machine learning has real world consequences for safety, fairness, etc. What is more, existing techniques based on neural networks (increasingly widespread) do seem to suffer significant flaws, especially exhibiting overconfidence when they should not. The paper offers a diagnosis in the form of a conjecture (Section 2.2): "failure to revert to the prior $p(\theta)$ for regions of the input space with insufficient evidence to warrant low entropy predictions." Figure 1 effectively visualizes this phenomenon in a toy setting, but no further proof is offered. Further, the assumption that prior reversion is the correct thing to do isn't examined (though that's a basic tenet Bayesian modeling, so we'll set that aside).

The proposed technique seems sensible, if complicated: add a generator network to produce OOD pseudo-samples during training and penalize the prediction network for making high confidence (low entropy) predictions on these pseudo-samples. We can consider this a form of model-driven (vs. heuristic) data augmentation. The generator loss, given in Equation (5), looks correct to my non-expert eye, and I suspect it's immediately comprehensible to readers familiar with GANs, VAEs, and Bayesian neural nets. The intuition for the OOD samples resonates with me: they should be close enough to real data to be plausible but far enough away that the predictor would be unjustified in assigning a high confidence or departing from the prior.

The regularization term in Equation (7) is a bit more arcane at first glance, but it's intuitive: the conditional prediction distribution should be close to the prior for pseudo-data points far from the real training distribution. The derivations of the K-L term for regression and categorical classification are given in the appendix, but these aren't critical details for judging the significance of the paper (they're quite straightforward).

The design of the experiments is sound: they simulate OOD settings by clustering each dataset and using distinct clusters for training and test splits and measure both accuracy and calibration. I don't have an opinion about the choice of the Kuleshov metric for calibration. The chosen baselines look strong, but I am not up-to-date on the relevant literature so I would not be able to identify a non-obvious missing baseline.

The experimental results are promising. A PAD variant is usually (but not always) the best for each task and metric (exceptions include GP for Naval/accuracy and R1BNN for Power/calibration). Perhaps more important PAD does generally seem to improve both accuracy and calibration across both variants (DE, MC, SWAG, etc.) and datasets. So in other words, if a modeler chooses to use one of the compatible Bayesian neural networks, in most cases they should also use PAD.

The work and manuscript have a few weaknesses that prevent me from more strongly recommending acceptance. For one, some of the exposition around training is unclear, in particular, how the objectivees in Equations (5) and (8) are combined during training.

I praised the results above, but I think the manuscript's interpretation of its results (Section 4.3) is still more generous than mine. PAD does consistently improve accuracy and calibration, but the margin is sometimes small, raising the question aboout whether the added complexity is worthwhile in all cases. The paper argues that PAD consistently improves the calibration curves in Figure 3, at least for poorly calibrated models, but that does not seem obvious to me. This might be because the curves are somewhat cluttered, but I see a number of exceptions where the PAD look potentially worse: Energy, Naval (SWAG), and Yacht.

I think perhaps the real problem is not the results themselves, which overall are strong, but rather the manuscript's rather cursory discussion of the results and its failure to offer any insights or guidance about, e.g., when PAD should be expected to help (based on task or dataset) or which baselines it works best with.

One last note: I don't want to over-index on a toy figure included for illustration purposes, but I don't find the results in Figure 1 convincing! Perhaps I am misunderstanding what behavior we desire (if so, please correct me). I agree that the PAD distribution does a better job of capturing the uncertainty in the central low-data area, but at the left- and righthand ends, the baselines actually look preferable in that the uncertainty is often appropriately wider and contains or at least follows the true function. It looks like PAD might be over-regularizing things in these cases.

Here are some actionable suggestions for improvements:
- Clarify how the objectives are combined and how training proceeds. Consider adding, e.g., an "Algorithm" summary figure.
- Expand the results discussion beyond simply restating the results (which are displayed in the Tables). For the raw accuracy and calibration numbers, perhaps you could compute some summary statistics for the baseline vs. PAD differences, so readers could get a quick sense of whether PAD usually beats the baseline. Maybe also some counts for how often a PAD variant has the best performance.
- Also in the discussion, try to distill out some illustrative patterns that could be turned into insights or practical guidance.
- For the calilbration curve plots (Figure 3), consider reducing the clutter by removing redundant curves. For most of the tasks, most baselines (and corresponding PAD variants) are quite similar so perhaps you could show a representative subset for each task (and then put the complete figures in the appendix).

Here's a laundry list of questions:
- How does training proceed? Is it a typical alternating adversarial optimization, i.e., optimize generator, then discriminator, repeat?
- What additional computational complexity does PAD introduce during training?
- Under which conditions PAD should be expected to help most: type or distribution of data, task structure, baseline model, etc.?
review_title: Clever model-driven data augmentation solution for OOD predictions with promising results

final_score: rating: 3: Clear rejection
review_content: This paper proposes a data augmentation scheme (named PAD) to improve accuracy and calibration of NNs. The idea is to generate OOD data, close to the training data, where the model is overconfident, and force a higher entropy for their corresponding predictions.

This topic is very relevant to the ICLR community, the paper is clear, and I was excited with the goal in a first place. However, the paper as it is has major drawbacks.

1. The biggest drawback is that the proposed approach is ad-hoc, a heuristic with no guarantees that it will work as desired. In fact, recent work has shown that Data augmentation on top of Ensembles can be harmful, the authors should discuss this in the paper (see [Wen et.al, 2020: Combining Ensembles and Data Augmentation can Harm your Calibration). For this paper to be accepted, the authors should explore the properties of the proposed approach with careful controlled toy scenarios, and bring further insights on when the approach is expected (ideally, guaranteed) to work. 
2. Using PAD on top of other probabilistic approaches destroys the probabilistic interpretation.
3. Experimental results are extensive, but not convincing: Figure 1 lacks the GP reference, and shows bad performance on the left extreme; the Ablation study suggest that Equation (5) could be simplified; finally results in Table 2-4 suggest that the proposed approach hurts in high-dimensional scenarios (Energy and Kin8nm datasets), the reported numbers also strongly depend on model selection and tuning of PAD and other baselines, information which is currently missing.
4. The authors do not compare nor mention recent advances on calibrating DNNs, for example:
	* (Antoran et.al, 2020) Depth Uncertainty in Neural Networks
	* (Liu et.al, 2019) Simple and principled uncertainty estimation with deterministic deep learning via distance awareness
	

More comments:

* the proposed model does not seem to scale to high-dimensions, as "filling the gaps" with the OOD data generator becomes infeasible (this is reflected in the tables, where both accuracy and calibration are systematically worse for the high-dim Kin8nm dataset). Up to how many dimensions would this approach be useful?

* the OOD dataset produce an "equally sized pseudo dataset". Yet, one might think that the amount of data needed to robustify uncertainty would depend on the manifold geometry.
* location of OOD samples is chosen as an interpolation of latent representations for the observed data. That means that many generated datapoints will NOT bee out-of-sample.

* From the ablation study (Tables 4 and 5), "without AB" gives similar results to Regular (always within the reported error bars of "regular". That seems to indicate that terms A and B are not that relevant. Am I missing something?

* Figure 1: the authors should include one column for the GP behavior, since the authors claim that the observed behavior is similar to that. Otherwise, it is unclear by eye what is best. In particular, PAD

* Could the proposed approach suffer from the opposite issue, i.e., deliver too high uncertainty in the augmented OOD data? How do you avoid this issue?
* How does the proposed approach compare to a DNN whose last layer is GP or Bayesian RBF network? (see http://www.gatsby.ucl.ac.uk/~balaji/udl2020/accepted-papers/UDL2020-paper-009.pdf)
* The proposed method encourages a reversion to a *specific* prior (0 mean functions)

Minor:

* the authors mention limited expressiveness of GPs, but this is subject to a simple kernel. If the kernel is complicated enough, then GPs are as expressive as we would like to (see equivalences between DNN and GPs in [Neil, 1997] and [Lee et.al, 2017]). Please clarify this statement.
* Figure 3 is hard to read, I suggest to highlight the PAD curves by changing the color scheme).
review_title: A heuristic approach with no guarantees on performance

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overview:

The authors propose a data augmentation scheme that generates samples out of distribution and helps with uncertainty estimates. Comparisons are to various bayesian methods in uci regression and mnist/cfar for classification. PAD seems to give some improvements in out of distribution uncertainty quantification.

The major concern is that the gains seem relatively small, and the objective is ad-hoc. It would be nice to see either more substantial, uniform gains (so that the authors can justify the procedure on the results alone) or more solid conceptual motivation of the method, especially from the Bayesian side. It seems like the motivation and intro is clear, and section 3 onwards becomes very ad-hoc and loses much of this. It would be nice to be convinced that there are a set of assumptions and conditions under which this is the right way to do uncertainty quantification. 

Positives:

The evaluations are extensive, and it's commendable that they include both positive and negative results in their regression evaluations. 

Uncertainty estimation out of distribution is an important and timely problem.

Negatives:

Minor: I'm not sure why there is a claim that the problems with uncertainty estimation comes from p(theta|D) and not p_theta(y|x). The fact that non-bayesian methods have similar issues with uncertainty quantification would suggest that the latter is certainly an issue.

Figure 1 doesn't seem like a compelling argument for the narrative in the paper. MC dropout and deep ensembles both have decent behavior outside the support (x<0, x>1) but suffer in the gap between 0.25 to 0.75 which is arguably due to overaggressive interpolation.

Term A in equation 5 is justified as "generating data where f is overconfident" but I don't see how this is true. It's just generating data where there's low prediction entropy... this includes areas where it is confident for the right reasons.

Somewhat minor, but the sum of term A and term B seems a bit problematic, since A will be in terms of discrete entropy in classification, and term B is going to be differential entropy in general. Rescaling X also seems like it would arbitrarily shift the weights between AB and C? The weird thresholding on the C penalty for regression problems does not inspire confidence.

Overall, equation 5 gives a sense of a fairly ad-hoc criterion. I'd like to be convinced that this is actually the right way of doing things, especially from a Bayesian perspective.

Looking at equation (7) it seems like learning the distribution of tilde X is alot of work to regularize the KL towards the marginal with a squared-exponential penalty away from the training data. Is it really not possible to post-process the model distribution to achieve the same thing? 

The experiments are extensive, but a bit mixed. The dataset construction for regression seems like it would naturally favor PAD-type methods, because the clustering occurs on the basis of feature distances, and PAD enforces uncertainty based on feature distances (via the squared-exp term in equation 7). In terms of results, I think overall PAD gives gains, but it's not uniform and in cases like SWAG on table 2 seems to hurt more than it helps. 

The corruptions in MNIST / CIFAR must also be pretty aggressive, as the accuracy numbers are quite low for both. Does PAD do similarly well on milder or no distribution shift settings? I am slightly concerned that the evaluations here focus so much on the large distribution setting and that PAD is tuned to that case.

Minor:

Inline equation involving sines is missing a closing parenthesis.

Notation. g_phi is a 'generative model' in section 2 but it seems to be the output of an autoencoder in section 3.1 q_phi seems to be the actual generative model?
review_title: Would like to see a stronger conceptual motivation and defense of the objective function. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: I think the paper is interesting and well-written. I agree with the mis-calibration can be caused by out-of-distribution data, even though it is still commonly observed without such discrepancy. Addressing OOD data is an important direction, and I think the author proposed a reasonable approach to prevent models from overfitting on data points that are rarely observed during training. However, I believe there are some limitation of the method at the current stage, and the experiments did not fully convince me.

Strength:
1. The paper addresses an important question of models being over-confident on out-of-distribution data. The method is practical for applications where uncertainty estimation is needed. 
2. The adversarial generation of OOD data is intersting, and the rationale is well explained.
3. The authors included a good selection of datasets and experiments. The PAD-based methods are also compared to a good variety of baselines.

Weakness:
1. To determine whether a data point is out-of-distribution, both the adversarial model and main model rely on the L2 distance and the length scale parameter \ell. My concern here is about 1) the heterogeneity in different dimensions, and 2) \ell seems particularly important and difficult parameter to tune. I would like the authors to give more details on how it is chosen.
2. I believe the approach here is to revert to prior when evaluated data points are far away from the observed ones. Thus the difference in accuracy depends on how good the starting prior is, and how much bias the baseline model learned from OOD data. Table 3 shows some of the trade-off, but I think it also be good to show the difference when there's no OOD data, because it's not necessarily known in advance.
3. Looking at figure 3, I don't think the PAD method shows significant improvement in most of the datasets. Housing seems to be the only one here. In figure 5, it looks like the OOD data are mostly in the convex hull of observed data (at least in this low dimensional embedding). It is unclear how to differentiate those from the region where models should be interpolating. Moreover, all the OOD data are artificially constructed. I think it would be more convincing to test the methods on some OOD data that arises naturally. One such source could be temporal data where distribution could shift over time.

Minor comments:
1. In section 5 "excessive computation for large models an datasets", an->and.

-----------

Thank the authors for lot of these responses. I'm still around neutral for this paper, but I will raise my score to marginally above acceptance.





review_title: The paper improves the calibration on out-of-distribution dataset, by introducing a regularization term using pseudo OOD data generated during training. The proposed method shows improvement in negative log likelihood and calibration error on some dataset in the experiment, where prior is a better approximation than the over-confident baseline metohds.

Meta-review: This paper studies the problem of uncertainty estimation under distribution shift. The proposed approach (PAD) addresses this under-estimation issue, by augmenting the training data with inputs that the network has unjustified low uncertainty estimates, and asking the model to correct this under-estimation at those augmented datapoints. Results show promising improvement over a set of common benchmark tasks in uncertainty estimation, with comparisons to a number of existing approaches.

All the reviewer agreed that the experiments are well conducted and the empirical results are very promising. However, they also had a shared concern on the justification of the approach. Reviewers are less willing to accept a paper merely for commending its empirical performance.

I share the above concern as the reviewers, and I personally found the presentation of the approach a bit rush and disconnected from the motivation. For example, the current presentation feels like the method is motivated by BNNs but it is not clear to me how the proposed objective connects to the motivation. Also no derivation of the objective is included in either main text or appendix. 

In revision, I would suggest a focus on improving the clarity and theoretical justification of the proposed objective function.
Decision: Reject