Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this work, the authors develop the discriminative jackknife (DJ), which is a novel way to compute estimates of predictive uncertainty. This is an important open question in machine learning and the authors have made a substantial contribution towards answering the question of "can you trust a model?" DJ constructs frequentist confidence intervals via a posthoc procedure. Throughout, the authors provide excellent background and exposition. They develop an exact construction of the DJ confidence intervals in Section 3.1. This is an intuitive approach that the authors explain well. Next, they explain and then develop the concept of higher order influence functions. They do a great job of communicating this concept. Section 3.4 provides the theoretical guarantees for DJ. The related work section is extensive and thorough. The authors have thoughtful experiments that demonstrate positive attributes of DJ. 

I suggest that this paper is weak accepted. On synthetic and real data, the DJ empirically works well, based on Figure 4 and Table 2. In addition, the theoretical exposition is very clear and compelling. The intuition provided in Sections 3.1 and 3.2 helps readers really understand what's going on, then 3.3 and 3.4 give theoretical justifications of the utility of DJ. 

However, I have a few suggestions for improvement that lead to the "weak" acceptance. First, I'll cover minor quibbles, then more major points. 

In Figure 1: At first, the blue dots and blue shading were not clear to me. In the legend, maybe explain that the blue shading indicates coverage and the blue dots indicate regions of higher/lower discrimination. 

In Equation 6: Looks like there is a misplaced parentheses. I think the last two terms of the equations should be Q(Vn-)+Q(Vn+) not Q(Vn-+Q(Vn+))

In Equation 7, and throughout: I think a better notation for the function would be \mathcal{I}^{(1)}_{\hat{\theta}} rather than \mathcal{I}^{(1)}_{\theta} since the influence function is a derivative with respect to the optimal parameters. 

Below equation 8: you should probably have an additional k exponent in the numerator of the kth order influence term, i.e. = \frac{\del^k \hat{\theta}_{i, \epsilon}}{\del \epsilon^k}

In Theorem 1: could you calculate this without \grad L(D, \theta), since L a function of \ell ? Maybe mention this in the appendix

It could be nice for an appendix study on how approximating the Hessian impacts performance for cases when we can compute the Hessian exactly. 

Table 1 could be expanded to include comparisons on computational bottlenecks and if there's retraining in these other methods. 

In Figure 4, please indicate the order of the IF used in the DJ procedure. DJ(m=?)? 

Major issues: 

Why weren't the other jackknife procedures used as baselines as well? I realize DJ has advantages compared to them, but an apples to apples comparison would be useful. For some researchers, LOO CV might not be prohibitive. This could be a chance to really sell your method: if it does well enough compared to more expensive LOO jackknife procedures, that would be a compelling reason to choose DJ. 

Could you please check this reference and let us know if it substantial is different from Influence functions that you develop? "Higher order influence functions and minimax estimation of nonlinear functionals" Robins 2008 DOI: 10.1214/193940307000000527 Robins et al. develop a way to compute higher order influence functions, which you do claim you're the first to do "to the best of your knowledge." 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper studies how to construct confidence intervals for deep neural networks with guaranteed coverage. The authors propose an algorithm, “discriminative jackknife”, based on the standard jackknife confidence interval estimate which they augment by a “local uncertainty estimate” based on the variability of the n leave-one-out fitted versions of the underlying algorithm (n = # data points). The whole study is concluded with toy and real-world examples showing the proposed algorithm is competitive with existing methods while also achieving the desired coverage.

I am currently leaning towards recommending rejection of the paper. The main reasons for this are: (i) A potential failure to cite and acknowledge the prior contribution of [1] which seems to have non-trivial overlap with this paper (on arxiv since end of July which is more than 30 days before the ICLR submission deadline and thus should be treated as prior work); (ii) The claims of guaranteed frequentist coverage are not backed up as, according to thm.2, they only hold when n >> 0 and the number of influence functions used goes to infinity (ideally, the authors would provide non-asymptotic bounds as in [1], but at the very least, these limitations should have been clearly pointed out and their practical implications discussed). Finally, I would like to say that I am holding the paper to a higher standard due to its 10 page length as instructed by the guidelines.


Major comments:

- Can you please explain the relation of this work to [1]? It seems that [1] already proposes use of the higher order expansions, provides an efficient implementation based on forward mode autodiff (do you plan to release code?), and moreover provides non-asymptotic bounds which are not present in your work?! 

- On a related note, Giordano et al. provide a careful analysis and discussion of the assumptions in their sect.4. Can you please clarify which of the assumptions you also make, and why you don’t need the others (if any)? 

- Throughout the paper (e.g., in and around eq.1), you seem to assume that there exists a unique minimiser of the objective which generally won’t be true (especially in your application to deep neural networks). In [1], the technical assumptions ensure this is true but I don’t see how this is handled in your case?! Can you please help me understand how to interpret your work in case there are multiple (possibly local) minima, and whether this has any effect on the results in thm.2?

- I think it would be beneficial to the reader if you could please provide a discussion and/or formula for how large n has to be for Theorem 2 to apply.

- On p.8, you say “To ensure a fair comparison, the hyper-parameters of the model f(x; θ) were the same for all baselines.” I am not sure I agree this is a fair comparison. Commonly, one has the opportunity to select hyperparameters for their algorithm (e.g., using a validation set, cross-validation, etc.) so it seems it would have been fair to run each algorithm with its best hyperparameters. Can you please provide a discussion of how this would affect the reported results?


Minor comments:

- On p.3, you say “We do not pose any assumptions on how the loss function in (1) is optimized.” Do you mean to say that you do not assume anything **but** that the chosen optimiser reaches a (global?!) minimum of the loss function? It seems like you would want to exclude pathological optimisers (e.g., one that always outputs zero) but also more realistically think about the known pathologies of commonly used optimisers (see, e.g., [2]).

- Can you please clarify if and how the use of the algorithm from (Agarwal et al., 2016) for approximation of the Hessian products affects accuracy of your confidence intervals?

- In fig.3, it seems like some of the methods are not properly tuned. For example, MC-dropout should not have zero uncertainty around zero (did you by any chance set bias variance to zero?!), and BNN-SGLD does not seem to have converged (can you please provide plots providing some evidence that the MCMC sampler has mixed + information about how the hyperparameters were selected?).

- I am somewhat confused by the statement “The only hyper-parameter involved in our method is the number of HOIFs m — this was tuned by optimizing the evaluation metrics ...” on p.8. Wouldn’t thm.2 suggest that you should use as high m as possible?

- Also on p.8, can you please clarify how you selected the threshold for the evaluation of “discriminative power”? In particular, thm.2 seems to suggest that what one would desire is that the ranking based on width of predictive intervals is equivalent to the ranking based on the actual prediction error. This would suggest, for example, comparing these two rankings using Kendall’s tau coefficient (perhaps a binned modification where one would count only the number of times true error puts a point into a different bin than the width of its confidence interval). Note that I am not suggesting the above method is perfect either (it completely ignores the actual sizes of the intervals), but I’m currently having trouble interpreting the results you report, so it would be very helpful to understand how you selected this particular measure of “discriminative power” and why alternatives like the example above were discarded please.


References:

[1] Ryan Giordano, Michael I. Jordan, Tamara Broderick. A Higher-Order Swiss Army Infinitesimal Jackknife. https://arxiv.org/abs/1907.12116

[2] Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. https://arxiv.org/abs/1705.08292
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors provide an interesting study on uncertainty estimation for deep learning for regression problems. The submission is well written and well structured w.r.t. the motivation and underlying theory, which are accompanied by extensive derivations/proofs in the annexes. However, experiments are presented for small-scale tasks only.

The title of the submission addresses deep learning in general. However, the submission specifically is limited to regression, which is only mentioned scarcely throughout the paper and only as early as Sec. 2. This limitation should be mentioned in title and abstract.  Also, even though deep learning is addressed and covered in principle by the theory, the experiments do not really seem to cover deep models, being limited to two-layer networks of limited size. In Annex D.3 and specifically Fig. 5, network depth is addressed, but virtually no further details or performance measures are given for these experiments.

Many of today's deep learning tasks go beyond simple classification and handle complex tasks based on e.g. sequential or higher dimensional data strongly exploiting context, being trained on huge amounts of data, and exploiting considerable deep models with huge numbers of parameters. The experiments presented here are limited to fairly small tasks and models of limited size and especially limited depth. It would be interesting to see a discussion on how the proposed approach is expected to scale w.r.t. the size of the data set and w.r.t. the complexity of the modeling. Also, it would be interesting, to understand how prediction in context would alter the described findings.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose using influence functions to efficiently estimate pointwise confidence intervals for regression models. Their central idea is to construct a confidence interval around a point $x$ as a function of how much the model $f$ changes around $x$ when individual training samples are left out. Their technical innovation is to combine a marginal error term that does not depend on $x$ (which ensures coverage) with a local variability error term that does depend on $x$ (to allow for greater variability in areas where the model is more uncertain and there is less data). The authors provide experimental support for the superiority of their method (in terms of coverage and discrimination) as well as theoretical support for consistency.

In my opinion, the ideas in the paper are exciting; the paper is clear and well-written; and the experimental evaluation is quite comprehensive in terms of baselines. However, I have concerns that prevent me from recommending an accept at this time:

1) Similar recursive formulations for HOIFs have appeared in the literature, so the claims in Section 3.3 should be toned down. See for example lemma 3 in Giordano, Jordan, and Broderick, 2019 (https://arxiv.org/abs/1907.12116); or for an older reference that’s a bit more specialized, see Debruyne, Hubert, and Suykens, 2008 (http://www.jmlr.org/papers/volume9/debruyne08a/debruyne08a.pdf).

2) The paper refers repeatedly to how their proposed method can be applied to deep learning models and, in particular, state-of-the-art deep learning models. In my opinion, the evidence in the paper does not support these claims. The largest experiments are run on neural networks with 100 hidden units, and it is not clear how to scale up their method to state-of-the-art models and large datasets. In particular, the Hessian-vector computations and the need to iterate through the dataset scale poorly with model and dataset size, and efficient approximations for these are non-trivial and an area of active research.

3) Related to the above point, the theorem statements apply to general differentiable loss functions. Are they true in such a general setting? For example, in Section 3.3, the assumption that the inverse Hessian needs to be positive definite is noted but this does not appear in the theorem statements. More importantly, it seems like some notion of strong convexity is required; for example, does the convergence of the von Mises series in Appendix A require that the parameters don’t change too much with $\epsilon$? This might not be true in a non-convex model. The paper also elides the fact that the global minimizer $\hat{theta}$ cannot in general be computed in non-convex models like neural networks.

The paper is otherwise compelling to me, and I believe that the above points can be remedied by being more careful and circumspect with the claims in the paper.
review_title: Official Blind Review #3

Meta-review: In this work, the authors develop a method for providing frequentist confidence intervals for a range of deep learning models with coverage guarantees.  While deep learning models are being used pervasively, providing reasonable uncertainty estimates from these models remains challenging and an important open problem.  Here, the authors argue that frequentist statistics can provide confidence intervals along with rigorous guarantees on their quality.  They develop a jack-knife based procedure for deep learning.  The reviews for this paper were all borderline, with two weak accepts and two weak rejects (one reviewer was added to provide an additional viewpoint).  The reviewers all thought that the proposed methodology seemed sensible and well motivated.  Among the cited issues, major topics of discussion were the close relation to related work (some of which is very recent, Giordano et al.) and that the reviewers felt the baselines were too weak (or weakly tuned).  The reviewers ultimately did not seem convinced enough by the author rebuttal to raise their scores during discussion and there was no reviewer really willing to champion the paper for acceptance.  Unfortunately, this paper falls below the bar for acceptance.  It seems clear that there is compelling work here and addressing the reviewer comments (relation to related work, i.e. Robbins, Giordano and stronger baselines) would make the paper much stronger for a future submission.
Decision: Reject