Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors the Locally Valid and Discriminative (LVD) method to construct confidence intervals. The idea has three steps. (1) First is to take the embeddings from a neural network and train a kernel regression model on the task, learning a Gaussian kernel $K$. (2) Second is to, on a separate held-out "conformal" set, compute the residuals between the model and ground truth labels. (3) Third is to, at inference time, compute the empirical desired $(1-\alpha)$ interval on the "conformal" set residuals, where each residual is re-weighted according to $K(x', x)$ where $x'$ is the query of interest. LVD has the appealing property that it is compatible with any (potentially pre-trained) deep learning model, with no further further training necessary.

The authors prove that under appropriate assumptions LVD achieves *local coverage*, i.e. coverage not only marginal over the entire feature space, but at a particular query $x$. Empirically they demonstrate that LVD improves upon popular neural network predictive uncertainty methods on the well-known UCI datasets. They evaluate in terms of Mean Absolute Deviation, Marginal Coverage Rates, and Tail Coverage Rates (which is defined as the coverage rate for the left and right 10% of the tails of labels $y$).
main_review: On the whole I was quite impressed with this paper. The authors write clearly throughout. Experimental setup is reasonable, and Figure 1 in particular provides a succinct picture for their motivation.

I was left with a few questions and suggestions:

1/ How does the smoothness requirement affect the confidence interval estimates? It would be interesting to repeat the procedure of say Table 3 and 4 as an ablation study to determine its necessity.

2/ It would be helpful to explicitly define Mean Absolute Deviation (MAD) in Section 4.2. 

3/ Pearce et al. ICML 2018 directly measures the Mean Prediction Interval Width instead of the MAD, as a measure of discrmination. I'd be curious to see that added as a sub-table of Table 4 (perhaps in the Appendix), to compare the two evaluation metrics.

Miscellaneous comments, typos:

1/ It looks like references 18, 19 are duplicates of each other.

2/ Line 221: "Conforamlized Quantile Regression".

--

Update: I thank the authors for their response, and for pointing me to Tables 7-10 in the Appendix in particular. After reading through their response, and the other reviewers' comments, I maintain my recommendation that the paper be accepted.
limitations_and_societal_impact: Yes, the authors adequately addressed limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the problem of providing prediction intervals in the deep learning setting, leveraging the recent work "Conformal prediction with localization" by Guan [2019]. The prediction intervals provided by this framework are simultaneously marginally valid, as well as conditionally valid in a certain sense, unlike other conformal methods. 
main_review: The empirical study done by the paper is extensive and has not been done before. However, the paper **significantly** overstates the novelty in terms of theory and methodology. The main algorithm and theoretical results heavily rely on the work of Guan [2019, 2021], as can be verified by looking at the proof in Appendix A. In particular, the conditional-style coverage result (equation (4) or Theorem 3.1) is not novel. The way the paper is written strongly indicates otherwise. I expand further on this point in the 'Writing' section below. **I strongly suggest that the authors transparently discuss the relationship to Guan's work, starting with a citation in the abstract.** I request the authors to correct me if I am missing some important theory developed in this paper which was not present in Guan's work, in which case I will reconsider my assessment.

_Strong empirics_:

As far as I know Guan's proposed method of localization, although having very nice theoretical properties, has not been empirically validated in an extensive study. This paper provides such a study for real world datasets for deep learning models; Section 4 confirms that the localization framework leads to discriminative prediction intervals that are marginally valid as well as have better conditional performance. The authors have considered many standard real world datasets, compared their approach to multiple other UQ methods (conformal and non-conformal), and used multiple meaningful evaluation metrics. They comment also on the time required for training and prediction of their method vs other methods. 

_Methodology_: 

This paper discusses learning an adaptive kernel based on the Nadaraya-Watson estimator. Guan's paper considered the localization function or kernel as fixed; thus learning the kernel using data is a new methodological contribution.

Once the kernel is learnt, from what I can tell the conformalization technique is exactly the same as Guan's paper, and so are the theoretical guarantees. Is there further development regarding the conformal methodology or theory itself which I have missed? 

_Writing_: 

The writing is clear and easy to follow. However, I feel that there are two significant issues which make the writing unscholarly to the extent that it has affected my overall score.

- **The paper seems to significantly downplay the dependence of their methodology and theory on Guan's paper.** The first mention of Guan's paper appear in passing in Section 2.2, which I found very strange. Lines 54-56: "... most conformal methods are only marginally valid [24, 37, 19, 4]. We however seek to construct a CI conditioning on the input." Is there a reason for not discussing the work of Guan [12] in this context? To a reader, it appears that this paper proposes a new solution to the conditional validity problem, whereas in fact the solution used is exactly the one proposed by Guan. Similarly lines 58-66 incorrectly suggest that the method proposed by the authors is new and this problem has not been solved before. Line 286 "This paper introduces LVD, the first locally valid and discriminative CI estimator for DL" also suggests that the conformal methodology is new. 

- The standard terminology for intervals that cover the output Y_{N+1} in _all_ of the conformal literature is ‘prediction intervals’, whereas ‘confidence intervals’ would attempt to cover some parameter of the distribution of Y, such as E[Y_{N+1} \mid X_{N+1}]. It seems that the authors have made a deliberate choice to use the term confidence intervals. Could the authors clarify the reason for this choice? If there is a strong reason, I suggest the authors discuss this in the paper. Else, I suggest to use 'prediction intervals' as is done in other papers on conformal. 

_Additional minor comments on writing and related works_: 
- The abstract is quite long, and contains many finer details and terminologies. These can be postponed to the introduction or discussion sections. I suggest streamlining the abstract and focusing on the 'big picture' to reach a wider audience. 
- Line 39: “Post-hoc methods are ideal because ...”. While this is common, I believe it is not a necessary property of post-hoc methods.
- Line 50: “Such methods usually use the prediction errors from a hold-out set to ...” Such ‘split’ conformal methods are currently popular due to their efficiency and validity despite their simplicity. However the statement of the authors is a misrepresentation of the well-developed body of conformal prediction where many methods that use all of the data exist: full conformal in Vovk’s 2005 book, cross-conformal [2], jackknife+ [3], and out-of-bag conformal methods [4]. 
- The paper [1] is highly relevant and should be cited somewhere.
- Eq. (4): Should X have a \tilde above it? 
- Line 112: Missing period ‘.’ after closing parenthesis. 
- Line 146: The definition of R_i may be somewhat abrupt for readers unfamiliar with conformal prediction

[1] https://arxiv.org/abs/1905.10634 

[2] http://proceedings.mlr.press/v91/vovk18a.html

[3] https://arxiv.org/abs/1905.02928

[4] https://arxiv.org/abs/1910.10562 
limitations_and_societal_impact: I don't think the method has any serious limitations more than other conformal methods. The authors briefly mention this in Sec 2.2.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new method for locally valid (e.g. marginal coverage guarantees) and discriminative prediction* intervals by connecting two ideas: kernel regression and conformal inference. They evaluate their method on a suite of small to medium scale regression tasks and find that the proposed approach is locally valid


*note that the authors refer to these intervals as _confidence_ intervals which I believe to be an error. CIs seek to cover a population parameter, which for regression would be the true, unknown, and underlying mean function (which can only be assessed via a simulation). What the authors propose are _prediction_ intervals which seek to cover draws from this distribution and are inclusive of distributional noise, c.f. this reference for more:

https://www.stat.cmu.edu/~ryantibs/papers/conformal.pdf 
main_review: ### Main comments:
I like the combination of kernel regression and split conformal inference. Overall, I found the method compelling and backed by a solid theoretical argument, though I found the experimental results somewhat limited, both in terms of the interval construct methods that were evaluated, deep learning models, and datasets. There also is a semantic confusion (see above) around confidence vs prediction intervals that warrants addressing. 

Additionally, the method relies on the assumption that neural nets (i.e. the embedding function) learns a mapping such that "similar" inputs end up mapping near to each other in this new space. I believe there is a lot of evidence in the literature for this assumption but it is worth making this assumption explicit.


### Detailed comments:

__Originality__: The work is original and combines two ideas to produce a novel uncertainty quantification procedure for deep learning models. Note, however, very similar ideas have been explore for non-deep learning models in the statistical literature, e.g.:

https://www.tandfonline.com/doi/abs/10.1080/01621459.2012.751873

__Quality__: The experiments are of high quality and I offer some suggestions for strengthening them below:

- The investigation into conditional coverage is limited to tail coverage and, given it is listed as one of the central contributions of the paper, I would like to see more empirical evaluations where other, non-tail subsamples are evaluated. 

- Given the close relationship between kernel methods and Gaussian Processes, it would be nice to include a GP in this comparison. Given the size of some of the datasets, I appreciate GPs could be expensive but it would be nice to see the comparison of how a 90% posterior interval from a GP performs.

- Evaluations on other kinds of architectures (e.g. CNNs) would also greatly strengthen the results.

- Prior evaluations of marginal coverage properties of Bayesian and non-Bayesian deep learning models should be acknowledged:

https://arxiv.org/abs/2010.03039

__Clarity__: The work is clearly presented and I found text and experiments easy to follow.

__Significance__: The work addresses an important topic and will be of interest to applied and theoretical researchers.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper focuses on quantifying predictive uncertainty through conformal prediction framework. While classic conformal guarantees are marginal (over calibration and test data), the authors aim to take a step forward towards conditional validity, which is itself an important question as typically a user prefers an uncertainty set that is conditional on a given input.
main_review: A question of conditional validity for the conformal framework has received a lot of attention over the past years. On the theory and clarity side, the paper contains significant flaws. I believe that even formalizing the problem, e.g., validity in section 2.2 requires revisiting. The writing can also be improved substantially as the paper contains a large number of typos. Examples include (there are many more):
1. Links to the equations are entered manually instead of using hyperlinks (line 87 --> "E.q. 2", lines 101 and 103).
2. Line 112 --> missing fullstop. 
3. I believe that the citation given on line 93 is incorrect. 
4. Equations are parts of the sentences (punctuation is needed).
5. In general, it is important to differentiate between confidence and prediction intervals not to introduce confusion. Authors focus on the latter while referring to the former. 

Finally, regarding the empirical evaluation, I am not convinced that presented simulations illustrate the conditional validity of the proposed procedure. Conditional validity is easy to study at least through simulated examples (see, for example, Romano et al. "Classification with Valid and Adaptive Coverage", which is in the classification setting but might point out relevant ideas).

Unfortunately, the current work seems not to be ready to be accepted to the conference. 
limitations_and_societal_impact: The question of uncertainty quantification has a huge impact on the applicability of ML models in real-world settings and thus is receiving a lot of attention in the research community these days. Unfortunately, the current work does not clearly explain the limitations of the proposed procedure, and the comparison with alternative frameworks is questionable.
review_title: null

Meta-review: Three reviewers indicated acceptance, two of them with clearly positive scores. Their main arguments in favor of this paper were good methodological contributions, convincing experiments, clarity of motivation and arguments, and a convincing rebuttal that added many details and addressed most points of criticism. On the other hand, there was one clearly negative review, but during the discussion period, the other members of the review committee (including myself)  had the impression that the authors' rebuttal addressed all these pints of criticism in a detailed and convincing way. So I recommend acceptance of this paper.
Decision: Accept (Poster)