Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies conformal prediction under covariate shift and presents a weighted version of jackknife+ for solving it.
strengths_and_weaknesses: Strengths:
- The problem this paper studies is significant.
- Overall, the paper is well written.
- The theoretical analysis looks solid.

Weaknesses:
- The major problem of this paper is its originality. The authors claim that the weighted version of jackknife+ proposed in [Barber et al., 2022] uses fixed and not data-dependent weights (and thus differs from the authors' proposed weighted jackknife+). However, in my understanding, Barber et al. (2022) consider conformal prediction in a more general setting (non-exchangeable), and therefore, the weights in their method are not specified. More importantly, I think the weights should reduce to those appearing in the proposed method of this paper if we consider the covariate shift setting (a special case of non-exchangeability), which implies the proposed method of this paper can be regarded as a special case of [Barber et al., 2022]. And obtaining these weights under covariate shift looks not difficult as long as we follow [Tibshirani et al., 2019] and [Barber et al., 2022].
questions: - The notations in the paper seem not consistent. For example, the quantile functions in Eq. (4) and Eq. (6) use different notations.
- Is the equation of likelihood ratio correct in Line 105, Page 3? Does X_i follow the training distribution or the test distribution?
limitations: None.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a new method, JAW, that incorporates importance weighting into Jackknife+. They demonstrate that JAW extends Jackknife+’s coverage guarantee to situations with covariate shift. Finally, they demonstrate the effectiveness of this method on experiments.
strengths_and_weaknesses: Strengths:
* An valuable work in developing the Jackknife+ framework and extending its utility to an important use case.
* Good technical contribution

Weaknesses:
* The method so far seems infeasible for larger-scale problems
questions: * In line 144-145: $\hat{\mu}\_{-i}$ includes $(X\_{n+1}, Y\_{n+1})$, but that doesn’t seem to be the case for the definition of $\hat{\mu}_{-i}$ in [1]. Are these supposed to be (slightly) different things?
* I may have missed this, but what are the values of $\beta$ for Figs. 3 and 4?
* Also regarding Fig 3., it seems that JAW and Jackknife-mm have comparable performance when using random forests, while Jackknife-mm can have much larger interval widths with neural networks. Is there an intuitive reason for this?
* Why are Naive, Jackknife-mm, and Jackknife missing from Fig. 4? Particularly Jackknife-mm as it seemed like the most competitive baseline in Fig. 3.

[1] Barber, Rina Foygel, et al. "Predictive inference with the jackknife+." The Annals of Statistics 49.1 (2021): 486-507.
limitations: * The main limitations of this method are adequately discussed in the work: in particular, the computational difficulty and the variance of JAW.
* This method, like other IW-based methods, requires knowing the covariate shift from train to test, which may not be applicable in many settings. Is there a way to incorporate weight estimates?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Jackknife+ is an approach to construct a predictive interval with coverage guarantee under the exchangeable assumption. This paper extends Jackknife+ for covariate shift. In particular, the paper proves that the proposed Jackknife+ Weighted with data-dependent weight (JAW) predictive interval satisfies the same coverage guarantee as the original Jackknife+ given the likelihood ratio. The paper empirically demonstrated that the proposed JAW predictive interval satisfies a target coverage level, while generating a small interval width, compared to six baselines. 

strengths_and_weaknesses: ### originality
Compared to Barber et al 2022 (which is the most related work), this paper removes the strong assumption on the likelihood ratio (i.e., the likelihood ratio is fixed) to satisfy the coverage guarantee; this is novel, though it would be more instructive to state key differences on the generalization from Barber et al 2022. The proof technique seem to be a combination of Tibshirani et al. [2019] and Barber et al. [2021], but anyway this is a novel combination.

### quality
This paper may theoretically sound and the empirical results support the claim.

### clarity
Overall clearly written, but the following may enhance the readability.
* Barber et al 2022 also proposed the Jackknife+ extension for covariate shift, so it’s better to introduce it in Section 2 for clear comparison.
* line 145: I guess (X_{n+1}, Y_{n+1}) is a typo; the algorithm considers the point up to n.
* line 165: it contains an incomplete math notation. 
* references to Appendix are broken.
* line 234: Jacknife-mm is not well defined; i is not a free-variable.

### significance
Extending uncertainty quantification approaches for covariate shift is required. But, the main concern lies in the usefulness of Jacknife+ in practical setups (i.e., it needs to keep n models to construct a predictive interval). So, extending Jackknife+ for covariate shift may not be a significant issue

questions: Thanks for introducing an extension of Jackknife+ for covariate shift.

* As mentioned before my main concern lies in the practicality of Jackknife+ (though I believe that Jackknife+ could be useful) and the benefit of JAW compared to conformal prediction for covariate shift. In particular, Jackknife+ requires keeping n models to construct a predictive interval; in the case of a large dataset like ImageNet, which consists of 1.2M images, this means I need to keep 1.2M resnet152 to run Jackknife+, which does not sound feasible. In this sense, split conformal prediction is a significantly appealing approach, thus its extension to covariate shift (i.e., weighted split conformal prediction (WSCP) by Tibshirani et al. [2019]) would be enough for conformalized prediction intervals under covariate shift. In other words, the motivation to use JAW seems to be weak, though Jackknife+/JAW themselves have unique algorithmic features. Why or when can JAW be more useful than WSCP? If WSCP is less sample-efficient (as it needs a hold-out set), what is the range of the number of samples where JAW is better than WSCP? In practice, we anyway need a holdout set for model selection or likelihood ratio estimation, then why not properly exploit this holdout set along with WSCP for conformalized prediction sets?

* I believe that WSCP is also a good comparing approach, but it is missing in the main experiments, though SCP is included. I think a comparison to WSCP is needed to quantitatively understand the benefit of JAW as a conformalized prediction set under covariate shift. 

* Barber et al. 2022 is clearly related work, but the proposed approach is not well contrasted to Babar et al. 2022. Why does Babar et al. 2022 make a stronger assumption (i.e., a fixed weights)? What are the main challenges on removing the stronger assumption? What's the key ideas on addressing the challenges?
limitations: The authors mentioned the high variance of coverage rate and computational overhead. In addition to this, the assumption that the likelihood ratio is known should be part of the limitations since otherwise JAW cannot get the desired guarantee.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors focus on introducing a new method named JAW to estimate predictive inference under covariate shift, and it achieves superior performance over the baselines on different data sets and architectures of black-box model. The proposed method is a weighted version of the jackknife+, and the weights are not a new technique. What's more, the idea of a weighted version of conformal prediction under covariate shift is not new, so the innovation of this paper is limited.

strengths_and_weaknesses: Strength：
  The authors analyze the problem scenario and the motivation clearly. 
  
Weaknesses：
  1. There seems to be an error in line 145, the model is trained with the ith datapoint removed, but the definition A(·) seems to be trained with the ith datapoint. And compared with \hat{\mu} defined in line 77, \hat{\mu_{-i}} seems have one more element, do authors mean \hat{\mu_{-i}(X_{n+1})}? Imprecise use of symbols makes definitions ambiguous.

  2. Many symbols are not formally defined when used. Although this is understandable due to space constraints, the authors had better add the note "as defined in [ ]" at least, or it will cause reading difficulties, and the reader needs to guess the meaning of each symbol according to the previous papers.

  3. The paper lacks innovation, the idea is based on [1] and the technique is from [1], then this technique is added to [2]. Hope the authors can put forward more analysis and improvement.


[1] Ryan J Tibshirani, et al. Conformal prediction under covariate shift. NeurIPS, 2019.

[2] Rina Foygel Barber, et al. Predictive inference with the jackknife+. The Annals of Statistics, 2021.
questions:  1. The style of the citations is different for different sections.

  2. There are writing issues in line 107.

  3. There seems to be reference error in line 109 and line 110, line 168
limitations: See above
review_title: null

Meta-review: metareview: This paper studies conformal prediction under covariate shift and proposes JAW, a weighted version of jackknife+ to solve this problem. There is a consensus among the expert reviewers that the paper considers an important problem and has substantial contributions that are deemed adequate for publication at NeurIPS2022. The authors provided a rebuttal that has sufficiently addressed the reviewers' concerns as acknowledged by Reviewers `qDpq `, `3P4m`, and `AAyc`.

confidence: Less certain
Decision: Accept