Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of domain shift at test time. They consider how to use the shifted unlabeled test data to recalibrate the cutoff threshold. The method is shown to be valid for some natural distribution shifts. Their method is mainly based on QTC in equation 6. Validity is also shown in binary classification in section 5. Empirical methods are based on image net and breeds.
strength_and_weaknesses: Strength: this paper is clearly written. Most parts are easy to follow.

Weakness: 

1. The method proposed is not intuitive, I really double whether this method will work on multiple class classification problems theoretically. From the expression of 6, it seems it only based on the maximal logic, and unlike TPS and APS, based on all logic explicitly. A theoretical study beyond 2 classes is appreciated.

2. The experiments cannot justify a wide family of natural distribution shifts. More datasets should be discussed to show more comprehensive results.

In summary, the beauty of distribution-free uncertainty quantification is rigorous analysis and handy to use, but this paper missed these parts.
clarity,_quality,_novelty_and_reproducibility: Clear. See comments above.
summary_of_the_review: See comments above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors approach the recalibration of the prediction set in the conformal prediction procedure to overcome the distribution shift using only unlabeled data. Although this problem is intractable in general terms, it can be solved using the assumption about the family of shifts used. The authors propose a simple method Quantile Thresholded Confidence (QTC) which proposes to estimate the quantile on the target distribution on the desired level, after which a parameter beta is calculated to calibrate the predictor on the calibration data. To calculate threshold on source data, it is proposed either to search for parameter beta and estimate threshold from it, or to estimate threshold directly.
strength_and_weaknesses: Strengths:
* The work is neatly written and easy to read.
* The problem presented is relevant, which is confirmed by many examples of the lack of guarantees when the test distribution is shifted compared to the calibration set. 
* For the experimental study, authors use variations of ImageNet, which is a large and challenging dataset, which makes the obtained results rather strong.
* The paper also presents a comparison with a number of baselines and an extensive theoretical study of the proposed method.

Weaknesses:
* While reading is easy the main idea of predicting the quantile is not explained enough. To the best of my understanding, you characterize datasets by some features and try to predict the quantiles with some predictor based on this features. It is not that easy to grasp from the paper. I recommend giving some example in the main body of the paper.
* Are there any theoretical guarantees for the resulting method? Why do we expect it to work well?
* While experimental results are promising, I am really curious how stable is the solution. It worked well on give example but what will predictor output for some very new data (with more significant shift for example).
* The method is a direct adaptation of the analysis of Garg et al. 2022, so the contribution is limited.
clarity,_quality,_novelty_and_reproducibility: The presented method is simple both in terms of explanation and in terms of use, though some improvements (example) are needed (see above). The step of using only unlabeled data to calibrate conformal prediction is interesting, though its robustness is questionable. Perhaps I would suggest plotting similar figures for smaller datasets such as the cifar-10/100 to strengthen the results, namely whether the same positive effect of recalibration on a smaller amount of data would occur.
summary_of_the_review: The paper is well written and the results presented are interesting, but I have serious doubts on robustness of the method. No theoretical guarantees are given which is an important part of success for conformal prediction.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Conformal prediction methods guarantee validity only when the the calibration set comes from the same distribution as the test set. This paper aims to construct a more reliable conformal predictive set under distribution shift with unlabeled data. They consider predicting the cutoff threshold for a new distribution based on unlabeled data only. Experiments are conducted to demonstrate the performance of the proposed method in reducing the coverage gap.

strength_and_weaknesses: Strength:
Appropriate experiments are conducted with some comparative methods.

Weaknesses:
-- The coverage is not exact. Actually, the proposed method is even not guaranteed to achieve the desired coverage probability asymptotically, which largely limits the contribution and novelty of this paper.

-- I am curious about efficiency about the recalibration method. This paper does not provide enough analysis and experimental results about efficiency. Maybe the authors can provide some experiments to compare the size of the predictive sets of different methods (on synthetic or real data sets). Simultaneous analysis of validity and efficiency is more informative and meaningful.

-- Comparisions in more datasets with conformal methods under distribution shift are recommended. The current version only provides comparison with some covariate-shift-based conformal methods in Appendix B. From the results (e.g., Figure 5), the performance of the proposed method is not convincing.

-- It is not clear which one of the three recalibration methods we should use in practice. From experimental results, each one may fail to achieve the desired coverage on target data (the gap may be large). Can the authors provide more discussions and practical guidance?

-- Why in Figures 7 and 8, are only results of QTC-ST provided? From other results, QTC-ST seems more likely to perform worse than the other two recalibration methods. Please provide all relevant results.

-- The authors only provide some theoretical results under a toy example. For some other types of distribution shift such as covariate shift (unlabeled data seems sufficient), can the proposed method guarantee the coverage on the target data?

-- What is the main difference between the proofs in this paper and that in Garg et al. (2022)? Please state clearly about the additional effort required in this work.
clarity,_quality,_novelty_and_reproducibility: The paper is well organized, and the codes are available.
summary_of_the_review: Though the paper proposes a simple method to obtain a predictive set under distribution shift, the experimental results are not very convincing. Moreover, the coverage is not guaranteed, which is a big issue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for recalibrating a conformal predictor on a shifted data distribution with only unlabeled data. To this extent, they intend to estimate a confidence level beta that, if used for calibration on the original data distribution, would give empirical coverage 1 – alpha (the target coverage) on the target distribution. This is done using a unlabeled confidence-score.
strength_and_weaknesses: Strengths:
- Thorough introduction and problem statement and nice review of work on accuracy estimation with distribution shift and conformal prediction with distribution shift.
- The authors tackle two important problems in my opinion: First, recalibration under distribution shift, second, calibration with unlabeled data.
- Realistic experiments on ImageNet.
- Theoretical analysis based on prior work that may give some indication why the method works.

Weaknesses:
- The paper is a bit notation heavy due to the strong reliance on thresholds, confidence levels and quantiles and I feel giving a bit more intuition on the calibration processes could help the reader.
- Notation-wise, the opposite inequalities in (3) and (4) are confusing and I do not understand why 1 – alpha is used in (7). Generally, of course, there is some duality (e.g., replacing the 1 – tau with tau in equation (3) will correspond to calibrating against alpha instead of 1 – alpha), but it seems that this is not consistent across equations.
- Figure 1 is aimed at illustrating the method, but it does not work well for me. First, the steps in the caption where confusing at first (are these text or part of the caption) and context for the confidence levels and thresholds is missing, i.e. the figure is on its own not helpful. Also, I am missing the connection to the calibration with TPS/APS with labels.
- Some related work is missing in my opinion, e.g., online adaptive calibrators such as [a] that I feel are very similar.

[a] https://arxiv.org/pdf/2106.00170.pdf

- For the method, I think that I am missing an important point. The calibration on P is done with labels, correct? Meaning, we calibrate the threshold tau^P such that coverage 1 – alpha on P is obtained and the APS/TPS scores are used for that. However, there is no intuition or connection whatsoever between calibration of APS/TPS with labels and the unlabeled score s on P. For APS, the score is not even similar (while for TPS, s is just the maximum conformity score). This seems to be an integral part because calibrating against alpha on P using s might not mean anything for the actual TPS or APS scores. This is of course because the maximum confidence might not correspond with the true label.
- I am also slightly confused by the QTC-ST baseline. For the conformity score being the maximum confidence, this works. But if my scores are logits, equation (5) does not seem to work since tau^P could be outside of [0, 1] while the confidence levels are in [0, 1]. For tau^P > 1, equation (5) should not work/be meaningful.
- In section 3.2, it is also extremely difficult to follow why we are suddenly interested in regression baselines that predict accuracy. I expect that these are used to predict coverage from which beta is derived? The experiments section does, unfortunately not provide any more details.
- I am also missing a clear statement that this approach does not provide any guarantee. Often, coverage is associated with a guarantee and I think this should be discussed and stated accordingly.
- For the theoretical analysis, I would expect a proper statement in Theorem 1 beyond just saying “converges to 1 – alpha with high probability”. I feel without an exact statement, the toy example is not insightful. From the appendix, I deduce that this does not correspond to a standard coverage guarantee but, but a guarantee on that beta can be estimated reasonable close. However, only because beta is close, coverage does not need to be close, too. Or am I missing anything?
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: In its current form I am not fully convinced that this paper is ready for publication. There are several disconnects that are missing for me to fully understand the details and judge the method properly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of uncertainty quantification (UQ) of predictors using the conformal prediction framework when there is a change in the data distribution (source => target) using only unlabaled examples from the target distribution. This is a very important problem.

It proposes a method referred to as Quantile Thresholded Confidence (QTC) which estimates a confidence level beta that, if used for calibration on the source distribution, would give empirical target coverage 1 – alpha on the target distribution. This estimation is performed using a unlabeled confidence-score. QTC is inspired by and is an adaptation of Garg et al., 2022 which studied the problem of predicting out-of-distribution performance using unlabaled data. Overall, the paper is very well-written and ideas are easy to follow.

There is a consensus among the reviewers' that the paper is lacking in the following aspects and requires some more work before it is ready for publication.
1. The novelty of the QTC method is limited as it is an adapatation of Garg et al., 2022 for the conformal prediction setting. 
2. The key advantage of conformal prediction methods over prior methods for UQ is strong theoretical guarantees. There is some theoretical analysis on a toy setup, but lacks strong and general theoretical analysis.
3. Experimental evaluation could be more comprehensive by testing on a wide range of models and distribution shifts to understand the robustness of the proposed recalibration method.

In my mind, #1 is not a major concern if #2 and/or #3 was addressed sufficiently well. I'm recommending to reject this paper and strongly encourage the authors' to address at least one out of #3 and #2 for resubmission.
Decision: Reject