Title: Split conformal classification with unsupervised calibration
Venue: NeurIPS 2025
Accepted: yes
Keywords: conformal prediction

performance guarantees

conformal classification

unsupervised split conformal prediction
Abstract: Methods for split conformal prediction leverage calibration samples to transform any prediction rule into a set-prediction rule that complies with a target coverage probability. Existing methods provide remarkably strong performance guarantees with minimal computational costs. However, they require the use calibration samples composed by labeled examples different to those used for training. This requirement can be highly inconvenient, as it prevents the use of all labeled examples for training and may require acquiring additional labels solely for calibration. This paper presents an effective methodology for split conformal prediction with unsupervised calibration for classification tasks. In the proposed approach, set-prediction rules are obtained using unsupervised calibration samples together with supervised training samples previously used to learn the classification rule. Theoretical and experimental results show that the presented methods can achieve performance comparable to that with supervised calibration, at the expenses of a moderate degradation in performance guarantees and computational efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interesting new idea for using unlabelled calibrated examples for doing split CP. This comes with a worse coverage guarantee, but gives the benefit of using unlabelled data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- new method that enables calibrating split CP on unlabelled points\n- well-presented results\n- good theoretical analysis\n\nWeaknesses:\n- it's hard to understand in practice what is the price to pay in the validity guarantee (excess coverage gap)\n- no comparison against methods that try to achieve the same\n\nQUESTIONS:\n1)\nThis paper claims that the theoretical guarantee (validity) is \"comparable\" to that of split CP. However, there is no understanding of what the $G^F_{n,m}$ means in practical terms. What is its magnitude? Can you estimate its value?\nI fully appreciate the discussion provided in 3.2 surrounding this, but it's still very hard to understand what the validity guarantee means in practical terms.\nI think the claim that the proposal is comparable to split CP is unsubstantiated unless concrete numbers (or a more interpretable expression for this term) is given.\n\n2)\nThe method is only compared against split CP. However, there's numerous methods which try to achieve the same (i.e., to maximize the information available to CP's prediction sets with little data). I suggest a comparison is made with them (e.g., cross-CP and full CP) under the following setup:\n- the proposal works as usual, by seeing both training and unlabelled sets\n- the other two methods are only given the training data (i.e., no calibration data, for fairness).\nPresumably, the proposal should perform better, because it has access to additional data. However, it is possible that the other two better exploit the information that is available in the training set.\n\n3)\nThe fact that coverage guarantees are worse for larger $n$ and $m$ should be emphasized from the beginning.\n\n4)\nThe term \"probabilities\" is confusing (and wrong); a more appropriate one is \"confidence values\".\n\n>  the conventional approach can be seen as a particular case of Algorithm 2\n\nThis isn't correct. In Algorithm 2, the quantile is evaluated over a much larger multiset than the original algorithm, since it looks at all the possible labels.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a novel method to split conformal prediction for classification tasks, utilizing unsupervised calibration. The proposed method constructs set-prediction rules by integrating unsupervised calibration samples with supervised training samples previously employed to train the classification rule.

STRENGTHS AND WEAKNESSES:
Strengths:
- This paper is well-written and easy to follow.
- To the best of my knowledge, this paper is the first to investigate the use of unsupervised calibration for split conformal prediction.

Weaknesses:
- The paper lacks a proof of exchangeability between the calibration set and the test set, which is a critical assumption for the validity of conformal prediction methods.
- The experimental validation is limited to small datasets; extending the evaluation to larger datasets, such as ImageNet, would significantly strengthen the robustness of the findings.

QUESTIONS:
- If supervised training samples are directly incorporated into the calibration set, can the coverage guarantee still be maintained?
- Can eq (4) ensure that the calibration and training samples become statistically indistinguishable? Could you provide a more detailed explanation to support this?
- Does eq (4) ensure the validity of the formula presented on line 168?
- The coverage on the Letter dataset appears consistently low; is this within an acceptable range?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes a novel method for conformal prediction using unsupervised calibration data. The authors introduce a technique for assigning label probabilities to each calibration example by leveraging the labeled training data. These probabilities are obtained by solving an optimization problem that minimizes the integral probability metric between the distribution of labeled training samples and the pseudo-labeled calibration data.

STRENGTHS AND WEAKNESSES:
Strengths: The paper provides a solid theoretical foundation for the proposed method, including performance guarantees and computational complexity analysis. It is also well-written and clearly presented.

Weakness: Please check the questions.

QUESTIONS:
1-	Could the authors provide experimental results on runtime to compare the proposed method with supervised and naive unsupervised approaches?

2-	Theorem 2 states that the guarantees hold for any data distribution. Does this imply that the proposed algorithm remains valid even when the exchangeability assumption between test and calibration data is violated?

3-	How does the proposed method compare to existing weakly-supervised conformal prediction methods in terms of coverage and prediction set size?

4-	As noted in the paper, achieving a small coverage gap requires the function class $\mathcal{F}$ to accurately approximate the indicator function $\mathcal{X}_C$. Could the authors elaborate on how the choice of $\mathcal{F}$ affects the prediction set size in practice

LIMITATIONS:
The proposed method may not scale well to settings with a large number of labels. Its performance relies on the chosen function class $\mathcal{F}$ being able to accurately approximate the function $\mathcal{X}$,
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of obtaining a conformal prediction set where the conformal score thresholds are calibrated using an unlabeled calibration set. The proposed method learns class probabilities by matching the joint distribution between the labeled training set (X̃, Ỹ) and the pseudo-labeled calibration set $(X = x, Y = j)$, where each is weighted by its estimated probability $P(Y = j | X = x)$, measured using an integral probability metric (IPM). The paper then shows that, under mild assumptions on the function space used for distribution matching and on the relationship between this space and the conformal set indicator function, the proposed approach can provide bounds on the coverage rate.\n\nSTRENGTHS AND WEAKNESSES:\n(+) The proposed approach is sound and, to the best of the reviewer’s knowledge, novel. This is the first work to leverage the idea of joint distribution matching on an unlabeled calibration set to construct conformal prediction sets. The theoretical framework is compelling and naturally connects to the literature on two-sample testing, particularly via maximum mean discrepancy (MMD) (Gretton et al., 2012). The authors also extend their method to a kernel-based version, making this connection explicit and providing a practical implementation pathway.\n\n\n\n(–) That said, a key weakness of the paper lies in the lack of discussion of related work and practical guidance on method selection. Specifically:\n\n* Conformal prediction with unlabeled calibration data has been studied in prior work [1,2,3], A detailed comparison of assumptions, theoretical guarantees, and empirical performance would strengthen the paper: \n\n\n\n\n* Prediction-powered inference (PPI) [4] is a related framework where unlabeled data is paired with predicted probabilities to perform valid inference. While this paper does not directly use hard classifier outputs, the optimization still implicitly assumes reasonable prediction quality, creating conceptual overlap worth acknowledging.\n\n* The literature on classifier probability calibration [5, 6] focuses on adjusting individual predictive confidence. While such methods lack coverage guarantees like conformal prediction, they offer fine-grained, per-instance calibration that may be practically useful in similar contexts. A clearer discussion of the tradeoffs could help position this work.\n\n\n(1) Zhou et al., SemiCP: Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score \n\n(2) Han et al., Conformal Calibration under Distribution Shift\n\n(3) Lienen et al., Conformal Prediction with Weak or Credal Labels\n\n(4) Gibbs & Jordan, Prediction-powered inference\n\n(5) Guo et al. (2017), On Calibration of Modern Neural Networks\n\n(6) ull et al., Beyond Temperature Scaling: Obtaining Well-Calibrated Multi-Class Probabilities with Dirichlet Calibration\n\nQUESTIONS:\nAs discussed above, a central claim of the paper is that the proposed class-probability optimization does not require a highly accurate base classifier trained on the labeled data. However, the constraint involving the base classifier predictions (specifically the bound on Bp) appears to implicitly require that the learned probabilities align reasonably well with those from a classifier. Could the authors clarify: how accurate does the base classifier need to be in practice for the method to perform well? Is there a failure mode if the base classifier is poorly calibrated or highly inaccurate?\n\nIn addition, it would be helpful to include a more practical comparison with existing approaches for conformal prediction using unlabeled calibration data, as well as standard classifier calibration methods (e.g., temperature scaling, Dirichlet calibration, use splits of the labelled training set to calibrate). This would help contextualize the proposed method’s strengths and limitations in applied settings.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}