Title: Class conditional conformal prediction for multiple inputs by p-value aggregation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conformal Prediction

multiclass classification

aggregation

conformal p-value

Pl@ntnet
Abstract: Conformal prediction methods are statistical tools designed to quantify uncertainty and generate predictive sets with guaranteed coverage probabilities. This work introduces an innovative refinement to these methods for classification tasks, specifically tailored for scenarios where multiple observations (multi-inputs) of a single instance are available at prediction time. Our approach is particularly motivated by applications in citizen science, where multiple images of the same plant or animal are captured by individuals.
Our method integrates the information from each observation into conformal prediction, enabling a reduction in the size of the predicted label set while preserving the required class-conditional coverage guarantee. The approach is based on the aggregation of conformal p-values computed from each observation of a multi-input. 
By exploiting the exact distribution of these p-values, we propose a general aggregation framework using an abstract scoring function, encompassing many classical statistical tools. Knowledge of this distribution also enables refined versions of standard strategies, such as majority voting.
We evaluate our method on simulated and real data, with a particular focus on Pl@ntNet, a prominent citizen science platform that facilitates the collection and identification of plant species through user-submitted images.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses class-conditional conformal prediction with multiple input covariates. Building upon the perspective of p-values and mathematical tools for their joint distribution, the paper proposes techniques to merge multiple prediction sets by aggregation methods including and beyond majority voting, which corresponds to different choices of non-conformity score functions, and reduces the size of the uncertainty set compared to the original majority voting. Under the critical assumption that the scores within the same class are exchangeable, class-conditional coverage is guaranteed. The technique is validated by experiments on both synthetic data and realistic data from citizen science, with simulated multiple inputs.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper has unified aggregation methods for prediction sets as forming an envelope around their p-values. Subsequently, it is well-motivated from the observation that the envelope for majority voting is not tight enough. The paper generalizes aggregation methods through various choices of score functions taking p-values as input. The resulting framework reduces the size of the prediction set by design, and is also empirically validated.\n\n### Weaknesses\n1. The critical assumption that the scores within the same class are exchangeable (Assumption 3.3) weakens the applicability of the technique. In the context of citizen science, it implies an impractical assumption that images of the same plant are exchangeable with images of different plants from the same species. The paper also reports that the exchangeability assumption does not hold for the LifeCLEF Plant Identification Task 2015 dataset. Therefore, exchangeability is enforced by random grouping of images of the same class. Empirical results on natural datasets without synthetic transformation would be more convincing to test the robustness of the technique when the assumption is violated.\n\n### Comments\n1. The core mathematical tool for the exact distribution of multiple values is mostly based on Gazin et al, though the results have been extended to handle ties between scores, and directly extended to be conditioned on classes.\n\nQUESTIONS:\n1. Is it possible to empirically validate the technique with natural datasets containing multiple inputs, instead of synthetic construction of multiple inputs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework for reducing uncertainty in class-conditional conformal prediction, specifically tailored to scenarios where multiple observations (e.g., images) of a single instance are available at test time. This setting is well-motivated by citizen science applications such as Pl@ntNet. The core idea is to perform class-conditional conformal prediction on each individual observation to generate p-values, and then aggregate these p-values to make a final prediction.\n\nThe authors first propose a refinement of majority voting by replacing the heuristic threshold with a quantile from the Beta-Binomial distribution. Second, they introduce a more powerful class of aggregation methods based on score functions defined directly on the vector of p-values. These methods—including scoring based on quantiles or the geometric area of the p-value trajectory—better capture joint information from multiple observations.\n\nThe authors validate their methods on synthetic data and the LifeCLEF 2015 plant identification dataset, demonstrating a significant reduction in prediction set size compared to baselines, while maintaining the desired class-conditional coverage guarantees.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper addresses uncertainty quantification in a setting where multiple inputs are available at test time, a scenario where standard conformal prediction is not directly applicable (as discussed in Appendix B).\n\n- The derivation of the joint distribution of ordered conditional p-values (Theorem 3.4) provides a solid foundation for Sections 4 and 5.\n\n- The paper offers two methodological contributions: (1) a refined majority voting scheme using the Beta-Binomial distribution (Section 4), which is a direct and elegant improvement over existing heuristics; and (2) a general framework for scoring p-value vectors based on simulation (Section 5), which flexibly captures the structure of joint p-values.\n\n- Experiments on both synthetic and real-world data show that the proposed methods, particularly the p-value scoring envelopes, produce significantly smaller prediction sets than baselines, while maintaining valid coverage.\n\n### Weaknesses\n- A major concern is the strong assumption of exchangeability among the multiple inputs (i.e., views) of a test instance. In real-world multi-view settings—such as images taken from different angles, lighting conditions, or crops—this assumption is often violated due to inherent correlations between views. For example, an image and its 90° rotation are not identically distributed and thus not exchangeable. The theoretical guarantees of the proposed framework rest on Assumption 3.3, which requires exchangeability between test-time inputs and the calibration data from the same class. To sidestep this issue, the experimental setup simulates multi-view data using independently drawn samples from the same class, thereby artificially restoring exchangeability. While this limitation is acknowledged, it raises significant doubts about whether the method can be reliably applied to realistic multi-view scenarios, particularly in cases where users submit similar or low-quality images.\n\n- The evaluation primarily compares the proposed methods to majority voting—a simple heuristic that does not take into account the scale of p-values. For a fairer and more comprehensive comparison, it would be valuable to benchmark against more advanced aggregation techniques.\n\n- Most plots comparing coverage and prediction set sizes (except for Figure 5) do not include any indication of statistical uncertainty (e.g., error bars or confidence intervals). This limits the reader's ability to assess the robustness and significance of the reported improvements.\n\n- While the framework is general, the empirical validation is limited to basic softmax scores. Demonstrating the method’s effectiveness with stronger conformity scores, such as Adaptive Prediction Sets (APS) [1], would reinforce the claims—especially since APS has been shown to improve conditional coverage.\n\n- In Figure 2, the refined voting methods (Beta-Binomial and Binomial) consistently exhibit over-coverage, providing coverage significantly above the nominal 90% level. A discussion of the causes and potential corrections for this behavior would strengthen the analysis in Section 4.\n\n- Some methodological choices lack justification. For instance, the choice of quantile level $\\lambda$ in the quantile envelope method (line 271) is not explained. Similarly, the decision to approximate the Beta-Binomial distribution with a Binomial (Proposition 4.2) is unclear, particularly given that Beta-Binomial quantiles are efficiently computable (e.g., via `scipy.stats.betabinom`).\n\n- While the main results are strong, additional metrics could provide further insights. Reporting the median prediction set size (e.g., in the Appendix) would offer robustness to outliers. Furthermore, evaluating coverage conditional on the input features (e.g., worst-slab coverage, as reported in [1]) could show whether the aggregation methods improve this stricter form of conditional validity.\n\n[1] Romano, Yaniv et al. \"Classification with valid and adaptive coverage.\" NeurIPS 2020.\n\nQUESTIONS:\n- **Line 49**: In your setting description, is *m* (the number of observations) considered fixed or a random variable? In practice, users may submit different numbers of images. Does the framework still hold if *m* varies per test instance, as long as calibration for the p-value scores (Algorithm 1) is done for that specific *m*? This should be clarified.\n\n### Typos\n- **Line 161**: “is” should be “are.”  \n- **Line 143**: “conditionnal” should be “conditional.”\n\nLIMITATIONS:\nThe authors are transparent about the main assumption underpinning their framework (Assumption 3.3), and they strengthen the paper by providing an empirical analysis in Appendix E where this assumption is violated.\n\nA second, less discussed limitation is the potential computational cost of the p-value scoring methods (Section 5), which require separate Monte Carlo calibrations for each class. This may become prohibitive in settings with a very large number of classes (K), although the calibration is a one-time offline process.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a framework to integrate multi-input information into conformal prediction to further improve the predictive interval.

STRENGTHS AND WEAKNESSES:
Strength: original; I also enjoyed reading this paper.

Weakness. If I understand correctly,Section 5 p-value score aggregation requires a Monte Carlo simulation to calibrate the quantiles of the score v for every class? For datasets with thousands of classes, this could be computationally prohibitive. The paper does not discuss the computational complexity or potential optimizations for this critical step.

QUESTIONS:
Your results for the LifeCLEF dataset show a significant difference in performance based on the temperature scaling of the softmax outputs. This suggests a strong interplay between the quality of the base classifier's scores and the effectiveness of your aggregation. Could you expand on this? For example, would applying a more sophisticated calibration method (like Ensemble Temperature Scaling) to the base classifier's scores before computing p-values further improve the reduction in set size?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a conformal prediction framework for multiple observations of the same instance, motivated by citizen science applications like plant identification. The authors aggregate conformal p-values from individual observations rather than predictions directly, preserving exchangeability assumptions. Key contributions include deriving the exact joint distribution of class-conditional conformal p-values and proposing two aggregation strategies: refined majority voting using Beta-Binomial quantiles and p-value aggregation methods with various score functions.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality:** The theoretical development is rigorous with proper handling of exchangeability assumptions and tied scores. However, the experimental evaluation lacks a critical baseline comparing against multi-view models with standard conformal prediction.\n\n**Clarity:** Well-written with clear mathematical exposition, though some technical details could be better integrated into the main text.\n\n**Significance:** Addresses a practically important problem with meaningful theoretical extensions to conformal prediction. Empirical results show substantial improvements within the tested framework.\n\n**Originality:** The p-value aggregation approach represents a novel contribution with original theoretical analysis of joint p-value distributions.\n\nQUESTIONS:\n**Missing Baseline:** The experiments lack comparison with multi-view models combined with standard conformal prediction. This baseline would involve end-to-end multi-view architectures applying standard conformal prediction to their outputs. Without this comparison, the relative advantages of p-value aggregation versus direct multi-view modeling remain unclear. Could you provide guidance on when each approach is preferable?\n\n**Exchangeability:** How sensitive are coverage guarantees to violations of Assumption 3.3? What diagnostic tools could practitioners use to assess this assumption?\n\n**Score Function Selection:** What guidance exists for choosing between quantile, area-based, and ℓq-envelope score functions under different conditions?\n\nLIMITATIONS:\nThe authors adequately address several important limitations, particularly the exchangeability assumption and its empirical violations. The computational requirements discussion is appropriate. However, more comprehensive baseline comparisons and specific practical implementation guidance would strengthen the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}