Title: Synthetic-powered predictive inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Distribution-free Inference

Conformal Prediction

Uncertainty Quantification

Domain Adaptation

Generative Models
Abstract: Conformal prediction is a framework for predictive inference with a distribution-free, finite-sample guarantee. However, it tends to provide uninformative prediction sets when calibration data are scarce. This paper introduces Synthetic-powered predictive inference (SPI), a novel framework that incorporates synthetic data---e.g., from a generative model---to improve sample efficiency. At the core of our method is a score transporter: an empirical quantile mapping that aligns nonconformity scores from trusted, real data with those from synthetic data. By carefully integrating the score transporter into the calibration process, SPI provably achieves finite-sample coverage guarantees without making any assumptions about the real and synthetic data distributions. When the score distributions are well aligned, SPI yields substantially tighter and more informative prediction sets than standard conformal prediction. Experiments on image classification---augmenting data with synthetic diffusion-model generated images---and on tabular regression demonstrate notable improvements in predictive efficiency in data-scarce settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called Synthetic-Powered Predictive Inference (SPPI), which focuses on improving Conformal Prediction (CP) when the amount of real calibration data is small. The key idea is to use synthetic data from a different distribution, and apply a score transporter that maps scores from the real data to match the distribution of synthetic scores. The method provides nearly tight coverage guarantee as well as the worst case coverage guarantee, and is shown to work well in both theory and experiments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- This paper is based on a brilliant idea and presents a solid method. It proposes a new way to build prediction sets when applying conformal prediction with a pretrained model and only a very small amount of calibration data is available. Also, the authors carefully prepare for possible drawbacks in Section 3.4, which I really appreciated.\n\n- The presentation of the paper is also very clear. When the authors explain the kind of problems their method can be applied to, they describe the feasibility of the situation very well in lines 25–34. This is actually an important and practical issue in the CP literature. Also, they clearly state the theoretical results of their method.\n\n**Weaknesses**\n\n- The only question I had while reading this paper is the following. Theorem 3.5 provides a safety bound when $P$ and $Q$ are not close. While the authors acknowledge this potential discrepancy, their explicit lower bound on coverage when $P$ and $Q$ are not the same is described somewhat vaguely. In particular, there is no quantitative analysis showing, under some reasonable experimental settings, how the lower bound behaves for a fixed value of $m$ and different levels of discrepancy between $P$ and $Q$.\n\nQUESTIONS:\nI think this is a solid work. But if the authors clearly answer the questions and suggestions below, I’d be happy to further increase my score.\n\n**Questions**\n\n- Is there a specific analysis which shows the empirical behavior of lower bounds when $P$ and $Q$ show different levels of discrepancy?\n\n- Since this algorithm has an extra hyperparameter $\\beta$ that controls the window size, I’m curious about how sensitive the prediction sets are to different values of $\\beta$.\n\n**Suggestions**\n\n- The idea in Section 3.4.1 about matching the synthetic scores depending on real scores deserves more than just a short remark. To me, using another model (e.g. least squares) to align synthetic calibration scores to match the distribution of real scores might raise concerns about overfitting when new test points come in. But the authors show that this doesn’t happen, which is surprising and kind of counterintuitive. I think it would be great if the paper could discuss this part in more detail.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SPPI, a framework for utilizing synthetic data in constructing conformal prediction sets. The authors propose to use a score transporter to bring the real scores to synthetic scores, and then use the large set of synthetic scores to calibrate the prediction set. Theoretical guarantees in terms of the distributional gap between real and synthetic data, and in terms of worst-case performance are provided. The proposed method is applied to three applications which show the advantages of the proposed methods in scenarios where high-quality data are scarce.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper studies an important problem\n2. The experiments cover rich applications \n3. The paper is clearly written and relatively easy to follow \n\nWeaknesses: (see Questions for detailed questions)\n\n1. The theoretical guarantees are a bit hard to interpret \n2. The intuitions of the proposed approaches are not adequately discussed \n3. The choice of hyperparameters is not clear in the experiments\n\nQUESTIONS:\n1. In Theorem 3.3, the term $\\varepsilon_{P,Q}^{m+1}$ depends on the TV distance of order statistics, which is a bit mysterious. How might this term depend on the TV distance between P and Q? \n\n2. The slack term $\\beta$ is discussed in lines 167-172 that only takes effect in extreme scenarios. Is there a way to derive a bound for that extreme event in order to obtain a sharper analysis?\n\n3. How should one interpret the quantities in Theorem 3.5? In particular, what does $R_j^+$ and $R_j^-$ quantify, intuitively? \n\n4. In general, I think the theoretical ideas behind the method are not discussed adequately. It might be helpful to include some high-level discussion on why the method could improve the efficiency and how the coverage is affected theoretically. \n\n5. It is also unclear why one might improve statistical efficiency by matching the distribution of real and synthetic scores. Intuitively, any efforts to align the scores will be subject to the intrinsic $1/\\sqrt{m}$ uncertainty in the real scores. Is this uncertainty still there, while you address the \"bias\" due to insufficient data? \n\n6. In Section 3.4, the theoretical results are not very illuminating, since attaining the bounds in Theorem 3.5 really doesn't tell us anything about how the coverage is improved. Can you derive some theory regarding the coverage/efficiency that is related to the quality of learning the synthetic score function? \n\n7. Why do you choose $\\beta=0.4$ in the experiments? How do the results change with the choice of $\\beta$? How would you recommend people to choose the value of $\\beta$? \n\n8. In Section 4.1.2, it is intuitively unclear to me why selecting samples from different label classes can help the prediction for a specific label class. Could you explain? \n\n9. The choice of $m=15$ is extremely small for Section 4.2. How would the results change with the value of $m$? Can the practitioners have a sense of how large $N$ is compared with $m$ to ensure that one gets some improvement?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework to leverage synthetic dataset for applying conformal prediction, in the case of scarce real data. To do so, it constructs a mapping called \"transporter\" using both real and synthetic calibration scores. Applying this mapping to the real test score allows to compare it to the synthetic quantile instead of the real quantile where the former is computed with much more samples than the latter.\n\nThey provide with a fast computation of their proposed conformal set with theoretical guarantee.\n\nIn the case of iid samples both for real and synthetic cases, upper and lower bounds for the marginal coverage are provided, which essentially depend on the total variation distance between the synthetic and real data distributions.\n\nExperiments are provided for the task of image classification using either diffusion models or unseen classes as synthetic dataset to illustrate their findings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem tackled of data augmentation for conformal prediciton is of interest. The paper writing is clear and enjoyable to read.\n- The transporter proposed is sound and intuitive.\n- Theoretical guarantees are provided and describe the deviation to the targeted $1-\\alpha$ coverage.\n- Experimental results show the applicability of their method.\n\nWeaknesses:\n- There is no in-depth discussion about the error terms in the bounds of Theorem 3.3 and 3.5. In particular, it seems that the sum $\\varepsilon + 1/(N+1)$ is a bias-variance trade-off.\n- The lower bound in Theorem 3.3 seems conservative as taking large values of $\\beta$ compared to $\\alpha$ in the experiments still yields good coverage.\n- I am slightly dubious about the assertion Line 167 \"when P=Q [...] our procedure provides a tighter upper bound than split conformal prediction\". If one has P=Q then there is no problem anymore to solve in the first place.\n\nQUESTIONS:\n- Could $\\beta$ be correlated to the difference between P and Q ? Since the transporter makes the real and synthetic scores \"similar\", I would assume that $\\beta$ could be used to calibrate that step similarly to results for weighted exchangeability, distributional shift. In particular, could deriving Lemma 3.1 in the case where P is not equal Q (with additional error terms that illustrate the shift) provide tighter coverage bounds ? \n- Do authors have an idea of how the bias variance (mentioned above) scale in $m,N$ even in simple cases and in particular whether there is some intuition in how to choose the scaling of $N$ compared to $m$ ?\n- The framework is extremely general and could include weighted exchangeability, distributional shift. For these specific cases, does the algorithm proposed obtain similar guarantees to existing ones ?\n\nI believe the proposed method is interesting but the guarantees do not allow to understand how to fully leverage it. Addressing the questions raised above could make me increase my score to accept the paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Synthetic-powered Predictive Inference (SPPI), a new framework that enhances the sample efficiency of conformal prediction by incorporating synthetic data generated from large generative models or alternative real datasets. The main innovation is a score transporter: an empirical quantile mapping that aligns nonconformity scores between real and synthetic calibration data. SPPI provides finite-sample, distribution-free coverage guarantees regardless of the similarity between real and synthetic data, and can substantially tighten prediction sets when synthetic and real data distributions are well aligned. The approach is demonstrated on both multi-class classification (ImageNet, using diffusion models) and tabular regression (MEPS panel data), showing improved efficiency over standard conformal prediction in data-scarce settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe method tackles a core limitation of conformal prediction—its inefficiency in low-data regimes—by leveraging abundant synthetic data in a theoretically principled manner. The authors provide finite-sample coverage guarantees,\n\nWeaknesses:\n1. The paper could further discuss or systematically analyze the impact of different types of synthetic data and score functions on efficiency and coverage—especially when using less realistic or lower-quality generators.\n\n2. SPPI method does not guarantee that the prediction set (or interval) will always be smaller than (or even no larger than) the one constructed using only real data. \n\n3. More practical guidance on choosing hyperparameters (e.g., β for windows, subset size for selection), diagnostics for misalignment, and post-hoc adjustment strategies would make the framework easier to deploy for non-expert users.\n\nQUESTIONS:\n1. Can the authors suggest practical ways to diagnose or quantify when the synthetic and real score distributions are too dissimilar for SPPI to be effective? Are there warning signs or automated checks practitioners could use?\n\n2. How does the method perform with synthetic data from weaker generators or less relevant domains? Can the authors comment on robustness in such scenarios?\n\n3. : For large-scale settings, how does the computational cost of SPPI compare to standard conformal prediction or other recent efficient conformal methods?\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}