Synthetic-powered predictive inference
Authors: Meshi Bashari
Roy Maor Lotan
Yonghoon Lee
Edgar Dobriban
Yaniv Romano
Venue: NeurIPS 2025 poster ()
Keywords: Distribution-free Inference
Conformal Prediction
Uncertainty Quantification
Domain Adaptation
Generative Models
Primary area: probabilistic_methods
TL;DR: Predictive inference framework that leverages synthetic data to improve sample efficiency in data-scarce settings, while providing rigorous, distribution-free finite-sample coverage guarantees.

Abstract: Conformal prediction is a framework for predictive inference with a distribution-free, finite-sample guarantee. However, it tends to provide uninformative prediction sets when calibration data are scarce. This paper introduces Synthetic-powered predictive inference (SPI), a novel framework that incorporates synthetic data---e.g., from a generative model---to improve sample efficiency. At the core of our method is a score transporter: an empirical quantile mapping that aligns nonconformity scores from trusted, real data with those from synthetic data. By carefully integrating the score transporter into the calibration process, SPI provably achieves finite-sample coverage guarantees without making any assumptions about the real and synthetic data distributions. When the score distributions are well aligned, SPI yields substantially tighter and more informative prediction sets than standard conformal prediction. Experiments on image classification---augmenting data with synthetic diffusion-model generated images---and on tabular regression demonstrate notable improvements in predictive efficiency in data-scarce settings.

Synthetic-Powered Predictive Inference
Meshi Bashari* Roy Maor Lotan" Yonghoon Lee Edgar Dobriban 2 Yaniv Romano 1.3
Department of Electrical and Computer Engineering, Technion- Israel Institute of Technology Department of Statistics and Data Science, The Wharton School, University of Pennsylvania USA 'Department of Computer Science, Technion-Israel Institute of Technology
Abstract
Conformal prediction is framework for predictive inference with distribution- free, finite-sample guarantee. However, it tends to provide uninformative prediction sets when calibration data are scarce. This paper introduces Synthetic- synthetie-powered predictive inference (SPI), novel framework that incorporates synthetic data- e.g., from generative model-to improve sample efficiency. At the core of our method is a score transporter: an empirical quantile mapping that aligns nonconformity scores from trusted, real data with those from synthetic data, By carefully integrating the score transporter into the calibration process, SPI provably achieves finite-sample coverage guarantees without making any assumptions about the real and synthetic data distributions. When the score distributions are well aligned, SPI yields substantially tighter and more informative prediction sets than standard conformal prediction. Experiments on image classification- augmmnting data with synthetic feffusion-modr generated images- and on tabular regression demonstrate notable improvements in predictive efficiency in data-scarce settings.
powered
Introduction
1.1 Background and motivation
is general framework for quantifying predictive uncertainty, providing fimite-sample statistical guarantees for any machine learning model. Given a test instance with an unknown label (c.g.. image), conformal prediction constructs prediction set-a collection of plausible labels guaranteed to include the true label with a user- specified coverage probability (e.g.. 95%). To do so, relies labeled holdout calibration set to compute nonconformity scores, which measure how well model's prediction aligns with the true labeled outcome. These scores are then used assess uncertainty in future predictions. Crucially, the coverage guarantee holds whenever the calibration and test data are exchangeable (e.g., i.i.d),, without any assumption on the sampling distribution.
Conformal prediction 48 58 59]
While conformal prediction offers a powerful coverage guarantee, its reliance on a holdout set limits its effectiveness when labeled data is setree-becomingg unstable and highly variable in coverage, overly conservative and uninformative. As a result, offers limited value in applications where labeled data is inherently limited, such as those requiring personalization Importantly, this is not merely an abstract concern [7] for example, in medical settings, it is natural seek valid inference tailored specific patient characteristics such as age, health condition, and/or other group identifiers of interest, see e.g., 1 38) Similarly, in image classification tasks, one may wish ensure that coverage holds for the true class label, see e.g., [57]. In these cases and many others, we often have only a few representative holdout examples for each group or class, which severely restricts the applicability of standard conformal prediction.
Slogroup-speciffee guarantees.
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Meanwhile, we are witnessing rapid progress in the ability to train accurate machine learning models even under data-scarce settings, driven by the rising quality of synthetic data produced by modern generative models and by advances in domain adaptation, see e.g., [TI]. These developments inspire the question we pursue in this work: Can we rigorously enhance the sample efficiency of conformal prediction by leveraging large pool of synthetic data-such as labeled datapoints from related subpopulations, or even data sampled from generative models?
At first glance, it may appear hard to use synthetic data to boost sample efficiency in a statistically valid way. After all, the distribution synthetic data can be completely different from that of the data SeeeresttOverr
of interest | Overcoming this challenge, we propose a principled framework that unlocks conformal prediction with the ability to incorporate synthetic data while preserving rigorous, -agoosiag provides a coverage bound that
non-asympiotic coverage guarantees. Crucially, our method-Stt requires no assumptions about the similarity between the real and synthetic data distributions. Still, when the distribution of synthetic and real scores is close, our approach yields a substantial boost in sample efficiency-rreuutinng in more informative prediction sets than those produced by standard conformal prediction. A discussion of related literature is deferred to Appendix B
1.2 Preview of the proposed method and our key contributions
Our key innovation is the introduction of the score transporter: a data-driven empirical quantile mapping function that transports the real calibration scores to resemble the synthetic scores. This mapping enables the construction of prediction sets for new test datapoints, leveraging the abundance of synthetic data. Crucially, the score transporter does not require data splitting, allowing full use of the real and synthetic calibration data. Furthermore, we develop computationally efficient algorithm with runtime complexity similar that of standard conformal prediction. A pictorial illustration of our proposed calibration framework is provided in Figure I
T(
synthetic scores
pred. interval/set for test point (Xrestr Yest ?) I
Limited real data
real
Scores
model calibration on real-to-syn
Abundant synthetic data
transported scores
Predictive
model
(black-box)
Figure I: A high-level overview of the proposed method. The approach leverages small labeled real dataset alongside a large labeled synthetic dataset. The score transporter maps scores from the real domain to the synthetic one. Calibration is then performed using the transported real scores and the synthetic scores.
We support our proposed SPI framework with two theoretical guarantees, where the final coverage rate bound the tighter of the two. The first shows that when the synthetic and real score distributions are close, the achieved coverage closely matches the desired level. More generally, it characterizes how the distributional shift between the real and synthetic scores and the construction of the score transporter affect the realized coverage.
The second theoretical result complements the first by providing worst-case bounds on the coverage probability, even if the synthetic data are of poor quality. This bound is directly controlled by the user, allowing them set a "guardrail," i.e.. a lower bound on the coverage probability (say 90%) that holds regardless of the distribution of the synthetic data. Remarkably, this bound holds even when the synthetic data depend on the real calibration set. This flexibility enables users to adapt or filter the synthetic data to improve efficiency, for example, by selecting datapoints that resemble the real ones all without requiring any data splitting.
We demonstrate the practicality of our method on multi-class classification and regression tasks. For image classification ImageNet, we explore two practical strategies for constructing synthetic data. The first leverages a generative model (Stable Diffusion [46] or FLUX (30) to generate artificial images for each class. The second uses another set of real data, drawn from different
We refer to the limited dataset of interest as the real calibration set, to distinguish it from the (potentially synthetic) calibration data.

---

distribution, as the synthetic data. In the regression setting, we consider tabular panel data, using past panels as synthetic data and a recent panel the real calibration set. Across all experiments, our method shows improvements in statistical efficiency- even when the real calibration set is very small, with as few as 15 datapoints. Software for reproducing the experiments available at https//ggihub..com/Mehhiii
2 Problem setup
Consider the standard setting of prediction problem where we have m i.i.d. (real) calibration datapoints (X; (X,,
(mm) Px,y Px x Pyjx, for i E [m] == (1...., m on X' x x), where each X, E represents the features and Y; E y denotes the label or outcome for the i-th datapoint. Given
new test input Xm+1 Px. the task is to construct prediction set ((Xm+1) for the unknown label Mm+1 with the following distribution-fres coverage guarantee:
P (x { {Ym+1 E (((m+1)) 2 21-a, a, for any distribution Px, on X x V, (I)
where 1 - a E 1) is a predetermined target level of coverage. Here and below, we abbreviate by (a:) vectors (a), a2..... ....... .Q)) While standard conformal prediction 44 39 58 59 has this property, its efficiency can be limited when the calibration sample size m is small.
Suppose now that we also have access to a set of synthetic datapoints (X,, JENNE iid could be datapoints collected from related distributions, sampled from generative model, obtained otherwise. We hope that xx is close to Px,y, but do not assume this. We are interested in the setting where & N, aiming to improve inference with small calibration set by leveraging large synthetic dataset. To make this concrete, we aim to construct a prediction set map C X + P())) where P()) is the set of subsets y-ass a function of the datasets (X,, Xe) and (X,, Y,) JE(N), such that the prediction set C(Xm+1) satisfies, for any distributions Px,y and Qx.y on X x y,
Qx.y. These
(X.. " v. (88.. #xx, {Ym+1 E (((Xmt1)) 2
P
(2)
For the classification task where Y is discrete, we extend our discussion beyond the marginal coverage guarantee in ① and consider the following label-conditional coverage guarantee [57 58
P (YYm1 E C(Xm++1) Ym+1 = y N 1 - a, for all y E y,
(3)
where, as before, we aim for a distribution-ffre guarantee, under any distributions Px, and Qx,- although the inequality in (3 is written in a simplified form.
2.1 Background: split conformal prediction
Split conformal prediction [39] is an approach to attain the coverage guarantee ① The first step is to
construct a nonconformity score function s : X' x y + ER from an independent dataset 3 Next, we compute the scores on the calibration datapoints: S. = s(X,, Y;) for i E [m]. The prediction set is then given as
Ō(Xm+1)= == {ur E E s(xm+1, v) S Q )
where Q denotes the [(1 am 1)7-th smallest score from the (multi-)or (ssti-))setttt (S:)xe|m)-
If the scores (St)ie(m) distinct almost surely, then the split conformal prediction set attains the following coverage bounds [39, 588 [59]:
(5) If m is very small, the split conformal set might be conservative. In particular, if m + 1 < 1/a, then the only way to achieve 1 coverage with m datapoints is by producing trivial prediction set that
1 1- S P Ymtz C(Xm+1)) S + 1/mmm + 1).
"Some of our results rely on weaker assumption than i.i.d.- i..d.-namely, exchangeability of the real calibration datapoints.
'A typical example for regression problems is s(x, y) = ly [a(x)), where is predictor pre-trained on separate dataset; e.g.. 58) etc.
3

---

includes all labels. For typical value of a = 0.05, this is the case when m < 19. Since we aim to handle situations with very low sample sizes, this motivates us to develop a procedure capable of producing more informative prediction sets by leveraging synthetic data.
3 Methodology
3.1 Synthetic-powered predictive inference
In this section, we introduce our method Soridd--P----- -PI-S-PISIS is designed to leverage the synthetic dat- apoints to effectively increase the sample size, thereby producing non-conservative prediction set. We construct split-conformal-yyee method that performs inference based on pre-constructed nonconformity scores. Throughout the section, we assume that the score function X' y
R is
fixed, and denote the real and synthetic scores as S; = s( ; 3, Y,) for i E [m + 1] and 5 = 8 XX, for j [AV], respectively. Here Sm+1 the unobserved test score.
that maps a real score to a synthetic score- as function of the observed scores. We then run split conformal prediction on the synthetic scores and apply T to obtain a prediction set for the real score Sm+1- 1- A carefully constructed map T can generate prediction with theoretically controlled coverage rate, while effectively leveraging the large synthetic dataset. The procedure has three steps.
Our strategy is to construct a score-transporter
Step 1. Construct windows in the space of synthetic scores. Denote by S(I)...., ((m+1) the real scores arranged in increasing order. We first define a "window" Im (r) designed to contain the r-th score (r)) for each r E [m + 1], as follows:
R = max {t E [N + 1]: F(t- - 1) G RI = min min (effteNN+] F(t) 3 (6)
where B E (0, 1) is predefined level, and F == FU,N,r r is defined a
F(t) Σ (kk),.(k),, with (k) (KtzT (N (N+m- (Mtm+)). (M*+))))))) m-r+1
Then, with the synthetic scores $t))...,,EEN in increasing order, construct the window as [m (r) = (Tm Um (r)], where I.n (r. == {R;)) and Um == (RZ)
and where ((N+1) = + co. This window is designed to satisfy the following property, where we denote the distribution of s(X, Y) under Px,y and Qx,y by P and Q, respectively:
Lemma 3.1. If Q and both are continuous distributions, then (S(n) E [m(r)) 2 1-18fon for all [m + 1).
The proof is deferred to Appendix G
Intuitively, the window (r) represents a region in the synthetic score space where S(r) is likely to lie, and the transporter we construct in the next step maps the real score to an element within its corresponding window.
Step 2. Construct the score-transporter. We now define the map T(.) = T(; (Si)semm) (5)))ENN) mapping real to synthetic scores as follows. For a scalar let Tn = -m the rank of 1 among (S1...., Sm.n) n) in increasing order, and with Lmx Un from (7) define
;= 1 (S, < n + 1 denote
Sm.
(r)))),
if 2 (r)),
T(n) = NN (N if Lm (r,)) I Um(rg); (r,)))
(8)
if L (Fn),
where the lower nearest neighbor NN is defined as
{ Bi6) S 5n]
NN (r, n) = max
RF SiSRT
'Here, for non non-negative integers < ( ll/(al(b a)!) denotes the binomial coefficient, where
x! = x (x 1) .... is the factorial of nen-ttteeive integer x. Also, (kk,..(() is the probability mass fiunction of the t-th order statistic from a random sample of size m + -1 drawn without replacement from finite [e.g.. 62 p. 243].
population of size N

---

Roughly speaking, the score score- transsrranprrtte T' maps 7) to synthetic score in the corresponding window Im (Ty) that closest 1). The lower nearest neighbor NN,, chosen carefully bound on the score, ensuring that the coverage can be tightly controlled.
lower
Step 3. Conformal prediction after transport-mapping. Applying the score- troe-trranportter T to hypothetical score s( s(Xm+1 y), we construct the prediction set as those y values for which this mapped value lies in the conformal prediction region constructed from the synthetic data:
c(Xm+1) {u y: T
T (s(X 9) S Q1-a)
Here Q1-a the [(1-a) + -th smallest score in (S)) We term this procedure Syntheric- powered predictive inference (SPI). Figure ST presents schematic overview of SPI with two candidate labels, illustrating each of the steps discussed above. Building on the ideas of [57 [38 we extend our proposed method to achieve label-conditional coverage guarantees Appendix
3.2 Simplifying the computation of SPI
Since T(-) ((: (S))) (SE))E(N) depends (S) and (S)) the prediction set C(Xm+1) in (9) has an a priori potentially complex dependence on Fortunately, the prediction set simplifies to the following formula, which is fast compute:
(1a = {ur : 3) I
((tt)) S(m+1) = t00.
(10)
Here Q4 is the (1(1-a)(N 1)1 + 1) smallest score among (S)))e(N)- and
R = max E [m + 1] R s [(1 a) + 1)13.
(11)
The following result shows that the prediction set Clast is equivalent the prediction set here, for two sets A and B, AAB denotes the symmetric set difference (A n B*) U (A B). Proposition 3.2. Recall the prediction sets C from
( and Clast from TOp. IfQ is continuous, then
P P(tYmt E Ö(Xm41)) A (Ym+1 Etast (Xm+))) =0
Based on this simplification, we present the complete SPI procedure in Algorithm !
3.3 Theoretical guarantees
We now derive bounds the coverage rate of the SPI prediction set 9 The first bound shows that when the real and synthetic scores are similar (as measured by total variation distance), our method has tight coverage around the desired level.
Theorem 3.3 (Coverage depending the closeness of real and synthetic distributions). Suppose the real calibration set (Xi, Y:)selml is exchangeable with the test point (Xm+1, Ym+1) and the synthetic calibration datapoints (X,, Y))jE(N) are drawn i..dd. where the distribution Q of their scores continuous. Let pm-1 and Qu++1 denote the distribution of the r-th order statistic among
(r) m + 1 i...d. draws from P and Q, respectively. Then the prediction set C(Xm+1)
( satisfies
1-a m++ S IP Ym+1 ((Xm+1)) 51-a+ P.Q
{
m++1 +1/(N+1
m++1 where E.Q
Σm-1 drv (Pttt Q and dry denotes the total variation distance.
m+I
Note that m++ is bounded above by the total variation distance between P and Q. When P= Q. P.Q
and thus our procedure provides tighter upper bound than split conformal prediction using only the real calibration data (5) when 3 + 1/NN + < 1/(m- + 1). When N > m and B & 1/m, our method offers a tighter coverage. In practice, however, we often observe tight coverage even for relatively large ß-in the proof, the + term arises from a union bound that accounts for the case where the test score Sm+1 is not covered by the corresponding window.
we have P.Q
.++1
5

---

Remark 3.4. The continuity of the score distribution required in Theorem can generally be attained conveniently. For example, in settings where the originally constructed score outputs discrete values, one can simply add a negligible amount of i.i.d. Uniform[- (-0, 6] noise to the scores, so that the perturbed scores scores-which are nearly identical to the original ones have continuous distribution.
When the distributions P and Q differ greatly, the bounds in Theorem 3.3 may be loose, they do not sufficiently account for the adjustment introduced by the map T under distribution shift. Below, we provide alternative worst-case bounds for the coverage rate of the SPI prediction set, which depend only on the sample sizes, and hold regardless of the relationship between Q and P.
Theorem 3.5 (Worst-case coverage), Suppose that the real calibration set (X1,
Ym) is exchangeable with the test point (Xm+1: Ym+1) 1), and that the synthetic score distribution Q is continuous. Then the prediction set C(Xm+1) in (9) satisfies
Kj E [m + 11: R
[(1 a) 117H
S P vm+1 E (((M+t))
m
H im + 1]: Rj S [(1- H)TH m m+1
While this result is somewhat non-explicit, the bounds can be computed fast, and remain close to the target level a, as illustrated in Section D through a set plots. We emphasize that the bounds hold due to the careful construction of the score transport map from (8), and would not hold if we were to simply mix together the real and synthetic data. Moreover, the bounds impose no condition on the distribution of the synthetic scores is even allowed for the synthetic scores to depend on the real calibration set. This provides significant flexibility in the choice of synthetic data even with a separate score function; see Section 3.4
The bounds in Theorem 3.5 depend solely on the constants m, N, a, and B. This allows to adjust the levels a and /3 to achieve a specific lower bound, say 1 - a', for predetermined value of w'-hhic implies that the guarantee (2) can be achieved. However, we advise using our procedure without level adjustment, in the spirit of Theorem 3.3 since Theorem 3.5 provides worst-case bounds. In practice, we recommend setting to meet a user-tolerable guardrail level-for instance, 90% worst-case coverage when the target level is compute such a 3 for given N, m, and a, ensuring the user-specified guardrail coverage bound is satisfied.
95%. Algorithm 4 outlines a procedure to
Having stated the two theoretical guarantees separately, we pause highlight the following: Remark 3.6 (Effective coverage guarantee of SPI). The coverage guarantees of SPI are adaptive to the quality of the synthetic data, in the sense that the effective theoretical bound automatically takes the tighter of Theorems
and 3.5
3.4 Improving the quality of synthetic scores
The quality of the SPI prediction set depends on how well the distribution of the synthetic score Q approximates the true score distribution P. as supported by Theorem 3.3 In practice, the alignment between synthetic and real scores can be assessed using standard goodness-of-fe tests on the empirical cumulative distribution functions (CDFs), such Cramér von Mises or Dolmogorov-Sminen tests. These tests can help detect when the synthetic scores deviate substantially from the real scores, indicating that the synthetic data may be less useful for the inference task could be modified to better align with the real data by carefully constructing the synthetic scores.
For instance, we can seek map 9 such that the distribution of the adjusted synthetic score S = .($)) better approximates the true distribution P. More generally, we may construct a separate score function
)/ + R for the synthetic data, so that the distribution the synthetic score 5(X, Y) better approximates that of the real score s(X, Y). Or, we may select a subset of the synthetic scores that is expected to provide better approximation. Below, we present two approaches to improve the quality of synthetic scores.
3.4.1 Constructing a separate synthetic score function
We first discuss the approach of constructing a separate synthetic score function 5. For example, one might choose construct š using split of the real data and split of the synthetic data. However, if

---

the original data sample size m is mmall-which is the main focus of this work we may prefer to reuse the data both for constructing the adjustment function or synthetic scores and for performing inference. Therefore, in this section, we focus on data-dependent score construction, while the details of the data-splitiig-
approach are deferred to Section F
For example, one might consider constructing an adjustment function 9 as s + g(s) == 01s + 62. where the parameters (01, 02) are fitted using the calibration scores (Si)ie(mi and (S))JE(N) via least squares:
(01.0) = argmin a.b
Σ la S(EeN/m]) + S(a)
and then constructing the adjusted synthetic scores (g( (g(())) setting = o 8; see for more sophisticated approach to learning the score function 8.
For such synthetic score constructed in a data-dependent manner, can still expect provable coverage bound? The answer is yes, since the bounds in Theorem 3.5 hold for synthetic scores with arbitrary dependence on the real calibration set.
Corollary 3.7. Suppose the synthetic score function constructed using both the real data (X,, Y,) [emm and the calibration data (X,,Y)) (j)]ENN. Then the prediction set C from (9) constructed
using 5, = =(x,, for j E [N], attains the bounds stated in Theorem 3.5
3.4.2 Constructing a subset of synthetic data
Now, we shift to a different approach for improving the quality of synthetic scores: constructing subset of the synthetic data that is more relevant for inference on the real data. This approach is particularly useful when the synthetic data comes from different sources, rather than sampled from a generative model. The idea is to select synthetic datapoints based how well they approximate the real data. Then, we form subset consisting of points with high approximation quality. Again, since Theorem 3.5 imposes condition the joint distribution of the synthetic scores, the bounds also hold for the SPI prediction set constructed with this subset of synthetic scores.
Corollary 3.8. Let muinet = (ii,... 3 C [N] denote the indices of a subset of synthetic data points, and suppose that N [ubsel is fixed. Then the prediction set C from (9) constructed using (3R))EM as the synthetic scores, satisfies the bounds stated in Theorem 3.5 with N replaced by N.
Note that this result requires the number of selected points Ñ to be fixed For example, one can use nearest- neighbor procedure, in which we partition the synthetic data into subsets of fixed size n, and then select subsets whose score distributions most closely resemble that of the real data, resulting in N = nk synthetic data points (see Algorithm 2)
4 Experiments
In this section, we compare the performance of the proposed SPI procedure to that of standard conformal prediction in setting where small real calibration set and a large synthetic calibration set are available, each drawn from distinct and unknown distributions. Further experiments on simulated data, where the underlying distributions are known, are presented Appendix !
Setup and performance metrics We randomly sample two disjoint subsets from the real data, assigning one as the real calibration and the other as the test Additionally, we sample synthetic calibration set from the synthetic data, which is intentionally larger than the real calibration set, aligning with the focus of this paper. The test is used to evaluate the procedure based on two metrics: the coverage rate, and the prediction set size (for classification problems) or prediction interval width (for regression problems). We report the results from 100 repeated trials, each with different random calibration, test, and synthetic datasets.
Methods
We compare the following methods: OnlyReal split conformal prediction 39 using the real calibration set; SalySSnthh-coffomm prediction applied to the synthetic calibration
'More generally, if N is random but independent of the real scores, the same bounds hold for the conditional probability P(Ym+1 E C(Xm+1) N}.
7

---

set as if it were real, which does not provide coverage guarantees; and SPI (ours) the proposed applied with 3 0.4.
procedure outlined in Algorithm [
4.1 Multi-class classification on the ImageNet data
We begin by evaluating our method on multi-class classification task using the ImageNet dataset [4] In particular, we aim for marginal ① and label-conditional coverage guarantees (3); the latter requires hold-out data for each class. Since our experiments involve generating thousands of images per class, restrict our study
subset of 30 classes (listed in Table (T) that form the real population.
We consider two scenarios for constructing the synthetic data. In the first scenario, we apply a generative model to produce synthetic images. In the second scenario, the synthetic set is formed using real images drawn from classes not included the real population.
Across all experiments and methods, we use a CLIP model
4% as the predictive model, along with We also include additional experiments with
the adaptive prediction sets (APS) score function 45 the homogeneous prediction sets (HPS) score function (58) in Appendix [.3 Importantly, CLIP is not trained on ImageNet images. Additional details on the score functions and pre-trained model are respectively.
provided in Appendices C.T and H.3
4.1.1 SPI with generated synthetic data
We use Stable Diffusion [66 to generate synthetic images resembling those ImageNet. Figure 2 shows representative examples, including images from an additional generative model discussed later. Additional examples and further details are provided in Figure S3 and Appendix H.4
Figure 2: Examples of real and generated images for the golden retriever class. The first column displays real ImageNet images, while the remaining columns show generated samples. The top row contains images generated by Stable Diffusion [46], and the bottom
FLUX (30].
For the marginal coverage experiments, we randomly select m 15 ImageNet images from the real data, chosen from among 30 classes, to construct the real calibration set. The test set consists of 15,000 real images, and the synthetic calibration includes N 1, 000 generated images, sampled uniformly across all classes. For the label-conditional experiments, we randomly select m 15 real images for each of the 30 classes form the real calibration set (resulting mk 450 real data points), 500 real images per class to form the test set, and n = 1, 000 generated images per class form the synthetic calibration set (resulting in N nk 30, 000 synthetic data points).
Figure presents the performance of various methods under both marginal and label condiccontiiinal 0.95. The label-conditional results shown for five
guarantees target coverage level representative classes. The observations below apply to both the marginal and label-conditional settings. We observe that OnlyReal controls the coverage level 1 conservative due to the small size of the real calibration set, which results trivial prediction sets. The OnlySynth approach fails achieve the target coverage level of 1 a, exhibiting undercoverage for some classes. This violation arises from the distribution shift between the real and synthetic data.
0.95. However, it remains
In contrast, the proposed method, SPI, achieves coverage within the theoretical bounds established in Theorem 3.5 For example, for the "Siberian husky" class, where the synthetic images differ significantly from the real ones, SPI still produces informative prediction sets. For classes where the synthetic and real data are more aligned, such as the "lighter" class, SPI shows low variance in coverage with smaller prediction set sizes.
We provide results for additional a levels in Appendix J.1 Further experiments on the effect of the size of the real calibration set and the parameter 3 on the performance of SPI are presented
8

---

in Appendices J.1.1 and J.1.2 respectively. In addition, we provide experiments using the FLUX generative model 30 which exhibit similar trends those observed with Stable Diffusion; see
Appendix I...3 Examples of generated images and details are provided in Figures and S6
and Appendix respectively.
1.0
30
OnlyReal OnlySynth
Coverage
10
Size
20
SPI
0.6
Lower-bound
Marginal
Siterrian buskyy
Baaver robun
American ran
Liphtor for.
Light
Barranouta
Marginal
Sibertan husky
Beateer robun
American Light
Lighter.
Barraconta
Upper-bound
Class
Class
Figure 3: Results for the ImageNet data: Coverage rates of OnlyReal, OnlySynth, and SPI at target level 0.95, averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box each group) and label-conditional coverage settings. Label-conditional results shown for selected classes; Table S3 for results across all classes.
4.1.2 SPI with synthetic data from k-nearest subset selection
We now explore the performance of the subset-based variant of our approach, referred to as SPI-Subset and described in Algorithm 2 The experiments in this section reflect scenarios where generative model unavailable.
As before, we aim to control both marginal and label-conditional coverage. In the marginal setting, randomly select real images from the same classes to form the test set. In the label-conditional setting, we randomly sample m 15 real images per class to form the real calibration for testing. In both cases, the synthetic calibration set consists of = 1, 500 annotated ImageNet images, drawn from 100 classes that are disjoint from the real classes, with n 15 images per class.
15 real images, across 30 classes, form the real calibration and 15,000
and 500 real images per class
We apply the subset selection approach to improve the quality of the synthetic data, using a k = 20 nearest-subset selection strategy, leading to Ñ = nk = 300 selected synthetic datapoints (see Algorithm 2). We compare this SPI-Subsed variant of our method to SPI-Whole where the latter denotes the SPI procedure run with the entire synthetic set. Additionally, as baseline, we include standard conformal prediction applied the real set, OnlyReal.
Figure shows the performance different methods with marginal and label-conditiona guarantees at target coverage level 1 a 0.98. The habel-conditional results are shown for five representative classes. We see that OnlyReal controls the coverage the 1 a level as expected, but it produces overly conservative-ia fact, trivial previl--rediition sets that contain all 30 possible labels. This is not surprising as split conformal prediction needs at least 50 datapoints to produce a nontrivial prediction set at level 0.02.
Both SPI-Whole and SPI-Subset achieve coverage within the theoretical bounds, generating smaller prediction sets compared to OnlyReal. In the label-conditional setting, SPI-Subset achieves coverage that more closely aligns with the target 1 a and produces smaller prediction sets, outper- forming SPI-Whole This highlights the benefit of aligning the synthetic set more closely with the real distribution through k-nearest subset selection.
1.0
30
Coverage
15
OnlyReal
Size
20
SPI-Whole
SPI-Subset a
0.8
Lower-bound
Marginal
Admatra Light.
Lighter
Bauver
Torch rubia
Amterican
Maryinal
Adhutral Canutrriiiii Light
Lighter
Boorer
Toroh rolin
Americaa
Upper-bound
Class
Class
Figure 4: Results for the ImageNet data: Coverage rates of OnlyReal, SPI-Whole, and SPI-Subset at level --a & 0.98, averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box each group) and label-conditional coverage settings. Label-condittonal results shown for selected classes; Table S8 for results across all classes.

---

However, in the marginal setting, where the real calibration set includes images from 30 different classes, selecting a small subset of synthetic classes does not necessarily improve alignment with which uses only a subset of the synthetie data (300
the real distribution. Consequently, images), exhibits higher variance in coverage compared SPI-Whole, which leverages the entire synthetic calibration set of 1,500 images.
We provide results for all classes appearing in the real data at additional values of the target level in Appendix J.2 In Appendix y.2.1] we further illustrate the performance of the SPI-Subset procedure for different values of the hyperparameter k.
4.2 Regression on the MEPS dataset
In this experiment, evaluate method regression task using the Medical Expenditure Panel Survey (MEPS) datasets [3]. We first fit a regression model on MEPS panel survey number 19. MEPS panel survey number 20 is then used as the synthetic data, and panel survey number 21 serves as the real data. This setup reflects a scenario in which large historical panels are leveraged as synthetic data to improve calibration use the conformalized quantile regression (CQR) score function [44]. Further details on the score function and the regression model are provided in Appendices C.1 and H..3 respectively. For each age group, we construct a real calibration set with m = 15 examples and a synthetic calibration set with N 1,000 examples.
recent, smaller weal-world population. For all methods,
Figure 5 presents the coverage and interval length results for OnlyReal, OnlySynth, and SPI at target coverage level 1
0.9, across different age-groups. Similarly the classification experiments, OnlyReal attains valid coverage but has a higher variance due to the small size of the real calibration set. Following that figure, see that the synthetic and real data are well aligned. However, OnlySynth, which relies solely on synthetic data, lacks formal coverage guarantees. In contrast, SPI achieves coverage close to the nominal level of 0.9, as predicted by Theorem 3.3 We
provide results for additional a levels and further experiments in Appendix K
1.0
OnlyReal
Coverage
OnlySynth
0.8
Longth
SPI
0.7
Lower-bound
0.5
2
Upper-bound
0.20
20:40
40:60
60
0.20
20-40
40.60
60
Age range
Age range
Figure 5: Results for the MEPS dataset: Marginal coverage and interval length for each age-group, obtained by OnlyReal, OnlySynth, and SPI. Target coverage is 1 0.9; experiments are repeated for 100 trials.
5 Discussion
In this work, presented novel framework that enhances the sample efficiency of conformal prediction by leveraging synthetic data in theoretically grounded manner. While we focused on marginal and label-conditional coverage, many applications require feature-conditional guarantees. Extending our approach to such settings-e.g. by drawing on ideas from Gibbs et al. 12 - an important direction for future work. Another limitation the assumption that the real calibration data and the test point are i.i.d., which may not hold in practice. We believe our results can be extended
beyond the i.i.d. setting by building on techniques developed in 28 42 53).
Naturally, the quality of the synthetic data affects the performance of SPI. While we found our data- dependent k-nearest subset approach to be effective, exploring alternative strategies srategies--particulatt particularly those suited to settings with multiple synthetic data sources-mas further enhance performance. Another future direction that we are pursuing is generalize SPI to enable the broader of synthetic data across statistical inference methods, including conformal risk control, hypothesis testing, and multiple hypothesis testing (9]
As this work aims to advance the field of uncertainty quantification in machine learning, it has potential social implications, similar to other research in the field.
10

---

Acknowledgments and Disclosure of Funding
M. B., R. M. L.,, and R. were supported by the European Union (ERC, SafetyBounds, 101163414). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. This research was also partially supported by the Israel Science Foundation (ISF grant 729/21). E. D. and L. were partially supported by the US NSF, NIH, ARO, AFOSR, ONR, and the Sloan Foundation. Y. R. acknowledges additional support from the Career Advancement Fellowship the Technion, and is deeply grateful to Shai Feldman and Jeremias Sulam for their insightful discussions and valuable feedback.
References
[1] Ageney for Healthcare Research and Quality. Medical expenditure panel survey, panel 20, 2020.
[2] Agency for Healthcare Research and Quality. Medical expenditure panel survey, panel 21, 2020.
[3] Ageney for Healthcare Research and Quality. Medical expenditure panel survey, panel 19, 2025.
[4] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[5] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana
Zmncc Prediction- inference. Science, 2023.
[6] Konstantina Bairaktari, Jiayun Wu, and Steven Wu. Kandinsky conformal prediction: Beyond class- and covariate-oon coniiiii-ooniiionalll coverage. In Forty-second International Conference on Machine Learning, 2025.
[7] Christopher RS Banerji. Tapabrata Chakraborti, Chris Harbron, and Ben D MacArthur. Clinical tools must convey predictive uncertainty for each individual patient. Nature medicine, 2023.
[8] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 2023.
[9] Meshi Bashari, Yonghoon Lee, Roy Maor Lotan, Edgar Dobriban, and Yaniv Romano. Sta- tistical inference leveraging synthetic data with distribution- guarantees. arXiv preprint arXiv:2509.2034. 2025.
[10] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 2023.
[11] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. the opportunities and risks of foundation models. arXiv preprint vrXiv:2108.07255. 2021.
[12] Victor Chernozhukov, Kaspar Wuthrich, and Yinchu Zhu. Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data. In Proceedings of the 31st Conference On Learning Theory, PMLR, 2018.
[13] Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Toward personalized inference individual treatment effects. Proceedings of the National Academy of Sciences, 2023.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009.
[15] Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-conditional conformal prediction with many classes. Advances neural information processing systems, 2023.

---

[16] Robin Dunn, Larry Wasserman, and Aaditya Ramdas. Distribution-fiee prediction sets for two-layer hierarchical models. Journal of the American Statistical Association, 2022.
[17] Shiladitya Dutta, Hongbo Wei, Lars van der Laan, and Ahmed Alaa. Estimating uncertainty in multimodal foundation models using public internet data. In RO-FoMo: Robusiness of Few-shot and Zero-shot Learning in Large Foundation Models, 2024.
[18] Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty- aware classifiers with conformalized deep learning. Advances in Neural Information Processing Systems, 2022.
[19] Bat-Sheva Einbinder, Liran Ringel, and Yaniv Romano. Semi-supervised risk control via cerdiction-poweree inference. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025.
[20] Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning PMLR, 2021.
[21] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 2021.
[22] Isaac Gibbs, John J Cherian, and Emmanuel J Candès, Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2025.
[23] Leying Guan. Localized conformal prediction: A generalized inference framework for confor- mal prediction. Biometrika, 2023.
[24] Leying Guan. A conformal test of linear models via permutation-auugmenet regressions. The Annals of Statisties, 2024.
[25] Leying Guan and Robert Tibshirani. Prediction and outlier detection in classification problems. Journal of the Royal Statistical Society: Series B, 2022.
[26] Rohan Hore and Rina Foygel Barber. Conformal prediction with local weights: randomiza- tion enables robust guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2025.
[27] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. Version 0.1, Zenodo.
[28] Sunay Joshi, Shayan Kiyani, George Pappas, Edgar Dobriban, and Hamed Hassani. Likelihood- ratio regularized quantile regression: Adapting conformal prediction to high-dimensional covariate shifts. arXiv preprint arXiv:2502. /3030, 2025.
[29] Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Batch multivalid conformal prediction. In International Conference on Learning Representations, 2023.
[30] Black Forest Labs. Flux: High-fidelity text-to-image generation with transformer diffusion models, 2024.
[31] Yonghoon Lee, Eric Tchetgen Tchetgen, and Edgar Dobriban. Batch predictive inference. arXiv preprint arXiv:2409.13990 2024.
[32] Jing Lei and Larry Wasserman. Distribution-fre free prediction bands for non- parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2014.
[33] Jing Lei, James Robins, and Larry Wasserman. Distribution-fret prediction sets. Journal of the American Statistical Association, 2013.
[34] Jing Lei, Alessandro Rinaldo, and Larry Wasserman. A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 2015.
[35] Jing Lei, Max G' Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 2018.
12

---

[36] Ziyi Liang. Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for powerful ditto---ditributionn testing with labeled outliers. arXiv preprint arXiv:2208. 11110, 2022.
[37] Ziyi Liang, Yanfei Zhou, and Matteo Sesia. Conformal inference is (almost) free for neural networks trained with early stopping. In International Conference on Machine Learning, 2023.
[38] Keli Liu and Xiao-Li Meng. There is individualized treatment. why not individualized inference? Annual Review of Statistics and Its Application, 2016.
[39] Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning. Springer, 2002.
[40] Sangdon Park, Edgar Dobriban, Insup Lee, and Oshert Bastani. PAC prediction sets under covariate shift. In International Conference on Learning Representations, 2022.
[41] Sangwoo Park, Kfir M Cohen, and Osvaldo Simeone. Few-shot calibration of set predictors via meta-learned poss-validation based conformal prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[42] Aleksandr Podkopaev and Aaditya Ramdas. Distribution-fiee uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence. PMLR, 2021.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PLMR, 2021.
[44] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 2019.
[45] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 2020.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[47] Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least Ambiguous Set- Valued Classifiers With Bounded Error Levels. Journal of the American Statistical Association, 2019.
[48] Craig Saunders, Alexander Gammerman, and Volodya Vovk. Transduction with confidence and credibility. In IJCAI, 1999.
[49] Henry Scheffe and John W Tukey. Non-parametric estimation. validation of order statistics. The Annuls of Mathematicul Statistics, 1945.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 2022.
[51] Matteo Sesia, YX Rachel Wang, and Xin Tong. Adaptive conformal classification with noisy labels. Journal of the Royal Statistical Society Series B: Statistical Methodology. 2024.
[52] David Stutz, Abhijit Guha Roy, Tatiana Matejovicova, Patricia Strachan, Ali Taylan Cemgil, and Araad Doucet. Conformal prediction under ambiguous ground truth. Transactions Machine Learning Research, 2024.
[53] Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 2019.
[54] John W Tukey. Non-parametric estimation ii statistically equivalent blocks and tolerance regions-the the continuous case. The Annals of Mathematical Statistics, 1947.
13

---

[55] John W Tukey, Nonparametric estimation, iii. statistically equivalent blocks and multivariate tolerance regions- discontinuous case. The Annals of Mathematical Statistics, 1948.
[56] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning. PMLR, 2012.
[57] Vladimir Vovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman. Mondrian confidence machine. Technical Report, 2003.
[58] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science Business Media, 2005.
[59] Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-leamiing applications of algorithmic randomness. In International Conference on Machine Learning, 1999.
[60] Abraham Wald. An extension of wilks' method for setting tolerance limits. The Annals of Mathematical Statistics, 1943.
[61] Samuel S Wilks. Determination of sample sizes for setting tolerance limits. The Annals of Mathematical Statistics, 1941.
[62] Samuel $ Wilks. Mathematical statistics. Wiley, 1962.
[63] Ran Xie, Rina Barber, and Emmanuel Candes. Boosted conformal prediction intervals, Advances in Neural Information Processing Systems, 2024.
[64] Yao Zhang and Emmanuel J Candès. Posterior conformal prediction. arXiv preprint (Tt::2409.199122 2024.
14

---

A
Algorithmic details
Algorithm 1
Synthetic- predictive inference (SPI)
1: Input: Real calibration set (X,, Ei)imm)i synthetic calibration set (X,, ke test input Xm+1; score function target coverage level at parameter for window construction B.
2: Compute the real scores S; = s s(X,, X,), for i [m]]
3: Compute the synthetic scores 5; = s (x 8j). for j E [M]. and let Qi - -())++
4: Compute R. and R for r E [m + 1.. according to (6)
5: Compute At and Rtt, according to (IT)
6: Compute the bound Q = max maix(min((e Q1-a: S (R,))
7: Compute C(Xm+1) = fy V s(Xm+1. s Q}.
8: Output: Prediction set ((Xm+1)).
Algorithm SPI with data-dependent k-nearest subset selection
1: Input: Real calibration set (X,, kemj] subsets of synthetic calibration set (X), Yj) Je [n]
1,2.... L; test input X score function s; target coverage level 1 - a; parameter for window construction 3; parameter for selection k.
2: Compute the real scores S. = 8 X; Yi), for i E [m]]
3: Compute the synthetic scores S = s s(X), ) for j E [N] and le [4].
4: for in [L] do
5: Distances(] [1] <- Craame--oon-Mise-SStaieee ({ i [m]]. {$} j E [nl)) (Algorithm)
6: end for
7: Let L be the set of & subsets in [L] with the smallest values in Distances.
8: Apply Algorithm with (XX) y : E [n), L) as the synthetic calibration data.
9: Output: Prediction set ((mm+)).
Algorithm 3 Cramer-von Mises two-sample test statistic
1: Input: (X))) (Yi)e(M) (all distinct)
2: Let W = (X,,..... xx) U (Yi...,, YMM be the set all datapoints.
Compute the ranks:
r; (the rank of X; in W) for i E [V], and = (the rank of Y; in W) for E [M].
4: Let r(1) <... T(N) and 8(1) < (MM) be the order statisties of (ri)vetN) and (st)ee)))
respectively.
5: Compute
U N Σ
(ro i)
M Σ
(80 -)
J=1
6: Compute the Cramer-von Mises test statistic T as:
4MN N
T NM +
6(M + N)
7: Output: T.
15

---

Algorithm 4 f-selection
1: Input: Real calibration set size m: synthetic calibration set size N; target coverage level 1 - a: desired worst-case lower bound L; step size e.
2: Set B 4- 6.
3: Compute R with for E [m + 1]. according to 6
4: Compute L + lfi [m + I]: R = [(1 + (m 1).
5: while L do
6: B+ =
7: Compute B with 8 for E [m + 1], according to (6
8: Compute + ii E [m + 1] R z [(1 aN + 1)1H/ / (m + 1).
9: end while
10: Output: B.
B Related work
The concept of prediction sets dates back to foundational works such as Wilks 61 Wald [60]
Scheffe and Tukey [49], and Tukey [54] [5]. The initial ideas behind conformal prediction introduced by Saunders et al. [48] and Vovk et al. [59]] Since then, with the rise of machine learning,
conformal prediction has emerged
widely used framework for constructing distribution-free
prediction sets [e.g.. 4 10 12
16
18
21 23 33
40
45
47
[66] [88
More recently, there has been growing interest in extending conformal prediction to offer more refined guarantees beyond standard marginal coverage. In particular, several works aim to offer approximate local coverage guarantees in the feature space [23 26 64]; group- conditoonii coverage, which aims to guarantee valid coverage across pre-defined groups based on features and/or labels [6 22, 22 [7];
and cluster-conditional coverage, which focuses label-conditioned subgroups [15]. However, these approaches still face the inherent limitations of conformal inference in settings where labeled data for the group-of caap--of-nnteee interest is limited, previously discussed.
In contrast, we are interested obtaining exact label- or group-of-interest conditional coverage guarantees even when the dataset from our distribution of interest is small. To this end, we take different approach, aiming to enhance sample efficiency by incorporating synthetic data.
A related line of work explores the use of unlabeled data to improve sample efficiency 19). These methods assume that the unlabeled data drawn from the same distribution as the labeled calibration set. In contrast, we consider settings where this assumption is violated and develop methods that remain valid under such unknown distributional shifts. Moreover, the above methods cannot be applied in the label-conditional setting, as they require knowing the labels of the unlabeled data.
Another related line of work is few-shot conformal prediction 20 which addresses settings where only limited data is available for the target task, along with additional auxiliary tasks. These approaches leverage related but distinct tasks improve sample efficiency. Fisch al. 20 provides asymptotic lask-conditional coverage guarantees, whereas our focus is on linie--sample guarantees. Park et al. [41] mitigate the small-sample challenge using cross-validation, but their methods remain constrained by the overall number of available datapoints- wwhch we assume to be small in our setting. Dutta et al. [17] propose to retrieve web images to enable conformal prediction in zero-shot settings, by leveraging conformal prediction with ambiguous ground truth [52], but do not provide coverage guarantees for their method.
C Technical background
C.1 Score functions
For classification tasks, we assume that the pre-trained model outputs an estimated probability vector [0, 1] where K is the number of classes and each entry represents the estimated probability of the corresponding class. We consider the APS score function that is defined for given pair (X, Y) as follows: Let (X) 2 (X)
Adaptive prediction sets (APS) 45
2 ((K)) (X) be the sorted values of the probability vector *(X), and let rYY, +(X)) denote the rank of the label Y
16

---

within this sorted vector. The nonconformity score is then given by:
s(X = -x) + (2) + +... (x(X)(())
(X) - U ((0rx))))))
(12)
where U is a uniform random variable on [0, 1], independent of everything else.
Homogeneous prediction sets (HPS) 58
As complementary score function for classification
lasks, we consider the HPS score. Using the same notations as in the APS paragraph, for a given pair (X, the HPS nonconformity score is defined
s s(X, Y) = (X)()
(13)
For the regression task, suppose we have pre-trained quantile regression model that estimates the 7-th quantile of the distribution Y X, denoted as 7(;;7). The conformalized quantile regression (CQR) score is then defined as
Conformalized quantile regression 44
s sXX, Y) max (#(X; a/2) Y,Y - ((X; a/2))-
(14)
Applying conformal prediction with this score, the prediction set takes the form
Ö(x++1)= = (((x+11://2) Qi-a: ((xx(((*))) ntii
-a/2) + Q Q1-a]
17

---

D Explaining the score transporter
In this section, we provide further intuition about our proposed procedure, as well as the theoretical bounds established in Theorem 3,5
Figure ST provides schematic overview of the procedure. For clarity, we illustrate the construction of the prediction set as described in Equation (9). In practice, we use the computationally efficient procedure described in Equation (TO), which have shown to be equivalent in Proposition 3.2
We begin by computing nonconformity scores for both real and synthetic data points. We assume access to fixed score function (e.g., pre-trained black-box model), along with small real calibration set (X;, Yi) a large synthetic calibration set (X,, JE) and test point X +1+
largest
largest
synthetic scores
real scores
Q1-a
sort
sort
(+)
(Xm+1131)
neighbor
smallest
smallest
(a) Candidate label y1: the test score (purple) ranks second among real scores. Its mapped synthetic neighbor computed via ( and outlined in purple- falls below the empirical quantile Q Q1-a, hence y1 ((m++1).
largest
largest
synthetic scores
real scores
T(
21-ar
nearest neighbor
sort
sort
3Xm+11)22)
smallest
smallest
(b) Candidate label y2: the test score (purple) ranks fourth among the real scores. Its mapped synthetic neighbor computed via and outlined in purple-excedd exceeds the empirical quantile Qi-a. thus y/2 & C(Xm+1).
ranks
Figure S1: Illustration of the synthetic-powered predictive inference for two candidate labels. Each panel displays sorted nonconformity scores: real scores on the left and synthetic scores on the right. The rectangle indicates the window in the synthetic space to which the test score can be mapped (as defined in The black-outlined circle indicates the (1 - a1 + x )th empirical quantile of the synthetic scores, Q Q0-
For each candidate label y, we compute the nonconformity score of the test pair (Xm+1, v), along with those of the real and synthetic calibration sets. These scores are depicted as circles in
18

---

Figure ST] displayed in sorted order. Crucially, the scores for the real and synthetic calibration sets are computed once and then reused for all candidate labels.
Next, following Equation 7 each real score S(), E [m 1], is associated with a window in the synthetic space. These windows (illustrated rectangles connected each real score) define the set of synthetic scores to which each real score can potentially be mapped. Importantly, these windows depend only on the sample sizes m, N, and the parameter 3-not the values of the scores themselves.
For the test score s(Xm+1, v). we identify its synthetic neighbor after mapping, within its associated window according to (8), denoted by ((s(Xm+1.B)) This synthetic score is then compared to the (1 - a1 + N )hh empirical quantile the synthetic scores, Q1-a (outlined black). v)) - Q1-ac the candidate label is included in the prediction set C(Xm+1)) otherwise, it is excluded.
To illustrate this, Figure STa shows the procedure for candidate label E Y. The corresponding test score (marked in purple) ranks second among the real scores, and its mapped synthetic neighbor within the corresponding window (outlined in purple) lies below the quantile threshold: ( (x5mm+11,00 B1)) I
Qi-a. Thus, y1 E C()m+1). In contrast, Figure S1b depicts the case for label y/2 E y (also marked in purple), where the test score ranks fourth, and its mapped neighhor exceeds the threshold: T(s(mm+ )2)) > Q1-a- Therefore, U2 & C(Xm+1) This procedure is repeated for each y E y to construct the full prediction set C((m++1).
We now illustrate the theoretical bounds established in Theorem 3.5
using the same schematic from Figure ST] Real and synthetic nonconformity scores shown circles in sorted order, with each real score connected to window in the synthetic score space (depicted as rectangles). The - (1 + N 1th empirical quantile of the synthetic scores, Q is outlined in black.
largest
largest
R5
largest
R
largest
R5
Q1-c
Q1-a
R
R4
R
R5
R
R
smallest
less
R
smallest
smallest
(a) Values of R (in blue), representing the upper endpoints synthetic windows. smallest three real scores satisfy Rt < Qi-a and thus guaranteed be mapped synthetic scores below the threshold Q....
Values of R (in blue), representing the lower endpoints of synthetic windows. fifth real score satisfies R Qi-a- meaning it is neces- sarily mapped above the threshold Qi-a-
Figure S2: Illustration of the quantities used in the worst-case coverage bounds from Theo- rem 3.5 Real and synthetic nonconformity scores are shown circles in sorted order. Each real score 1s connected to a window in the synthetic score space (depicted as rectangles). The synthetic (1 - (1 + N/ 1t empirical quantile Q1-a a is outlined in black.
Figure S2 visualizes the quantities used to derive the worst-case coverage bounds. For each score S(n) for [m + 1], denote the endpoints of its associated window by R, and Rt, as introduced
19

---

in 16. These correspond to the smallest and largest ranks, respectively. of the synthetic scores that (r) can be mapped to. For convenience, we refer to R, and R as the synthetic scores those ranks.
Figure $2a shows the values RT (in blue), which used to compute the lower bound. Since the transported score ((((in)ddefieed as the nearest synthetic score within the window among those that are smaller than (ir) always less than equal to Rt any real satisfying R Qi-a must necessarily satisfy (()())) & Q1-a. Consequently, the corresponding label will always be included in the prediction set.
Figure S2b shows the values Rt (in blue), which are used to compute the upper bound. Real scores for which Rे 2 Q1-a may be mapped to a synthetic score below the threshold and thus may be included in the prediction set-for example, the bottom four real scores in the figure. In contrast, if RT Qi-a (as for the fifth score), then the transported score must exceed the threshold, and the corresponding label is guaranteed to excluded.
By exchangeability, the test score is equally likely to take any of the m + possible ranks among the real calibration scores. Therefore, the coverage probability bounded between the fraction of real scores whose R S Qiaa and the fraction whose R, S Q1-a. as formalized in Theorem 3.5 In our example, these correspond to 3/5 and 4/5, respectively.
D.1 Coverage guarantee bounds
To illustrate the distribution-Fie bounds in Theorem 3.5 we present several visualizations. These bounds are determined solely by the sample sizes and N, the parameter 3, and the target coverage level a.
Figure S3 presents the upper and lower bounds in Theorem 3.5 as functions of the calibration set size m and the level a, with fixed N 1000 and 0.4. As m increases, the bounds become tighter around the target level 1 a.
1.0
1.0
1.0
Coverage Bounds
Coverage Bounds
Bounds
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
5
15
25
15
25
35
45
(a) a = 0.02
(b) = 0.05
(c) a = 0.1
Figure $3: Illustration of the coverage bounds in Theorem 3.5 as function of the real calibration
set size m. The synthetic calibration size is N = 1000, and we set /3 = 0.4. Results are presented for a 0.02 (a), 0.05 (b), and 0.1 (c). The shaded regions represent the area between the lower and upper bounds for each level, with the dashed black lines indicating the target coverage level 1 a.
Next, Figure S4 illustrates how the bounds vary with the parameter 3-under m 15, N = 1000,
and different values of a. As 3 decreases, the bounds become looser. This trend can be explained as follows: by the construction of the windows in (6), smaller values of 3 lead to wider windows. As result, fewer Rt values (for r E [m + 1)) fall below the (1 - a )h empirical quantile, loosening the lower bound. At the same time, more Rt values fall below this quantile, resulting in looser upper bound. This trend is consistent across various a, as shown in the figure. Further, the bounds exhibit a stepwise pattern due to their discrete nature-ther values change in increments of 1/(m + 1).
practice, one may be interested using our method while ensuring that the lower bound guaranteed by Theorem 3.5 is no smaller than user-specified level L. To this end, we provide an algorithm (see Algorithm 4 for selecting 3 based on the sample sizes m and N, the target miscoverage level a, step size (c.g., 0.01), and the desired lower bound L. As shown in Figure S4 multiple 3 values may
20

---

1.0
Coverage Bounds
Coverage Bounds
Bounds
0.8
0.8
0.8
0.6
0.6
0.6
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
B
B
0.02
0.05
(c) 0.1
Figure S4: Illustration of the coverage guarantee bounds from Theorem 3.5 as a function of B. The real calibration set contains m 15 datapoints. Other details are as Figure
yield the same lower bound. In such cases, the algorithm selects the smallest that results in lower bound greater than or equal to L, inspired by the result in Theorem 3.3
21

---

E Constructing a separate synthetic score function with data splitting
In this section, we provide details on constructing the synthetic score function independently of the calibration data, adding to the discussion in Section 3.4 For instance, suppose we apply data splitting (to both the real and the synthetic data) and use one split training data construct the synthetic score function 3. If the data has already been split to construct the same split can be used for constructing Then we use both real and synthetic (training) data to construct s, to ensure that its distribution better approximates that of the real score. Throughout this section, we condition everything on the training datasets.
We begin with the method of constructing an adjustment function and using the transformed score function š go gos for the synthetic data. In this case, the prediction set Ca (Xm+1))) constructed according to ( given by
(X = tu E y T((((((smmmmiii ( (Si) (o(33)) (((S))))N) & &9-ah
where Qi_a denotes the [(N + 1) - smallest value among { {g(5,) : [[]]] Example One option is construct an affine transformation g(s) 01S + 02 to adjust the scale and bias of s(X,Y) Denote the training sets by (X,, Y?),..., (X/m. Y V and
(X1, Yi),.... (X Munis" Y and denote the corresponding real and synthetic (training) scores by [E and (S)))(((Nmn)) respectively. Then, we can set 02 and 0g via least squares:
02) aargminn 0,
Σ a (LiNrrnn/ Minin)) 1) b 5/3)
where S/1) and (3)) denote the order statistics the real and synthetic training scores, respectively.
More generally, suppose construct new score function using the training data, such that the distribution of 3(X, Y) approximates that of s(X, Y), where (X, Y) Qx,y and (X, Y) P. The prediction set is then constructed according ( using the synthetic scores S, 5 ((x,, Yj):
Õ(Xm+1) = ty E y: ((x(Xmmsx: v): (St) ($,) & Q1-a).
(15)
Denoting the distribution of 3(x, Y) as Q we have the following result as a direct consequence of Theorem 3.3 and 3
Corollary E.1. Suppose the distribution Q is continuous. Then the prediction set ((Xm+ from 15 constructed using S 5(X,, , Y2) for [N], satisfies
1 1 a m+1 S IP Y Ym+1+ E ((Xm+11) 11- 51-a+1 + m+ p.
E.. + 1/(N + 1).
Moreover, the bounds stated in Theorem 3.5 also hold for ((Xm+1) from [5
F Predictive inference with label-conditional coverage control
Here, we review the standard approach (88 for achieving the label-conditional coverage guarantee and then discuss a variant of this approach based SPI.
The basic idea to partition the calibration set by classes, run conformal prediction within each class, and then combine the results construct prediction set. Specifically, the prediction set is constructed as follows:
Ō(Xm+1) (y ::::s:sss.... s(Xm+1, 3) S Q.-a)
where Qr denotes the [[1- - (n,, + 1) smallest element among (S: i [m], Y; y). and Ny denotes the number of calibration points labeled with y.
"The proofs these theorems build upon the setting Si...., Sn+1 and S...., Sw Q, and do not depend directly on the datasets or the score function. Therefore, the results in Corollary B. follow directly by applying the same arguments with Št...., Syy iid Q)
22

---

Now, we introduce the SPI-based method that ensures the label-conditional coverage guarantee. The idea follows the same logic as the standard method: We run SPI within each class-specific partition of the real and synthetic calibration sets and then combine the results, For a class that does not appear in the synthetic dataset, we run the procedure with the entire synthetic dataset.
To formalize this idea, define
J = Ey y: Y, = y for some j E [M]] and 3o = {v E y: Y, * for all j [N]]. Let I, E [m] Y, y) for each y 3/ and J, = E [N] Y, = for each E 3/. Then for each y V4, we define the function (V(-) (()) ($,)ee,,), following the definition in (8). For y E Vo, we let (-)- = ($))))), Then we define Qi for each y as follows:
Q4 = [[(1-10 a:) (15n) | + 1)1 smallest element in {$} j: Jy),
if V1, if e Vo.
([(1-a) (((1-a)) (N + 1)]-th smallest element in {Š₁ j E [N]}
Then we construct the prediction set as
Ö(Xm+1)= = V. y)) - Q² }
(16)
As direct consequence of Theorem 3.5 the prediction set (16) attains the following label-conditional coverage control:
Ki E [m+
1
R3 & (N + 1)TH
S {Y Ym+z (((M+11) Ym+1 =
m + 1
Hí E [m + 1] R $ [(1 - aN + 1]]H
for all y E e,,
m + 1
where we let N2 = |Jyl for E V1 and N₂ N for Vo.
G Mathematical proofs
G.1 Proof of Lemma 3.1
The result follows directly from the work of Lee et al. (31, but we provide the proof for completeness. Define
R2 = min E [N + 1| S(r) 2 ((r)
for each [in + 1J. where we let R, - N + 1 (()) 2 S(N). Note that R2 is random, whereas R and R are not. Then by the exchangeability of (S,)re(m+1) and (S)) the distribution of the vector (Ri, R2, Rm41) is given by
(R1, ... Rm+11 Unif([((1,.... Sm+1) : 11 5......55... Smm1 x++1)) Therefore, for each & E [N + 1], we have
P (Rr = k) I((Si,... (m+1) S Si S ... S Sm+1 S N + and &. = k]]
[((I, ... (m+1) ; S G1 5.... S Sm+1 < N +1H|
[(G1(r-1): : < G1 5 S S k)l (G(+1)):(++) k G+1 S .... Gm+1 N N I|]
H((1, Sm+1): 1 < C1 ... 5m+1 & N + +1}
( (N 4mm
HH ' -222 Hm-r
N+1
(N (N+m+1)
where we use the notation H, denote the number of ways to select items with replacement from items. Therefore,
P (() [m(rr)
s SSn) S (m))) (3 S Si,) S S3 (m!))
= P & R2 < R F(Rt) ) - F(R, - 2 - 3-
where the inequality follows from the definition of R, and R
23

---

G.2 Proof of Theorem 3.3
Let rm+11 - 1 (S, < Sm+1] denote the rank of Sm41 the increasing order among Si...., Sm, 5m+1- Observe that T(S+11) I Sm+1 holds if Lm (m)) < Sm+1, by the con- struction of the mapping T. Therefore, writing L, (r)r) and U, Un(r)) for simplicity, we have
P (YYmi E ((Xm+i)} IP (((m+)) S 21-03
P (T((Sm++) S Q1-a;, Sm+1 E [Lro U Urmtil}
+P(((mm+1) S SQ Smi $ [Trmat Urmetl)
2 { Qiaa
Sm+1 I U Urmtal).
We can condition on Fm+11 to write that this equals
E PP Sm+1 S Qi-a; Sm+1 E [Tr 12
rm+1)
}}
Further, since Tm++1 ~ Uni ........m... m + 1) by the exchangeability of (S,) (S.) and since Tm++1 is independent of the order statistics -(), S ((m+1). 1). the expression further simplifies to
P ((r) S Qi Jy (()) Ur
mm+1 =r]
m+1
Σ m +
p P(s S br
Ly S S() s U.}.
Now we fix r E [m + 1] and examine the probability in the summation. The event inside the probability a function S()) ~ Pott and Si,... .N iid Q. Thus, we have
P pm SNN~QN {st) S Qi-ai b, SSn }
P a BINNQN (n S Q1,,, Lr s S/n s {Un}
- ddv(p x QN, 30 x Q*))
= SNN~QN {sin s Q1-a:4. S(n s U,} div(P 2
Therefore, putting everything together, we have
Yymt E ((X
m+1
Σ
P Suy SEX-QN {sim I L, s U.}
m +
m+
Crv(pmtt.
P f1t { {mmt1 S Qi,a, E Lr U Urm=i] - -EP.Q
m+1
The probability in the last term is equivalently taken with respect to St, ... "t,...,,mm... S,, ,SN itd and thus we have
~ Q
P S fy Qm+ , SLNNON (Sm+1 S Qu-a) 2 a,
by the standard conformal prediction coverage guarantee (). and
m+1
Po (tt.siinmttttt...00000 {Sm+1 E [1rmirs Urmiall
((S E [1r,, 21-3.
m +1
24

---

by Lemma 3.1 Therefore, by the union bound, we have
P Ym+ E (((m++)) -1-0-
m
Next, defining Qi in Section 3.2. the events Sm+1 [Trmmt -m--il and Sm-1 < Pi-a together imply T(Sm+1) S Q a- by the construction of T, and thus we have
(T(Sm++) S Q Q1-a} s (Sm+1 Qi-a or Sm+1 [Lrm-r UrmtelJ.
Therefore, applying arguments analogous to the ones above, have
P {Ym+1= E ((Xmt1))
$ PP S 1 SuNNQQ
m+1 (Smt1 < Qi-a or Sm+1 & (Trmer Urmmil) + EPP
1
$ - +
EP
m+1
+
N
+1
since { Ūi-a) 5 a
N+1
under exchangeability due the standard conformal
prediction coverage guarantee (5)
G.3 Proof of Theorem 3.5
Let us define rm+11 as in the proof of Theorem 3.3 scores are almost surely all distinct, and their order is well-defined. Now observe the deterministic relation
By the continuity assumption on Q, the synthetic
(R Fm+1 $T(1- + 1)1} =
& Qi-aa C. {Ym+1 E Õ(Xm+1)3
(17)
c (Lm(( S Q1-a] ( S (11- ((11-)) a)(N 1)1),
which holds by the construction of 0(Xm+1) and the definition of the interval [Tm((mm1)) Um(mm+1)]- Therefore, the desired inequalities directly follow from the fact that Tmtt1 Unif[[m + 1)) due the exchangeability of the scores (S:)xe(m++]-
G.4 Proof of Proposition 3.2
We show that for any x E X, the following relation holds:
((x) A Fhatt (x) C {v E 3 : s(x, v) E ( :j [N]]]
The claim then follows directly from the continuity of Q.
Fix any X. is sufficient to prove that for any in the set A == (3/ ssxx v') & ($ ;j [N]]).
we have E ((x) if and only if y E Ötast holds.
Let us first take any E Clast (x²) D.A. A, and define ,.(.,)) have the following:
Em 1 1s: < s(1,1)) + 1. Then we
v E Etast (x) =((s(x,)) S @i Jana]nffe (( (f S (( (
u)) v) s S(tt))
((s(z.)) <Qiiaa } n (22))) R- }) U (a:1) < <Rt
since y E A
(fs(z.)) ai-a } n Tam (r (w(w))) Q ) U (n & Q Q1-a3
and the final set can be expressed as disjoint union of two events:
s(x,g) v) < Qi-a and Lm (r((a))) & Qi-a Um ((aa))) (ii) Una (1.2)) S Qi-a-
Note that in the case (ii). T(s(E, v)) Q directly follows, since T(s(x, v)) S Un (1(a)) holds deterministically. In the case (i), we have s(x, v) I Qi-o Um (r and thus T(s(2, y)) is
25

---

equal to either Lm (r(-1) NN (ritt) sxx, v)), which are both less than equal to Qi-a: the first by the condition (i), the second by the definition of NN Therefore, in either case, have y E C(z).
Next, to prove the contrapositive, let y & ( more precisely, y E Ö (x) n.A From the observations above, have
( & Chesst (x)} (((2)))>(0i-a) u Lmm (r((11)))) > -i-aa) n Um Um((mtt (2.00) > Q1-a }
(( (<<(x.)) > 0i-a n {mm((1nt1) (
(a.1) a1 @1-a]) u (m (1.2))
2 )
where the second equality applies De Morgan's 's law. The final set a disjoint union of the following two events:
(i v( > Qi and Lmn (1144)) S Qi-a Un (m(tt)))
(ii) Lm (f (att) Qi-a.
In case (ii), have (ss y)) Qi-a: since ((s(I, f)) Lm (m( holds deterministically. In case (i), Ts((,, v)) is equal to either Um (rm44) or (r (a.1) s(2, v)), which are both larger than Qi-a. This can be concluded as follows: In this case, have s(x,)) > Qi-a Z Q1-a- If Lm Lm(r (1(4)) s(x,y) Um(r by the construction of [s((2, y)), we have T(s 9) NN m (xx, 2 Ūi-a 2 Qi-a. and moreover we cannot have equality, since E A. Otherwise, s(xr, y) 2 Um(r ), and therefore, T(s(2,)) = Um (m(rm-)) (2.2)) Q Therefore, in
(x.9)
both cases, have & ((x), as desired.
26

---

H Experimental details
H.1 Setup and environment
The experiments were conducted system running Ubuntu 20.04.6 LTS, with 192 CPU cores of Intel(R) Xe) Gold CPUs at 2.40 GHz, TB of RAM, and 16 NVIDIA A40 GPUs. The software environment used Python 3.11.5, PyTorch 2.6, and CUDA 12.2.
H.2
Datasets
Our experiments involve two datasets: ImageNet for image classification tasks and the Medical Expenditure Panel Survey (MEPS) for regression tasks.
ImageNet [4]
We use the training split of ImageNet, focusing on 30 selected classes, which are listed in Table ST
MEPS: The MEPS dataset is a medical survey used for regression tasks, with the goal of predicting healthcare expenditures. For the regression experiments: MEPS-19 []. MEPS-20 [. and MEPS-21 [2. Each survey includes 139 features, such as demographic information (e.g., age, gender), and clinical data (e.g., chronic conditions, medical history).
H.3 Model details
We have applied the following models to compute the nonconformity scores:
ImageNet experiments: We employed a CLIP model based ViT-B/32 backbone, pre-trained on the LAION-2B dataset 227 50 using the HuggingFace API. Table reports the top-1 and top-2 accuracies of this model on the ImageNet training set for the subset of classes used in our experiments.
MEPS experiments: The dataset was filtered to include only non-Hispanic White and non- White individuals. Panel-specific variables were renamed for consistency across panels 19- 21, and rows with missing or invalid values were removed. A healthcare utilization variable was computed as the sum of expenditures across outpatient, office-based, emergency room, inpatient, and home health services, serving as the regression target. Preprocessing steps included retaining common features across panels, standardizing covariates, and applying log transformation to the target variable to reduce skewness.
A deep neural network was trained estimate the lower and upper quantile bounds of healthcare utilization using quantile regression approach, with different a levels used for each experiment. The network architecture consisted of four hidden layers (256, 128, 64, and 32 units) with LeakyReLU activations, dropout regularization (rate = 0.3), and optional batch normalization. The model was optimized using the pinball loss function and trained on 2019 data with early stopping based on validation loss (up to 50 epochs, batch size 128, learning rate le-4).
H.4 Data generation
H.4.1
Stable Diffusion
We generated synthetic images using the Stable Diffusion v1.5 model [46). For each class listed in we generated 2,000 images using the following configuration:
Table ST
Prompt: "A photo of [class name]", where (class name) refers to the corresponding ImageNet label, as shown be effective in 43
Inference steps: 260
Guidance scale: 7.5
Figure S5
presents examples of generated images alongside real ImageNet training images from the same class.
27

---

H.4.2 FLUX
We generated synthetic images using the FLUX.1 model (30 from Black Forest Labs, For each class listed Table [ST] we generated 2,000 images using the following configuration:
Prompt: "A photo of class name", where class name refers to the corresponding ImageNet
label, as shown to be effective in
Inference steps: 50
Images were generated using the FluxPipeline from the diffusers library, utilizing NVIDIA GPUs for acceleration with mixed-precision (float16) computation. The generation process was parallelized across multiple GPUs. Figure [ presentss examples of generated images alongside real ImageNet training images from the same class.
Table S1: Per-class accuracies of the pre-trained CLIP model with VIT backbone on ImageNet. The first columns (Top-l and Top-2 accuracy) are computed over all ImageNet classes, while the last two columns are computed only over the subset of classes shown in this table.
Class
Top-1 (%)
Top-2 (%)
Top-1 (%)
Top-2 (%)
Junco, snowbird
91.8
95.1
94.7
98.3
Bulbul
89.8
96.2
96
99.5
Jay
9.6
22
29
58.3
Magpie
88.2
93.2
94.5
97.8
Golden retriever
66.5
78.4
83,9
95.5
Labrador retriever
53.8
66.3
83.9
94.2
English springer
58.7
79.9
96.2
97.7
Kuvasz
65.5
83.5
97.9
Siberian husky
13.8
40
87.3
94.2
Marmot
47.5
66
75.4
98.5
Beaver
59.6
73.5
93.6
98.5
Bicycle
91.5
96.2
97.8
99.9
Lighter, Light
35.3
44.3
72.2
84.8
Muzzle
52.5
62.1
89.2
94.5
Tennis ball
65.4
76.8
88.4
93.8
Torch
44.8
60.7
85.2
95.2
Unicycle
66.3
80.5
83.2
96.5
White wolf
63.9
79.5
87.3
95.4
Water ouzel
88.9
93.5
93.1
96.1
American robin
87.2
92.3
94.5
98.2
Admiral
0.1
0.1
0.1
0.1
Rock beauty
4.6
28.9
66.6
89.5
Papillon
59.8
73
93.8
97.3
Lycaenid butterfly
70.1
92.8
95.3
99.4
Gyromitra
0.1
0.1
0.1
0.1
Coral fungus
78.9
91
87.6
98.8
Stinkhorn
65.3
79.2
87.1
97.7
Barracouta
1.4
7.1
4.4
41.8
Garfish
48.2
64.8
85.8
94.3
Tinca tinca
89.4
94.2
96.5
98.8
28

---

Figure S5: Comparison between real and Stable Diffusion- Diffusion-generatel images for selected ImageNet classes. Each row corresponds to a class, with the first column showing a real ImageNet image and the remaining columns showing generated datapoints.
Figure S6: Comparison between real and FLUX-generated images for selected ImageNet classes. Each row corresponds to class, with the first column showing a real ImageNet image and the remaining columns showing generated datapoints.
29

---

I Simulated data experiments
In this section, we present controlled experiments on simulated data, where the distributions of the real and synthetic scores are known. The goal is to compute and visualize the bounds from Theorem 3.3 and Theorem 3.5 The former provides bounds that depend on the total variation (TV) distance between the score distributions of the real and synthetic data, while the latter provides user-specified bounds (via the parameter 3) that not depend on these distributions. Our theoretical guarantees rely on both theorems, where the effective bound in each case is given by the tighter of the two, as illustrated in the following experiment.
Data and setup. We consider a simple regression setting with X = X = O, real outcomes ~ (5,, 1), and synthetic outcomes Y ~ N(u, 1). where μ E {4, 4.4, 4.6, 4.9.4.9, 5} controls the discrepancy between the two distributions. We use the absolute residual score function s(X, (x(X) Yl. where jx(X) = X. The real dataset includes m = 15 samples, while the synthetic dataset includes N 000 samples. The test set consists of 1, 000 real samples. We set = 0.05, and report results averaged over 100 independent trials.
function of the TV distance for target levels 0.05 and 0.1. The bounds derived from Theorem 3.5] denoted [WC], remain constant across different TV distances since they do not depend on the underlying data distributions. In contrast, the bounds from Theorem 3.3 denoted [TV]. vary with the TV distance and become looser as the discrepancy between the real and synthetic distributions increases. The shaded yellow line indicates the effective bounds-definee as the tighter the two in each case.
Figure S7 shows the performance of all methods
Following that figure, SPI achieves coverage within the effective bounds, as guaranteed by The- orems 3.3 and 3.5 Notably, for large TV distances whhaee the synthetic and real data differ substantially the bounds from Theorem 3.5 dominate, providing the tighter guarantee even for small /3 value. Moreover, as the user-specified parameter increases, the [WC] bounds approach the nominal level 1
OnlyReal
1.0
OnlySynth
Coverage
Coverage
SPI
0.8
0.8
Effective bound
0.6
Lower bound [WC]
Upper bound [WC]
0.04
0.159
0.236
0.311
0,383
0.04
0.08
0.159
02236
0,311
0.383
Lower bound [TV]
TV distance
TV distance
Upper bound [TV]
(a) a 0.05
(b) 0.1
Figure S7: Results simulated data: Coverage rate function of the TV distance between the
real and synthetic score distributions. Bounds from Theorem 3.3 are labeled [TV], and those from Theorem 3.5 are labeled [WC]. Effective bounds comnimum/maximun of the two) are shown in yellow. Results are shown for a = 0.05 (a) and 0.1 (b), with 8 = 0.05.
J Additional ImageNet experiments
This section provides supplementary results that complement those in Section including additional experiments on the ImageNet dataset. The following two subsections Appendices J.I and [.2 correspond Sections 4.1.I and 4.1.2 of the main manuscript, respectively, and follow the same experimental settings.
J.1 Experiments with generated synthetic data
Figure presents the performance under both marginal and label-conditional guarantees at levels = 0.02 and 0.1. We observe a similar trend to that seen in Figure Following that figure, we can see that the standard conformal prediction method, OnlyReal, controls the coverage at the a level as expected, but it produces overly conservative prediction sets. OnlySynth method fails to
30

---

achieve the target coverage level of 1 under under-covering some classes, while in others, it becomes overly conservative, depending on the unknown distribution shift between the real and synthetic data. In contrast, the proposed method, SPI, stays within the theoretical bounds and produces informative prediction sets.
Coverage
Sizee
20
0.8
Marginal
Siborian barsty
Beaver rabio
American
teg
Harracouta
Marginal
Sibertian husky
Heover rabin
American Light
Lighter:
Harracouta
Light
Class
Class
0.02
1.0
30
OnlyReal
Coverage
0.8
Size
20
OnlySynth
0.6
10
1-
Lower-bound
Marginal
Siberion busky
Banver robuo
Americaa
Light
Lightor.
Barracouth
Manginal
Siwrran busky
Beaver robin
American
Lighter Light
Baraccuta
Upper-bound
Class
Class
(b) 0.1
Figure S8: Results for the ImageNet data: Coverage rates of OnlyReal, OnlySynth, and SPI run at level 0.02 (a) and 0.1 (b), averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box in each group) and label-conditional coverage settings. Label-conditional results are shown for selected classes; see Tables S2 anddSA for results across all classes
Tables $2 to present results for classes in the real calibration set corresponding to Figure in the main manuscript and Figures S8a and S8b above.
31

---

Table S2: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials. Standard errors are shown in parentheses. The target coverage level is a = 0.98. The theoretical coverage guarantees for SPI are in the range [93.7, 100). Other details are as in Figure S8
Coverage (%)
Size
Class
Only Real
Only Synth
SPI
Only Real
Only Synth
SPI
Admiral
100 (+ 0)
6.9 (# 0.3)
(3.6
30 (+ 0)
4.5 0)
6.1
American robin
100 (+ 0)
94.8 0.1)
(6.6
30 (=)
2. (+
(.
Barracouta
100 (# 0)
99.8 0)
(*
30 (=
7+2
Beaver
100 (+ 0)
(1.1
95.3(#
30
(=
(+
Bicycle
100 (# 0)
(7.1(+ (6.3
(0
(+
(+5
Bulbul
100 (+ 0)
(9)4
(9)5
(0
2.6(#))
(.7
Coral fungus
100
(9
(9.4
30
2+7
(+3
English springer
100 0)
94.1(#0.1)
(6.5()
(0
2.9(00)
4.1(
Garfish
100 0)
(2.1(
(5.3()
(4
4.6(#0)
(.9
Golden retriever
100 (= 0)
(4.1
(6.2))
(4
4*3(00)
5.8
Gyromitra
100 (= 0)
(2.2(
(6.3))
(0
(.1
3+7
Jay
100 (+ 0)
92.0(#0.))
95.6(+
(0
6.4
8.0
Junco, snowbird
100 0)
07.)
98.0(#0.1)
(0
2.3(0))
3.1
Kuvasz
100 0)
00.1
96.5
(0
2.8(20)
4.0
Labrador retriever
100 0)
(6.2(
97.((0.1))
(0
4.6(#0)
6.3
Lighter, Light
100 (- 0)
08.0(#0.1)
98.3(
(0
4.9
6.2
Lycaenid butterfly
100 0)
(3.5
(4.7)
(0
2.9
4.1
Magpie
100 0)
06.7
(..1
(0
2.5
3.4
Marmot
100 (= 0)
(a.1
(7.0)
(=
4.0(00)
5.6
Muzzle
100 (= 0)
(7)5
0..0
(0
3.5
4.9
Papillon
100 (+ 0)
(2.2)
(4.44
(0
(+1
(.3
Rock beauty
100 (+ 0)
90.6
(5.3)
(0
4.8(#0)
6.0
Siberian husky
100 (# 0)
(9.7
(4.1
(0
3,2
4.6
Stinkhorn
100 (+ 0)
(..1
(8.2(+
30
4.6(#0)
5.5
Tennis ball
100 (+ 0)
97.5
98.0(#0.1)
30
(,1
4.0(#))
Tinca tinca
100 ()
(8))
98.6(+
30
3.3(0))
4.2
Torch
100 (+ 0)
98
98.7(+
(0
5.6
7.1
Unicycle
100 ()
(6.3
(4.1
30
(+2
5.5
Water ouzel
100 (#
(5.7
97.0(+0.2)
30
2.5
3.3
White wolf
100 (# 0) (5.5
(4.5)
(0
(+
(+0
32

---

Table S3: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials. Standard errors are shown parentheses. The target coverage level is a = 0.95. The theoretical coverage guarantees for SPI are in the range [93.7, 100). Other details are in Figure
Coverage (%)
Size
Class
Only Real
Only Synth
SPI
Only Real
Only Synth
SPI
Admiral
100 (+ 0)
0.6 (# 0)
(3.6
30 (+ 0)
3.6 0)
5.8 0)
American robin
100 (+ 0)
90.5
95.4
(=)
1+8
3+
Barracouta
100 (# 0)
99.3 ()
(*
30 (=
(+
6.8
Beaver
100 (= 0)
(5.1
(4.55)
(=
(=
(*9
Bicycle
100 (+ 0)
(2.5
0..3
(0
2+3(+0)
3+3
Bulbul
100 (+ 0)
(8.4
(8.6()
(0
2.0(#0)
3.4
Coral fungus
100 (+ 0)
(8.0(
30
2.1(00)
3.1
English springer
100 0)
90.1
95.2
(0
2,2(00)
3.8
Garfish
100 0)
(6.1
(4.2
(0
3.7(00)
(.6(+))
Golden retriever
100 (+ 0)
(8.2
(4.4)
(4
3.3(=0)
5.3
Gyromitra
100 (= 0)
80.3(#0.))
(5..)
(0
2.7
3.6(#0)
Jay
100 (+ 0)
(0.
93.7
(0
4.8(#0)
7.4
Junco, snowbird
100 (= 0)
(4.7(#0.1))
(6.2
(0
1.7
(+9
Kuvasz
100 0)
01.6(#0.11
95.1)
30
2.1
3.7
Labrador retriever
100 0)
(1.1
95.6(#0.3)
(0
3.6(00)
5.8
Lighter, Light
100 (= 0)
(5.1
96.8
(0
3.9
5.8
Lycaenid butterfly
100 0)
(8.1
94.5(#04))
(0
2.3(0))
4.0
Magpie
100 0)
0..1
(5.2
30
1.9
3.1
Marmot
100 (= 0)
(3.1(
96.2
(0
3,2(00)
5.2
Muzzle
100 (= 0)
(5.1
(..2)
30
2.8(00)
4.6(
Papillon
100 (+ 0)
94.1 (z.2
(0
2.3(0))
(.0(#0)
Rock beauty
100 (+ 0)
(z.5
(4.5
30
3.5(0))
5.5
Siberian husky
100 (# 0)
(6.3
(..1
(0
2.3(0))
(.2
Stinkhorn
100 (+ 0)
05.6(#0.11
96.6(#
30
3.6(#0)
5.1
Tennis ball
100 (+
94.3(
96.0
30
2.5(0))
(,7
Tinca tinca
100 (=
(6.5
(0.1
30
2.6(0))
4.0
Torch
100 (+ 0)
96.7
07.6
30
6.6
Unicycle
100 ()
(2.1(
(5.3)
30
(+
5.1
Water ouzel
100 (= 0)
92.6(
95.7
30
(+
3.1
White wolf
100 (# 0)
(..2)
(3.9
(4
(+
(+7
33

---

Table S4: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials. Standard errors are shown in parentheses. The target coverage level is 1 a = 0.9. The theoretical coverage guarantees for SPI are in the range (81.2, 93.7]. Other details as in Figure S8
Coverage (%)
Size
Class
Only Real
Only Synth
SPI
Only Real
Only Synth
SPI
Admiral
(3
0.2 0)
(1
5. 0)
3.1 (+
4.3
American robin
(4 (.
(..5))
(+2
(=)
199
Barracouta
97.8 0z.4)
6.1 (4.4))
4.9 (=
Beaver
0..6
(8.1
83.6
3.6(#..1)
(=8
2.3
Bicycle
(7.1 (4.5(+
88.2(+0.))
3,2(=0)
2.0
2.3(#0)
Bulbul
(..
95.1(=
92.6(=0.)
3.((0)
1.6(#))
2.1
Coral fungus
(4.6))
97.7(#0.1)
93.2(#0.))
299(0)
1.7
2.1
English springer
(..9(#
83.1
(5.3(=04)
3.6(0))
1.5(0))
2,2(0))
Garfish
(4.6
79.2(#02)
5.4(0..1 83.((+0.5)
322(0))
4.0(#0)
Golden retriever
(..1))
82.2(+
85.8(=0.5)
5.0(0.11)
2.3(#0)
3.1
Gyromitra
95.0
64.1
85.2
3.3(=))
2.3(=0)
2.5(00)
Jay
(3.7
(..1)
80.9(#09)
6.8(0..))
3.7
4,8(#0)
Junco, snowbird
90.0(#0.1) (4.5))
(9.44
2.7(0))
1.300))
1.7
Kuvasz
85.0 (..66)
(6.3
3.5(=))
1.5(0))
2.0
Labrador retriever
(3.5()
84.6(#0.))
85.5(
5.4(0.1)
2.7
3,5
Lighter, Light
(..2(+
90.4(
89.6(#04)
5.4(0.1)
322
3.9(0))
Lycaenid butterfly
94.0(+0.5)
(H.22
(5.7
3.9(0..))
1.9
2.5
Magpie
(..6
(2.2)
(..5))
3.0(=))
1.4
1.9
Marmot
94.0(+0.6)
89.8(#0.1)
(4.44
4.9(0..)
266(0))
3.1
Muzzle
0..6
(2.1(
(..5(
43((0)
2.3
3.0
Papillon
(.06))
75.8(#022)
82.4
3.8(0.1))
(.7(00)
2.4+0))
Rock beauty
44.6 (+.5)
1..0
5.1
2.5
3.8(00)
Siberian husky
(4.3 (4.6))
(0.1
4.0(#0)
1.5
24((0)
Stinkhorn
92.2(+0.1) (+06))
(0.7(+
4.5(0..)
28
3,2
Tennis ball
(+.6))
88.6(
88.6(#0.))
3,5(0))
2.0
2.5
Tinca tinca
93.2(+0.1) (3.2
91.2(+0.5)
3.8(00)
2.1(=0)
2.7
Torch
93.9(#0.1) (4.5)
92.6(+0.4)
6.2
3.8
4.6(2
Unicycle
(4..6)
(6.5
(2.14)
4.8(00)
288
3.3
Water ouzel
(4.5)
82.7(
88.3))
3.0(0))
1.4
1.9
White wolf
(4.6))
(4.1
(2.7
3.4
1.5
2.1 (+
34

---

J.1.1 The effect of the real calibration set size
Here, we evaluate the performance of different methods as function of the real calibration set size m, following the same setup described Section [4..T]] This parameter directly affects the performance of both the standard conformal prediction method, OnlyReal, and proposed method, SPI, including theoretical bounds established in Theorem 3.5 In contrast, OnlySyuth, which relies solely on the synthetic calibration set, is unaffected by changes in m. As such, it serves useful baseline for assessing how well the synthetic calibration set aligns with the real one.
Figure S9 presents the performance of all methods for the "Lighter" class across varying values of m and a levels. Notably, although OnlySynth does not have formal coverage guarantees, its empirical coverage closely matches the target level - a. This alignment suggests that the synthetic calibration data approximate the real distribution well.
Coverage
0.98
0.96
Size
0.94
5-
m
0.02
1.00
Covorage
0.95
0.90
Size
0.85
10
15
50
10
15
25
50
(b)
0.05
1.0
30
OnlyReal
Coverage
0.9
10
OnlySynth SPI
0.8
0.74
Size
1-a a
Lower-bound Upper-bound
0.6
0.5
25
60
15
50
(c)
Figure S9: Results for the ImageNet data: Coverage rate for OnlyReal, OnlySynth, and SPI on the "Lighter" class as function of the real calibration set size m, for levels a 0.02 (a), a 0.05 (b), and 0.1 (c).
Figure $9a presents results for a = 0.02. At this low level, the standard conformal prediction, OnlyReal, controls the coverage at level o, but-as expected-producce. trivial prediction sets when m < 50.
In contrast, proposed method, SPI, achieves coverage within the theoretical bounds even for small m, with reduced variance in coverage and smaller prediction sets for m > 15. Interestingly, for 0.02 and m 5 or 10, the theoretical lower and upper coverage bounds are both equal to unity, indicating that we know a priori that the proposed method yields trivial prediction sets for this window construction.
respectively), we observe similar trends. Our method, SPI, consistently achieves coverage within the theoretical bounds, remaining close to the target coverage level - at while also exhibiting reduced variance in coverage and producing smaller, more informative prediction sets compared to the baseline, OnlyReal.
For a 0.05 and = 0.1 (Figures 59b and S9c
35

---

Additionally, Figure S1O presents the same experiment as in Figure S9 but for the "Beaver" class. In this case, the OnlySynth method yields coverage that falls significantly below the target level a. indicating that the synthetic calibration set differs substantially from the real data. Nevertheless, our proposed method, SPI, achieves coverage within the theoretical bounds across all levels and calibration set sizes, while also producing informative prediction sets.
Coverage
0.975
0.950
Size
0.925
0.900
0.875
10
0.02
1.00
30
Coverage
10-
Size
0.85
0.80
10
15
25
50
15
25
50
0.05
OnlyReal OnlySynth
Coverage
0.9
SPI
Size
1 - a Lower-bound Upper-bound
0.7-
m
m
0.1
Figure S10: Results for the ImageNet data: Coverage rate for OnlyReal, OnlySynth, and SPI on the Beaver class as function of the real calibration size m, for levels 0.02 (a), 0.05 (b), and a = 0.1 (c).
36

---

J.1.2 The effect of the hyperparameter B
In this section, we examine the effect of the hyperparameter 3 on the performance of our proposed method, SPI. At a theoretical level, the sensitivity of SPI to depends on the similarity between the synthetic and real score distributions. Specifically, this can be seen in the score-transportation step: when the two score distributions differ substantially, the transported score T(Sm+1) is likely to lie at one of the endpoints of the corresponding window-eve under small B. As a result, the
coverage of the SPI prediction set approaches the guardrail bound in Theorem 3.5 indicating that we gain little from the synthetic data.
On the other hand, when the synthetic scores closely resemble the real scores, the transported score is likely to lie within the corresponding window rather than in its endpoints
for relatively large values of B. In this case, the SPI prediction set resembles conformal prediction with larger sample size, achieving coverage close to the nominal 1 - a level, and the effect of the hyperparameter B becomes relatively minor.
Figure STI illustrates this behavior, showing the coverage of all methods for different 3 values across plasses-lighted (left column) and Siberian husky (right column) classes-al at target levels = 0.02, 0.05, and 0.1. Following that figure, for all B values, the empirical coverage of SPI lies within the theoretical bounds of Theorem 3.5
The results align with the behavior described above: for the lighter class (left column), the synthetic scores closely resemble the real ones, and accordingly, the performance of SPI remains roughly the same across different 3 values, achieving coverage close to the nominal 1 - a level. In contrast, for the Siberian husky class (right column), where the synthetic scores deviate significantly from the real scores, the coverage of SPI closely follows the guardrail bounds. Note that this figure also presents how the lower bound of Theorem 3.5 increases with B, becoming closer to the nominal 1 level.
1.00
Coverage
Coverage
0.90
0.85
0.01
011
0.2
0.3
0.4
0.5
B
B
0.02
1.00
Coverage
0.95
Coverage
0.9
0.90
0.8
0.8.5
0.7
0.80
0.6-
8
18
0.75
0.4
B
(b)
0.05
OnlyReal
OnlySynth SPI 1-a
Coverage
Coverage
0.8
Lower-bound
0.7
Upper-bound
0.6
0.4-
0.005
0.01
0.05
0.1
0.2
0.4
0.5
0.005
0.01
0.05
0.1
0.4
0.5
B
8
(c) a 0.1
Figure S11: Results for the ImageNet data: Coverage for OnlyReal, OnlySynth, and SPI function 3. Results are displayed for the Siberian husky (right column) and lighter (left column) classes, at levels 0.02 (a), 0.05 (b), and 0.1 (c).
37

---

J.1.3 Results for SPI with FLUX- FLUX-genereee -eenerttdd synthetic data
In this section, we evaluate the performance of the proposed method, SPI, using synthetic images generated by the FLUX.1 model [30]. The experimental setup follows the same procedure described in Section of the main manuscript. As before, we aim for both marginal and label-conditional coverage guarantees.
Figure S12 presents the marginal and label-conditional coverage of various methods at levels a 0.02, 0.05, and 0.1. The results for cubelcconittionl guarantees are presented for representative classes; results for all classes in the real population are detailed in Tables [S5 to S7 We observe similar trends to those observed using synthetic images generated by Stable Diffusion. The standard conformal method, OnlyReal, controls the coverage at the 1 a level; however, it yields overly conservative prediction sets due the small sample size. OnlySynth fails control the coverage at the desired level, exhibiting under-coverage for some classes and over-coverage for others. In contrast, the proposed method, SPI, achieves coverage within the theoretical bounds while providing smaller, more informative prediction sets.
1.0
30
30
Coverage
10
Size
20-
0.8
Marginal
Siborsalf Wotrrrrrowww pitrever
Labradaf F
American
Magple
Iicyycle
Marginal
Stbortb6 tt treviar
Labraddpplon
Amerkca
Magple
Bicycle
busky
busky
Class
Class
(a)
0.02
Coverage
0.8-
Size
0.6
Mardinal
Siberddf gir periover
Labrado Pobia
American
Magpie
Bicyclo
Marginal
Sther Blerrgrriee
Labraci
American
Mouple
Bleccl
husky
husky
Class
Class
0.05
1.0
OnlyReal OnlySynth
Coverage
0.8
Size
SPI
1 - a
Lower-bound
Marginal
Stbor patriovar busk
Labrado Fabin
America
Magpie
Micycle
Marcinal
Stbers (arrover
Labradd Pabin
Amorican
Magpin
Blcycle
hasky
Upper-bound
Class
Class
(c) 0.1
Figure S12: Results for the ImageNet data using FLUX-generated synthetic images: Coverage rates of OnlyReal, OnlySynth, and SPI run level 0.02 (a), 0.05 (b), and 0.1 (b), averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box in each group) and label-conditional coverage settings. Label-conditional results are shown for selected classes; see Tables S5 to S7 for results across all classes.
38

---

Table S5: Per-class conditional coverage (in %) and prediction size for each method, computed
over 100 trials using FLUX-generated synthetic data. The target coverage level is 1 = 0.98. The theoretical coverage guarantees for SPI in the range [93.7, 100). Standard errors shown in parentheses. Other experimental details follow Figure ST2
Coverage (%)
Size
Class
Only Real
Only Synth
Only Real
Only Synth
SPI
Admiral
100 (# 0)
0.2 (# 0)
93.6 (+
30 (#
4.7 (# 0)
6.5 0)
American robin
100 (+ 0)
95.5 (+
96.8 (+2))
30 (=
2.7 (# 0)
4.3
Barracouta
100 (# 0)
(*)
30 (=
(+
8.2
Beaver
100 (+ 0)
(..7
(4.44
30 (=
(=
()
Bicycle
100 (= 0)
(*
()
30 (=
(.8
4.8
Bulbul
100 (+ 0)
(z.3
(7.11
(=
3,2
(.7
Coral fungus
100 0)
(9)6
(9
(0
2.9
(+
English springer
100 0)
96.2
(7.3(
(=
3,3
(=1
Garfish
100 (1 0)
(0.1
(+.3)
(0
4.9
(.9
Golden retriever
100 (+ 0)
(3.6(
(6.0()
(=
4.6(#0)
6.9
Gyromitra
100 (+ 0)
(7.2
95.0(#
30
3.6
(+5
Jay
100 0)
(0.1(
93.3
(0
6.9
9.1
Junco, snowbird
100 0)
(9.2
99.3
30
2.4
(+6
Kuvasz
100 0)
99)4
99.4
30
(.6
Labrador retriever
100 0)
91.6(#02)
95.5(#
30
4.9
(#4
Lighter, Light
100 0)
(5.6
94.4
30
(.7
(=0
Lycaenid butterfly
100 0)
(5.2) (2.1)
(0
(.8
4.7 (+7
Magpie
100 (+ 0)
(z.0(
97.6(
(0
3.0
(+
Marmot
100 (+ 0)
(8.0(#0.1)
(8.0))
30
4.9
6.9
Muzzle
100 (+ 0)
(..1
(1.2
(0
3.7(#0)
(.
Papillon
100 (+ 0)
(9)9
(9.9
(0
2.5
(#4
Rock beauty
100 (+ 0)
03.6(#0.1)
(..0
(0
4.9
(+1
Siberian husky
100 (+
(8.2
94.1
(0
3.4(0)
(*5
Stinkhomn
100 0)
98)3
98.5()
30
5.1
(+5
Tennis ball
100 (=
(9.1
94.5
(0
34(00)
(+
Tinca tinca
100 ()
(9))
(9)4
(0
3.3
5.0
Torch
100 (+
(a.11
95.3
(0
6.0
8.5
Unicycle
100 (+
(9)
99.8
30
4.8
6.6
Water ouzel
100 (+ 0)
99.1
99,2
(0
2.
(.
White wolf
100 (+ 0)
03.2
(4.5
(0
3.4
5.1 (=
39

---

Table S6: Per-class conditional coverage (in %) and prediction size for each method, computed
over 100 trials using FLUX-generated synthetic data. The target coverage level is 1 = 0.95. The theoretical coverage guarantees for SPI in the range [93.7, 100). Standard errors shown in parentheses. Other experimental details follow Figure ST2
Coverage (%)
Size
Class
Only Real
Only Synth
Only Real
Only Synth
SPI
Admiral
100 (# 0)
0.2 (# 0)
(
30 (+ 0)
4.4 (# 0)
6.3 (# 0)
American robin
100 (+ 0)
91.7 0.1)
(..3)
30 (=)
2.0 (+ 0)
3.8 (+
Barracouta
100 (# 0)
99.9 (+
99.9 ()
30 (=
(+
7.7
Beaver
100 (+ 0)
81.0
(4.1)
(= (=
(+
Bicycle
100 (= 0)
())
(9
30 (=
3.1
(+2
Bulbul
100 (+ 0)
(3.1(#
96.2()
(0
2.6
(+
Coral fungus
100 0)
(9 (9)4
(0
2.4
(+6(2
English springer
100 0)
(2.1
(4.2)
(=
3.0
(.9(2))
Garfish
100 (= 0)
(4.7
(+.5)
(0
(.0
(.3
Golden retriever
100 (+ 0)
(..1
(5.0
(0
(,
6.5
Gyromitra
100 (+ 0)
46.6(#0.2)
(5.0
30
3.0
4.1
Jay
100 0)
(1.7
93.7
(0
5.7
8.5
Junco, snowbird
100 (+ 0)
(7.1
(0..))
30
2.0
3.3
Kuvasz
100 0)
99))
99)3
30
2.3
(+3
Labrador retriever
100 0)
(.02)
(4.5)
30
4.1
6.9
Lighter, Light
100 0)
(..2
94.2(#0.6)
30
3.9
6.7
Lycaenid butterfly
100 0)
(2.1)
94.6(+0.4)
(0
3.3(0))
(.6(#))
Magpie
100 (+ 0)
0..0(
95.8(+
(0
2.5
(.0
Marmot
100 (+ 0)
(6.1
(7.7
30
3.9
6.2
Muzzle
100 (+ 0)
(..1
(6.2
(0
3.2(00)
(+
Papillon
100 (+ 0)
(9)9
(9.9
(0
2.2
(+3
Rock beauty
100 (+ 0)
(..1
(..7
(0
4.4
(+
Siberian husky
100 (+ 0)
(1.2
94.1
(0
2.9
(.2
Stinkhomn
100 ()
97.7( (0.1
30
4.4
(.0
Tennis ball
100 (=
(4.2
93.7
(0
2.8
(+4
Tinca tinca
100 ()
(8
(8.1()
(0
2.7
(.6
Torch
100 ()
(4.2)
95.2
30
5.2
8.0
Unicycle
100 (=
(9)
99.8
30
4.0
6.0
Water ouzel
100 (= 0)
98.9
99.0
30
2+0
(=
White wolf
100 (+ 0)
(..2)
(3.6)
30
2.8
4.7 (#
40

---

Table S7: Per-class conditional coverage (in %) and prediction size for each method, computed over 100 trials using FLUX-generse synthetic data. The target coverage level is - a 0.9. The theoretical coverage guarantees for SPI are in the range [81.2, 93.7]. Standard errors are shown in parentheses. Other experimental details follow Figure ST2
Coverage (%)
Size
Class
Only Real
Only Synth
SPI
Only Real
Only Synth
SPI
Admiral
93.6 (.66)
0.1 0)
5.6 (1
4.1 (# 0)
(.6
American robin
(.22 (.
(..4)
(=
1.5 (+)
(+0
Barracouta
0..4)
(9
(..44)
(.1
(=9
5.1
Beaver
9..0(+
(..2))
82.5(+
3.6(#..1)
(=4
2.5(
Bicycle
0.55
(6.1()
93.6(#0.5)
32((0)
2.7
2.5
Bulbul
93.6(+
88.2()
88.4(+0.4)
(+)(=))
1.9
2,2(=0)
Coral fungus
(..6
()
(3.6
29(=0)
2.0(00)
2,2
English springer
(..6
(3.01)
91.3(#0.4)
3.6(#))
2.5
2.5(0))
Garfish
(..6))
(2.2
5.4 (2.7
3.3
4.1
Golden retriever
(..6))
84.0(#0.2)
86.5()
5.0(0..1)
(.0(=0)
3.3
Gyromitra
(5.0
38.2(#
1..8
3.3(0))
2.
2.6(00)
Jay
23.5 (3.77
10.5
6.8(0.1)
(.)(=))
5,1
Junco, snowbird
(5.3)) (..5
(2.4(#0.)
2.7(0))
1.6
1.8
Kuvasz
(..6))
99.0
93.3(#0.))
3,5(=))
1.8(0))
2.1
Labrador retriever
(3.5(+
(1.5
84.5(+0.5)
5.4(0..1
3.3(=0)
3.7
Lighter, Light
(..2))
57.8(=02)
81,5(#09)
5.((0.1)
(.1
3,8(00)
Lycaenid butterfly
94.0(+0.5)
82.2
86.2(+0.5)
3.9(0..))
3.0(00)
3.0
Magpie
(..66)
(..9
88.7(+0.5)
3.0(0)
1.8
2.0
Marmot
94.0(+0.6)
96.0(#0.1)
92.9(+0.))
4.9(0..)
3.1
3.3(0))
Muzzle
(3.6
(0.5(
(0.5(#0.5)
43(00)
26
3.1
Papillon
93.6(+0.))
(9.7
93.8(#0.))
3.)(0.1)
19900)
25((0))
Rock beauty
(..5)
(0.2
84.5
5.1
3.7
4.1
Siberian husky
(4.1
03.2
80.6(#1.0)
4.0(#0)
2.2
2.6(20)
Stinkhom
(..66)
96.0(#..1)
(2.7(+
4.5(
(,5
3.4
Tennis ball
93.5(+0.))
(2.2
(2.7
3,5(0)
2.1
2.5
Tinca tinca
(3.2
(..1(
93.0(#0.6)
3.8(0))
2.2
2.8
Torch
(9.5)) (405)
84.5(+0.5)
6.2
(44
4.8
Unicycle
(..66)
99.7
(..6
4.8(=))
3.4
3.5
Water ouzel
(+0.5
98.8
94.2(+0.5)
3.0(#0)
1.6
2.0
White wolf
(..66)
(2.2
(2.8)
3.4
2+2
2.3
41

---

J.2 Experiments with auxiliary labeled data
In this section, we follow the experimental setup described in Section 4.1.2 where the synthetic data comprise 100 classes, none of which are included in the real calibration set.
Figure ST3 presents the results for both marginal and label-conditional guarantees at levels 0.05 and 0.1, demonstrating trends similar to those observed in Figure The standard conformal prediction, OnlyReal, conservatively controls coverage the target level a, but results in larger and noisier prediction sets due to the limited sample size. In contrast, both SPI-Whole and SPI-Subset substantially reduce the size and variance of the prediction sets and, as expected, achieve coverage within the theoretical bounds.
Notably, for the "American robin" and "Torch" classes, the SPI-Subset variant achieves coverage more tightly aligned with the target level a, outperforming the standard SPI-Whole method.
We include the results for all real classes in Tables S8 to STO corresponding to Figures ST3a and S13b respectively.
Coverage
0.9
10-
0.8
10
0.7
Marginal
Admiral
Lighter Lihht
Reaver
Torch robii
American
Marginal
Admira/
Lightor Light
Beaver
Torch rbin
americaz
Class
Class
0.05
1.0
20
Coverage
OnlyReal
0.8
Size
20:
SPI-Whole
SPI-Subset
0.6
1 a
Marginal
Atmiral Light
Lighter:
Beavor
Torch robia
American
Marginal
Admiral Aightt
Lighter
Braver
Tonch reban
Amercca
Lower-bound
Upper-bound
Class
Class
(b) 0.1
Figure S13: Results for the ImageNet data: Coverage rates of OnlyReal, SPI-Whole, and SPI-Subset at level 0.05 (a) and 0.1 (b), averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box each group) and label- conditional coverage settings. Labellconditional results shown for selected classes; Tables 59 and for results across all classes
42

---

Table S8: Per-class conditional coverage (in %) and prediction size for each method, computed over 100 trials. Standard errrrs are shown in parentheses. The target coverage level is 1 = 0.98. The theoretical coverage guarantees for both SPI-Whole and SPI-Subset are in the range [93.7, 100]. Other details are as in Figure
Coverage (%)
Size
Class
Only Real
SPI
Only Real
SPI Whole
Whole
Subset
Subset
Admiral
100 (# 0)
93.6 0.6)
99.8 ()
30 (=
7.6 (# 0.1)
6.5 (# 0)
American robin
100 (+ 0)
99.9 (+ (..1)
(..0()
(=
5.0 0.1))
4.4 (=
Barracouta
100 (# 0)
98.0
(=
9.
7.8 (.
Beaver
100 (+ 0)
(*
(8.1
(=
6.1
(+
Bicycle
100 (= 0)
(=
09.2(
(0
4.7
(+0
Bulbul
100 (+ 0)
(9)
99.0(#
(0
5.2
(+
Coral fungus
100 0)
99.7
(8.1(
30
4.1
(.7
English springer
100 0)
(90)
(8.1(
(=
5.7
(.7(#))
Garfish
100 (1 0)
(9)4
(+.2)
(0
8.0
(.6
Golden retriever
100 (+ 0)
(9)
98.0(#
(0
(..0).1)
6.5
Gyromitra
100 (+ 0)
(5.6
95.0()
30
4.6
(+3
Jay
100 0)
0..3
(0.1
(0
11.4
8.7
Junco, snowbird
100 0)
(9.9
(8.7(+
30
4.3
3.5
Kuvasz
100 0)
99)5
98.5
30
5.7
(.
Labrador retriever
100 0)
(4)3
97.6(+
30
8.7(#0.1)
(..1)
Lighter, Light
100 0)
0..1
(6.9
30
8.4(0.1))
6.6
Lycaenid butterfly
100 0)
(9)
(9.1))
30
5.2(0)
5.2
Magpie
100 (+ 0)
(9.7
98.6(#
30
4.7(0))
3.9
Marmot
100 (+ 0)
(9)7
(8.5()
30
7.5
6.4
Muzzle
100 (+ 0)
(..1
96.6)
30
6.7(2..1)
5.4+0)
Papillon
100 (+ 0)
(9)2
97.6(+0.2)
(0
5.7
5.1
Rock beauty
100 (+ 0)
0..1)
0..5
(0
8..
6.2
Siberian husky
100 (+ 0)
98.8(+
30
6.1
5,2
Stinkhomn
100 0)
()
(8.3()
30
7.1
6.0
Tennis ball
100 0)
08.6
96.6(#0.3)
(0
5.3(0)
(+
Tinca tinca
100 ()
(9))
(8.1()
(0
5.3
(+
Torch
100 ()
99)6
98.4(40.1)
30
10.3
(.
Unicycle
100 ( 0)
(9.7
(8.2
30
8.1
(.1
Water ouzel
100 (+ 0)
99.2
98.0(#0.1)
(0
5.0
4.0
White wolf
100 (+ 0)
(9.1
(7.1)
(0
5.5
4.6 (.
43

---

Table S9: Per-class conditional coverage (in %) and prediction size for each method, computed over 100 trials. Standard errоss are shown in parentheses. The target coverage level is 1 = 0.95. The theoretical coverage guarantees for both SPI-Whole and SPI-Subset are in the range [93.7, 100]. Other details are as in Figure S13
Coverage (%)
Size
Class
Only Real
SPI
Only Real
SPI Whole
Whole
Subset
Subset
Admiral
100 (# 0)
93.6 (= 0.6)
95.9
30 ()
5.9 ()
5,8 0)
American robin
100 (+ 0)
95.3 0.1))
(..6
30
3.4 ()
3.3 (+
Barracouta
100 (+ 0)
(5.3))
30 (=
6.9 7.1
Beaver
100 (+ 0)
0..6
(5.3
3.9 (=
(+7
Bicycle
100 (= 0)
99.3
(7.2(+
(0
3.4
(+
Bulbul
100 (+ 0)
(9)
96.2
(00)
3,5(=0)
(+4
Coral fungus
100 0)
(..1)
(6.3))
30
3.1
3.0
English springer
100 0)
(8.1
(7.2(
(0
3.9
3.8
Garfish
100 (# 0)
(6.2
(..2)
(0
5.8
5.6
Golden retriever
100 (+ 0)
(..7
(5.3)
(0
5.5
5.3
Gyromitra
100 (+ 0)
(5.0
(5.0
30
3.6(#0)
(+5
Jay
100 0)
(4.5(
95.3)
(0
7.9
7.2
Junco, snowbird
100 (+ 0)
08.1
96.6
30
3.0
2.8
Kuvasz
100 0)
08.5(
96.5
(0
3.9(0))
3.7
Labrador retriever
100 0)
(0.1
(5.3
30
6.0
5.7
Lighter, Light
100 0)
95.0(#044)
95.0(#0.5)
30
5.7
5.5
Lycaenid butterfly
100
(9)2
(6.2))
(0
4.2(00)
4.1
Magpie
100 (+ 0)
0..1
(6.4))
(0
3.2
3.1
Marmot
100 (+ 0)
(8.1
96.4
30
5.1
5.1
Muzzle
100 (+ 0)
(5.3
94.3
(0
4.6(#0)
4.5
Papillon
100 (+ 0)
(5.3) (z.1)
(0
4.0
4.0(#0)
Rock beauty
100 (+ 0)
(H.44
(5.6)
30 (0
5.6
5.3
Siberian husky
100 (+
(6.5
97.0(#0.2)
(0
4.3
4.1(#0)
Stinkhomn
100 0)
0..1
(5.2
30 (0
5.0
4.8(#0)
Tennis ball
100 0=
94.6(+0.4) (5.3)
(0
3.0
3.6(#0)
Tinca tinca
100 (=)
(..1)
(..2)
(0
4.0
(.9
Torch
100 (2)
(a.1
96.3
30
6.7
6.4
Unicycle
100 (
(..1
9..6
30
5.3
5.0
Water ouzel
100 (+ 0)
98.0(#0.11)
96.2
(0
(+
3.1
White wolf
100 (+ 0)
06.6
95.1 (+:.4)
(0
(+7
3.6
44

---

Table S10: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials. Standard errors are shown in parentheses. The target coverage level is 1 - 0.9. The theoretical coverage guarantee for both SPI-Whole and SPI-Subset are in the range [81.2, 93.7]. Other details are as in Figure S13
Coverage (%)
Size
Class
Only Real
SPI
SPI Subset
Only Real
SPI Subset
Whole
Whole
Admiral
23
(+
(.
5.6 0)
(.6
4.5 (+ 0)
American robin
(.55 (.
(= (+.5))
2.0 (=
Barracouta
0..4)
(3.4
(.1 (..6))
(=0
4.9
Beaver
0..6(+
(0.4(#
87.5(+0.5)
3.6(
2.2 (=3
Bicycle
(3.5() 0.55
(0.4()
32
2.3
2.3
Bulbul
9..6(+
93.0(#0.))
90.0(#0.6)
(=1
2.0
Coral fungus
(..6
(1.5(
(8.5)
29(=0)
2.1
2.0
English springer
(..9
(3.5(
90.4(
3.6(#))
2.2
222
Garfish
(..6))
(..4))
(6.5)
5.4
3.9
3.9
Golden retriever
(..6))
(1.7(
88.8(+0.5)
5.0(0..1)
3,2
3,1
Gyromitra
(6.0
8..8(=
84.8(#10))
3.3(0)
2.5
24(+0)
Jay
(3.77)
1.05
(3.7
6.8(0.1)
(.8
4,6(#0)
Junco, snowbird
(3.0 (..5
90.6(#
277(0))
1.8
1.7
Kuvasz
(..6))
(2.5
90.1(#
3,5(=))
2.1
2.1
Labrador retriever
(3.5(+
88.5
87.0(+0.6)
5.4(0..1)
3.4
3.4
Lighter, Light
(..26)
83.0(#0.))
84.2
5.((0.1)
(,5
3,5(0))
Lycaenid butterfly
94.0(+0.5)
(3.44)
(0.5(
3.9(0..))
2.7
2.7(00)
Magpie
(..6
92.4(20.))
(0.1(
3.0(0)
1.9(0))
19((0)
Marmot
94.0(+0.6)
90.9(#
88.3
4.9(0..)
3.1
3.0
Muzzle
(3.6
86.1
(5.9
43(00)
288
288(0))
Papillon
93.6(+0.))
(0.4
87.5
3.8(0.1)
2.4(=0)
2.(00)
Rock beauty
(0.9 (..5)
84.6(#0.5)
5.1
3.7
3.7
Siberian husky
(4.1
84.3
(9.1
4.0(#))
2.3
2.(=0)
Stinkhom
(..6))
91.5(#0.5)
(9.3
4.5(
(,0
3.0
Tennis ball
93.5(+0.))
87.3)
85.8(+0.5)
3,5(0)
2.
Tinca tinca
(3.2
92.6(+0.5)
(0.5
3.8(0))
2.6
2.6
Torch
(405)
(2.44
(..44)
6.2
(
4.3
Unicycle
(..66)
(..3
(8.2
4.8(=0)
3.3
3.2
Water ouzel
(2.4(=0.4) (+0.5
89.94)
3.0(#0)
2.0
1.9
White wolf
93.9
(9.2
(+.5)
3+4
2.1
2.1
45

---

J.2.1 Results for SPI-Subset with different hyperparameter values
In this section, we present results for the SPI- SPI-Subset procedure across different values of k. the number of subsets selected construct the synthetic calibration We compare the performance of SPI-Subset with whole-which uses all 100 synthetic classes-and and the standard conformal prediction, OnlyReal.
Figure S14 presents the performance of all methods for the "American robin" class as a function of k different values of the level a. Notably, for all values of k and a, SPI-Subset achieves coverage within the theoretical bounds.
The two methods, SPI-Subset and SPI-Whole coincide when k 100, both use the full synthetic calibration set. However, for smaller values of k, the two methods exhibit significant differences. While SPI-Whole tends to produce more conservative prediction sets, the SPI-Subset procedure more tightly achieves the target coverage level across different settings.
For the case 0.02 and ho = 5, both the theoretical lower and upper bounds on coverage are equal unity, implying that SPI-Subset yields trivial prediction sets that include all possible classes. This outcome is known a priori and can be avoided by selecting different hyperparameter for window construction.
1.00
Coverage
10-
Size
0.06
100
0.94
5
20
30
50
100
5
10
20
30
50
100
(a)
0.02
1.000
Coverago
0.975
0.950
Size
0.925
8
5
10
20
30
50
100
5
20
30
50
100
k
(b) a
0.05
OnlyReal
SPI-Whole
Coverage
SPI-Subse!
Lower-bound
Upper-bound
Lower-bound [s]
20
50
Upper-bound [s]
k
(c) 0.1
Figure $14: Results for the ImageNet data: Coverage rate for SPI--ool,,, and SPI-Subset the American robin class function of the number of subsets k. for levels a = 0.02 (a), = 0.05 (b), and a 0.1 (c).
Figure S15 shows the results for the "Beaver" class. For 0.02 and 0.05, observe the same trend as in Figure(S14 SPI-Whole yields relatively conservative coverage, while SPI-Subset with k 100 achieves coverage closer to the nominal level 1 a.
For a 0.1, SPI-Whole-whieh uses the full synthetic set-already achieves coverage close to the target level 1 a, suggesting that the empirical (1 a:)th quantile of the synthetic data closely
46

---

matches that of the real data. Consequently, in this setting, using only a subset of the synthetic data results in an increase the variance of the coverage rate.
1.000
Coverage
0.050
Size
0.925
3
10
20
50
100
6
10
20
30
50
100
(a)
0.02
1.00
30
Coverage
10
0.95
Size
8.90
000
5
10
20
30
50
100
5
10
20
30
50
160
k
(b) a = 0.05
OnlyReal
SPI-Whole SPI-Subset
Coverage
0.9
Size
0.8
Lower-bound
0.7
Upper-bound
Lower-bound [s]
20
50
20
100
Upper-bound [s]
k
(c)
0.1
Figure S15: Results for the ImageNet data: Coverage rate for OnlyReal, SPI-Whole, and SPI-Subset on the Beaver class as a function of the number of subsets for levels 0.02 (a), a = 0.05 (b), and a = 0.1 (c).
J.3 Additional ImageNet experiments with the HPS score function
Here, we replicate the ImageNet experiments using the HPS score function cettiio--commllmmeennini the APS- based results in the manuscript-acreee across three synthetic datasets: Stable-Diffusion (Figure FLUX (Figure S17). and auxiliary data [188 Each figure corresponds to the same experimental setup used with the APS score function. Further details on the score function provided in Appendix C.1
The score function provides a heuristic measure of the model's uncertainty, and APS and HPS capture this uncertainty in different ways. Recall that we utilize the synthetic data in the score space, where quality of SPI prediction set depends on how well the distribution of the synthetic score approximates the real score distribution. Consequently, different score functions may induce different aligmments between the real and synthetic scores. Therefore, while we generally expect similar trends. the exact per-class coverage is not necessarily preserved across score functions. Indeed, in Figure S16 the per-class results largely follow those of the APS-based experiment (Figure (3). In contrast, Figure SI7 for the class Magpie, OnlySynth under-covers substantially, whereas under the APS score in Figure S12b it achieves coverage closer to the target level a.
Given these class-specific differences, the overall trends remain similar to the APS- results. The OnlyReal method yields overly conservative and uninformative prediction sets due to the limited sample size, whereas OnlySynth lacks coverage guarantees, leading to under-coverage for some classes. In contrast, SPI achieves coverage within the theoretical bounds while producing smaller, more informative prediction sets.
47

---

1.00
30
30-
OnlyReal OnlySynth SPI
Coverage
0.75
a Lower-bound Upper-bound
0.25
Morginal
Siberian
Beaver rabic
American
Lightr
Baracoata
Marginal
Siborian bunky
Beatee robin
American iight
Lightor,
Barvwoota
husky
Light
Class
Class
Figure S16: Results for the ImageNet data using the HPS score function: Same experimental setup as in Figure 3 but using the HPS score instead of APS. Coverage rates of OnlyReal, OnlySynth, and SPI at target level 11-a & = 0.95, averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box in each group) and label-conditional coverage settings. Label-conditional results are shown for selected classes; see Table SII for results across all classes.
Coverage
OnlyReal OnlySynth
0.75
Size
20
0.50
SPI
0.25
Lower Lower-bound
Marginal
Sibersif Ma fitriver
Labra dol (Piin
Amscican
Magple
Bicyclo
Marginal
Siberdifren voee
Labradel /lm
Amurican
Magple
Birycle
Buddy
Bansly
Upper-bound
Class
Class
Figure S17: Results for the ImageNet data using FLUX- -eD-eeeretted synthetic images and the HPS score function: Same experimental setup as in Figure ST2b using the HPS score instead of APS. Coverage rates of Onl OnlyReal, OnlySynth, and SPI target level 1 a 0.95, averaged over 100 trials. Left: Average coverage. Right: Average prediction size, both under marginal (leftmost box in each group) and label-conditional coverage settings. Label-conditional results shown for selected classes; see Table $12 for results across all classes.
presents the performance of the subset-based variant of our approach, SPI-Subset, compared to SPI-Whole, which uses the entire synthetic dataset. We observe a similar trend to the APS-based results in Figure both SPI variants control the coverage within the theoretical bounds. For some classes, SPI-Subset achieves coverage closer to the nominal level while producing smaller prediction sets. In the marginal setting, however, using only subset of the synthetic data does not improve the performance.
Finally, Figure $18
Coverage
1.0
30
30-
15
OnlyReal
0.9
Size
SPI-Whole SPI-Subset
10-
1 1-a
Lower-bound
Marginal
Admara
Lightet:
Light
Beaver
Torch rabin
American
Marginal
Admira labi
Lighter
Beaver
American Torch rohin
Upper-bound
Class
Class
Figure S18: Results for the ImageNet data using the HPS score function: Same experimental setup as in Figure 4 but using the HPS score instead of APS. Coverage rates of OnlyReal, SPI-Whole, and SPI-Subset at target level a 0.98, averaged over 100 trials. Left: Average coverage. Right: Average prediction set size, both under marginal (leftmost box in each group) and label-conditions coverage settings. Label-conditional results are shown for selected classes; see Table S13 for results across all classes.
48

---

Table S11: Per-class conditional coverage (in %) and prediction set size for each method, computed
over 100 trials using the HPS score function. Standard errors are shown in parentheses. The target
coverage level is 1 a 0.95. The theoretical coverage guarantees for SPI are the range 193.7, 100). Other details are in Figure S16
Coverage (%)
Size
Class
Only Real
Only Synth
Only Real
Only Synth
SPI
Admiral
100 (# 0)
0.7 (# 0)
(+
30 (+ 0)
3.2 (# 0)
5.5
American robin
100 (+ 0)
87.4 0.1.)
(4.04)
30 (=
2+1
(+
Barracouta
100 (# 0)
98.4 (+
0..6
30 (=
(*
6.2
Beaver
100 (+ 0)
(9.2
(3.6)
(=
2=2
3.5
Bicycle
100 (= 0)
(1.1(
(5.3
(0
2.1
3.1
Bulbul
100 (+
(8))
99.0
(0
2,2(#0)
3.7
Coral fungus
100 0)
(8)0
(...()
(0
2.6(20)
3.6
English springer
100 0)
77.0(#0.1)
(3.6
(=
2.3(0))
3.
Garfish
100 (= 0)
(3.7 (+.2)
(0
3.3
5.2
Golden retriever
100 0)
(0.4))0.)
(*.2))
(0
3,0(#0)
(.1
Gyromitra
100 (+ 0)
100
(#
30
2.9
388
Jay
100 0)
(2.2(+
(3.1
(0
3,0(#0)
6.0
Junco, snowbird
100 0)
(4.0(#0.1)
(5.2)
30
2.2
(.)
Kuvasz
100 0)
(1.9
93.1
30
2,2(00)
(.
Labrador retriever
100 0)
(2.2
94.5
30
2.9
5.3
Lighter, Light
100 0)
94.8(#0.1)
96.5
30
3,2
4.9
Lycaenid butterfly
100
(2.)
94.6(+0.5)
(0
2.1
(...)
Magpie
100 (+ 0)
(4.3))
93.3
(0
2.0(2))
3.
Marmot
100 (+ 0)
(2.1)
(5.5
30
2.6(0))
(..
Muzzle
100 (+
(z.1
(H.7
(0
2.3(0))
(.1
Papillon
100 0=
(2.2
(3.9
(0
2,2(0))
(..
Rock beauty
100 (+
(1.8
(4.3)
(0
2.7
(.0
Siberian husky
100 0)
(2.2))
94.4(+0.5)
30
2.1
(.
Stinkhomn
100 (
(0.1)
96.5
30
3.0(20)
4.5
Tennis ball
100 0)
(2.2(
(5.4
(0
2.(+0)
3+4
Tinca tinca
100 (=)
(
(..1
(0
2.6
(.9
Torch
100 (+
(..6
97.0(
30
(4
(.1
Unicycle
100 ()
8..6
(4.5)
30
2.4
4.1
Water ouzel
100 (=
91.3
(5.2
30
(+
3.3
White wolf
100 (+ 0) (0.2
03.7
30
(+
3.7 (.)
49

---

Table S12: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials using FLUX-generated synthetic data and the HPS score function. The target coverage level is a 0.95. The theoretical coverage guarantees for SPI Standard errors are shown in parentheses. Other experimental details follow Figure S17
the range [93.7, 100].
Coverage (%)
Size
Class
Only Real
Only Synth
Only Real
Only Synth
SPI
Admiral
100 (# 0)
0.1 (# 0)
93.5 (+)7)
30 (#
3.5 (# 0)
(.
American robin
100 (+ 0)
71.7 0.1)
93.7 (..6)
30 (=)
1.4 (+
(.
Barracouta
100 (# 0)
(9
(*
30 (=
3.7
(.
Beaver
100 (+ 0)
(
(3.6(
(=
1.9
4.2
Bicycle
100 ( 0)
(+
()
30 (0
3.0
(.0
Bulbul
100 (+ 0)
(1.3(#
95.2(#0.3)
(0
2.3(=0)
4.5
Coral fungus
100 0)
(8)6
98.7
(0
2.3(20)
(.0
English springer
100 0)
(2.1
(3.6
(=
2.4(0))
(.
Garfish
100 (= 0)
(3.2
93.7
(0
3.1
(.
Golden retriever
100 0)
(0.9
(4.2
(0
2=1
(.0
Gyromitra
100 (+ 0)
(2.5
(9.3
30
2.8(00)
Jay
100 0)
(8.4
93.0
(0
3,6(=0)
(..
Junco, snowbird
100 0)
(7.1
(7.)
30
1.7
(.
Kuvasz
100 0)
98))
(8..(
30
1.9
(.
Labrador retriever
100 0)
(2.2)
93.5
30
2.6
(.
Lighter, Light
100 0)
(0.2
94.2
30
2.6(0))
5.6
Lycaenid butterfly
100 0)
(0.2
(4.5(
(0
3.2
4.6
Magpie
100 (+ 0)
22.2
93.77
(0
1.5
(.
Marmot
100 (+ 0)
(4.1)
96.2()
30
2.6
(.1
Muzzle
100 (+
(8.1))
(4.3
(0
2.6(2))
4.8
Papillon
100 (= 0)
(9)9
(9.9
(0
1.6(#))
4.0
Rock beauty
100 (+ 0)
(..2)
(..3)
(0
3.4
6.0
Siberian husky
100 (+
(6.0
94.4(+0.5)
(0
2.1
(.9
Stinkhomn
100 (
(2.1)
95.4
30
3.6(0))
5.4
Tennis ball
100 (=
93.3(+0.6)
(0
1.5(0)
(.0
Tinca tinca
100 ()
(H.1( (..3)
(0
2.3
4.4
Torch
100 ()
(4.2)
94.6(40.5)
30
6.7
Unicycle
100 ()
(9
99.4
30
3.4
5.1
Water ouzel
100 0) 98.5
(8.7
(0
1.6
(.
White wolf
100 (+
(..5)
(3.7
(0
2+4
4.7 (+1
50

---

Table S13: Per-class conditional coverage (in %) and prediction set size for each method, computed over 100 trials using the HPS score function. Standard errors are shown in parentheses. The target
coverage level is 1 a 0.98. The theoretical coverage guarantees for both SPI-Whole and SPI-Subset are in the range [93.7, 100). Other details are as in Figure S18
Coverage (%)
Size
Class
Only Real
SPI Whole
Only Real
SPI Whole
Subset
Subset
Admiral
100 (# 0)
93.5
95.3 0.44)
30 ()
0.10)
5.6 0)
American robin
100 (+ 0)
0..1)
(+.22)
(=
3.9 0.1))
(.
Barracouta
100 (+ 0)
(5.4)
(5.4))
30 (=
6.1
(.1
Beaver
100 (+ 0)
(z.1
(6.3))
(=
3.1()..1)
(.5
Bicycle
100 (= 0)
()
(7.7
(0
3.2
(,2
Bulbul
100 (+ 0)
(+)
(7.2(
(00)
(..1)
3.7
Coral fungus
100 0)
(8.5
(+.2)
30
3.7
English springer
100 0)
98.0(#0.11
(7.2
(0
3.7(=
(.7
Garfish
100 (1 0)
95.3
(..44
(0
5.1(
(.1
Golden retriever
100 (+ 0)
(6.2(
(5.3)
(0
5.2(
(.1
Gyromitra
100 (+ 0)
(9.3
(9.7
30
388(0))
(*7
Jay
100 0)
(..66
(4.44
(0
6.8
(.9
Junco, snowbird
100 (+ 0)
08.)
(6.2
30
3.5(#..))
(.2
Kuvasz
100 0)
08.0(#
96.8
(0
4.0(#..1)
(.
Labrador retriever
100 0)
95.9
95.3(+
30
5.3(0..))
(..
Lighter, Light
100 0)
(4.5
94.5
30
4.7(0..1)
(.7
Lycaenid butterfly
100 0)
(9)5
(8.9(+
(0
4.5(0))
(.4
Magpie
100 (+ 0)
(..7
(6.2
(0
3.5
(.
Marmot
100 (+ 0)
08.1(#
(+.2)
30
4.1()
0.1.)
Muzzle
100 (+ 0)
(H.5
(3.9
(0
4.0(
0.0
Papillon
100 (+ 0)
(6.5
95.6(#
(0
(..
(.9
Rock beauty
100 (+ 0)
(z.5)
(..44
(0
5.1
(.
Siberian husky
100 (+
(6.1
96.1)
(0
4.0(
0.0
Stinkhomn
100 0)
(0.1)
96.6
30
4.1(
O.
Tennis ball
100 0)
(5.3
(3.9
(0
3.((0)
(z)
Tinca tinca
100 ())
(9)0
(..2))
(0
(.9
3.8 (.
Torch
100 (2)
(6.2
(6.5
30
5.4
5.4
Unicycle
100 (
(..6
(..74
30
4.1
(.1
Water ouzel
100 (= 0)
97.1
(5.3)
3.6 30
(.1
White wolf
100 (+ 0)
96.6
(5.3)
(0
3.1
3.8 0.10)
51

---

K Additional MEPS experiments
In this section, we present additional results for the MEPS regression experiments, complementing those reported in Section 4.2
Figure ST9 reports the coverage rates and prediction interval lengths for all age groups, evaluated 0.02, and 0.05. As in the main paper, we observe that SPI achieves coverage rates that remain within the theoretical bounds, with lower variance compared OnlyReal.
Coverage
0.98
Length
0.96
0:20
40.00
Age range
Age range
0.02
1.00
OnlyReal OnlySynth
Coverago
0.90
Length
6-
SPI
0.96
0.94
Lower-bound
Upper-bound
0.92
0.20
20-40
40:60
60
0:20
20-10
40-80
60+
Age range
Age range
(b) = 0.05
Figure S19: MEPS dataset results: coverage and interval length for each age group, obtained by OnlyReal, OnlySynth, and SPI, at target coverage levels Experiments are repeated over 100 trials. OnlyReal produces trivial (infinite) prediction intervals; thus, its interval length is omitted.
0.98 (a), and 0.95 (b).
K.1 The effect the real calibration set size
We replicate the experiments from Appendix J.I.I on the MEPS dataset, evaluating the performance of different methods as function of the real calibration set size, m.
Figure S20 and Figure S2I present the performance of all methods for age groups 0-20 and 20-40, respectively, across different a levels and values of m. The standard conformal method, OnlyReal, conservatively controls the coverage at the target level o; however, it results in larger and noisier prediction intervals due the small sample size.
Similar to the trends observed in the main manuscript, OnlySynth achieves coverage close to the nominal 1 level, indicating that the synthetic data align well with the real one. However, this approach does not have coverage guarantees.
In contrast, the proposed method, SPI, achieves coverage within the theoretical bounds, closely matching the target informative prediction intervals.
level while reducing coverage variance and producing smaller, more
For 0.02 with small calibration sizes (m one under this window construction. This implies that the proposed method produces trivial prediction intervals. This behavior is known priori and was also observed in the ImageNet experiment, where we used the same window construction parameters. Nevertheless, it can be avoided by employing different window construction.
10), the theoretical coverage bounds equal to
52

---

Coverage
0.975
Length
0.950
m
0.02
Coverage
0.95
Length
0.90
10
15
25
50
10
15
2b
m
(b)
0.05
1.0
OnlyReal
OnlySynth
Coverage
Length
8
SPI
0.7
Lower-boute Upper-bound
0.6
15
25
50
10 15 25
(c) a 0.1
Figure $20: MEPS dataset results: coverage and interval length for the 0-20 age group, obtained by OnlyReal, OnlySynth, and SPI, at target coverage levels a = 0.98 (a), 0.95 (b), and 0.9
(c). Experiments are repeated over 100 trials. For a 0.02 and 0.05, methods that produce trivial (infinite) prediction intervals are omitted from the interval length panel.
53

---

Coverage
0.98
Length
0.96
0.52
15
m
m
0.02
Coverage
Length
0.90
0.85
10
15
25
50
10
15
2b
m
(b)
0.05
1.0
OnlyReal
Coverage
0.9
Length
000
OnlySynth
SPI
6
Lower-boute
Upper-bound
15
25
50
10
15
25
(c) 0.1
Figure $21: MEPS dataset results: coverage and interval length for the 20-40 age group, obtained by OnlyReal, OnlySynth, and SPI, at target coverage levels a = 0.98 (a), 0.95 (b), and 0.9
(c). Experiments are repeated over 100 trials. For 0.02 and 0.05, methods that produce trivial (infinite) prediction intervals are omitted from the interval length panel.
54

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are supported by the theoretical results Section 3.3 and the experimental findings in Section and Appendices K
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of method in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
55

---

The problem setup and all assumptions are detailed in Section and complete proofs for the theoretical results in Section 3.3 are provided in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are provided in Section and Appendix H in- cluding dataset information. Software for reproducing the experiments is available at https://ithub.com/Meiibbaaa Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. For closed-souree models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path reproducing or verifying the results.
5. Open access to data and code
56

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The software package implementing our method and reproducing the exper- iments is available at httts://github.ccm soom/Mshhba//pii including the code used for image generation with Stable Diffusion via Hugging Face. The ImageNet and MEPS datasets used are publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.c)
guw.ees//ooeeuumssss/tt/. soteeesuubmsiiioooooocooc.... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https / httpsssccppbblicc/cc healllic/guieeee guw Saaee//CooSSubiiseeee wwwabbmmsssiooooooiii.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting is described in Section with additional technical details provided in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All plots are presented as boxplots to reflect variability across runs, and tables report means with standard errors; see Section 4 and Appendices J and K for the corresponding figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
57

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix H.I we provide detailed information on the computational resources used in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttga///oorrpps.c./p/tttt/ttttt
Answer: [Yes]
Justification: The research conducted in this paper follows the NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential social impact of our work in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
58

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of any pretrained models, image generators, or datasets that pose high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and sources used in this paper are properly cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
59

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. However, provide a well- documented software package.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
60

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology in this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http::://neurps.... http:///aaarreccc//////eeerre/// cc//oonferencee/22255/LI. for what should or should be described.
61


Reviews:
[{"review_id": "hTjUwB4ZQj", "reviewer": "Reviewer_pG2H", "summary": "This paper introduces Synthetic-powered Predictive Inference (SPPI), a new framework that enhances the sample efficiency of conformal prediction by incorporating synthetic data generated from large generative models or alternative real datasets. The main innovation is a score transporter: an empirical quantile mapping that aligns nonconformity scores between real and synthetic calibration data. SPPI provides finite-sample, distribution-free coverage guarantees regardless of the similarity between real and synthetic data, and can substantially tighten prediction sets when synthetic and real data distributions are well aligned. The approach is demonstrated on both multi-class classification (ImageNet, using diffusion models) and tabular regression (MEPS panel data), showing improved efficiency over standard conformal prediction in data-scarce settings.", "questions": "1. Can the authors suggest practical ways to diagnose or quantify when the synthetic and real score distributions are too dissimilar for SPPI to be effective? Are there warning signs or automated checks practitioners could use?\n\n2. How does the method perform with synthetic data from weaker generators or less relevant domains? Can the authors comment on robustness in such scenarios?\n\n3. : For large-scale settings, how does the computational cost of SPPI compare to standard conformal prediction or other recent efficient conformal methods?", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe method tackles a core limitation of conformal prediction—its inefficiency in low-data regimes—by leveraging abundant synthetic data in a theoretically principled manner. The authors provide finite-sample coverage guarantees,\n\nWeaknesses:\n1. The paper could further discuss or systematically analyze the impact of different types of synthetic data and score functions on efficiency and coverage—especially when using less realistic or lower-quality generators.\n\n2. SPPI method does not guarantee that the prediction set (or interval) will always be smaller than (or even no larger than) the one constructed using only real data. \n\n3. More practical guidance on choosing hyperparameters (e.g., β for windows, subset size for selection), diagnostics for misalignment, and post-hoc adjustment strategies would make the framework easier to deploy for non-expert users.", "was_revised": true, "final_justification": "Overall, this is a solid contribution (score: 4). However, the paper does not provide efficiency guarantees; Therefore, the procedure may produce wider intervals than standard methods, constraining practical utility.", "rebuttal": "Thank you for your thoughtful review. It appears that your major concern is with the robustness of our method to the quality of the synthetic data. We start by addressing this point and then proceed by responding to the rest of your comments.\n>1. How does the method perform with synthetic data from weaker generators or less relevant domains? Can the authors comment on robustness in such scenarios?\n\nFirst, we would like to point out that our method relies on the synthetic data only through their scores---i.e., the prediction error of the fitted model---not through the raw inputs or labels. \nWhen the synthetic data are of low quality (i.e., the synthetic score distribution differs significantly from the real scores), **SPPI provably remains robust and guaranteed to yield prediction sets with coverage rate that falls within the theoretical guardrail bounds established in Theorem 3.5. These bounds hold regardless of the quality of the synthetic data.**\nMoreover, the lower guardrail bound is tunable: for given sample sizes $m,N$, and target level $\\alpha$, the hyperparameter $\\beta$ can be set to meet the user-specified guardrail coverage bound (as detailed in Algorithm 4 in the appendix).\n\nAt the same time, if the synthetic data are of high quality (i.e., the synthetic score distribution closely resembles the real scores), SPPI benefits from using the synthetic data. The empirical coverage is tightly regulated around the target level $1-\\alpha$, with low variability. This aligns with the results in Theorem 3.3, as the total variation distance is small in this case.\n\n**Importantly, our method also allows for improving the alignment between synthetic and real scores using the real scores themselves, and without requiring any data-splitting. For instance, our K-nearest subset approach selects subsets of synthetic scores that are more aligned with the real ones to improve statistical efficiency.** Crucially, such refinements do not compromise the coverage guarantees; indeed, we demonstrate in the K-nearest subset experiments (Figure 2 and Appendix I.2) that using this approach can improve performance.\n>2. Discuss or systematically analyze the impact of different types of synthetic data and score functions on efficiency and coverage—especially when using less realistic or lower-quality generators.\n\nWe thank the reviewer for the suggestion. Note that the experiments in the manuscript already include four types of synthetic data: for image classification, we consider images generated by Stable Diffusion, Flux, and images from different classes; for the MEPS dataset, we use past panel data.\n\nWe agree that further comparisons across different synthetic sources and score functions are valuable. To that end, we include in the table below new results for image classification using the HPS score function---complementing the APS-based results in the manuscript---evaluated across three different synthetic datasets. We observe a similar trend to the one observed in the manuscript.\n\nRegarding the effect of the quality of synthetic data on performance, we kindly refer the reviewer to our response to Reviewer aUpi, Item 1, where we present new experiments with synthetic data for which the total variation distance between the real and synthetic score distributions can be computed. These experiments demonstrate how varying levels of the quality of synthetic data affect the theoretical coverage bounds, and also the empirical coverage is always within the effective bounds defined as the tighter of Theorems 3.3 and 3.5, as our theory suggests. We will include all the new results in the revised version.\n\n**Table: Results for the ImageNet data using the HPS score function**  \nMarginal and per-class conditional coverage for three synthetic datasets. The target coverage level is 1 − α = 0.95. Other details are as in Figure 1 from the manuscript.\n||Stable Diffusion||FLUX||Auxiliary data||\n|-|-|-|-|-|-|-|\n||`OnlySynth`|`SPPI`|`OnlySynth`|`SPPI`|`SPPI-Whole`|`SPPI-Subset`|\n|**Marginal**|89.9 (± 0.1)|95.3 (±0.3)|93.1 (±0.1)|95.9 (±0.3)|94.9 (±0.4)|94.8 (±0.4)|\n|**Siberian husky**|32.4 (±0.2)|94.4 (±0.5)|36.0 (±0.2)|94.4 (±0.5)|96.1 (±0.3)|96.1 (±0.3)|\n|**Beaver**|79.7 (±0.2)|93.5 (±0.6)|35.8 (±0.2)|93.4 (±0.6)|97.4 (±0.1)|96.0 (±0.3)|\n|**American robin**|87.4 (±0.1)|94.4 (±0.4)|71.7 (±0.1)|93.7 (±0.6)|98.9 (±0.1)|96.2 (±0.2)|\n|**Lighter,Light**|94.8 (±0.1)|96.5 (±0.2)|30.2 (±0.2)|94.2 (±0.7)|94.5 (±0.5)|94.5 (±0.5)|\n|**Barracouta**|98.4 (±0.0)|98.6 (±0.1)|99.6 (±0.0)|99.6 (±0.0)|95.1 (±0.4)|95.6 (±0.4)|\n>3. SPPI method does not guarantee that the prediction set (or interval) will always be smaller than (or even no larger than) the one constructed using only real data.\n\nThere are scenarios in which the SPPI prediction set is always shorter than the conformal prediction set based solely on real data. For example, when the real data size is very small---specifically, $m < 1/\\alpha + 1$---the naive conformal prediction always yields a trivial prediction set, whereas SPPI can produce nontrivial, finite-width prediction sets.\n\nBut, more generally, we would like to note that while ``constructing a **valid** prediction set whose length (or more generally, measure) is almost surely shorter than that of the naive conformal prediction\" may sound appealing, it may be very hard or even infeasible target in the distribution-free setting---especially when leveraging synthetic data whose distribution is unknown. The same holds even for weaker targets, such as achieving a smaller expected width. Intuitively, the reason is that the distribution of the synthetic data can range from very good to very bad, and so the only way to make sure that we do not harm the coverage guarantee of standard conformal prediction (which is already very tight) would be to ignore synthetic data. Our SPPI method sidesteps this challenge by introducing the guardrails, as explained in the paper.\n \nNevertheless, as discussed in Item 1, when the synthetic scores are aligned with the real ones, SPPI yields shorter, more efficient prediction sets. When the synthetic data are of low quality, the method remains safe, with coverage bounded by the worst-case guarantees of Theorem 3.5. These two properties are validated through our extensive experiments: \nin nearly all of our experimental results, the proposed procedure outperforms naive conformal prediction in terms of prediction set width, while still achieving reasonable coverage rates.\n>4. More practical guidance on choosing hyperparameters, diagnostics for misalignment, and post-hoc adjustment strategies.\n\nWe thank the reviewer for this suggestion, and we will add the following discussion in the revised version.\n\nFor the hyperparameter $\\beta$, we recommend selecting its value based on the desired guardrail bounds, e.g., 90\\% coverage with the ideal target level $1-\\alpha=95\\%$. Specifically, Algorithm 4 provides a procedure to obtain such a $\\beta$ for given values of $N$, $m$, $\\alpha$, that meets the user-specified guardrail coverage bound.\n\nRegarding the subset size for selection, we suggest following our experiments and fixing $N$ to be reasonably large (e.g., at least 300–500) and then selecting the $K$ nearest synthetic subsets.\n\nAs for diagnostics and post-hoc adjustment strategies, we first recall that only the distribution of the scores matters for our method, not the raw inputs or labels. Therefore, diagnostics should be done in the score space. To assess similarity between the real and synthetic scores, one can apply any 1-dimensional distributional tests such as the Cramér–von Mises or Kolmogorov–Smirnov statistics, or visually inspect the histograms of the real and synthetic scores. However, since $m$ is often relatively small, these diagnostics will be noisy but still useful to detect major deviations.\nImportantly, such diagnostics do not compromise the validity of our coverage guarantees, as discussed below.\n>5. Suggest practical ways to diagnose or quantify when the synthetic and real score distributions are too dissimilar for SPPI to be effective. Are there warning signs or automated checks practitioners could use?\n \nOur main focus in this work is on the setting where the real data is limited and insufficient to provide useful inference---in such a setting, diagnostics based on the real data may be noisy. Importantly, our approach is supported with distribution-free guardrails: regardless of the quality of the synthetic data, the coverage rate always lies between some fixed coverage levels $1-\\alpha - \\varepsilon_1$ and $1-\\alpha + \\varepsilon_2$ (this is essentially the result of Theorem 3.5), where the lower coverage level can be pre-defined by the user. **Put simply, there is no need for any ``warning signs.''**\n\nNevertheless, one can apply the diagnostics described in the previous item to detect when the synthetic scores are far off from the real scores, indicating that the synthetic data is not useful for the inference task or could be modified to better align with the real data. In fact, the latter is precisely the approach we take in our K-nearest subsets method, which allows us to benefit from better-aligned synthetic data. As demonstrated in the paper, with this approach, the coverage rate becomes closer to $1-\\alpha$ rather than hitting the guardrails.\n>6. For large-scale settings, how does the computational cost of SPPI compare to standard conformal prediction or other recent efficient conformal methods?\n\nSPPI incurs almost no additional computational cost compared to standard conformal prediction at inference time. To see this, note that conformal prediction requires finding a sample quantile of the scores, while SPPI performs essentially the same computation three times. Specifically, the simplified formula for SPPI (Section 3.2) consists of three terms: $S_{(\\tilde{R}^-)}$ and $S_{(\\tilde{R}^+)}$---sample quantiles of the real scores---and $\\tilde{Q}_{1-\\alpha}'$, a sample quantile of the synthetic scores."}, {"review_id": "F0hrhScRsk", "reviewer": "Reviewer_aUpi", "summary": "This paper proposes a framework to leverage synthetic dataset for applying conformal prediction, in the case of scarce real data. To do so, it constructs a mapping called \"transporter\" using both real and synthetic calibration scores. Applying this mapping to the real test score allows to compare it to the synthetic quantile instead of the real quantile where the former is computed with much more samples than the latter.\n\nThey provide with a fast computation of their proposed conformal set with theoretical guarantee.\n\nIn the case of iid samples both for real and synthetic cases, upper and lower bounds for the marginal coverage are provided, which essentially depend on the total variation distance between the synthetic and real data distributions.\n\nExperiments are provided for the task of image classification using either diffusion models or unseen classes as synthetic dataset to illustrate their findings.", "questions": "- Could $\\beta$ be correlated to the difference between P and Q ? Since the transporter makes the real and synthetic scores \"similar\", I would assume that $\\beta$ could be used to calibrate that step similarly to results for weighted exchangeability, distributional shift. In particular, could deriving Lemma 3.1 in the case where P is not equal Q (with additional error terms that illustrate the shift) provide tighter coverage bounds ? \n- Do authors have an idea of how the bias variance (mentioned above) scale in $m,N$ even in simple cases and in particular whether there is some intuition in how to choose the scaling of $N$ compared to $m$ ?\n- The framework is extremely general and could include weighted exchangeability, distributional shift. For these specific cases, does the algorithm proposed obtain similar guarantees to existing ones ?\n\nI believe the proposed method is interesting but the guarantees do not allow to understand how to fully leverage it. Addressing the questions raised above could make me increase my score to accept the paper.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The problem tackled of data augmentation for conformal prediciton is of interest. The paper writing is clear and enjoyable to read.\n- The transporter proposed is sound and intuitive.\n- Theoretical guarantees are provided and describe the deviation to the targeted $1-\\alpha$ coverage.\n- Experimental results show the applicability of their method.\n\nWeaknesses:\n- There is no in-depth discussion about the error terms in the bounds of Theorem 3.3 and 3.5. In particular, it seems that the sum $\\varepsilon + 1/(N+1)$ is a bias-variance trade-off.\n- The lower bound in Theorem 3.3 seems conservative as taking large values of $\\beta$ compared to $\\alpha$ in the experiments still yields good coverage.\n- I am slightly dubious about the assertion Line 167 \"when P=Q [...] our procedure provides a tighter upper bound than split conformal prediction\". If one has P=Q then there is no problem anymore to solve in the first place.", "was_revised": true, "final_justification": "While the theoretical analysis could benefit from even tighter investigation, their proposed method is of interest as it requires little assumption, is flexible and authors have demonstrated strong numerical evidence of its applicability.", "rebuttal": "Thank you for your thoughtful review. We are happy that you found the paper clear and enjoyable to read. It appears that your major concern is with the tightness of our theoretical results. We start by addressing this point and then proceed by responding to the rest of your comments.\n\n>1. The lower bound in Theorem 3.3 seems conservative as taking large values of beta compared to alpha in the experiments still yields good coverage.\n\nWe would like to argue that the bounds provided in this work are already quite tight. **Our theoretical result is based on both Theorems 3.3 and 3.5, where the effective bounds are the tighter of the two in each case.**\n\nWe first mention that the bounds in Theorems 3.3 and 3.5 are **distribution-free uniform bounds**---specifically, the lower bound in Theorem 3.3 essentially states the following:\n\n$\\inf_{\\text{all distributions $P$ and $Q$}} \\left[\\mathbb{P}(Y_{m+1} \\in \\widehat{C}(X_{m+1})) - (1-\\alpha-\\beta+\\varepsilon_{P,Q}^{m+1})\\right] \\geq 0.$\n\nThis bound was constructed to control this infimum, and thus the ``conservativeness\" should be discussed in terms of how close the above infimum is to zero. For specific distributions $P$ and $Q$, it is possible that the situation is better and we achieve a higher coverage.\n\nIn practice, one would prefer a tight bound that holds for any unknown data distribution. Such a uniformly tight bound is a challenging goal and is likely infeasible (how can we be both distribution-free and distribution-adaptive?).\nHowever, our second result---Theorem 3.5---essentially addresses this concern. Recall that Theorem 3.5 provides ``guardrail\" bounds, which approach $1 - \\alpha$ as $\\beta$ increases (discussed in Section D.1 as well). Importantly, these bounds do not depend on the distributions of the real or synthetic scores. Further, the lower guardrail bound is tunable: for given sample sizes $m,N$, and target level $\\alpha$, the hyperparameter $\\beta$ can be set to meet the user-specified guardrail coverage bound (as detailed in Algorithm 4 in the appendix).\n\nIn fact, if $\\beta$ is large, then---depending on the quality of the synthetic scores---the bound in Theorem 3.5 may dominate that of Theorem 3.3 and provide a tighter guarantee.\n\nTo illustrate this point, we ran new experiments with synthetic data where the total variation (TV) distance between the real and synthetic score distributions can be computed exactly. The table below presents the coverage rates for all methods, along with the theoretical bounds from Theorems 3.3 and 3.5, and the effective bounds in each case. We will include the new results, along with this discussion, in the revised version.\n\n**Table: Results on synthetic data.**  \nCoverage rates across different TV distances between the real and synthetic score distributions, along with bounds from Theorems 3.3, and 3.5, and the effective bounds (minimum/maximum of the two). Standard errors are shown in parentheses. Results are shown for $1-\\alpha=0.95$ and $\\beta=0.05$.\n||Coverage|||Theoretical|bounds||\n|-|-|-|-|-|-|-|\n|**TV distance**|`OnlyReal`|`OnlySynth`|`SPPI`|`THM 3.3`|`THM 3.5`|`Effective bounds`|\n|0|100 (±0)|95.2 (±0.1)|95.3 (±0.1)|(0.9, 1.0)|(0.812, 1.0)|(0.9, 1.0)|\n|0.04|100 (±0)|94.1 (±0.1)|94.2 (±0.1)|(0.86, 1.0)|(0.812, 1.0)|(0.86, 1.0)|\n|0.08|100 (±0)|92.8 (±0.1)|93.0 (±0.1)|(0.82, 1.0)|(0.812, 1.0)|(0.82, 1.0)|\n|0.159|100 (±0)|89.5 (±0.2)|90.2 (±0.2)|(0.741, 1.0)|(0.812, 1.0)|(0.812, 1.0)|\n|0.236|100 (±0)|85.3 (±0.2)|87.1 (±0.3)|(0.664, 1.0)|(0.812, 1.0)|(0.812, 1.0)|\n|0.311|100 (±0)|80.2 (±0.2)|84.2 (±0.5)|(0.589, 1.0)|(0.812, 1.0)|(0.812, 1.0)|\n|0.383|100 (±0)|74.2 (±0.3)|82.2 (±0.7)|(0.517, 1.0)|(0.812, 1.0)|(0.812, 1.0)|\n\n>2. I am slightly dubious about the assertion Line 167 \"when P=Q [...] our procedure provides a tighter upper bound than split conformal prediction\". If one has P=Q then there is no problem anymore to solve in the first place.\n\nWe are considering the case where **the quality of the synthetic data is unknown in advance**, as is often the case in practice. In particular, we do not know beforehand whether $Q$ equals $P$ or not.\n\nHowever, we can still leverage synthetic data to improve inference without such prior knowledge, and that is precisely the goal of our work. We address this problem by proposing a procedure with guardrails---with the following properties (which hold despite us not knowing whether $P \\approx Q$ or not!):\n\n- When $P \\approx Q$, SPPI behaves like conformal prediction with a large sample size, thereby improving upon the naive conformal prediction based solely on limited real data.\n\n- When the $Q$ doesn't approximate $P$ well, SPPI still provides meaningful inference to some extent, supported by the distribution-free guardrail bounds established in Theorem 3.5.\n\nThese properties are also validated empirically: our experiments show that SPPI yields more efficient prediction sets when $P \\approx Q$, and continues to provide valid inference when this approximation is poor.\n\n>3. There is no in-depth discussion about the error terms in the bounds of Theorem 3.3 and 3.5. In particular, it seems that the sum $\\varepsilon+1/(N+1)$ is a bias-variance trade-off.\n\nThank you for pointing this out. We agree with the interpretation in terms of the bias-variance tradeoff.\nSpecifically, the fundamental goal of this work is to develop a method that leverages synthetic data—which may follow a different distribution from the real data and thus introduce some bias—to improve the sample efficiency of the inference.\nOne important point to note is that what is being traded for the bias is not only the variance. Increasing the sample size in conformal prediction has two effects:\n\n- It reduces the variability of the prediction set: a larger sample size leads to smaller variance in the score bound $S_{(\\lceil(1-\\alpha)(n+1)\\rceil)}$, which is a sample quantile of the calibration scores.\n\n- It reduces the conservativeness of the prediction set: Recall that in split conformal prediction, a mass of $1/(n+1)$ is assigned to $+\\infty$ to account for the unobserved test score. This is the source of the upper bound $1 - \\alpha + 1/(n+1)$.\n\n**The proposed SPPI method allows for some bias from the synthetic data in order to gain advantages in both aspects.**\n\n>4. Do authors have an idea of how the bias variance (mentioned above) scale in m,N even in simple cases and in particular whether there is some intuition in how to choose the scaling of N compared to m?\n\nWe would say that ``a larger $N$ is always better.” Specifically, regarding the `bias' term $\\varepsilon_{P,Q}^{m+1}$, it does not depend on the synthetic sample size $N$, whereas the term $1/(N+1)$ decreases as $N$ grows. Thus, at least in terms of $N$, there is no tradeoff---a larger $N$ represents a strict improvement.\n\nMoreover, the sample size of the real data, $m$, affects the granularity of the bounds in Theorem 3.5. These bounds can be interpreted as $1-\\alpha -\\epsilon_1$ and $1-\\alpha+\\epsilon_2$, where a larger $m$ allows for tighter lower worst-case bound (for an appropriate choice of $\\beta$), thereby reducing the worst-case bias.\n\n>5. The framework is extremely general and could include weighted exchangeability, distributional shift. For these specific cases, does the algorithm proposed obtain similar guarantees to existing ones?\n\nThank you for this interesting question. Extending the method to the covariate shift setting is in fact not straightforward. Specifically, the SPPI method relies on a score transporter that maps each order statistic of the real scores to the space of synthetic scores. A key property used in the theoretical analysis is that the rank of the test score is independent of the (ordered) values of the scores. However, this independence no longer holds under covariate shift. We believe this presents an interesting direction for further investigation.\n\n>6. Could beta be correlated to the difference between P and Q ? Since the transporter makes the real and synthetic scores ``similar\", I would assume that could be used to calibrate that step similarly to results for weighted exchangeability, distributional shift. In particular, could deriving Lemma 3.1 in the case where P is not equal Q (with additional error terms that illustrate the shift) provide tighter coverage bounds?\n\nWe appreciate this insightful question. If we can somehow construct a window satisfying Lemma 3.1 in the case of $P \\neq Q$ (through some $(P,Q)$-dependent $\\beta$), then yes---the SPPI procedure will achieve the bounds in Theorem 3.3, but without the $\\varepsilon_{P,Q}^{m+1}$ error terms.\n\nHowever, it seems that such a construction of the window could require knowledge of the exact distributions $P$ and $Q$, rather than merely a similarity measure between them. Moreover, our primary focus is on settings with a limited number of real data points, where accurately estimating $P$ or the similarity between $P$ and $Q$ a priori is not a reasonable target.\n\nNotably, the weighted conformal method that works under general distribution shift (``Conformal Prediction Beyond Exchangeability'' by Barber et. al, mentioned by the reviewer) requires the weights to be set a priori and relies on the TV distance between the weighted score distributions to quantify the drop in coverage due to the shift. Consequently, it is unclear how the latter could tighten Theorem 3.3, especially in light of our Theorem 3.5 that allows users to adapt the synthetic scores to better align with the real scores in a data-driven manner, without compromising the validity of our guardrail coverage guarantees and without requiring data splitting.\n\n## A closing remark\n\nThank you for the opportunity to clarify why we argue that our theoretical analysis is quite tight, and for your thoughtful feedback on our method. If any part remains unclear, we would be happy to elaborate further. We sincerely believe our response addresses your concerns and will be helpful as you reconsider your evaluation."}, {"review_id": "02rs6fpSUr", "reviewer": "Reviewer_Rc3Q", "summary": "This paper proposes SPPI, a framework for utilizing synthetic data in constructing conformal prediction sets. The authors propose to use a score transporter to bring the real scores to synthetic scores, and then use the large set of synthetic scores to calibrate the prediction set. Theoretical guarantees in terms of the distributional gap between real and synthetic data, and in terms of worst-case performance are provided. The proposed method is applied to three applications which show the advantages of the proposed methods in scenarios where high-quality data are scarce.", "questions": "1. In Theorem 3.3, the term $\\varepsilon_{P,Q}^{m+1}$ depends on the TV distance of order statistics, which is a bit mysterious. How might this term depend on the TV distance between P and Q? \n\n2. The slack term $\\beta$ is discussed in lines 167-172 that only takes effect in extreme scenarios. Is there a way to derive a bound for that extreme event in order to obtain a sharper analysis?\n\n3. How should one interpret the quantities in Theorem 3.5? In particular, what does $R_j^+$ and $R_j^-$ quantify, intuitively? \n\n4. In general, I think the theoretical ideas behind the method are not discussed adequately. It might be helpful to include some high-level discussion on why the method could improve the efficiency and how the coverage is affected theoretically. \n\n5. It is also unclear why one might improve statistical efficiency by matching the distribution of real and synthetic scores. Intuitively, any efforts to align the scores will be subject to the intrinsic $1/\\sqrt{m}$ uncertainty in the real scores. Is this uncertainty still there, while you address the \"bias\" due to insufficient data? \n\n6. In Section 3.4, the theoretical results are not very illuminating, since attaining the bounds in Theorem 3.5 really doesn't tell us anything about how the coverage is improved. Can you derive some theory regarding the coverage/efficiency that is related to the quality of learning the synthetic score function? \n\n7. Why do you choose $\\beta=0.4$ in the experiments? How do the results change with the choice of $\\beta$? How would you recommend people to choose the value of $\\beta$? \n\n8. In Section 4.1.2, it is intuitively unclear to me why selecting samples from different label classes can help the prediction for a specific label class. Could you explain? \n\n9. The choice of $m=15$ is extremely small for Section 4.2. How would the results change with the value of $m$? Can the practitioners have a sense of how large $N$ is compared with $m$ to ensure that one gets some improvement?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper studies an important problem\n2. The experiments cover rich applications \n3. The paper is clearly written and relatively easy to follow \n\nWeaknesses: (see Questions for detailed questions)\n\n1. The theoretical guarantees are a bit hard to interpret \n2. The intuitions of the proposed approaches are not adequately discussed \n3. The choice of hyperparameters is not clear in the experiments", "was_revised": true, "final_justification": "revisions addressed some of my concerns", "rebuttal": "Thank you for your thoughtful review. We're glad you found the paper easy to follow.  It appears that your major concerns are related to high-level theoretical aspects of our method and the data-driven adaptation of the synthetic scores. We start by addressing these two concerns and then proceed by responding to the rest of your comments.\n>Q4: It might be helpful to include some high-level discussion on why the method could improve the efficiency and how the coverage is affected theoretically.\n\nWe thank the reviewer for this suggestion. As discussed in the introduction, the key question is how to leverage synthetic data---which can be abundantly generated through generative AI, for instance---to improve the sample efficiency of predictive inference in settings where data is scarce.\n\nA naive approach of simply applying existing methods to the large synthetic dataset leverages the increased sample size, but coverage is not guaranteed due to the unreliability of the synthetic data. One might suggest evaluating the quality of the synthetic data in advance, but such evaluation is not feasible in the setting of interest, where real data is scarce.\n\nThe proposed method, SPPI, addresses this issue by providing distribution-free guardrail bounds, enabling the procedure to achieve both of the following properties:\n\n- When $P \\approx Q$, SPPI behaves like conformal prediction with a large sample size, improving efficiency and reducing coverage variability. This is essentially the result of Theorem 3.3.\n- When the $Q$ doesn't approximate $P$ well, SPPI provably remains robust and guaranteed to yield prediction sets with coverage rate that falls within the theoretical guardrail bounds established in Theorem 3.5.\n\n**Our theoretical result is based on both Theorems 3.3 and 3.5, where the effective bounds are the tighter of the two in each case**. Specifically, Theorem 3.3 provides coverage bounds that depend on the distance between the real and synthetic score distributions---the smaller this distance, the tighter the bound approaches the nominal $1-\\alpha$ level. On the other hand, Theorem 3.5 provides bounds that are determined solely by the sample sizes $N,m$, and levels $\\alpha,\\beta$. In fact, one can set $\\beta$ in advance to control the lower bound on the coverage, as detailed in Algorithm 4 of the Appendix.\n\nIn sum, **SPPI does not require prior knowledge of how well $Q$ approximates $P$, and yet it automatically adapts to the quality of synthetic data, ensuring valid coverage.**\n>Q6-A: In Section 3.4, the theoretical results are not very illuminating, since attaining the bounds in Theorem 3.5 really doesn't tell us anything about how the coverage is improved.\n\nAs noted above, Theorem 3.5 does not quantify the impact of synthetic data quality on coverage. Instead, it provides guardrail coverage guarantees that hold regardless of the synthetic score distribution.\n\nPerhaps the reviewer refers to Theorem 3.3, as that characterizes how the coverage bounds depend on the similarity between the synthetic and real score distributions. In Section 3.4, we discuss strategies to improve the quality of synthetic data. When such strategies reduce the total variation distance between the real and modified synthetic score distributions, Theorem 3.3 implies that the empirical coverage will better follow the nominal $1-\\alpha$ level.\n>Q6-B: Theory for the coverage/efficiency that is related to the quality of learning the synthetic score function\n\nWhen data splitting is used to improve the quality of the synthetic scores, we already provide a variant of Theorem 3.3 that is formally stated in Appendix Section E.\n\nFor a more challenging setup, where synthetic scores are modified without data splitting, the real and synthetic scores are no longer independent, and thus Theorem 3.3 does not apply. Yet, the bounds of Theorem 3.5 continue to hold! In practice, we expect better empirical performance when the synthetic scores are better aligned---as we observe in our K-nearest subset experiments (Figure 2 and Appendix I.2).\n\nLastly, we kindly refer the reviewer to a related discussion on theoretical analysis for efficiency (i.e., prediction set size) in our response to Reviewer pG2H, Item 3.\n>Q5: Unclear why one might improve statistical efficiency by matching the distribution of real and synthetic scores. Any efforts to align the scores will be subject to the intrinsic $1/\\sqrt{m}$ uncertainty in the real scores. Is this uncertainty still there, while you address the ``bias\" due to insufficient data?\n\nYes, the intrinsic uncertainty still remains. The goal of the score transformation is to shift the synthetic score distribution $Q$ appropriately so that it better resembles the real score distribution $P$, and the ``uncertainty\" in the real scores is part of the information they provide about $P$. Yet, in some cases, we can still benefit from the larger sample size of the transformed synthetic scores. This is evident by our K-nearest subsets experiments, which we elaborate on in Q8 below.\n>Q8: Why can selecting samples from different label classes help the prediction for a specific label class?\n\nOur method relies on synthetic data only through their scores---i.e., the prediction error of the fitted model---not through the raw inputs or labels. The experimental setting is motivated by prior work (e.g., ``Class-conditional conformal prediction with many classes'' by Ding et al.), which suggests that certain classes may have score distributions more similar to the class of interest. Selecting a subset of such classes can help align the synthetic and real score distributions, potentially improving statistical efficiency.\n>Q1: In Theorem 3.3, $\\varepsilon_{P,Q}^{m+1}$ depends on the TV distance of order statistics, which is a bit mysterious. How might this term depend on the TV distance between P and Q?\n\nThe following relation holds for the term $\\varepsilon_{P,Q}^{m+1}$:\n\n$\\varepsilon_{P,Q}^{m+1} \\leq \\mathrm{d}_\\text{TV}(P^{m+1},Q^{m+1}),$\n\nwhere $P^{m+1}$ denotes the joint distribution of $m$ i.i.d. samples from $P$, and $Q^{m+1}$ is defined similarly. Thus, the quantity is upper bounded by the TV distance between the product distributions $P^{m+1}$ and $Q^{m+1}$. Although the right-hand side is simpler and more intuitive, we chose to include the $\\varepsilon_{P,Q}^{m+1}$ term to present a tighter bound. We will add this discussion to the revised version.\n>Q2: The slack term $\\beta$ is discussed in lines 167-172 that only takes effect in extreme scenarios. Is there a way to derive a bound for that extreme event to obtain a sharper analysis?\n\nThank you for pointing this out---we found that the statements in lines 167–172 are somewhat vague, and we will make the necessary revisions. We apologize for the confusion. Specifically, what the term $\\beta$ accounts for is the case where test score $S_{m+1}$ is not covered by the corresponding window, rather than ``extreme” cases. Although the step where $\\beta$ appears involves a tight inequality, the final bounds we obtain can be loose, as some of the proof steps apply the union bound---which we believe is inevitable for the goal of deriving distribution-free bounds.\n\nIn fact, we claim that the bounds provided in this work are already quite tight. We kindly refer the reviewer to Item 1 in our response to Reviewer aUpi for further discussion on this topic and new experimental results.\n>Q3: How should one interpret the quantities in Theorem 3.5? In particular, what does $R_j^+$ and $R_j^-$ quantify, intuitively?\n\nFigure S3 in the appendix provides a concrete example of the real and synthetic scores along with their corresponding windows, illustrating how the guardrail bounds are derived. Specifically, $R_j^+$ and $R_j^-$ are ranks within the set of synthetic scores. The ranks $R_j^+$ and $R_j^-$ are chosen appropriately so that ``the $j$-th smallest real score is between $R_j^-$-th and $R_j^+$-th synthetic scores with sufficient probability\" holds. Therefore, roughly speaking, the interval $[R_j^-, R_j^+]$ is a prediction interval for the rank of $S_{(j)}$ in the set of synthetic scores.\n\nThe lower bound in Theorem 3.5 represents the proportion of windows that entirely lie below the conformal threshold $\\tilde{Q}_{1-\\alpha}$ based on synthetic data---which is equivalent to the proportion of $R_j^+$ that are less than $\\lceil(1-\\alpha)(N+1)\\rceil$.\n>Q7-A: Why do you choose beta=0.4 in the experiments? How would you recommend choosing the value of beta?\n\nWe choose $\\beta = 0.4$ based on the guardrail bounds in Theorem 3.5, and we recommend selecting its value following the same principle. Specifically, given the sample sizes $m, N$ and the target level $\\alpha$, Algorithm 4 in the appendix provides a way to set $\\beta$ to meet a user-specified lower bound on coverage.\n>Q7-B: How do the results change with the choice of beta? \n\nWe kindly refer the reviewer to the discussion and new results provided in Item 3 of our response to Reviewer UUws.\n>Q9: m=15 is extremely small for Section 4.2. How would the results change with the value of m? How large N is compared with m to ensure that one gets some improvement?\n\nNote that relevant discussions are provided in Sections I.1.1 and J.1, ``The effect of the real calibration set size,” along with supporting experiments---in case this was previously overlooked. There, we demonstrate meaningful improvements for larger values of $m$, such as 50. But, naturally, the degree of improvement from leveraging synthetic data tends to diminish as $m$ grows.\n\nThe amount of improvement depends not only on the synthetic size $N$ but also on the quality of the synthetic scores. If the synthetic score distribution is nearly identical to the real score distribution, then any $N > m$ will lead to improvement. However, if the synthetic scores are of low quality such that the procedure frequently hits the guardrail, the degree of improvement will be relatively minor, even with a large $N$."}, {"review_id": "LfVappc3fs", "reviewer": "Reviewer_UUws", "summary": "This paper proposes a new method called Synthetic-Powered Predictive Inference (SPPI), which focuses on improving Conformal Prediction (CP) when the amount of real calibration data is small. The key idea is to use synthetic data from a different distribution, and apply a score transporter that maps scores from the real data to match the distribution of synthetic scores. The method provides nearly tight coverage guarantee as well as the worst case coverage guarantee, and is shown to work well in both theory and experiments.", "questions": "I think this is a solid work. But if the authors clearly answer the questions and suggestions below, I’d be happy to further increase my score.\n\n**Questions**\n\n- Is there a specific analysis which shows the empirical behavior of lower bounds when $P$ and $Q$ show different levels of discrepancy?\n\n- Since this algorithm has an extra hyperparameter $\\beta$ that controls the window size, I’m curious about how sensitive the prediction sets are to different values of $\\beta$.\n\n**Suggestions**\n\n- The idea in Section 3.4.1 about matching the synthetic scores depending on real scores deserves more than just a short remark. To me, using another model (e.g. least squares) to align synthetic calibration scores to match the distribution of real scores might raise concerns about overfitting when new test points come in. But the authors show that this doesn’t happen, which is surprising and kind of counterintuitive. I think it would be great if the paper could discuss this part in more detail.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n- This paper is based on a brilliant idea and presents a solid method. It proposes a new way to build prediction sets when applying conformal prediction with a pretrained model and only a very small amount of calibration data is available. Also, the authors carefully prepare for possible drawbacks in Section 3.4, which I really appreciated.\n\n- The presentation of the paper is also very clear. When the authors explain the kind of problems their method can be applied to, they describe the feasibility of the situation very well in lines 25–34. This is actually an important and practical issue in the CP literature. Also, they clearly state the theoretical results of their method.\n\n**Weaknesses**\n\n- The only question I had while reading this paper is the following. Theorem 3.5 provides a safety bound when $P$ and $Q$ are not close. While the authors acknowledge this potential discrepancy, their explicit lower bound on coverage when $P$ and $Q$ are not the same is described somewhat vaguely. In particular, there is no quantitative analysis showing, under some reasonable experimental settings, how the lower bound behaves for a fixed value of $m$ and different levels of discrepancy between $P$ and $Q$.", "was_revised": true, "final_justification": "The authors addressed my (minor) concerns about coverage change under the level of distribution shift, and the sensitivity under the choice of $\\beta$ with clear theoretical and empirical support. I think this work is very solid, and I increase my rating.", "rebuttal": "Thank you for your thoughtful review and encouraging feedback. Below, we address your comments in detail.\n\n>1. The only question I had while reading this paper is the following. Theorem 3.5 provides a safety bound when P and Q are not close. While the authors acknowledge this potential discrepancy, their explicit lower bound on coverage when P and Q  are not the same is described somewhat vaguely. In particular, there is no quantitative analysis showing, under some reasonable experimental settings, how the lower bound behaves for a fixed value of $m$ and different levels of discrepancy between $P$ and $Q$.\n\n**The bounds in Theorem 3.5 are ``distribution-free guardrail bounds\" that do not depend on either $P$ or $Q$, or their level of discrepancy**. In other words, the lower bound is for following quantity:\n\n$\\inf_\\text{all distribution $P$ and $Q$} \\mathbb{P}(\\text{coverage event}),$\n\nand similarly, the upper bound is for the supremum of the coverage rates.\n\nThese bounds are determined solely by the sample sizes $N$, $m$, and levels $\\alpha$, $\\beta$, and are independent of any unknown distributions or observed data. Therefore, once $N$, $m$, $\\alpha$, and $\\beta$ are fixed, these bounds are constant and known. In fact, one can set $\\beta$ in advance to control the lower bound on the coverage, as detailed in Algorithm 4 of the Appendix. In our experimental plots (e.g., Figures 1 and 2), \nthese bounds are also shown (labeled as \"Lower bound\" and \"Upper bound\"), and we observe that, across different levels of synthetic data quality, the coverage of the SPPI method always lies within these bounds---as guaranteed by Theorem 3.5.\n\nThis is, in fact, one of the main advantages of SPPI: it provides a guardrail bound on the quality of inference. While high-quality synthetic data is certainly preferable, the method still yields meaningful inference even when the synthetic data is of low quality---which is important, given that we typically lack prior knowledge about the data quality.\n\n>2. Is there a specific analysis which shows the empirical behavior of lower bounds when P and Q show different levels of discrepancy?\n\nRegarding the lower bounds in Theorem 3.5, we believe that our response to the previous comment, along with the experimental results in the manuscript, sufficiently addresses the question.\n\nTheorem 3.3, on the other hand, provides coverage bounds that indeed depend on the discrepancy between the real and synthetic score distributions. The final theoretical bounds are given by the tighter of the two---that is, the maximum of the lower bounds and the minimum of the upper bounds from Theorems 3.3 and 3.5.\n\nIf the reviewer is referring to the bounds in Theorem 3.3, \nwe kindly refer the reviewer to the new results presented in Item 1 in our response to Reviewer aUpi. There, the table presents both the bounds from Theorems 3.3 and 3.5---as well as the best of the two (denoted as ``effective bounds\")---on synthetic data where the total variation distance between the real and synthetic score distributions is known. These results demonstrate how the bounds adapt to different levels of discrepancy between the real and synthetic score distributions: Theorem 3.3 dominates the bound when $P$ and $Q$ are close, whereas Theorem 3.5 dominates when $P$ and $Q$ are more distinct.\n\n>3. Since this algorithm has an extra hyperparameter beta that controls the window size, I’m curious about how sensitive the prediction sets are to different values of beta.\n\nWe agree that it would be interesting to examine the performance of our method under different choices of hyperparameters. We have conducted additional experiments to explore this, and we will include the relevant results in the revision (due to space limitation and the changed rules that do not allow pdf responses, we are only able to provide a summary here). We appreciate the suggestion.\n\nAt a theoretical level, the sensitivity depends on the similarity between the synthetic and real score distributions. Specifically, this can be seen in the score-transportation step: if the two score distributions differ substantially, then the transported score $T(S_{m+1})$ is likely to lie at one of the endpoints of the corresponding window---even under a small $\\beta$. As a result, the coverage of the SPPI prediction set will reach the guardrail bound in Theorem 3.5, indicating that we gain little from the synthetic data.\n\nOn the other hand, if the synthetic scores closely resemble the real scores, the transported score is likely to lie within the corresponding window rather than in its endpoints---even for relatively large values of $\\beta$.\nIn this case, the SPPI prediction set resembles conformal prediction with a larger sample size, achieving near $1-\\alpha$ coverage, and the effect of the hyperparameter becomes relatively minor.\n\nWe include experiments on the ImageNet data below. The table reports the performance of all methods across different values of $\\beta$ for the Siberian husky and lighter classes. The results align with the behavior described above: for the lighter class, the synthetic scores closely resemble the real ones, and accordingly, the performance of SPPI remains roughly the same across different $\\beta$ values. In contrast, for the Siberian husky class, where the synthetic scores deviate significantly from the real scores, the performance of SPPI closely aligns with the guardrail bounds. Note that this table also presents how the lower bound of Theorem 3.5 increases with $\\beta$, becoming closer to the nominal $1-\\alpha$ level.\n\n**Table: Results for the ImageNet data as a function of $\\beta$**  \nConditional coverage (%) and prediction set size across 100 trials for different β values, for *lighter* and *Siberian husky* classes. The target coverage is 95%. Standard errors are in parentheses. See Figure 1 (main text) for additional setup details.\n||||**Coverage (%)**|||**Size**|||\n|-|-|-|-|-|-|-|-|-|\n|**Class**|**β**|**THM 3.5**|`OnlyReal`|`OnlySynth`|`SPPI`|`OnlyReal`|`OnlySynth`|`SPPI`|\n|Lighter, light|0.005|75|100 (±0)|95.2 (±0.1)|95.3 (±0.1)|30 (±0)|3.9 (±0)|4.5 (±0)|\n||0.01|75|100 (±0)|95.2 (±0.1)|95.3 (±0.1)|30 (±0)|3.9 (±0)|4.5 (±0)|\n||0.05|81.2|100 (±0)|95.2 (±0.1)|95.3 (±0.1)|30 (±0)|3.9 (±0)|4.5 (±0)|\n||0.1|87.5|100 (±0)|95.2 (±0.1)|95.5 (±0.1)|30 (±0)|3.9 (±0)|4.8 (±0)|\n||0.2|87.5|100 (±0)|95.2 (±0.1)|95.5 (±0.1)|30 (±0)|3.9 (±0)|4.8 (±0)|\n||0.3|87.5|100 (±0)|95.2 (±0.1)|95.5 (±0.1)|30 (±0)|3.9 (±0)|4.8 (±0)|\n||0.4|93.8|100 (±0)|95.2 (±0.1)|96.8 (±0.2)|30 (±0)|3.9 (±0)|5.8 (±0)|\n||0.5|93.8|100 (±0)|95.2 (±0.1)|96.8 (±0.2)|30 (±0)|3.9 (±0)|5.8 (±0)|\n|Siberian husky|0.005|75|100 (±0)|56.7 (±0.3)|74.9 (±1.0)|30 (±0)|2.3 (±0)|2.7 (±0)|\n||0.01|75|100 (±0)|56.7 (±0.3)|74.9 (±1.0)|30 (±0)|2.3 (±0)|2.7 (±0)|\n||0.05|81.2|100 (±0)|56.7 (±0.3)|80.8 (±1.0)|30 (±0)|2.3 (±0)| 2.8 (±0)|\n||0.1|87.5|100 (±0)|56.7 (±0.3)|87.4 (±0.9)|30 (±0)|2.3 (±0)| 3.2 (±0)|\n||0.2|87.5|100 (±0)|56.7 (±0.3)|87.4 (±0.9)|30 (±0)|2.3 (±0)| 3.2 (±0)|\n||0.3|87.5|100 (±0)|56.7 (±0.3)|87.4 (±0.9)|30 (±0)|2.3 (±0)| 3.2 (±0)|\n||0.4|93.8|100 (±0)|56.7 (±0.3)|94.1 (±0.6)|30 (±0)|2.3 (±0)| 4.2 (±0)|\n||0.5|93.8|100 (±0)|56.7 (±0.3)|94.1 (±0.6)|30 (±0)|2.3 (±0)| 4.2 (±0)|\n\n>4. The idea in Section 3.4.1 about matching the synthetic scores depending on real scores deserves more than just a short remark. To me, using another model (e.g. least squares) to align synthetic calibration scores to match the distribution of real scores might raise concerns about overfitting when new test points come in. But the authors show that this doesn’t happen, which is surprising and kind of counterintuitive. I think it would be great if the paper could discuss this part in more detail.\n\nThank you for pointing this out; we will clarify this issue in the revised paper.\nThe core concern---how a transformation that depends on the real scores can avoid overfitting and still provide valid inference---is precisely what is addressed by Theorem 3.5 and its extensions (Corollaries 3.6 and 3.7). These results show that **the guardrail coverage bounds hold without any restrictions on the synthetic scores---they do not have to be i.i.d. and can even depend on the real data.**\nRoughly speaking, this is because, in the derivation of the bounds, any effect of the synthetic data is eliminated by a deterministic component inherent in the structure of the SPPI procedure.\n\nTherefore, at least in terms of maintaining the guardrail bounds, we are allowed to do ``anything”---including double dipping, overfitting, and other steps that would typically compromise statistical validity in other settings and procedures.\n\nThis theoretical flexibility allows users to adapt the synthetic scores to better align with the real scores in a data-driven manner, without compromising the validity of our guardrail coverage guarantees and without requiring data splitting. In our experiments (Figure 2 and Appendix I.2), we demonstrate that the K-nearest subset approach can improve performance by bringing the empirical coverage rate closer to $1-\\alpha$.\nThis method is motivated by prior work (e.g., ``Class-conditional conformal prediction with many classes'' by Ding et al.), which suggests that certain classes may have score distributions more similar to the class of interest.\n**More generally, whenever there is reason to believe that a subset of the synthetic data---or a transformation thereof---better aligns with the real scores, it is worthwhile to apply such methods**.\n\n\n## A closing remark\n\nThank you for the opportunity to clarify our theoretical results and for your thoughtful feedback on our method. If any part remains unclear, we would be happy to elaborate further. We sincerely believe our response addresses your concerns and will be helpful as you reconsider your evaluation."}]

Author rebuttal:

Meta-review: Claims and findings: This paper is about conformal prediction with synthetic data, motivated by data-starved scenarios. The basic idea is to show that it is possible to use synthetic samples by building mappings between the relevant quantities (e.g., scores) derived from synthetic and real samples. 

Strengths: The problem is important. The idea the authors develop is intuitive but is also pretty powerful. The paper is written in a clear way. 

Weaknesses: There are some practical questions (e.g., how to set some of the hyperparameters, how robust the method is to difference choices).

Decision: This is a nice paper that has a lot of intriguing results. It is worth accepting. 

Rebuttal discussion: The reviewers brought up some great questions, especially around practical use of the method. The authors’ answers were very helpful and will be valuable for the final version of the paper.
Decision: Accept (poster)