Conformal Bounds on Full-Reference Image Quality for Imaging Inverse Problems
Authors: Jeffrey Wen
Rizwan Ahmad
Philip Schniter
Venue: Accepted by TMLR ()

Abstract: In imaging inverse problems, we would like to know how close the recovered image is to the true image in terms of full-reference image quality (FRIQ) metrics like PSNR, SSIM, LPIPS, etc. This is especially important in safety-critical applications like medical imaging, where knowing that, say, the SSIM was poor could potentially avoid a costly misdiagnosis. But since we don’t know the true image, computing FRIQ is non-trivial. In this work, we combine conformal prediction with approximate posterior sampling to construct bounds on FRIQ that are guaranteed to hold up to a user-specified error probability. We demonstrate our approach on image denoising and accelerated magnetic resonance imaging (MRI) problems.

Published in Transactions on Machine Learning Research (05/2025)
Conformal Bounds on Full-Reference Image Quality for Imag- ing Inverse Problems
Jeffrey Wen
wen.254 @osu. edu
Deportment of Electricnl and Computer Engineering The Ohio Stote University
Rizwan Ahmad
rizwan, www.aaaaaaaddsu.mmooooo
Department of Biomedical Engineering The Ohio State University
Philip Schniter
scloiter. 1 edu
Department of Electricol and Computer Engineering The Ohio State University
Reviewed on OpenReview: https:// opwnreveee..........mm net/ forum? WADLPccB6o
Abstract
In imaging inverse problems, would like to know how close the recovered image is to the true image in terms of full-reference image quality (FRIQ) metrics like PSNR, SSIM, LPIPS, etc. This is especially important in safety-critical applications like medical imaging, where knowing that, say, the SSIM was poor could potentially avoid a costly misdiagnosis. But since we don't know the true image, computing FRIQ is non-trivial. In this work, we combine conformal prediction with approximate posterior sampling to construct bounds on FRIQ that are guaranteed to hold up to uscr-specified error probability. We demonstrate our approach on image denoising and accelerated magnetic resonance imaging (MRI) problems. Code is available at https: // thub. com/ homm///en377qqaliiie
1 Introduction
In imaging inverse problems, one aims to recover a true image from noisy/distorted/ nois/ddistoreediinomptt 'incomplete measure- ments Yo = A(xu) (Arridge et al.) 2019). Denoising, deblurring, inpainting, super-rcsol super-rcsolutin, limited-angle computed tomography, and accelerated magnetic resonance imaging (MRT) are examples of linear inverse problems, while phase-retrieval, de-quantization, low-light imaging, and image-to-imaae translation are examples of non-linear inverse problems. Such problems ill-posed, in that many hypotheses of zo consistent with both the measurements Yo and prior knowledge about Zg. To complicate matters, different recovery methods are biased towards different plausible image hypotheses, leading to important differences in reconstruction quality. For example, modern deep-network approaches can sometimes hallucinate (Cohen et al. 2018 Belthangady Royer 2019 Hoffman et al. 2021 Muckley et al., 2021 Bhadra et al. 2021
Gottschling et al. 2023 Tivnan al. 2024), i.e., generate visually pleasing recoveries that differ in importani. ways from the true image &p- Thus, there a strong nced to quantify the accuracy of a given recovery,
especially in safety-critical applications like medical imaging (Chu et al. 2020 Banerji et al. 2023
In image recovery, "accuracy" be defined in different ways. Classical metrics like mean-squared error (MSE), or its scaled counterpart peak signal-to-noise ratio (PSNR), are convenient for theoretical analysis but do not always correlate well with human perceptions of image quality. This fact inspired the field of full-reference image-quality (FRIQ) assessment [Lin & Kuo 2011 Wang 2011), which led to the well-known Structural Similarity Index Measure (SSIM) (Wang et al. (2004) that still popular today. However, progress continues be made. Most recent methods leverage the internal features of deep neural networks, which are said to mimie the processing architecture of the human visual cortex (Yamins & DiCarlo, 2016 Lindsay

---

Published in Transactions on Machine Learning Research (05/2025)
2021 A popular example of the latter is Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al. 2018). In the end, though, the best choice of metric may depend on the application. For example, in magnetic resonance imaging (MRI), the goal is to provide the radiologist with an image recovery that leads to an accurate diagnosis. A recent clinical MRI study (Kastryulin et al. 2023 found that, among 35 tested metrics, Deep Image Structure and ure Similarity (DISTS) (Ding el al. 2020a) correlated best with radiologists' ratings of perceived noise level, contrast level, and presence of artifacts when comparing reconstructions to the fully sampled ground-truth.
In this work, our goal is to provide rigorous bounds the FRIQ m(3o, zo) of recovery To h(30) relative to the true image Tp. Here, k(-)) is an arbitrary imago-recovery scheme and m(:,-) is an arbitrary FRIQ metric. The key challenge is that zo is unknown. To our knowledge, there exists no prior work on providing FRIQ guarantees in image recovery. Our contributions are as follows.
1.
We propose a framework to bound the FRIQ m(Gu, _n) of a recovered image Ty without access to the true image Ti- Our framework uses conformal prediction (Vovk et al. 2005 Angelopoulos & Bates) 2023) to construct bounds that hold with probability at least 1 a under certain exchangeability assumptions and where a E (0, is chosen by the user.
We show how posterior-sampling--- that adapt to the measurements 1/o and reconstruction To.
image recovery can be used to construct conformal bounds
3. We demonstrate our approach on two linear inverse problems: denoising of FFHQ faces (Karras et al., 2019) faces and recovery of fastMRI knee images (Zbontar et al. 2018 from accelerated multicoil measurements.
From the perspective of uncertainly quantification (UQ), one could say that our goal is to bound the uncertainty on FRIQ m() that arises due to 2o being unknown. As such, our approach to UQ differs from typical ones in image recovery. There, uncertainty is typically quantified on individual pixels, with the overall result being a pixel-wise uncertainty map. To construct these maps, it's popular to use (approximate)
posterior samplers (Adler & Öktem 2018 Tonolini et al.
2020
Edupuganti et al.
2021 Jalal et al. 2021
Sun & Bouman 2021|
Laumont et al.) 2022 Zach et al. 2023
Wen et al. 2023a
Bendel et al.) 2023 Wu
ct al. 2024) or Bayesian neural networks (BNNs) (Kendall Gal
2017 Xue et al. 2019 Barbano et al. 2021
Ekmneei & Cetin 2022 Narnhofer et al. 2022 to draw many reconstructions from the distribution of plausible (x.))_((13))), (-136)), from which pixel-wise standard-devvaaiion
20 for given Yo (i.c., the posterior distribution Px./1%
can be estimated. An alternative is to utilize conformal prediction to produce pixel-wise intervals that are
guaranteed to contain the true pixel value with high probability (Angelopoulos et al. 2022b Horwitz &
Hoshen 2022 Teneggi et al., 2023 Kutiel et al.) 2023 Narnhofer et al., 2024). Although these uncertainty maps can be visually interesting, they do not quantify uncertainty on multi-pixel structures of interest, such as hallucinations or anatomical features relevant to medical diagnosis (e.g., tumors).
To our knowledge, there exist relatively few works on multi-pixel UQ, and none target FRIQ. For example, Durmus et al. (2018) use hypothesis testing to infer the presence presence/absence absence of structure-of-intersst within the maximum a posteriori (MAP) image recovery, but their method relies on inpainting to construct the structure-absent hypothesis, which may not be accurate. (Sankaranarayanan et al. (002 use conformal prediction to compute uncertainty intervals on the presence/ /absence of semantic attributes (e.g., whether a face has a smile, glasses, etc.) but their method requires a "disentangled" generative adversarial network (GAN) that generates image samples given a set of attribute probabilities. Belhasin et al. (2023) compute conformal prediction intervals on the principal components of the posterior covariance matrix. Lastly, given mcasurements Vo A(za) and downstream imaging task ud-) R (e.g., soft-output classification), Wen et al. (2024) compute conformal bounds the true task output u(zz). While interesting, none of the above works quantify the uncertainty on FRIQ metrics like PSNR, SSIM, LPIPS, DISTS, etc., due to zo being unknown. As metrics like SSIM, DISTS, and LPIPS are designed to quantily higher-level perceptual similarities, bounds on these metrics can provide insight into the differences between multi-pixel structures within the reconstruction and the true image. For applications like accclerated MRI, where DISTS has been shown to correlate With radiologists' perceptions (Kastryulin et al. 2023), rigorous bounds on FRIQ can provide useful information the clinical utility of an accelerated reconstruction.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Background
Conformal prediction (CP) (Vovk et al. 2005 Angelopoulos & Bates 2023) is a powerful framework for computing uncertainty intervals on the output of any black-box predictor. CP makes no assumptions on the distributiin of the data, yet provides probabilistic guarantees that the true larget lies within the ronstr uncertainty interval. In this paper, we focus on the common variant known as split CP (Papadopoulos et al. Lei et al. 2018).
2002
We now provide brief background split CP. Given features uo U, the goal of CP is to construct set Cx (2)) that contains an unknown target 2o E 2 with high probability. Here, 2g = fuo) is some prediction from a black-box model f()), and Cx(-) is constructed so that (30) C (x)) for any À < X and En. Split CP accomplishes this goal by calibrating À using a dataset of feature and target pairs {(u,, 2)) that has not been used to train f(-). In particular, it first constructs the set deal _ ((2, 2, using 2 f(u,) and
then finds A(deal) to provide the marginal coverage guarantee (Lei & Wasserman 2014
Pr Zoo E CxIDa) (2o) 21-a,
(1)
where a is a user-chosen error rale. Here and in the sequel, we use capital letters to denote random variables and lower-case letters to denote their realizations. In words, ( guarantees that the unknown target Zo falls
within the interval (xp,a) (Zo) with probability at least a when averaged over the randomness in the test
data (Zo, Zo) and calibration data Deal.
While there are a number of ways to describe CP calibration À (Vovk et al. 2005 2023), we will focus on the method from Angelopoulos et al. (2022a). starts by defining the empirical miscoverage as
Angelopoulos Bates
1 Σ 12 $ Cx))),
P., deal)
(2)
where 11-3 is the indicator function. The empirical miscoverage measures the proportion of targets 2 that land outside of Cx in the calibration set deat- The calibration procedure then sets À at
A(das) = inf {A (M; deal) a -a--1p 1=11 1-a],
(3)
which can be found using a simple binary search. Intuitively, the X chosen in 13 yields an empirical
miscoverage that is slightly more conservative than the desired a in order to handle the finite size of the calibration set. When (Zo. Zo). (21, Zi). (Zn, Zn)} are statistically exchangeable, (3) ensures that (1)
holds (Angelopoulos et al. 2022a). See the overviews (Angclopoulos & Bates 2023 Vovk al.) 2005) for
more conformal prediction.
For exchangeability, it suffices that the feature/targer pairs (U Zi) are i..d across test and calibration samples i, although independence is not strictly necessary. However, when there is distribution shift between the test and calibration samples, exchangeability will be lost. Works like Tibshirani et al. (2019); Barber et al. )2023; Cauchois et al. 2024 propose methods to handle distribution shift. We discuss distribution
shift further in Sec. 4.3
3 Proposed approach
Consider an imaging inverse problem, where we observe distorted, incomplete, and/or noisy measurements Y0 = A(zo) of true image 2g. Suppose that 2o h(yo) is reconstruction of zo provided by some image recovery method h(-)) and that 2g m(Fa, æo) E IR is some FRIQ metric on To with respect to the truc We would like to know oo, especially in safety critical applications. For example, if 20 was unacceptable, then perhaps we could use a different. recovery method 1(-)) or collect more measurements yo- But cannot. be directly computed because, in practice, Ho unknown.
Our key insight that it's possible construct a set Cx that is guaranteed to contain the unknown FRIQ 2o with high probability. This can be done using CP, at least when has access to calibration data
3

---

Published in Transactions on Machine Learning Research (05/2025)
( us));= of true image and measurement pairs that agrees with the test (Zu, yo) in the sense that the resulting FRIQ pairs {(2, x))== are statistically exchangeable. As in Sec. 2 the calibration data must be distinct from the data used to train h(-) or any other other model.
Our general approach as follows. Using y:),,:))==1 we compute the image recovery E, = h(y.) and the corresponding true FRIQ 2; = m(( 24)) for each i = 1,....n.. n, Then we construct an estimator f()) that produces an FRIQ estimate 2 = f(u)) for some choice of u; Several choices of f(-) and u, will be described in the sequel. We then collect the results into the set deal {(2,, 4)]==1 and calibrate the À parameter of the FRIQ prediction interval (x)2) using CP.
We now describe our choice of prediction interval (+). In the sequel, we will refer to those metrics m(:-) for which higher value indicates better image quality (e.g., PSNR, SSIM) as higher-preferred (HP) metrics, and those for which a lower value indicates better image quality (e.g. LPIPS, DISTS) as lower-prefereed (LP) metrics. We choose to construct the prediction set for the i-th sample as
(xi2)) = (B(E,, A), oc) for HP metrics and Cx(2,) = (-o,, A)] for LP metrics,
(4)
where we choose the lower/ upper bound ((ss)) as
8(2, A) = 2, > for HP metrics and B A) = 2g + A for LP metrics.
(5)
Although our prediction intervals have infinite width, they ensure that the empirical miscoverage in is monotonically non-increasing in X, as required by Angelopoulos el. al. (022a)).
By calibrating the bound parameter X as X(dea) using (3, we obtain the following marginal coverage guarantee for the test sample (70; Zo):
Pr Zo E
C (D (2o)) 2 -
(6)
which holds when ((Zo, 20), (2 2), (Z,. }n)} are exchangeable (Angelopoulos et al. 2022a In partic- ular, (Zo,, X(Deai)) lower-bounds the unknown true HP metric value Zo, or upper-bounds the unknown true LP metric value Zo, with probability at least 1 a, where is selected by the user. A smaller error-rate a will tend to yield looser bound, but importantly the coverage guarantee (6) will hold for any chosen a (0,1), 1). In the sequel, we will refer to 8(2,, X(deai) as the "conformal bound" on 20- Note that the conformal bound can "adapt" to the test measurements yo and reconstruction To through 2n = f(oo for appropriate choices of f()) and uo.
Below we describe different ways to construct f(-) and uo, which in turn yield conformal bounds with different properties. Section 4.3 investigates violations of the exchangeability assumption.
3.1 A non-adaptive bound
snlovered-image FRIQ
As a simple bascline, we start with the choice f(-) = 0. In this case, to is inconsoquential and 2. 0, and so the conformal bound ( X(dea)) will depend the calibration set deal but not the test measurements yo or reconstruction Eg. We refer to such bounds as "non-adaptive." As we demonstrate in Sec. 4 non-adaptivity leads to conservative bounds. Still, this non-adaptive bound is valid in the sense of guaranteed marginal coverage (6) under the exchangeability assumption.
3.2 Intuitions constructing adaptive FRIQ bounds
Our approach to constructing adaptive FRIQ bounds is based the following probabilistic viewpoint. Conditioned on the observed measurements yo, we can model the unknown FRIQ as Zo m Xo) for ã = ((v)) and X, Px,,,%,-lbu). (-190). The distribution Px.1%% (-190) is often referred to as the posterior distribution on Xo given the measurements Yo = yo-
Let us first consider the idcal and unrcalistic case that the -o-conditional FRIQ distribution Pz. (:19)) is known. And let's consider the case of HP metrics, noting that all of our arguments can be easily modified to cover I.P metrics. If Pz, (:190) was known, then constructing lower-bound on Zo that holds with
4

---

Published in Transactions on Machine Learning Research (05/2025)
True Image
Recovered Image Eo
Prediction Interval Construction
Posterior Samples {=] Le t jj=1
20
Metric ml.,.)
Metric m(.,.)
{0 1j=1
True Image Quality 20
E I
Estimator f
Prediction Interval (dda))
Calibration
Image Quality
20
deal
20
Cx(dea)) (20)
Conformal Prediction
{ 2i)
Goal: Pr(Zo Cx(Daa) (2n) 2
Figure 1: Overview of method: Given a recovery To of true image xn, approximate posterior samples {x(ا]] j==11 and a calibration set deal, we construct prediction interval (raai) (3o) that guaranteed to contain the unknown true FRIQ 2g m(To, 2o) with probability at least 1 -
probability 2 1 - could be directly accomplished by finding the 3 E R that satisfies Pr(Zo 2 3|Y6 yo) - which is known as the ath quantile of Zo|Yo= = yoo
Now suppose that the distribution of Zo|Yo = You was unknown, but instoad one had access to an infinito z(j) number of perfect posterior image samples { By "perfect" mean that Io are independent
realizations of YY you From them, could construct posterior FRIQs (2 using 26 m(20. =0) Importantly, {2o, (11) x(2) x(3) on Yo yo that holds with probability - a, one could use the empirical quantile {-(1)) } i..,
20 ..... are iid. realizations of Zo Yo yo= Thus, to construct lower bound B
B lim EmpQuant (a, (20 )5=1),
c000
which converges to the ath quantile of Yo Fristedt & Gray 2013
In practice, one will not have access to an infinite number of perfect posterior image samples. However, it is Jÿ=1- From them, could
not difficult to obtain finite number of approcimate posterior samples [
estimate the ath quantile of Zo) Yo =Woo yo and subsequently calibrate that (imperfect) estimate using conformal prediction. Two such strategies are described below.
3.3
An adaptive bound recovered-image FRIQ
Suppose that, for each E (0, 1.......,, n). we have access to 2 1 approximate posterior image samples (26) J produced by a black-box posterior image sampler such as those listed in Sec. [ Guided by the intuitions from Sec. 3.2 we propose the following for HP metrics. For each
first compute the corresponding approximate (2(1) 1- using z(i) = m(2,, a(1)) and then set 2n at their empirical quantile
posterior FRIQs {
f(-) EmppQuant(a.)) u; (1)
2 EmpQuant (a, (2(0)))))) )=1) = f(u)) for
(8)
210) 2 T ERC.
We then use deal ((2,, 2) 2 to calibrate the bound parameter À using yielding (ddaa. Finally, we plug this and & into (5) to get B(2, A(dea)), which our conformal bound on the true FRIQ 2g- From Sec. 2 we know that this conformal bound satisfies the coverage guarantee (6 under the exchangeability assumption. Furthermore, it adapts to the measurements yo and reconstruction To through their effect
5

---

Published in Transactions on Machine Learning Research (05/2025)
So and (2 1j=1> unlike the non-adaplive bound from Sec. 3.1 We refer to these conformal bounds as the "quantile" bounds.
Recalling Sec. 3.2 one could interpret Eo rough estimate of the ath quantile of Zu Yo = and X(dea) as an additive correction that accounts for the finite and approximale nature of the posterior image samples (+) ==1 used to construct 8g. For LP metrics, would instead compute the (1 a-)-mpirical quantile in F Figure illustrates the overall methodology.
3.4 A learned adaptive bound on recovered-image FRIQ
In Sec. 3.2 we reasoned that the ath quantile of -o|Yo= yo yields a valid HP FRIQ bound, but we noted that this quantile is not directly observable. Thus, in Sec. 3.3 we used the oth empirical quantile of 1 a rough estimate "2;" of the desired quantile, after which we used CP to correct this estimate and obtain a valid HP FRIQ bound. However, is well known from the CP literature that inaccurate base estimators cause loose conformal bounds (Angclopoulos Bates
2023
Thus, in this scction, we aim to improve our
estimate of the ath quantile of Zo| yo.
Inspired by conformalized quantile regression Romaao et al. 2019 propose to estimate the ath quantile of Zo Yo y/ using
2, = fu;;; 0) with 4 =
ER;
(9)
where 9 are predictor parameters trained using quantile regression (QR) (Koenker & Bassett example () is given in App. In the case of an HP metric, this manifests as
1978). An
arg min
(a -0,00) (1 a) max(0,22 -z:)) + 7p((),
(10)
Mpain that is independent. of the calibration samples {(u,, x)) and using a training set draiin {wu, 2;) R=n+ au test sample (uo, 20).
The first term in (10) is the pinball loss [Koenker & Bassctt
1978 which encourages an a-fraction of training samples to violate the HP bound & < z;. The p(-) term in 10 is regularization that avoids overfitting 0 to the training set. The regularization weight y can be tuned using k-fold cross validaaai....on. The d-dependence of 2, made explicit in 10).
Once the predictor ((;0) is trained, it can be used to obtain the quantile estimates (2: Then deat {(2, 2)) can be used to calibrate the bound parameter À using (3) As before, the resulting conformal bound ((2,, X(daa)) will enjoy the coverage guaranlee (6) under the exchangeability assumption. To handle LP metrics, we would swap with 1-a a in (10). Note that any estimation function f(:;0) can be used in 9 and the best choice will vary with the application. Through the remainder of the paper, describe these bounds the "regression" bounds.
4 Numerical experiments
We now consider two imaging inverse problems: image denoising and accelerated MRI. For each, we evaluate the proposed bounds using the PSNR, SSIM (Wang al.] 2004), LPIPS (Zhang al. 2018), and DISTS Ding et al. 2020a metrics.
4.1
Denoising
Data:
For true images, we use a random subset of 4000 images from the Flickr Faces HQ (FFHQ) (Karaa ct al. 2019) validation dataset, which we added white Gaussian noise of standard deviation a 0.75 create the measurements Vo- The first 1000 images were used to train the predictor f(::0) in 19 and the remaining 3000 used for calibration and testing.

---

Published in Transactions on Machine Learning Research (05/2025)
DISTS L
PSNR
Measurement
True Image
Recovered Image
Bound 4
0.30
0.25
16.
LPIPS
SSIM
0.6
Bound p
0.6
0.4
0.
0.3
0-4
0.2
0.6
0.6
0.
0.4
0
0.9
True Quality z
True Quality Z
non-adaptive bound
quantile bound
Figure 2: Scaller plots show the non-adaptive (purple) and quantile (green) bounds ( ((deaa) versus the true FRIQ 2μ: over FFHQ test samples k. The black line shows where 3 2, and fraction a 0.05 of samples are the side of the line that violates the bound. The quantile bound tracks the true zạ much better than the non-adaptive bound. The red and blue stars correspond to the images in the red and blue boxes: the red recovery represents better FRIQs and blue represents worse.
True Image
Recovered Image
True Image
Recovered Image
True Image
Recovered Image
0.44
0.42
N
N
0.45
0.26
0.28
0.33
YN
0.58
0.61
0.58
0.55
0.58
0.57
Figure Examples from the FFHQ denoising experiment. Top row: true image and low-LPIPS recovery. Bottom row: true image and high-LPIPS recovery. True LPIPS reported in blue and quantile upper-bound in red. (Recall that LPIPS assigns lower values to better recoveries.)
the Denoising Diffusion Restoration Model (DDRM) (Kawar et al. 2022a). Following Kawar et al. (2022a), we run DDRM with Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020) pretrained on the CelebA-HQ dataset (Karras et al. 2018 To increase sampling diversity, we used 7 = 1 and 77 = 0.5 but set all other hyperparameters at their default values. For each measurement yı, we use one DDRM sample for the image estimate T, and e independent. samples for {- (1) )-
Recovery: To recover Zy from Yo: denoising task,
Conformal bounds:
3.3 and 3.4 which we We evaluate the proposed bounding methods from Secs. 3.1
refer to as the non-adaptive, quantile, and regression bounds, respectively. For the regression bound, use quantile predictor f(,,)) that takes the form of a linear spline with two knots (sce Appendix F for more details).

---

Published in Transactions on Machine Learning Research (05/2025)
non-adaptive bound
quantile bound
regression bound
DISTS
LPIPS
PSNR
S5IM T
Mean Conformal Bound
0.36
0.55
0.50
0.54
19 5
0.49
0.53
0.48
19 0
0.52
0.47
0.33
0.51
18.5
0.46-
12 8
16
32
12 4
16
32
12 4
8
16
32
12.4
8
26
32
Number of Image camples
Number of Image Samples
Number of Image Samples
Number of Image Samples
Figure 4: Mean conformal bound versus number of posterior samples for FFHQ denoising.
non-adaptive bound
quantile bound
regression bound
DISTS
LPIPS
PSNR
SSIM
Mean Absolute Difference
0.140
0.070
2.4
0.135
0.065
2.0
0.130
0 08
0.060
1.8
0.125
0 07
12 1
6
16
32
12
4
8
16
32
12
4
8
16
32
12 4
16
32
Number of Image Samples
Number of Image Samples
Number of Image Samples
Number Image Samples
Figure 5: Mean absolute difference between the bound and true FRIQ versus number of posterior samples e for FFHQ denoising.
Validation procedure: Because the coverage guarantee 6 involves random calibration data and test data, evaluate methods using T Monte-Carlo trials. For each trial E {1,...,?)) T) we randomly select 70% of the 3000 non-training samples to create the calibration set deai[t] with indices E Icai[t), and we use the remaining 30% of the non-training samples for test fold with indices k E Itealt). In particular, we compute X using dra(tt and then, for each test sample index k compute the bound 3(2,, ((d)al(t))) Finally, we average performance across the test indices of trial t and then average those results across the T trials. Unless specified otherwise, we used error rate = 0.03, T = 10 000, and c = 32 samples for the adaptive bounds.
Bound versus true metric: Ideally, an FRIQ bound should track the true FRIQ in the sense that the bound is small when the true FRIQ is small and large when the true FRIQ is large. To assess this tracking behavior, Fig. shows scatter plots of the uon-adaptive and quantile bounds 3(2,, ((deal(()) versus the true FRIQ mą for the test indices [tes[[t] of a single Monte Carlo trial, along with the true image Iy and recovery E4 for two test samples. The sample highlighted in red has better subjective visual quality compared to the one in blue, and this ranking is reflected in both the true FRIQ metrics zµ and the corresponding quantile bounds, but not the non-adaptive bound. Figure g shows six additional samples from the FFHQ denoising experiment, three with low (true) LPIPS and three with high (true) LPIPS, along with the respective true images. The quantile upper-bound on LPIPS superimposed on each recovery. We see that the bounds are valid the sense that they did not under-predict the true LPIPS and adaptive the sense that the bounding value lower when the true LPIPS is lower.
Empirical Coverage: To verify the coverage guarantee in 6 is satisfied, we compute the empirical coverage
EC||
A
Σ kEIwa [t]
1( E E (ADD,)) (2.)),
(11)
Trest
for cach Monte Carlo trial In Table D we report the average empirical coverage and standard error across T 100 trials for all three methods on the FFHQ data using a target error rate of a 0.05. For all methods, the average empirical coverage is very close to the theoretical coverage of 1 a 0.95 regardless of
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Mean empirical coverage for all bounds with a = 0.05 and T 100000 on the FFHQ denoising task (# standard error). Quantile and regression bounds are computed with c 32.
Bound
DISTS
LPIPS
PSNR
SSIM
Non-adaptive
0.95000 = 0.00009
0.95016 + 0.00009
0.95004 = 0.00009
0.95010 + 0.00009
Quantile
0.95002 = 0.00009
0.95013 + 0.00009
0.95003 = 0.00009
0.95006 + 0.00009
Regression
0.95013 + 0.00009
0.95026 + 0.00009
0.95001 + 0.00009
0.95006 + 0.00009
the metric, demonstrating close adherence to the theory. In Appendix we further demonstrate that this adherence holds independent of the choice of c.
Bound tightness versus bounding method and number of posterior samples e: To assess the tightness of the conformal bounds, we average the bound ((2,, ((dcai())))) over the test indices & E Itea(t) and the Monte Carlo trials / to yield the "mean conformal bound" (MCB). We also average the absolute difference between the bound and true FRIQ I [(x,, {()cai(()) over the test indices and Monte Carlo trials to find the "mean absolute difference" (MAD). Figure 4 plots the MCB versus the number of posterior samples used for the adaptive bounds. The figure shows that the non-adaptive bound is looser (i.e., smaller for the HP metrics PSNR and SSIM and larger for the LP metrics DISTS and LPIPS) than the two adaptive bounds. Figure 5 plots the MAD versus the number of posterior samples c. Similarly, this figure shows the non-adaptive bound further from the true FRIQ compared to the quantile and regression bounds. For both adaptive bounds, Figs. and show only minor bound improvement with increasing c, suggesting that the adaptive bounds robust to the choice of c, and that small values of could suffice if sample-generation was computationally expensive. (We discuss computation time below.)
Interestingly, Figs. g and show relatively little improvement when going from the quantile bound to the regression bound. This may be due to our choice of a linear spline with two knots for f(: but experiments with higher spline orders and/or more knots did not yield improved results, and neither did experiments with XGBoost (Chen & Guestrin 2016) models for f(: Additional experiments that hold the number of test samples al 900 and vary Aeerain and Teaal such that Aezzain Teat 3100 (see Appendix B) also show little change in the performance of the quantile and regression bounds. Thus, for our experimental data, the effort to train the estimation function f(:;0) from (9) may not be justified, given the good performance of the simple empirical-quantile estimation function f(-) from ()) But the behavior may be different with other datasets.
Using a single NVIDIA V100 GPU with 32GB of memory, computing a single DDRM sample takes approximately 2.73 soconds. Once the calibration constant ((dea)) is known, computing c == FRIQ samples { ) and B(2o, ((deai)) takes around 217ms, 320ms, 5ms, and 6ms for DISTS, LPIPS, PSNR, and SSIM, respectively.
Computation time:
4.2 Accelerated MRI
We now simulate our methods on accelerated multicoil MRI (Knoll et al. 2020: Hammernik et al. 2023 MRI is a medical imaging technique known for excellent soft tissue contrast without subjecting the patient to harmful ionizing radiation. MRI has slow scan times, though, which reduce patient throughput and comfort. In accelerated MRI, one collects only 1/R of the measurements specified by the Nyquist sampling theorem, thus speeding up the acquisition process by rate R. For R > 1, however, the inverse problem may become ill- ill-posed, in which case may be interested in bounding the FRIQ of the recovered image.
We utilize the non-fat-suppressed subset of the multicoil fastMRI knee dataset (Zbontar et al. 2018), yielding 17286 training images and 2188 validation images. To simulate the imaging process, we rel rospectivell sub-sample in the spatial Fourier domain (the "k-space") using random Cartesian masks that give acceleration rates {16, 8, 4,2). See App. E for additional details.
Data:
Recovery: To generate approximate posterior samples for the adaptive bounds, we utilize the conditional normalizing flow (CNF) from Wen et al. (022a)). Although in App. D investigate what happens when
9

---

Published in Transactions on Machine Learning Research (05/2025)
DISTS L
LPIPS
PSNR
SSIM
05
Bound
0.20
0.15
0.10
0.2
0.6
0.8 Quality.
Qualitty,
Quality,
Quality,
ndaptivetive bound
quantil bound
Figure 6: Scatter plots show the non-adaptive (blue) and quantile (orange) bounds _(2,, M(deal(t)) versus the true FRIQ over MRT test indices & E Trstt at acceleration R = 8. The black line shows where = A fraction of a = 0.05 samples are on the side of the line that violates the bound. Note that the quantile bound tracks the true zạ much better than the non-adaptive bound.
non-adaptive
quantile
non-adaptive
quantile
1.0
1.0
Fraction Accepted
Fraction Accepted
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
16
8
2
16
8
2
1
Acceleration, R
Acceleration, R
Figure 7: Fraction of accepted slices versus fi- nal acceleration rate for multi-round MRI using DISTS with = 0.16. Error bars show standard deviation.
Figure 8: Fraction of accepted slices versus fi- nal acceleration rate for multi-round MRI using PSNR with t = 33dB, Error bars show standard deviation.
the same CNF also used as the recovery network h(.), in this section we use the well-known E2E- VarNet [Sriram al. 2020a deterministic reconstruction approach, for I(())
Both networks are trained work well with all four acceleration rates R. (See App. F for training details.) Similar to Sec. 4.1 we found that the regression bound did not provide significant gain over the quantile bound and so, to stramlline our discussion,
consider only the quantile and non-adaptive bounds for MRT. As before, we evaluate performance over T = 10 000 Monte Carlo trials with a random 70% calibration and 30% test split of the validation data. All experiments use an error-rate a = 0.05. Methods are separately calibrated for each acceleration rate.
Bound versus truc-metrie: Figure shows scatter plots of the true FRIQ 2 versus the non-adaptive and quantile bounds ((2,, (())) for the test indices à Tsest [t] in a single Monte-Carlo trial The results are shown for R 8 acceleration and 32 samples in the adaptive bounds. Except for few outliers, the quantile bound closely tracks the true FRIQ 2k. demonstrating good adaptivity, while the non-adaptive bounds remain constant with Z
To showcase the practical impact our bounds, we adapt the multi-round measurement protocol from Wen et al. (2024), where measurements are collected over multiple rounds until the uncertainty bound falls below a threshold. In our setting, measurements are first collected at acceleration R 16, image recovery is computed, and conformal upper-bound its DISTS is computed. Tf the
Multi-round measurement:
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Average results for a multi-round MRI simulation where measurement collection stops once the DIST bound is below a user-sct threshold т. Results shown for T 000 trials using 0.05, 0.16, and c 32 (+ standard error).
Method
Average
Acceptance
R 16 Empirical
R 8
R Empirical
R Empirical
Acceleration
Empirical
Empirical
Coverage
Coverage
Coverage
Coverage
Coverage
Non-adaptive Quantile
2.000 0.000 3.973 + 0.001
0.9504 + 0.0001
0.9505 + 0.0001
0.9505 0.0001
0.9505 0.0001
0.9506 + 0.0001
0.9323 + 0.0001
0.9506 + 0.0001
0.9505 + 0.0001
0.9505 = O. 0001
0.9503 = 0.0001
Table Average results for a multi-round MRI simulation where measurement collection stop once the PSNR bound is above a user-set threshold T. Results shown for T 10 000 trials with a 0.05, T 33.0 dB, and 32 (+ standard error).
Method
Average Acceleration
Acceptance Empirical
R 16 Empirical
R = Empirical
R=4 Empirical
R = Empirical Coverage
Coverage
Coverage
Coverage
Coverage
Non-adaptive Quantile
2.000 0.000 4.048 + 0.001
0.9503 + 0.0001
0.9504 0.0001
0.9504 0.0001 0.9505 + 0.0001
0.9504 = 0.0001
0.9504 = 0.0001
0.9514 + 0.0001
0.9503 + 0.0001
0.9504 = 0.0001
0.9503 = 0.0001
bounding value is lower than a pre-determined threshold T, signifying that the recovery is (with probability a:) of sufficient diagnostic quality (Kastryulin ct al., (023), then measurement collection stops. If not, additional measurements are collected and combined with the previous ones to yield an acceleration of / = 8, and the process repeats. We allow up to five measurement rounds, corresponding to final accelerations of R E {16, 8, 1,
Once again, we report average results across T = 10 000 trials. We set the DISTS acceplance threshold at T == 0.16, which requires the non-adaptive approach to use acceleration R = 2 in order to guarantee 1 - empirical coverage. Figure / plots the fraction of test image slices accepted by the multi-round protocol at each acceleration rate R with rounds (i.e., R 4) in more than 80% of the cases. With the non-adaptive bound, the measurements stop after four rounds (i.c., R 2) in all cases. Table 2 shows that, with the quantile bound, the multi-round protocol allains an average acceleration of A = 3.973, which far surpasses the R = 2 acceleration achieved with the non-adaptive bound, Since a separate conformal predictor is calibrated for each acceleration rate, Table 2 shows that each predictor satisfics the coverage guarantees (6) when evaluated on all test samples at the corresponding acceleration rate. We also show the acceptance empirical coverage, which computes the coverage when the slices are accepted. Despite only having coverage guarantees for individual acceleration rates, the acceptance empirical coverage is very close to a. Figure g shows examples of the image-error, the true DISTS, and its quantile upper-bound for each measurement round. With the threshold set at 0.16, the example on the top would collect two rounds of measurements (i.e., R = 8) while the example at the bottom would collect three rounds of measurements (i.e., R = 4), as demarcated by the red squares. See App.
0.16. With the quantile bound, the measurements stop after three of fewer
C for additional qualitative results.
We now repeat the multi-round experiment using the (perhaps more familiar) metric of PSNR. We set the acceptability threshold at 7 == 33.0 dB, which requires the non-adaptive approach to use accoleration R 2 to guarantee - a empirical coverage. Similar to what happened when DISTS was used, Fig. S shows that the quantile bound allows large proportion of the slices to be collected at R > 4. In Tab. we see that this results in an average accepted acceleration rate of R = 4.048, over twice the acceleration achieved with the non-adaptive bound.
The E2E- VarNet takes approximately 104ms to generate a single posterior sample, while the CNF take about 1.22 seconds to generate 32 posterior samples (corresponding to c c= 33) on single NVIDIA V100. The computation time of the metries and bounds on par with the times reported for the FFHQ experiments.
Computation time:
11

---

Published in Transactions on Machine Learning Research (05/2025)
True Image
R 16
R 8
R 4
R = 2
10
0.20 0.18
0.13
0.09
0.04
0.15
0.11
0.06
0.03
0.10
0.5
0.05
0.0
0.00
1.0
0.25
0.20
0.16
0.09
0.15
0.21
0.15
0.10
0.05
0.10
0.5
0.00 0.05
Figure 9: Examples of the multi-round MRI measurement procedure with DISTS at 0.05, 7 0.16, and c 32. Error images at each acceleration R are shown with the quantile bound (orange) and true metric (white). The red box indicates the measurement round at which the bound falls below the threshold T and the measurement procedure concludes.
4.3 Investigation of distribution shift in MRI
As previously mentioned, a general limitation CP methods like Angelopoulos et al. (2022a) is the requirement of exchangeability, which in our case applies to the pairs {(Z,, ((),, J/) This requirement may be violated when there is a distributional shift between the test data (Io. and the calibration data {( M)) which can then cause a distributional shift. between the corresponding FRIQ quantities (Zo, =0) and {(2,, =1)
In the case of MRI, such distributional shifts may arise for various reasons, some of which would be easy to prevent while others would be more difficult. For example, if the CP method was calibrated on knee images, one would not want to immediately test on brain images, but instead recalibrate CP method brain images. Likewise, if the CP method was calibrated with data from manufacturer and/or strength of scanner, then it would be best to test on data from the same manufacturer and/or strength of scanner. Still, due to limited calibration data, situations may arise where a distribution shift is inevitable. Thus, we perform a study to analyze the scnsitivity of our proposed method to distribution shifts.
For this study, we use the validation fold of the nou-eal-suppreseed multicoil fastMRT knee dataset (Zontt et al.] 20188. which contains 100 3D volumes. volume contains all the images collected for a single patient, with each image showing different slice of the knoe (from front to back). To induce realistic yet controllable distribution shift, we choose calibration images from only the center slices of these volumes, and refer to the center slices as "location 1 = O." We then create one test set with images from slice locations = 0, another test set with images from slice location / = 1, and so on, until slice location / = 10 (which typically corresponds to an edge slice). Example images from various slice locations are shown in Fig. 10
We first evaluate the coverage of the quantile bound using T 10 000 Monte Carlo trials, error-rate a 0.1, acceleration f 8, an E2E- VarNet (Sriram et al. 2020a) sample for T;, and 32 posterior samples for u.- For each trial t (I,...,T). we construct the calibration set by randomly sampling 70 of the 100 center slices. For the same the test data at locations 1. > 0 by randomly sampling 30 of the 200 available slices. Figure 12 plots the mean empirical coverage over the T trials as function of test. slice location 1. As expected, the desired a coverage is met when = 0. However, the behavior of the empirical coverage for | >> 0 varies depending on the metrics. The coverage for LPIPS tends to decrease slightly as the slice location / increascs, and the coverage for PSNR only falls below a after I 7. Surprisingly, for the DISTS and SSIM metrics, the coverage remains well above 1 for all slice locations, suggesting the bounds remain valid, but become
form the test data at location / = 0 using the remaining 30 slices, and form
12

---

Published in Transactions on Machine Learning Research (05/2025)
Slice Location 0
Slice Location 5
Slice Location 10
Figure 10: Qualitative examples of images from different slice locations. Slice location 0 indicates the center slice of volume while larger slice locations are further towards the edges of volume.
13

---

Published in Transactions on Machine Learning Research (05/2025)
DISTS
PSNR
Ca braiion
Callbration
noat Loc:
1.0
Test Loc:
nedt Loc:
Rost Loc.
Loc: 10
Rest Loc 10
Density
0.6-
1.0
0.2
00
0.10
Coverage
0.8
LPIPS
SSIM
25
Calbration
Calbbration
Loc
Desired
"oer 10
30-
Rest Loc
Rest Loc 10
0.6
PSNR
Density
LPIPS
15-
10
Slice Location
10
The average empirical cover- age across T = 10000 trials for test sets at different slice locations. All trials are calibrated with images from slice location with 0.1, R 8, and 32.
Figure 12:
-0.10
-0.25
0.00
0.10
2/:
Figure 11: Histograms of the difference between the true FRIQ z4: and the FRIQ estimate 2 for test indices & in the test fold Testt [t] of a single trial. Histograms are shown for test slice locations 0,5, 10. Note the increasing shift in distribution from the calibration set as increases.
slightly over-conservative for 1 > 0. Overall, the results demonstrate our bounds are quite robust to small distributional shifts with only a minor loss in coverage for certain metrics.
To visualize the distribution shift versus test location I, we consider the difference between the true FRIQ 2n and the FRIQ estimate 24 for each test index E Test [t] in single trial t. This difference is 2 2 for LP metrics and & 2 for HP metrics. Figure 11 shows the histogram of this difference for test locations E 50,5, 10). As expected, these histograms deviate more the Lest location / increases, although the amount of deviation depends on the FRIQ metric. For PSNR, we see the histogram shifting slightly to the right, while for SSIM, the histogram starts to shrink in width.
Figure 12 suggests that one could select more conservative a to ensure sufficiently high coverage under small distributional shifts, but at the cost of more conservative bounds, In fact, this is largely the mechanism behind distributionally robust CP extensions like Tibshirani et al. (2019) Barber et al. (2023); Cauchois
et al. 2024).
We leave such generalizations to future work.
5
Conclusion
For imaging inverse problems, we used conformal prediction to construct bounds the FRIQ of a recovered image relative to the unknown true image. When constructed using a calibration set that is statistically exchangeable with the test sample, our bounds are guaranteed to hold with high probability. Two of our perpiiiaate-pssttto-sasaaae
methods leveraged the measurements and reconstruction. Our approaches were demonstrated on image denoising and accelerated multicoil MRI, illustrating the broad applicability of our work.
schemes to yield tighter conformal bounds that adapt to
Limitations
We acknowledge multiple limitations in our proposed methodology. 1) Our methods require access to calibration data {(x,, ) that is similar enough the test data (æo- Mo) for the FRIQ pairs (2,,,,) &) to be modeled as statistically exchangeable. More work required make our methods robust to distribution shift. (see Sec. 4.3), although Tibshirani et al. (2019); Barber et. al. 2023) Cauchois et al. (2024) suggest.
14

---

Published in Transactions on Machine Learning Research (05/2025)
some paths forward. 2) Our methods will be most impactful when there exists evidence that the FRIQ metric is well matched to the application (e.g., DISTS for MRT (Kastryulin et al. 2023 )- For some applications, additional work is required to determine which metrics are more appropriate. 3) Our MRI application ideas are preliminary and not ready for practical use; rigorous clinical trials are needed to tune and validate the methodology on a much larger and diverse cohort of data. 4) The learned adaptive bound from Sec. 3.4 requires training a quantile regression model, and our FFHQ denoising experiment suggests that may not be easy to significantly outperform the simpler adaptive bound from Sec. 3.3 5) The posterior samplers that considered in our numerical experiments target only aleatoric uncertainty, and sharper conformal bounds might be attained with posterior samplers that also target epistemic uncertainty (e.g., Ekmekci & Cetin 2023 ) 6) Because our methods are based on CP (or, equivalently, conformal risk control under the indicator loss (Angelopoulos et al. 2022a)); the marginal guarantee (G) holds with probability 1 - a over random test data (c.g., Zo. Zo) and calibration sets Deal- A more fine-grained coverage could be achieved via the Risk-Controlling Prediction Sets (RCPS) framework from Bates et al. 2021). which employs two user-selected error rates o, E (0, 1) to yield coverage guarantees like
Pr [P Zoo (LDD) (Pa)) 11 a 21-5
(12)
in place of 60 In
12
controls the Dea-conditional error while å controls the error over Deal.
Broader Impact Statement
By providing conformal bounds on the FRIQ of recovered images, we anticipate that our framework will positively impact the field of imaging inverse problems by providing rigorous guarantees on recovery accuracy, Our methods may help to give confidence that recovered images can be trusted, especially in safety-critical applications. However, the marginal coverage guarantee in (6) holds average across random calibration and test data, but not conditionally for given measurement yo and/or calibration set deal. Furthermore, before our methods are ready for practical use, clinical studies with large and diverse datasets must be performed to guide the choice of FRIQ metrics, error rates a, recovery schemes h(-), and conformal bounding strategics.
Acknowledgements
This work was supported in part by the National Institutes of Health under Grant R01-EB029957.
15

---

Published in Transactions on Machine Learning Research (05/2025)
References
Jonas Adler and Ozan Öktem. Deep Bayesian inversion. arXiv:1811.05910, 2018.
Martin Andersen, Joachim Dahl, and Lieven Vandenberghe. CVXOPT, 2023. URL http::t//vxoott.orggg
copyright. html
Anastasios N. Angelopoulos and Stephen Bates, Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494 591, 2023. ISSN 1935-8237. doi: /5261/2200001101.
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lci, and Tal Schuster. Conformal risk control. wrXiv:220.002811 2022a..
Anastasios N. Angelopoulos, Amit P. Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to- to-inage regression with distribution-free uncertainty quantification and applications in imaging. In Proc. Intl. Conf. on Machine Learning, 2022b.
doi: 4445500/aXiivv 220.0.200..
Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb. Solving inverse problems using data-driven models. Acla Numerica, 28:1 18:1-174, June 2019.
Christopher R. S. Banerji, Tapabrata Chakraborti, Chris Harbron, and Ben D. MacArthur. Clinical AI tools must convey predictive uncertainty for each individual patient. Nature Medicine, 29(12):2996-25 2998, 2023. doi: 00.1088//41131--0232222222
Riccardo Barbano, Chen Zhang, Simon Arridge, and Bangti Jin. Quantifying model uncertainty in inverse problems via Bayesian deep gradient descent. In Proc. IEEE Intl. Conf. on Pattern Recognition, pp. 1392-1399, 2021. doi:
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prodiction beyond exchangeability. Annals of Slatistics, 51 (2):)811 845, 2023.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6), 2021. doi: 10.1145/3478535
Omer Belhasin, Yaniv Romano, Daniel Freedman, Ehud Rivlin, and Michael Flad. Principal uncertainty quantification with spatial correlation for image restoration problems. IEEE Trans. on Pattern Analysis and Machine Intelligence, 46:3321 3333, 2023.
Chinmay Belthangady and Loic A Royer. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nature Methods, 16(12):1215-122) 2019.
Matthew Bendel, Rizwan Ahmad, and Philip Schniter. A regularized conditional GAN for posterior sampling in inverse problems. In Proc. Neural Information Processing Systems Conf., 2023.
Sayantan Bhadra, Varun A Kelkar, Frank J Brooks, and Mark A Anastasio. On hallucinations in tomographic image reconstruction. IEFE Trans. on Medical Imaging, 40(11):3249 3260, 2021.
Yochai Blau and Tomer Michaeli. The perception-diitortton tradeoff. Tn Proc. IEFE Conf. on Computer Vision and Pattern Recognition, pp. 6228 6237, 2018.
Nicki Skafte Detlefsenand Jiri Borovec, Justus Schock, Ananya Harsh, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. Torchmetrics, 2022. URL https
7/ httpab/bc.coo..iaaaannna///oecee
Stéphane Caron, Daniel Arnström, Suraj Bonagiri, Antoine Dechaune, Nikolai Flowers, Adam Heins, Takuma Ishikawa, Dustin Kenefake, Giacomo Mazzamuto, Donato Meoli, Brendan O'Donoghue, Adam A. Oppenheimer, Abhishck Pandala, Juan José Quiroz Omaña, Nikitas Rontsis, Paarth Shah, Samuel St-Jean, Nicola Vitucci, Soeren Wolfers, and Fengyu Yang. qpsolvers: Quadratic programming solvers in python, 2024. URI. htttp:://gihhbbb..c/pppovveer./
16

---

Published in Transactions on Machine Learning Research (05/2025)
Maxime Cauchois, Suyash Gupta, Almur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association, pp. 1 1-66, 2024.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proc. Inll. Conf. on Knowledge Discovery and Data Mining, pp. 785-799 195,794, 2016.
Linda C Chu, Anima Anandkumar, Hoo Chang Shin, and Elliot K Fishman. The potential dangers of artificial intelligence for radiology and radiologists. Journal of the American College of Radiology, 17(10): 1309-1311, 2020.
Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hallucinate features in medical image translation. In Proc. Intl. Conf. on Medical Image Compulation and Compuler-Assiseed Intervention, pp. 529-536 2018.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero Simoncelli. Image quality assessment: Unilying structure and
texture similarity. IEEE Trans, on Pattern Anolysis and Machine Intelligence, 44(3):2567-2) 2581, 2020a.
Keyan Ding, Kede Ma, Shiqi Wang, and Ecro P. Simoncelli. DISTS, 2020b. URL https://githu: com/ dingkeyan33/IISTS
Alain Durmus, Eric Moulines, and Marcelo Pereyra. Efficient Bayesian computation by proximal Markov chain Monte Carlo: When Langevin meets Morean. STAM Journal on Imaging Sciences, 11(1):473 -006, 2018.
Vineet Edupuganti, Morteza Mardani, Shreyas Vasanawala, and John Pauly, Uncertainty quantification in doep MRI reconstruction. IEEE Trans. on Medical Imaging, 40(1):233922) 250, Jamuary 2021.
Canberk Ekmekei and Mujdat Cetin. Uncertainty quantification for deep unrolling-based computational imaging, IEEE Trans. on Computational Imaging, 8:1195-1209, 2022. doi: 10.1100/TC1.220223333333
Canberk Ekmekei and Mujdat Cetin. Quantifying generative model uncertainty in posterior sampling methods for computational imaging. In Proc. Neural Information Processing Systems Workshop, 2023.
William Falcon al. Pytorch lightning, 2019. pytorch- lightning
URL hattee//gghhhbb.coo/////ochhittp
Bert E Fristedt and Lawrence F Gray. Modern Approuch to Probability Theory. Springer, 2013.
Nina M Gottschling, Vegard Antun, Anders C Hansen, and Ben Adcock. The troublesome kernel On hallucinations, no free lunches and the accuracy-stability tradc-off in inverse problems. arXiv:2001.0125: 2023.
Kerstin Hammernik, Thomas Küstner, Burhaneddin Yaman, Zhengnan Huang, Daniel Rueckert, Florian Knoll, and Mehmet Akçakaya. Physics-driven deep learning for computational magnetic resonance imaging: Combining physics and machine learning for improved medical imaging. IEEE Signal Processing Magazine, 40(1):98-11) 114, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. Neural Information Processing Systems Conf., volume 33, pp. 6840 6851, 2020.
David P Hoffman, Isaac Slavitt, and Cascy A Fitzpatrick. The promise and peril of deep learning in microscopy. Nature Methods, (8(2):131-133) 2021.
Eliahu Horwitz and Yedid Hoshen. Conffusion: Confidence intervals for diffusion models. ar 2rXiv.2211.0979. 2022. doi: 10.48550/ arXiv.2211.0977
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alex Dimakis, and Jonathan Tamir. Robust compressod sensing MRI with deep generative priors. In Proc. Newral Information Processing Systems Conf., 2021.
Mihir Joshi, Aaron Pruitt, Chong Chen, Yingmin Liu, and Rizwan Ahmad. Technical report (v1.0) random cartesian sampling for dynamic MRI. arXin:2206.036% 2022.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. Iril. Conf. on Learning Represenlations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style- based generator architecture for generative adversarial networks. In Proc. TEEE Conf. on Computer Vision and Pattern Recognation, pp. 4396-4405, 2019.
Sergey Kastryulin, Jamil Zakirov, Nicola Pezzotti, and Dmitry V Dylov. Image quality assessment for magnetic resonance imaging. IEEE Access, 11:14154 14168, 2023.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Proc. Neural Information Processing Systems Conf., 2022a.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Downloaded from http:////gctubb..oooo//aattttaat
May 2022b.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Proc. Neural Information Processing Systems Conf., 2017.
Florian Knoll, Kerstin Hammernik, Chi Zhang, Steen Moeller, Thomas Pock, Daniel K Sodickson, and Mehmet Akcakaya. Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues. IEFE Signal Processing Magazine, 37(1):128-140, January 2020.
Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica, 46(1), 1978. doi: 10. 10.2307 10.2307/1913643 1913643.
Gilad Kutiel, Regev Cohen, Michael Elad, Danicl Froedman, and Ehud Rivlin. Conformal prediction masks: Visualizing uncertainty in medical imaging. In Proc. Intl. Conf. on Learning Representations, 2023.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: When Langevin meets Tweedie. SIAM Journal on Imaging Sciences, (52))7711-737 2022.
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Stalistical Sociely, 76, 2014. doi: 10.1111/rssb.2022
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 2018
Weisi Lin and C-C Jay Kuo. Perceptual visual quality metries: A survey. Journal of Visual Communication and Image Representation, 22(4):297-38) 312, 2011.
Grace W Lindsay. Convolutional neural networks as model of the visual system: Past, present, and future. Journal of Cognitive Neuroscience, 23(10))201722033 2021.
Matthew Muckley, Bruno Riemenschneider, Alireza Radmanesh, Sunwoo Kim, Geunu Jeong, Jingyu Ko, Yohan Jun, Hyungseob Shin, Dosik Hwang, Mahmoud Mostapha, al. Results of the 2020 fastMRI challenge for machine learning MR image reconstruction. JEEE Trans. Medical Imaging, 40(9):2306 2317, 2021.
Dominik Narnhofer, Alexander Effladd, Erich Kobler, Kerstin Hammernik, Florian Knoll, and Thomas Pock. Bayesian uncertainty estimation of learned variational MRI reconstruction. IEEE Trans. on Medical Imaging, 41(2):279 291, 2022.
Dominik Narnhofer, Andreas Habring, Martin Holler, and Thomas Pock. Posterior-variance--esr error quantification for inverse problems in imaging. SIAM Journal on Imaging Sciences, 17(1):301 333, 2024.
Harris Papadopoulos, Kostas Procdrou, Volodya Vork, and Alex Gammerman. Inductive confidence machines for regression. In Proc. European Conf. on Machine Learning, pp. 345 245-356, 2002. doi:
10.1007/3-540-36765-11 29.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca. Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, peh-performance deep learning library. In Proc. Neural Information Processing Systems Conf., pp. 8024 8035, 2019.
Peter B Roemer, William A Edelstein, Cecil E Hayes, Steven P Souza, and Otward M Mueller. The NMR phased array. Magnetic Resonance in Medicine, 16(2):192-25) 225, 1990.
Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Proc. Neural Information Processing Systems Conf. pp. 3544-3553, 2019. doi: 10.48550/arXiv.nin 10.......110002...
Swami Sankaranarayanan, Anastasios N. Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola. Semantic uncertainty intervals for disentangled latent spaces. In Proc. Neural Information Processing Systems Conf., 2022. doi: 10.48550 wtt.:.2207.10077..
Amuroop Sriram, Jure Zbontar, Tullic Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated MRI reconstruction. In Proc. Intll Conf- on Medical Image Compulation and Compuler-Assisted Intervention, pp. 64-73, 2020a.
Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated MRT reconstruction. 2020b.
Rhea Sanjay Sukthanker, Zhiwu Huang, Suryansh Kumar, Radu Timofte, and Lue Van Gool. Generative flows with invertible attentions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2022. doi: 11.48550/arXiv...2 21....0956.9
He Sun and Katherine L Bouman. Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging. In Proc. AAAI Conf. Artificial Intelligence, volume 35, pp. 1192--2631 2021.
Jacopo Teneggi, Matthew Tivnan, J. Webster Stayman, and Jeremias Sulam. How to trust your diffusion model: A convex optimization approach to conformal risk control, 2023.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In Proc. Neural Information Processing Systems Conf., volume 32, 2019.
Matthew Tivnan, Siyeop Yoon, Zhennong Chen, Xiang Li, Dufan Wu, and Quanzheng Li. Hallucination index: An image quality metric for generative reconstruction models. In Proc. Intl. Conf. on Medical Image Compulation and Cempuler-Assisled Intervention, pp. 449-458, 2024.
Francesco Tonolini, Jack Radford, Alex Turpin, Daniele Faccio, and Roderick Murray-Smith. Variational inference for computational imaging inverse problems. Journal of Machine Learning Research, 21(179): 1 46, 2020.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, New York, 2005.
Zhon Wang. Applications of objective image quality asscssment methods. IEEE Signal Processing Magazine, 28(6):137 142, 2011.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Truns. on Image Processing, (3(4):000 612, April 2004.
Jeffrey Wen, Rizwan Ahmad, and Philip Schniter. A conditional normalizing flow for accelerated multi-coil MR imaging. In Proc. Inll. Conf. on Machine Learning, 2023a.
Jeffrey Wen, Rizwan Ahmad, and Philip Schniter. MRI CNF, 2023b. URL httpp:://gtthbb.comm/yeee/
mri cnf
19

---

Published in Transactions on Machine Learning Research (05/2025)
Jeffrey Wen, Rizwan Ahmad, and Philip Schniter. Task-driven uncertainty quantification in inverse problems via conformal prediction. arXic:2405. 18527, 2024.
Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue, and Katherine L Bouman. Principled
probabilistic imaging using diffusion models plug-and-play priors. 188882,55.8888, 2024.
Yujia Xue, Shiyi Cheng, Yunzhe Li, and Lei Tian. Reliable deep-learning- phase imaging with uncertainty quantification. Optica, 6(5), 2019. ISSN 2334-2536. doi:
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356 365, 2016.
Martin Zach, Florian Knoll, and Thomas Pock. Stable deep MRI reconstruction using generative priors. IEEE Trans. Medical Imaging, 2023.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal, Adriana Romero, Michael Rabbat, Pascal Vincent, James Pinkerton, Duo Wang, Nafissa Yakubova, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: An open dataset and benchmarks for accelerated MRI. 1Xi::1811008833 2018.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Mean empirical coverage for the quantile method with a = 0.05 and T = 100000 on the FFHQ denoising task (+ standard error)
C
DISTS
LPIPS
PSNR
SSIM
0.95002 = 0.00009
0.94997 = 0.00009
0.95013 + 0.00009
0.94989 + 0.00009
2
0.95006 = 0.00009
0.95003 = 0.00009
0.95001 + 0.00009
0.95022 + 0.00009
4
0.94997 = 0.00009
0.95008 = 0.00009
0.94986 + 0.00009
0.94999 + 0.00009
8
0.95020 + 0.00009
0.95015 + 0.00009
0.95019 + 0.00009
0.94991 + 0.00009
16
0.94998 + 0.00009
0.94999
0.00009
0.95009
0.00009
0.95008 + 0.00009
32
0.95002 0. 0.00009
0.95013 0.00009
0.95003 0.00009
0.95006 + 0.00009
Table 5: Mean empirical coverage for the regression method with 0.05 and T 10000 on the FFHQ denoising task (+ standard error)
DISTS
LPIPS
PSNR
SSIM
0.94994 + 0.00009
0.94970
0.00009
0.95009 + 0.00009
0.95014 + 0.00009
0.95011
0.00009
0.94953
0.00009
0.94985
0.00009
0.95004 + 0.00009
0.94996
0.00009
0.94946
0.00009
0.95003
0.00009
0.94995 0.00009
0.95004 + 0.00009
0.94964
0.00009
0.94999 + 0.00009
0.95017 + 0.00009
16
0,94986 = 0.00009
0.94964 = 0.00009
0.95007 = 0.00009
0.94987 = 0.00009
32
0.95013 + 0.00009
0.95026 + 0.00009
0.95001 + 0.00009
0.95006 + 0.00009
A Empirical coverage
we empirically demonstrated that the coverage guarantees in (6) are met for the non-adaptive, quantile, and regression bounds in the FFHQ denoising experiments. Here, we further demonstrate that these guarantees hold regardless of the number of posterior samples used to compute the adaptive bounds. Tables 4 and 5 show the average empirical coverage for the quantile and regression method, respectively, across T = 100 trials for different values of e and a = 0.05. The same number of posterior samples e is used during calibration and to compute the adaptive bounds during testing. Again, we observe that the average empirical coverage is very close to the desired 1 in all cases though there are very slight deviations as a result of finite trials, number of calibration samples, and number of testing samples.
In Sec. 4.1
In Table 6 we report the mean empirical coverage for the quantile method in the MRI experiments with = 0.05, 32, and acceleration rate R E (2,4, 8, 16) across T = 10000 trials. For any value of R, we see the empirical coverage is very close to the theoretical 1 - a = 0.95 coverage; thus, once again, our method shows close compliance to the theory.
B Additional FFHQ denoising experiments
Effect of training and calibration set size: For FFHQ denoising, we now investigate how the amount of training and calibration data affect the mean conformal bound. Following the same Monte Carlo procedure as Sec. 4.1 we fix the number of testing samples 900 but change the proportion of Atrain versus Teal for the remaining 3100 samples. For the non-adaptive and quantile bounds, the training samples are unused. In Fig. show the mean conformal bounds as the proportion of training samples varies, starting with 0.1 and going up to 0.95, for T = 100000 e = 32, and = 0.05. Both adaptive methods still provide noticcable gains over the non-adaptive bound. Even with additional training samples, however, the regression bounds show relatively little improvement over the quantile bounds. Based on (3), the conformal bounds should grow more consorvative as the number of calibration points decreases for the non-adaptive and quantile bounds. However, this effect is not evident until very small calibration set sizes (c.g., when the fraction of calibration samples is 0.05).
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Mean empirical coverage for the quantile method across accelerations with a = 0.05, c= 32, and T 10000 on the accelerated MRI task (+ standard error). All coverages are above the expected coverage of a 0.95
R
DISTS
LPIPS
PSNR
SSIM
2
0.9503 0.0001
0.9503 + 0.0001
0.9504 + 0.0001
0.9504
0.0001
4
0.9504 0.0001
0.9503 = 0.0001
0.9505 = 0.0001
0.9504
0.0001
8
0.9503 0.0001
0.9504 + 0.0001
0.9503 = 0.0001
0.9503 = 0.0001
16
0.9504 + 0.0001
0.9504 + 0.0001
0.9505 = 0.0001
0.9506 + 0.0001
non-adaptive bound
quantile bound
regression bound
DISTS
LPIPS
PSNR
S5/M 1
Mean Conformal Bound
0.36
20.0
0.50
0.54
0.35
19.5
19.0
0.33
18.5
0.46
0.32
18.0
0.1
0.3
0.5
0.7
0.9
0.1
0.5
0.7
0.9
0.1
0.5
0.7
0.9
0.3
0.5
0.7
0.9
Fraction of Training Samples
Fraction of Training Samples
Fraction of Training Samples
Fraction Training Samples
Figure 13: Mean conformal bound versus the proportion of training samples for FFHQ denoising with
Ttrrin naal = 3300 samples.
Correlation between conformal bound and true FRIQ: Figure visually demonstrates that the quantile bound tracks the true FRIQ much better than the non-adaptive bound. To quantify this tracking behavior, we compute the Pearson correlation coefficient between each conformal bound ( ((dailtt) and the true FRIQ 2* over the test indices E [est[t] for each Monte-Carlo trial t. Fig. 14 we plot the mean (across = 10000 trials) Pearson correlation coefficient versus с for each bound. Since the non-adaptive bound constant with žk, its correlation equals O. However, the two adaptive approaches demonstrate correlation coefficient above 0.5, and up to 0.7, depending on the metric. These correlation coefficients quantify the adaptivity of our bounds and explain, in part, why the adaptive bounds led to better average acceleration rates than the non-adaptive bound in the multi-round measurement experiment of Sec. 4.2
Performance across different error rates a: In Fig. 15 we plot the mean empirical coverage for different values of a across T 10000 trials. Results are shown for each bound on the LPIPS metric with c c= 32. As expected, the empirical coverage for all three bounds very close to the desired - a for all values of a, further demonstrating adherence conformal prediction thcory. For the same experimental sctup, we plot the mean absolute difference between the conformal bound and true image quality against the desired coverage 1 in Fig. 16 Once again, see the quantile and regression bounds give tighter bounds on average compared to the non-adaptive bounds. The tightest bounds for all three methods are found when = 0.5 since the conditional median is known to be the minimum mean absolute error (MMAE) estimator.
C
Additional MRI experiments
Effect of number of posterior samples e in conformal bound: For the case of FFHQ denoising, Sec. demonstrated the number of posterior samples has limited effect on the conformal bounds for the FFHQ experiments. We now investigate whether the same occurs with MRI. Figure 17 plots the percent improvement in MCB as e increases relative to the MCB for e 1. From the figure, we see less than 1.5% improvement over e = 1 for any metric, suggesting that the quantile method is indeed robust to the choice of с for both experiments.
Multi-round measurement samples: In Fig. 18 we show the fill-fille measurement, recovered image, and absolute-error map each acceleration rate. The conformal bound is imposed the reconstructions for
22

---

Published in Transactions on Machine Learning Research (05/2025)
non-adaptive bound
quantile bound
regression bound
DISTS
LPIPS
PSNR
SSIM
0.8
Correlation Coeficiemt
0.6
6
0.6
0.4
0.4
0.2
0.
0.2
0.0
0.0
12
8
16
32
12
16
32
12
8
16
32
12
16
32
Number of Image camples
Number of Image Samples
Number Image Samples
Number of Image Samples
Figure 14: Mean Pearson correlation coefficient between each conformal bound and the true FRIQ versus the number of posterior samples for FFHQ denoising.
Mean Empirical Coverage
non-adaptive
Mean Absolute Difference
non-adaptive
0.8
quantile
0.08
quantile regression
regression
0.07
0.6
desired
0.06
0.4
0.05
0.2
0.04
0.2
0.4
0.6
0.8
0.03
0.2
0.4
0.6
0.8
Desired Coverage 1 (
Desired Coverage - a
Figure 15: The average empirical coverage across T = 10000 trials for different values of 1 - Results are shown for LPIPS with c = 32.
Figure 16: The mean absolute difference between the conformal bound and true image quality for varying
1 - a. Results are shown for LPIPS with e e= 32.
the case when a = 0.05, == 0.16, and c = 32, Following the multi-round measurement protocol described in Sec. 4.2. the reconstruction R 8 (marked red) would be docmed sufficient (B < T), and the measurement collection would end.
D MRI reconstruction from CNF
In the MRT experiments Sec. 4.2 we used the F2F VarNet. from Sriram et al. (2020a) for MRT image recovery and the CNF from Wen et al. (2023a)
adaptive bound computation. In this scction, we investigate what happens when the CNF from Wen al. 2023a is also used for image recovery.
When using an approximate posterior sampler for image recovery, one has the option of averaging p 2 posterior samples. For example, when one is interested in estimating H; from Vi with high PSNR, equivalently low MSE, it makes sense lo set. G; as the minimum MSE (MMSE) or conditional-mean estimate E(X,(Y;y: The MMSE estimate can be approximated by the empirical mean
ctp
(13)
j=c+1
with large p. The indices on j in 13
chosen to avoid the samples (5(") S- used for the adaptive bounds.
23

---

Published in Transactions on Machine Learning Research (05/2025)
R-16
R-8
R-4
R-2
0.8
1.25
% Improvement over
0.6
0.4
1.00
0.4-
0.4
0.75
0.50
dists
plxc
0.2
1pgs
tdlps
pls
2577
ppnr
psr
0.25
p5or
0.0
3mm
0.0
sim
0.0
eim
0.00
5mm
16
12
12
Number of Image Samples,
Number of Image Samples,
Number Image Samples,
Number of Image Samples,
Figure 17: Percent improvement in MCB versus number of samples used in the quantile bound for the accelerated MRI experiments.
Table 7: Average reconstruction performance on the fastMRI Zbootar et al. 2018 knce validation set for
R = 16 (+ standard error)
Network
DISTS +
LPIPS 4
PSNR f
SSIM 1
E2E- VarNet
0.209 0.001
0.354 + 0.001
30.301
0.043
0.807 0.001
CNF (p- 1)
0.183 0.001
0.312 = 0.001
28.244 + 0.039
0.688 + 0.002
CNF (p- 2)
0.167 = 0.001
0.292 = 0.001
29.091 + 0.039
0.730 = 0.001
CNF (p 4)
0.165 + 0.001
0.287 + 0.001
29.588 + 0.039
0.755 + 0.001
CNF (0 8)
0.173 + 0.001
0.296 + 0.001
29.862 + 0.039
0.770 + 0.001
CNF (p = 16)
0.184 0.001
0.314 0.001
30.006 + 0.039
0.777 + 0.001
CNF (p D032)
0.193 = 0.001
0.333 = 0.001
30.080 + 0.039
0.781 = 0.001
Table 8: Average reconstruction performance on the fastMRI
Zbontar al.
2018
knee validation set for
R (= standard error)
Network
DISTS 4
LPIPS 4
PSNR f
SSTM +
E2E- VarNet
0.151 + 0.001
0.262 + 0.001
33.459 + 0.047
0.864 + 0.001
CNF (p 1)
0.136 + 0.000
0.248 = 0.001
30.796 + 0.044
0.761 + 0.002
CNF (p- 2)
0.118 + 0.000
0.225 + 0.001
31.754 + 0.044
0.799 + 0.001
CNF (p- = 4)
0.000
0 0.001
32.329 + 0.043
0.821 = 0.001
CNF (p- 8)
0.128
0.228 = 0.001
32.650 + 0.043
0.834 0.001
CNF (6)
0.138
0.243
32.819
0.840 0.001
CNF (2)
0.145
0.255 + 0.001
32.907 + 0.043
0.843 + 0.001
However, because the MMSE estimate can look unrealistically smooth, one may instead be interested in producing an image estimate with good SSTM, DISTS, or LPPPS performance. In this case, smaller values of may be more appropriate. For example, Bendel et al. (2023) found that, for multicoil brain MRI at acceleration = with their conditional GAN, the best choices were p p= 8 for SSIM and p = 2 for both DISTS and LPIPS. This can explained by the percept fon-distortion tradeoff (Blau & Michaeli 2018 which says that, as the MSE distortion improves, the perceptual quality tends to decrease. In the end, cach FRIQ metric prefers a particular tradeoff between perceptual quality and distortion, and thus particular p when sample averaging. In the sequel, we consider different choices for p when using the CNF for image recovery.
Average reconstruction performance: To get sense for how the CNF compares to the E2E-VarNet, and for how the choice of affects the CNF, we report average PSNR, SSIM, DISTS, and LPIPS on the con-fat-suppreseed subset of the fastMRI knee validation set at acceleration rates = 16, 8, 4, and 2 Tables and 10 respoctively. There we see that the E2E-VarNet outperforms the CNF PSNR and SSIM all accelerations, while the CNF outperforms the E2E- VarNet in DISTS and LPIPS in all cases other than LPIPS at R 2.
24

---

Published in Transactions on Machine Learning Research (05/2025)
True Image
Measurement
Recovery
Error-Map
10
0.20
0.20
021
0.18
0.18
o0o0
R
=
0.5
16
000
0000
0.0
0.13
0.13
013
0.11
0.11
o.10
R
=
8
0005
00'0
0
600
0.09
0.06
0.06
0.10
R
=
4
00'5
00'0
0.00
0.04
ST
0.03
0.03
o1'0
R
R
2
000
00'0
Figure 18: Qualitative example of the multi-round MRI experiment with DISTS at a = 0.05, -= 0.16, and c= The measurement, recovery, and absolute error are shown for all accelerations. The quantile bound (orange) and true DISTS (white) are imposed on the reconstructions. The red box indicates the accepted reconstruction where the bound first falls below the threshold T.
Empirical Coverage: We first investigate empirical coverage using the same Monte-Carle validation procodure described in Sec. 4.2 again with 10000 trials. Table II reports the average empirical coverage 8. As with all previous experiments, the coverage is
for the quantile bounds with different choices of above and very close the desired - value for all choices of p.
Effect of acceloration rate R and choice of recovery method: Figure 19 plots the mean conformal bound (MCB) of the quantile method with c = 32 versus the acceleration rate R for different image isates 2, The image estimate computed using either the E2E- VarNet point estimate or a p-sample average of p. In all cases, the MCB improves as the acceleration R decreases, expected. But, as discussed above, each metric benefits from different choice of p. DISTS and LPIPS prefer p (2,4) while PSNR and SSIM prefer p 32. The figure also shows that the MCB for the p-optimized
from the CNF with different values
25

---

Published in Transactions on Machine Learning Research (05/2025)
Table 9: Average reconstruction performance on the fastMRI Zbontar et al.
2018
knee validation set for
R (= standard error)
Network
DISTS +
LPIPS +
PSNR f
SSIM 1
E2E- VarNet
0.110 0.001
0.181 = 0.001
36.030 = 0.053
0.905 = 0.001
CNF (p- 1)
0.100 0.000
0.191 = 0.001
33.090 = 0.048
0.826 + 0.001
CNF (p 2)
0.087 = 0.000
0.170 = 0.001
34.073 + 0.048
0.856 + 0.001
CNF (p 4)
0.090 0.000
0.166 + 0.001
34.666 + 0.048
0.873 + 0.001
CNF (p= 8)
0.171 0.001
$4.998 0.047
0.882 = 0.001
CNF (6=16)
0.106 0.001
0.178 = 0.001
35.174 0.047
0.887 = 0.001
CNF (2=32)
0.110
0.184 = 0.001
35.265 + 0.047
0.889 0.001
Table 10: Average reconstruction performance on the fastMRI
Zbontar et al.
2018 knee validation set for
R = 2 (= standard error)
Network
DISTS +
LPIPS 4
PSNR f
SSTM +
E2E- VarNet
0.059 0.000
0.094 + 0.001
39.692
0.060
0.947 + 0.001
CNF (p 1)
0.059 0.000
0.118 = 0.000
36.810 + 0.054
0.907 = 0.001
CNF (p- 2)
0.054 0.000
0.105 + 0.000
37.667 + 0.054
0.923 + 0.001
CNF (p- 4)
0.055 = 0.000
0.100 = 0.000
38.171 + 0.054
0.931 = 0.001
CNF (p 8)
0.058 + 0.000
0.099 = 0.000
38.448 + 0.054
0.935 + 0.001
CNF (p (p=16)
0.060 + 0.000
0.099 = 0.000
38.593 + 0.054
0.937 + 0.001
CNF (p- (D=32)
0.061 + 0.000
0.099 + 0.000
38.668 + 0.054
0.939 + 0.001
CNF-based method is better than the MCB for the E2E- Saee-aareetaaaaae method with both DISTS and LPIPS but not with PSNR and SSTM. Thus, the recovery method that yields the tightest bounds may depend on the metric of interest.
Multi-round MRI: Since Figure 19 reveals the tightest bounds on DISTS are obtained when the CNF posterior average with 4 is used for the image estimate 2, we repeat the multi-round experiment from Sec. 4.2 with this sctup. As before, we set a = 0.05 and c= 32. The DISTS acceptance threshold is set at T = 0.11, where the non-adaptive approach requires of slices that were collected each acceleration rate. Here, the aon-adaptive approach always accepts slices at R = 2 while the quantile bound accepts nearly 50% of the slices at R = 4. Table 12 shows that this equates to average accepted acceleration rate of 2.596, with empirical coverage of 0.9434 acceptance. This demonstrates that, when considering multi-round recovery performance, the relative performance of different bounding strategies may depend on which image recovery model is used. In any case, the modularity of our proposed framework allows one to improve multi-round performance through the choice of the recovery model, the posterior sampler, and/or for the conformal bound.
for acceptance. In Figure 20 we plot the number
Distribution Shift: We repeat the distribution shift analysis from Sec. 4.3 using single CNF posterior sample as the image estimate E, i.e p= with 0.1, 32, and R 8. As before, we visualize the distribution shift by looking at the histograms of the difference between the true FRIQ 2 and FRIQ estimate 2 for each test index & E [te as the test location / inercases. Figure 21 shows the histograms for test locations / E {0,5 5, 10). It shows more distinct distributional shifts compared to Sec. 4.3 as increases, the PSNR histogram noticeably shifts to the right and widens, while the histogram for LPIPS becomes bimodal. Not surprisingly, these more dramatic shifts lead to decrease in coverage for all metrics in Fig. 22 as increases. We do note, however, that both SSIM and PSNR retain coverage at or above 1 a until 4, demonstrating a certain level of robustness.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 11: Mean empirical coverage for the quantile method with a = 0.05, c= 32, and T = 10000 on the R accelerated MRI task (+ standard error). All coverages are above the expected coverage of 1 a 0.95
P
DISTS
LPIPS
PSNR
SSIM
1
0.9503
0.0001
0.9503 + 0.0001
0.9505 + 0.0001
0.9504
0.0001
2
0.9505 0.0001
0.9503 + 0.0001
0.9504 + 0.0001
0.9505
0.0001
4
0.9503 0.0001
0.9503 = 0.0001
0.9505 = 0.0001
0.9504
0.0001
8
0.9505 = 0.0001
0.9504 + 0.0001
0.9504 = 0.0001
0.9505 = 0.0001
16
0.9505 + 0.0001
0.9502 + 0.0001
0.9504 = 0.0001
0.9504 + 0.0001
32
0.9504 + 0.0001
0.9506 + 0.0001
0.9504 + 0.0001
0.9505 + 0.0001
p
p
p
p
E2E-VarNet
LPIPS
SSIM
Mcan Confermal Bound
0.35
37.5
0.90
0.20-
0.30
35.0
0.80
0.15
32
0.75
0.20
0.10
0.70
0.15
27.5
0.65
2
16
16
Acceleration
Acceleration
Acceleration
Acceleration
Figure 19: Mean conformal bound versus acceleaation R for accelerated MRI. Results shown for the quantile bound with & computed from the E2E- Var point estimate (shown in pink) or the p-sample average from CNF posteriors. Various p shown.
E
MRI subsampling mask details
For the MRI experiments, we simulate the collection of measurements at four acceleration rates R = 1,6,8,4422 2]. These measurements are collected in the 2D spatial frequency domain known as k-space, and the pattern with which these samples are collected is called a sampling mask.
For this study, we use Cartesian sampling procedure where full lines of the 2D k-space are collected progressively. Starting with R = 16, we utilize Golden Ratio Offsee (GRO) (Joshi et al. 2022) sampling mask with GRO-specific parameters == 15 and 8. This gives fully sampled region of 9 lines in the center of k-space known as the autocalibration signal (ACS) region. To simulate the iterative collection of measurements, we build upon this mask for R 8. We first collect central lines to obtain an ACS region of 16 lines before sampling additional k-space lines with a sampling probability inversely proportional to the distance from the center. Additional lines are collected until the desired acceleration R = 8 is met. This procedure is repeated for = 4 and = 2 acquire masks with ACS widths of 24 and 32, respectively. Fig. 23 illustrates examples of the resulting masks.
F Training Modell details
For the regression bound in Sec. 3.4 where us Re, we use quantile predictor of the form
fuu; 0) = (uu)) T w + b with 0 = [w. b]
(14)
where (-)) is a linear spline with two knots, t and f2, implemented via the truncated power basis
v(u)) = [ug (w: 1:1)4: (w: t2])+]ERR
(15)
with 1 the c-dimensional vector of ones and (x) A max (x, 0). The two knots were placed al the and empirical quantiles of the mean training feature ( E; 2(1 tost li=natt, respectively. Essentially, for each feature in ui, (14) implements a piece- wisel regression function with three distinct pieces. To promote
27

---

Published in Transactions on Machine Learning Research (05/2025)
non-adaptive
quantile
1.0
Fraction Accepted
0.8
Table 12: Average results for a multi-round MRI simulation where measurement collection stop once bounds are below a user-set threshold т. Results shown for T 10000 trials using the DISIS metric with 0.05, 0.11, p 4, and c 32 (# stan- dard error).
0.6
0.4
0.2
Method
Average Acceleration
Acceptance Empirical
0.0
16
8
2
Acceleration, R
Coverage
Non-adaptive Quantile
2.000 + 0.000
0.9504 0.0001
Figure 20: Fraction of accepled slices versus fi- nal acceleration rate for multi-round MRI using DISTS. Both methods use a p-postcrior average for the recovery with p = 4, c 32, a 0.05, and 0.11. Error bars show standard deviation.
2.596 1 0.001
0.9434 + 0.0001
DISTS
PSNR
Calbbattion
Calbration
30
Test Loc
Test Loc:
1.0-
Text Loc: 3
Test Loc:
Tex: Loc: 10
Taet Lot: 10
Density
0.6-
1.0
0.2-
-2
Coverage
0.6
LPIPS
SSIM
Desired
35-
Loci
DISTS
20-
30-
Loc:
Lot:
0.2
SSIM
PSNR
Density
LPIPS
20-
10-
15-
Slice Location
Figure 22: The average empirical cover- age across = 10000 trials for test sets at different slice locations. All trials are calibrated with images from slice location 0 with 0.1, R = p 1, and 32.
-0.05
0.50
0.05
0.10
0.15
0.20
0.25
0.30
26.00
Figure 21: Histograms the difference between the true FRIQ 2 and the FRIQ estimate & for test samples in the test fold of single trial. Histograms are shown for test. slice locations = 5,5, 10. Note the increasing shift in distribution from the calibration set as increases.
consistency in u, (2(1) ((2)
across different i, the spline function Φ(-) first sorts the values
(2("))z_ within each For p(0) in 10 we use ridge regularization on the weights w. The resulting (10 is a quadratic program, which can be optimized using any convex solver. To tune the regularization weight 7, we use K-fold cross validation with K folds and select the weight that provides the lowest mean pinball
loss across the 5 folds.
For DDRM, we use the author's implementation KKwar al. 2022b), which is publicly available under an MIT license.
28

---

Published in Transactions on Machine Learning Research (05/2025)
R R=1 16
R = 8
4=4
R R=
Figure 23: MRT sampling masks k-space for each acceleatiin rate R. White pixels indicate the measurement was collected for that location in k-space. The masks are designed in a nested fashion where each mask contains all the measurements of higher R.
Both fastMRI reconstruction models were trained with all four acceleration rates. For each sample in epoch, one of the four sampling masks is randomly drawn, allowing the model to each sample at different acceleration throughout the training.
With the E2E- VarNet, we use the author's codebase (Sriram et al. 2020b), which is released under an MIT license. For training, we utilize the default hyperparameters provided by the authors for the model on the lastMRT knee leaderboard. The model was trained for 50 epochs with a batch size of 16 and learning rate of 0.0001 using SSLM (Wang al. 2004 as the loss function. This takes around 38 hours on single NVIDIA V100 with 32GB of memory.
To train the CNF, start with the author's implementation (Wnn et al. 2023b) that is available under an MIT license. We modify the architecture slightly in order to better handle multiple accelerations. First, we include an invertible atlention module, iMAP (Sukthanker et al., 2022). to the end of the base flow step. Then, we increase the number of initial channels in the conditioning network to 256, Using layers and 10 flows steps each layer, we train the CNF to minimize the negative log- objective. The model is trained for 150 epochs with batch size 8 and learning rate 0.0001. On a single NVIDIA V100, this takes around 335 hours.
To compute the quadratic program for Sec. 4.1 we use the qpsolver CCaron et al.
2024 package under a
LGPL 3.0 license along with the CVXOPT Andersen et al. 2023) package under a GNU General Public License.
We use the TorchMetrics (Borovec et al. 2022 package under the Apache 2.0 license to compute PSNR, SSIM, and LPIPS. We use the author's code (Ding et al.) (020b)) for DISTS under a MIT license. For multicoil MRT, we first compute the magnitude images using the "root-sum-of-squures" (RSS) (Roemer al. 1990p before computing any metric. Since DISTS and LPIPS require 3-channel image, we repeat the magnitude image for all three channels and normalize the values to be between 0 and 1 before computing either metric.
All models use the PyTorch Paszke et al. 2019h framework with a custom license allowing open use. The E2F- VarNet and CNF are implemented using PyTorch Lightning (Falcon al. 2019) under an Apache 2.0 license.
G Datasets
The Flick--Faces-HC (FFHQ) KKrras al. 2019 is publicly available under the Creative Commons BY-NC- SA 4.0 license. The fastMRI (Zbontar et al. 2018) datasets is available under royalty-free license for internal research and educational purposes by the NYU fastMRI initiative. The providers have deidentified and manually inspected images and metadata for protected health information (PHI) as part of an IRB-approved study.
29


Reviews:
[{"review_id": "HZ2Nf03R0N", "reviewer": "Reviewer_ydvr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths:***\n* Writing is clear and easy to follow.\n* The application of split conformal prediction framework to image similarity metrics is quite an interesting and important direction.\n\n***\n***Weaknesses:***\n* The practical utility of the current method is not clear beyond rigorous guarantees.\n* Please see the requested changes section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, authors combine conformal prediction with approximate posterior sampling to construct bounds on full-reference image-quality (FRIQ) metrics that are guaranteed to hold up to a user-specified error probability. Authors demonstrate their technique on image denoising (on FFHQ dataset) and accelerated magnetic resonance imaging (MRI) (on fastMRI knee dataset) problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "***Questions/Proposed changes critical for recommendation for acceptance:***\n\n* In the introduction section, authors mention that \"...Although these uncertainty maps can be visually interesting, they do not quantify uncertainty on multi-pixel structures of interest, such as hallucinations or anatomical features relevant to medical diagnosis (e.g., tumors).\". While I agree with the authors, it is not clear to me how bounds on FRIQ remedy these problems. That is, what is the advantage of rigorous bounds on FRIQ rather than pixel-level set predictions?\n\n***\n***Questions/Proposed changes that would strengthen the work:***\n\n* In Table 2, why is the empirical coverage of the quantile method is not close to 0.95?\n* For the image denoising section, I believe that one DDRM sample results in bad quality reconstructions visually. For instance, reconstructions in Figure 3 are not great. Ideally, a better pipeline could demonstrate the efficacy and the practical utility of the model much better. I recognize that running this would take more time.\n* Would it be possible to obtain rigorous upper and lower bounds *together* rather than one-sided bounds?\n* Do you have ablations on the size of the calibration set $d_{cal}$? \n* Could you provide some intuition or explanation for why the correlation of bounds with the true metrics appears to be weaker for SSIM compared to the other metrics (e.g. in Figure 2)?\n* Although there is no reason for it to not work, demonstrating the method for different choices of $\\alpha$ would make the paper more convincing to the reader.", "broader_impact_concerns": "No ethical concerns that are not addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6DlNulAhft", "reviewer": "Reviewer_8ViD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper introduces a method to bound full-reference image quality (FRIQ) metrics without access to the ground truth. This approach is particularly valuable for safety-critical applications, such as MRI reconstruction. The introduction of posterior sampling is a unique contribution that provides adaptive and probabilistically valid bounds on FRIQ metrics. While the focus is on MRI, the approach could be extended to other imaging inverse problems, including CT, PET, and super-resolution imaging.\n\nWeaknesses: The method assumes access to a paired calibration dataset, where the ground-truth image is known. This requirement limits the applicability to scenarios where no paired dataset is available. While the approach is tested on simulated undersampling, its clinical impact remains untested. Real-world invivo MRI acquisitions could violate the exchangeability assumption, as it's more complicated than the simple retrospective undersampling. Pathological variations (e.g., normal vs. diseased scans) can also lead to non-exchangeable calibration and test sets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach for quantifying uncertainty in full-reference image quality (FRIQ) metrics for imaging inverse problems, particularly in applications such as MRI reconstruction and image denoising. The paper introduces a method to compute probabilistic bounds on FRIQ scores (e.g., PSNR, SSIM, LPIPS, DISTS) without requiring access to the ground-truth image. The method leverages approximate posterior sampling to refine the conformal bounds, enabling the construction of adaptive FRIQ estimates. The framework is tested on the fastMRI knee dataset using multiple acceleration rates (R = 2,4,8,16).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarify the Exchangeability Assumption and Its Implications\n\t•\tThe paper relies on exchangeability between the calibration and test datasets, but real-world MRI scans often violate this assumption due to scanner differences, patient variability, and pathology.\n\t•\tSuggested revision:\n\t•\tExplicitly discuss when the assumption holds and when it may fail in practical settings.\n\t•\tConsider adding an empirical test or discussion on how violations affect the validity of the FRIQ bounds.", "broader_impact_concerns": "The method does not provide absolute guarantees of image quality, yet it may be misused as a regulatory benchmark for AI-based MRI reconstructions. Regulators or hospitals might over-rely on the conformal bounds without considering clinical validation, leading to unwarranted trust in AI-generated images.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tZrQsutvMm", "reviewer": "Reviewer_59Bf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well-written and easy to follow.\n- The proposed approach is theoretically solid and addresses an important practical problem in (blind) quality assessment in imaging inverse problems.\n\n**Questions/Weaknesses**\n\nSome details of methodology and experiments can be better explained. Please see detailed questions below.\n\n- Section 3:\nThe proposed approach constructs one-sided bounds using conformal prediction. Based on the reviewer's knowledge, the standard conformal prediction method and theory study two-sided intervals. \n  - Has one-sided bound previously been explored in conformal prediction? Do the coverage guarantees for two-sided intervals apply to the one-sided case?\n  - Does a one-sided bound satisfy the condition that $|C_\\lambda|$ is non-decreasing in $\\lambda$, given that $|C_\\lambda|$ is infinite in this case?\n\n- Section 3.3:\nIt is my understanding that the approach works best when the distribution of approximate posterior samples is close to the true posterior distribution. If there is a large mismatch between the approximate and true posterior distributions, although the calibration step will still guarantee coverage of the bound, the resulting bound may be too loose or even vacuous. Have the authors studied the effect of this distribution mismatch on the resulting bound, either empirically or theoretically? Are there potential ways to characterize and mitigate this effect?\n\n- Section 3.4:\nMultiple datasets are potentially used in the proposed framework, all requiring ground-truth images:\n\n    - Training dataset of the image recovery model $h$\n    - Training dataset of the posterior sampler\n    - Calibration dataset for the metric bound $C_\\lambda(\\hat{z})$\n    - Training dataset of the quantile regression model $\\hat{z}(\\theta)$\n\nCan the authors clarify what assumptions on the above datasets are? Must all of them be disjoint? If not, which datasets can be shared?\n\n- Figure 2:\nThe quantile bound seems much more conservative for SSIM compared to that for PSNR. Do the authors have any insights about why this is the case?\n\n- Figure 2:\nThe figure shows that failure of coverage mainly happens when the recovery result is bad, i.e., when the true metric is poor (low for PSNR and SSIM, and high for DISTS and LPIPS). Do the authors have any insights on this?\n\n- Figure 4:\nIt would be helpful to also plot the mean difference between the bound and the true metric value. A smaller difference would indicate a tighter bound (with signs of the difference selected appropriately for higher-preferred and lower-preferred metrics respectively). This would more clearly show the gap between the bound and the true metric value.\n\n- Section 4.1:\nThe authors noted that the regression bound is not significantly better (tighter) than the quantile bound, even with more complex splines. Have the authors tried decreasing the number of posterior samples? The current number of samples (32) may be large enough to achieve a good estimate by empirical quantile, but the accuracy of empirical quantile may decrease when using fewer samples. The regression bound may be more competitive in this case. Using fewer posterior samples is desirable because of computation time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for constructing bounds on reference-based image quality metrics (e.g., PSNR, SSIM, etc.) when the ground-truth image is unknown, using conformal prediction (CP) to provide high-probability guarantees on the coverage of the bound. Two methods for providing point estimates of the metric are presented, including using the empirical quantile and a quantile regression method. Then, a calibration set is used to construct bounds from the point estimates, guaranteed to cover the true metric value with a given probability. Experiments on image denoising and accelerated MRI reconstruction show that the proposed approach achieves the desired coverage, provides tighter bounds than a baseline approach, and achieves more acceleartion in MRI for a given target on the metric value.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Consider adding clarifications for some of the above questions to the manuscript.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please ensure that all the changes and improvements made during the discussion period are properly implemented in the final version of the paper.
Decision: Accept (poster)