pcaGAN: Improving Posterior-Sampling cGANs via Principal Component Regularization
Authors: Matthew C Bendel
Rizwan Ahmad
Philip Schniter
Venue: NeurIPS 2024 poster (2024)
Keywords: Image recovery
inverse problems
MRI
posterior sampling
GAN
Primary area: machine_vision
TL;DR: For image-recovery problems, we propose a fast and accurate posterior sampler by regularizing a cGAN to enforce correctness in the K principal components of the posterior covariance matrix, and in both the trace-covariance and conditional mean.

Abstract: In ill-posed imaging inverse problems, there can exist many hypotheses that fit both the observed measurements and prior knowledge of the true image. Rather than returning just one hypothesis of that image, posterior samplers aim to explore the full solution space by generating many probable hypotheses, which can later be used to quantify uncertainty or construct recoveries that appropriately navigate the perception/distortion trade-off. In this work, we propose a fast and accurate posterior-sampling conditional generative adversarial network (cGAN) that, through a novel form of regularization, aims for correctness in the posterior mean as well as the trace and K principal components of the posterior covariance matrix. Numerical experiments demonstrate that our method outperforms competitors in a wide range of ill-posed imaging inverse problems.

pcaGAN: Improving Posterior-Sampling cGANs via Principal Component Regularization
Matthew Bendel Dept. ECE
Rizwan Ahmad Dept. BME The Ohio State University Columbus, OH 43210 ahmad 460osu. edu
Philip Schniter Dept. ECE
The Ohio State University
The Ohio State University Columbus, OH 43210 schniter: 1@osu. edu
Columbus, OH 43210 bendel. 00oll..8ssueeee edu
Abstract
In ill- ill-posed imaging inverse problems, there can exist many hypotheses that both the observed measurements and prior knowledge of the true image. Rather than returning just one hypothesis of that image, posterior samplers aim to ex- plore the full solution space by generating many probable hypotheses, which later be used to quantify uncertainty or construct recoveries that appropriately navigate the perception/ derstotion/ddstoottonn trade-off. In this work, we propose fast and accurate pesterior-sampling that, through novel form of regularization, aims for correctness in the posterior mean as well as the trace and K principal components of the posterior covariance matrix. Numerical experiments demonstrate that our method outperforms contem- porary cGANs and diffusion models in imaging inverse problems like denoising, large-scale inpainting, and accelerated MRI recovery. The code for our model can be found here: htpp::///tthan com/matt bendel bendel/pcaGAN,
conditional generative adversarial network (cGAN)
Introduction
In image recovery, the goal is to recover the true image æ from noisy/distoree//ncooplee measure- ments y = M(x).. This arises in linear inverse problems such as denoising, deblurring, inpainting, and magnetic resonance imaging (MRJ), well as in non-linear inverse problems like phase-retrieval and image-to-image translation. For such problems, impossible to perfectly recover æ from y.
In much of the literature, image recovery is posed as point estimation, where the goal is to return single best estimate w. However, there several shortcomings of this approach. First, it's clear how to define "best," since L2- or LI Ll-minimizing z are often regarded as too blurry, while efforts to make perceptually pleasing can sacrifice agreement with the true image and cause hallucinations [1-5], as we show later. Also, many applications demand not only recovery ~ but also some quantification of uncertainty that recovery [6. 7].
As an alternative to point estimation, posterior- sartiii-ssampiing--aaaaa image recovery [8-27] aims to generate P 2 samples { 1 from the posterior distribution P-yy(- Posterior sampling facilitates numerous strategies to quantify the uncertainty in estimating æ, or any function of æ, from y [6. 7]. It also can help with visualizing uncertainty and increasing robustness adversarial attacks [28]. That said, the design of accurate and computationally-eficei remains an open problem.
posterior samplers
Given a training dataset of image/mcasuremmnt pairs {(x, v.) 1T T our goal is to build a generator Go that, for given y, maps random code vectors I) to samples of the posterior, iee, = Go Go(z,y) y) ~ (x y). There are many ways to approximate the ideal generator. The recent literature has focused conditional variational autoencoders (cVAEs) [12-14], conditional
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

normalizing flows (cNFs) [8-11], conditional generative ad adversarial networks (cGANs) [1]-19], and sangeviisscore/difftussoooooo
generative models [20-27]. Although diffusion models have garnered a great share of recent attention, they tend to generate samples several wyders-of-magnituu slower than their cNF, cVAE, and cGAN counterparts.
With the goal of fast and accurate posterior sampling, recent progress in cGAN training has been made through regularization. For example, Ohayon al. [29] proposed enforce correctness in the generated y-conditional mean using novel form of L2 regularization. Later, Bendel et al. [19] proposed to enforce correctness in the generated y- g-conditional mean and trace-covariance using L1 regularization plus correctly weighted their "reGAN" competes with contemporary diffusion samplers on MRI and inpainting tasks. Soon after, Man et al. [30] showed that L2 regularization and a variance reward are effective when training cGANs for JPEG decoding. More details are given in Section
(SD) reward, and gave evidence that
In separate line of work, Nehme et al. [31] trained a Neural Posterior Principal Components (NPPC) network to directly estimate the eigenvectors and eigenvalues of the y-conditional covariance matrix, which allows powerful insights into the nature of uncertainty in an inverse problem.
Our contributions. Inspired by regularized cGANs and NPPC, we propose novel "peaGAN" that encourages correctness the principal components of the y y-conditional covariance matrix, as well the conditional mean and trace covariance, when sampling from the posterior. We demonstrate that our pcaGAN outperforms existing cGANs [16, 17, 19, 29] and diffusion models [22, 25- on posterior sampling tasks like denoising. large-scale inpainting, and accelerated MRI recovery, while sampling orders-of-magnitude faster than those diffusion models. We also demonstrate that PeaGAN recovers the principal components more accurately than NPPC using approximately the same runtime.
2 Background
Our approach builds on the reGAN regularization framework from [19], which itself builds on the cGAN framework from [16]. both of which we now summarize. Let X, y, and Z denote the domains of æ, z, and y. respectively. The goal is design generator Go : Z x y + where, for given y. the random ã Go(z, y) induced by the code vector z Pz (with z independent of y) has distribution that is close the true posterior (:y(:))) in the Wasserstein- distance, given by
Wi (Prly P.) = sup Exy y)} ( y))-
DEL,
Here, D : x/ > R is "critic" or "discriminator" that tries distinguish between the true and generated z given and L1 denotes the set of I-Lipschitz functions. A loss is constructed by averaging (I) over y ~ Py, which takes the form
By (Wi (Pay(;s v). (3
sup Ex y) D(Go(z, v)}, DEL1
using the fact that the expectation commutes with the supremum [16]. D is then implemented as neural network De with parameters Φ. Finally, (0, Φ) are learned by minimizing
Lea x) A Ex.zy (aaeyyDDssee
(2s y) DG y).
over 0 and minimizing - La Φ)) + Lg (T) over Φ, where (o) is a gradient penalty that encourages D& E L1 [32]. In practice, the expectation in (3) is approximated by a sample average over the training data { }.
In the typical case that the training data includes only single image x, for each measurement vector yr minimizing Ladu(B (0, +) alone does not encourage the generator produce diverse samples. Rather, it leads to form of mode collapse where the code z is ignored. In [16], Adler and Öktem proposed two-sample discriminator that encourages diverse generator outputs without compromising the Wasserstein objective (2). In [19], Bendel et al. instead proposed to regularize Ladv Φ) in way that encourages correct posterior means and trace-covariances, i.e..
Paly Mxlyy
for Haly A E }
and Pxly E(
(4)
tr( (r(E)))= = tr tx(Ex)y
for Eny Cov
and Σxyy Cov(aly).
(5)

---

To do this, [19] replaced (0,v Φ) with the regularized adversarial loss
LicGAN Φ) - Ladv Φ) + P. (0) SSDCSD.P.(())
(6)
which involves the Pre -ample supervised-fi loss and standard-deviation (SD) reward terms
(() A Exz {1 :xp; 111}
(7)
LSD.r(0) - E Ea....yy { E6
(8)
where typically Pre 2. Here, } are the generated samples and (ip) is their P-sample average:
3, Go y) for = ,...,, and 3 -== S..
(9)
The reward weight ASD in (6) is automatically adjusted to accomplish (5) during training [19]. We note that the (0) regularization proposed in [19] is closely related to the regularization
(2)) e Ex..1....z... [|| p) [12] proposed earlier by Ohayon et al. in [29]. A detailed discussion
of the advantages and disadvantages of various cGAN regularizations can be found in [19].
3 Proposed method
Whereas reGAN aimed for correctness the posterior mean and posterior coverccoe statistics, our proposed peaGAN also aims for correctness in the K principal components of the posterior covariance matrix Σgy, where K is user-selectable. To do this, pcaGAN adds two additional regularization terms to the reGAN objective:
(OC Φ) - CreGAN(O, Φ) + (eve + Ppca Ceval (0)
(10)
Levec (0) -E, Ex +l olyy EE=Ilo 10 (x M Muly |
(11)
(eval()) A E, Ex.p,...apyyy EE (1- (1- >o 11)}} |4}
(12)
Here, ((0g, ) k== denote the principal eigenvectors and eigenvalues of the O O-dependent generated covariance matrix Eny and {(Uk, ) denote the principal eigenvectors and eigenvalues of the true covariance matrix Exly- Because (11) is the classical PCA objective [33], minimizing Lever (0) over 0 will drive the generated principal eigenvector Üy towards the true principal eigenvector uy for each k = L,..., K. Likewise, minimizing (ev over O will drive the generated principal eigenvalue Å towards the true principal eigenvalue X for each k ,,...,K K. Based experiments, putting ' in the denominator works better than the numerator and the squared error in (12) works better than an absolute value.
In practice, the expectations in (11)-(12) are replaced by sample averages over the training data. In the typical case that the training data includes only a single image æ, for each measurement vector yr, the quantities Muly and { in (11)-(12) are unknown and non-trivial to estimate for each y Hence, when training the peaGAN, we approximate them with learned quantities. This must be done carefully, however. For example, if μxly in (11) was simply replaced by the O-dependent quantity May: then minimizing Levec(0) over θ would encourage Maly to become overly large in order to drive Levee () to a large negative value.
Algorithm 1 details our proposed approach to training the PeaGAN. In particular, it describes the steps used to perform a single update of the generator parameters 0 based on the training batch {wvs, yb) ) Before diving into the details, we offer brief summary of Algorithm 1. For the initial epochs, the reGAN objective frEGAN alone is optimized, which allows the generated posterior mean Maly to converge to the vicinity of Mxly- Starting Eevec epochs, the Levec (0) regularization from (11) is added, but with Hxly approximated StopGrad The of StopGrad ad forces Lever (0) to minimized by manipulating the eigenvectors {Øx) and not the generated posterior mean May These eigenvectors are computed using an SVD of centered approximate prrsttiate-posterior samples. To reduce the computational burden imposed by this SVD, a "lazy regularization" [34] approach adopted, which computes Levec()) only once every M training steps. Training proceeds this manner until the eigenvectors {ūt) converge. Starting at Eeval epochs, the Leval (0) regularization from (12) is added, but with A approximated as
A StopGrad (1 10 ( Molys 31 Maly:.... P 13
(13)
3

---

Algorithm 1 pcaGAN generator-training iteration
Require:
number of estimated eigen-components K, number of samples used for eigenvector and eigenvalue regularization Ppea. number of samples used for reGAN regularization Pre. epoch at which to involve eigenvector regularization Eevec. epoch at which to involve eigenvalue regularization Eeval. lazy update period M. adversarial loss weight Sadv. std regularization weight $SD. cigenvector and eigenvalue regularization weight Spca. training batch { yo) current model parameters 0, current training epoch e, the current training step 8
1: [(0) < 0
2:
3: for 1, -....,, B do
4:
2, ~ 10, for i= I...,.., Pre
5:
T. + Go(zi, Vb) for i i= 1,..,, Pre
6:
(()) - [(0) Badv Eie, ( ybb
7:
Lin = 1 E 3;
Ph
8:
(()) + (()) + | - i inlll 1 - ADD E Em {1āt 2mmll
9:
10:
ife Fevee and 8 mod M = () then
11:
2j ~ NoO, I) for ,,..,, P pta
12:
2j + Ga(zj, vo) for = 1,..., Po
pca
13:
p StopGrad Pps Σ)=1 33j)
14:
ÚSV + +VD((@ N1,
..... Pps [1]
15:
ūR + [i for k- = 1,... K
16:
(()) [(0) Spca Σ K
-i=1
T
1 (ars- ]2)
17:
end if
19:
if'
e
> Levall and s mod M = 0 then
20:
A f [ for k = l...., K..., K
21:
X [ pr, 3b μ, .2
8 ...... 3t P... A
22:
((0) [()) + Spca k= (1 As
StopGrad PartT 100 X 11)
23:
end
24:
end for
25:
26:
0 <- Adam(0, VC(O))
where StopGrad is used so that the optimization focuses on {7x J. The rational behind (13) is that, when 0. UR and Maly Mxly- the terms [ (xx, M2y)]] and 10 (3) May)! Vj all equal A in y,, -o-nonnitional expectation. This expectation is approximated using + Ppta)-term sample average in (13) via the squared norm. The eigenvalues {Aa} in (12) are computed using the previously described SVD and the regularization schedule is again M Mlaazy.
We now provide additional details on Algorithm After the loss is initialized in line 1, the following steps are executed for each measurement vector yı, in the batch. First, approximate posterior samples [ are generated in line 5, where Pre = 2 as per the suggestion in [19]. Using these samples, the adversarial component of the loss added line and the reGAN regularization is added in line 8. Starting epoch Eevec lines 11-16 executed whenever the training iteration is multiple of M. Nominally, Levec is set where the validation PSNR of f (an empirical approximation of
Pare Mayy) stabilizes and M = 100. Within those lines, samples (Φ, j== are generated in line 12 (where nominally Ppca = 10K), their sample mean is computed line 13, and the SVD of the centered samples computed in line 14. The top K right singular vectors are then extracted in line 15 in order to construct the Levec (0) regularization, which is added to the overall generator loss (()) in line 16. Starting epoch Eeval, where nominally Eeval = Eevec + 25, lines 20-22 are executed whenever the training iteration a multiple of M. In line 20, the top K eigenvalues ) are constructed from the previously computed singular values and, in line 22, the ( regularization is constructed

---

(a) W2 W2 PRIy) vs. M
(b) (2 Paly) vs. K
(c) W2(Paly Pay pay)
10"
13
0.2
0.3
Figure 1: Gaussian experiment. Wasserstein-2 distance versus (a) lazy update period M for pcaGAN with d = 100 K, (b) estimated eigen- componetss for peaGAN with d 100 and M 100. and (c) problem dimension d for all methods under test with K = d and M 100.
and added to the overall training loss. The construction of Leval (0) was previously described around (13). Finally, once the losses for all batch elements have been incorporated. the gradient V (0) is computed using back-propan proppppppaaaionn and gradient-descent step of 0 is performed using the Adam optimizer [35] in line 26.
Numerical experiments
We now present experiments with Gaussian data, MNIST denoising, MRI, and FFHQ face inpainting. Additional implementation and training details for each experiment are provided in Appendix D.
4.1 Recovering synthetic Gaussian data
Here our goal is to recover Nu Ex) D Rd from y Mæ + Rd, where M masks
at even indices and noise w ~ _N(0, o2) is independent of with o² = 0.001. Since a and y are jointly Gaussian, the posterior is Gaussian posterior with Mxly = 14 + En Iy Hy) and Exyy - Eny where 1xx My: Ex, Ey are marginal and Exy Eyx are joint statistics.
We generate random μ. NoO, I) and Σ, with half-normal eigenvalues d (see additional details in App. A), and consider a sequence of problem sizes d 10, 20, 30...., 100. For each d, generate 70000 training, 20000 validation, and 10000 test samples. The generator and discriminator are simple multilayer perceptrons (see App. D.1) trained for 100 epochs with K d, Eever 10, Bad = 10-3, and 3pca = 10-2
Competitors. We compare the proposed pcaGAN reGAN [19] and NPPC [31]. reGAN uses the same generator and discriminator architectures as pcaGAN and is trained according to (6) with Badv 10 and Pe For NPPC, we use the authors' implementation [36] with K d and some minor modifications to work with vector data. To evaluate performance, we use the Wasserstein-2 (W2) distance between Pxly and Pxlys which in the Gaussian case reduces to
(2 Paly = | Mxly 11 + Exly + Exyy 2(E
(14)
For the cGANs, compute Pujy and Σxly empirically from 10d samples, while for NPPC the conditional mean, eigenvalues, and eigenvectors returned by the approach.
Results. Figure 1a examines the impact of the lazy update period M on pcaGAN's W2 distance at d = 100 with K = d. Based on this figure, to balance performance with training overhead, we set M 100 for all future experiments. Figure 1b examines the impact of K on W2 distance for the pcaGAN with d = 100. It shows that using K < d causes relatively mild increase in W2 distance, as expected due to the half-normal distribution on the true eigenvalues dς- Figure 1e shows that the proposed pcaGAN outperforms reGAN and NPPC in W2 distance for all problem sizes d.
4.2 MNIST denoising
Now our goal recover an MNIST digit æ E [o, 2|2822 from noisy measurements y x with (O, I). We randomly split the MNIST training fold into 50 000 training and 10000
5

---

Table 1: Average MNIST denoising results.
Model
rMSR]
REM,
CFID
Time(118))
NPPC (Nehme et [31])
3,94
3.63
ms
reGAN (Bendel et al. [19]
4.04
3.41
63.44
118
peaGAN (Ours, K 5)
4.02
3,31
61.48
118 ms
peaGAN (Ours, K = 10)
4.02
3.25
60.16
118 ms
(a) peaGAN (K 10)
(b) NPPC
1
22
3
U4
45
1
42
3
44
Ø5
S
5
$
0--3 3
a 2
o=0
o=
o=3
n=D
o=2
o=3
S
6
6
5
S
S
p
9
6
5
Figure 2: For (a) pcaGAN and (b) NPPC, this figure shows the true image æ, noisy measurements y. the conditional mean Pxly. principal eigenvectors {ū and two perturbations of Mxly
validation images, and we use the entire MNIST fold set for testing. For pcaGAN and reGAN, we UNet [37] generator and the encoder portion of the same UNet followed by dense layer as the discriminator, pcaGAN was trained for 125 epochs with Eever = 25, Badv = 10 8pca = 10 10-1 and K E 10).
Competitors. We again compare the proposed peaGAN reGAN and NPPC. For reGAN we used the same generator and discriminator architectures pcaGAN and trained according to (6) with Badv = 10-3 and Pre 2. For NPPC, we used the authors' MNIST implementation from [36].
Following the NPPC paper [31], we evaluate performance using root MSE (rMSE) Ex.y [1 P lla}
and residual error magnitude (REMs) Exy {I] - ve where e = x Puly and V 3 an 28 x 5 matrix whose kth column equals the keth principal eigenvector G₁. For the cGANs, we use J = EM and compute { } from the SVD of a matrix of centered samples ] both with P = 100. For NPPC, we the conditional means and eigenvectors returned by the approach. For performance evaluation, we also consider Conditional Frechet Inception Distance (CFID) [38] with inception features. CFID is analogous to Frechet Inception Distance (FID) [39] but applies to conditional distributions (see App. B for more details).
Results. Table shows rMSE, REM,, CFID, and the reconstruction time for batch of 128 images on the test fold. (NPPC does not generate image samples and so CFID does not apply.) The table shows that the proposed pcaGAN wins in all metrics, except for rMSE where NPPC wins. This is not surprising because NPPC computes Pxly using a dedicated network trained to minimize MSE loss. NPPC also generates its eigenvectors slightly quicker than PcaGAN generates samples. Table also shows that pcaGAN performance improves K increases from 5 to 10, despite the fact that REM; uses only the top 5 cigenvectors. Figure E. shows examples the 5 principal cigenvectors and posterior mean learned by peaGAN and NPPC. The eigenvectors of pcaGAN are more structured and less noisy than those of NPPC. Figure E. also shows Pxly + too, for E [-3, 3. and E {1, 4). Additional figures can be found in App. E.I.
4.3 Accelerated MRT
We now consider accelerated MRI, where the goal is recover complex-valued multicoil image from masked frequency-domaii (i.e., "k-space") measurements y. To build the image data (æt). we follow the approach in the reGAN paper [19], which uses the first 8 slices of all fastMRI [40] T2

---

PcaGAN (ours)
rcGAN (Bendel)
pScGAN (Ohayon)
cGAN (Adler)
Langevin (Jalal)
Truth
Average
(P-32)
Sample
Truth
E2E-VarNer
Sample
Figure Example MRI recoveries at R 8. Arrows highlight meaningful variations.
Table 2: Average MRI results at acceleration R E {4,8)
R 4
==
Model
CFID'L
CFID"
CRID
FIDL
APSD
Time
CFID!
CFID'
CFIP%)
(Sriram (43)
316ms
Langevin (Jabl et al. (23))
33.05
31.43
5.9c-6
14 min
52.62
7.6e-6
min
cGAN (Adler & Üktem [16]
19.00
12.05
7,00
29.77
3.9c-5
217 ms
59.94
40.24
26,10
31.81
7.7e-6
217 ms
pscGAN (Ohayon
(29))
13.74
10.56
7.53
37.28
7.2c-8
217
39:67
31.81
24.06
43.39
7.7c-7
217
JeGAN (Beadel (19])
1.69
25.02
3.Ne-I
13.20
3.83
28.43
217
PoaGAN (Ours)
25.02
brain volumes with at least coils, crops 384 384 pixels, and compresses 8 virtual coils [41].
This yields 200 training, 2 276 testing, and 784 validation images. To create each Ye, we transform at to the k-space, subsample using the Cartesian GRO mask [42] accelerations R and R and transform the zero-filled k-space measurements back to the image domain.
We train pcaGAN for 100 epochs with K 1, Levec 25, Badv 10-5, and Spca 10 and select the final model using validation CFID computed with VGG- 16 features.
Competitors. We compare the proposed pcaGAN to reGAN [19], pscGAN [29], Adler & Öktem's cGAN [16], the Langevin approach [22], and the E2E- VerNNe [43]. All eGANs use the generator and discriminator architectures from [19] and enforce consisonn [44]. For reGAN and the
Langevin approach, we did not modify the authors' implementations from [45] and [46] except to use
the GRO sampling mask. For E2F- VarNet, we use the GRO mask, hyperparameters, and training procedure from [22].
Following [19], we convert the multicoil outputs T; to complex-valuee images using SENSE-based coil combining [47] with ESPIRiT-estimted [48] coil sensitivity maps, and compute performance on magnitude images. All feature-based metrics (CFID, FID, LPIPS, DISTS) were computed with AlexNet features to show that peaGAN does not overfit to the VGG-16 features used for validation. It was shown in [49] that image-quality metrics computed using ImageNet-trand trained feature generators like AlexNet and VGG-16 perform comparably metrics computed using MRI-trained feature generators in terms of correlation with radiologists' scores.
Results. Table 2 shows CFID, FID, APSD ( E N -, 1/) and 4-sample generation time, we evaluate the CFID, FID, and APSD of the Langevin technique [22] using the 72-image test from [19]. But due to the bias of CFID at small sample sizes [38], we evaluate the other methods using all 2376 test images (CFID") and again using all 14 576 training and test images (CFID3). Table 2 shows that pcaGAN yields better CFID and FID than the competitors. All eGANs generated samples 3-4 ofaggn-d faster than the Langevin approach [22].
time for the methods under test. Due its slow

---

Table 3: Average PSNR, SSIM, LPIPS, and DISTS of ((p)) versus P for MRI at R =
PSNRT
SSIMT
Model
P=2
P=4
3=
P=1
P=2
P= P-16
3=
EZE. VarNet (Sriram et al. (43))
36.49
0.9220
Langevin (Jatal et al. (22])
52.17
32.83
33.45
33.74
33.83
33.90
0,8725
0.8919
0.90131
0.9091
0.9120
0.9137
cGAN (Adler Oktem [16])
32.92
0.5865
0.9045
0.9102
0.9095
PecGAN (Ohagon et al. (29])
34.89
34,90
34.90
34.90
34.91
34.92
0.9222
0.9217
0.9213
0.9211
0.9211
0.9210
reGAN (Bendel [19])
32.32
33.67
34.53
35.0
35.27
35.42
0,9030
0.9199
0.9252
0.9257
0.9251
0.9246
peaGAN (Ours)
0.9136
0.9257
0,9262
0,9253
LPIPS)
DISTS)
Model
P=1
P=2
P=4
P=8
P=16
32
P=1
P=2
P=1
P=
P=16
P2
I: VarNet (Sriam [43])
0,0575
0.1253
Langevin (lala et al. (22))
0,0769
0.0619
0.0579
0.0589
0.0611
0.0611
0.1341
0.1136
0.1086
0.1119
0.1175
0.1212
cGAN (Adler & Oktem [16])
0,008
0.0614
0.0625
0.0667
0.07704
0.0727
0.1447
0.1262
0.1252
0.1291
0.1334
0.1361
pscAN (Obayon et (29p)
0,0536
0.0539
0.0540
0.1128
0.1143
0.1157
0.1158
rGAN (Bendel et al. (191)
0.0118
0.0379
0.0421
0.0476
0.0516
0.0539
0.0906
0.0877
0.0965
0.1063
0.1135
0.1177
PcaGAN (Ours)
0.0358
0.0344
0.0391
0.0442
0.0479
0.0499
0.0604
0.0799
0.0920
0.1026
0.1099
0.1144
Table 4: Average FFHQ inpainting results.
Model
CHID
FID
LPIPS
Time (40 samples)
DPS (Chung et al. [26])
7.26
2.00
0.1245
14 min
DDNM (Wang et 127D
11.30
3.63
0,1409
30
DDRM (Kawar ct al. (25))
13.17
5.36
0.1587
5:
PseGAN (Ohayon et al. [29])
18.44
8.40
0,1716
325 ms
CoMotGAN (Zhao al. [17])
7.85
0.1290
ms
reGAN (Bendel et al. (19]
7.51
2.12
0.1262
325 ms
PcaGAN (Ours)
0.1230
ms
Table 3 shows PSNR, SSIM, LPIPS [50], and DISTS [51] for the P-sample average T.my P E {1,2, 4, 8, 16, 32} and R = (See App. for = 4.) It has been shown that DISTS correlates particularly well with radiologist scores [52]. The E2E- VarNet achieves the best PSNR, but the proposed cGAN achieves the best LPIPS and DISTS when P and the best SSIM when P = This P-dependence is related to the perception-distortion trade-off [53] and consistent with that reported in [19].
Figure 3 shows zoomed versions of two recoveries T; and the sample average p,p with P = 32. Appendices E.2 and E.3 show additional plots of (1p) that visually demonstrate the perception- distortion trade-off.
4.4 Large-scale inpainting
Our final goal is to inpaint a face image with a large randomly generated masked region. For this task, we use 256 x 256 FFHQ face images [54] and the mask generation procedure from [17]. We randomly split the FFHQ training fold into 45 000 training and 5 500 validation images, and we use the remaining 20 000 images for testing.
For pcaGAN, epochs using K- = 2, Eevec 25, Padv = 5 x 10-3, and 3pca = 10-3
CoModGAN's [17] generator and discriminator architecture and train for 100
Competitors. We compare with CoModGAN [17], pscGAN [29], reGAN [19], and state-of-the art diffusion methods DDRM (20 NFEs) [25], DDNM (100 NFEs) [27], and DPS (1000 NFEs) [26]. CoModGAN, pscGAN, and reGAN differ from pcaGAN only in generator regularization and CoModGAN's use discriminator MBSD [55]. For reGAN, DDNM, DDRM, and DPS, we use the authors' implementations from [45], [56], [57], and [58] with mask generation from [17]. FID and
CFID were evaluated on our 200 image test set with P=1.
Results. Table 4 shows test CFID, FID, LPIPS, and 40-sample generation time. The table shows that the proposed peaGAN wins in CFID, FID and LPIPS, and that the four cGANs generate samples 3-4 orders-of-magnieee
faster than DPS. Figure 4 shows five generated samples for each method under test, along with the true and masked image. peaGAN shows better subjective quality than the competitors, well good diversity, Additional figures can be found in App. E.4.
8

---

Original
PcRGAN
Masked
TeGAN
CoModGAN
DPS
DDNM
DDRM
Figure 4: Example of inpainting randomly generated mask on 256 x 256 FFHQ face image.
5
Discussion
When training cGAN, the overall goal is that the samples (..) generated from a particular y accurately represent the true posterior ( Achieving this goal is challenging when training from paired data { ), because such datasets provide only one example of x for each given y- variation among {æ;) but did not aim for the correct variation. The reGAN from [19] focused on providing the correct amount of variation by enforcing tr( Exy) tr( Exjy), and the proposed pcaGAN goes farther by encouraging Enyy and Exly to agree along K principal directions. Our experiments demonstrate that pcaGAN yields a notable improvement over reGAN and outperforms contemporary diffusion approaches like DPS [26).
Early methods like [15-17] focused providing
PCA principles have also been used unconditional GANs, where the goal is to train a generator Go that turns codes ~ (O I) into outputs ã Gø(z) that match the true marginal distribution px from which the training samples (æ;} are drawn. For example, the eigenGAN from [59] aims to train in such way that semantic attributes are learned (without supervision) and can be independently controlled by manipulating individual entries of z. But their goal is clearly different from ours.
Limitations. We acknowledge several limitations of our work. First, generating Ppea 10K samples during training can impose burden on memory when æ is high dimensional. MRI experiment, æ E Rd for d = 2.4e6, which limited us to K = 1 at batch size 2. Second, although our focus on designing fast and accurate posterior sampler, more work is needed on how to best the generated samples across different applications. Using them to compute rigorous uncertainty
multicoil

---

intervals seems like a promising direction [60-62] 62]. Third, the application to MRI is preliminary; additional tuning and validation is needed before it can be considered for clinical practice.
Conclusion
In this work, we proposed peaGAN, novel image-recovery cGAN that enforces correctness in the K principal components of the conditional covariance matrix Egly as well as in the conditional mean Maly and trace-covariance tr (aay)) ). Experiments with synthetic Gaussian data showed pea outperforming both reGAN [19] and NPPC [31] in Wasserstein-2 2 distance across a range of problem sizes. Experiments on MNIST denoising, accelerated multicoil MRI, and large-scale image inpainting showed pcaGAN outperforming several other cGANS and diffusion models in CFID, FID, PSNR, SSIM, LPIPS, and DISTS metrics. Furthermore, peaGAN generates samples 3-4 orders-of- -fdder-of-maanniuuee faster than the tested diffusion models. The proposed peaGAN thus provides fast and accurate posterior sampling for image recovery problems, which enables uncertainty quantification, fairness in recovery, and easy navigation of the perception/diisootion trade-off.
Acknowledgments and Disclosure of Funding
The authors are funded in part by the National Institutes of Health under grant R01-EB0299977.
References
[1] C. Belthangady and L. Royer, "Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction," " Nature Methods, vol. 16, no. 12, pp. 1215-1225, 2019.
[2] D. P. Hoffman, L. Slavitt, and C. A. Fitzpatrick, "The promise and peril of deep learning in microscopy."
Nature Methods, vol. 18, no. 2. pp. 131-132, 2021. 1
[3] M. J. Muckley, B. Riemenschncider, A. Radmanesh, S. Kim, G. Jcong, Ko, Y. Jun, H. Shin, D. Hwang,
M. Mostapha, et al., "Results of the 2020 fastMRI challenge for machine learning MR image reconstruction."
IEEE Trans. Med. Imag., vol. 40, no. 9, pp. 2306-2317, 2021. 1
[4] S. Bhadra, A. Kelkar, F. Brooks, and M. Anastasio, "On hallucinations tomographic image reconstruction," IEEE Trans. Med. Imag., vol. 40, no. 11. pp.
2021. 1
[5] N. M. Gottschling, V. Antun, A. C. Hansen, and B. Adeock, "The troublesome kernel- OO ballucinations. free lunches and the accuracy-stabilli trade-off in inverse problems." arXiv:2001.01255. 2023.
[6] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Ficguth, X. Cao, A. Khosravi, U. R. Acharya, et al., "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Info. Fusion, vol. 76, pp. 243-297, 2021.
[7] B. Lambert, Forbes. S. Doyle, Dehaene, and M. Dojat, "Trustworthy clinical Al solutions: unified review of uncertainty quantification in deep learning models for medical image analysis." Artificial Intelligence in Medicine, 102830, 2024. 1
[8] L. Ardizzone,, C. Lüth, J. Kruse, Rother, and U. Köthe, "Guided image generation with conditional invertible neural networks," " arXiv:/907.000 /907.02.92, 2019. 1,2
[9] C. Winkler, D. Worrall, E. Hoogeboom. and M. Welling, "Learning likelihoods with conditional normaliz- ing flows." arXiv:1912.00044 2019. 1,2
[10] H. Sun and K. L. Bouman, "Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging." in Proc. AAAI Conf. Artificial Intell., vol. 35, pp. 2628-2637, 2021. 2021. 1,2
[11] J. Wen, R. Ahmad, and Schniter. "A conditional normalizing flow for accelerated multi-coil MR imaging." in Proc. Intl. Conf. Mach. Learn., 2023. 1,
[12] V. Edupuganti, M. Mardani, S. Vasanawala, and J. Pauly, "Uncertainty quantification in deep MRI reconstruction," IEEE Trans. Med. Imag., vol. 40, pp. 239-250, Jan. 2021.
[13] F. Tonolini, J. Radford, A. Turpin. D. Faccio, and R. Aurray-Smith "Variational inference for computa- tional imaging inverse problems," J. Mach. Learn. Res., vol. 21, no. 179, pp. 1-46, 2020.
[14] K. Sohn, H. Lee, and X. Yan, "Learning structured output representation using deep conditional generative models," in Proc. Neural Info. Process. Syst. Conf., 2015.
[15] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, "Image- to image translation with conditional adversarial networks." in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp. 1125 1134, 2017. 2.
10

---

[16] J. Adler and O, Öktem, "Deep Bayesian inversion," arXiv: 2018. 2, 7, 14
[17] S. Zhao, J. Cui, Y. Sheng, Y. Dong. X. Liang, E. I.C. Chang, and Y. Xu, 'Large scale image completion via co- no-modulated generative adversarial networks," in Proc. Intl, Conf. Learn. Rep., 2021. 1, 2, 8, 9, 16
[18] H. Zhao, H. Li, S. Maurer-Stroo Stroh, and L. Cheng, "Synthesizing retinal and neuronal images with generative adversarial nets," Med. Image Analysis, vol. 49, 07 2018. 1.2
[19] M. Bendel, R. Ahmad, and P. Schniter, "A regularized conditional GAN for posterior sampling in inverse problems." in Proc. Neural Info. Process. Syst. Conf. 2023. 2, 4. 5, 6, 7, 8, 9, 10, 13, 14, 15, 16
[20] M. Welling and Y. W. Teh, "Bayesian learning via stochastic gradient Langevin dynamics." in Proc. Intl. Conf. Mach. Learn., pp. 68 681-688, 2011. 1,2
[21] Y. Song and Ermon, "Improved techniques for training score-based generative models," in Proc. Neural Info. Process. Syst. Conf. 2020. 1,
[22] A. Jalal, M. Arvinte, G. Daras, E. Price, A. Dimakis, and J. Tamir, "Robust compressed sensing MRI with deep generative priors," in Proc. Neural Info. Process. Syst. Conf. 2021. 1, 2, 7, 8, 14, 15
[23] Y. Song, Sohl-DDcksteins D. P. Kingma. A. Kumar, S. Ermon, and B. Poole, Sscore-based generative modeling through stochastic differential equations." " in Proc. Int. Conf. Learn. Rep., 2021. 2
[24] Y. Song, L. Shen, L. Xing, and S. Ermon, "Solving inverse problems in medical imaging with score-based generative models," " in Proc. Intl. Conf. Learn. Rep., 2022. 1,2
[25] B. Kawar, M. Elad, S. Ermon, and J. Song, "Denoising diffusion restoration models," in Proc. Neural Info.
Process. Syst. Conf. 2022. 1,
[26] H. Chung, J. Kim. M. T. McCann, M. L. Klasky, and J. C. Ye, "Diffusion posterior sampling for general noisy inverse problems," in Proc. Intl. Conf. Learn. Rep., 2023. 2, 8, 9, 16
[27] Y. Wang, J. Yu, and J. Zhang, "Zero-shot image restoration using denoising diffusion null-space model," in Proc. Intl. Conf. Learn. Rep., 2023. 1, 2, 8
[28] G. Ohayon, T. J. Adrai, M. Elad, and T. Michaeli, "Reasons for the superiority of stochastic estimators over deterministic ones: Robustness, consistency and perceptual quality," in Proc. Intl. Conf. Much. Learn.,
pp. 26474-26494, 2023. 1
[29] G. Obayon, T. Adrai, G. Vaksman, M. Elad, and P. Milanfar, "High perceptual quality image denoising with posterior sampling CGAN," in Proc. IEEE Intl. Conf. Comput. Vis. Workshops, vol. 10, pp. 1805-1813. 2021. 3, 7, 8, 14
[30] S. Man, G. Ohayon, T. Adrai, and M. Flad, FHigh-perceptud quality JPEG decoding via posterior sampling, Proc. IEFF Conf. Comp. Vision Pattern Recog. Workshop, 2023. 2
[31] E. Nehme, O. Yair, and T. Michaeli, "Uncertainty quantification via neural posterior principal components."
in Proc. Neural Info. Process. Syst. Conf., 2023. 2, 5. 10, 15
[32] I. Guirajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, "Improved training of Wasserstein GANs," in Proc. Neural Info. Process. Syst. Conf.. p. 5769-5779, 2017.
[33] I. T. Jolliffe, Principal Component Analysis, vol. 2. New York: Springer Verlag, 2002. 3
[34] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, "Analyzing and improving the image quality of StyleGAN," in Proc. IEEE Conf. Comp. Vision Pattern Recog.. pp. 111081119 2020. 3, 16
[35] D. Kingma and J. Ba, "Adam: method for stochastic optimization." " in Proc. Intl. Conf. Learn. Rep.. 2015. 5, 15
[36] E. Nehme, O. Yair, and T. Michaeli, "Uncertainty quantification via neural posterior principal components." Downloaded from https://githut: com/EliasNehma.eee.eoo /NPPC, Jan. 2023. 5, 6, 15
[37] O. Ronneherger, Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image seg- mentation," in Proc. Inal. Conf. Med. Image Comput. Comput. Assist. Intervent., pp. 234-241, 2015.
[38] M. Soloveitchik, T. Diskin, Morin, and A. Wiesel, "Conditional Frechet inception distance." axXiv:2103.1152/, 2021. 6, 7, 13
[39] M. Heusel, H. Ramsauer, Unterthiner, B. Nessler, and S. Hochreiter, "GANs trained by two time-scale update rule converge to a local Nash equilibrium," " in Proc. Neural Info. Process. Syst. Conf. vol. 30, 2017.
6, 13
[40] Zbontar, Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. Geras, J. Katsnelson.
H. Chandarana, Z. Zhang. M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, J. Pinkerton, D. Wang. N. Yakubova, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, "fastMRI: An open dataset and benchmarks for accelerated MRI," arXiv:1811 1.8839, 2018. 6
11

---

[41] T. Zhang, J. M. Pauly. S. S. Vasanawala, and M. Lustig. "Coil compression for accelerated imaging with Cartesian sampling," Magn. Reson. Med., vol. 69, no. pp. 571-582, 2013.
[42] M. Joshi, A. Pruitt, C. Chen, Y. Liu, and R. Ahmad, "Technical report (V1.0)ppeeudo-radoom cartesian
sampling for dynamic MRI," (tth:220060363000 2022. 7
[43] A. Sriram, Zbontar, Murrell, A. Defazio, C. Zitmick, Yakubova, Knoll, and Johnson. "End-to-end variational networks for accelerated MRI reconstruction," in Proc. Inil. Conf. Med. Image Comput. Comput. Assist. Intervent., pp. 64-73, 2020. 7, 8, 14, 15
[44] C. K. Sønderby, J. Caballero, Theis, W. Shi, and F. Huszár, "Amortised MAP inference for image supcr-resolution." in Proc. Intl. Conf. Learn. Rep., 2017. 7
[45] M. Bendel, R. Ahmad, and P. Schniter, "A regularized conditional GAN for posterior sampling in inverse problems." Downloaded from bestts://giiiie httpa//.....nnaaabbeeeeeeeeeeeee bendel/rcGAM, May 2023. 7, 7,8, 13
[46] A. Jalal, M. Arvinte, G. Daras, E. Price, A. Dimakis, and J. Tamir, gesgm-mri-langewwe mri -ra--m-nen-aannnnnn...aa 2021. Accessed: 2021-12-05. 7, 15
https://
gi wtt..b/ccmttsssaabbeeaa..
[47] K. Pruessmann, M. Weiger, M. Scheidegger, and P. Boesiger, "SENSE: Sensitivity encoding for fast
MRI," Magn. Reson. Med., vol. 42, no. 5, pp. 952-962, 1999.
[48] M. Uecker, Lai, M. Murphy, P. Virtue, M. Flad, M. Pauly, S. Vasanawala, and M. Lustig, "ESPIRIT an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA," Magn. Reson. Med., vol. 71, no. 3, pp. 990-1001, 20144,
[49] P. M. Adamson, A. D. Desai, J. Dominic, C. Bluethgen, J. P. Wood, A. B. Syed, R. D. Boutin, K. J. Stevens, S. Vasanawala, J. M. Pauly, A. S. Chaudhari, and B. Gunel, "Using deep feature distances for evaluating MR image reconstruction quality," in Proc. Neural Info. Process. Syst. Workshop, 2023. 7
[50] R. Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as perceptual metric." Proc. IEEE Conf. Comp. Vision Puttern Recog., pp. 386-595,00 2018. 13
[51] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, "Image quality assessment: Unifying structure and texture similarity," IEEE Trans, Pattern Anal. Mach. Intell., vol. 44, no. pp. 2567-2581, 2020. 8, 13
[52] S. Kastryulin, J. Zakirow, N. Pezzotti, and D. V. Dylov, "Image quality assessment for magnetic resonance imaging." IEEE Access, vol. 11, pp. 14154-14168, 2023. 8
[53] Y. Blau and Michaeli, "The perception-distorton tradeoff," Proc. IEEE Conf. Comp. Vision Pattern Recog,, PD.
2018. 8
[54] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks." in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp. 4396-4405, 2019.
[55] T. Karras, T. Aila, S. Laine, and J. Lehtinen, "Progressive growing of GANs for improved quality, stability, and variation." " in Proc. Intl. Conf. Learn. Rep., 2018. 8. 14, 15
[56] Y. Wang, "DDNM." Downloaded from http:://itthu...o//// wwwt::/www.....w/....a.aaaaa DyD./// Sept. 2023. 8, 16
[57] B. Kawar, M. Elad, S. Ermon, and Song. "Denoising diffusion restoration models." Downloaded from com/babjat-kawaa /ddrm. May 2022. 8, 16
https://gi
[58] H. Chung, J. Kim, M. T. McCann, M. L. Klasky, and J. C. Ye, "diffusion- posterior-sampllneee https: //github. wwwpc//ppp//////iioooooo http:///iifffssiooooootrriiieiie
sampling. Mar. 2023. 8, 16
[59] Z. He, M. Kan, and S. Shan, "EigenGAN: Layer-wise eigen-learning for GANs." in Proc. IEEE Inil. Conf.
Comput. Vis., pp. 14408-14417 2021. 9
[60] A. N. Angelopoulos, A. Kohli, S. Bates, M. Jordan, J. Malik, T. Alshaabi, Upadhyayula, and
Y. Romano, 'Image-to-imanl image regression with distribution-fice uncertainty quantification and applications in imaging," in Proc. Inil. Conf. Mach. Learn., 2022. 10
[61] J. Teneggi, M. Tivnan, J. W. Stayman, and J. Sulam, "How to trust your diffusion model: A convex
optimization approach to conformal risk control," 2023. 10
[62] D. Namhofer, A. Habring, M. Holler, and T. Pock, cnsterior-variance--be
error quantification for
inverse problems in imaging." " SIAM Imag. Sci., vol. 17, no. pp. 301-333, 2024. 10
[63] Y. Zeng, "ca-mod-gaan-pyorrch co-mod- mood-gan-pytorchh Sept. 2022. 16
Downloaded from https: //gp:s////thhb..../ com/ cowm/engrianyuu//////oo..ooo///e
12

---

Supplementary Materials
A Synthetic Gaussian priors
Algorithm captures how we construct the Gaussian priors used Sec. 4.
Algorithm 2 Gaussian prior generation
ux
(100)
NN(0, I 100)
2: (1000)
~ (N 1) for k = 1,..., 100
(100)
= -x% [ for k 1......1000 100
3: x
(100) -N for k = 1..... 100
4: Ux
5: IQ. R = QRdecomp(( (100), (200) 100
u1 122
(100)
6: 0
(100)
iQ for k
100
7:
8: for d 90, 80..... 10 do
9:
fx (d) = [plx (100)
load
(d) b (100) for k
10:
k
d
(100)
11:
U 0k,0.d for A: = I....,d d
QQ. R = QRdecomp((ee (0) u1 (d)
u (d)
12:
uE [Q): for ,,...,d
14: end for
We begin with dimension d 100, generating random mean px E R 100 and eigenvalues (Ag (x(100) === To construct the eigenvectors { (00) ) Jk=1, we perform QR decomposition on
(100)
100x 100 matrix with i... (O entries and set V (d)
(100) the kth column of Q. For each remaining
(d) E {90, 80, .... 10), we construct μxx {>E and un by truncating the previous quantities to ensure some level of continuity across
(d)
B Conditional Fréchet inception distance
We quantify posterior-approximee accuracy using conditional Fréchet inception distance (CFID) [38], which approximates the conditional Wasserstein distance
CWD & (y2((2 Pxy(
(B.1)
In (B.1), W2(pa, Pb) denotes the Wasserstein-22 distance between distributions and Pb. defined as
W2(Pa,Pb)
min P.D
Ea bll2),
(B.2)
where II(P.. Pb) Pa.b P. = f Pa,b db and Pb = Pa.o Pa.b with marginal distributions Pa and Pb- Similarly to how FID [39] is computed for marginal distributions, CFID approximates (B.1) for conditional distributions. In particular, the random vectors xc, 3. and y replaced by low-dimensional embeddings æ, 2. and y, generated by the convolutional layers of a deep network (e.g.. Inception V3 or VGG-16 or AlexNet), and the embedding distributions Paly and Pay are approximated by multivariate Gaussians. We compute CFID using the code from [45], which is described in [19].
Pa, da} denotes the set of joint distributions
C Additional MRI results for acceleration R 4
Table 5 shows PSNR, SSIM, LPIPS [50], and DISTS [51] for the P-sample average B.m, P E {1, 2, 4, 8, 16, 32) for 4. this case, the E2E--arNee attains the best PSNR and SSIM, while pcaGAN performs best in LPIPS and DISTS when P 2 and P. 1, respectively. These results,
13

---

Table 5: Average PSNR, SSIM, LPIPS, and DISTS of (ip) versus P for R 4 MRI
PSNR
Model
P-1
P-2
p
P-8
P-16
P--2
P-1
P-2
P-4
P-
P-16
P-32
E2E. VarNet (Sriam et al. (43])
39.90
0.9941
Langevin et al. (22])
0.9138
cGAN (Adier Oktem [16])
35.63
36.64
31.24
37.
97.73
37.82
0.9330
0.9445
0.9478
0.9480
0.9477
0.9475
pscGAN (Ohayon al. (29])
39:44
39.46
39.46
39:47
39:47
39:47
0.9558
0.9516
0.9539
0.9535
0,9533
0.9532
(Bendel [19])
0.9537
pcaGAN (Ours)
37.32
38.43
39.11
39:47
39.67
39.77
0.9463
0.9541
0.9557
0,9553
0.9546
0.9542
LPIPS.
DISTS)
Model
P-2
P-4
P-
P-16
P-
P-1
P-2
P-4
P-8
P-16
P-32
EZE- VurNet (Sriram et al. (43))
0.0316
0.0859
Langevin (Jalal et al. (22))
0.0545
0.0394
0.0336
0.0320
0.0317
0.0316
0.1116
0.0921
0.0828
0.0793
0.0781
0.0717
(Adier Oktem [16D
0.0273
0.0967
pecGAN (Ohayon et al. (29))
0.0245
0.0247
0.0248
0,0269
0.0244
0.0049
0.0767
0.0790
0.0801
0.0807
0.0810
0.0811
reGAN (Bendel et al. [19])
0.0175
0.0164
0.0188
0.0216
0.0235
0.0245
0.0446
0.0563
0.0667
0.0755
0.0809
0.0837
peaGAN (Ours)
0.0164
0.0159
0.0188
0,0214
0.0231
0.0342
0.0542
0.0548
0.0662
0.0754
0.0811
0.0843
PcaGAN
roGAN (Bendel)
PScGAN (Ohayon)
cGAN (Adler)
Langevin (Jalal)
(ours)
Average
(P-3e)
Sample
Truthh
E2E E2E-Variet
Sample
Figure C.I: Example MRI recoveries at R = 4. Arrows highlight meaningful variations.
in conjunction with the R = 8 results discussed in Sec. 4.3, show that pcaGAN yields a notable improvement over reGAN in all metrics.
Figure C. shows zoomed versions of two recoveries ã, and the sample average i,p with = 32.
D Implementation details
In each experiment, all cGANs were trained using the Adam optimizer with a learning rate of 10-3 81 0, and 32 = 0.99 as in [55].
D.1 Synthetic Gaussian recovery
cGAN training. We choose Bady = 10 rbbtcch = 64, P.c =P 2, and train for 100 epochs for both reGAN and pcaGAN. Running PyTorch server with 4 Tesla A100 GPUs, each with 82 GB of memory, the cGAN training for 100 takes approximately hours, with training time decreasing with smaller d. For pcaGAN, we choose K d for each d in this experiment (unless otherwise noted) and 3pca = 10-2
We exploit the Gaussian nature of the problem, constructing Go with two dense layers; one which takes y as input and one which takes as input. The output of each layer is added, yielding æ. Similarly, D& is comprised of a single dense layer which takes in the concatenation of / and y and outputs a sealar. We use this architecture for both reGAN and pcaGAN. Note that there is no listed license for reGAN.
cGAN architecture.
14

---

NPPC. For NPPC, we use the suggested hyperparameters from [31] and opt to train the MMSE network before training NPPC. We use the suggested architectures from their Gaussian denoising experiment and train for 100 epochs with rbbathh = 64. We leverage the authors' implementation in [36], modifying it for this Gaussian problem. There is no listed license for NPPC.
D.2 MNIST denoising
The MNIST dataset is available under the GNU general public license, which we respect through our use.
cGAN training. We choose Sadv 10 5 Pbbtchh = 64, Pre = 2, and train for 125 epochs for both reGAN and pcaGAN. Running PyTorch on server with 4 Tesla A100 GPUs, each with 82 GB of memory, the cGAN training for d 100 takes approximately 8 hours, with training time decreasing with smaller For peaGAN, we train two models, one with K and one with K 10. In both Fever = 25, and Eeval = 50.
cases, pca 10-1,
cGAN architecture. For both cGANs, the generator is the standard UNet which takes in the concatenation of y and code 2. The network consists of 3 pooling layers and 32 initial channels. The convolutions use kernel of size x 3, instance normalization, and leaky ReLU activations with negative slope of 0.1. For the encoder portion of the UNet, we use max pooling with a kernel size of reduce spatial resolution by factor of 2. For the decoder portion of the UNet, we use nearest-neighbor interpolation to increase spatial resolution by factor of 2. The discriminator is simply the encoder portion of the UNet with an additional dense layer appended to map the UNet's latent space to scalar output. Note that there is no listed license for reGAN.
NPPC. For NPPC, we do not modify the authors' implementation in [36] in any way. We first train the MMSE reconstruction network and then train the NPPC network. There is no listed license for NPPC.
D.3 Accelerated MRI
For our MRI experiments, we use the fastMRI dataset which is available under the MIT license, which we respect through our use.
Padv = 10-5, Spca = 10-2, Abattch = 2, Pre = K 1, Eevec 25, and E 50 for peaGAN. For reGAN, pscGAN and Adler's cGAN,
cGAN training. We choose
the hyperparameters and training procedure described in [19]. All models were trained for 100 epochs using the Adam optimizer [35] with learning rate of 10-3, 91 and B2 0.99, in [55]. Running PyTorch on a server with 4 Tesla A100 GPUs, each with 82 GB of memory, the training of each MRI cGAN took approximately day. Note that there is listed license for reGAN, pscGAN, or Adler and Öktem's cGAN.
cGAN architecture. All four eGANs used the same generator and discriminator architectures described in [19], except that Adler and Öktem's discriminator used extra input channels to facilitate the 3-input loss.
E2E-VarN VarNet. For the Sriram et al.'s E2E- VarNet [43], we use the same training procedure and
hyperparameters outlined in [22] with modification to the sampling pattern as in [19]. As in [22], we use the SENSE-based coil-combined image as the ground truth instead of the RSS image. The 12E-Vrreet is available under the MIT license, which we respect.
Langevin approach. For Jalal et al.'s MRI approach [22], we do modify the authors' implemen- tation from [46] other than replacing the default sampling pattern with the GRO sampling mask. We borrow both generated samples and results from [19]. The authors' code available under the MIT license, which we respect.
D.4 FFHQ Inpainting
For our inpainting experiment, we use the FFHQ dataset which available under the Creative Commons BY-NC-SA 4.0 license, which we respect through our use.
15

---

cGAN training, We choose Padv - 10-3, Spea 10-3, Rbatch = 5. Pre 2, K 2. Eever 25, and Eeval = 50 for peaGAN. Running PyTorch on server with Tesla A100 GPUs, each with 82 GB of memory, the training of our cGAN took approximately 1.5 days.
cGAN architecture. As in [19], use the CoModGAN networks from [17] which extend the StyleGAN2 [34] network. The StyleGAN2 architecture is available under the NVIDIA Source Code License, which respect.
reGAN. We follow the training procedure outlined [19], only modifying the inpainting mask to be random. The total training time server with 4 NVIDIA A100 GPUs, each with 82 GB of memory, is roughly day. There is no listed license for reGAN.
PscGAN. We use the same training procedure outlined in [19], modifying the inpainting masks to be random and using the L2.r objective described briefty in Sec. 2 with P=8. The total training time on server with 4 NVIDIA A 4100 GPUs, each with 82 GB of memory, is roughly 1.5 days. There is no listed license for pscGAN.
We use the Py Torch implementation of CoModGAN from [63] and train the model. The total training time on a server with 4 NVIDIA A100 GPUs, each with 82 GB of memory, is roughly day. There is no listed license for CoModGAN, beyond the NVIDIA Source Code License.
CoModGAN.
D.4.1 Diffusion Methods
For all three diffusion methods, we use the pretrained weights from [26].
DPS, We use the suggested settings for the 256 x 256 FFHQ dataset and the code from the official PyTorch implementation [58]. We found the LPIPS LPIPS-minimizing step-size Ç via grid search over a 1000 image validation set. We generate sample for all 000 images in our test set, using a batch-size of and 1000 NFEs. The total generation time server with 4 NVIDIA A100 GPUs, each with 82 GB of memory, is roughly 9 days. There is no listed license for DPS.
DDNM. We use the code from the official PyTorch implementation [56]. We generate I sample for all 20 000 images in our test set, using a batch-size and 100 NFEs. The total generation time on server with NVIDIA A100 GPUs, each with 82 GB of memory, is roughly 1.5 days. There is no listed license for DDNM.
DDRM. We use the code from the official PoTorch implementation [57]. We generate I sample for all 20 000 images in our test set, using batch-size of and 20 NFEs. The total generation time on server with 4 NVIDIA A100 GPUs, each with 82 GB of memory, is roughly 5.5 hours. There is no listed license for DDRM.
16

---

E
Additional reconstruction plots
E.1 MNIST denoising
(a) PeaGAN (K
10)
(b) NPPC
1
2
83
04
05
02
Ø3
04
05
4
Tgc 20
4
4
4
4
4
4
4
4
21
C
8
Figure E.1: For (a) pcaGAN and (b) NPPC, this figure shows the true image æ, noisy measurements y. the conditional mean Pxly- principal eigenvectors {+x.) and two perturbations of jxyy
(a) pcaGAN (K
10)
(b) NPPC
2
3
44
55
22
V3
04
Ø5
0mmmn 3
Q--2
2
a
Q
3
4 3
Q-- 2
Q-
0-2
4-3
7
7 7
7
7 7
7
7
7
7
7
7
7
7
7
7
7 8
7
7
7
7
7
7
Figure E.2: For (a) peaGAN and (b) NPPC, this figure shows the true image æ. noisy measurements y, the conditional mean Mu principal eigenvectors {8x), , and two perturbations of jx
(a) pcaGAN (K = 10)
(b) NPPC
1
22
03
04
45
1
D2
04
Ü5
0=-3 33
a=-2
a=0
o=2
a=3
Q=-3
a=-2
a=0
a=2
o=3
Pelv
9
9
0
7
3
9
4
4
4
9
Pry ot
8
8
9
3
9
4
4
4
Figure E.3: For (a) pcaGAN and (b) NPPC, this figure shows the true image æ. noisy measurements y. the conditional mean Muly, principal eigenvectors { and two perturbations of Mxyy
17

---

E.2
MRI at acceleration R = 4
E2E-VarNet
pcaGAN (ours)
rcGAN (Bendel)
PSCGAN Ohayyn))
cGAN (Adler)
Langevin (Jalal)
Std. Dev. Mag
7e-05
0e-C0
Average
(P-32)
Average (P-4)
Average (P-2)
Truith
Sample
Sample
Figure E.4: Example R = 4 MRI reconstruction. Row one: pixel-wise SD with P = 32, Row two: Ci,, with P 32, Row three: H.n, with Row four: 20,, with P 2, Rows five and six: posterior samples. The arrows indicate regions of meaningful variation across posterior samples.
18

---

E2E-VarNet
PcaGAN (ours)
rcGAN (Bendel)
PECGAN Ohayon)
cGAN (Adler)
Langevin (Jalal)
Std. Dev. Map
0e:00
Average (P-32)
Average (P-4)
Average (P-2)
Sample
Truth
Sample
Figure E.5; Example At = 4 MRI reconstruction. Row one: pixel-wise SD with P = 32, Row two: Lip, with P 32, Row three: E.P, with Row four: E.e, with P = 2, Rows five and six: posterior samples. The arrows indicate regions of meaningful variation across posterior samples.
19

---

E.3
MRI at acceleration R 8
E2E-VarNet
pcaGAN (ours)
rcGAN (Bendel)
PSCGAN Ohayon)
cGAN (Adler)
Langevin (Jalal)
Std. Dev. Mag
36.05
0e-C0
Average
(P-32)
Average (P-4)
Average (P-2)
Truith
Sample
Sample
Figure E.6: Example R = 8 MRI reconstruction. Row one: pixel-wise SD with P = 32, Row two: Ci,, with P 32, Row three: R.,, with Row four: 20,, with P 2, Rows five and six: posterior samples. The arrows indicate regions of meaningful variation across posterior samples.
20

---

E2E-VarNet
PcaGAN
rcGAN (Bendel)
PECGAN Ohayon)
cGAN (Adler)
Langevin (Jalal)
Std. Dev. Map
0e-00
Average (P-32)
Average (P-4)
Average (P-2)
Sample
Sample
Figure E.7: Example Rt = 8 MRI reconstruction. Row one: pixel-wise SD with P = 32, Row two: Lip, with P 32, Row three: E.P, with Row four: E.P, with P 2, Rows five and six: posterior samples. The arrows indicate regions of meaningful variation across posterior samples.
21

---

E,4
Inpainting
Original
pcaGAN
Masked
ToGAN
CoModGAN
DPS
DDNM
DDRM
Figure E.8: Example of inpainting a randomly generated mask on a 256 256 FFHQ face image.
22

---

Original
poRGAN
Masked
reGAN
CoModGANN
DPS
DDNM
DDRM
Figure E.9: Example of inpainting a randomly generated mask on a 256 x 256 FFHQ face image.
23

---

Original
PoRGAN
Masked
reGAN
CoModGANN
DPS
DDNM
DDRM
Figure E.10: Example of inpainting a randomly generated mask on a 256 x 256 FFHQ face image.
24

---

Original
PoRGAN
Masked
reGAN
CoMMdGGAN
DPS
DDNM
DDRM
Figure E.11: Example of inpainting a randomly generated mask on a 256 x 256 FFHQ face image.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contribution and results described in the abstract and introduction detailed in Section and Section 4, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
We do not provide any theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental settings are described in Section 4 with additional details given in Appendix D. Should this work accepted, we will release our source code and link to in the camera-ready submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Upon acceptance, we will release source code to reproduce each experiment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.co wttpccccppcc/ceeeeeeeccecccccltt httpc//cccccc.dce///e/e////////e meuidde/Cooeeutlee ieabbbisssionPe sssisnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Experimental settings are described in Section with additional details given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are omitted due to computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are described in Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics https:/ http:////aarsr...c//////////////
Answer: [Yes]
Justification:
This research abides by the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are discussed in the Introduction (our method facilitates uncertainty quantification, fairness, and perception-distortion trade-offs) and MRI-specific impacts are discussed in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licenses discussed in Appendix D and are properly respected. Credit is given for the various data/code/models we use throughout the paper when appropriate.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswith thcode www.coo//a/////c condsatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Upon release, the source code will be well-documented with instructions for reproducing the experimental results and extending the code.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This research did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "wd6KrQbWKi", "reviewer": "Reviewer_n2zA", "summary": "This work considers using posterior sampling to solve inverse problems via conditional GANs. Previous work has shown that cGANs are competitive in terms of reconstruction quality as compared to other generative models to sample from the posterior (e.g., diffusion models), while also having the advantage that, once trained, sampling is fast as it simply requires a forward pass. In this paper, the authors build on such approaches by adding regularization terms to encourage the cGAN to learn K principal components of the posterior covariance matrix, where K is user-defined quantity. Due to the computational burden of such an approach, a training scheme is devised where learning of the eigenvectors and eigenvalues is done after optimizing a regularized cGAN objective. The method is validated on a variety of linear inverse problems (denoising, accelerated MRI, inpainting) and datasets (mnist, fastmri, ffhq).", "questions": "- As the parameter $K$ is fixed prior to training with knowledge of the inverse problem, it would be good to discuss how the choice of $K$ relates to the particular inverse problem itself. For example, how does the choice of $K$ relate to how challenging the inverse problem is? I presume that ill-posedness plays a role here (e.g., comparing a linear inverse problem with a non-linear inverse problem, such as phase retrieval).", "limitations": "The authors discuss some of the above weaknesses I mentioned near the end of their paper, which I appreciate. They also discuss other issues regarding how to properly incorporate this approach for rigorous uncertainty quantification.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to read. In particular, the idea of adding regularization to fit to principal components of the posterior distribution is a simple and natural idea. It is also one of importance, as being able to visualize uncertainty in reconstructions is crucial in understanding which regions of the reconstruction can be “trusted”\n- The method is validated on a nice range of problem types and datasets.\n- Using a conditional cGAN allows for fast sample generation, especially as compared to diffusion models. \n- The additional use of the pca regularization term appears to improve performance of rcGAN, while also allowing for visualization of directions of uncertainty.", "weaknesses": "- The proposed approach is a slight, potentially somewhat incremental, improvement to the prior rcGAN approach of [21]. In particular, rcGAN also aims to better model the posterior distribution by matching certain statistics of the posterior mean and covariance. This work takes this reasoning a step further by aiming to fit the principle components of the covariance. This has also been done (in a different way) in prior work [29], that the authors compare to. The bonus for this work, however, is that the authors can sample quickly, whereas in [29] sampling is not possible.\n- Coupled with the concern regarding novelty, the additional regularization terms are computationally expensive to optimize, especially for high-dimensional problems. This is evidenced by, for example, only being able to set K = 1 in the MRI experiments and K = 2 in the large-scale inpainting experiments, and is discussed by the authors in their limitations. Moreover, the training strategy becomes a bit more complicated in order to aid in efficiency, as one needs to first optimize the rcGAN objective and then integrate the PCA terms into the learning process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and time spent reviewing.\n\n- **Reviewer**: Slight, potentially somewhat incremental, improvement to the prior rcGAN approach of [21].\n\n    **Response**: Please see the global rebuttal.\n\n- **Reviewer**: Additional regularization terms are computationally expensive to optimize in high-dimensional problems, forcing $K=1$ for multicoil MRI and $K=2$ for 256x256 image inpainting.\n\n    **Response**: True, our limited GPU memory forced us to use small $K$. \n    But even with $K=1$ and $K=2$, pcaGAN achieved noticeable performance gains over rcGAN (see global rebuttal).\n    And, with the memory advances predicted by Moore's law, pcaGAN's advantage will grow.\n    Regarding time complexity, pcaGAN is only 1.3x more expensive than rcGAN in training, and equal to rcGAN in inference.\n    \n- **Reviewer**: How does the choice of $K$ relate to the inverse problem?\n\n    **Response**: Since $K$ is the number of posterior covariance principal components that pcaGAN aims to match, the effect of $K$ in pcaGAN is similar in spirit to the effect of $K$ in standard PCA. For example, pcaGAN's recovery performance should be non-decreasing in $K$, as corroborated by Fig.1b.  But when the singular values of the posterior covariance matrix decay quickly, a small $K$ will suffice for good recovery performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkAsN9EcNo", "reviewer": "Reviewer_s7hc", "summary": "The paper focuses on learning posterior samplers for inverse problems, specifically aiming to accurately estimate the principal components of the posterior covariance matrix. To achieve this, the authors consider training a conditional generative adversarial network (cGAN), building upon previous work [21]. They introduce two new regularization terms: one for the eigenvalues and another for the eigenvectors. Recognizing the challenge of estimating these statistics from a single ground truth solution per input measurement, the authors propose gradually adding these terms during training, while utilizing intermediate results to estimate the statistics.\nThe authors present a series of experiments to demonstrate their method, ranging from toy examples to real-world MRI data. Comparisons are primarily made with rcGAN and NPPC.", "questions": "* Please address the weaknesses above.\n* I suggest conducting an ablation study to assess the impact of key priors in the method, such as the different terms and $P_{rc}$.\n* I suggest expanding the experiments to include comparisons with more recent posterior samplers in standard image inverse problems, like super-resolution and inpainting.\n* Please provide plots illustrating the variation in the generated statistics over the training epochs.\n* Do the additional regularization terms theoretically change the optimal solution, or does it remain the same?", "limitations": "See weaknesses.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* The paper is well-written, with clear motivation, method presentation, and background (mostly).\n* The detailed algorithm description, beyond pseudocode, significantly enhances understanding and reproducibility.\n* The authors nicely addressed the problem of estimating the desired statistics from a single ground truth vector.\n* The authors provide sufficient evidence to demonstrate their method (but additional evidence could further strengthen their claims).", "weaknesses": "Major Issues:\n* The paper lacks an ablation study which is crucial. While the authors followed [21] and set $P_{rc}=2$, I still expect an ablation study on this parameter, especially since using 2 samples seems insufficient to estimate statistics properly. \nFurthermore, there is no ablation study regarding the different regularization terms and their relative weights. Given the newly regularization terms, it is possible that other terms, introduced in [21], are redundant, especially when $K=d$. Since accurately recovering the eigenvalues implies perfect recovery of the trace, for example.  \n\n* I find the comparison to competing methods limited. I suggest adding comparisons to additional recent posterior sampling techniques. For example, the authors of NPPC presented comparison to DDRM, DDNM, RePaint, and MAT.\n\n*  Unlike work [21], the paper present no proposition or theoretical guarantees regarding the estimated statistics, limiting the novelty of the paper. \n\n* The description of work [21] is imprecise. Specifically, the usage of the notations $P$ and $P_{rc}$ is incorrect, leading to confusion.\n\n* The authors state in lines 103-104 that they apply the regularization term (11) only after allowing the generated posterior mean to converge near the true mean. This raises the question of whether (11) is necessary or if its impact is marginal, as it only leads to small variations in the generated posterior mean. The same question applies to the generated eigenvalues. I suggest adding plots to illustrate the variation in generated statistics over training epochs, comparing them to their final values or their values before applying the new regularization term during training.\n\n* While the relevance of NPPC to this work is clear, the experimental comparison seems unfair as NPPC is not designed for generation tasks. Furthermore, the authors demonstrate a speedup against DPS, but neglect to discuss the computational load of their method compared to NPPC.\n\nMinor:\n* The authors in lines 24-26 correctly point out that bias can be an issue in single-sample recovery. However, this is also a potential concern for recovery methods that yield multiple solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and time spent reviewing.\n\n- **Reviewer**: No ablation study of the rcGAN parameter $P_{rc}$, especially since $P_{rc}=2$ seems insufficient to estimate statistics properly.\n\n    **Response**: We use $P_{rc}=2$ when training an rcGAN model, which is the first step in training pcaGAN. To train rcGAN, the paper [21] analytically proved that $P_{rc}=2$ is sufficient and empirically found that $P_{rc}=2$ is optimal.\n    In the second and third stages of pcaGAN training, when the eigenvector and eigenvalue regularizations are introduced, we see no good reason to change $P_{rc}$ (nor any of the other rcGAN regularization parameters) because doing so would likely destabilize the training.\n    Given the clear rationale for $P_{rc}=2$, we feel that an ablation study is unnecessary, especially since it would take many weeks of compute.\n\n- **Reviewer**: The rcGAN posterior variance regularization is redundant when $K=d$.\n\n    **Response**: True, but pcaGAN is typically used with $K<d$, in which case variance regularization is essential.  Even when $K=d$, Fig.1b shows that there is no penalty to this redundancy, and Fig.1c shows that pcaGAN gives a significant improvement over both rcGAN and NPPC.\n\n- **Reviewer**: The comparison to competing methods is limited.  I suggest adding comparison to DDRM, DDNM, RePaint, and MAT.\n\n    **Response**: As described in the global rebuttal, have added results for DDRM and DDNM to the rebuttal pdf, and we have tuned DPS since the original submission.\n    We will include these results in our final paper if accepted.\n\n- **Reviewer**: No theoretical guarantees, limiting the novelty of the paper.\n\n    **Response**: pcaGAN's novelty is that it's the first posterior-sampling method to aim for correctness in the principal components of the posterior covariance matrix.  That novelty is independent of the presence/absence of theoretical guarantees.\n\n- **Reviewer**: The description of work [21] is imprecise. Specifically, the usage of the notations $P$ and $P_{rc}$ is incorrect.\n\n    **Response**: Definitions (7)-(9) are stated in terms of a generic variable $P$, and their usage in (6) correctly invokes $P=P_{rc}$ for rcGAN.  That said, we do see one typo in (7), where $P_{rc}$ should be replaced by $P$.  Is this what the reviewer was referring to?  \n\n- **Reviewer**: There is a question of whether the eigenvector regularization (11) is necessary or if its impact is marginal. The same question applies to the eigenvalue regularization (12). I suggest adding plots to illustrate the variation in generated statistics over training epochs.\n\n    **Response**: There must be a misunderstanding here.  The regularizations in (11)-(12) are what distinguishes the proposed pcaGAN from the existing rcGAN.  To understand the impact of (11)-(12) on performance, one needs only to compare the rcGAN and pcaGAN entries in our figures and tables. \n    A summary of the performance gains is given in the global rebuttal.\n\n- **Reviewer**: The experimental comparison to NPPC seems unfair as NPPC is not designed for generation tasks.\n\n    **Response**: There must be a misunderstanding here, since we evaluate NPPC only for non-generative tasks.  Recall that NPPC aims to directly estimate the principal components of the posterior covariance matrix, and that the NPPC paper [29] evaluated performance using Wasserstein-2 distance, residual MSE, and residual error magnitude.  We compare to NPPC using exactly these metrics, which is fair.\n\n- **Reviewer**: The authors demonstrate a speedup against DPS, but not NPPC.\n\n    **Response**: That's a good point.  For a batch of 128 MNIST samples, NPPC's inference time is 112 ms while pcaGAN's is 118 ms, a 5\\% increase.  These results will be added to Tab.1.\n\n- **Reviewer**: Do the added regularization terms theoretically change the optimal solution, or does it remain the same?\n\n    **Response**: The added regularization terms do not change the optimal solution.  Recall that the pcaGAN regularizations aim to make the cGAN's $y$-conditional mean, trace-covariance, covariance eigenvectors, and covariance eigenvalues all agree with the respective statistics of the *true* posterior distribution.  So if the cGAN's output statistics are optimal, then the regularizations will have no effect.  But if the cGAN's $y$-conditional statistics are suboptimal, the regularizations will try to to correct them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zehCnEzFd0", "reviewer": "Reviewer_GSQK", "summary": "The authors address inverse problems and aim to improve GAN-based posterior samplers by adopting regularization to correct the principal components of the covariance matrix of the approximated posterior distribution.", "questions": "1. While diffusion models are slower compared to GANs, they have demonstrated success over GAN-based models in enhancing posterior richness for conditional tasks. A fair comparison with diffusion models would be valuable, as it remains uncertain whether the proposed method outperforms diffusion models. For faster sampling approaches, the authors might consider DDIM[1], which accelerates the diffusion process.\n\nReferences:\n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\" (2020).", "limitations": "The authors explained the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The idea presented in the paper is novel and clear, especially given the tendency of GAN-based posterior samplers to experience mode collapse. This serves as a strong motivation for the proposed method in the paper.\n3. The authors conducted empirical study with a variety of datasets.", "weaknesses": "1. For each dataset in the empirical study, a different set of baseline methods was chosen, possibly due to the requirement of pretraining. However, for a fair comparison, this is necessary for a complete research.\n2. The authors claim that the proposed method outperforms existing diffusion models (line 52). However, the experiments conducted with diffusion models are not comparable due to the much smaller test size (line 210 in MRI experiments and line 235 in inpainting experiments).\n3. Figure 4 presents inpainting posterior samples, but for the diffusion model referred to as DPS, the masking was not applied, which results in incorrect and misleading visualization.\n4. Some minor errors:\n    - In line 28, “There the goal\" should be revised.\n    - In lines 33-34, there is a grammar mistake.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and time spent reviewing.\n\n- **Reviewer**: For each dataset in the empirical study, a different set of baseline methods was chosen.\n\n    **Response**: Yes, this was done because there exists no single baseline method that is state-of-the-art for all datasets considered.  Also, we wanted to compare against cGANs (because we propose a cGAN), diffusion-based methods (because they are recently popular), and NPPC (which offers a direct way to estimate posterior covariance principal components).  We don't think that using a strong, dataset-specific set of baselines would be a weakness.\n\n- **Reviewer**: The experiments conducted with diffusion models are not comparable due to the much smaller test size (line 210 in MRI experiments and line 235 in inpainting experiments).\n\n    **Response**: There must be a misunderstanding, since we always use a common test dataset when comparing any collection of methods.  For example, the CFID$^1$ and FID columns in Tab.2 evaluate all five methods on a common 72-image dataset.  Likewise, the CFID column in Tab.3 evaluates all four methods on a common 51k-sample dataset, and the FID column in Tab.3 evaluates all four methods on a common 6k-sample dataset. \n\n    Line 210 explains that, beyond the CFID$^1$ experiment that compared both cGANs and diffusion methods, we conducted *additional* cGAN-only experiments that used larger test datasets, which led to the CFID$^2$ and CFID$^3$ columns in Tab.2.  This was needed because Jalal's diffusion sampler is very slow to generate samples.  For example, evaluating Jalal for the CFID$^3$ columns in Tab.2 would require 72 days on our hardware.\n    Line 235 simply says that large datasets were used to avoid small-sample bias.\n\n\n- **Reviewer**: Figure 4 presents inpainting posterior samples, but for DPS, the masking was not applied, which results in incorrect and misleading visualization.\n\n    **Response**: Yes, our original submission used DPS under a default parameter setting, which was suboptimal for our experimental setup. The global rebuttal describes how DPS has since been tuned, and the rebuttal pdf shows the performance of tuned DPS.\n\n- **Reviewer**: A fair comparison with diffusion models would be valuable, as it remains uncertain whether the proposed method outperforms diffusion models. DDIM could be considered to accelerate sampling.\n\n    **Response**: Yes, we agree. The global rebuttal adds a comparison to DDRM and DDNM, which both use DDIM for acceleration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MUxbLk7pPk", "reviewer": "Reviewer_cM4t", "summary": "To solve image inverse problems (denoising, super resolution, etc.), many works attempt to sample from the posterior distribution of ground truth images given a degraded measurement. One line of works propose to train a stochastic neural network as a CGAN (conditioned on the degraded image and an additional random seed as inputs), while regularizing the conditional mean and the conditional covariance matrix of the outputs. This paper proposes yet another, improved regularization term for such a CGAN approach. Specifically, the authors propose to correct (regularize) the K principal components of the conditional covariance matrix in additional to the conditional mean and the trace of the covariance.", "questions": "Please address each of the weaknesses.", "limitations": "The authors do provide a dedicated limitations section, but I observe additional limitations in this paper which are not addressed, such as incremental novelty of the proposed algorithm over previous work, limited experimental section (comparing only to one predecessor), and more, as explained in the weaknesses section.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper is mostly written well. The proposed method is demonstrated on toy examples (such as Gaussian random signals and MNIST images), which I find appealing to gain intuition into the advantages of the proposed approach. The paper shows results on MRI reconstruction, which is not very common to report but an incredibly important inverse problem to solve.", "weaknesses": "Even though I personally like this type of work and the proposed approach, unfortunately I see several major weaknesses in this paper, starting even with wrong/inaccurate claims in the introduction & motivation. I'll list some of the weaknesses below.\n\n1) I disagree with most of the claims made in L17 onwards regarding the shortcomings of point estimators (introduction). The perception-distortion tradeoff theorem in [6] does not prove anything specific about point estimators (a claim made in L20-L21). In fact, this theorem proves that distortion can only be at odds with perceptual quality (but now always), regardless of whether the estimator is \"deterministic\" or not. Moreover, [6] defines perceptual quality as the statistical distance (e.g., KL) between $p_{\\hat{X}}$ (the distribution of the reconstructions) and $p_{X}$ (the distribution of the source images), and not as the distance between $\\hat{x}$ and the manifold of natural images (as claimed in L22). What is the definition of such a distance? Lastly, the perception-distortion tradeoff is not always a strict one, as hinted in L21. This tradeoff  depends on the distortion measure, as discussed by the authors of [6]. Some distortion measures (e.g., \"perceptual\" losses in the deep image features space) virtually do not contradict with perceptual quality (as demonstrated experimentally in [6]). I would be pleased to see a complete revision of this paragraph in which the errors are corrected, or otherwise a clear explanation for why am I wrong. This is the entry point of the paper, it should at least be accurate.\n\n2) I also do not agree with L27: \"to address these shortcomings, posterior-sampling-based image recovery has been proposed\". Posterior sampling as a solution for inverse problems has been proposed and suggested, e.g., with Langevin dynamics (though, it didn't work properly for high-dimensional data) much before we knew anything about the perception-distortion tradeoff or fairness issues in image reconstruction. As I understand it, the true motivation for the rise of posterior-sampling is the following: We are often dealing with ill-posed inverse problems, and performing posterior sampling allows us to conduct uncertainty quantification of the inverse task at hand, explore the space of solutions visually, and make more informed decisions in down-stream tasks. It has also been shown that stochastic estimators are more robust to adversarial attacks [1]. I don't see why the rise of posterior sampling is linked to the perception-distortion tradeoff and/or to fairness. Please provide evidence (citations and explanation) if that's the case.\n\n3) L30: \"posterior samples will display attributes that reflect minority and majority populations in the training data\". As far as I know, this is often incorrect. In fact, we may often require to produce an intractable amount of posterior samples to observe an outcome for a minority. Please see [2] as an example.\n\n4) L43: You are missing citations (and comparison) with highly related work that proposed to use posterior sampling using a CGAN with conditional mean and conditional covariance regularizations, prior to the work of Bendel et al. For example, [3] and [4]. In fact, the work of Bendel et al. is based on the idea proposed in [2] (to regularize the neural network by enforcing agreement of the average posterior sample with the posterior mean), and in [3] (to use a second-moment penalty of the covariance matrix as well), as you suggest in L75. As far as I understand, your algorithm is a different (and maybe improved) version of [4], but I don't know which one is better since a comparison with [3] and [4] is not reported in the paper. Please address the differences in your work so I could better understand why it's not appropriate to cite such papers and/or include them in your analysis.\n\n5) I would suggest explaining L76-L79 better, by adding appropriate definitions for all the terminology used. For instance, the definition of a $P_{rc}$-sample supervised-$l_{1}$ loss is missing and unclear. I would be happy to see a more self-contained background section. Make sure every notation you are using is clearly defined before using it.\n\n6) Some results seem wrong to me. For example, the results of DPS in Figure 4. I have been using DPS in many different image inverse problems, and never observed such inconsistencies between the outputs and the inputs (see, e.g., Figure 14 in the DPS paper appendices, which shows almost perfect consistency with the measurements for severe inpainting tasks). The FID results of DPS also seem odd to me. I find it very hard to believe that a CGAN algorithm would outperform DPS, given that the hyper-parameters of DPS are properly tuned. Can you please provide the code you used to execute these experiments so I could verify them? Maybe the hyperparemeters you used to run DPS are suboptimal?\n\n7) The contribution of this work is quite limited in my opinion. The proposed algorithm only slightly improves its predecessor (Badler et al.), and the experimental evaluation is limited. A comparison with more works is missing, such as [3] and [4], and other, more recent state-of-the-art stochastic image restoration methods that produce better results than DPS. What about DDNM, DDRM, etc.? Why are these not included in the analysis? (besides the fact that they don't \"promise\" posterior sampling). I would be happy if you could clarify the contribution of your work and its impact on the field, as it seems quite incremental to me.\n\n\n[1] Reasons for the Superiority of Stochastic Estimators over Deterministic Ones: Robustness, Consistency and Perceptual Quality, Guy Ohayon et al., ICML 2023\n\n[2] From Posterior Sampling to Meaningful Diversity in Image Restoration, Noa Cohen et al, ICLR 2024.\n\n[3] High Perceptual Quality Image Denoising with a Posterior Sampling CGAN, Guy Ohayon et al., ICCV workshops 2021.\n\n[4] High-Perceptual Quality JPEG Decoding via Posterior Sampling, Sean Man et al, CVPR workshops 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and time spent reviewing.\n\n- **Reviewer**: I disagree with most of the claims made regarding the shortcomings of point estimators. The perception-distortion (PD) tradeoff theorem in [6] does not prove anything specific about point estimators.\n\n    **Response**: We'll try to explain again.  We'll refer to the full paper\n    >    [PD] https://arxiv.org/pdf/1711.06077v4\n    \n    not the IEEE version [6] missing figures and appendices.\n\n    [PD] considers the setting in which a *single* (possibly random) image estimate $\\hat{X}(y)$ is constructed from a given measurement $y$.\n    Examples in [PD] include the conditional-mean estimate $\\hat{X}(y)=E\\\\{X|Y=y\\\\}$, which achieves optimal MSE distortion but suboptimal perceptual quality, and the \"conditional draw\" estimator, where $\\hat{X}(y)$ is a single random draw from the posterior $p(X|Y=y)$, which achieves perfect perceptual quality but suboptimal distortion (see Thm.3 and Fig.7 in [PD]).\n    The key takeaway from the PD tradeoff is that no $\\hat{X}(y)$ can *simultaneously* achieve optimal distortion and perfect perceptual quality.\n    To achieve $K$ points along the tradeoff, [PD] proposed to train a GAN $K$ times, with a different loss each time. \n\n    Our point is that by drawing *many* posterior samples $\\\\{\\hat{X}\\_i(y)\\\\}\\_{i=1}^P$, where $\\hat{X}\\_i(y)\\sim$ iid $p(X|Y=y)$, one circumvents the PD tradeoff in the sense that\n    optimal MSE distortion is attained by the sample-mean estimator $\\frac{1}{P}\\\\sum_{i=1}^P \\hat{X}\\_i(y)$ as $P\\rightarrow\\infty$, while perfect perceptual quality is attained by $\\hat{X}_i(y)$ for any $i$. \n    Thus, given $\\\\{\\hat{X}\\_i(y)\\\\}\\_{i=1}^P$ with large $P$, there is essentially no PD tradeoff.\n    \n    The construction of a *single* (possibly random) estimate $\\hat{X}(y)$ from $y$ is  known as *point estimation*, even when $\\hat{X}(y)$ is a random draw from the posterior. See  \n    >   Lehmann and Casella, *Theory of Point Estimation*, Springer 2006.\n\n    Because the PD tradeoff assumes the use of a *single* estimate $\\hat{X}(y)$ from $y$, it proves a property of point estimation. \n    We hope this clarifies our point.\n\n    That said, the PD tradeoff is non-essential to motivating pcaGAN, and removing it from the paper would in no way diminish our contribution.\n    Thus, **we plan to remove the PD discussion from our Introduction**.\n\n- **Reviewer**: I don't see why the rise of posterior sampling is linked to the PD tradeoff.\n\n    **Response**: See above.  However, since this point is non-essential to motivating pcaGAN, **we plan to remove the PD discussion from our Introduction**.\n    \n- **Reviewer**: L30: \"posterior samples will display attributes that reflect minority and majority populations in the training data\". As far as I know, this is often incorrect.\n\n    **Response**: A proof of what we meant is given in App.A of [21].  However, since this statement is non-essential in motivating pcaGAN, **we plan to remove it**.\n\n- **Reviewer**: You are missing citations (and comparison) with highly related work proposed prior to rcGAN [21], for example [Ohayon'21] and [Man'24]. In fact, rcGAN [21] is based on [Ohayon'21] (regularize by enforcing agreement of the average posterior sample with the posterior mean) and on [Man'24] (use a second-moment penalty of the covariance matrix as well).\n\n    **Response**: We'd be happy to cite and compare to [Ohayon'21]; see the rebuttal pdf. As for [Man'24], it's highly specialized to the JPEG decoding application, and so it's not clear which modifications would be fair for MRI or inpainting.\n    We don't think that rcGAN is \"based on\" [Ohayon'21] or [Man'24], though.\n    The rcGAN paper [21] cites the 2017 Isola cGAN paper [17] for posterior-mean regularization.\n    And rcGAN predates [Man'24], as evident from the arXiv release dates.\n\n- **Reviewer**: I would suggest explaining L76-L79 better and including a self-contained background section\n\n    **Response**: Sure, we'd be happy to do so. \n\n- **Reviewer**: Some results seem wrong to me. For example, the results of DPS in Figure 4, which show inconsistencies between the outputs and the inputs. Also, I find it very hard to believe that a CGAN algorithm would outperform DPS when the hyper-parameters of DPS are properly tuned.\n\n    **Response**: Please see the global rebuttal and the rebuttal pdf.\n    Consistent with the reviewer's intuition, tuned DPS outperforms all existing cGANs under test.\n    However, it does not outperform the proposed pcaGAN, which attests to the importance of principal-component regularization.\n\n- **Reviewer**: The contribution of this work is quite limited in my opinion. 1) The proposed algorithm only slightly improves its predecessor, 2) A comparison with [Ohayon'21] and [Man'24] is missing, as well as recent works like DDNM and DDRM. 3) I would be happy if you could clarify the contribution of your work and its impact on the field, as it seems quite incremental to me.\n\n    **Response**: All three points are addressed in the global rebuttal.\n\n- **Reviewer**: I personally like this type of work and the proposed approach.\n\n    **Response**: We are glad to hear that you like our work.  We feel, however, that your final rating of \"3\" is at odds with this statement and with the remainder of your review.\n    For example, your first 3 critiques targeted non-essential statements in our Introduction.\n    Your 4th critique pertained to the missing references/comparisons to [Ohayon'21] and [Man'24], which are easily remedied.\n    Your 5th critique asked for more detail, which is easy to provide, and the remaining two critiques asked for comparisons to DDRM, DDNM, and tuned DPS, which we've included in the rebuttal.\n    With all this in mind, we kindly ask you to reconsider your rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable feedback and time spent reviewing.
In this global rebuttal, we address remarks made by multiple reviewers.

- **Reviewers n2zA and cM4t**: Incremental contribution over the prior work rcGAN [21]?

    **Response**: No.
    rcGAN enforces accuracy in the posterior mean and trace-covariance.
    The proposed pcaGAN goes beyond that by also forcing accuracy in the principal components of the posterior covariance matrix. 
    To our knowledge, pcaGAN is the first-ever generative method to do so. 
    Accomplishing this task was non-trivial, as it required novel eigenvalue and eigenvalue regularizations, as well as a novel training protocol to properly integrate them with the posterior mean and trace-covariance regularizations.
    
    There are two main advantages to pcaGAN over rcGAN.
    First, pcaGAN allows one to ascertain the *dominant modes of reconstruction uncertainty*. 
    Although this is also attempted by NPPC [34], the Gaussian and MNIST results in Figs.1-2 and Tab.1 show that the eigenvectors and eigenvalues estimated by pcaGAN are significantly more accurate.
    
    Second, pcaGAN generates significantly more accurate posterior samples than rcGAN.
    Summarizing the results from our experiments, we find the following improvements from rcGAN to pcaGAN:
    
    - 1e-2/1e-3=10x improvement in $\\mathcal{W}_2$ distance in Fig. 1
    
    - 1.69/1.29=1.3x or 30\% improvement in CFID$^3$ in Tab.2 at $R=4$ 
    
    - 3.83/3.21=1.16x or 16\% improvement in CFID$^3$ in Tab.2 at $R=8$
    
    - 0.0379/0.0344=1.1x or 10\% improvement in LPIPS in Tab.3
    
    - 0.0877/0.0799=1.1x or 10\% improvement in DISTS in Tab.3
    
    - 3.74/3.45=1.08 or 8\% improvement in CFID in Tab.4

    Finally, Tab.A of the rebuttal pdf shows that *pcaGAN outperformed DPS in all three quality metrics (CFID, FID, and LPIPS), whereas rcGAN did not outperform DPS in any quality metrics*.  Thus pcaGAN is a significant improvement over rcGAN.

- **Reviewers GSQK and cM4t**: The DPS images do not look right.  Was DPS properly tuned?

    **Response**: In our original submission, we set the DPS step-size $\zeta'=1.0$, as suggested by the DPS paper [28] for FFHQ inpainting. But, with this value of $\zeta'$, the DPS inpainting recoveries were inconsistent with the original images at the unmasked pixels.  Thus, when computing the CFID and FID values in Tab.4, we replaced the DPS-recovered pixels with the true ones at the unmasked locations, which greatly improved the CFID and FID values reported for DPS.
    
    Just after the NeurIPS deadline, we discovered that DPS can be improved by tuning $\zeta'$ to our experimental setup. 
    In particular, we found that, over the grid [0.5: 0.25: 3.0], the value $\zeta'=2.0$ minimizes LPIPS on a 1k-sample validation dataset.
    We used LPIPS because, as a feature-based metric, it correlates well with CFID and FID.
    (Note that CFID and FID themselves are inadmissible for tuning, because their values are unreliable with 1k-sample validation [36].)
    
    In the rebuttal pdf, Fig.A shows tuned-DPS inpainting recoveries, which look as expected.
    Also, Tab.A reports the performance of tuned-DPS on a 20k-sample test set.
    There we see that tuned-DPS outperforms all tested diffusion methods and cGANs in CFID, FID, and LPIPS *except* for pcaGAN.


- **Reviewers s7hc, GSQK, and cM4t**: How does pcaGAN compare to recent diffusion posterior samplers like DDRM, DDNM, or RePaint?

    **Response**: For multicoil fastMRI brain data, Jalal [24] is, to our knowledge, the state-of-the-art diffusion posterior sampler. We are unaware of any pretrained diffusion denoisers that can be used with samplers like DPS, DDRM, or DDNM to yield competitive performance on this dataset.
    We tried training our own diffusion model on the multicoil fastMRI brain data, but the results were not competitive. 
    
    For FFHQ face data, however, there are readily available pretrained diffusion models that are compatible with DPS, DDRM, DDNM, and RePaint. 
    Thus, in the rebuttal pdf, we add new results for DDNM (ICLR'23) and DDRM (NeurIPS'22).
    (We did not test RePaint, because DDNM has been shown to outperform it.)
    Fig.A shows example recoveries for pcaGAN, DPS (@ 1000 NFEs), DDNM (@ 100 NFEs), and DDRM (@ 20 NFEs).
    Tab.A shows CFID, FID, LPIPS, and the time to generate 40 samples.
    As expected from their NFEs, when going from DPS to DDNM to DDRM, we see an increase in recovery speed but a decrease in recovery quality.
    Notably, *pcaGAN outperforms all diffusion methods in both recovery quality and speed*.
    
    That said, comparisons between cGANs and diffusion methods should be taken with a grain of salt.
    cGANs are trained to handle a specific inverse problem, whereas diffusion samplers must handle a wide range of problems without retraining.
    Thus it is not surprising that a well-designed cGAN can outperform today's diffusion samplers. 
    On the other hand, because cGANs use one NFE but diffusion samplers use many, it would not be surprising if tomorrow's diffusion samplers outperformed pcaGAN.


- **Implementation details for the new results in the rebuttal pdf**:
    - For DDNM, we use the authors' implementation and the suggested DDIM setup of 100 steps with $\eta=0.85$.
    - For DDRM, we use the authors' implementation and the suggested DDIM setup of 20 steps with $\eta=0.85$. Per the authors' suggestion, we also set $\eta_b=1$.
    - DDRM, DDNM, and DPS all use the same FFHQ diffusion model.
    - For pscGAN, we use the authors' suggestion to set $\lambda_{MM}=0.001$ and $M=8$ during training. To ensure a fair comparison, other components of the training/validation/test process are identical across all cGANs.

Meta-review: In this paper, the authors present a method for using a conditional GAN to solve inverse problems by posterior sampling.  Whereas prior work enforces accuracy in the posterior mean and trace-covariance, this work further enforces accuracy in a user-specified number of principal components of the posterior covariance matrix.  The proposed work yields more accurate posterior samples than baselines with only modest increases in computational cost during training (relative to the most similar method to the proposed work).  Based on reviewer comments, there is some room for improvement in terms of thorough ablations and explanation of prior literature.  Based on the author-reviewer discussions, these concerns have been resolved enough to warrant acceptance.  The authors should implement the promised changes in the camera-ready.
Decision: Accept (poster)