A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
Authors: Matthew C Bendel
Rizwan Ahmad
Philip Schniter
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative adversarial network
inverse problems
posterior sampling
cGAN
GAN
TL;DR: A novel regularization technique that elevates conditional GANs to the state-of-the-art in posterior image sampling.

Abstract: In image recovery problems, one seeks to infer an image from distorted, incomplete, and/or noise-corrupted measurements.
Such problems arise in magnetic resonance imaging (MRI), computed tomography, deblurring, super-resolution, inpainting, phase retrieval, image-to-image translation, and other applications. Given a training set of signal/measurement pairs, we seek to do more than just produce one good image estimate. Rather, we aim to rapidly and accurately sample from the posterior distribution. To do this,
we propose a regularized conditional Wasserstein GAN that generates dozens of high-quality posterior samples per second. Our regularization comprises an $\ell_1$ penalty and an adaptively weighted standard-deviation reward. Using quantitative evaluation metrics like conditional Fréchet inception distance, we demonstrate that our method produces state-of-the-art posterior samples in both multicoil MRI and large-scale inpainting applications. The code for our model can be found here: https://github.com/matt-bendel/rcGAN.

A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
Matthew Bendel Dept. ECE
Rizwan Ahmad Dept. BME The Ohio State University Columbus, OH 43210 ahmad 460osu. edu
Philip Schniter Dept. ECE
The Ohio State University Columbus, OH 43210 bendel. 88osu...@..... edu
The Ohio State University Columbus, OH 43201 schniter. 1@osu. edu
Abstract
In image recovery problems, one seeks infer image from distorted, incomplete, and/or noise-corrupted measurements. Such problems arise in magnetic resonance imaging (MRT), computed tomography, deblurring, super-resolution, inpainting, phase retrieval, image-to-imase translation, and other applications. Given training set of signal/meaasurement pairs, we seek do more than just produce one good image estimate. Rather, we aim to rapidly and accurately sample from the posterior distribution. To do this, we propose a regularized conditional Wasserstein GAN that generates dozens of high-quality posterior samples per second. Our regularization comprises an ε penalty and an adaptively weighted standard-deviation reward. Using quantitative evaluation metrics like conditional Fréchet inception distance, we demonstrate that our method produces state-of-the-art posterior samples in both multicoil MRI and large-scale inpainting applications. The code for our model can found here: https : ///g//gttt///////
1 Introduction
We consider image recovery, where one observes measurements y = .M(x) the true image æ that may be masked, distorted, and/or corrupted æ with noise, and the goal is to infer from y. This includes linear inverse problems arising in, e.g., deblurring, super-resolution, inpainting, colorization, computed tomography (CT), and magnetic resonance imaging (MRI), where y Ax + with known linear operator A and noise w. But also includes non-linear inverse problems like those arising in phase-retrieval and dequantization. cases, it is impossible to perfectly infer æ from y.
well image-to tangge translation problems. In all
Image recovery is often posed finding the single "best" recovery ă, which is known as a point estimate æ [1]. But point estimation problematic due to the perception- distortion tradeoff [2], which establishes a fundamental tradeoff between distortion (defined as some distance between ă and x)) and perceptual quality (defined as some distance between z and the set clean images). For example, the minimum squared error (MMSE) recovery Emmse is optimal in terms of l distortion, but can be unrealistically smooth. Although one could instead compute an approximation of the maximum posteriori (MAP) estimate [3] or minimize some combination of perceptual and distortion losses, it's s unclear which combination would be most appropriate.
Another major limitation with point estimation is that it's unclear how certain one can be about the recovered T. For example, with deep-learning-basee recovery, it's possible to hallucinate nice-looking E. but is it correct? Quantifying the uncertainty in き is especially important in medical applications such as MRI, where a diagnosis must be made based on the measurements y. Rather than simply reporting our best guess of whether pathology is present absent based on ã. we might want to report the probability that the pathology is present (given all available data).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Yet another problem with point estimation is that the estimated @ could pose issues with faimess [4]. For example, say we are inpainting a face within an image. With a racially heterogeneous prior distribution, mmmse (being the posterior mean) will be biased towards the most predominant race. The same will be true of many other point estimates ät
To address the aforementioned limitations of point estimation, we focus on generating samples from the posterior distribution Pxyy(-1)). which represents the complete sate-of-kknoweee about given the measurements y. The posterior correctly fuses prior knowledge with measurement knowledge, thereby alleviating any concerns about fairesss (assuming the data used to represent the prior is fairly chosen [4]). Furthermore, the posterior directly facilitates uncertainty quantification via, e.g.. pixel-wise standard deviations or pathology detection probabilities (see Appendix A). Also, if it was important to report a single "good" recovery, then posterior sampling leads to an easy navigation of the perception-diisortionn tradeoff. For example, averaging P 2 posterior samples gives a close iiistorteddbut-overrmmn
approximation to the with small P. Additionally, posterior sampling unlocks other important capabilities such as adaptive acquisition [5] and counterfactual diagnosis [6].
mmmse with large P and sharp-but-more col--morr-dssoooeed
Concretely, given a training dataset of image/measuremmnt pairs {at, yo) our goal is to learn a generating function Go that, for a new y. maps random code vectors z ~ NOO,, I) to posterior sam- ples Gz(z. y) ~ Pxiy(- (3y). There exist several well-known approaches this task, with recent literature focusing on conditional generative adversarial networks (cGANs) [7, 8, 9. 10], conditional variational autoencoders (c (AEs) [11, 12, 13], conditional normalizing flows (cNFs) [14, 15. 16]. and score/diffusion/I Lngeevin based generative models [17, 18, 19, 20, 21J. Despite it being long- standing problem, posterior image sampling remains challenging. Although ccoreddiffusion/llanee approaches have dominated the recent literature due to advances in accuracy and diversity, their sample-generatin speeds remain urders-of-magniute
behind those of eGANs, c cVAEs, and cNFs.
We choose to focus on eGANs, which are typically regarded as generating samples of high quality but low diversity. Our proposed cGAN tackles the lack-of-diversio issue using a novel regularization that consists of supervised-E1 1 loss plus an adaptively weighted standard- deviatioo (SD) reward. This not a heuristic choice; we prove that our regularization enforces consistency with the true posterior mean and covariance under certain conditions.
Experimentally, we demonstrate completion/inpainting We consider these applications for three main reasons. First, uncertainty quantification in MRI, and fairness in sice-generation,, are both of paramount importance. Second, posterior-sampling has been well studied for both applications, and fine-tuned cGANs [9] and score/Langevin-basee approaches [19, 20] are readily available. Third, the linear operator " "A" manifests very differently in these two applications, which illustrates the versatility of our approach. To quantify performance, we focus on conditional Fréchet inception distance (CFID) [22]. which is principled way to quantify the difference between two high- -ighenimnni posterior distributions, although we also report other metrics. Our results show the proposed regularized cGAN (reGAN) outperforming existing cGANs [8, 23, 9] and the score/diffusion/lanng Laneevin approaches from [19] and (20] in all tested metrics, while generating samples ~ 10 times faster than [19, 20].
regularized cGAN accelerated MRI and large-scale face
2 Problem formulation and background
We build on the Wasserstein cGAN framework from [8]. The goal is to design a generator network Gg : Z x y + X such that, for fixed y, the random variable ø Gø(z, y) induced by P2 has a distribution that best matches the posterior (xl((-1)) in Wasserstein- 1 distance. Here, X', y, and Z denote the sets of x, y. and z, respectively, and is drawn independently of y.
The W distance can be expressed as
W1 (1 Pay (+)()))) = sup Eny(D(a, v)} - (Dy DEL,
where L1 denotes functions that are 1 I-Lipschitz with respect to their first argument and D X x + R is "critic" or "discriminator" that tries to distinguish between true x and generated 0 given
In MRI, the forward operator acts locally in the frequency domain but globally in the pixel domain, while in inpainting, the operator acts locally in the pixel domain but globally in the frequency domain.

---

Since we want the method to work for typical values of we define loss by taking an expectation of (1) over y ~ Py- Since the expectation commutes with the supremum in (1), we have [8]
Wy Pay b)))
sup (xy( y)} Ez,y ( (D y)
(2)
DEL1
sup Fx,,y D y) D(Go y).
(3)
DEL1
In practice, D is implemented by neural network DΦ with parameters Φ. and (0, Φ) are trained by alternately minimizing
Lad (0, ④) Ex y) Do(GG(z,))
(4)
with respect to and minimizing - Ladv (0, Φ)) + (g)( with respect to Φ, where Cgp (Φ) is a
gradient penalty that is used encourage D& E L1 [24]. Furthemore, the expectation over x and y in (4) is replaced in practice by a sample average over the training examples { y.) y.))
One of the main challenges with the cGAN framework in image recovery problems that, for each measurement example y,, there is only single image example æ;. Thus, with the previously described training methodology, there is no incentive for the generator to produce diverse samples sa-p, for fixed This can lead to the generator learning to ignore the code vector which
G(z,, l> causes a form of "mode collapse."
Although issues with stability and mode collapse are also present in unconditional GANs (uGANs) or discretely conditioned cGANs [25], the causes are fundamentally different than in continuously conditioned cGANs like ours. With continuously conditioned cGANs, there is only one example of valid D for each given y. whereas with uGANs there are many I; and with discretely conditioned cGANs there are many 2, for each conditioning class. As result, most strategies that are used to combat mode-collapse in uGANs [26, 27, 28] are not well suited to eGANs. For example, mini-batch discrimination strategies like MBSI) [29], where the discriminator aims to distinguish a mini-batch of true samples (a:e) } from mini-batch of generated samples [. don't work with eGANs because the posterior statistics are very different than the prior statistics.
To combat mode collapse in cGANs, Adler & Öktem [8] proposed use a three-input discriminator Dadler X''s x X' x x + IR and replace Ladv from (4) with the loss
Cadier (0, x) A Ex,2,.2.1 { ( (xx, Ge(zi, 4) + 2 D (Go y), x,
Dadler ( y), (e(22. v)).
(5)
which rewards variation between the first and second inputs to D They then proved that minimizing Cadler in place of Cadv does not compromise the Wasserstein cGAN objective, i.c., arg ming Cadler (0, Φ) arg ming (0,v(O, Φ). As we show in Section 4, this approach does prevent complete mode collapse, but it leaves much room for improvement.
3 Proposed method
3.1 Proposed regularization: supervised-(, plus SD reward
We now propose a novel cGAN regularization framework. To train the generator, we propose to solve arg ming ( Bav (0, Φ) + Li. (0, {sp)}
(6)
with appropriately chosen Badv, 3sD and Perain 2, where the regularizer
L,.DD. (0, {sD) - LI (0) CSD (0)
(7)
is constructed from the Ptrain-Sampl supervised-(, loss and standard-deviation (SD) reward terms
Pr. (0) A Ex.z,....2.. 20. { [||x (P
(8)
Poan ( 2Poan
Panin Ex iill
zply {1 3 (
(9)
and where { denote the generated samples and Dip, their P-sapple average:
e Ge(z,, y), e A DL_,
(10)
The use of supervised-E, loss and SD reward in (7) not heuristic. As shown in Proposition 3.1, encourages the samples (ã.) } to match the true posterior in both mean and covariance.
3

---

[-]
(d)
Ooet
Cest
25
Corect
Figure 1: The contours show the regularizer value versus 0 [M, a. T for four different regularizers: (a) supervised-E, plus SD reward with ASD = SSD at Pirain = 2. (b) supervised-Ei plus SD reward with ASD = -/S at Prrain= 8, (c) supervised-fz at Pirain = 8, and (d) supervised-fz plus variance reward at Ptrain = 8. The red star shows the true posterior parameters lug. oo]
Proposition 3.1. Suppose Ptrain and 0 has complete control over the y-conditional mean and covariance of Ct. Then the parameters 0 = arg ming L1, (8, (00))) with
(11)
7 Paria Prrant
yield generated statistics
Ealy (0.))l)) Exly(xl13 mmmse
(12a)
Cov zly (0.)))) = Covxly {
(12b)
when the elements of T; and x are independent Gaussian conditioned on y. Thus, minimizing Pean encourages the y-conditional mean and covariance of ã; match those of the true æ.
See Appendix B for proof. In imaging applications, T; and æ may not be independent Gaussian conditioned on and so the value of 3sD in (11) may not be appropriate. Thus we propose method automatically tune 3sp in Section 3.2.
Figure 1 shows a toy example with parameters 0 = [u. _ generator Go y) = μ + aze and NO, 1). giving generated posterior Pzy(zly) = (Z;; μi, o²). Assuming the true Pxly
(a; no; oî) Figs. ((a)-(b) show that, by minimizing the proposed Ci,sD, (0, 13 regularization over 0 jM, a]] for any Perain 2 2, recover the true 0,- [mo, They also show that the cost function steepens as Ptrain decreases, with agrees with our empirical finding that Ptrain = 2 tends to work best in practice.
We note that regularizing a cGAN with supervised-f, loss alone is not new; see, e.g., [7]. In fact, the use of supervised-,, loss is often preferred over &g in image recovery because it results in sharper, more visually pleasing results [30]. But regularizing a cGAN using supervised-E, loss alone can push the generator towards mode collapse, for reasons described below. For example, in [7], & -induccdd mode collapse led the authors to use dropout to induce generator variation, instead of random z;.
Why not supervised-22 regularization? One may wonder: loss plus an SD) reward in (7) and not more conventional choice like supervised-Ez loss plus variance reward, or even supervised-fz loss alone? We start by discussing the latter.
Why regularize using supervised-f, 1
The use of supervised-f2 regularization in cGAN can be found in [7]. In this case, to train the generator, one would aim to solve arg ming ( Ladv (0, +) + XL2(O)) with some À > 0 and
(2( Exyy {||= - E, (z H1²).
(13)
Ohayon et al. [23] revived this idea for the explicit purpose of fighting mode collapse. In practice, however, the Ez term in (13) must be implemented by finite-sample average, giving
L2, P.ai (0) A Ex,z,....z..y {||
E
Prnin Go(z,, v))
(14)
Poste
for some Purain 2 2. For example, Ohayon's 's implementation [31] used Perain = As we show in Proposition 3.2, L2, induces mode collapse rather than prevents it.
Proposition 3.2. Say Purain finite and 0 has complete control over the g-conditional mean and covariance of w. Then the parameters 0_ = arg ming L2, Prain (0) yield generated statistics
Ezly ( (0. )l [y = Ex Emmse
(15a)
Cov zly ( (0. (0))1) y) 0. Thus, minimizing Po.n encourages mode collapse.
(15b)

---

A50
AsD
PSD
Aso 1.2,%%
A0o
1.4%g
1.6,%
Ao 1.8,%b
40
Actual
94
10
10'
10P
10'
10a
10'
16"
10'
10
10'
10"
10'
Number of averaged outputs, P. on a log scale
Figure 2: Example PSNR of p;; versus P. the number of averaged outputs, for several training AsD
and MRI recovery at R 4. Also shown is the theoretical behavior for true-posterior samples.
The proof (see Appendix C) follows from the biri-vari decomposition of (14), i.e.,
L P (0) Posin
Ey [|mse Exiy ly (0) Peain tr(Covz,ly (2:)))) Exly [ 12 (16)
where emss
x mmmse is the MMSE error. Figure I(c) shows that L2, regularization causes mode collapse in the toy example, and Section 4.2 shows that it causes mode collapse MRI.
Why not supervised lg plus a variance reward? To mitigate P... incentive for mode-collapse, the second term in (16) could be canceled using variance reward, giving
Car Pesin (0) Pr (0) Peai
Lvar, Ponin (
(17)
with foaar Poain ( Punn-1 Zi=1
Pinin
E.,....,
E zp.......... y a (0) (ey (pr(()) (0)
(18)
since Appendix D shows that Loar, Posin (0) is unbiased estimator of the posterior trace-covariance:
Lvar, (0) Ey (/ (0) (0)
for any Pirain 2
(19)
However, the resulting Doai regularizer in (17) does not encourage the generated covariance to match the true posterior covariance, unlike the proposed L1.SD, Pram regularizer in (7) (recall Proposition 3.1). For the toy example, this behavior is visible in Fig. 1(d).
3.2 Auto-tuning SD reward weight AsD
We now propose a method to auto-tune AsD in (7) for given training dataset. Our approach is based on the observation that larger values of AoD tend to yield samples T; with more variation. But more variation is not necessarily better; we want samples with the correct amount of variation. To assess the correct amount of variation, we compare the expected lz error of the P-sample average "(p) to that of Ten. When } } are true-posterior samples, these errors follow particular relationship, as established by Proposition 3.3 below (see Appendix E for a proof).
Proposition 3.3. Say 2, (-ly(- independent samples of the true posterior and, for any P P 1, their P-sample average is eip) P EP=1 2. Then
Ep 2 E x1(201 y) P+ Emmse. which implies
&L
2P P+1
(20)
Experimentally we find that Ej/Ep grows with the SD reward weight ASD (See Fig. 2.) Thus, we propose to adjust ASD that the observed SNR-gain ratio ΣI/E,,, matches the correct ratio (2.Pai)) (Pval + 1) from (20), for some Pval 2 2 that does not need to match Perain. (We use Pval = 8 in Section 4.) In particular, at each training epoch 5, we approximate &p.,, and & as follows:
EP $ DV
Pial EIn Go. (Zi,u: Uv) x
x
(21)
& 1 EV
Ge.
(21 yo) xvl12.
(22)
with validation set ((I,, y ) V and i.i.d codes {z 1 We update AsD using gradient descent:
BsD = BSDDr USD (Lε E - 12Pval/ (Pval + 1))aB) By for T = 2,.......
(23)
with IsD,0 33% some MSD 0, and [m]s e 10 10
5

---

Numerical experiments
4.1 Conditional Fréchet inception distance
As previously stated, our goal is to train generator Go so that, for typical fixed values of y. the generated distribution Pay(-ly) matches the true posterior Pxy(:1 quantitative metric for evaluating performance with respect to this goal. For example, it is not enough that the generated samples are "accurate" in the sense of being close to the ground truth, nor enough that they "diverse" according to some heuristically chosen metric.
essential to have
We quantify
quality using the conditional Fréchet inception distance (CFID) [22], computationally efficient approximation to the conditional Wasserstein distance
CWD & (W2((yy (.))) Phy( y))}
(24)
In (24), W2(Pa. Pb) denotes the Wasserstein-2 distance between distributions Pa and pb. defined as min W2(Pa.Pb) P.
Eaabilla bll2),
(25)
where [p)). {Pa.b Ps Pa.b db and pb da, denotes the set of joint distributions Pa.b with prescribed marginals Pa and pb- Similar to how FID [32]-a popular uGAN metric- is computed, CFID approximates CWD (24) as follows: the random vectors 2. and y are replaced by low-dimensional embeddings æ. ã, and y. generated by the convolutional layers of a deep network, and ii) the embedding distributions Pxly and Paly are approximated by multivariate Gaussians. More details on CFID are given in Appendix F.
4.2
MRI experiments
We consider parallel MRI recovery, where the goal is to recover a complex-valued multicoil image æ from zero-filled measurements y (see Appendix G for details).
Data. For training data {z:}, we used the first 8 slices of all fastMRI [33] T2 brain training volumes with at least coils, cropping 384 384 pixels and compressing to 8 virtual coils [34], yielding 200 training images. Using the same procedure, 2 testing and 784 validation images obtained from the fastMRI T2 brain testing volumes. From the 2 testing images, 72 were randomly selected to evaluate the Langevin technique [19] in order to limit its sample generation days. To create the measurement ye- we transformed æ; to the Fourier domain, sampled using pscudo-random GRO patterns [35] at accelcration It = 4 and R = 8, and Fourier-transformed the zero-filled k-space measurements back the (complex, multicoil) image domain.
Architecture. We use a UNet [36] for our generator and a standard CNN for our generator, along with data-consistency in Appendix H. Architecture and training details given in Appendix I.
Competitors. We compare our cGAN to the Adler and Öktem's cGAN [8], Ohayon et al.'s cGAN [23], Jalal et al.'s fastMRI Langevin approach [19], and Sriram et al.'s VarNet [37].
The cGAN from [8] uses generator loss Padv Cadler (f, Φ)) and discriminator loss - Cadler (0, Φ) adv a1 (gp Cariri (Ø), while the cGAN from [23] uses generator loss Cadv (0, Φ) + ( and discriminator loss LLav (0, 00,00 Φ) + a1 (z) in the original paper. All cGANs used the same generator and discriminator architectures, except that [8] used extra discriminator input channels to facilitate the 3-input loss (5). For the fastMRI Langevin approach [19], did not modify the authors' implementation in [38] except
az ( Each used the value of Badv specified
the GRO sampling mask. For the E2E- VarNet [37], we use the same training procedure and hyperparameters outlined in [19] except that use the GRO sampling mask.
Testing.
To evaluate performance, we converted the multicoil outputs क; to complex-valus images using SENSE-based coil combining [39] with ASPPIRiT-sstimate [40] coil sensitivity maps, as described in Appendix G. Magnitude images were used to compute performance metries.
Results. Table shows CFID, FID, APSD (TP EP [ 2; 112)1/2 and 4-sample generation time at R E {4,8). (C)FID was computed using VGG-16 (not Inception-v3) to better align with radiologists' perceptions [41]. As previously described, the Langevin method was evaluated using only 72 test images. Because CFID is biased at small sample sizes [22], we re-evaluated the other

---

Table 1: Average MRI results at R E {4,8). CFID', FID, and APSD used 72 test samples and
P= 32, CFID2 used 2 376 test samples and P 8 8, and CFID3 used all 14 576 samples and P P=1
R 4
R
Model
CFID'
CFID-|
CFID')
FIDI
AFSD
Time (4),
CFID"L
CFID")
CFID"L
FIDI
APSD
Time
E2F: VatNer (Srimm al. (37])
6.61
7.82
6.31
0.0
316ma
Langevin (Jola) et al. (19])
5.29
6.12
5.9e-6
14
7.34
7.66-6
min
cGAN (Adler& & Oktem (SI)
6.39
4.27
3.82
5:25
3.9c-6
217 ms
10.10
6.30
5.72
10.77
1.7c-6
217 ms
oGAAN (Ohayon et (231)
4,06
3.27
2.95
6.45
7.2=-8
217
6.04
4.59
4.27
11.05
7.76-7
217 ms
cGAN (Ours)
3,10
1.54
1.29
3,75
3.8e-6
217
4.87
2.23
1.79
7.72
7
217
Table 2: Average PSNR, SSIM, LPIPS, and DISTS of Be) versus for R 4 MRI
PSNRT
SSIMI
Model
P-1
P-2
P-4
P-8
P-16
P 32
P-1
P-2
P-4
P-8
P-16
P-
EZE. VarNet (Stirm et al. [37])
39,93
0.9641
Langevin (aaal al. (191)
37.6
37.
0.8989
0.9138
0.9260
0.9281
0.9292
cGAN (Adler Oktem (8))
35.63
36.64
37.24
37.56
37.73
37.82
0.9330
0.9445
0.9478
0.9480
0.9477
0.9473
cGAN (Olayon (25)
39.44
39.46
39.46
39.47
39.47
39.47
0.9558
0.9546
0.9539
0.9535
0,933
0.9532
(Ours)
38.14
38.84
39.55
0.9440
0.9542
0,9537
0.9533
LPIPS]
DISTS
Model
P-1
P-2
P-4
p
P- 16
P-
P-1
P-2
P-
P-8
P-16 16
P-32
E2E VarNet (Stirm [37])
0.0316
0.0859
Langevin (Jalal et al. (199)
0.0555
0.0594
0.036
0.0200
0.0317
0.0316
0.1116
0.0921
0.0828
0.0793
0.0781
0.0777
cGAN (Adler & Oktem (8))
0.0285
0,2255
0.0233
0,0298
0.0316
0.0327
0.0972
0.0857
0.0878
0.0930
0.0967
0.0990
(Olayon (23))
00248
0.0767
0.0810
cGAN (Ours)
00175
0.0164
00188
01216
0.0235
0,0245
0.0546
0.0563
0.0667
0.0755
0.0809
0.0837
methods using all 2376 test images, and again using all 14 576 training and test images. Table 1 shows that our approach gave significantly better CFID and FID than the competitors. Also, the APSD of Ohayon et al.'s cGAN was an order-of- of-mgnniudd smaller than the others, indicating mode collapse. The cGANs generated samples 800 times faster than the Langevin approach from [19].
Tables 2 and 3 show PSNR, SSIM, LPIPS [42], and DISTS [43] for the P-sample average (()) P {1, 2, 4, 8, 16,32) 32) and R {4, 8). respectively. While the E2E-VarNee achieves the best PSNR at R E {4.8] and the best SSIM at R = 4. the proposed cGAN achieves the best LPIPS and DISTS performances at R E {4.8] 8} when P and the best SSIM at 8 when p 8. The P dependence can be explained by the perception-distorticn transitions from better pereeptual quality to lower l distortion. PSNR favors P - oo (e.g., &2 optimality) while the other metrics favor particular combinations of perceptual quality and distortion. The plots in Appendices K.1 and K.2 show zoomed-in versions of E.P, that visually demonstrate the perception-distoriee tradeoff at P E f1, 2,2,4 4, 32}: smaller P yield sharper images with more variability from the ground truth, while larger yield smoother reconstructions.
tradeoff [2]: as P increases, ((P)
Figure 3 shows zoomed versions of two posterior samples C. as well as pip, at P 32 and R= 8. The posterior samples show meaningful variations for the proposed method, essentially no variation for Ohayon et al.'s cGAN, and vertical or horizontal reconstruction artifacts for Adler & Öktem's cGAN and the Langevin method, respectively. The (Pp) plots show that these artifacts are mostly suppressed by sample averaging with large P.
Figure 4 shows examples of Pyy along with the corresponding pixel-wise absolute errors [ and pixel-wise SD images ( E (x - 12// for P = 32 and R = The absolute-error image for the Langevin technique looks more diffuse than those of the other methods in the brain region. The fact that it is brighter in the air region (i.c., near the edges) is a consequence of minor differences in sensitivity- the E2E- VarNet, which does not generate posterior samples, as well as Ohayon et al.'s cGAN, due to mode collapse. The Langevin wixel-wise SD images show localized hot-spots that appear to be reconstruction artifacts.
estimation. The pixel-wise SD images show a lack of variability for
Appendices K.1 and K.2 show other example MRI recoveries with zoomed pixel-wise SD images R 4 and R = 8, respectively. Notably, Figures K. and K. show strong hallucinations for Langevin recovery at R = 8, as highlighted by the red arrows.
7

---

Table 3: Average PSNR, SSIM, LPIPS, and DISTS of Di)) versus P for R 8 MRI
PSNRT
SSIMI
Model
P-1
P-2
P-4
P-8
P-16
P-32
P-1
P-2
P-4 P- P-16 P-32
EZE- VarNet (Srirum et al. (37])
36.49
0.9220
Langevin et al. [19])
0.8725
0.8919
0.9031
cGAN (Adler & Oktem (8))
31.31
32.31
32.92
33.26
33.42
33.51
0.8865
0.9045
0.9103
0.9111
0,9102
0.9095
cGAN (Ohayon [23))
34.89
34.90
34.90
34,90
34,91
34.92
0.9222
0.9217
0.9213
0.9211
0,9211
0.9210
(Ours)
33.67
34.53
35.01
0.9030
0.9251
[PPPS]]
DISIS)
Model
P=1
P-2
P-4
P-
P-16
P-32
P-1
P=2
P=4
P-8
P-16
P-32
VarNet et (37])
0.1253
Langevin (Jala) et al. [19]
0.0769
0.0619
0.0579
0.0589
0.0611
0.0611
0.1341
0.1136
0.1086
0.1119
0.1175
0.1212
cGAN (Adler Oktem (8I)
0.0698
0.0614
0.0623
0.0667
0.0704
0.0727
0.1407
0.1262
0.1252
0.1291
0.1334
0.1361
(Olayen (23))
0.0536
0.0539
0.0540
0.1128
0.1143
0.1157
0.1158
cGAN (Ours)
0.0418
0.0379
0.0421
0.0476
0.0516
0.0539
0.0906
0.0877
0.0965
0.1063
0.1135
0.1177
cGAN (ours)
cGAN (Ohayon)
cGAN (Adler)
Langevin (Jalal)
Average (P-32)
Truth
Sample
E2E-VarNer
Sample
Figure 3: Example R = 8 MRI reconstructions. Arrows show meaningful variations across samples.
4.3 Inpainting experiments
In this section, our goal is to complete large missing square in face image.
Data. We used 256 256 CelebA-HQ face images [29] and a centered 128 x 128 missing square. We randomly split the dataset, yielding 27 000 training, 2 000 validation, and 1 000 testing images. Architecture. For our cGAN, our proposed Pi.. regularization. Unlike [9], we do not use MBSD [29] at the discriminator.
the CoModGAN generator and discriminator from [9] with
Traininngvaiidaaioo/ttssnnn We use the same general training and testing procedure described in Section 4.2, but with Badv = 5c-3, Tbbatch = 100, and 110 epochs of cGAN training. Running PyTorch on server with 4 Tesla A100 GPUs, each with 82 GB of memory, the cGAN training takes approximately 2 days. FID was evaluated on 1000 test images using P = 32 samples per measurement. To avoid the bias that would result from evaluating CFID only 1 000 images (see Appendix J.1), J.1), CFID was evaluated on all 30 000 images with P 1.
Competitors. We compare with the tfate-of-the-ar CoModGAN [9] and Score-based SDE [20] approaches. For CoModGAN, we use the PyTorch implementation from [44]. CoModGAN differs from our cGAN only in its use of MBSD and lack of Li.sD,P.... regularization. For Song et al.'s SDE, we use the authors' implementation from [45] with their pretrained weights and the settings they suggested for the 256 x 256 CelebA-HQ dataset.
8

---

Truth
E2E-VarNet
cGAN (ours)
cGAN (Ohayon)
cGAN (Adier)
Langevin (Jalal)
Se-05
0c+00
5e-05
0e+00
Figure 4: Example R = 8 MRI reconstructions with P = 32. Row one: P-sample average iryy-
Row two: pixel-wise absolute error (pP) x Row three: pixel-wise SD ( Ei (ā; )
Table 4: Average results for inpainting: FID was computed from 1 000 test images with P= 32, while
CFID was computed from all 30 images with P P=1
Model
CFID
FID|
Time (128))
Score-SDE (Song et [20))
5.11
7.92
48 min
CoModGAN (Zhao et al. [9])
5.29
8.50
217 ms
cGAN (ours)
4.69
7.45
217 ms
Results. Table 4 shows test CFID, FID, and 128-sample generation time. The table shows that our approach gave the best CFID and FID, and that the eGANs generated samples 000 times faster than the score-based method. Figure 5 shows methods under test. The samples are all quite good, although a few generated by CoModGAN and the score-based technique have minor artifacts. Some samples generated by technique show almond-shaped eyes, demonstrating fairness. Additional examples are given Appendix K.3.
example of five generated samples for the three
Conclusion
We propose a novel regularization framework for image-recoven cGANs that consists of supervised- &, loss plus appropriately weighted saadard-deviation reward. For the case of an independent Gaussian posterior, we proved that our regularization yields generated samples that agree with the true-posterior samples in both mean and covariance. We also established limitations for alternatives based on supervised-Ez regularization with or without a variance reward. For practical datasets, we proposed method to auto-tune our standard-dewe deviation reward weight.
Experiments on parallel MRI and large-scale face inpainting showed our proposed method outper- forming all cGAN and score-based competitors in CFID, which measures vosterior-appooxmmatio quality, as well as other metrics like FID, PSNR, SSIM, LPIPS, and DISTS. Furthermore, it generates samples thousands of times faster than Langevin/scoree based approaches.
In ongoing work, we are extending our approach so that can be trained to handle a wide range of recovery tasks, such as MRI with a generic acceleration and sampling mask [46], or inpainting with generic mask. We are also extending our approach to other applications, such super-resolution, deblurring, compressive sensing, denoising, and phase retrieval.

---

cGAN (ours)
Original
CoModGAN
Masked
Score SDE
Figure 5: Example of inpainting 128 128 square 256 x 256 resolution CelebA-HQ image.
Limitations. We acknowledge several limitations of our work. First, while our current work focuses on how to build fast and accurate posterior sampler, it's not yet clear how to best exploit the resulting posterior samples each given application. For example, MRI, where the posterior distribution has the potential to help assess uncertainty in image recovery, it's still not quite clear how to best convey uncertainty information to radiologists (e.g., they may not gain much from pixel-wise SD images). More work is needed this front. Second, we acknowledge that, because radiologists are risk-averse, more studies are needed before they will feel comfortable incorporating generative deep-learning-baseee -based methods into the clinical workflow. Third, acknowledge that the visual quality of our R = MRI reconstructions falls below clinical standards. Fourth, some caution needed when interpreting
CFID, FID, and DISTS perceptual metries because the VGG-16 backbone used to compute them was trained on ImageNet data. Although there some evidence that the resulting DISTS metric correlates well with radiologists perceptions [41], there is also evidence that Imagener-trained features may discard information that is diagnostically relevant in medical imaging [47]. Thus results need to be validated with pathology-centric radiologist study before they can be considered relevant to clinical practice.
Acknowledgments and Disclosure of Funding
The authors are funded in part by the National Institutes of Health under grant R01-EB029957 and the National Science Foundation under grant CCF-1955587.
References
F. I.. Lehhman and G. Casella, Theory point estimation. Springer Science Business Media, 2006.
[2] Y. Blau and T. Michaeli, "The perception-distortee tradeoff." in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp. 6228-6237, 2018. 1.7
[3] C. K. Sønderby, J. Caballero, L. Theis, W. Shi, and E. Huszár, "Amortised MAP inference for image super-resolution," in Proc. Int. Conf. on Learn. Rep., 2017. 1, 19
[4] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, "A survey on bias and faimess in machine learning," ACM Comput. Surveys, vol. 54, no. 6, 1-35, 2021. 2
[5] T. Sanchez, I. Krawezuk, Sun, and V. Cevher, -Unnertainty-driven adaptive sampling via GANs," Proc. Neural Inf. Process. Syst. Workshop, 2020. 2
C.-H. Chang, Creager, A. Goldenberg, and D. Duvenaud, "Explaining image classifiers by counterfactual generation." in Proc. Int. Conf. on Learn. Rep., 2019. 2
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial
networks." in Proc. IEEE Conf. Comp. Vision Patter Recog., pp. 1125-1134, 2017. 2,
[8] J. Adler and O, Öktem, "Deep Bayesian inversion." arXiv:1811.05910 2018. 2. 6, 7, 19, 20. 21
10

---

[9] S. Zhao, J. Cui, Y. Sheng. Y. Dong. X. Liang, E. I.-.. Chang. and Y. Xu, "Large scale image completion via co-modulated generative adversarial networks." in Pruc. Ini. Conf. Learn. Rep., 2021. 2, 8, 9. 20, 21
[10] H. Zhao, H. Li, S. Maurer-Stroh, and L. Cheng, "Synthesizing retinal and neuronal images with generative adversarial nets," Med. Image Analysis, vol. 49, 07 2018.
[11] V. Edupuganti, M. Mardani, S. Vasanawala, and J. Pauly, "Uncertainty quantification in deep MRI reconstruction," IEEF Trans. Med. Imag. vol. 40, pp. 239-250, Jan. 2021.
[12] F. Tonolini, J. Radford, A. Turpin, D. Faccio, and R. Murray-Smine "Variational inference for computa- tional imaging inverse problems." J. Mach. Learn. Res., vol. 21. no. 179, pp. 1-46, 2020. 2
[13] K. Sohn, H. Lee, and X. Yan, "Learning structured output representation using deep conditional generative models," in Proc. Neural Inf. Process. Syst. Conf., 2015. 2
[14] L. Ardizzone, C. Lüth, J. Kruse, C. Rother, and U. Köthe, "Guided image generation with conditional invertible neural networks," vrXiv:/907.02392 2019.
[15] C. Winkler, D. Worrall, E. Hoogehoom, and M. Welling, "Learning likelihoods with conditional normaliz- ing flows," arXiv preprint arXiv:1912.00042, 2019. 2
[16] H. Sun and K. L. Bouman, "Deep probabilistic imaging: Uncertainty quantification and multi-moda solution characterization for computational imaging," in Proc. AAAI Conf. Artificial Intell., vol. 35, pp. 2628-2637, 2021. 2
[17] M. Welling and Y. W. Tch, "Bayesian learning via stochastic gradient Langevin dynamics," in Proc. Int.
Conf. Mach. Learn., pp. 681-688, 2011. 2
[18] Y. Song and Ermon, "Improved techniques for training score-based generative models," in Proc. Neural Inf. Process. Syst. Conf., 2020. 2
[19] A. Jalal, M. Arvinte, Daras, E. Price, A. Dimakis, and J. Tamir, "Robust compressed sensing MRI with deep generative priors," in Proc. Neural Inf. Process. Syst. Conf. 2021. 2, 6, 7, 8, 20,
[20] Y. Song, Sohl-Dickstein, D. P. Kingma, Kumar, Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations." " in Proc. Int. Conf. on Learn. Rep.. 2021. 8, 9,
[21] Y. Song, L. Shen, L. Xing, and S. Ermon, "Solving inverse problems in medical imaging with score-based generative models," " Proc. Int. Conf. on Learn. Rep., 2022. 2
[22] M. Soloveitchik, T. Diskin, E. Morin, and A. Wiesel, "Conditional Frechet inception distance." arXiv:210311522,, 2021. 2, 6, 18,
[23] G. Ohayon, T. Adrai, G. Vaksman, M. Flad, and P. Milanfar, "High perceptual quality image denoising with posterior sampling CGAN," in Proc. IEFF Int. Conf. Comput. Vis. Workshops, vol. 10, pp. 1805-1813, 2021. 2, 4. 6, 7,8, 8. 20. 21
[24] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, "Improved training of Wasserstein GANs," in Proc. Neural Inf. Process. Syst. Conf. p. 5769-5779, 2017. 19
[25] M. Mirza and S. Osindero, "Conditional generative adversarial nets," arXiv:1411.1788 2014. 3
[26] E. Schonfeld, B. Schiele, and A. Khoreva, "A U-Net based discriminator for generative adversarial networks," in Proc. IEEF Conf. Comp. Vision Pattern Recog., pp. 8207-8216, 2020. 3
[27] T. Karras, M. Aittala. J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, "Training generative adversarial
networks with limited data," in Proc. Neurul Inf. Process. Syst. Conf.. vol. 33, pp. 22104-12114, 2020. 3
[28] Z. Zhao, S. Singh, H. Lec, Z. Zhang, A. Odena, and H. Zhang, "Improved consistency regularization for GANs," in Proc. AAAI Conf. Artificial Intell., vol. 35, pp. 11033-11041, 2021. 3
[29] T. Karras, T. Aila, S. Laine, and J. Lehtinen, "Progressive growing GANs for improved quality, stability, and variation," " in Proc. Int. Conf. on Learn. Rep., 2018. 3, 8, 19
[30] H. Zhao, O. Gallo, Frosio, and J. Kautz, "Loss functions for image restoration with neural networks." IEEE Trans. Comput. Imag., vol. 3. pp. 47-57, Mar. 2017. 4
[31] G. Ohayon, T. Adrai, G. Vaksman, M. Elad, and P. Milanfar, "High perceptual quality image denoising with posterior sampling CGAN." Downloaded from /ttpp://giiiee //gtt//////tbbbb// com/ www.thhehcppheeeee July 2021.
[32] M. Heusel, H. Ramsauer, Unterthiner, B. Nessler, and S. Hochreiter, "GANs trained by two time-scale update rule converge to a local Nash equilibrium," " Proc. Neural Inf. Process. Syst. Conf., vol. 30, 2017.
[33] Zbontar, Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. Geras, J. Katsnelson.
H. Chandarana, Z. Zhang. M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, J. Pinkerton, D. Wang. N. Yakubova, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, "fastMRI: An open dataset and benchmarks for accelerated MRI," arXiv:1811 088839, 2018. 6, 19
11

---

[34] T. Zhang, J. M. Pauly. S. S. Vasanawala, and M. Lustig. "Coil compression for accelerated imaging with Cartesian sampling," Magn. Reson. Med., vol. 69, no. pp. 571-582, 2013.
[35] M. Joshi, A. Pruitt, C. Chen, Y. Liu, and R. Ahmad, "Technical report (v1.0) (.l0)-pseudoorrnddo cartesian
sampling for dynamic MRI," UrXi:22006.03633 2022. 6
[36] O. Ronneberger. Fischer, and Brox. "U-Net: Convolutional networks for biomedical image seg- mentation," in Proc. Intl. Conf. Med. Image Comput. Compul. Assist. Intervent. pp. 234-241, 2015. 6,
20
[37] A. Sriram, Zbontar, Murrell, A. Defazio, C. Zitmick, N. Yakubova, Knoll, and Johnson, "End-to-end variational networks for accelerated MRI reconstruction," in Proc. Inl. Conf. Med. Image Comput. Comput. Assist. Intervent., pp. 64-73, 2020. 6, 7, 8, 20
[38] A. Jalal, M. Arvinte, G. Daras, F. Price, A. Dimakis, and J. Tamir, "exgm-mri-langeevnn" https:// 2021, Accessed: 2021-12-05. 6, 20
www/etcsslaa//eeeeecccoeoeoeeeee wawpaaaaengmmminannneeeeeeeee github. github.com/utceee
[39] K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, "SENSE: Sensitivity encoding for fast MRI," Magn. Reson. Med., vol. 42, no. pp. 952-962, 1999. 6, 18
[40] M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig, "ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA," Magn. Reson. Med., vol. 71, no. pp. 1900-101, 2014. 6, 18
[41] S. Kastryulia, J. Zakirow. N. Pezzotti, and D. V. Dylov, "Image quality assessment for magnetic resonance imaging." arXiv:2203, 07809, 2022. 6. 10, 18
[42] R. Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as perceptual metric," in Proc. IEEE Conf. Comp. Vision Pattern Recog.. pp.
2018.
[43] K. Ding, K. Ma. S. Wang, and E. P. Simoncelli, "Image quality assessment: Unifying structure and texture similarity," IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. pp. 2567-2581, 2020.
[44] Y. Zeng, "co-mod-gan -omo--g---yttrrhh... co- co-mod-get ee--mod-ga-ppy- gan-aattpptth,, Sept. 2022. 8,
Downloaded from https: https://github... com zemmm//engxaaauu////m
[45] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score- generative mod- eling through stochastic differential equations." Downloaded from attps://gitee httptthhuu..o.cyaaaeeeeeeee saryyyannggoog'
score_sde_pytore
Oct. 2022. 8, 21
[46] M. C. Bendel, R. Ahmad, and P. Schniter, agnss-agnostic posterior sampling MRI via conditional GANs with guided reconstruction," in Proc. NeurlPS Workshop on Deep Inverse Problems, 2023.
[47] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen, "The role of imagenet classes in fréchet inception distance," " arXiv:2203.088 0060266 2022. 10
[48] F. C. Leone, I.. S. Nelson, and R. Nottingham, "The folded normal distribution," Technometrics, vol. 3, 4. pp. 543 550, 1961. 14
[49] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition." "
140. 2014. 18
[50] C. Szegedy, V. Vanhoucke, loffe, Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," in Proc. IEEE Conf. Comp. Vision Pattern Recog., 2016. 18
[51] F. Ong and M. Lustig. "SigPy: A python package for high performance iterative reconstruction." in Proc. Annu. Meeting ISMRM, vol. 4819, 2019.
[52] D. Chen and M. E. Davies, "Deep decomposition learning for inverse imaging problems," in Proc. European Conf. Comp. Vision, pp. 510-526,66602111010 526, 2020. 19
[53] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proc. Int. Conf. on Learn, Rep.. 2015. 19
[54] P. Deora, B. Vasudeva, S. Bhattacharya, and P. M. Pradhan, "Structure preserving compressive sensing MRI reconstruction using generative adversarial networks," Proc. IEEE Conf. Comp. Vision Pattern Recog. Workshop, pp. 221122212222 22199,, June 2020. 20
12


Reviews:
[{"review_id": "6c5u4kfY9n", "reviewer": "Reviewer_pyUh", "summary": "This paper proposed a cGAN (conditional generative adversarial network) method for general image inverse problems (IR). Conventional cGAN methods for IR usually suffer from mode collapse that the generators learned to ignore latent inputs and are unable to generate diverse image reconstruction samples. Motivated by existing work, this work proposes to incorporate multiple latent input to the GAN model during training to mitigate the mode collapse effect. Accordingly, a sophisticated designed supervised training loss function is also presented in this work. The numerical results show that the proposed method generates comparable or even better results to cGAN and score-based baseline methods on compressive MRI and image inpainting.", "questions": "1), There seems to have a gap between proposition 3.3 and the gradient descent on the $\\beta_{SD}$ and could be clarified for reader’s better understanding. For example, there is no clear formular between $\\beta_{SD}$ and the ratio in Eq 20., making the derivative of Eq. 21-23 a bit confusing. It is unclear how Eq. 21-22 is formulated and how this led to the update of $\\beta_{SD}$. Please clarify this part in the revision.\n\n2), At the same line above, it is unclear why the number of latent $P$ is not the same for training and testing (e.g., line 166,” for some $P_{\\text{val}} $ that does not need to match $P_{\\text{train}} $.”). Is this means the performance of this proposal for each image is sensitive to the choice of value $P$? It is unclear when there is a lack of enough validation dataset to finetune $\\beta$ or if there is a mismatch between train and testing dataset. As a result, the effects of hyper-parameter settings for different settings are unclear. \n\n3), The paper lacks sufficient discussion on the CS-MRI experiments.  For instance, based on the description of Sec. 4.2, “we transformed $x_t$ to the Fourier domain”, seems the authors do not follow the trends of other methods in the fast MRI leaderboard that directly perform down-sampling on the kspace data. This seems like an important distinction and must be made clear in the paper. There is no discussion on how the measurement noise is modeled or how it affects the reconstruction performance. Also, while there are some technical discussions on the CSMRI formulation in the appendix, there is no description of how the forward model simulation in this paper differs from the actual forward model of the real MRI system.\n\n4), One benefit of posterior sampling is for model uncertainty estimation. While this work can generate multiple samples and show the pixel-wise standard deviation in Figure 4, it is difficult to evaluate if it can indeed reflect the mean-absolute error for image reconstruction. It would be great to have a quantitative result for this study.\n\n5), There is a lack of comparison to more recent diffusion model-based IR methods, where many have been demonstrated good posterior sampling performance for various imaging tasks, such as [R1, R2, R3].\n\n6),  The run time comparison to  the selected Langevin dynamic baselines seems to be suboptimal, as there are many efficient sampling strategies can be applied for IR diffusion/score-based methods, where fewer iteration steps could result in appealing reconstruction [R3].\n\n7), It is unclear which forward model/dataset are used for Figure 2 in the paper. If it is plotted using images, it is unclear how is the curve in theory plotted. Moreover, it would be more informative if Figure2 can be directly represented by $\\epsilon_1/\\epsilon_P$ and $\\beta_{SD}$. \n\n[R1] Score-based diffusion models for accelerated MRI, Chung. etal, 2021.\n\n[R2] Diffusion posterior sampling for general noisy inverse problems, Chung. etal, 2022.\n\n[R3] Pseudoinverse-Guided Diffusion Models for Inverse Problems, Song. etal, 2023.\n", "limitations": "The authors discussed their work's limitations in the submission. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1, The paper is overall well motivated, clearly written and easy to follow. \n\n2, The proposed posterior sampling-based method is effective for generating multiple samples compared to some score-based methods. The multiple sampling can also be applied for model uncertainty estimation, which is important for many medical imaging problems.\n\n3, The theoretical interpretation on the mean and covariance of predicted posterior distribution is informative, though it is sometimes unrealistic for most real-world imaging problems.\n", "weaknesses": "1, One contribution of this paper seems still to be based on the idea of Adler & Oktem [9], where multiple latent vectors are as inputs to the discriminator and generator module instead of only one input. In the meantime, as pointed out by the author as well, adding extra regularization term in the adversarial GAN loss during training is not completely new. \n\n2, This proposal ignores a body of work for posterior sampling based on Bayesian deep learning such as Monte Carlo dropout, deep ensembles, etc. There’s also a lack of numerical comparisons with these methods, making the baseline methods not sufficient enough. \n\n3, While some ablation studies provided in its original submission, the effects of regularization parameter $\\beta_{SD}$ and number $P$ on the final imaging reconstruction are still not comprehensive and could be improved. For example, a visual illustration or numerical comparison of several proposed sampling results due to different settings of $\\beta_{SD}$ is preferable.\n\n4, The assumption for proposition 3.1 that samples $\\widehat x_i$ and $x$ are independent Gaussian conditioned on y is not true for imaging applications, weekend the theoretical contribution. \n\n5, Some technical details and notations are confusing and could be improved in their current state.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Weakness**:\nAdding extra regularization to the GAN loss is not completely new.  \n**Response**:\nPlease see the global rebuttal.\n\n- **Weakness**:\nNo discussion of Monte-Carlo dropout or deep ensembles.  \n**Response**:\nPosterior samplers aim to accurately sample from the true posterior $p(\\boldsymbol{x}|\\boldsymbol{y})$, which is a property of the data $\\\\{(\\boldsymbol{x}_i,\\boldsymbol{y}_i)\\\\}$ and not any trained network. In contrast, Monte-Carlo dropout and deep ensembles heuristically perturb a given deep network and quantify its uncertainty according to the variance induced by those perturbations; see (Gal'16).  They make no attempt to accurately sample from the posterior, which is a non-trivial task with high dimensional data such as images.  Image posterior sampling is tackled by cGANs, cNFs, cVAEs, and Langevin/scores/diffusion methods. To be consistent with that literature, we didn't discuss/compare to Monte-Carlo dropout or deep ensembles. But if the reviewer has particular papers in mind, we could include a discussion/comparison in the final version (if accepted).\n\n- **Weakness**:\nThe effect of $\\\\beta_{\\\\mathsf{SD}}$ and $P$ on final reconstruction are not well described.  \n**Response**:\nNeither are tunable parameters.  There is a correct value for $\\\\beta_{\\\\mathsf{SD}}$, and our method estimates it as described in Section 3.2.  For more on $\\\\beta_{\\\\mathsf{SD}}$ and $P$, please see our responses below.\n\n- **Question**: \nWhat is the relationship between Prop. 3.3 and the update on $\\beta_{\\mathsf{SD}}$?  \n**Response**: \nJust after equation (20), we write that the ratio \"$\\\\mathcal{E}\\_1 / \\\\mathcal{E}\\_P$ grows with the SD reward weight $\\\\beta_{\\\\mathsf{SD}}$,\"\nand we provide evidence of this behavior in Figure 2. Thus, if the ratio $\\\\mathcal{E}\\_1/\\\\mathcal{E}\\_P$ is smaller than the value $2P/(P+1)$ specified in Prop. 3, then $\\\\beta_{\\\\mathsf{SD}}$ is too low. Conversely, if the ratio is larger than $2P/(P+1)$, then $\\\\beta_{\\\\mathsf{SD}}$ is too high. The $\\\\beta_{\\\\mathsf{SD}}$ update (23) simply increases or decreases $\\\\beta_{\\\\mathsf{SD}}$ depending on whether $\\\\mathcal{E}_1/\\\\mathcal{E}_P$ is smaller or larger than $2P/(P+1)$.\n\n- **Question**: \nWhy does $P$ differ between training and testing?  \n**Response**:\n$P$ is simply the number of generated samples. Changing $P$ does not affect the quality of those samples. When training, Prop. 3.1 shows that any $P\\geq 2$ can be used. In practice, larger values of $P$ consume more GPU memory, and so we use the minimum value of 2. When testing, the choice of $P$ depends on the task. If the goal is to generate a single posterior sample, then $P=1$ suffices. If the goal is to approximate the conditional mean (i.e., MMSE) estimate, then larger $P$ will give lower MSE (see Prop. 3.3) but consume more computational resources. When validating during modeling training, we roughly estimate the MSE at each epoch, for which we used $P=8$ to strike a balance between accuracy and computation.\n\n- **Question**:\nAre you using a non-standard MRI setup?  \n**Response**:\nNo, we are using a standard MRI setup. Because the fastMRI samples vary in dimension, we center-crop them to a consistent dimension of 384x384. The cropping must be done in the image domain, not the k-space, else the sampling resolution will change. This is common practice: see page 15 of the fastMRI paper https://arxiv.org/pdf/1811.08839.pdf. In summary, we start with fully sampled k-space data, convert to the image domain, center-crop it to 384x384, convert back to the k-space, and then sub-sample it. No additional measurement noise is added.\n\n- **Question**:\nDoes pixel-wise standard deviation reflect the mean-absolute error (MAE) of image reconstruction?  \n**Response**:\nNot quite. Say $x$ is a single pixel of the unknown image. The posterior distribution $p(x|\\boldsymbol{y})$ is the probability distribution of $x$ given the measurements $\\boldsymbol{y}$. Then for a given pixel estimate $\\hat{x}$, we have MSE($\\hat{x}$) $=\\int |x-\\hat{x}|^2 \\\\, p(x|\\boldsymbol{y}) dx$ and MAE($\\hat{x}$) $=\\int |x-\\hat{x}| \\\\, p(x|\\boldsymbol{y}) dx$.  The posterior variance, $\\int |x-\\mathrm{E}\\\\{x|\\boldsymbol{y}\\\\}|^2 \\, p(x|\\boldsymbol{y}) dx$, equals the minimum possible MSE for any $\\hat{x}$, and the standard deviation is simply its square root.  Note that MSE and MAE are functions of $\\hat{x}$ while the posterior standard deviation is not.\n\n- **Question**:\nWhy not compare to more recent diffusion models like (Chung'21), (Chung'22), or (Song'23)?  \n**Response**:\nPlease see the global response.\n\n- **Question**:\nWhy didn't you use sampling strategies that make diffusion/score-based methods more efficient?  \n**Response**:\nWe did not want to modify the existing Langevin and Score-SDE techniques from (Jalal'21) and (Song'21) because we could be blamed if the resulting modified techniques under-performed our approach. So we used the authors' recommendations. But we note that more recent diffusion methods like DDRM at 20 steps are still 275x slower than our cGAN (see global rebuttal).\n\n- **Question**: \nWhich dataset was used for Figure 2? And can the figure be represented by $\\\\mathcal{E}\\_1 / \\mathcal{E}\\_P$ and $\\beta_\\text{SD}$?  \n**Response**:\nFor Figure 2, we used the fastMRI brain data at $R=4$. We will describe this in the final paper if accepted.  We plot $1/\\mathcal{E}\\_P$ rather than $\\mathcal{E}\\_1 / \\mathcal{E}\\_P$, because the former contains information about $\\mathcal{E}\\_1$ that is lost in the latter.  But in the dB domain, the two curves are related by a simple vertical shift.  Finally, Figure 2 prints the value of $\\beta_\\text{SD}$ in the title of each subplot.\n"}, {"review_id": "1CIxvQaSQE", "reviewer": "Reviewer_N79n", "summary": "The paper presents a new method based on conditional GANs (cGANs) for generating posterior samples for solutions of inverse problems. The motivation comes from the fact that standard approaches to inverse problems typically use point estimates, which due to the perception-distortion tradeoff will lead to blurry images. Conversely, reconstruction via sampling can be used to navigate the perception-distortion tradeoff by either sampling small counts to have sharp images, or averaging many samples to approximate the point estimate reconstruction.\n\nAll of the above is known from the previous literature. The present paper's specific contributions largely center on the loss function and regularization design for cGANs to avoid posterior collapse and increase sample quality. The regularizer design is based on matching the means and covariances of the samples to the means and covariances of the true distribution. The manuscript chooses to compute these statistics over the $\\ell_1$ norm rather than the $\\ell_2$ norm, and the paper shows that $\\ell_2$ norm can in fact lead to collapse.\n\nIn numerical experiments the paper demonstrates superior FID and conditional FID performance for the proposed method, indicating that it is superior at distribution matching than the previous methods considered. For distortion metrics, the experiments demonstrate that superior distortion performance can be achieved by averaging over samples.", "questions": "I am positive on this work, but a number of aspects on the presentation could be improved. I would appreciate the authors' responses on improving the following, mostly related to the limitations of the work and contextualization with existing literature on medical image reconstruction.\n\n1. Although sampling-based reconstruction can do a better job of distribution-matching and uncertainty estimates, the artifacts introduced are unpredictable. For medical imaging, calibrated radiologists can work around predictable artifacts such as blurring. In fact, even point-estimate reconstruction has not fully taken over the medical field, because simple linear reconstructions have more predictable artifacts than deep learning point estimates. The use of generative sampling compounds this effect and is likely to be substantial limitation on the present work. I think a discussion on this should be added as limitation (some sketch of the proposed discussion would be appreciated during the rebuttal).\n2. The paper primarily examines distribution matching over ImageNet features, which are likely to be quite different than the features useful for medical imaging. (Kynkäänniemi, 2023) shows the presence of perceptual nullspaces that could very well ignore small pathologies. Even the self-supervised solution of (Kynkäänniemi, 2023) is likely to be of benefit for medical imaging analysis, because self-supervised features rely on large objects at the center of the image.\n3. Although the classifier matching proof in Appendix A seems theoretically sound, it relies on a few limits and calibration assumptions that are likely to deviate from reality, so I am skeptical this is a grounded proof that classifiers trained on fully-sampled images will perform adequately on the sampled images.\n4. The paper could benefit with a comparison to the End-to-End variational network, which is a standard comparison method in the MRI literature and was previously used in the Jalal paper. Although the present manuscript has shown superiority to (Jalal, 2021) on metrics, sampling-based methods have not yet taken the general field of medical image reconstruction, so the comparison is still valid.\n5. A cursory overview of the Plug-and-Play reconstruction literature would provide a more complete picture of previous work.", "limitations": "As discussed under Questions, the paper could expand its discussion of limitations, including the issues of 1) issues artifact consistency posterior sampling methods with human observers and 2) the use of metrics based on ImageNet feature alignment for medical imaging tasks.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "**Originality**\n- Interesting new regularization method for training cGANs for image restoration tasks.\n\n**Quality**\n- The paper's quantitative results indicate SotA performance across a variety of tasks for both MRI reconstruction and image inpainting.\n- Principled use of VGG weights instead of Inception weights for FID calculation and correlations to radiologist perception.\n- A derivation motivating the use of both the supervised standard deviation reward and the application of this award over $\\ell_1$ norms.\n\n**Clarity**\n- Excellent presentation of perception-distortion tradeoff and its relevance to imaging inverse problems.\n- Detailed outline of the proposed method with the appendices, as well as relevant mathematics concerning the various tasks in the results.\n\n**Significance**\n- There is substantial interest in the image reconstruction/restoration community now, much of which has focused on conditional diffusion models. The establishment of competitive performance in this space with cGANs (as well as the superior performance to past cGAN methods) will be of interest.", "weaknesses": "**Quality**\n- The MRI images still demonstrate substantial artifacts from all methods at 8X acceleration. It is possible to find a number of features and fine details that are missing, so I doubt these would be accepted for clinical use.\n- Although VGG may correlate better with radiologist scores, it is still an ImageNet feature backbone and contains nullspaces that may be poor for the tasks under consideration (Kynkäänniemi, 2023).\n- There is a literature dedicated to so-called \"plug-and-play\" methods that are relevant to the present work, but was not considered. See (Venkatakrishnan, 2013) and follow-up papers.\n- The U-Net is used as a surrogate for point-estimate methods in the MRI example, but it is known in the MRI community that the U-Net is a poor reconstructor. A more standard comparison would be to use the End-to-End Variational Network (Sriram 2020), which is available from the fastMRI GitHub repository. This comparison was included in the cited paper of (Jalal 2021).\n- The competing methods for inpainting seem out of date. Dozens of methods based on conditional probabilistic diffusion have emerged over the last year (e.g., DDRM (Kawar 2022) and follow-ups), so I do not find these experiments as strong as the MRI ones.\n\n1. Kynkäänniemi, Tuomas, et al. \"The Role of ImageNet Classes in Frechet Inception Distance.\" ICLR (2023).\n2. Venkatakrishnan, Singanallur V., Charles A. Bouman, and Brendt Wohlberg. \"Plug-and-play priors for model based reconstruction.\" 2013 IEEE global conference on signal and information processing. IEEE, 2013.\n3. Sriram, Anuroop, et al. \"End-to-end variational networks for accelerated MRI reconstruction.\" Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. Springer International Publishing, 2020.\n4. Kawar, Bahjat, et al. \"Denoising diffusion restoration models.\" Advances in Neural Information Processing Systems 35 (2022): 23593-23606.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Weakness**:\nAt 8x acceleration, it is possible to find a number of features and fine details that are missing, so I doubt these would be accepted for clinical use.  \n**Response**:\n8x acceleration is perhaps too aggressive for clinical practice, but keep in mind that we are zooming the plots significantly. For example, the E2E-VarNet paper (Sriram'20) shows only non-zoomed 8x plots that look good from casual inspection.  But by looking closely, one can spot artifacts and missing details that would be clearly evident in a zoomed plot. \n\n- **Weakness**:\nThe competing methods for inpainting seem out of date. Dozens of diffusion methods have emerged over the last year (e.g., DDRM and follow-ons).  \n**Response**:\nPlease see the global response.\n\n- **Question**: \nRadiologists are okay with blurring, but not with unpredictable artifacts. Will radiologists benefit from the use of generative sampling?  \n**Response**: \nThere exists a fundamental uncertainty in what one can infer about the true image from sub-sampled k-space measurements, and the posterior distribution gives a full description of that uncertainty. But we agree that it's not yet clear how to best distill that uncertainty information for radiologists, and this limits the short-term impact of our work. Ignoring uncertainty altogether, however, does not seem to be a good solution. Thus we are optimistic that radiologists will eventually be persuaded to integrate uncertainty information into their workflows. In ongoing work, we are investigating various ways of doing so, including rigorous confidence bounds on classifier outputs, counterfactual generation, and visualizations of dominant uncertainty eigenspaces.\n\n- **Question**:\nAre metrics computed using a VGG backbone appropriate for MRI?  \n**Response**:\nThis is an excellent question. (Kastryulin'22) studied 35 contemporary MRI metrics and found that, of the four methods that best correlated with radiologists perceptions, two used features from ImageNet-trained VGG. But we agree that improved metrics are likely result from non-ImageNet backbones, and we hope to soon see studies that validate them with radiologists. In the meanwhile, however, we worry that readers may not trust our evaluations if they are based on novel metrics. In any case, if our final paper is accepted, we will discuss the limitations of using ImageNet-trained backbones for MRI performance metrics.\n\n- **Question**:\nThe classifier matching proof relies on a perfectly calibrated classifier.  Can this be generalized to practical classifiers?  \n**Response**:\nBased on our ongoing work, the answer seems to be \"yes\". For our Neurips submission, though, we feel that this is a bit out of scope.  \n\n- **Question**:\nThe UNet is a weak baseline. Compare to an end-to-end (E2E) VarNet, as in (Jalal'21)?  \n**Response**:\nPlease see the global rebuttal.\n\n- **Question**:\nDiscuss plug-and-play methods?  \n**Response**:\nSince our paper focuses on posterior-sampling, we did not review point-estimation approaches like compressed sensing, PnP/RED, unrolled networks, deep equilibrium, etc.  That said, adding noise to each PnP/RED iteration gives a posterior sampler known as the unadjusted Langevin algorithm (Durmus'17), which is a precursor to the Langevin/score/diffusion techniques that we consider.  We will discuss these PnP connections in the final paper if accepted."}, {"review_id": "hlmRe7Qon3", "reviewer": "Reviewer_ur5t", "summary": "This work uses conditional GANs for solving image recovery problems including face-inpainting and accelerated MRI. In particular, a regularization scheme is proposed that tackles the lack-of-diversity issue related to posterior sampling from cGANs. The authors show that the proposed regularization allows sampling from the true posterior under the assumption of a Gaussian signal prior. Empirically the method outperforms a U-Net trained end-to-end, existing cGANs and two score/diffusion/Langevin approaches in terms of accuracy and the latter also significantly in terms of speed.", "questions": "No additional questions.", "limitations": "Limitations are discussed adequately by the authors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is written very clearly and is well organized.\n2. The proposed regularization significantly improves over existing cGAN approaches and hence will be of interest for the community.\n3. The theoretical discussion of a toy example with Gaussian signal prior is insightful and contributes to the understanding of the proposed regularization.", "weaknesses": "1. The authors argue that sampling from the posterior distribution alleviates any concerns about fairness and facilitates uncertainty quantification and further that uncertainty quantification in MRI, and fairness in face-generation, are both of paramount importance. In Appendix A they show that a classifier trained on images from the prior distribution maintains its performance when averaged over many samples from the posterior but not when evaluated on a point estimate.\nHowever, the empirical experiments in Section 4 do not evaluate these points in detail. Regarding uncertainty quantification for MRI Figure 4 shows one example of the pixel-wise standard deviation (SD) but a discussion on the usefulness and practical applicability is missing. To me it seems that in the depicted SD map the model is equally uncertain about every edge in the image making me wonder about the usefulness of this map to a radiologist.\nRegarding fairness in face-generation 5 different reconstructions of one image are shown and the authors argue that the fact that some samples show almond-shaped eyes demonstrates fairness. \n    Here, a more detailed evaluation would strengthen the contribution although I acknowledge that this might be out the scope for a single paper.\n\n2. In terms of reconstruction accuracy the U-Net is shown to be outperformed but I would assume that a stronger baseline trained end-to-end like a Var-Net https://arxiv.org/abs/1704.00447 would perform significantly better in terms of PSNR and SSIM. Further, the conditioning makes the model lose the advantage of unconditioned generative models for solving inverse problems to be independent from the forward map. \n    Since the authors also do not convincingly demonstrate cGANs to be superior in terms of fairness and uncertainty estimation (see previous pont), I wonder about the general usefulness of cGANs as proposed in this work.\n    However, I acknowledge that this work significantly outperforms previous cGAN based approaches and thus is a valuable contribution to the field of generative models for solving inverse problems. Also the authors discuss the dependence on a specific forward model as a limitation and refer to ongoing work that removes this constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Weakness**: \nThe experiments did not demonstrate fairness or classifier calibration advantages due to posterior sampling. Also, it's not clear how a radiologist would benefit from a pixel-wise standard-deviation map.  \n**Response**: \nOur goal for this paper was to design a fast and accurate posterior sampler and to evaluate the quality of its samples using principled metrics like CFID. In ongoing work, we are currently studying the *application* of posterior sampling to particular fields.  This includes studying how posterior sampling can be leveraged to ensure fairness, how it can be leveraged to quantify the uncertainty of downstream tasks like classification, and how uncertainty information can be best communicated to radiologists. Although these application-related questions are important, we feel that they are outside the scope of our NeurIPS submission.  \n\n- **Weakness**: \nThe UNet is a weak baseline.  Better to compare to the E2E-VarNet.  \n**Response**: \nPlease see our global rebuttal. \n\n- **Weakness**: \nDue to a dependence on the forward model and the lack of demonstration of fairness and uncertainty estimation, the general usefulness of cGANs in posterior sampling is unclear.  \n**Response**: \nIt's true that the dependence of cGANs on the forward model is a disadvantage, although we are addressing this limitation in ongoing work.  But unconditional approaches like Langevin/score/diffusion methods have their own disadvantages, such as much slower sample-generation time and the need to have an explicit model for the likelihood function $p(\\\\boldsymbol{y}|\\\\boldsymbol{x})$. "}, {"review_id": "cZTI7A5oC5", "reviewer": "Reviewer_qA2y", "summary": "This paper introduces a regularization term, comprising an L1 penalty and a standard-deviation reward, for the conditional GAN with Wasserstein loss. The objective is to ensure the generation of high-quality samples for a specific input observation y. The proposed approach is evaluated on two image recovery tasks: MRI reconstruction and face image inpainting. The numerical results illustrate its effectiveness in comparison to alternative methods.", "questions": "See [Weaknesses].", "limitations": "Yes, the authors have discussed their limitations in the Conclusion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed regularization term is simple and easy to implement and be added to GAN models;\n2) Discussed different regularization options and demonstrated why they did not work in this case;\n3) It includes theoretical analysis and empirical results;\n4) The overall writing and presentation are good and easy to follow.", "weaknesses": "1) The novelty of this paper sounds a little bit limited. The supervision-L1 to regularize conditional GAN is proposed by previous work. So it looks like it only added a new reward term on top of previous work.\n\n2) The two experimental tasks are all linear image restoration tasks. However, it is curious how the proposed method would work on nonlinear tasks.\n\n3) It would also be interesting to see whether the proposed method would work on a more complicated real-world dataset for the inpainting task.\n\n4) The current experimental image datasets are low resolution. Will it still outperform other models in terms of quality and computational cost on higher-resolution image datasets?\n\n5) How to determine the hyper-parameter: belta_adv for different datasets?\n\n6) Minor issue: there are some writing inconsistencies (e.g., the capitalization of the headline of 4.2 /4.3 is different from the headline of 3.1/3.2).\n\n\n#------------------------------------------------------------------------------------------------#\n\n#------------------------------------------------------------------------------------------------#\n\nMost of my previous concerns have been addressed by the authors during the rebuttal, thus I have increased my rating accordingly. However, it would be great if the authors could add some experiments about non-linear image inverse problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Weakness**: \nNovelty is a bit limited. Only added a new reward term.  \n**Response**: \nPlease see the global rebuttal.\n\n- **Weakness**: \nOnly linear inverse problems were demonstrated.  How would it perform on nonlinear ones?  \n**Response**:\nOur method should work equally well for nonlinear inverse problems, since it makes no assumption about model linearity. In fact, it does not assume any particular forward model, and instead infers the posterior distribution from a supervised training set $\\\\{(\\\\boldsymbol{x}_i,\\\\boldsymbol{y}_i)\\\\}$. We chose to experiment with MRI and large-scale inpainting because they are well-studied problems with strong baselines.\n\n- **Weakness**:\nHow would the proposed method work on a more complicated real-world dataset for the inpainting task?  \n**Response**: \nOur primary contribution is a cGAN regularization that prevents mode-collapse. To our knowledge, there is nothing about our regularization that favors particular datasets over others. For good reconstruction quality, a cGAN must have a good generator, but the literature contains many examples of high-performance generators for a wealth of applications. \n\n- **Weakness**:\nThe current experimental image datasets are low resolution. Will it still outperform other models in terms of quality and computational cost on higher-resolution image datasets?  \n**Response**:\nFor MRI, we use 384x384 pixel, 8 coil, complex-valued brain images, which contain 2.4 million unknowns. This size is standard for clinical practice, so it would not be correct to call it low resolution. For inpainting, we use the 256x256 version of CelebA-HQ, which is the standard resolution of CelebA-HQ in the literature. For example, the 256x256 size was needed access a public pretrained CelebA-HQ model for the Score SDE baseline.  In any case, there is no reason to believe that our proposed regularization would work poorly with larger images.\n\n- **Weakness**: \nHow to choose $\\beta_{\\mathsf{adv}}$?  \n**Response**: \nIn our experience, $\\beta_{\\mathsf{adv}}$ does not require precise tuning. Our strategy has been to set $\\beta_{\\mathsf{adv}}$ to roughly equate the size of the adversarial loss term and the regularization term.\n"}]

Author rebuttal:
We sincerely thank the reviewers for the time spent reading and reviewing our manuscript.  The feedback is very useful and will help to improve the quality of the final paper if it is accepted.  In this global rebuttal we respond to questions that were raised by multiple reviewers.

- **Weakness**:
Novelty is a bit limited. Supervised-L1 regularization has been proposed and so you are only adding a reward term.  
**Response**:
Yes, using supervised-L1 regularization in cGANs is well known. Our contributions include proving that the typical L1 and L2 regularizations actually *induce* mode collapse in cGANs, and proposing a novel regularization that prevents mode collapse. Our framework also addresses the tuning of the standard-deviation reward weight, which is a difficult problem in practice.  Our regularization provably leads to true-posterior sampling in simple cases and empirically yields high-fidelity samples in real-world imaging applications. 

- **Question**:
Isn't the UNet a weak baseline?  
**Response**:
There are many variations of the UNet. The one we used for both our baseline point-estimator and for our generator is considerably more powerful than the UNet described in the fastMRI paper (Zbontar'18), to which comparisons are typically made. First, our UNet operates on complex multicoil images and we coil-combine its output, while the UNet in (Zbontar'18) operates on single-coil magnitude images after coil-combining its input. Second, we employ several architectural modifications, as detailed in Appendix I.1.2 of our paper. Third, we apply data-consistency as described in Appendix H. In any case, our main motivation to compare to this UNet is that it differs from our proposed method only in the training loss, and therefore illustrates the effect of our proposed loss/regularization.

- **Question**:
Why not compare to an end-to-end (E2E) trained VarNet, as in (Jalal'21)?  
**Response**:
Thanks, this is a good idea.  We trained an E2E VarNet using the authors' code at 8x acceleration with the same sampling mask and training data used for the other methods under test. The results in the attached pdf show our method outperforming the E2E VarNet in all metrics except PSNR. We will include these results in the final paper. Like (Jalal'21), we used ESPIRIT-based ground truth for training and performance evaluation, as justified in Appendix A.1 of (Jalal'21).

- **Question**:
Why not compare to more recent diffusion models like (Chung'21), DPS (Chung'22), DDRM (Kawar'22), or PiDGM (Song'23)?  
**Response**:
We chose well-known diffusion baselines rather than the absolute latest methods. Regarding the cited papers:  
-- The MRI method from (Chung'21) is not practical in that it requires the inputs to be precisely normalized, which the authors do by peeking at the ground truth. See line 52 in their code:
https://github.com/HJ-harry/score-MRI/blob/main/inference_multi-coil_hybrid.py. We found that their method fails when the normalization constant is estimated from sub-sampled data.  
-- As for DPS, the main contribution is a way of handling significantly *noisy* inverse problems, whereas our MRI and inpainting problems have very high SNR. Also, it is not clear how to best apply this method to complex-valued multicoil MRI, which is a nontrivial undertaking. 
-- We attempted to run DDRM as suggested for our inpainting problem, but no pre-trained CelebA-HQ model is currently available (see https://github.com/bahjat-kawar/ddrm/issues/24).  
-- For PiDGM, we could not find public code.  
In any case, a persistent issue with diffusion methods is their slow evaluation speed.  For example, DDRM at 20 steps would be the fastest among the above methods, but after running it on our hardware with batch size of 128 and image size of 256x256, we found that it is still 275x slower than our cGAN. 

- **General comment**:
Our main contribution is a novel cGAN regularization for posterior sampling. Our regularization can be used with any generator architecture, the choice of which will strongly affect performance in any given application.  Although the performance we achieved on MRI and inpainting tasks was good, it should not be taken as a ceiling, because our choice of generator architecture can certainly be improved upon. For example, the Langevin/score/diffusion models and E2E-VarNet effectively use a series of many CNNs, with data-fidelity steps in-between, whereas we use only a single CNN. 

Meta-review: The paper presented a conditional GAN method for general imaging problems that addresses the posterior collapse problem using an L1 instead of L2 penalty. The four reviewers all agreed the paper should be accepted, with three reviewers raising their scores in light of the author discussion period. The reviewers recognized the impressive performance of the regularization technique for sampling images and one reviewer appreciated the added discussion of areas where the method may have limitations.
Decision: Accept (poster)