Title: A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative adversarial network

inverse problems

posterior sampling

cGAN

GAN
Abstract: In image recovery problems, one seeks to infer an image from distorted, incomplete, and/or noise-corrupted measurements.
Such problems arise in magnetic resonance imaging (MRI), computed tomography, deblurring, super-resolution, inpainting, phase retrieval, image-to-image translation, and other applications. Given a training set of signal/measurement pairs, we seek to do more than just produce one good image estimate. Rather, we aim to rapidly and accurately sample from the posterior distribution. To do this,
we propose a regularized conditional Wasserstein GAN that generates dozens of high-quality posterior samples per second. Our regularization comprises an $\ell_1$ penalty and an adaptively weighted standard-deviation reward. Using quantitative evaluation metrics like conditional Fréchet inception distance, we demonstrate that our method produces state-of-the-art posterior samples in both multicoil MRI and large-scale inpainting applications. The code for our model can be found here: https://github.com/matt-bendel/rcGAN.

A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems

Abstract

In image recovery problems, one seeks to infer an image from distorted, incomplete, and/or noise-corrupted measurements. Such problems arise in magnetic resonance imaging (MRI), computed tomography, deblurring, super-resolution, inpainting, phase retrieval, image-to-image translation, and other applications. Given a training set of signal/measurement pairs, we seek to do more than just produce one good image estimate. Rather, we aim to rapidly and accurately sample from the posterior distribution. To do this, we propose a regularized conditional Wasserstein GAN that generates dozens of high-quality posterior samples per second. Our regularization comprises an /lscript 1 penalty and an adaptively weighted standard-deviation reward. Using quantitative evaluation metrics like conditional Fr´ echet inception distance, we demonstrate that our method produces state-of-the-art posterior samples in both multicoil MRI and large-scale inpainting applications. The code for our model can be found here: https://github.com/matt-bendel/rcGAN .

1 Introduction

We consider image recovery, where one observes measurements y = M ( x ) of the true image x that may be masked, distorted, and/or corrupted x with noise, and the goal is to infer x from y . This includes linear inverse problems arising in, e.g., deblurring, super-resolution, inpainting, colorization, computed tomography (CT), and magnetic resonance imaging (MRI), where y = Ax + w with known linear operator A and noise w . But it also includes non-linear inverse problems like those arising in phase-retrieval and dequantization, as well as image-to-image translation problems. In all cases, it is impossible to perfectly infer x from y .

Image recovery is often posed as finding the single 'best' recovery ̂ x , which is known as a point estimate of x [1]. But point estimation is problematic due to the perception-distortion tradeoff [2], which establishes a fundamental tradeoff between distortion (defined as some distance between ̂ x and x ) and perceptual quality (defined as some distance between ̂ x and the set of clean images). For example, the minimum mean-squared error (MMSE) recovery ̂ x mmse is optimal in terms of /lscript 2 distortion, but can be unrealistically smooth. Although one could instead compute an approximation of the maximum a posteriori (MAP) estimate [3] or minimize some combination of perceptual and distortion losses, it's unclear which combination would be most appropriate.

Another major limitation with point estimation is that it's unclear how certain one can be about the recovered ̂ x . For example, with deep-learning-based recovery, it's possible to hallucinate a nice-looking ̂ x , but is it correct? Quantifying the uncertainty in ̂ x is especially important in medical applications such as MRI, where a diagnosis must be made based on the measurements y . Rather than simply reporting our best guess of whether a pathology is present or absent based on ̂ x , we might want to report the probability that the pathology is present (given all available data).

Yet another problem with point estimation is that the estimated ̂ x could pose issues with fairness [4]. For example, say we are inpainting a face within an image. With a racially heterogeneous prior distribution, ̂ x mmse (being the posterior mean) will be biased towards the most predominant race. The same will be true of many other point estimates x .

Concretely, given a training dataset of image/measurement pairs { ( x t , y t ) } T t =1 , our goal is to learn a generating function G θ that, for a new y , maps random code vectors z ∼ N ( 0 , I ) to posterior samples ̂ x = G θ ( z , y ) ∼ p x | y ( ·| y ) . There exist several well-known approaches to this task, with recent literature focusing on conditional generative adversarial networks (cGANs) [7, 8, 9, 10], conditional variational autoencoders (cVAEs) [11, 12, 13], conditional normalizing flows (cNFs) [14, 15, 16], and score/diffusion/Langevin-based generative models [17, 18, 19, 20, 21]. Despite it being a longstanding problem, posterior image sampling remains challenging. Although score/diffusion/Langevin approaches have dominated the recent literature due to advances in accuracy and diversity, their sample-generation speeds remain orders-of-magnitude behind those of cGANs, cVAEs, and cNFs.

̂ To address the aforementioned limitations of point estimation, we focus on generating samples from the posterior distribution p x | y ( ·| y ) , which represents the complete state-of-knowledge about x given the measurements y . The posterior correctly fuses prior knowledge with measurement knowledge, thereby alleviating any concerns about fairness (assuming the data used to represent the prior is fairly chosen [4]). Furthermore, the posterior directly facilitates uncertainty quantification via, e.g., pixel-wise standard deviations or pathology detection probabilities (see Appendix A). Also, if it was important to report a single 'good' recovery, then posterior sampling leads to an easy navigation of the perception-distortion tradeoff. For example, averaging P ≥ 1 posterior samples gives a close approximation to the less-distorted-but-oversmooth ̂ x mmse with large P and sharp-but-more-distorted ̂ x with small P . Additionally, posterior sampling unlocks other important capabilities such as adaptive acquisition [5] and counterfactual diagnosis [6].

We choose to focus on cGANs, which are typically regarded as generating samples of high quality but low diversity. Our proposed cGAN tackles the lack-of-diversity issue using a novel regularization that consists of supervised/lscript 1 loss plus an adaptively weighted standard-deviation (SD) reward. This is not a heuristic choice; we prove that our regularization enforces consistency with the true posterior mean and covariance under certain conditions.

Experimentally, we demonstrate our regularized cGAN on accelerated MRI and large-scale face completion/inpainting. We consider these applications for three main reasons. First, uncertainty quantification in MRI, and fairness in face-generation, are both of paramount importance. Second, posterior-sampling has been well studied for both applications, and fine-tuned cGANs [9] and score/Langevin-based approaches [19, 20] are readily available. Third, the linear operator ' A ' manifests very differently in these two applications, 1 which illustrates the versatility of our approach. To quantify performance, we focus on conditional Fr´ echet inception distance (CFID) [22], which is a principled way to quantify the difference between two high-dimensional posterior distributions, although we also report other metrics. Our results show the proposed regularized cGAN (rcGAN) outperforming existing cGANs [8, 23, 9] and the score/diffusion/Langevin approaches from [19] and [20] in all tested metrics, while generating samples ∼ 10 4 times faster than [19, 20].

2 Problem formulation and background

We build on the Wasserstein cGAN framework from [8]. The goal is to design a generator network G θ : Z × Y → X such that, for fixed y , the random variable ̂ x = G θ ( z , y ) induced by z ∼ p z has a distribution that best matches the posterior p x | y ( ·| y ) in Wasserstein-1 distance. Here, X , Y , and Z denote the sets of x , y , and z , respectively, and z is drawn independently of y .

The Wasserstein-1 distance can be expressed as

where L 1 denotes functions that are 1-Lipschitz with respect to their first argument and D : X × Y → R is a 'critic' or 'discriminator' that tries to distinguish between true x and generated x given y .

Since we want the method to work for typical values of y , we define a loss by taking an expectation of (1) over y ∼ p y . Since the expectation commutes with the supremum in (1), we have [8]

In practice, D is implemented by a neural network D φ with parameters φ , and ( θ , φ ) are trained by alternately minimizing

with respect to θ and minimizing -L adv ( θ , φ ) + L gp ( φ ) with respect to φ , where L gp ( φ ) is a gradient penalty that is used to encourage D φ ∈ L 1 [24]. Furthemore, the expectation over x and y in (4) is replaced in practice by a sample average over the training examples { ( x t , y t ) } T t =1 .

One of the main challenges with the cGAN framework in image recovery problems is that, for each measurement example y t , there is only a single image example x t . Thus, with the previously described training methodology, there is no incentive for the generator to produce diverse samples G ( z , y ) | z ∼ p z for a fixed y . This can lead to the generator learning to ignore the code vector z , which causes a form of 'mode collapse.'

Although issues with stability and mode collapse are also present in unconditional GANs (uGANs) or discretely conditioned cGANs [25], the causes are fundamentally different than in continuously conditioned cGANs like ours. With continuously conditioned cGANs, there is only one example of a valid x t for each given y t , whereas with uGANs there are many x t and with discretely conditioned cGANs there are many x t for each conditioning class. As a result, most strategies that are used to combat mode-collapse in uGANs [26, 27, 28] are not well suited to cGANs. For example, mini-batch discrimination strategies like MBSD [29], where the discriminator aims to distinguish a mini-batch of true samples { x t } from a mini-batch of generated samples { ̂ x t } , don't work with cGANs because the posterior statistics are very different than the prior statistics.

To combat mode collapse in cGANs, Adler & ¨ Oktem [8] proposed to use a three-input discriminator D adler φ : X × X × Y → R and replace L adv from (4) with the loss which rewards variation between the first and second inputs to D adler φ . They then proved that minimizing L adler adv in place of L adv does not compromise the Wasserstein cGAN objective, i.e., arg min θ L adler adv ( θ , φ ) = arg min θ L adv ( θ , φ ) . As we show in Section 4, this approach does prevent complete mode collapse, but it leaves much room for improvement.

3 Proposed method

3.1 Proposed regularization: supervised/lscript 1 plus SD reward

We now propose a novel cGAN regularization framework. To train the generator, we propose to solve

with appropriately chosen β adv , β SD > 0 and P train ≥ 2 , where the regularizer is constructed from the P train -sample supervised/lscript 1 loss and standard-deviation (SD) reward terms

and where { ̂ x i } denote the generated samples and x ( P ) their P -sample average:

̂ ̂ ̂ The use of supervised/lscript 1 loss and SD reward in (7) is not heuristic. As shown in Proposition 3.1, it encourages the samples { ̂ x i } to match the true posterior in both mean and covariance.

(a)

(b)

(c)

(p)

3

Cost

3

Cost

3

Cost

Cost

2.5

Correct

2.5

Correct

2

2.5

Correct

2.5

Correct

5.5

2

1.8

C2

1.8

2

2

5

1.5

1

1.6

1.4

6

1.5

1.6

1.4

6

1.5

1.5

4.5

3.5

0.5

1.2

0.5

0.5

0.5

3

-1

1

2

Q

-1

Q

1

2

1.2

1

2

-1

0

1

2

2.5

μ

μ

Proposition 3.1. Suppose P train ≥ 2 and θ has complete control over the y -conditional mean and covariance of ̂ x i . Then the parameters θ ∗ = arg min θ L 1 , SD ,P train ( θ , β N SD ) with yield generated statistics

̂ See Appendix B for a proof. In imaging applications, ̂ x i and x may not be independent Gaussian conditioned on y , and so the value of β SD in (11) may not be appropriate. Thus we propose a method to automatically tune β SD in Section 3.2.

̂ when the elements of ̂ x i and x are independent Gaussian conditioned on y . Thus, minimizing L 1 , SD ,P train encourages the y -conditional mean and covariance of x i to match those of the true x .

Figure 1 shows a toy example with parameters θ = [ µ, σ ] /latticetop , generator G θ ( z, y ) = µ + σ z , and z ∼ N (0 , 1) , giving generated posterior p ̂ x | y ( x | y ) = N ( x ; µ, σ 2 ) . Assuming the true p x | y ( x | y ) = N ( x ; µ 0 , σ 2 0 ) , Figs. 1(a)-(b) show that, by minimizing the proposed L 1 , SD ,P train ( θ , β N SD ) regularization over θ = [ µ, σ ] /latticetop for any P train ≥ 2 , we recover the true θ 0 = [ µ 0 , σ 0 ] /latticetop . They also show that the cost function steepens as P train decreases, with agrees with our empirical finding that P train = 2 tends to work best in practice.

We note that regularizing a cGAN with supervised/lscript 1 loss alone is not new; see, e.g., [7]. In fact, the use of supervised/lscript 1 loss is often preferred over /lscript 2 in image recovery because it results in sharper, more visually pleasing results [30]. But regularizing a cGAN using supervised/lscript 1 loss alone can push the generator towards mode collapse, for reasons described below. For example, in [7], /lscript 1 -induced mode collapse led the authors to use dropout to induce generator variation, instead of random z i .

Why not supervised/lscript 2 regularization? One may wonder: Why regularize using supervised/lscript 1 loss plus an SD reward in (7) and not a more conventional choice like supervised/lscript 2 loss plus a variance reward, or even supervised/lscript 2 loss alone? We start by discussing the latter.

The use of supervised/lscript 2 regularization in a cGAN can be found in [7]. In this case, to train the generator, one would aim to solve arg min θ {L adv ( θ , φ ) + λ L 2 ( θ ) } with some λ > 0 and

Ohayon et al. [23] revived this idea for the explicit purpose of fighting mode collapse. In practice, however, the E z term in (13) must be implemented by a finite-sample average, giving

L 2 ,P train ( θ ) /defines E x , z1 ,..., zP , y {∥ ∥ x -1 P train ∑ P train i =1 G θ ( z i , y ) ∥ ∥ 2 2 } (14) for some P train ≥ 2 . For example, Ohayon's implementation [31] used P train = 8 . As we show in Proposition 3.2, L 2 ,P train induces mode collapse rather than prevents it.

̂ Thus, minimizing L 2 ,P train encourages mode collapse.

Proposition 3.2. Say P train is finite and θ has complete control over the y -conditional mean and covariance of ̂ x i . Then the parameters θ ∗ = arg min θ L 2 ,P train ( θ ) yield generated statistics E zi | y { ̂ x i ( θ ∗ ) | y } = E x | y { x | y } = ̂ x mmse (15a) Cov zi | y { x i ( θ ∗ ) | y } = 0 . (15b)

β

SD

= 0

β

SD

=

β

N

SD

β

SD

= 1

.

2

β

N

SD

β

SD

= 1

.

4

β

N

SD

β

SD

= 1

.

6

β

N

SD

β

SD

= 1

.

8

β

N

SD

Number of averaged outputs,

P

, on a log scale

40

38

36

Theory

Actual

34

100

101

100

101

100

101

100

101

100

101

100

101

The proof (see Appendix C) follows from the bias-variance decomposition of (14), i.e.,

̂ ̂ ̂ where e mmse /defines x -x mmse is the MMSE error. Figure 1(c) shows that L 2 ,P train regularization causes mode collapse in the toy example, and Section 4.2 shows that it causes mode collapse in MRI.

Whynot supervised /lscript 2 plus a variance reward? To mitigate L 2 ,P train 's incentive for mode-collapse, the second term in (16) could be canceled using a variance reward, giving

since Appendix D shows that L var ,P train ( θ ) is an unbiased estimator of the posterior trace-covariance:

L var ,P train ( θ ) = E y { tr[Cov zi | y { ̂ x i ( θ ) | y } ] } for any P train ≥ 2 . (19) However, the resulting L 2 , var ,P train regularizer in (17) does not encourage the generated covariance to match the true posterior covariance, unlike the proposed L 1 , SD ,P train regularizer in (7) (recall Proposition 3.1). For the toy example, this behavior is visible in Fig. 1(d).

3.2 Auto-tuning of SD reward weight β SD

We now propose a method to auto-tune β SD in (7) for a given training dataset. Our approach is based on the observation that larger values of β SD tend to yield samples ̂ x i with more variation. But more variation is not necessarily better; we want samples with the correct amount of variation. To assess the correct amount of variation, we compare the expected /lscript 2 error of the P -sample average ̂ x ( P ) to that of ̂ x (1) . When { ̂ x i } are true-posterior samples, these errors follow a particular relationship, as established by Proposition 3.3 below (see Appendix E for a proof).

Proposition 3.3. Say ̂ x i ∼ p x | y ( ·| y ) are independent samples of the true posterior and, for any P ≥ 1 , their P -sample average is x ( P ) /defines 1 P ∑ P i =1 x i . Then

Experimentally we find that E 1 / E P grows with the SD reward weight β SD . (See Fig. 2.) Thus, we propose to adjust β SD so that the observed SNR-gain ratio E 1 / E P val matches the correct ratio (2 P val ) / ( P val +1) from (20), for some P val ≥ 2 that does not need to match P train . (We use P val = 8 in Section 4.) In particular, at each training epoch τ , we approximate E P val and E 1 as follows:

̂ E 1 , τ /defines 1 V ∑ V v =1 ‖ G θ τ ( z 1 ,v , y v ) -x v ‖ 2 2 , (22) with validation set { ( x v , y v ) } V v =1 and i.i.d. codes { z i,v } P val i =1 . We update β SD using gradient descent:

4 Numerical experiments

4.1 Conditional Fr´ echet inception distance

As previously stated, our goal is to train a generator G θ so that, for typical fixed values of y , the generated distribution p ̂ x | y ( ·| y ) matches the true posterior p x | y ( ·| y ) . It is essential to have a quantitative metric for evaluating performance with respect to this goal. For example, it is not enough that the generated samples are 'accurate' in the sense of being close to the ground truth, nor is it enough that they are 'diverse' according to some heuristically chosen metric.

We quantify posterior-approximation quality using the conditional Fr´ echet inception distance (CFID) [22], a computationally efficient approximation to the conditional Wasserstein distance

̂ In (24), W 2 ( p a , p b ) denotes the Wasserstein-2 distance between distributions p a and p b , defined as

where Π ( p a , p b ) /defines { p a , b : p a = ∫ p a , b d b and p b = ∫ p a , b d a } denotes the set of joint distributions p a , b with prescribed marginals p a and p b . Similar to how FID [32]-a popular uGAN metric-is computed, CFID approximates CWD (24) as follows: i) the random vectors x , ̂ x , and y are replaced by low-dimensional embeddings x , ̂ x , and y , generated by the convolutional layers of a deep network, and ii) the embedding distributions p x | y and p ̂ x | y are approximated by multivariate Gaussians. More details on CFID are given in Appendix F.

4.2 MRI experiments

We consider parallel MRI recovery, where the goal is to recover a complex-valued multicoil image x from zero-filled measurements y (see Appendix G for details).

Data. For training data { x t } , we used the first 8 slices of all fastMRI [33] T2 brain training volumes with at least 8 coils, cropping to 384 × 384 pixels and compressing to 8 virtual coils [34], yielding 12 200 training images. Using the same procedure, 2 376 testing and 784 validation images were obtained from the fastMRI T2 brain testing volumes. From the 2 376 testing images, 72 were randomly selected to evaluate the Langevin technique [19] in order to limit its sample generation to 6 days. To create the measurement y t , we transformed x t to the Fourier domain, sampled using pseudo-random GRO patterns [35] at acceleration R = 4 and R = 8 , and Fourier-transformed the zero-filled k-space measurements back to the (complex, multicoil) image domain.

Architecture. We use a UNet [36] for our generator and a standard CNN for our generator, along with data-consistency as in Appendix H. Architecture and training details are given in Appendix I.

Competitors. We compare our cGAN to the Adler and ¨ Oktem's cGAN [8], Ohayon et al.'s cGAN [23], Jalal et al.'s fastMRI Langevin approach [19], and Sriram et al.'s E2E-VarNet [37]. The cGAN from [8] uses generator loss β adv L adler adv ( θ , φ ) and discriminator loss -L adler adv ( θ , φ ) + α 1 L gp ( φ ) + α 2 L drift ( φ ) , while the cGAN from [23] uses generator loss β adv L adv ( θ , φ ) + L 2 ,P ( θ ) and discriminator loss -L adv ( θ , φ ) + α 1 L gp ( φ ) + α 2 L drift ( φ ) . Each used the value of β adv specified in the original paper. All cGANs used the same generator and discriminator architectures, except that [8] used extra discriminator input channels to facilitate the 3-input loss (5). For the fastMRI Langevin approach [19], we did not modify the authors' implementation in [38] except to use the GRO sampling mask. For the E2E-VarNet [37], we use the same training procedure and hyperparameters outlined in [19] except that we use the GRO sampling mask.

Testing. To evaluate performance, we converted the multicoil outputs ̂ x i to complex-valued images using SENSE-based coil combining [39] with ESPIRiT-estimated [40] coil sensitivity maps, as described in Appendix G. Magnitude images were used to compute performance metrics.

Results. Table 1 shows CFID, FID, APSD /defines ( 1 NP ∑ P i =1 ‖ ̂ x ( P ) -̂ x i ‖ 2 ) 1 / 2 , and 4-sample generation time at R ∈ { 4 , 8 } . (C)FID was computed using VGG-16 (not Inception-v3) to better align with radiologists' perceptions [41]. As previously described, the Langevin method was evaluated using only 72 test images. Because CFID is biased at small sample sizes [22], we re-evaluated the other

methods using all 2 376 test images, and again using all 14 576 training and test images. Table 1 shows that our approach gave significantly better CFID and FID than the competitors. Also, the APSD of Ohayon et al.'s cGAN was an order-of-magnitude smaller than the others, indicating mode collapse. The cGANs generated samples 3 800 times faster than the Langevin approach from [19].

Tables 2 and 3 show PSNR, SSIM, LPIPS [42], and DISTS [43] for the P -sample average ̂ x ( P ) at P ∈ { 1 , 2 , 4 , 8 , 16 , 32 } and R ∈ { 4 , 8 } , respectively. While the E2E-VarNet achieves the best PSNR at R ∈ { 4 , 8 } and the best SSIM at R = 4 , the proposed cGAN achieves the best LPIPS and DISTS performances at R ∈ { 4 , 8 } when P = 2 and the best SSIM at R = 8 when P = 8 . The P dependence can be explained by the perception-distortion tradeoff [2]: as P increases, ̂ x ( P ) transitions from better perceptual quality to lower /lscript 2 distortion. PSNR favors P → ∞ (e.g., /lscript 2 optimality) while the other metrics favor particular combinations of perceptual quality and distortion. The plots in Appendices K.1 and K.2 show zoomed-in versions of ̂ x ( P ) that visually demonstrate the perception-distortion tradeoff at P ∈ { 1 , 2 , 4 , 32 } : smaller P yield sharper images with more variability from the ground truth, while larger P yield smoother reconstructions.

Figure 3 shows zoomed versions of two posterior samples ̂ x i , as well as ̂ x ( P ) , at P = 32 and R = 8 . The posterior samples show meaningful variations for the proposed method, essentially no variation for Ohayon et al.'s cGAN, and vertical or horizontal reconstruction artifacts for Adler & ¨ Oktem's cGAN and the Langevin method, respectively. The ̂ x ( P ) plots show that these artifacts are mostly suppressed by sample averaging with large P .

Figure 4 shows examples of ̂ x ( P ) , along with the corresponding pixel-wise absolute errors | ̂ x ( P ) -x | and pixel-wise SD images ( 1 P ∑ P i =1 ( ̂ x ( P ) -̂ x i ) 2 ) 1 / 2 , for P = 32 and R = 8 . The absolute-error image for the Langevin technique looks more diffuse than those of the other methods in the brain region. The fact that it is brighter in the air region (i.e., near the edges) is a consequence of minor differences in sensitivity-map estimation. The pixel-wise SD images show a lack of variability for the E2E-VarNet, which does not generate posterior samples, as well as Ohayon et al.'s cGAN, due to mode collapse. The Langevin pixel-wise SD images show localized hot-spots that appear to be reconstruction artifacts.

Appendices K.1 and K.2 show other example MRI recoveries with zoomed pixel-wise SD images at R = 4 and R = 8 , respectively. Notably, Figures K.10 and K.11 show strong hallucinations for Langevin recovery at R = 8 , as highlighted by the red arrows.

cGAN (ours)

cGAN (Ohayon)

cGAN (Adler)

Langevin (Jalal)

E2E-VarNet

Truth

Truth

Sample

Sample

Average (P=32)

4.3 Inpainting experiments

In this section, our goal is to complete a large missing square in a face image.

Data. We used 256 × 256 CelebA-HQ face images [29] and a centered 128 × 128 missing square. We randomly split the dataset, yielding 27 000 training, 2 000 validation, and 1 000 testing images.

Architecture. For our cGAN, we use the CoModGAN generator and discriminator from [9] with our proposed L 1 , SD ,P train regularization. Unlike [9], we do not use MBSD [29] at the discriminator.

Training/validation/testing. We use the same general training and testing procedure described in Section 4.2, but with β adv = 5e-3, n batch = 100 , and 110 epochs of cGAN training. Running PyTorch on a server with 4 Tesla A100 GPUs, each with 82 GB of memory, the cGAN training takes approximately 2 days. FID was evaluated on 1 000 test images using P =32 samples per measurement. To avoid the bias that would result from evaluating CFID on only 1 000 images (see Appendix J.1), CFID was evaluated on all 30 000 images with P = 1 .

Competitors. We compare with the state-of-the-art CoModGAN [9] and Score-based SDE [20] approaches. For CoModGAN, we use the PyTorch implementation from [44]. CoModGAN differs from our cGAN only in its use of MBSD and lack of L 1 , SD ,P train regularization. For Song et al.'s SDE, we use the authors' implementation from [45] with their pretrained weights and the settings they suggested for the 256 × 256 CelebA-HQ dataset.

Truth

E2E-VarNet

cGAN (ours)

cGAN (Ohayon)

cGAN (Adler)

Langevin (Jalal)

1e-04

5e-05

0e+00

5e-05

3e-05

Oe+00

Results. Table 4 shows test CFID, FID, and 128-sample generation time. The table shows that our approach gave the best CFID and FID, and that the cGANs generated samples 13 000 times faster than the score-based method. Figure 5 shows an example of five generated samples for the three methods under test. The samples are all quite good, although a few generated by CoModGAN and the score-based technique have minor artifacts. Some samples generated by our technique show almond-shaped eyes, demonstrating fairness. Additional examples are given in Appendix K.3.

5 Conclusion

We propose a novel regularization framework for image-recovery cGANs that consists of supervised/lscript 1 loss plus an appropriately weighted standard-deviation reward. For the case of an independent Gaussian posterior, we proved that our regularization yields generated samples that agree with the true-posterior samples in both mean and covariance. We also established limitations for alternatives based on supervised/lscript 2 regularization with or without a variance reward. For practical datasets, we proposed a method to auto-tune our standard-deviation reward weight.

Experiments on parallel MRI and large-scale face inpainting showed our proposed method outperforming all cGAN and score-based competitors in CFID, which measures posterior-approximation quality, as well as other metrics like FID, PSNR, SSIM, LPIPS, and DISTS. Furthermore, it generates samples thousands of times faster than Langevin/score-based approaches.

In ongoing work, we are extending our approach so that it can be trained to handle a wide range of recovery tasks, such as MRI with a generic acceleration and sampling mask [46], or inpainting with a generic mask. We are also extending our approach to other applications, such as super-resolution, deblurring, compressive sensing, denoising, and phase retrieval.

Original

SgcGAN (ours)N

SgMaskedN

SgCoModGAN N

SgScore-SDEN

Limitations. We acknowledge several limitations of our work. First, while our current work focuses on how to build a fast and accurate posterior sampler, it's not yet clear how to best exploit the resulting posterior samples in each given application. For example, in MRI, where the posterior distribution has the potential to help assess uncertainty in image recovery, it's still not quite clear how to best convey uncertainty information to radiologists (e.g., they may not gain much from pixel-wise SD images). More work is needed on this front. Second, we acknowledge that, because radiologists are risk-averse, more studies are needed before they will feel comfortable incorporating generative deep-learning-based methods into the clinical workflow. Third, we acknowledge that the visual quality of our R = 8 MRI reconstructions falls below clinical standards. Fourth, some caution is needed when interpreting our CFID, FID, and DISTS perceptual metrics because the VGG-16 backbone used to compute them was trained on ImageNet data. Although there is some evidence that the resulting DISTS metric correlates well with radiologists' perceptions [41], there is also evidence that ImageNet-trained features may discard information that is diagnostically relevant in medical imaging [47]. Thus our results need to be validated with a pathology-centric radiologist study before they can be considered relevant to clinical practice.

Acknowledgments and Disclosure of Funding

The authors are funded in part by the National Institutes of Health under grant R01-EB029957 and the National Science Foundation under grant CCF-1955587.

References

E. L. Lehmann and G. Casella, Theory of point estimation . Springer Science & Business Media, 2006. 1

Y. Blau and T. Michaeli, 'The perception-distortion tradeoff,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. , pp. 6228-6237, 2018. 1, 7

C. K. Sønderby, J. Caballero, L. Theis, W. Shi, and F. Husz´ ar, 'Amortised MAP inference for image super-resolution,' in Proc. Int. Conf. on Learn. Rep. , 2017. 1, 19

N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, 'A survey on bias and fairness in machine learning,' ACM Comput. Surveys , vol. 54, no. 6, pp. 1-35, 2021. 2

T. Sanchez, I. Krawczuk, Z. Sun, and V. Cevher, 'Uncertainty-driven adaptive sampling via GANs,' in Proc. Neural Inf. Process. Syst. Workshop , 2020. 2

C.-H. Chang, E. Creager, A. Goldenberg, and D. Duvenaud, 'Explaining image classifiers by counterfactual generation,' in Proc. Int. Conf. on Learn. Rep. , 2019. 2

P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros, 'Image-to-image translation with conditional adversarial networks,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. , pp. 1125-1134, 2017. 2, 4

J. Adler and O. ¨ Oktem, 'Deep Bayesian inversion,' arXiv:1811.05910 , 2018. 2, 3, 6, 7, 8, 19, 20, 21

S. Zhao, J. Cui, Y. Sheng, Y. Dong, X. Liang, E. I.-C. Chang, and Y. Xu, 'Large scale image completion via co-modulated generative adversarial networks,' in Proc. Int. Conf. on Learn. Rep. , 2021. 2, 8, 9, 20, 21

H. Zhao, H. Li, S. Maurer-Stroh, and L. Cheng, 'Synthesizing retinal and neuronal images with generative adversarial nets,' Med. Image Analysis , vol. 49, 07 2018. 2

V. Edupuganti, M. Mardani, S. Vasanawala, and J. Pauly, 'Uncertainty quantification in deep MRI reconstruction,' IEEE Trans. Med. Imag. , vol. 40, pp. 239-250, Jan. 2021. 2

F. Tonolini, J. Radford, A. Turpin, D. Faccio, and R. Murray-Smith, 'Variational inference for computational imaging inverse problems,' J. Mach. Learn. Res. , vol. 21, no. 179, pp. 1-46, 2020. 2

K. Sohn, H. Lee, and X. Yan, 'Learning structured output representation using deep conditional generative models,' in Proc. Neural Inf. Process. Syst. Conf. , 2015. 2

L. Ardizzone, C. L¨ uth, J. Kruse, C. Rother, and U. K¨ othe, 'Guided image generation with conditional invertible neural networks,' arXiv:1907.02392 , 2019. 2

C. Winkler, D. Worrall, E. Hoogeboom, and M. Welling, 'Learning likelihoods with conditional normalizing flows,' arXiv preprint arXiv:1912.00042 , 2019. 2

H. Sun and K. L. Bouman, 'Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging,' in Proc. AAAI Conf. Artificial Intell. , vol. 35, pp. 2628-2637, 2021. 2

M. Welling and Y. W. Teh, 'Bayesian learning via stochastic gradient Langevin dynamics,' in Proc. Int. Conf. Mach. Learn. , pp. 681-688, 2011. 2

Y. Song and S. Ermon, 'Improved techniques for training score-based generative models,' in Proc. Neural Inf. Process. Syst. Conf. , 2020. 2

A. Jalal, M. Arvinte, G. Daras, E. Price, A. Dimakis, and J. Tamir, 'Robust compressed sensing MRI with deep generative priors,' in Proc. Neural Inf. Process. Syst. Conf. , 2021. 2, 6, 7, 8, 20, 21

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-based generative modeling through stochastic differential equations,' in Proc. Int. Conf. on Learn. Rep. , 2021. 2, 8, 9, 21

Y. Song, L. Shen, L. Xing, and S. Ermon, 'Solving inverse problems in medical imaging with score-based generative models,' in Proc. Int. Conf. on Learn. Rep. , 2022. 2

M. Soloveitchik, T. Diskin, E. Morin, and A. Wiesel, 'Conditional Frechet inception distance,' arXiv:2103.11521 , 2021. 2, 6, 18, 21

G. Ohayon, T. Adrai, G. Vaksman, M. Elad, and P. Milanfar, 'High perceptual quality image denoising with a posterior sampling CGAN,' in Proc. IEEE Int. Conf. Comput. Vis. Workshops , vol. 10, pp. 1805-1813, 2021. 2, 4, 6, 7, 8, 20, 21

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, 'Improved training of Wasserstein GANs,' in Proc. Neural Inf. Process. Syst. Conf. , p. 5769-5779, 2017. 3, 19

M. Mirza and S. Osindero, 'Conditional generative adversarial nets,' arXiv:1411.1784 , 2014. 3

E. Schonfeld, B. Schiele, and A. Khoreva, 'A U-Net based discriminator for generative adversarial networks,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. , pp. 8207-8216, 2020. 3

T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, 'Training generative adversarial networks with limited data,' in Proc. Neural Inf. Process. Syst. Conf. , vol. 33, pp. 12104-12114, 2020. 3

Z. Zhao, S. Singh, H. Lee, Z. Zhang, A. Odena, and H. Zhang, 'Improved consistency regularization for GANs,' in Proc. AAAI Conf. Artificial Intell. , vol. 35, pp. 11033-11041, 2021. 3

T. Karras, T. Aila, S. Laine, and J. Lehtinen, 'Progressive growing of GANs for improved quality, stability, and variation,' in Proc. Int. Conf. on Learn. Rep. , 2018. 3, 8, 19

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, 'Loss functions for image restoration with neural networks,' IEEE Trans. Comput. Imag. , vol. 3, pp. 47-57, Mar. 2017. 4

G. Ohayon, T. Adrai, G. Vaksman, M. Elad, and P. Milanfar, 'High perceptual quality image denoising with a posterior sampling CGAN.' Downloaded from https://github.com/theoad/pscgan , July 2021. 4

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, 'GANs trained by a two time-scale update rule converge to a local Nash equilibrium,' in Proc. Neural Inf. Process. Syst. Conf. , vol. 30, 2017. 6

J. Zbontar, F. Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, Z. Zhang, M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, J. Pinkerton, D. Wang, N. Yakubova, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, 'fastMRI: An open dataset and benchmarks for accelerated MRI,' arXiv:1811.08839 , 2018. 6, 19

T. Zhang, J. M. Pauly, S. S. Vasanawala, and M. Lustig, 'Coil compression for accelerated imaging with Cartesian sampling,' Magn. Reson. Med. , vol. 69, no. 2, pp. 571-582, 2013. 6

M. Joshi, A. Pruitt, C. Chen, Y. Liu, and R. Ahmad, 'Technical report (v1.0)-pseudo-random cartesian sampling for dynamic MRI,' arXiv:2206.03630 , 2022. 6

O. Ronneberger, P. Fischer, and T. Brox, 'U-Net: Convolutional networks for biomedical image segmentation,' in Proc. Intl. Conf. Med. Image Comput. Comput. Assist. Intervent. , pp. 234-241, 2015. 6, 20

A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson, 'End-to-end variational networks for accelerated MRI reconstruction,' in Proc. Intl. Conf. Med. Image Comput. Comput. Assist. Intervent. , pp. 64-73, 2020. 6, 7, 8, 20

A. Jalal, M. Arvinte, G. Daras, E. Price, A. Dimakis, and J. Tamir, 'csgm-mri-langevin.' https:// github.com/utcsilab/csgm-mri-langevin , 2021. Accessed: 2021-12-05. 6, 20

K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, 'SENSE: Sensitivity encoding for fast MRI,' Magn. Reson. Med. , vol. 42, no. 5, pp. 952-962, 1999. 6, 18

M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig, 'ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA,' Magn. Reson. Med. , vol. 71, no. 3, pp. 990-1001, 2014. 6, 18

S. Kastryulin, J. Zakirov, N. Pezzotti, and D. V . Dylov, 'Image quality assessment for magnetic resonance imaging,' arXiv:2203.07809 , 2022. 6, 10, 18

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The unreasonable effectiveness of deep features as a perceptual metric,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. , pp. 586-595, 2018. 7

K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, 'Image quality assessment: Unifying structure and texture similarity,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 5, pp. 2567-2581, 2020. 7

Y. Zeng, 'co-mod-gan-pytorch.' Downloaded from https://github.com/zengxianyu/ co-mod-gan-pytorch , Sept. 2022. 8, 21

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-based generative modeling through stochastic differential equations.' Downloaded from https://github.com/yang-song/ score_sde_pytorch , Oct. 2022. 8, 21

M. C. Bendel, R. Ahmad, and P. Schniter, 'Mask-agnostic posterior sampling MRI via conditional GANs with guided reconstruction,' in Proc. NeurIPS Workshop on Deep Inverse Problems , 2023. 9

T. Kynk¨ a¨ anniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen, 'The role of imagenet classes in fr´ echet inception distance,' arXiv:2203.06026 , 2022. 10

F. C. Leone, L. S. Nelson, and R. Nottingham, 'The folded normal distribution,' Technometrics , vol. 3, no. 4, pp. 543-550, 1961. 14

K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' arXiv:1409.1556 , 2014. 18

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the inception architecture for computer vision,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. , 2016. 18

F. Ong and M. Lustig, 'SigPy: A python package for high performance iterative reconstruction,' in Proc. Annu. Meeting ISMRM , vol. 4819, 2019. 18

D. Chen and M. E. Davies, 'Deep decomposition learning for inverse imaging problems,' in Proc. European Conf. Comp. Vision , pp. 510-526, 2020. 19

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in Proc. Int. Conf. on Learn. Rep. , 2015. 19

P. Deora, B. Vasudeva, S. Bhattacharya, and P. M. Pradhan, 'Structure preserving compressive sensing MRI reconstruction using generative adversarial networks,' in Proc. IEEE Conf. Comp. Vision Pattern Recog. Workshop , pp. 2211-2219, June 2020. 20

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a regularization term, comprising an L1 penalty and a standard-deviation reward, for the conditional GAN with Wasserstein loss. The objective is to ensure the generation of high-quality samples for a specific input observation y. The proposed approach is evaluated on two image recovery tasks: MRI reconstruction and face image inpainting. The numerical results illustrate its effectiveness in comparison to alternative methods.\n\nSTRENGTHS:\n1) The proposed regularization term is simple and easy to implement and be added to GAN models;\n2) Discussed different regularization options and demonstrated why they did not work in this case;\n3) It includes theoretical analysis and empirical results;\n4) The overall writing and presentation are good and easy to follow.\n\nWEAKNESSES:\n1) The novelty of this paper sounds a little bit limited. The supervision-L1 to regularize conditional GAN is proposed by previous work. So it looks like it only added a new reward term on top of previous work.\n\n2) The two experimental tasks are all linear image restoration tasks. However, it is curious how the proposed method would work on nonlinear tasks.\n\n3) It would also be interesting to see whether the proposed method would work on a more complicated real-world dataset for the inpainting task.\n\n4) The current experimental image datasets are low resolution. Will it still outperform other models in terms of quality and computational cost on higher-resolution image datasets?\n\n5) How to determine the hyper-parameter: belta_adv for different datasets?\n\n6) Minor issue: there are some writing inconsistencies (e.g., the capitalization of the headline of 4.2 /4.3 is different from the headline of 3.1/3.2).\n\n\n#------------------------------------------------------------------------------------------------#\n\n#------------------------------------------------------------------------------------------------#\n\nMost of my previous concerns have been addressed by the authors during the rebuttal, thus I have increased my rating accordingly. However, it would be great if the authors could add some experiments about non-linear image inverse problems.\n\nQUESTIONS:\nSee [Weaknesses].\n\nLIMITATIONS:\nYes, the authors have discussed their limitations in the Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work uses conditional GANs for solving image recovery problems including face-inpainting and accelerated MRI. In particular, a regularization scheme is proposed that tackles the lack-of-diversity issue related to posterior sampling from cGANs. The authors show that the proposed regularization allows sampling from the true posterior under the assumption of a Gaussian signal prior. Empirically the method outperforms a U-Net trained end-to-end, existing cGANs and two score/diffusion/Langevin approaches in terms of accuracy and the latter also significantly in terms of speed.\n\nSTRENGTHS:\n1. The paper is written very clearly and is well organized.\n2. The proposed regularization significantly improves over existing cGAN approaches and hence will be of interest for the community.\n3. The theoretical discussion of a toy example with Gaussian signal prior is insightful and contributes to the understanding of the proposed regularization.\n\nWEAKNESSES:\n1. The authors argue that sampling from the posterior distribution alleviates any concerns about fairness and facilitates uncertainty quantification and further that uncertainty quantification in MRI, and fairness in face-generation, are both of paramount importance. In Appendix A they show that a classifier trained on images from the prior distribution maintains its performance when averaged over many samples from the posterior but not when evaluated on a point estimate.\nHowever, the empirical experiments in Section 4 do not evaluate these points in detail. Regarding uncertainty quantification for MRI Figure 4 shows one example of the pixel-wise standard deviation (SD) but a discussion on the usefulness and practical applicability is missing. To me it seems that in the depicted SD map the model is equally uncertain about every edge in the image making me wonder about the usefulness of this map to a radiologist.\nRegarding fairness in face-generation 5 different reconstructions of one image are shown and the authors argue that the fact that some samples show almond-shaped eyes demonstrates fairness. \n    Here, a more detailed evaluation would strengthen the contribution although I acknowledge that this might be out the scope for a single paper.\n\n2. In terms of reconstruction accuracy the U-Net is shown to be outperformed but I would assume that a stronger baseline trained end-to-end like a Var-Net https://arxiv.org/abs/1704.00447 would perform significantly better in terms of PSNR and SSIM. Further, the conditioning makes the model lose the advantage of unconditioned generative models for solving inverse problems to be independent from the forward map. \n    Since the authors also do not convincingly demonstrate cGANs to be superior in terms of fairness and uncertainty estimation (see previous pont), I wonder about the general usefulness of cGANs as proposed in this work.\n    However, I acknowledge that this work significantly outperforms previous cGAN based approaches and thus is a valuable contribution to the field of generative models for solving inverse problems. Also the authors discuss the dependence on a specific forward model as a limitation and refer to ongoing work that removes this constraint.\n\nQUESTIONS:\nNo additional questions.\n\nLIMITATIONS:\nLimitations are discussed adequately by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new method based on conditional GANs (cGANs) for generating posterior samples for solutions of inverse problems. The motivation comes from the fact that standard approaches to inverse problems typically use point estimates, which due to the perception-distortion tradeoff will lead to blurry images. Conversely, reconstruction via sampling can be used to navigate the perception-distortion tradeoff by either sampling small counts to have sharp images, or averaging many samples to approximate the point estimate reconstruction.\n\nAll of the above is known from the previous literature. The present paper's specific contributions largely center on the loss function and regularization design for cGANs to avoid posterior collapse and increase sample quality. The regularizer design is based on matching the means and covariances of the samples to the means and covariances of the true distribution. The manuscript chooses to compute these statistics over the $\\ell_1$ norm rather than the $\\ell_2$ norm, and the paper shows that $\\ell_2$ norm can in fact lead to collapse.\n\nIn numerical experiments the paper demonstrates superior FID and conditional FID performance for the proposed method, indicating that it is superior at distribution matching than the previous methods considered. For distortion metrics, the experiments demonstrate that superior distortion performance can be achieved by averaging over samples.\n\nSTRENGTHS:\n**Originality**\n- Interesting new regularization method for training cGANs for image restoration tasks.\n\n**Quality**\n- The paper's quantitative results indicate SotA performance across a variety of tasks for both MRI reconstruction and image inpainting.\n- Principled use of VGG weights instead of Inception weights for FID calculation and correlations to radiologist perception.\n- A derivation motivating the use of both the supervised standard deviation reward and the application of this award over $\\ell_1$ norms.\n\n**Clarity**\n- Excellent presentation of perception-distortion tradeoff and its relevance to imaging inverse problems.\n- Detailed outline of the proposed method with the appendices, as well as relevant mathematics concerning the various tasks in the results.\n\n**Significance**\n- There is substantial interest in the image reconstruction/restoration community now, much of which has focused on conditional diffusion models. The establishment of competitive performance in this space with cGANs (as well as the superior performance to past cGAN methods) will be of interest.\n\nWEAKNESSES:\n**Quality**\n- The MRI images still demonstrate substantial artifacts from all methods at 8X acceleration. It is possible to find a number of features and fine details that are missing, so I doubt these would be accepted for clinical use.\n- Although VGG may correlate better with radiologist scores, it is still an ImageNet feature backbone and contains nullspaces that may be poor for the tasks under consideration (Kynkäänniemi, 2023).\n- There is a literature dedicated to so-called \"plug-and-play\" methods that are relevant to the present work, but was not considered. See (Venkatakrishnan, 2013) and follow-up papers.\n- The U-Net is used as a surrogate for point-estimate methods in the MRI example, but it is known in the MRI community that the U-Net is a poor reconstructor. A more standard comparison would be to use the End-to-End Variational Network (Sriram 2020), which is available from the fastMRI GitHub repository. This comparison was included in the cited paper of (Jalal 2021).\n- The competing methods for inpainting seem out of date. Dozens of methods based on conditional probabilistic diffusion have emerged over the last year (e.g., DDRM (Kawar 2022) and follow-ups), so I do not find these experiments as strong as the MRI ones.\n\n1. Kynkäänniemi, Tuomas, et al. \"The Role of ImageNet Classes in Frechet Inception Distance.\" ICLR (2023).\n2. Venkatakrishnan, Singanallur V., Charles A. Bouman, and Brendt Wohlberg. \"Plug-and-play priors for model based reconstruction.\" 2013 IEEE global conference on signal and information processing. IEEE, 2013.\n3. Sriram, Anuroop, et al. \"End-to-end variational networks for accelerated MRI reconstruction.\" Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. Springer International Publishing, 2020.\n4. Kawar, Bahjat, et al. \"Denoising diffusion restoration models.\" Advances in Neural Information Processing Systems 35 (2022): 23593-23606.\n\nQUESTIONS:\nI am positive on this work, but a number of aspects on the presentation could be improved. I would appreciate the authors' responses on improving the following, mostly related to the limitations of the work and contextualization with existing literature on medical image reconstruction.\n\n1. Although sampling-based reconstruction can do a better job of distribution-matching and uncertainty estimates, the artifacts introduced are unpredictable. For medical imaging, calibrated radiologists can work around predictable artifacts such as blurring. In fact, even point-estimate reconstruction has not fully taken over the medical field, because simple linear reconstructions have more predictable artifacts than deep learning point estimates. The use of generative sampling compounds this effect and is likely to be substantial limitation on the present work. I think a discussion on this should be added as limitation (some sketch of the proposed discussion would be appreciated during the rebuttal).\n2. The paper primarily examines distribution matching over ImageNet features, which are likely to be quite different than the features useful for medical imaging. (Kynkäänniemi, 2023) shows the presence of perceptual nullspaces that could very well ignore small pathologies. Even the self-supervised solution of (Kynkäänniemi, 2023) is likely to be of benefit for medical imaging analysis, because self-supervised features rely on large objects at the center of the image.\n3. Although the classifier matching proof in Appendix A seems theoretically sound, it relies on a few limits and calibration assumptions that are likely to deviate from reality, so I am skeptical this is a grounded proof that classifiers trained on fully-sampled images will perform adequately on the sampled images.\n4. The paper could benefit with a comparison to the End-to-End variational network, which is a standard comparison method in the MRI literature and was previously used in the Jalal paper. Although the present manuscript has shown superiority to (Jalal, 2021) on metrics, sampling-based methods have not yet taken the general field of medical image reconstruction, so the comparison is still valid.\n5. A cursory overview of the Plug-and-Play reconstruction literature would provide a more complete picture of previous work.\n\nLIMITATIONS:\nAs discussed under Questions, the paper could expand its discussion of limitations, including the issues of 1) issues artifact consistency posterior sampling methods with human observers and 2) the use of metrics based on ImageNet feature alignment for medical imaging tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a cGAN (conditional generative adversarial network) method for general image inverse problems (IR). Conventional cGAN methods for IR usually suffer from mode collapse that the generators learned to ignore latent inputs and are unable to generate diverse image reconstruction samples. Motivated by existing work, this work proposes to incorporate multiple latent input to the GAN model during training to mitigate the mode collapse effect. Accordingly, a sophisticated designed supervised training loss function is also presented in this work. The numerical results show that the proposed method generates comparable or even better results to cGAN and score-based baseline methods on compressive MRI and image inpainting.\n\nSTRENGTHS:\n1, The paper is overall well motivated, clearly written and easy to follow. \n\n2, The proposed posterior sampling-based method is effective for generating multiple samples compared to some score-based methods. The multiple sampling can also be applied for model uncertainty estimation, which is important for many medical imaging problems.\n\n3, The theoretical interpretation on the mean and covariance of predicted posterior distribution is informative, though it is sometimes unrealistic for most real-world imaging problems.\n\nWEAKNESSES:\n1, One contribution of this paper seems still to be based on the idea of Adler & Oktem [9], where multiple latent vectors are as inputs to the discriminator and generator module instead of only one input. In the meantime, as pointed out by the author as well, adding extra regularization term in the adversarial GAN loss during training is not completely new. \n\n2, This proposal ignores a body of work for posterior sampling based on Bayesian deep learning such as Monte Carlo dropout, deep ensembles, etc. There’s also a lack of numerical comparisons with these methods, making the baseline methods not sufficient enough. \n\n3, While some ablation studies provided in its original submission, the effects of regularization parameter $\\beta_{SD}$ and number $P$ on the final imaging reconstruction are still not comprehensive and could be improved. For example, a visual illustration or numerical comparison of several proposed sampling results due to different settings of $\\beta_{SD}$ is preferable.\n\n4, The assumption for proposition 3.1 that samples $\\widehat x_i$ and $x$ are independent Gaussian conditioned on y is not true for imaging applications, weekend the theoretical contribution. \n\n5, Some technical details and notations are confusing and could be improved in their current state.\n\nQUESTIONS:\n1), There seems to have a gap between proposition 3.3 and the gradient descent on the $\\beta_{SD}$ and could be clarified for reader’s better understanding. For example, there is no clear formular between $\\beta_{SD}$ and the ratio in Eq 20., making the derivative of Eq. 21-23 a bit confusing. It is unclear how Eq. 21-22 is formulated and how this led to the update of $\\beta_{SD}$. Please clarify this part in the revision.\n\n2), At the same line above, it is unclear why the number of latent $P$ is not the same for training and testing (e.g., line 166,” for some $P_{\\text{val}} $ that does not need to match $P_{\\text{train}} $.”). Is this means the performance of this proposal for each image is sensitive to the choice of value $P$? It is unclear when there is a lack of enough validation dataset to finetune $\\beta$ or if there is a mismatch between train and testing dataset. As a result, the effects of hyper-parameter settings for different settings are unclear. \n\n3), The paper lacks sufficient discussion on the CS-MRI experiments.  For instance, based on the description of Sec. 4.2, “we transformed $x_t$ to the Fourier domain”, seems the authors do not follow the trends of other methods in the fast MRI leaderboard that directly perform down-sampling on the kspace data. This seems like an important distinction and must be made clear in the paper. There is no discussion on how the measurement noise is modeled or how it affects the reconstruction performance. Also, while there are some technical discussions on the CSMRI formulation in the appendix, there is no description of how the forward model simulation in this paper differs from the actual forward model of the real MRI system.\n\n4), One benefit of posterior sampling is for model uncertainty estimation. While this work can generate multiple samples and show the pixel-wise standard deviation in Figure 4, it is difficult to evaluate if it can indeed reflect the mean-absolute error for image reconstruction. It would be great to have a quantitative result for this study.\n\n5), There is a lack of comparison to more recent diffusion model-based IR methods, where many have been demonstrated good posterior sampling performance for various imaging tasks, such as [R1, R2, R3].\n\n6),  The run time comparison to  the selected Langevin dynamic baselines seems to be suboptimal, as there are many efficient sampling strategies can be applied for IR diffusion/score-based methods, where fewer iteration steps could result in appealing reconstruction [R3].\n\n7), It is unclear which forward model/dataset are used for Figure 2 in the paper. If it is plotted using images, it is unclear how is the curve in theory plotted. Moreover, it would be more informative if Figure2 can be directly represented by $\\epsilon_1/\\epsilon_P$ and $\\beta_{SD}$. \n\n[R1] Score-based diffusion models for accelerated MRI, Chung. etal, 2021.\n\n[R2] Diffusion posterior sampling for general noisy inverse problems, Chung. etal, 2022.\n\n[R3] Pseudoinverse-Guided Diffusion Models for Inverse Problems, Song. etal, 2023.\n\nLIMITATIONS:\nThe authors discussed their work's limitations in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}