SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {2, \operatorname {v a r}, P _ {\text {t r a i n}}} (\boldsymbol {\theta}) \triangleq \mathcal {L} _ {2, P _ {\text {t r a i n}}} (\boldsymbol {\theta}) - \frac {1}{P _ {\text {t r a i n}}} \mathcal {L} _ {\text {v a r}, P _ {\text {t r a i n}}} (\boldsymbol {\theta}) \tag {17}
$$
$$
\text {w i t h} \mathcal {L} _ {\operatorname {v a r}, P _ {\operatorname {t r a i n}}} (\boldsymbol {\theta}) \triangleq \frac {1}{P _ {\operatorname {t r a i n}} - 1} \sum_ {i = 1} ^ {P _ {\operatorname {t r a i n}}} \mathrm {E} _ {\mathbf {z} _ {1}, \dots , \mathbf {z} _ {\mathrm {P}}, \mathrm {y}} \left\{\| \widehat {\boldsymbol {x}} _ {i} (\boldsymbol {\theta}) - \widehat {\boldsymbol {x}} _ {(P)} (\boldsymbol {\theta}) \| _ {2} ^ {2} \right\}. \tag {18}
$$
since Appendix D shows that  $\mathcal{L}_{\mathrm{var},P_{\mathrm{train}}}(\pmb {\theta})$  is an unbiased estimator of the posterior trace-covariance:
$$
\mathcal {L} _ {\text {v a r}, P _ {\text {t r a i n}}} (\boldsymbol {\theta}) = \mathrm {E} _ {\mathrm {y}} \left\{\operatorname {t r} \left[ \operatorname {C o v} _ {\mathrm {z} _ {\mathrm {i}} \mid \mathrm {y}} \left\{\widehat {\boldsymbol {x}} _ {i} (\boldsymbol {\theta}) \mid \boldsymbol {y} \right\} \right] \right\} \text {f o r a n y} P _ {\text {t r a i n}} \geq 2. \tag {19}
$$
However, the resulting  $\mathcal{L}_{2,\mathrm{var},P_{\mathrm{train}}}$  regularizer in (17) does not encourage the generated covariance to match the true posterior covariance, unlike the proposed  $\mathcal{L}_{1,\mathrm{SD},P_{\mathrm{train}}}$  regularizer in (7) (recall Proposition 3.1). For the toy example, this behavior is visible in Fig. 1(d).
3.2 Auto-tuning of SD reward weight  $\beta_{\mathrm{SD}}$
We now propose a method to auto-tune  $\beta_{\mathrm{SD}}$  in (7) for a given training dataset. Our approach is based on the observation that larger values of  $\beta_{\mathrm{SD}}$  tend to yield samples  $\widehat{\pmb{x}}_i$  with more variation. But more variation is not necessarily better; we want samples with the correct amount of variation. To assess the correct amount of variation, we compare the expected  $\ell_2$  error of the  $P$ -sample average  $\widehat{\pmb{x}}_{(P)}$  to that of  $\widehat{\pmb{x}}_{(1)}$ . When  $\{\widehat{\pmb{x}}_i\}$  are true-posterior samples, these errors follow a particular relationship, as established by Proposition 3.3 below (see Appendix E for a proof).
Proposition 3.3. Say  $\widehat{\pmb{x}}_i\sim p_{\mathsf{X|Y}}(\cdot |\pmb {y})$  are independent samples of the true posterior and, for any  $P\geq 1$ , their  $P$ -sample average is  $\widehat{\pmb{x}}_{(P)}\triangleq \frac{1}{P}\sum_{i = 1}^{P}\widehat{\pmb{x}}_i$ . Then
$$
\mathcal {E} _ {P} \triangleq \mathrm {E} \left\{\| \widehat {\boldsymbol {x}} _ {(P)} - \boldsymbol {x} \| _ {2} ^ {2} | \boldsymbol {y} \right\} = \frac {P + 1}{P} \mathcal {E} _ {\text {m m s e}}, \quad \text {w h i c h i m p l i e s} \quad \frac {\mathcal {E} _ {1}}{\mathcal {E} _ {P}} = \frac {2 P}{P + 1}. \tag {20}
$$
Experimentally we find that  $\mathcal{E}_1 / \mathcal{E}_P$  grows with the SD reward weight  $\beta_{\mathrm{SD}}$ . (See Fig. 2.) Thus, we propose to adjust  $\beta_{\mathrm{SD}}$  so that the observed SNR-gain ratio  $\mathcal{E}_1 / \mathcal{E}_{P_{\mathrm{val}}}$  matches the correct ratio  $(2P_{\mathrm{val}}) / (P_{\mathrm{val}} + 1)$  from (20), for some  $P_{\mathrm{val}} \geq 2$  that does not need to match  $P_{\mathrm{train}}$ . (We use  $P_{\mathrm{val}} = 8$  in Section 4.) In particular, at each training epoch  $\tau$ , we approximate  $\mathcal{E}_{P_{\mathrm{val}}}$  and  $\mathcal{E}_1$  as follows:
$$
\widehat {\mathcal {E}} _ {P _ {\text {v a l}}, \tau} \triangleq \frac {1}{V} \sum_ {v = 1} ^ {V} \| \frac {1}{P _ {\text {v a l}}} \sum_ {i = 1} ^ {P _ {\text {v a l}}} G _ {\boldsymbol {\theta} _ {\tau}} \left(\boldsymbol {z} _ {i, v}, \boldsymbol {y} _ {v}\right) - \boldsymbol {x} _ {v} \| _ {2} ^ {2} \tag {21}
$$
$$
\widehat {\mathcal {E}} _ {1, \tau} \triangleq \frac {1}{V} \sum_ {v = 1} ^ {V} \| G _ {\boldsymbol {\theta} _ {\tau}} \left(\boldsymbol {z} _ {1, v}, \boldsymbol {y} _ {v}\right) - \boldsymbol {x} _ {v} \| _ {2} ^ {2}, \tag {22}
$$
with validation set  $\{(x_v, y_v)\}_{v=1}^V$  and i.i.d. codes  $\{z_{i,v}\}_{i=1}^{P_{\mathrm{val}}}$ . We update  $\beta_{\mathrm{SD}}$  using gradient descent:
$$
\beta_ {\mathbf {S D}, \tau + 1} = \beta_ {\mathbf {S D}, \tau} - \mu_ {\mathbf {S D}} \cdot \left([ \hat {\mathcal {E}} _ {1, \tau} / \hat {\mathcal {E}} _ {P _ {\text {v a l}}, \tau} ] _ {\mathrm {d B}} - [ 2 P _ {\text {v a l}} / (P _ {\text {v a l}} + 1) ] _ {\mathrm {d B}}\right) \beta_ {\mathbf {S D}} ^ {N} \text {f o r} \tau = 0, 1, 2, \dots . \tag {23}
$$
with  $\beta_{\mathsf{SD},0} = \beta_{\mathsf{SD}}^{\mathcal{N}}$ , some  $\mu_{\mathsf{SD}} > 0$ , and  $[x]_{\mathrm{dB}} \triangleq 10 \log_{10}(x)$ .
4.1 Conditional Fréchet inception distance
As previously stated, our goal is to train a generator  $G_{\theta}$  so that, for typical fixed values of  $y$ , the generated distribution  $p_{\mathbf{x}|\mathbf{y}}(\cdot |\mathbf{y})$  matches the true posterior  $p_{\mathbf{x}|\mathbf{y}}(\cdot |\mathbf{y})$ . It is essential to have a quantitative metric for evaluating performance with respect to this goal. For example, it is not enough that the generated samples are "accurate" in the sense of being close to the ground truth, nor is it enough that they are "diverse" according to some heuristically chosen metric.
We quantify posterior-approximation quality using the conditional Fréchet inception distance (CFID) [22], a computationally efficient approximation to the conditional Wasserstein distance
$$
\mathrm {C W D} \triangleq \mathrm {E} _ {\mathcal {Y}} \left\{W _ {2} \left(p _ {\mathrm {x} | \mathrm {y}} (\cdot , \boldsymbol {y}), p _ {\hat {\mathrm {x}} | \mathrm {y}} (\cdot , \boldsymbol {y})\right) \right\}. \tag {24}
$$
In (24),  $W_{2}(p_{\mathrm{a}}, p_{\mathrm{b}})$  denotes the Wasserstein-2 distance between distributions  $p_{\mathrm{a}}$  and  $p_{\mathrm{b}}$ , defined as
$$
W _ {2} \left(p _ {\mathrm {a}}, p _ {\mathrm {b}}\right) \triangleq \min  _ {p _ {\mathrm {a}, \mathrm {b}} \in \Pi \left(p _ {\mathrm {a}}, p _ {\mathrm {b}}\right)} \mathrm {E} _ {\mathrm {a}, \mathrm {b}} \left\{\left\| \boldsymbol {a} - \boldsymbol {b} \right\| _ {2} ^ {2} \right\}, \tag {25}
$$
where  $\Pi(p_{\mathrm{a}}, p_{\mathrm{b}}) \triangleq \left\{p_{\mathrm{a}, \mathrm{b}} : p_{\mathrm{a}} = \int p_{\mathrm{a}, \mathrm{b}} \, \mathrm{d}\boldsymbol{b} \text{ and } p_{\mathrm{b}} = \int p_{\mathrm{a}, \mathrm{b}} \, \mathrm{d}\boldsymbol{a} \right\}$  denotes the set of joint distributions  $p_{\mathrm{a}, \mathrm{b}}$  with prescribed marginals  $p_{\mathrm{a}}$  and  $p_{\mathrm{b}}$ . Similar to how FID [32]—a popular uGAN metric—is computed, CFID approximates CWD (24) as follows: i) the random vectors  $\boldsymbol{x}$ ,  $\hat{\boldsymbol{x}}$ , and  $\boldsymbol{y}$  are replaced by low-dimensional embeddings  $\underline{x}$ ,  $\hat{\underline{x}}$ , and  $\boldsymbol{y}$ , generated by the convolutional layers of a deep network, and ii) the embedding distributions  $p_{\underline{x}|\underline{y}}$  and  $p_{\hat{\underline{x}}|\underline{y}}$  are approximated by multivariate Gaussians. More details on CFID are given in Appendix F.
4.2 MRI experiments
We consider parallel MRI recovery, where the goal is to recover a complex-valued multicoil image  $\mathbf{x}$  from zero-filled measurements  $\mathbf{y}$  (see Appendix G for details).
Data. For training data  $\{\pmb{x}_t\}$ , we used the first 8 slices of all fastMRI [33] T2 brain training volumes with at least 8 coils, cropping to  $384 \times 384$  pixels and compressing to 8 virtual coils [34], yielding 12 200 training images. Using the same procedure, 2 376 testing and 784 validation images were obtained from the fastMRI T2 brain testing volumes. From the 2 376 testing images, 72 were randomly selected to evaluate the Langevin technique [19] in order to limit its sample generation to 6 days. To create the measurement  $\pmb{y}_t$ , we transformed  $\pmb{x}_t$  to the Fourier domain, sampled using pseudo-random GRO patterns [35] at acceleration  $R = 4$  and  $R = 8$ , and Fourier-transformed the zero-filled k-space measurements back to the (complex, multicoil) image domain.
Architecture. We use a UNet [36] for our generator and a standard CNN for our generator, along with data-consistency as in Appendix H. Architecture and training details are given in Appendix I.
Competitors. We compare our cGAN to the Adler and Oktem's cGAN [8], Ohayon et al.'s cGAN [23], Jalal et al.'s fastMRI Langevin approach [19], and Sriram et al.'s E2E-VarNet [37]. The cGAN from [8] uses generator loss  $\beta_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}^{\mathrm{ader}}(\pmb {\theta},\phi)$  and discriminator loss  $-\mathcal{L}_{\mathrm{adv}}^{\mathrm{ader}}(\pmb {\theta},\phi) + \alpha_1\mathcal{L}_{\mathrm{gp}}(\phi) + \alpha_2\mathcal{L}_{\mathrm{drift}}(\phi)$ , while the cGAN from [23] uses generator loss  $\beta_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}(\pmb {\theta},\phi) + \mathcal{L}_{2,P}(\pmb {\theta})$  and discriminator loss  $-\mathcal{L}_{\mathrm{adv}}(\pmb {\theta},\phi) + \alpha_1\mathcal{L}_{\mathrm{gp}}(\phi) + \alpha_2\mathcal{L}_{\mathrm{drift}}(\phi)$ . Each used the value of  $\beta_{\mathrm{adv}}$  specified in the original paper. All cGANs used the same generator and discriminator architectures, except that [8] used extra discriminator input channels to facilitate the 3-input loss (5). For the fastMRI Langevin approach [19], we did not modify the authors' implementation in [38] except to use the GRO sampling mask. For the E2E-VarNet [37], we use the same training procedure and hyperparameters outlined in [19] except that we use the GRO sampling mask.
Testing. To evaluate performance, we converted the multicoil outputs  $\widehat{\pmb{x}}_i$  to complex-valued images using SENSE-based coil combining [39] with ESPIRiT-estimated [40] coil sensitivity maps, as described in Appendix G. Magnitude images were used to compute performance metrics.
Results. Table 1 shows CFID, FID, APSD  $\triangleq$ $(\frac{1}{NP}\sum_{i = 1}^{P}\| \widehat{\pmb{x}}_{(P)} - \widehat{\pmb{x}}_i\|^2)^{1 / 2}$ , and 4-sample generation time at  $R\in \{4,8\}$ . (C)FID was computed using VGG-16 (not Inception-v3) to better align with radiologists' perceptions [41]. As previously described, the Langevin method was evaluated using only 72 test images. Because CFID is biased at small sample sizes [22], we re-evaluated the other
Table 1: Average MRI results at  $R \in  \{ 4,8\}$  . CFID  ${}^{1}$  ,FID,and APSD used 72 test samples and  $P = {32},{\mathrm{{CFID}}}^{2}$  used 2376 test samples and  $P = 8$  ,and  ${\mathrm{{CFID}}}^{3}$  used all 14576 samples and  $P = 1$
Model	R=4	R=8
CFID1↓	CFID2↓	CFID3↓	FID↓	APSD	Time (4)↓	CFID1↓	CFID2↓	CFID3↓	FID↓	APSD	Time (4)↓
E2E-VarNet (Sriram et al. [37])	7.47	6.99	6.61	8.84	0.0	310ms	7.82	6.81	6.31	8.40	0.0	316ms
Langevin (Jalal et al. [19])	5.29	-	-	6.12	5.9e-6	14 min	7.34	-	-	14.32	7.6e-6	14 min
cGAN (Adler & Öktem [8])	6.39	4.27	3.82	5.25	3.9e-6	217 ms	10.10	6.30	5.72	10.77	7.7e-6	217 ms
cGAN (Ohayon et al. [23])	4.06	3.27	2.95	6.45	7.2e-8	217 ms	6.04	4.59	4.27	11.05	7.7e-7	217 ms
cGAN (Ours)	3.10	1.54	1.29	3.75	3.8e-6	217 ms	4.87	2.23	1.79	7.72	7.6e-6	217 ms
Table 2: Average PSNR, SSIM, LPIPS, and DISTS of  $\hat{\pmb{x}}_{(P)}$  versus  $P$  for  $R = 4$  MRI
Model	PSNR↑	SSIM↑
P=1	P=2	P=4	P=8	P=16	P=32	P=1	P=2	P=4	P=8	P=16	P=32
E2E-VarNet (Sriram et al. [37])	39.93	-	-	-	-	-	0.9641	-	-	-	-	-
Langevin (Jalal et al. [19])	36.04	37.02	37.65	37.99	38.17	38.27	0.8989	0.9138	0.9218	0.9260	0.9281	0.9292
cGAN (Adler & Öktem [8])	35.63	36.64	37.24	37.56	37.73	37.82	0.9330	0.9445	0.9478	0.9480	0.9477	0.9473
cGAN (Ohayon et al. [23])	39.44	39.46	39.46	39.47	39.47	39.47	0.9558	0.9546	0.9539	0.9535	0.9533	0.9532
cGAN (Ours)	36.96	38.14	38.84	39.24	39.44	39.55	0.9440	0.9526	0.9544	0.9542	0.9537	0.9533
Model	LPIPS↓	DISTS↓
P=1	P=2	P=4	P=8	P=16	P=32	P=1	P=2	P=4	P=8	P=16	P=32
E2E-VarNet (Sriram et al. [37])	0.0316	-	-	-	-	-	0.0859	-	-	-	-	-
Langevin (Jalal et al. [19])	0.0545	0.0394	0.0336	0.0320	0.0317	0.0316	0.1116	0.0921	0.0828	0.0793	0.0781	0.0777
cGAN (Adler & Öktem [8])	0.0285	0.0255	0.0273	0.0298	0.0316	0.0327	0.0972	0.0857	0.0878	0.0930	0.0967	0.0990
cGAN (Ohayon et al. [23])	0.0245	0.0247	0.0248	0.0249	0.0249	0.0249	0.0767	0.0790	0.0801	0.0807	0.0810	0.0811
cGAN (Ours)	0.0175	0.0164	0.0188	0.0216	0.0235	0.0245	0.0546	0.0563	0.0667	0.0755	0.0809	0.0837
methods using all 2376 test images, and again using all 14576 training and test images. Table 1 shows that our approach gave significantly better CFID and FID than the competitors. Also, the APSD of Ohayon et al.'s cGAN was an order-of-magnitude smaller than the others, indicating mode collapse. The cGANs generated samples 3800 times faster than the Langevin approach from [19].
Tables 2 and 3 show PSNR, SSIM, LPIPS [42], and DISTS [43] for the  $P$ -sample average  $\widehat{\pmb{x}}_{(P)}$  at  $P \in \{1, 2, 4, 8, 16, 32\}$  and  $R \in \{4, 8\}$ , respectively. While the E2E-VarNet achieves the best PSNR at  $R \in \{4, 8\}$  and the best SSIM at  $R = 4$ , the proposed cGAN achieves the best LPIPS and DISTS performances at  $R \in \{4, 8\}$  when  $P = 2$  and the best SSIM at  $R = 8$  when  $P = 8$ . The  $P$  dependence can be explained by the perception-distortion tradeoff [2]: as  $P$  increases,  $\widehat{\pmb{x}}_{(P)}$  transitions from better perceptual quality to lower  $\ell_2$  distortion. PSNR favors  $P \to \infty$  (e.g.,  $\ell_2$  optimality) while the other metrics favor particular combinations of perceptual quality and distortion. The plots in Appendices K.1 and K.2 show zoomed-in versions of  $\widehat{\pmb{x}}_{(P)}$  that visually demonstrate the perception-distortion tradeoff at  $P \in \{1, 2, 4, 32\}$ : smaller  $P$  yield sharper images with more variability from the ground truth, while larger  $P$  yield smoother reconstructions.
Figure 3 shows zoomed versions of two posterior samples  $\widehat{\pmb{x}}_i$ , as well as  $\widehat{\pmb{x}}_{(P)}$ , at  $P = 32$  and  $R = 8$ . The posterior samples show meaningful variations for the proposed method, essentially no variation for Ohayon et al.'s cGAN, and vertical or horizontal reconstruction artifacts for Adler & Oktem's cGAN and the Langevin method, respectively. The  $\widehat{\pmb{x}}_{(P)}$  plots show that these artifacts are mostly suppressed by sample averaging with large  $P$ .
Figure 4 shows examples of  $\widehat{\pmb{x}}_{(P)}$ , along with the corresponding pixel-wise absolute errors  $|\widehat{\pmb{x}}_{(P)} - \pmb{x}|$  and pixel-wise SD images  $(\frac{1}{P}\sum_{i = 1}^{P}(\widehat{\pmb{x}}_{(P)} - \widehat{\pmb{x}}_i)^2)^{1 / 2}$ , for  $P = 32$  and  $R = 8$ . The absolute-error image for the Langevin technique looks more diffuse than those of the other methods in the brain region. The fact that it is brighter in the air region (i.e., near the edges) is a consequence of minor differences in sensitivity-map estimation. The pixel-wise SD images show a lack of variability for the E2E-VarNet, which does not generate posterior samples, as well as Ohayon et al.'s cGAN, due to mode collapse. The Langevin pixel-wise SD images show localized hot-spots that appear to be reconstruction artifacts.
Appendices K.1 and K.2 show other example MRI recoveries with zoomed pixel-wise SD images at  $R = 4$  and  $R = 8$ , respectively. Notably, Figures K.10 and K.11 show strong hallucinations for Langevin recovery at  $R = 8$ , as highlighted by the red arrows.
Table 3: Average PSNR, SSIM, LPIPS, and DISTS of  $\hat{\pmb{x}}_{(P)}$  versus  $P$  for  $R = 8$  MRI
Model	PSNR↑	SSIM↑
P=1	P=2	P=4	P=8	P=16	P=32	P=1	P=2	P=4	P=8	P=16	P=32
E2E-VarNet (Sriram et al. [37])	36.49	-	-	-	-	-	0.9220	-	-	-	-	-
Langevin (Jalal et al. [19])	32.17	32.83	33.45	33.74	33.83	33.90	0.8725	0.8919	0.9031	0.9091	0.9120	0.9137
cGAN (Adler & Öktem [8])	31.31	32.31	32.92	33.26	33.42	33.51	0.8865	0.9045	0.9103	0.9111	0.9102	0.9095
cGAN (Ohayon et al. [23])	34.89	34.90	34.90	34.90	34.91	34.92	0.9222	0.9217	0.9213	0.9211	0.9211	0.9210
cGAN (Ours)	32.32	33.67	34.53	35.01	35.27	35.42	0.9030	0.9199	0.9252	0.9257	0.9251	0.9246
LPIPS↓	DISTS↓
Model	P=1	P=2	P=4	P=8	P=16	P=32	P=1	P=2	P=4	P=8	P=16	P=32
E2E-VarNet (Sriram et al. [37])	0.0575	-	-	-	-	-	0.1253	-	-	-	-	-
Langevin (Jalal et al. [19])	0.0769	0.0619	0.0579	0.0589	0.0611	0.0611	0.1341	0.1136	0.1086	0.1119	0.1175	0.1212
cGAN (Adler & Öktem [8])	0.0698	0.0614	0.0623	0.0667	0.0704	0.0727	0.1407	0.1262	0.1252	0.1291	0.1334	0.1361
cGAN (Ohayon et al. [23])	0.0532	0.0536	0.0539	0.0540	0.0534	0.0540	0.1128	0.1143	0.1151	0.1155	0.1157	0.1158
cGAN (Ours)	0.0418	0.0379	0.0421	0.0476	0.0516	0.0539	0.0906	0.0877	0.0965	0.1063	0.1135	0.1177
Figure 3: Example  $R = 8$  MRI reconstructions. Arrows show meaningful variations across samples.
4.3 Inpainting experiments
In this section, our goal is to complete a large missing square in a face image.
Data. We used  $256 \times 256$  CelebA-HQ face images [29] and a centered  $128 \times 128$  missing square. We randomly split the dataset, yielding 27000 training, 2000 validation, and 1000 testing images.
Architecture. For our cGAN, we use the CoModGAN generator and discriminator from [9] with our proposed  $\mathcal{L}_{1,\mathrm{SD},P_{\mathrm{train}}}$  regularization. Unlike [9], we do not use MBSD [29] at the discriminator.
Training/validation/testing. We use the same general training and testing procedure described in Section 4.2, but with  $\beta_{\mathrm{adv}} = 5\mathrm{e} - 3$ ,  $n_{\mathrm{batch}} = 100$ , and 110 epochs of cGAN training. Running PyTorch on a server with 4 Tesla A100 GPUs, each with 82 GB of memory, the cGAN training takes approximately 2 days. FID was evaluated on 1000 test images using  $P = 32$  samples per measurement. To avoid the bias that would result from evaluating CFID on only 1000 images (see Appendix J.1), CFID was evaluated on all 30000 images with  $P = 1$ .
Competitors. We compare with the state-of-the-art CoModGAN [9] and Score-based SDE [20] approaches. For CoModGAN, we use the PyTorch implementation from [44]. CoModGAN differs from our cGAN only in its use of MBSD and lack of  $\mathcal{L}_{1,\mathrm{SD},P_{\mathrm{train}}}$  regularization. For Song et al.'s SDE, we use the authors' implementation from [45] with their pretrained weights and the settings they suggested for the  $256\times 256$  CelebA-HQ dataset.
Figure 4: Example  $R = 8$  MRI reconstructions with  $P = 32$ . Row one:  $P$ -sample average  $\widehat{\pmb{x}}_{(P)}$ . Row two: pixel-wise absolute error  $|\widehat{\pmb{x}}_{(P)} - \pmb{x}|$ . Row three: pixel-wise SD  $\left(\frac{1}{P}\sum_{i=1}^{P}(\widehat{\pmb{x}}_i - \widehat{\pmb{x}}_{(P)})^2\right)^{1/2}$ .
Table 4: Average results for inpainting: FID was computed from 1000 test images with  $P = 32$ , while CFID was computed from all 30000 images with  $P = 1$
Model	CFID↓	FID↓	Time (128)↓
Score-SDE (Song et al. [20])	5.11	7.92	48 min
CoModGAN (Zhao et al. [9])	5.29	8.50	217 ms
cGAN (ours)	4.69	7.45	217 ms
Results. Table 4 shows test CFID, FID, and 128-sample generation time. The table shows that our approach gave the best CFID and FID, and that the cGANs generated samples 13000 times faster than the score-based method. Figure 5 shows an example of five generated samples for the three methods under test. The samples are all quite good, although a few generated by CoModGAN and the score-based technique have minor artifacts. Some samples generated by our technique show almond-shaped eyes, demonstrating fairness. Additional examples are given in Appendix K.3.
We propose a novel regularization framework for image-recovery cGANs that consists of supervised- $\ell_1$  loss plus an appropriately weighted standard-deviation reward. For the case of an independent Gaussian posterior, we proved that our regularization yields generated samples that agree with the true-posterior samples in both mean and covariance. We also established limitations for alternatives based on supervised- $\ell_2$  regularization with or without a variance reward. For practical datasets, we proposed a method to auto-tune our standard-deviation reward weight.
Experiments on parallel MRI and large-scale face inpainting showed our proposed method outperforming all cGAN and score-based competitors in CFID, which measures posterior-approximation quality, as well as other metrics like FID, PSNR, SSIM, LPIPS, and DISTS. Furthermore, it generates samples thousands of times faster than Langevin/score-based approaches.
In ongoing work, we are extending our approach so that it can be trained to handle a wide range of recovery tasks, such as MRI with a generic acceleration and sampling mask [46], or inpainting with a generic mask. We are also extending our approach to other applications, such as super-resolution, deblurring, compressive sensing, denoising, and phase retrieval.
Figure 5: Example of inpainting a  $128 \times 128$  square on a  $256 \times 256$  resolution CelebA-HQ image.
Limitations. We acknowledge several limitations of our work. First, while our current work focuses on how to build a fast and accurate posterior sampler, it's not yet clear how to best exploit the resulting posterior samples in each given application. For example, in MRI, where the posterior distribution has the potential to help assess uncertainty in image recovery, it's still not quite clear how to best convey uncertainty information to radiologists (e.g., they may not gain much from pixel-wise SD images). More work is needed on this front. Second, we acknowledge that, because radiologists are risk-averse, more studies are needed before they will feel comfortable incorporating generative deep-learning-based methods into the clinical workflow. Third, we acknowledge that the visual quality of our  $R = 8$  MRI reconstructions falls below clinical standards. Fourth, some caution is needed when interpreting our CFID, FID, and DISTS perceptual metrics because the VGG-16 backbone used to compute them was trained on ImageNet data. Although there is some evidence that the resulting DISTS metric correlates well with radiologists' perceptions [41], there is also evidence that ImageNet-trained features may discard information that is diagnostically relevant in medical imaging [47]. Thus our results need to be validated with a pathology-centric radiologist study before they can be considered relevant to clinical practice.
The authors are funded in part by the National Institutes of Health under grant R01-EB029957 and the National Science Foundation under grant CCF-1955587.
[1] E. L. Lehmann and G. Casella, Theory of point estimation. Springer Science & Business Media, 2006. 1
[2] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp. 6228–6237, 2018. 1, 7
[3] C. K. Sønderby, J. Caballero, L. Theis, W. Shi, and F. Huszár, “Amortised MAP inference for image super-resolution,” in Proc. Int. Conf. on Learn. Rep., 2017. 1, 19
[4] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” ACM Comput. Surveys, vol. 54, no. 6, pp. 1–35, 2021. 2
[5] T. Sanchez, I. Krawczuk, Z. Sun, and V. Cevher, "Uncertainty-driven adaptive sampling via GANs," in Proc. Neural Inf. Process. Syst. Workshop, 2020. 2
[6] C.-H. Chang, E. Creager, A. Goldenberg, and D. Duvenaud, "Explaining image classifiers by counterfactual generation," in Proc. Int. Conf. on Learn. Rep., 2019. 2
[7] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in Proc. IEEE Conf. Comp. Vision Pattern Recog., pp. 1125-1134, 2017. 2, 4
[8] J. Adler and O. Oktem, "Deep Bayesian inversion," arXiv:1811.05910, 2018. 2, 3, 6, 7, 8, 19, 20, 21