SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $G_{\phi}$  :  $z \in \mathbb{R}^{128}(\sim \nu_0: \mathcal{N}(0,1)) \to 4 \times 4 \times 256(\text{Dense}, \text{Linear}) \to \text{ResBlock up } 256 \to \text{ResBlock up } 256 \to \text{ResBlock up } 256 \to \text{BN}, \text{ReLU}, \to 3 \times 3 \text{ conv}, 3 \text{Tanh}$ .
-  $F_{\gamma_1} \colon x \in [-1, 1]^{32 \times 32 \times 3} \to \text{ResBlock down } 128 \to \text{ResBlock down } 128 \to \text{ResBlock down } 128 \to \text{ResBlock } 128$ .
Table 5: Summary of FID and IS scores of methods on CIFAR10 (32x32), and CelebA (64x64)
Method	CIFAR10 (32x32)	CelebA (64x64)
FID (↓)	IS (↑)	FID (↓)
iMSW (L=100,T=10,M=0,N=1)	14.61±0.72	8.15±0.15	9.73±0.33
iMSW (L=100,T=10,M=9,N=1)	14.16±1.11	8.17±0.07	9.10±0.34
iMSW (L=100,T=10,M=5,N=1)	13.93±0.21	8.15±0.05	9.49±0.52
iMSW (L=100,T=10,M=0,N=2)	14.33±0.32	8.15±0.06	8.99±0.64
iMSW (L=10,T=100,M=0,N=1)	14.26±0.74	8.15±0.07	8.89±0.23
iMSW (L=10,T=100,M=99,N=1)	14.50±0.70	8.12±0.08	9.55±0.35
iMSW (L=10,T=100,M=50,N=1)	14.41±0.58	8.12±0.06	9.46±0.73
iMSW (L=10,T=100,M=0,N=2)	14.65±0.01	8.11±0.06	9.49±0.39
-  $F_{\gamma_2}$ :  $\mathbf{x} \in \mathbb{R}^{128 \times 8 \times 8} \rightarrow \mathrm{ReLU} \rightarrow \mathrm{Global~sum~pooling}(128) \rightarrow 1$  (Spectral normalization).
-  $F_{\gamma}(x) = (F_{\gamma_1}(x), F_{\gamma_2}(F_{\gamma_1}(x)))$  and  $H(F_{\gamma}(x)) = F_{\gamma_2}(F_{\gamma_1}(x))$ .
CelebA.
-  $G_{\phi}$  :  $z \in \mathbb{R}^{128}(\sim \nu_0: \mathcal{N}(0,1)) \to 4 \times 4 \times 256$  (Dense, Linear) → ResBlock up 256 → ResBlock up 256 → ResBlock up 256 → BN, ReLU, → 3 × 3 conv, 3 Tanh.
-  $F_{\gamma_1}\colon \pmb {x}\in [-1,1]^{32\times 32\times 3}\to \mathrm{ResBlock~down~}128\to \mathrm{ResBlock~down~}128\to$  ResBlock down  $128\rightarrow$  ResBlock  $128\rightarrow$  ResBlock 128.
-  $F_{\gamma_2}$ :  $\mathbf{x} \in \mathbb{R}^{128 \times 8 \times 8} \rightarrow \mathrm{ReLU} \rightarrow \mathrm{Global~sum~pooling}(128) \rightarrow$  1(Spectral normalization).
-  $F_{\gamma}(x) = (F_{\gamma_1}(x), F_{\gamma_2}(F_{\gamma_1}(x)))$  and  $H(F_{\gamma}(x)) = F_{\gamma_2}(F_{\gamma_1}(x))$ .
For all datasets, the number of training iterations is set to 50000. We update the generator  $G_{\phi}$  each 5 iterations while we update the feature function  $F_{\gamma}$  every iteration. The mini-batch size  $m$  is set 128 in all datasets. The learning rate for  $G_{\phi}$  and  $F_{\gamma}$  is 0.0002 and the optimizer is Adam [25] with parameters  $(\beta_{1}, \beta_{2}) = (0, 0.9)$ . We use the order  $p = 2$  for all sliced Wasserstein variants. We use 50000 random samples from estimated generative models  $G_{\phi}$  for computing the FID scores and the Inception scores. In evaluating FID scores, we use all training samples for computing statistics of  $\mathbf{d}\mathbf{s}\mathbf{t}\mathbf{s}^2$ .
Generated images. We show generated images on CIFAR10 and CelebA from different generative models trained by different distances in Figure 7 and in Figure 8 in turn. Overall, images are visually consistent with the quantitative FID scores in Table 2.
Studies on hyperparameters. We run some additional settings of iMSW to investigate the performance of the burning thinning technique and to compare the role of  $L$  and  $T$  in Table 5. First, we see that burning and thinning helps to improve FID score and IS score on CIFAR10 and CelebA in the settings of  $L = 100, T = 10$ . It is worth noting that the original purpose of burning and thinning is to reduce computational complexity and memory complexity. The side benefit of improving performance requires more investigation that is left for future work. In addition, we find that for the same number of total projections 1000 without burning and thinning, the setting of  $L = 10, T = 100$  is better than the setting of  $L = 100, T = 10$  on CIFAR10. However, the reverse direction happens on CelebA. Therefore, on different datasets, it might require hyperparameter tuning for finding the best setting of the number of projections  $L$  and the number of timesteps  $T$ .
2We evaluate the scores based on the code from https://github.com/GongXinyuu/sngan.pytorch.
viMSW
Figure 8: Random generated images of distances on CelebA.