SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that here  $\rho$  denotes a scalar hyperparameter following the EDM notation, and not the generated distribution as in the main paper.
Usual generative modeling tricks. For simplicity in our proof-of-concept experiments, we avoided using standard performance improvement tricks such as exponential moving average and truncation tricks (Brock et al., 2019), EDM sampling tricks (Karras et al., 2022), or spectral normalization (Miyato et al., 2018).
For all experiments we use one or two Nvidia V100 GPUs with CUDA 11.8. When using two V100 Nvidia GPUs, the training time of both Discriminator Flow and Score GAN models is at most one day for the largest dataset (CelebA).
As a means to evaluate each model's efficiency, we provide the approximate amount of time we used to train the tested models over MNIST on one NVIDIA V100 GPU:
for Discriminator Flow, 24 hours;
- for Score GAN, 10 hours (excluding the pretrained diffusion model);
for EDM, 6 hours;
- for GAN, 1 hour.
Table 4: Chosen hyperparameters for Discriminator Flows for each dataset. GP stands for Gradient Penalty (Gulrajani et al., 2017). IPM stands for Integral Probability Metric (Müller, 1997). BN stands for Batch Normalization (Ioffe & Szegedy, 2015).
Hyperparameters	Gaussians	MNIST	CelebA
GAN loss	Non-saturating vanilla	IPM	IPM
a (Equation (15))	log(1 - sigmoid)	id	id
b (Equation (15))	- log sigmoid	id	id
c (Equation (15))	- log sigmoid	id	id
R (Equation (15))	0	GP	GP
GP strength	0	0.04	0.05
GP center	—	0	0
π (Equation (1))	N(0, ID)
η (Algorithm 3)	20	[2,1]	2
N (Algorithm 3)	56	[64,128]	25
Network architecture	MLP	DCGAN	ResNet
Width	512	64	64
Activation	LeakyReLU, negative slope of -0.2
Depth	4	—	—
BN	No
Initialization	Normal	Orthogonal	Normal
Initialization gain	0.02	1.41	0.02
Time embedding type	Fourier
Frequency scale	16
Time embedding size	128
Batch size	128	128	64
Number of optimization steps	4000	200000	200000
Optimizer	Adam (Kingma & Ba, 2015)
Learning rate	0.0002
(β1,β2)	(0.5,0.999)
Model selection	—	Best validation FID
Validation frequency	—	1000
Number of validation samples	—	6000	1000
Our work aims to better understand recent generative modeling methods. As such, from a practical point of view, our work shares much of the impact of other work in this domain. While the models we propose do not yet have the quality required for broad application, generative models have a wide range of potential positive and negative impacts. Some of the positive impacts include enabling faster and more accurate natural language processing (Li et al., 2022), improving automated tasks like summarization (Liu et al., 2018), and automating certain aspects of content creation (Nichol et al., 2022). However, generative models are also susceptible to producing undesirable output, such as unethical text, adversarial attacks (Wang et al., 2021), and malicious manipulation of data. These models also have the potential to exacerbate issues such as bias and discrimination in AI systems (Lucy & Bamman, 2021), enabling the creation of more effective fake text and videos, and expanding the scope and complexity of cyberattacks that can be launched by malicious actors (Seymour & Tully, 2018). For a thorough discussion on the potential dangers of deepfakes, see Fallis (2021).
Discussions and debates around the responsible use of generative models and the potential broader impacts of deploying them more widely are still ongoing. We hope that our principled framework aimed at improving our understanding of generative models will contribute to these discussions and to better control of such models.
Table 5: Chosen hyperparameters for EDM for each dataset. Cf. Karras et al. (2022) and our code for more details.
Hyperparameters	Gaussians	MNIST	CelebA
σmin	0.002
σmax	40
σdata	0.5
ρ	7
Equation & Solver	Heun solver on the deterministic ODE of Equation (8)
Number of solver steps	7	32	25
Network architecture	MLP	EDM	EDM
Width	512	16	128
Number of residual blocks	—	1	2
Dropout	—	0.13	0.1
Depth	4	—	—
Activation	Leaky ReLU, slope of -0.2	—	—
Initialization	Uniform Kaiming for convolutions, unit weight and zero bias for group normalization layers, otherwise PyTorch default
Time embedding type	Fourier	Positional	Positional
Frequency scale	16	—	—
Time embedding size	128	256	256
Batch size	128	128	64
Number of optimization steps	10000	500000	100000
Optimizer	Adam (Kingma & Ba, 2015)
Learning rate	0.0002
(β1, β2)	(0.9, 0.999)
Model selection	—
Table 6: Chosen hyperparameters for GANs for each dataset. Hinge refers to Hinge GANs (Lim & Ye, 2017).
Hyperparameters	Gaussians	MNIST	CelebA
GAN loss	Non-saturating vanilla	Hinge
Number of discriminator
steps per generator update	1
Latent space size	128
pz(Definition 2)	N(0,Id)
Discriminator architecture	MLP	DCGAN	FastGAN
Width	512	64	32
Activation	LeakyReLU, negative slope of -0.2
Depth	4	—	—
BN	No	Yes	Yes
Generator architecture	MLP	DCGAN	FastGAN
Width	512	64	32
Activation	LeakyReLU,
slope of -0.2	ReLU	LeakyReLU,
slope of -0.2
Depth	4	—	—
BN	No	Yes	Yes
Initialization	Normal
Initialization gain	0.02
Batch size	128	128	64
Number of optimization steps	1900	10000	100000
Optimizers	Adam (Kingma & Ba, 2015)
Learning rate	0.0002
(β1,β2)	(0.5,0.999)
Model selection	—	Best validation FID
Validation frequency	—	1000
Number of validation samples	—	6000	1280
Table 7: Chosen hyperparameters for Score GANs for each dataset.
Hyperparameters	Gaussians	MNIST	CelebA
σmin(Appendix D.2)	0.1	0.32	0.32
σmax(Appendix D.2)	10	40	40
ρ(Appendix D.2)	3
K(Algorithm 1)	10	1	4
Latent space size	128
pz(Definition 2)	N(0,Id)
Data score spdataψ	EDM from Table 5
Gen. score sp architecture	MLP	EDM	EDM
σdata	0.5
Width	512	64	128
Number of residual blocks	—	2	2
Dropout	—	0.13	0
Depth	4	—	—
Activation	LeakyReLU, slope of -0.2	—	—
Initialization	EDM from Table 5
Generator architecture	MLP	DCGAN	FastGAN
Width	512	64	32
Activation	ReLU
Depth	4	—	—
BN	No	Yes	Yes
Initialization	PyTorch default	Normal	Orthogonal
Initialization gain	—	0.02	1.41
Batch size	128	256	32
Number of generator optimization steps	3150	100000	150000
Optimizers	Adam (Kingma & Ba, 2015)
Score learning rate	0.0002	0.001	0.0004
Generator learning rate (β1,β2)	0.0002
(0.9,0.999)
Model selection	—	Best validation FID	—
Validation frequency	—	2500	—
Number of validation samples	—	6000	—
(a) Initialization (sampling from  $\rho_0 = \pi$ ).
(b)  $t = \frac{1}{7} T$
(c)  $t = \frac{2}{7} T$
(d)  $t = \frac{3}{7} T$
(e)  $t = \frac{4}{7} T$
(f)  $t = \frac{5}{7} T$
(g)  $t = \frac{6}{7} T$
Figure 8: Sampling steps for EDM on a Gaussian mixture; 128 samples are shown for both the generated  $(\bullet)$  and the data  $(\diamond)$  distributions. The second-order solver was used with 13 NFEs. Arrows show the gradients  $\nabla h_{\rho_t}$  associated with each generated sample, corresponding to the direction provided by the score function, cf. Equation (8).
(h)  $t = T$
(d)  $[t = \frac{3}{7} T.$
(g)  $t = \frac{6}{7} T$
Figure 9: Sampling steps for Discriminator Flows on a Gaussian mixture; cf. Figure 8. Our chosen discretization yields 14 NFEs. The colored background represents the pointwise generator loss function  $-h_{\rho_t} = -c \circ f_{\rho_t}$  (darker is lower, renormalized for every snapshot), from which the particle gradients, shown in the figures, are derived.
(a) Initialization  $(\rho_0 = g_{\theta_0}\sharp p_z)$
(g)  $t = \frac{6}{7} T$
Figure 10: Training snapshots of a GAN on a Gaussian mixture; cf. Figure 8. Here time  $t$  represents training time, and  $T$  is the end of training. The colored backgrounds represent the pointwise generator loss function  $-h_{\rho_t} = -c \circ f_{\rho_t}$  (darker is lower, renormalized for every snapshot), from which the particle gradients, shown in the figures, are derived and then fed to the generator following Equations (11) and (13).
(g)  $t = \frac{6}{7} T$
Figure 11: Training snapshots of a Score GAN on a Gaussian mixture; cf. Figure 8. Here time  $t$  represents training time, and  $T$  is the end of training.
(g)  $t = \frac{6}{7} T$
Figure 12: Training snapshots of a Score GAN on a Gaussian mixture, identical to Figure 11, but with generated samples perturbed by Gaussian noise of standard deviation  $\sigma = 0.2$ . Arrows show the gradients  $\nabla h_{\rho_t}$  received by the generated particles at this noise level, corresponding to Equation (19), split into the data score (in blue) and minus the score of the generated distribution (in yellow). They are then fed to the generator following Equations (11) and (13).