SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Identifying the effectiveness of generative models in high-dimensional data is tricky, and we cannot directly visualize the mode coverage of the generated data. Also, popular metrics like FID and Inception Score can only be referred as quality evaluation. Thus, we test our models and the baseline DDGANs on generating a Mixture of Gaussians. To generate the data, we sample 25 groups of data from the Gaus
Model / steps	1	2	4	8	16
vanilla GANs	12.04	-	-	-	-
DDGAN [11]	-	7.27	0.99	0.49	0.53
SIDDMs(ours)	-	0.14	1.21	0.30	0.23
SIDDMs w/o AFD (ours)	-	21.19	53.22	7.04	14.37
Table 1: MOG 5x5 results, FID↓
sians independently and each distribution has a different mean but with the same variance. In Figure 3, we show the synthesized results of different models and the denoising steps and the quantitative results are shown in Table 1. We can observe that for the extremely few denoising steps of 2 and many as 16 steps, DDGANs fail to converge to real data distribution, and our proposed method can recover well on the different steps. Also, to identify if the adversarial term takes the main job in the generation, we remove the auxiliary forward term and find that our model cannot recover the original distribution, proving the proposed decomposition's effectiveness. To evaluate our model's effectiveness on real-world generation tasks, we test our models on the tiny image dataset CIFAR10, the fine-grained generation task on the Celeb-HQ, and the large-scale challenging dataset Imagennet. We pick a small amount of generated images from our model and
(a) Generated samples on CIFAR10.
(b) Generated samples on CelebHQ.
(c) Generated samples on ImageNet.
Figure 2: We pick a subset of images generated from our model, which produces the paper results. From the top left to the bottom, they are the generated results on CIFAR10, Celeb-HQ and ImageNet.
show them in the Figure 2. We also quantitatively evaluate the quality of the generated samples and collect all of the results in Table 2,3 and 4. Visually, our model generates samples with high fidelity and rich diversity. For the evaluation metric, we choose the Fréchet inception distance (FID) [47] and Inception Score [48] for sample fidelity and the improved recall score [49] to measure the sample diversity. Compared with the GANs-based method, our method achieves better sample quality and diversity, identifying the benefit of enhancing GANs with the diffusion formulation. Compared with the baseline DDGANs, our model shows superior quantitative results than the baseline DDGANs. Although our method does not reach the same quality as the diffusion-based (or score-based) models
Figure 3: We show the generative results for the  $5 \times 5$  Mixture of Gaussians, which can straightforwardly show the effectiveness of our proposed method. We include our full model, our model without the auxiliary forward diffusion term and the baseline DDGANs. Our model can recover the original distribution even in the small diffusion steps of 2, but the DDGANs fail. Also, without the completed formulation, our model creates biased results.
Model	IS↑	FID↓	Recall↑	NFE ↓	Time (s) ↓
SIDDMs (ours), T=4	9.85	2.24	0.61	4	0.20
DDGANs ([11]), T=4	9.63	3.75	0.57	4	0.20
Diffusion models
DDPM [1]	9.46	3.21	0.57	1000	80.5
EDM [21]	9.84	2.04	-	36	-
Score SDE (VE) [16]	9.89	2.20	0.59	2000	423.2
Score SDE (VP) [16]	9.68	2.41	0.59	2000	421.5
Improved DDPM [37]	-	2.90	-	4000	-
UDM [20]	10.1	2.33	-	2000	-
Distillation of Diffusion Models
DDPM Distillation [38]	3.00	-	-	4	-
CD [22]	2.93	9.75	-	2	-
GANs-based models
StyleGAN2 w/o ADA [31]	9.18	8.32	0.41	1	0.04
StyleGAN2 w/ ADA [31]	9.83	2.92	0.49	1	0.04
StyleGAN2 w/ Diffauq [30]	9.40	5.79	0.42	1	0.04
Table 2: Generative results on CIFAR10
Model	FID↓
SIDDMs (ours)	7.37
DDGANs ([11])	7.64
Score SDE [16]	7.23
LSGM [39]	7.22
UDM [20]	7.16
NVAE [40]	29.7
VAEBM [41]	20.4
NCP-VAE [42]	24.8
PGGAN [43]	8.03
Adv. LAE [44]	19.2
VQ-GAN [45]	10.2
DC-AE [46]	15.8
with a small gap, our model still has the large advantage of fast sampling. To be notified, we apply four denoising steps for the CIFAR10 and two steps for the CelebA-HQ generation, identical to the DDGANs' main results.
5.2 Generation on Real Data
For the ImageNet, we choose four denoising steps to get the best sample quality on the ImageNet. ImageNet has 1000 categories, each containing thousands of images and distinguishing itself as a large-scale dataset with diversity. To train a generative model with high fidelity on the dataset, we usually require other inductive biases such as regularizers or model scaling-up methods [13, 50]. The DDGANs fail to retain the high-fidelity samples on this dataset without additional inductive bias. However, with the same model capacity, our proposed method can achieve comparable results w.r.t. the diffusion-based models. This result shows that our model can handle large-scale generation and potentially be applied to boost the sampling speed of large-scale text2image models.
Table 3: Generative results on CelebA-HQ-256
Model	FID↓
SIDDMs (ours)	3.13
DDGANs ([11])	20.63
CT [22]	11.10
ADM [2]	2.07
EDM [21]	2.44
Improved DDPM [37]	2.90
CD ([22])	4.07
Table 4: Generative results on ImageNet1000
Table 5: Study the effect of Auxiliary Forward Diffusion (AFD), FID. The weight of 0.0 means that we remove the AFD term and only train the model with adversarial loss. The weight of  $\infty$  means we only keep the AFD term without adversarial loss. This table also evaluates our model performance with different denoising steps and without our regularizer for the discriminator. All of the scores are evaluated on the CIFAR10.
Weight of AFD	0.0	0.1	0.5	1.0	5.0	∞
SIDDMs(ours)	77.15	3.32	2.63	2.24	2.55	41.27
SIDDMs(ours) w/o Regularizer of Discriminator	51.67	4.64	4.79	3.20	2.95	51.55
Diffusion Steps	1	2	4	8	16	32
SIDDMs(ours)	27.39	3.17	2.24	2.31	2.15	2.46
(a) AFD weight 0.0.
(b) AFD weight 1.0.
Figure 4: Generated samples on CIFAR10, we show two severe cases where only adversarial term (weight 0.0) and only AFD term (weight  $\infty$ ). We also show our full model results in the middle for comparison.
(c) AFD weight  $\infty$
To identify the function of the components in our formulation, we conduct controlledexperiments on the effects of the adversarial and the AFD term. We set the weights of AFD to be  $[0.0, 0.1, 0.5, 1.0, 5.0, \infty]$ , where 0.0 represents only the adversarial term and  $\infty$  denotes only the AFD term in our training. We apply the same sampling strategy as our full models and adopt four denoising steps. The FID scores are reported in Table 5, and the generated samples are shown in Figure 4. We can see that if missing any of these two terms in our formulation, we cannot recover the original image distribution. In addition, we found that our model is not sensitive to the weights between these two components w.r.t. the FID scores when both terms participate in the training, which further identifies the effectiveness of our formulation. Also, we train the model without the regularizer for the discriminator, and we also identify that the proposed auxiliary tasks for the discriminator further enhance the model performance for our full formulation.
In conclusion, the quest for fast sampling with diverse and high-quality samples in generative models continues to pose a significant challenge. Existing models, such as Denoising Diffusion Probabilistic Models (DDPM), encounter limitations due to the inherent slowness associated with their iterative steps. On the other hand, Denoising Diffusion Generative Adversarial Networks (DDGAN) faces scalability issues when dealing with large-scale datasets. To address these challenges, we propose a novel approach that effectively addresses the limitations of previous models by leveraging a combination of implicit and explicit factors. Specifically, we introduce an implicit model that enables us to match the marginal distribution of random variables in the reverse diffusion process. Additionally, we model explicit distributions between pairs of variables in reverse steps, which allows us to effectively utilize the Kullback-Leibler (KL) divergence for the reverse distribution. To estimate the negative entropy component, we incorporate a min-max game into our framework. Moreover, we adopt the L2 reconstruction loss to accurately represent the cross-entropy term in the KL divergence. Unlike DDPM but similar to DDGAN, we do not impose a parametric distribution for the reverse step in our approach. This design choice empowers us to take larger steps during the inference process, contributing to enhanced speed and efficiency. Additionally, similar to DDPM, we effectively leverage the exact form of the diffusion process to further improve our model's performance. Our proposed approach exhibits comparable generative performance to DDPM while surpassing models with fewer sampling steps.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. cite arxiv:1505.04597Comment: conditionally accepted at MICCAI 2015.
[37] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
[38] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
[39] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in neural information processing systems, 2021.
[40] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in neural information processing systems, 2020.
[41] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021.
[42] Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems, 2021.
[43] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
[44] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020.
[45] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
[46] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradictinistive generative autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
[47] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 2017.
[48] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 2016.
[49] Tuomas Kynkänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint arXiv:1904.06991, 2019.
[50] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. volume abs/2201.00273, 2022.