Nested Diffusion Models using Hierarchical Latent Priors

Authors: Xiao Zhang, Ruoxi Jiang, Rebecca Willett, Michael Maire
Primary area: generative models
Keywords: Hierarchical Generative Models, Diffusion Models.

Abstract:
We introduce nested diffusion models, an efficient and powerful hierarchical generative framework that substantially enhances the generation quality of diffusion models, particularly for images of complex scenes.  Our approach employs a series of diffusion models to progressively generate latent variables at different semantic levels.  Each model in this series is conditioned on the output of the preceding higher-level model, culminating in image generation.  Hierarchical latent variables guide the generation process along predefined semantic pathways, allowing our approach to capture intricate structural details while significantly improving image quality. 
To construct these latent variables, we leverage a pre-trained visual encoder, which learns strong semantic visual representations, and apply a series of compression techniques, including spatial pooling, channel reduction, and noise injection, in order to control the information capacity at each level of the hierarchy.  Across multiple benchmarks, including class-conditioned generation on ImageNet-1k and text-conditioned generation on the COCO dataset, our system demonstrates notable improvements in image quality, as reflected by FID scores.  These improvements incur only slight additional computational cost, as more abstract levels of our hierarchy operate on lower-dimensional representations.  Our method also enhances unconditional generation, narrowing the performance gap between conditional generation and unconditional generation that leverages neither text nor class labels.

Decision: reject

Reviews and discussion:

Metareview: