Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
Authors: name: Tariq Berrada; name: Jakob Verbeek; name: Camille Couprie; name: Karteek Alahari
Published: 2023-12-20 09:39:19

Abstract: Semantic image synthesis, i.e., generating images from user-provided semantic
label maps, is an important conditional image generation task as it allows to
control both the content as well as the spatial layout of generated images.
Although diffusion models have pushed the state of the art in generative image
modeling, the iterative nature of their inference process makes them
computationally demanding. Other approaches such as GANs are more efficient as
they only need a single feed-forward pass for generation, but the image quality
tends to suffer on large and diverse datasets. In this work, we propose a new
class of GAN discriminators for semantic image synthesis that generates highly
realistic images by exploiting feature backbone networks pre-trained for tasks
such as image classification. We also introduce a new generator architecture
with better context modeling and using cross-attention to inject noise into
latent variables, leading to more diverse generated images. Our model, which we
dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
surpassing recent diffusion models while requiring two orders of magnitude less
compute for inference.