Semantica: An Adaptable Image-Conditioned Diffusion Model
Authors: name: Manoj Kumar; name: Neil Houlsby; name: Emiel Hoogeboom
Published: 2024-05-23 17:58:03

Abstract: We investigate the task of adapting image generative models to different
datasets without finetuneing. To this end, we introduce Semantica, an
image-conditioned diffusion model capable of generating images based on the
semantics of a conditioning image. Semantica is trained exclusively on
web-scale image pairs, that is it receives a random image from a webpage as
conditional input and models another random image from the same webpage. Our
experiments highlight the expressivity of pretrained image encoders and
necessity of semantic-based data filtering in achieving high-quality image
generation. Once trained, it can adaptively generate new images from a dataset
by simply using images from that dataset as input. We study the transfer
properties of Semantica on ImageNet, LSUN Churches, LSUN Bedroom and SUN397.