Unsupervised Semantic Correspondence Using Stable Diffusion
Authors: name: Eric Hedlin; name: Gopal Sharma; name: Shweta Mahajan; name: Hossam Isack; name: Abhishek Kar; name: Andrea Tagliasacchi; name: Kwang Moo Yi
Published: 2023-05-24 21:34:34

Abstract: Text-to-image diffusion models are now capable of generating images that are
often indistinguishable from real images. To generate such images, these models
must understand the semantics of the objects they are asked to generate. In
this work we show that, without any training, one can leverage this semantic
knowledge within diffusion models to find semantic correspondences -- locations
in multiple images that have the same semantic meaning. Specifically, given an
image, we optimize the prompt embeddings of these models for maximum attention
on the regions of interest. These optimized embeddings capture semantic
information about the location, which can then be transferred to another image.
By doing so we obtain results on par with the strongly supervised state of the
art on the PF-Willow dataset and significantly outperform (20.9% relative for
the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
CUB-200 and SPair-71k datasets.