Unsupervised Semantic Correspondence Using Stable Diffusion
Authors: Eric Hedlin
Gopal Sharma
Shweta Mahajan
Hossam Isack
Abhishek Kar
Andrea Tagliasacchi
Kwang Moo Yi
Venue: NeurIPS 2023 poster (2023)
Keywords: Semantic Correspondence
Stable Diffusion
Optimization-based Inference
TL;DR: We optimize the text embeddings of a pre-trained Stable Diffusion network to estimate semantic correspondences without any supervision.

Abstract: Text-to-image diffusion models are now capable of generating images that are often indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they are asked to generate. In this work we show that, without any training, one can leverage this semantic knowledge within diffusion models to find semantic correspondences – locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention on the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing so we obtain results on par with the strongly supervised state of the art on the PF-Willow dataset and significantly outperform (20.9% relative for the SPair-71k dataset) any existing weakly- or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.

Unsupervised Semantic Correspondence Using Stable Diffusion
Eric Hedlin', Gopal Sharma', Shweta Mahajan Hossam Isack', Abhishek Kar), Andrea Tagliasacchi* 4.5 Kwang Moo Yi University of British Columbia, Vector Institute for AI, Google, Simon Fraser University, University of Toronto
Abstract
Text-to-image diffusion models indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they asked generate. In this work we show that, without any training, one can leverage this semantic knowledge within diffusion models to find semantic correspondences locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing we obtain results on par with the strongly supervised state of the art on the Pilloww dataset and significantly outperform (20.9% relative for the SPair-7 Ik dataset) any existing weakly or unsupervised method on PF-Willow, CUB-200 and SPair-7 lk datasets.
capable of generating images that often
Introduction
Estimating point correspondences between images is a fundamental problem in computer vision, with numerous applications in areas such image registration (1) object recognition (2), and 3D reconstruction [3] Work on correspondences can largely be classified into geometric and semantic correspondence search. Geometric correspondence search aims to find points that correspond the same physical point of the same object and are typically solved with local feature-based methods [4] 5 6 7 S and optical flow methods
g 10]. Semantic correspondence search the core application
of interest in this paper focuses on points that correspond semantically [
13 14]
[5
16), not
necessarily of the same object, but of similar objects of the same related class. For example, given the selected kitten paw in the (source) image of Figure I we would like to automatically identify kitten paws in other (target) images. Whether geometric semantic correspondences, common
image
attention
target image
target attention
target image
target attention
Figure 1: Teaser - We optimize for the prompt embedding of the diffusion model that activates attention the region of the 'paw' for the source image, marked in yellow. With this embedding, the attention highlights semantically similar points in various target images, which we then use to find semantic correspondences. This holds even for "out of distribution" target images (LEGO cats).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) input image
"paw" attention
'collar' attention
'shower' attention
'ears' attention
Figure 2: Semantic knowledge in diffusion models - Given an input image and text prompts describing parts of the image, the attention maps of diffusion models highlight semantically relevant areas of the image. We visualize the attention maps superimposed atop the input image.
recent trend is to learn to solve these problems as in many other areas of computer vision.
While learning- based methods provide superior performance
16
15
17
18 often these methods
need to be trained on large supervised datasets [7 19. For geometric correspondences, this is relatively straightforward, as one can leverage the vast amount of photos that exist on the web and utilize points that pass sophisticated geometric verification processes [[7 20 19] For semantic correspondences. this is more challenging, one cannot simply collect more photos for higher- quality ground truth-automatic verification of semantic correspondences difficult, and human labeling effort is required. Thus, research on unsupervised learned semantic correspondence has been trending [21] 22 (23].
In this work, we show that one may not need any ground-truth semantic correspondences between image pairs at all for finding semantic correspondences, or need to rely on generic pre-trained deep features [24 25 we can instead simply harness the knowledge within powerful text-to-
can generate reats-realistic images from text prompts only, there must be knowledge about semantic correspondences built-in within them. For example, for a diffusion model to successfully create an image of a human face, it must know that human face consists of two eyes, one nose, and mouth, as well as their supposed whercabouts in other words, it must know the semantics of the scene it is generating. Thus, should one be able to extract this knowledge from these models, trained with billions text-image pairs [30 31] one should be able to solve semantic correspondences as a by-product. Note here that these generative models also be thought of as form of unsupervised pre-training, akin to self-supervised pre- pre-training methods 32 25 but with more supervision from the provided text-image relationship.
image models. Our key insight is that, given that recent diffusion models [26] 27
28
29
30
Hence, inspired by the recent success of prompt-to-prome build our method by exploiting the attention maps of latent diffusion models. These maps attend to different portions of the image as the text prompt is changed. For example, given an image of a cat, the text prompt is 'paw', the attention map will highlight the paws, while if the text prompt is 'collar', they will highlight the collar; see Figure 2] Given arbitrary input images, these attention maps should respond to the semantics of the prompt. In other words, if one can identify the 'prompt' corresponding to a particular image location, the diffusion model could be used to identify semantically similar image locations in new, unseen, image. Note that finding actual prompts that correspond words a discrete problem, hence difficult to solve. However, these prompts do not have to correspond to actual words, as long as they produce attention maps that highlight the queried image location. In other words, the analogy above holds when we operate on the continuous embedding space of prompts, representing the core insight over which we build our method.
33 for text-based image editing, we
With this key insight, we propose to find these (localized) embeddings by optimizing them so that the attention map within the diffusion models corresponds points of interest, similarly to how prompts are found for textual inversion 34 35). As illustrated in Figure I given an (source) image and a (query) location that we wish to find the semantic correspondences of, we first optimize randomly initialized text embedding to maximize the cross-attention at the query location while keeping the diffusion model fixed (i.e. stable diffusion (30). We then apply the optimized text embedding to another image (i.e. target) to find the semantically corresponding location the pixel attaining the maximum attention map value within the target image.
Beyond our core technical contribution, we introduce a number of important design choices that deal with problems that would arise from its naive implementation. Specifically, (1) optimize

---

on single image when finding the embeddings, to prevent overfitting we apply random crops; (2) avoid the instability and randomness of textual inversion [6 35] we find multiple embeddings starting from random initialization; (3) we utilize attention maps at different layers of the diffusion network to build a multi-scale notion of semantic matching. These collectively allow our method to be on par with prrongl-ssupevvise state of the art on the PF-Willow dataset [13] and outperform all weakly- and un-supervised baselines in PF-Willows CUB-200 [37]. and SPair-7 Ik datasets [[4] on the SPair-71k Ik dataset we outperform the closest weakly supervised baseline by 20.9% relative.
We emphasize that our method does not require supervised training that is specific semantic point correspondence estimation. Instead, we simply utilize an thf-th diffusion model, without fine-tuning, and are able achieve state-of-the-arr results. To summarize, we make the following contributions:
we show how to effectively extract semantic correspondences from an off-the-shelf Stable Dif- fusion [30] model without training any new task-specific neural network, or using any semantic correspondence labels;
we introduce set of design choices random crops, multiple embeddings, multi-scale attention
that are critical to achieving state-of-the-ar performance;
we significantly outperform prior state of the art based weakly supervised techniques on the SPair-71k [14). PF-Willow [13), and CUB-200 (37] datasets (20.9% relative on SPair-7Ik) and is on par with the strongly-superviied state of the art the PF-Willow [13 dataset.
2
Related work
We first review work on semantic correspondences and then discuss work focusing reducing supervision. We also discuss work on utilizing pre-trained diffusion models.
Semantic correspondences. Finding correspondences is long-standing core problem in computer vision, serving as a building block in various tasks, for example, optical flow estimation 38 10], structure from motion 33 40
5] and semantic flow [II. While a large body of work exists, including those that focus more finding geometric correspondences that rely on local features [4] 7 and matching them [1
43 or directly via deep networks [44 45 17 18), here we focus
only semantic correspondences in images that are of the same "semantic" e.g., paws of the cat in Figure T For wider review of this topic, we refer the reader to [47
15 46), that is, the task of finding corresponding locations
Finding semantic correspondences has been of interest lately, as they allow class-specific alignment of data [46 48), which can then be used to, for example, train generative models [49], or transfer content between images [T (5, [5]] [ To achieve semantic correspondence, as in many other fields in computer vision, the offte-of-the-arr is to train neural networks [16 [5]. Much research focus was aimed initially at architectural designs that explicitly allow correspondences to be discovered within learned feature maps [10, 45 52 [6]. For example using pyramid architectures [52]. or
architectures [16] that utilize both convolutional neural networks [53] and transformers [54]. However,
these approaches require large-scale datasets that are either sparsely [14 or densely (55) labeled,
limiting the generalization and scalability of these methods without additional human labeling effort.
Learning semantic correspondences with less supervision. Thus, reducing the strong supervision requirement has been of research focus. One direction in achieving less supervision is through the use of carefully designed frameworks and loss formulations. For example, Li et al. (2 use a probabilistic student-teachh framework to distill knowledge from synthetic data and apply it to unlabeled real
image pairs. Kim et al. 22 form a semi-supervised framework that uses unsupervised losses formed via augmentation.
Another direction is to view semantic correspondence problem is utilizing pre-trained deep features. For example, Neural Best-Buddies (56) look for mutual nearest neighbor neuron pairs of a pre-trained CNN to discover semantic correspondences. Amir et al. [57] investigate the use of deep features from pre-trained Vision Transformers (ViT) (58), specifically DINO-ViT ViT (25). More recently, in ASIC (23 rough correspondences from these pre-trate pre-trained networks have been utilized to train a network that maps images into canonical grid, which can then be used to extract semantic correspondences.
These methods either rely heavily on the generalization capacity of pre-trained neural network
representations [56]5] (5) require training new semantic correspondence network 21 22 23 In
3

---

this work, we show how one can achieve dramatically improved results over the current state-of the- art, achieving performance similar that of strongly-supervia methods, even without training any new neural network by simply using a Stable Diffusion network.
Utilizing pre-trained diffusion models.. Diffusion models have recently emerged as a powerful class of generative models, attracting significant attention due to their ability to generate high- quality samples [30, (6 27
28 29 59 60]. Diffusion models generate high-quality samples, with
text-conditioned versions incorporating textual information via cross-attention mechanisms.
is a popular model of choice thanks to its lightweight and open- source nature. While training a latent diffusion model is difficult and resource-consuming, various methods have been suggested to extend its capabilities. These include model personalization [61] through techniques such as Low Rank Adaptation (LoRA) 62] and textual inversion 341, including new conditioning signals via ControlNet [63], or repurposing the model for completely new purposes such as text-to-3D (64] and text-driven image editing [33] While the applications vary, many of
Among them, Stable Diffusion [30]
these applications are for generative tasks, that is, creating images and 3D shapes.
Of particular interest us is the finding from Prompt-to- toorrpp 333 that the cross-attention maps within diffusion models can effectively act as a segummnttttmtnttioonnnn for given text query -in in other words, it contains semantic information. This is further supported by the fact that intermediate features of diffusion models can be used for semantic segmentation [65]. In this work, with these observations, and with the help of textual inversion (34), we show that we can utilize Stable Diffusion (30) for not just generative tasks, but for the task of finding semantic correspondences, completely different task from what these models are trained for, all without any training by repurposing the attention maps, similarly as in [66] but for an entirely different application and framework.
Concurrently to our work, work utilizing feature representations within Stable Diffusion for corre- spondences has been proposed [67 68 69]. These methods look into how to use the deep features within the Stable Diffusion Network effectively, similar to how VGG19 features (70] widely used for various tasks. Our method, on the other hand, looks into how we can alter the attention maps within Stable Diffusion our will, in other words taking into account how these features are supposed to be used within Stable Diffusion we optimize word embeddings, By doing so we show that one can perform alternative tasks than simple image creation such as the semantic correspondence task we demonstrated. However, this is not the end of what our framework can do. For example, a recent followup to our preprint demonstrated an extension of our method segmentation 77
3 Method
Our method identifies semantic correspondences between pairs of images by leveraging the attention maps induced by latent diffusion models. Given pair of images (respectively source and target), and a query point in the source image domain, we seek to compute an attention map that highlights areas in the target image that are in semantic correspondence with the query. While classical diffusion models work on images directly, we employ latent diffusion models [30]. which operate on encoded images. Further, rather than being interested the generated image, we use the attention maps generated as a by-product of the synthesis process. Our process involves two stages; see Figure 3 In the first stage (optimization), we seek to find an embedding that represents the semanties the query region in the source image by investigating the activation map of the denoising step at time t. In the second stage (inference), the embeddings from the source image are kept fixed, and attention maps for a given target image again at time are computed. The location attaining the highest value of attention in the generated map provides the desired semantic correspondence. We start by reviewing the basics of diffusion models in /eection 3.1, detail our two-stage algorithm in Section 3.2 and augmentation techniques to boost its performance in [Section 3.3
3.1 Preliminaries
Diffusion models are class of generative models that approximate the data distribution by denoising a base (Gaussian) distribution (59) In the forward diffusion process, the input image I is gradually transformed into Gaussian noise over series of T timesteps. Then, sequence of denoising iterations eo (I,,), parameterized by 0, and t I....,T I take as input the noisy image I, at each timestep and

---

Source Image
Diftuse
Source Attention Map
Denoise
Prediction
Optimization
Image Encoder
Query
Cross
A
Attention
Layers
Text Embedding
Image Encoder
Inference
Cross
Argmax
Zo
2x.1
Attention
Layers
Denoise
Target Image
Diffuse
Target Attention Map
Figure 3: Method (Top) Given a source image and a query point, we optimize the embeddings so that the attention map for the denoising step at time highlights the query location in the source image. (Bottom) During inference, we input the target image and reuse the embeddings for the same denoising step I, determining the corresponding point in the target image the argmax of the attention map. The architecture mapping images to attention maps is pre-trained Stable Diffusion model 30 which is kept frozen throughout the entire process.
predict the noise added for that iteration The diffusion objective is given by:
LDm (I,t..~~((0.1) [||c (T. t)][
Rather than directly operating on images I, latent diffusion models (LDM) 30 execute instead on latent representation z, where an encoder maps the image I into a latent z, and a decoder maps the latent representation back into an image. The model can additionally be made conditional on some text y. by providing an embedding e Ty(y) using a text encoder тo to the denoiser:
LLOM Ee,,
c2 t, 1)
The denoiser for a text-conditional LDM implemented by a transformer architecture involving combination of self-attention and cross-attention layers.
30)
3.2 Optimizing for correspondences
In what follows, we detail how we compute attention masks given an image/embedding pair, how optimize for an embedding that activates a desired position in the source image, and how to employ this optimized embedding to identify the corresponding point in the target image.
Atention masks. Given an image I, z(t) be its latent representation within the diffusion model at the t-th diffusion step. We first compute query Q (z==tt L and key K₁ (e), where (-). and (-(-) are the 1-th linear layers of the U-Net [72) that denoises in the latent space. The cross-attention at these layers are then defined as:
( CrossAttention Q:: Ki) softmax (Q.K7 /Vdi
where the attention map M' E (CCx(xx) and P. C, h, and w respectively represent the number of tokens, the number of attention heads in the transformer layer, the height, and width of the image
We select the I 1=8 diffusion step of 7'=50 steps diffusion model via hyper-parameter tuning.

---

(a) Source image
(b) Layer? 7
(c) Layer
(d) Layer 9
(e) Layer 10
(l) Average
(g) Target image
(h) Layer 7
(i) Layer
ii Layer
(k) Layer
Average
Figure 4: Attention response of different layers We show example attention maps (b-e) for particular U-Net layers for the optimized embedding that correspond to the location marked with the yellow star on the source image (a) and corresponding average in image (f). We use the same embedding for another (target) image (g) and display its attention map as well (h-k) and their average (I). The ground-truth semantically corresponding region in the target image marked also with the yellow star. Earlier layers respond more broadly, whereas later layers are more specific. To utilize these different characteristics of each layer average that of multiple layers into single attention map.
at the particular layer in the U-Net. Here, Q E ([(xx))xd; and K E IR Px where di represents the dimensionality of this layer, and softmax denotes the softmax operation along the P dimension.
As mentioned earlier in [Section 1 and observed in [65], different layers of the U-Net exhibit different "level" of semantic knowledge; see Figuuee 4] Thus, to take advantage of the different characteristics each layer, we average along both the channel axis and across a subset of U-Net layers M" E (
(M)) Note that the size of the attention maps M; differ according to each layer, hence we utilize bilinear interpolation when averaging. Hence, with M(u e, I) we denote indexing a pixel u of the attention map M"[1] IR (B(hww) via bilinear interpolation, where [1] extracts the first of the P available attention maps"] that is, we use the first token of the embedding to represent our query. Examples of these attention maps for embeddings e derived from natural text are visualized in Figure 2
Optimization. Given source image I, and query pixel location E 1J we interested in finding the corresponding pixel u, in the target image I,. We emulate the source attention map for the query Gaussian of standard deviation centered at the query location u,:
M.(u) exp (-Ilu u u.ll3/20²).
The Gaussian map represents the desired region focus of the attention mechanism. We then optimize for the (text) embedding that reproduces the desired attention map
arg min Σ M M. (u)
Inference. With the optimized embedding e", we can then identify the target point (the point uy in I, most semantically similar to u; in I by computing the attention map for the target image, and finding the spatial location that maximizes it. We write:
U arg max M(uee", I,).
We do not employ the first or last token as typically these are special termination tokens.
'We empirically observed that the choice of which token does have significant impact on final outcome as all other tokens (i.e., P entries of e) all are also optimized regardless due to the softmax that we apply along the P axis optimization will find the prompts (or more exactly the embeddings) that match the chosen token location.

---

Successuul
Mixed
Failure
Spair-7lk
CUB-200
PF-Willow
Figure 8: Qualitative examples correspondences estimated from our method are colored in blue correct in orange if wrong according to PCK (Top) successful cases, (middle) mixed cases, and (bottom) failure cases. Interestingly, even when our estimates disagree with the human annotation (thus shown as orange), they are arguably at plausible locations.
3.3 Regularizations
As discussed [Section I optimizing text embeddings on a single image makes susceptible to overfitting Furthermore, the process of finding embeddings via optimization, textual inversion, has recently been shown
sensitive to initialization [35 36). To overcome these issues, we apply various forms of regularization. Note that while we describe these one at time, these two augmentations are simultaneously enabled.
Averaging across crops. Let (I) be a cropping operation with parameters c, and (T( the operation placing the crop back to its original location; that is, Ca(H4e(x.) Xx for some crop Xe- For the we reduce the image dimensions to 93% (found via hyperparameter sweep) and sample uniformly random translation within the image we will denote this sampling as c D. We augment the optimization in Equation 5 by averaging across cropping augmentations as:
arg min EAED Σ |[C.(M( L.) Cc(MM,
similarly, at inference time, average the attention masks generated by different crops:
= arg max EcED Vc(M(u;;', ,C,(I,)))
(8)
Averaging across optimization rounds. We empirically found that doing multiple rounds of optimization is one of the keys obtaining good performance; see Figure 9 To detail this process, let us abstract the optimization in Equation 7 as e* = (e, L),, where e is its initialization. We then average the attention masks induced by multiple optimization runs as:
u, arg EE-D M(u; ((e, I)),
u
4 Results
evaluate semantic correspondence search on three standard benchmarks: SPair-71k [14 the largest standard dataset for evaluating semantic correspondences composed of 958 image pairs of 18 different classes. Since we do not perform any training, we only use the 12, 234 correspondences
of the test set for evaluation; PF Wi-lillow [33 comprises four classes wine bottle, duck, motorcycle,

---

Table 1: Quantitative results We report the Percentage of Correct Keypoints (PCK), where bold numbers are the best results amongst weakly- or un-supervised methods. Our method outperforms all weakly supervised baselines (we use the numbers reported the literature). Note also that for PF-Willow, our method outperforms even the strongly-supervieed state of the art in terms of PCK
CUB-200
PF-Willow
SPair-7Ik
PCKg0.05
PCKg0.1
PC
PCK
PCKg0.0s
PCKgo.
PWarpC-CC-Ne NC-Neelrretta
48.0
76.2
21.5
37.1
CHM
52.7
79.4
27.2
46.3
Strong supervision
VAT
81.6
35.0
55.5
CAT+++
56.7
81.2
59.8
PMD
40.3
74.7
26.5
PSCNet-SE
42.6
75.1
27.0
Weak supervision
VGG+MLS
18.3
25.8
41.2
63.2
27.4
52.0
67.0
31.1
WWarp-NCC-Net.aal
45.0
75.9
18.2
35.3
ASIC
57.9
75.9
53,0
76.3
36.9
DINO+NN
52.8
68.3
40.1
60.1
33.3
Unsupervised
Our method
61.6
77.5
53.0
84.3
28.9
45.4
and car - with 900 correspondences in the test set; CUB-200 [37 includes 200 different classes of birds. Following ASIC 46 we select the first three classes, yielding total of 1, 248 correspondences in the test set.
Metries. We measure the performance of each method via the standard metric 13
23 of
Percentage of Correct Keypoints (PCK) under selected thresholds we report how many semantic correspondence estimates are within the error threshold from human-annotation. Following the standard protocol 13 14 23 for SPair-7Ik and PF-Willow thresholds are determined with respect to the bounding boxes of the object of interest, whereas for CUB-200, the thresholds are set relative to the overall image dimensions. We report results for thresholds 0.05 and 0.1, as annotations themselves are not pixel-perfect due to the nature of semantic correspondences i.e., there is no exact geometric correspondence.
Implementation details. We use Stable Diffusion version 1.4 In our ablations, we find that more augmentation help, but with the caveat of (linearly) requiring more computation. Hence, on CUB-200 and PF- Willow datasets we use 10 optimization rounds for the embeddings and 30 random crops for the inference. For the larger SPair-71k dataset we use less 5 embeddings and 20 crops. We choose our hyperparameters based on the validation subset of SPair-71k and PCK PCK 20.05 via fully-randomized search and applied them to all datasets. We use the Adam [73 optimizer to find the prompts. For detailed hyperparameter setup see Supplementary Material.
show qualitative examples of the semantic correspondences estimated by our method in Figure 8 Interestingly, our method, even when it disagrees with the human annotation, provides results that can arguably interpreted as plausible. For example, as shown in the wine bottle example at the bottom-right of Figure 8) source points occluded by the wine glasses are correctly mapped to another wine glass the target image, which disagrees with the human annotator's label which points to the wine bottle.
Qualitative highlights.
Quantitative results. We report the performance of our method in Table 1 We compare our method against wcakly-supervised basclines [2 52 46), as well as baselines from ASIC [46) that are based on general-purper purpose deep features using VGG (4 features with Moving Least Squares (MLS)
and DINO 25 features with Moving Least Squares (MLS) or Nearest Neighbor (NN) matching.
Our method outperforms all compared weakly supervised baselines. Note that the performance gap in terms of average PCK compared to the second best method, ASIC [66, large- 20.9% relative. Note also that in the case of the Willoow dataset, our method is on par with the current strongly supervised state-of fate-off-he--rt. Even in the case the SPair-71k dataset, our results are comparable to very recent method VAT 15 besides the few problematic classes bottle, bus, plant, train, tv. These problematic classes are typically those that exhibit strong symmetry, which we do not explicitly deal with. For detailed per-class results, see our Supplementary Material. Considering that our method is

---

90
PCKi80.0
45
PCKiBn.05
30
10
16 Number crops
Layurwithin netwerk
Namber of embooddings optiiiized
(a) Individual layer performance
(b) embeddings vs performance
(c) fecops vs performance
Figure 9: Ablations We ablate performance measured by the mo.00 metric the PF- Willow [33 dataset. (a) Using multiple layers; (b) Using optimization augmentations; (c) Using crop augmentations. Dashed-line denotes the performance of full method. Note that in (a) individual layer performance is significantly worse, showing that the information within layers is complimentary. In (b) and (c) using more embeddings and crops leads improved performance.
Figure 10: Semantic correspondence between different classes We show examples of applying our method to pairs of image from different classes. We manually mark those that seem arguably correct with blue. Our method in many cases generalizes well.
fully unsupervised when comes to the task of semantic correspondences, these results are quite surprising.
Ablations. To ablate our model, we the PF-Willow 13 dataset and report PCK In Figure 9a| we show that using individual layers leads to significantly worse results than using multiple layers together; this demonstrates the importance of capturing semantic knowledge across multiple receptive fields. FFiiurrr 9b] we show that using multiple optimized embeddings significantly boosts performance, and in Figure 9c see how using inference. Finally, besides crops during inference, we disable random crops during embedding optimization PCKa..os drops 45.5 (vs. 53.0 in our full model).
crops further leads improvements during
Beyond semantic correspondences of the same class.. We further experiment with correspondences across different classes e.g., correlating different animals. As shown in Figure 10 in many cases method provides correspondences that arguably correct even for objects that are of different classes. This includes being from similar classes (sheep and dog) to more different classes (bird and airplane), and to wildly different classes (chair and horse).
Conclusions
We have demonstrated the remarkable potential leveraging diffusion models, specifically Stable Diffusion [30], for the task of semantic correspondence estimation. We have shown that by simply optimizing and finding the embeddings for location of interest, one can find its semantically corre- sponding locations in other images, although the diffusion model was never trained for this task. We further introduced design choices that significantly impact the accuracy of the estimated correspon- dences. Ultimately, our approach significantly outperforms existing weakly supervised methods on SPair-7Ik [4), Wi-Willow [13), and CUB-200 datasets [37] (20.9% relative for SPair-7Ik), and even achieves performance on par with strongly supervised methods in PF-Willow dataset [3
Limitations and future work. This study highlights the emergent properties of training large models on vast amounts data, revealing that model primarily designed for image generation can also be utilized for other tasks, specifically semantic correspondences. The success of our method

---

underscores the importance of exploring the hidden capabilities of these large text-to- text-to-image generative models and suggests that there may be other novel applications and tasks that can benefit from the vast knowledge encoded within these models. For example, as an immediate application, our method can be used to scale up training of 3D generative models such as FigNeRF (49 with images from the web without human supervision.
We note that our method does have its limitations. Many of the failure modes include dealing with symmetric objects. An explicit way to deal with these cases with more refined techniques to extract correspondences may help solve this problem. It also requires significant compute. On an NVIDIA RTX 3090 GPU, finding a single prompt for single keypoint takes 30 seconds. We suspect that one could potentially train a semantic correspondence network based on our estimates achieve both scalability in terms of training data and fast inference time.
Acknowledgments and Disclosure of Funding
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google, Digital Research Alliance of Canada, and by Advanced Research Computing at the University of British Columbia.
References
[1] B.B. Hansen and B.S. Morse. Proceedings of the ieee conf. on computer vision and pattern recognition. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition. 1999.
[2] Serge Belongie. Jitendra Malik, and Jan Puzicha. Shape context: A new descriptor for shape matching and object recognition. Advances in Neural Information Processing Systems, 13, 2000.
[3] Johannes Lutz Schönberger and Jan-Michael Frahm. Structureffrommmotton revisited. In Conference on Computer Vision and Pattern Recognition, 2016.
[4] David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004.
[5] Alex Fisher, Ricardo Cannizzaro, Madeleine Cochrane, Chatura Nagahawatte, and Jennifer L Palmer. Colmap: memory-efficient occupancy grid mapping framework. Robotics and Autonomous Systems, 2021.
[6] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference Computer Vision, 2016.
[7] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceeuings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2018.
[8] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2020.
[9] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2018.
[10] Zachary Teed and Jia Deng. Raft: Recurrent all- pairs field transforms for optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision, 2020.
[11] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 2010.
10

---

[12] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 2021.
[13] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: correspondences from object proposals. In arXiv Preprint, 2017.
Semantic
[14] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-7lk: A large-scale benchmark for semantic correspondence. arXiv Preprint, 2019.
[15] Sunghwan Hong. Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation. In European Conference on Computer Vision, 2022.
[16) Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[17] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: Correspondence Transformer for Matching Across Images. In International Conference Computer Vision, 2021.
[18] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector- free local feature matching with transformers. Conference on Computer Vision and Pattern Recognition, 2021.
[19] Zhengqi Li and Noah Snavely. Megadepth: photos. In Conference Computer Vision and Pattern Recognition, 2018.
Learning single-view depth prediction from internet
[20] Yuhe Jin, Dmytro Mishkin, Anastasiaa Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. International Journal of Computer Vision, 2021.
[21] Xin Li, Deng-Ping Fan, Fan Yang, Ao Luo, Hong Cheng, and Zicheng Liu. Probabilistic model distillation for semantic correspondence. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2021.
[22] Jiwon Kim, Kwangrok Ryoo, Junyoung Seo, Gyuseong Lee, Dachwan Kim, Hansang Cho, and Seungryong Kim. Semi-Supervised Learning of Semantic Correspondence with Pseudo-Labels. In Conference on Computer Vision and Pattern Recognition, 2022.
[23] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. arXiv Preprint, 2023.
[24] Shuying Liu and Weihong Deng. Very deep convolutional neural network based image clas- sification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015.
[25] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in superviid vision transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pages 9650-9660, 2021.
[26] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv Preprint, 2022.
[27] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
[28] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, arXiv Preprint, 2022.

---

[29] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Hya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv Preprint, 2021.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, 2022.
[31] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
[32] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[33] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2022.
[34] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv Preprint, 2022.
[35] Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion. arXiv Preprint, 2023.
[36] Anton Voronov, Mikhail Khoroshikh, Artem Babenko, and Max Ryabinin. Is This Loss Informative? Speeding Up Textual Inversion with Deterministic Objective Evaluation. arXiv Preprint, 2023.
[37] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical report, California Institute of Technology, 2011.
[38] Bruce D Lucas and Takeo Kanade. An Iterative Image Registration Technique With an Applica- tion Stereo Vision. In International Joint Conference on Artificial Intelligence, 1981.
[39] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Rick Szeliski. Building rome in a day. Communications of the ACM, 2011.
[40] Jan-Michael Frahm, Pierre Fite-Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, and Marc Pollefeys. Building rome on a cloudless day. In Kostas Danilidiss Petros Maragos, and Nikos Paragios, editors, European Conference on Computer Vision, 2010.
[41] Ondrej Chum, Tomas Werner, and Jiri Matas. Two- View Geometry Estimation Unaffected by Dominant Plane. Conference on Computer Vision and Pattern Recognition, 2005.
[42] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to Find Good Correspondences. In Conference on Computer Vision and Pattern Recognition, 2018
[43] Weiwei Sun, Wei Jiang, Andrea Tagliasacchi, Eduard Trulls, and Kwang Moo Yi. Attentive context normalization for robust permutation-cquivarian learning. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2020.
[44] Prune Truong, Martin Danelljan, and Radu Timofte. GLU-Net: Global-local universal network for dense flow and correspondences. In Conference on Computer Vision and Pattern Recognition, 2020.
[45] Prune Truong, Martin Danelljan, Lue Van Gool, and Radu Timofte, GOCor: optimized correspondence volumes into your neural network, In Advances in Neural Information Processing Systems, 2020.
Bringing globally
12

---

[46] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. arXiv Preprint. 2023.
[47] Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision, 2020.
[48] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Conference on Computer Vision and Pattern Recognition, 2019.
[49] Christopher Xie, Keunhong Park, Ricardo Martin- Bartinn and Matthew Brown. Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling. International Confer- ence on 3D Vision, 2021.
[50] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2016.
[51] Gopal Sharma, Kangxue Yin, Subhransu Maji, Evangelos Kalogerakis, Or Litany, and Sanja Fidler. Mvdecor: Multi-view dense correspondence learning for fine-grained 3d segmentation. In European Conference on Computer Vision, 2022.
[52] Sangryul Jeon, Seungryong Kim. Dongbo Min, and Kwanghoon Sohn. Pyramidal semantic correspondence networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
[53] Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech, and Time Series. 1998.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[55] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, 2012.
[56] Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics, 2018.
[57] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv Preprint, 2021.
[58) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[59] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, In International Conference on Machine Learning, 2021.
[60] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021.
[61] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation. In Conference Computer Vision and Pattern Recognition, 2023.
[62] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv Preprint, 2021.
[63] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
13

---

[64] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3e 3d using 2d diffusion. arXiv Preprint, 2022.
[65] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-Efficient Semantic Segmentation with Diffusion Models. In International Conference Learning Representations, 2022.
[66] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural Descriptor Fields: Squiv-aqui Object Representations for Manipulation. In International Conference on Robotics and Automation, 2022.
[67] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures:
Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 2023.
[68] Junyi Zhang. Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 2023.
[69] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 2023.
[70] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv Preprint, 2014.
[71] Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamareh. Slime: Segment like me. arXiv Preprint, 2023.
[72] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Conference on Medical Image Computing and Computer Assisted Intervention, 2015.
[73] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv Preprint, 2014.
14


Reviews:
[{"review_id": "QvtAk5ky45", "reviewer": "Reviewer_5igN", "summary": "The paper proposes an approach for unsupervised semantic correspondence by employing a pre-trained Stable Diffusion generator. The basic idea is that since text-to-image diffusion models are capable of generating realistic images, they must understand the semantics of the objects they generate and thus be capable of finding semantic similarities in images.\n\nTechnically, the approach feeds a real image (or, better, its latent vector) into a diffusion pipeline, recover attention matrices between pixel values and the textual prompt, and optimizes the textual prompt embedding so that the cross-attention focuses on a query pixel location. Once optimization is over, the same text embedding can be employed to find semantic correspondences in a target image. Cross-attention matrices are averaged across channels and a subset of U-Net layers, using bilinear interpolation. Further, the target attention map is simulated with a Gaussian distribution centered on the query location, and a L2 loss is employed for optimization.\n\nAuthors also propose two regularization strategies to prevent over-fitting during optimization, i.e. averaging across crops of the query image, and averaging across multiple rounds of optimizations/initialization choices.\n\nExperimental results are conducted over three standard benchmarks for the task, i.e. SPair-71k, PF-Willow and CUB-200. The proposed approach achieves superior results when compared to unsupervised or weakly-supervised approaches, and is also competitive when compared to models employing strong supervision.", "questions": "See above.", "limitations": "The authors have successfully discussed the limitations of the work in the appropriate section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The main idea of the paper (extracting/exploiting the semantics embedded in a pre-trained a SD network) is novel and interesting. It is in-line with some recent pre-prints working on related tasks (e.g. image segmentation), but to my knowledge the proposed approach is novel wrt published papers.\n- The technique for extracting semantics is simple and effective, and also are the two regularization techniques.\n- Experimental results are solid and confirm the appropriateness of the approach.\n- The paper is well written, clear and a pleasure to read. The supplementary material is also interesting and comprehensive.", "weaknesses": "- While the idea behind the paper is very general and could have been applied to a variety of tasks involving semantics (e.g. image segmentation, panoptic segmentation, classification), the paper only focuses on finding semantic correspondences. This limits the overall impact of the work to a single task of Computer Vision. I wonder if authors have tried to extend to other tasks, and how difficult that would be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for recognizing that our proposed approach is novel, effective and our experimental results are solid. We address the reviewer's comments below:\n\n### ***Discussion on the extension of the approach to tasks such as semantic segmentation, classification etc.***\n\nWe thank the reviewer for suggesting an extension to this work. Indeed image segmentation tasks could possibly be accomplished by optimizing an embedding to attend to a region as opposed to a single point.  We plan to continue exploring potential extensions -- as the reviewer pointed out, our framework is indeed very general after all, and can be seen as a direction forward for extracting learned knowledge from these large generative models."}, {"review_id": "Kehbz0HwD2", "reviewer": "Reviewer_pziK", "summary": "This paper explores unsupervised semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first optimize the prompt embeddings of stable diffusion model, to maximize attention on the region of interest, then the optimized prompts are used for semantic correspondence.\nExperimental results on PF-Willow, CUB-200 and SPair-71k datasets show that proposed method significantly outperforms weakly / unsupervised methods.\n\n", "questions": "please refer them to the weakness session", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper shows that one could explore diffusion models for semantic correspondence tasks.\n\n\n- the writing is clear and easy to follow.\n\n- The experiments show that the proposed method could even outperform supervised method on PF-Willow.\n", "weaknesses": "\n\n1. How important is the optimization for prompts? Could the authors show results if the prompts are a vanilla sentence, e.g. \"an image of a cat\"?\n2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models:\n\n[1*] Luo, Grace, et al. \"Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence.\" arXiv preprint arXiv:2305.14334 (2023).\n\n[2*] Tang, Luming, et al. \"Emergent Correspondence from Image Diffusion.\" arXiv preprint arXiv:2306.03881 (2023).\n\n[3*] Zhang, Junyi, et al. \"A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence.\" arXiv preprint arXiv:2305.15347 (2023).\n\nCould the authors explain the differences if possible?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and for recognizing that our approach achieves state-of-the-art results. We address the reviewer's concerns below:\n\n### ***W-1: Importance of prompt-optimization for the correspondence task***\n\nOptimization of the prompts is critical. Without it, one cannot truly “localize”. As an example, we are attaching the attention map corresponding to the prompt “An image of a cat”. For the token “cat” the entire body of both cats is attended to rather than a single cat or parts of the cat. See respectively **Figures 1 and 4** in the provided rebuttal PDF above. \n\n### ***W-2: Comparison with papers submitted to Arxiv after Neurips deadline***\n\nWe note that the papers mentioned by the reviewer appeared on Arxiv after the NeurIPS submission deadline, and we believe that, per the NeurIPS guidelines, they should be counted as contemporary work and should not affect the evaluation of our paper.\n\nNonetheless, in the revision, we plan to mention them as they are quite relevant, and in fact, highlight the core difference that our method has compared to them -- they all look into how to use the deep features within the Stable Diffusion Network effectively, similarly to how VGG19 features are widely used for various tasks.  Our method, on the other hand, looks into how we can alter the attention maps within Stable Diffusion to our will, in other words taking into account how these features are supposed to be used within Stable Diffusion. To do so we optimize embeddings. And by doing so we show that one can perform alternative tasks than simple image creation such as the semantic correspondence task we demonstrated. However, this is not the end of what our framework can do as *reviewer 5igN* pointed out. For example, an immediate straightforward extension of our method could be to learn an embedding for a part of the cat, e.g., paw, using multiple images and not just one. We note that our current work was a first investigation about whether this was at all possible, and that whether one solution is more general and useful than the other, is a question for future works.\n\nIn more detail, Diffusion Hyperfeatures [1*] consolidates multi-scale and multi-timestep feature maps from Stable Diffusion into per-pixel feature descriptors with a lightweight aggregation network. A Tale of Two Features [3*] introduces a fusion approach that capitalizes on the distinct properties of Stable Diffusion (SD) features and DINOv2 by extracting per-pixel features from each. Emergent Correspondence from Image Diffusion [2*] extracts per pixel features from Stable Diffusion.\n\n[1*] Luo, Grace, et al. \"Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence.\" arXiv preprint arXiv:2305.14334 (2023).\n\n[2*] Tang, Luming, et al. \"Emergent Correspondence from Image Diffusion.\" arXiv preprint arXiv:2306.03881 (2023).\n\n[3*] Zhang, Junyi, et al. \"A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence.\" arXiv preprint arXiv:2305.15347 (2023)."}, {"review_id": "IIQ4MsX7oc", "reviewer": "Reviewer_Vgvc", "summary": "This paper focuses unsupervised semantic correspondence and proposes to leverage the semantic knowledge within fashionable text-to-image diffusion model to accomplish this task. The method optimizes a learnable text prompt to maximize source image attention value of query location. Then, the optimized text prompt is used to find corresponding location of target image by finding the max value of target image attention value. Related experiments show the effectiveness of the proposed method.", "questions": "Please see “Weakness” part.", "limitations": "The authors objectively discuss the limitations of the proposed method, e.g., the slow inference time caused by the introduction of diffusion model. This paper doesn’t have negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The idea of leveraging semantic knowledge of text-to-image diffusion model sounds interesting. This paper builds a new type of unsupervised semantic correspondence method based on text-to-image diffusion model.\n\n2.The designed method is effective and achieve promising correspondence results on common datasets.\n\n3.The design choices are reasonable, e.g., random crop for preventing overfitting when finding the text prompts.", "weaknesses": "1.The usage of text-to-image (t2i) diffusion model is not well-motivated. The proposed method mainly leverages the cross-modal ability of t2i diffusion model but underuses the generative ability of t2i diffusion model. The role of t2i diffusion model in proposed method can be replaced by other cross-modal pretraining model, e.g., CLIP, BLIP, GLIP. Please claim the unique significance for adopting t2i diffusion model rather than other cross-modal pretraining model.\n\n2.The proposed method has to execute dozens of denoising steps for each query location. As diffusion model is notorious for its inference speed and cost, the total overhead is expensive. Are there any mitigation measures? For example, grouping the query locations or modifying the diffusion model architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for recognizing that our approach is effective and our results on unsupervised semantic correspondences are promising. We address the reviewer's comments below:\n\n### **W-1: Missing motivation of using text-to-image (t2i) diffusion models and comparison with CLIP, BLIP, or GLIP networks.**\n\nOur work is motivated by the observation that text-to-image diffusion models are capable of generating high-quality images conditioned on textual prompts, indicating that these models understand the semantics of the objects being generated. This is owing to the specialized architecture used in Stable Diffusion which produces attention maps using cross-attention between textual and image features.\n\nIt is indeed possible to leverage cross-modal models like BLIP within our proposed framework, although suboptimal. Taking into account the reviewer's suggestion, we use BLIP to get semantic correspondence between a pair of images by capturing the output from the cross-attention layers between the text embeddings and the image features. This is done similarly to our method across multiple layers in the model and aggregated. We get significantly worse performance on PF-Willow (PCK 13.1% for PCK @0.05 and 31.7% for PCK @0.1 ) in comparison to our method (PCK of 53.0% for PCK @0.05 and 84.3% for PCK @0.1). This underperformance may be attributable to factors such as the dataset size the model was trained on, the number of parameters in the model itself, or the model outputting text tokens as opposed to denoising the image. We thank the reviewer for this suggestion and will include these results in the revision.\n\n### **W-2: Can grouping the query location reduce the computation overhead of SD inference?**\n\nThis is an interesting observation. This would indeed reduce the computation overhead significantly. We leave this research to be explored in the future."}, {"review_id": "Nxwf3AEYO2", "reviewer": "Reviewer_6281", "summary": "This paper proposes a model that finds semantic correspondence between a pair of images, with a pretrained diffusion model. By optimizing the prompt embeddings, such correspondence can be read out via attention maps in the UNet. Empirical study shows the good performance of this proposed model over exising baselines.", "questions": "1. I'm wondering how would the trained prompt compare to text embeddings, e.g. how would a prompt optimized from a bird's eye compare to the text embedding of \"a bird's left eye\"?\n2. One textual baseline that doesn't appear in the experiments is the attention map of an actual text prompt, i.e. would the prompt of \"a bird's left eye\" have its highest attention to the eye's location?", "limitations": "The authors has addressed valid limitations of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method has a simple form but is effective. \n2. The presentation is clear and easy to follow.\n2. The qualitative results are impressive, especially the results between different classes.\n3. Some quantitative results are significantly better than baselines, or comparable to supervised results. ", "weaknesses": "1. More information should be provided for the optimization set up and implementation, e.g. the optimization algorithm for Eq (5) and Eq (9).\n2. The complexity and efficiency are needed for better assessment of the algorithm, especially in comparison to baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback. We appreciate that the reviewer finds our approach effective and our qualitative results across classes impressive. We address the reviewer's concerns below:\n\n### **W-1: More information about optimization setup and implementation**\n\nWe provide detailed information about the implementation in the Supplementary, as it is difficult to fit into the main paper.\n\nWe guarantee we will release the code upon acceptance of the manuscript to facilitate reproduction.\n\n### **W-2: The complexity and efficiency are needed for a better assessment of the algorithm, especially in comparison to baselines.**\n\nWe only optimize the text embeddings, hence 59k parameters. No stable diffusion weights are touched. As reported in the submission (Section 5), this results in ~30 seconds per single prompt on a 3090 GPU. Note that while this is the case, compared to baselines, this could result in less compute time if evaluating only a few images -- this is because the weakly supervised baselines require training a deep network (which takes approximately 3 hours per dataset for the best-performing baseline, ASIC), which is not required by our method.\n\n### **Q-1 and Q-2: How would a prompt optimized from a bird's eye compare to the text embedding of \"a bird's left eye\"? What does the attention map look like?**\n\nTo effectively visualize the optimized embeddings we look into the attention maps for the optimized embedding and the natural prompt.\n\nIn the provided rebuttal PDF above, **Figure 2** illustrates an initial image of a bird, while **Figure 3** shows the corresponding optimized attention map focused on the bird's eye. The optimized attention map notably aligns with the Gaussian distribution it was supervised to imitate.\n\nFurthermore, **Figure 5** in the same PDF also visualizes the attention maps for each token corresponding to the sentence \"a bird's left eye.” The bird’s eye gets highlighted as we would expect, but not as localized as the optimized embedding."}, {"review_id": "tHVnTPJeTx", "reviewer": "Reviewer_DBqD", "summary": "The authors propose an approach to establish semantic correspondences, ie correspondences between different instances of the same object, using a pre-trained stable diffusion models, and without any task-specific finetuning. In particular, they leverage the fact that intermediate attention layers of unet diffusion model respond to the semantic of the text prompt. For a source image and a particular query pixel location for which a match should be computed, they optimize the prompt embedding that leads to a cross-attention map with a high activation at the query location. Once the prompt is recovered, they can apply the reverse attention process on a target image conditioned on the optimized prompt. The intermediate cross-attention layers of the diffusion model highlight the matching location. They also introduce technical ‘tricks’ to make it work, for example averaging over multiple crops of an image or starting from different random embeddings. This process must be repeated for each query pixel location. ", "questions": "In addition to the weakness above:\n\nA) What is the resolution of the extracted feature/attention maps and at what resolution is the matching done? for example are the features interpolated to higer resolution before computing the cross-attention weights? \nIs a similar resolution used in competitor works like DINO? \nThis is important because the resolution of the feature maps is of crucial importance for obtaining fine-grained/accurate correspondences. It has been the limiting factor of many previous approaches that needed to compute all-to-all correspondences, because of memory constraint. Just using a higher resolution feature map for the cross-attention would lead to an improvement in results, not necessarily related to the features themselves. \n\n\nB) L.230 the authors mention it takes ‘30s’ to optimize the prompt, does it include only finding the prompt? or finding the match? A real run-time analysis of time versus performance for different settings would be interesting here (depending on how many crops are used, how many initializations ect).\n\nC) clarification needed: \nC.1) L.178: M′ has dimension C×(h×w)×P, where is the channel coming from? According to equation, M’ is computed with the standard attention mechanism, which is just a matrix multiplication followed by a softmax. As a result, the dimensions should be (hxw)xP. \n\nC.2) L.185, 187, it would be helpful to specify the shape of M′′ and M, to better follow the different steps.\nWhat is the dimension of P for the learnt embedding?\n\nC.3) L213, is it equation 5 or equation 7? The authors say they average the optimization over multiple crops and over multiple initialization of the embeddings. Is this happening at the same time, i.e. for different initialization, the optimization is done over multiple crops? If so, then it should be equation (7). \n\nC.4) in Tab1, what is meant by ‘weakly-supervised (test-time optimization)’? DINO with NN is for me unsupervised since it only requires single images for the contrastive learning part. \n\n\nsome typos:\nL.207 hyperparemter\nL.53-54 Note that while finding actual prompt thatcorrespond to words is a discrete problem, hence difficult to solve. -> remove while\n", "limitations": "they have adequately addressed the limitations an negative impact of their work. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The insight that stable diffusion models inherently learn semantic correspondences without task-specific training is interesting. How to extract and use such knowledge has not been explored in the past. \n\n- The results are convincing - the approach obtains near state-of-the-art in semantic matching without any task-specific knowledge or funetuning. \n\n- The paper reads well overall\n", "weaknesses": "1) Missing comparisons and citations in results: PWarpC [Truong et al. CVPR 2022]: \n- the paper only includes comparisons to two weakly-supervised semantic matching approaches, with relatively weak results, which allows the authors to claim that their approach is 20% better than the state-of-the-art in weakly-supervised semantic matching. \n- The authors did not include results of PWarpC, a weakly-supervised approach for semantic matching presented at CVPR 2022. The results of PWarpC significantly outperform the baseline results of this paper for SPair-71K (38.0 on SPair-71K for PWarpC-NC-Net), making the above claim invalid. \n- on PF-Willow: to double check since the two papers use different metrics but PWarpC (weakly-supervised) outperforms CATs, which obtains better metrics than the proposed approach. It might therefore outperform the proposed approach. \n- please include this comparisons and revise the claims/analysis accordingly\n\n2) while the paper presents an interesting analysis, the proposed approach is currently very impractical and almost unusable, since as the authors say, it takes 30s to generate a single match. I believe it would be interesting to have even a small experiment training a semantic matching model based on pseudo ground-truth matches obtained from the diffusion model.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for recognizing that our approach provides new insights for the estimation of semantic correspondences with stable diffusion. We are glad that the reviewer finds our results convincing. We address the reviewer's concerns below:\n\n### **W-1: Missing comparison with PWarpC**\n\nThank you for pointing out this baseline. Below is the updated Table with PWarpC. As there are many variants, we have chosen the one that performs the best. Here we provide the updated table. The complete updated table can be found in **Table 1** in the included rebuttal PDF above. \n\n|  | CUB-200 |  | PF-Willow |  | SPair-71k |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | PCK@0.05 | PCK@0.10 | PCK@0.05 | PCK@0.10 | PCK@0.05 | PCK@0.10 |\n| Strong supervision |  |  |  |  |  |  |\n| PWarpC-NC-Net* res101 [1] | -- | -- | 48 | 76.2 | 21.5 | 37.1 |\n| Weak supervision |  |  |  |  |  |  |\n| PWarpC-NC-Net res101 [1] | -- | -- | 45 | 75.9 | 18.2 | 35.3 |\n| Unsupervised |  |  |  |  |  |  |\n| DINO+NN [2] | 52.8 | 68.3 | 40.1 | 60.1 | -- | 33.3 |\n| Our method | 61.6 | 77.5 | 53 | 84.3 | 28.9 | 45.4 |\n\n[1] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Probabilistic warp consistency\nfor weakly-supervised semantic correspondences. In Conference on Computer Vision and\nPattern Recognition, 2022.\n\n[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual\ndescriptors. arXiv Preprint, 2021.\n\n### **W-2 and Q-B: Practicality / Amount of time it takes to optimize the prompt**\n\nIt takes 30 seconds to optimize a text embedding; however, inference time is much quicker since it only needs to be run once per crop. For a single crop, it takes around 0.16 seconds, so for the 20 crops we used in our experiments, it takes 3.2 seconds. Inference time can be sped up by reducing the number of crops and Figure 9C of our paper shows the corresponding effect on final performance. This can be helpful for the case where a single optimized keypoint is applied to many images.\n\nWe would like to further note that while it takes 30 seconds, there is no network training involved. Other baselines, such as the weakly supervised ones on the table would require training. As an example, the best-performing weakly supervised baseline, ASIC, requires training for around 3 hours per dataset.\n\nRegarding supervising another model with output from our method as pseudo ground truth, due to the limited size of the training dataset, any form of supervised training is shown to have an overfitting effect.\n\nAs future research direction for a more practical solution, we will explore optimizing text embeddings for multiple query points jointly that will significantly speed up our approach (also suggested by *reviewer Vgvc*), or methods such as those suggested recently in the textual inversion literature using hypernetworks [3, 4]. We would like to also note that, while early methods have been slow, such as the original NeRF and AlexNet (at the time), follow-ups have significantly sped them up by orders of magnitude, for example as in Instant NGP and MobileNet.\n\n[3]. Ruiz, Nataniel, et al. \"HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models.\" *arXiv preprint arXiv:2307.06949* (2023).\n\n[4]. Arar, Moab, et al. \"Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models.\" *arXiv preprint arXiv:2307.06925* (2023).\n\n### **Q-A: Resolution of the features being used**\n\nWe only upsample the attention maps after the cross-attention layer to stay in line with the original stable diffusion model. Cross-attention layers with sizes 16x16 and 32x32 are upsampled to the original image size and then combined.\n\n### **Q-C: M′ has dimension C×(h×w)×P, where is the channel coming from**\n\nThank you for pointing this out. We will include the fact in the paper that the cross attention is applied separately over the C channels, leading to the dimension of M′ being C×(h×w)×P.\n\n### **Q-C.2: shape of M′′ and M and dimension of P**\n\nThank you for pointing this out. M'' is of shape [(H*W), P]. M is [H, W] and the dimension of P is 77. We will revise to make this clear.\n\n### **Q-C.3: Is this happening at the same time, i.e. for different initialization, the optimization is done over multiple crops? If so, then it should be equation (7).**\n\nThank you for pointing out this error. We will correct it. The reviewer is indeed correct that line 213 should be referring to equation (7), not equation (5). This then accurately reflects how the embeddings used at this stage are optimized with different crops.\n\n### **Q-C.4: DINO with NN is for me unsupervised since it only requires single images for the contrastive learning part.**\n\nThank you for pointing this out. We will update the table by putting DINO+NN in an unsupervised row. The provided table also reflects this change."}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We are glad that the reviewers acknowledge that our method is novel and provide new insights, with solid experimental results, and clear exposition. We provide detailed responses in the individual rebuttals and provide the figures and the tables in the attached PDF.


Meta-review: This dissertation deals with unsupervised semantic correspondences between a pair of images. They propose to exploit the semantic knowledge within fashionable text-image diffusion model to accomplish this task. By optimizing the text prompt embeddings of the stable diffusion model, such a correspondence can be read out via attention maps in the deep network. The resulting optimized prompts are used for semantic correspondence.  Related experiments on PF-Willow, CUB-200 and SPair-71k datasets clearly demonstrate the effectiveness of the proposed method. All reviewers emphasized the originality of the proposed work, but also asked for explanations and clarifications.
After a period of rebuttal and discussion, all were satisfied with the answers. Although it is a pity that the article focuses solely on semantic correspondences, this paper makes a solid contribution that should be accepted.
Decision: Accept (poster)