Title: Unsupervised Semantic Correspondence Using Stable Diffusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Semantic Correspondence

Stable Diffusion

Optimization-based Inference
Abstract: Text-to-image diffusion models are now capable of generating images that are often indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they are asked to generate. In this work we show that, without any training, one can leverage this semantic knowledge within diffusion models to find semantic correspondences – locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention on the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing so we obtain results on par with the strongly supervised state of the art on the PF-Willow dataset and significantly outperform (20.9% relative for the SPair-71k dataset) any existing weakly- or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.

Unsupervised Semantic Correspondence Using Stable Diffusion

Abstract

Text-to-image diffusion models are now capable of generating images that are often indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they are asked to generate. In this work we show that, without any training , one can leverage this semantic knowledge within diffusion models to find semantic correspondences - locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention on the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing so we obtain results on par with the strongly supervised state of the art on the PF-Willow dataset and significantly outperform (20.9% relative for the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.

1 Introduction

Estimating point correspondences between images is a fundamental problem in computer vision, with numerous applications in areas such as image registration [1], object recognition [2], and 3D reconstruction [3]. Work on correspondences can largely be classified into geometric and semantic correspondence search. Geometric correspondence search aims to find points that correspond to the same physical point of the same object and are typically solved with local feature-based methods [4, 5, 6, 7, 8] and optical flow methods [9, 10]. Semantic correspondence search - the core application of interest in this paper - focuses on points that correspond semantically [11, 12, 13, 14, 15, 16], not necessarily of the same object, but of similar objects of the same or related class. For example, given the selected kitten paw in the (source) image of Figure 1, we would like to automatically identify kitten paws in other (target) images. Whether geometric or semantic correspondences, a common

(a) input image

'paw' attention

'collar' attention

'shower' attention

'ears' attention

recent trend is to learn to solve these problems [17], as in many other areas of computer vision.

While learning-based methods provide superior performance [16, 15, 17, 18] often these methods need to be trained on large supervised datasets [17, 19]. For geometric correspondences, this is relatively straightforward, as one can leverage the vast amount of photos that exist on the web and utilize points that pass sophisticated geometric verification processes [17, 20, 19]. For semantic correspondences, this is more challenging, as one cannot simply collect more photos for higherquality ground truth-automatic verification of semantic correspondences is difficult, and human labeling effort is required. Thus, research on unsupervised learned semantic correspondence has been trending [21, 22, 23].

In this work, we show that one may not need any ground-truth semantic correspondences between image pairs at all for finding semantic correspondences, or need to rely on generic pre-trained deep features [24, 25] - we can instead simply harness the knowledge within powerful text-toimage models. Our key insight is that, given that recent diffusion models [26, 27, 28, 29, 30] can generate photo-realistic images from text prompts only, there must be knowledge about semantic correspondences built-in within them . For example, for a diffusion model to successfully create an image of a human face, it must know that a human face consists of two eyes, one nose, and a mouth, as well as their supposed whereabouts - in other words, it must know the semantics of the scene it is generating. Thus, should one be able to extract this knowledge from these models, trained with billions of text-image pairs [30, 31], one should be able to solve semantic correspondences as a by-product. Note here that these generative models can also be thought of as a form of unsupervised pre-training, akin to self-supervised pre-training methods [32, 25], but with more supervision from the provided text-image relationship.

Hence, inspired by the recent success of prompt-to-prompt [33] for text-based image editing, we build our method by exploiting the attention maps of latent diffusion models. These maps attend to different portions of the image as the text prompt is changed. For example, given an image of a cat, if the text prompt is 'paw', the attention map will highlight the paws, while if the text prompt is 'collar', they will highlight the collar; see Figure 2. Given arbitrary input images, these attention maps should respond to the semantics of the prompt. In other words, if one can identify the 'prompt' corresponding to a particular image location, the diffusion model could be used to identify semantically similar image locations in a new, unseen, image. Note that finding actual prompts that correspond to words is a discrete problem, hence difficult to solve. However, these prompts do not have to correspond to actual words, as long as they produce attention maps that highlight the queried image location. In other words, the analogy above holds when we operate on the continuous embedding space of prompts, representing the core insight over which we build our method.

With this key insight, we propose to find these (localized) embeddings by optimizing them so that the attention map within the diffusion models corresponds to points of interest, similarly to how prompts are found for textual inversion [34, 35]. As illustrated in Figure 1, given an (source) image and a (query) location that we wish to find the semantic correspondences of, we first optimize a randomly initialized text embedding to maximize the cross-attention at the query location while keeping the diffusion model fixed ( i.e. stable diffusion [30]). We then apply the optimized text embedding to another image ( i.e. target) to find the semantically corresponding location - the pixel attaining the maximum attention map value within the target image.

Beyond our core technical contribution, we introduce a number of important design choices that deal with problems that would arise from its naive implementation. Specifically, (1) as we optimize

on a single image when finding the embeddings, to prevent overfitting we apply random crops; (2) to avoid the instability and randomness of textual inversion [36, 35] we find multiple embeddings starting from random initialization; (3) we utilize attention maps at different layers of the diffusion network to build a multi-scale notion of semantic matching. These collectively allow our method to be on par with strongly-supervised state of the art on the PF-Willow dataset [13] and outperform all weakly- and un-supervised baselines in PF-Willow, CUB-200 [37], and SPair-71k datasets [14] - on the SPair-71k dataset we outperform the closest weakly supervised baseline by 20.9% relative.

We emphasize that our method does not require supervised training that is specific to semantic point correspondence estimation. Instead, we simply utilize an off-the-shelf diffusion model, without fine-tuning, and are able to achieve state-of-the-art results. To summarize, we make the following contributions:

we show how to effectively extract semantic correspondences from an off-the-shelf Stable Diffusion [30] model without training any new task-specific neural network, or using any semantic correspondence labels;

we introduce a set of design choices - random crops, multiple embeddings, multi-scale attention that are critical to achieving state-of-the-art performance;

we significantly outperform prior state of the art based on weakly supervised techniques on the SPair-71k [14], PF-Willow [13], and CUB-200 [37] datasets (20.9% relative on SPair-71k) and is on par with the strongly-supervised state of the art on the PF-Willow [13] dataset.

2 Related work

We first review work on semantic correspondences and then discuss work focusing on reducing supervision. We also discuss work on utilizing pre-trained diffusion models.

Semantic correspondences. . Finding correspondences is a long-standing core problem in computer vision, serving as a building block in various tasks, for example, optical flow estimation [38, 10], structure from motion [39, 40, 3, 5], and semantic flow [11]. While a large body of work exists, including those that focus more on finding geometric correspondences that rely on local features [4, 6, 7] and matching them [41, 42, 43, 8] or directly via deep networks [44, 45, 17, 18], here we focus only on semantic correspondences [11, 16, 15, 46], that is, the task of finding corresponding locations in images that are of the same 'semantic' e.g. , paws of the cat in Figure 1. For a wider review of this topic, we refer the reader to [47].

Finding semantic correspondences has been of interest lately, as they allow class-specific alignment of data [46, 48], which can then be used to, for example, train generative models [49], or to transfer content between images [11, 50, 51]. To achieve semantic correspondence, as in many other fields in computer vision, the state-of-the-art is to train neural networks [16, 15]. Much research focus was aimed initially at architectural designs that explicitly allow correspondences to be discovered within learned feature maps [10, 45, 52, 16]. For example using pyramid architectures [52], or architectures [16] that utilize both convolutional neural networks [53] and transformers [54]. However, these approaches require large-scale datasets that are either sparsely [14, 13] or densely [55] labeled, limiting the generalization and scalability of these methods without additional human labeling effort.

Learning semantic correspondences with less supervision. . Thus, reducing the strong supervision requirement has been of research focus. One direction in achieving less supervision is through the use of carefully designed frameworks and loss formulations. For example, Li et al. [21] use a probabilistic student-teacher framework to distill knowledge from synthetic data and apply it to unlabeled real image pairs. Kim et al. [22] form a semi-supervised framework that uses unsupervised losses formed via augmentation.

Another direction is to view semantic correspondence problem is utilizing pre-trained deep features. For example, Neural Best-Buddies [56] look for mutual nearest neighbor neuron pairs of a pre-trained CNN to discover semantic correspondences. Amir et al. [57] investigate the use of deep features from pre-trained Vision Transformers (ViT) [58], specifically DINO-ViT [25]. More recently, in ASIC [23] rough correspondences from these pre-trained networks have been utilized to train a network that maps images into a canonical grid, which can then be used to extract semantic correspondences.

These methods either rely heavily on the generalization capacity of pre-trained neural network representations [56, 57] or require training a new semantic correspondence network [21, 22, 23]. In

this work, we show how one can achieve dramatically improved results over the current state-of-theart, achieving performance similar to that of strongly-supervised methods, even without training any new neural network by simply using a Stable Diffusion network.

Utilizing pre-trained diffusion models. . Diffusion models have recently emerged as a powerful class of generative models, attracting significant attention due to their ability to generate highquality samples [30, 26, 27, 28, 29, 59, 60]. Diffusion models generate high-quality samples, with text-conditioned versions incorporating textual information via cross-attention mechanisms.

Among them, Stable Diffusion [30] is a popular model of choice thanks to its lightweight and opensource nature. While training a latent diffusion model is difficult and resource-consuming, various methods have been suggested to extend its capabilities. These include model personalization [61] through techniques such as Low Rank Adaptation (LoRA) [62] and textual inversion [34], including new conditioning signals via ControlNet [63], or repurposing the model for completely new purposes such as text-to-3D [64] and text-driven image editing [33]. While the applications vary, many of these applications are for generative tasks, that is, creating images and 3D shapes.

Of particular interest to us is the finding from Prompt-to-Prompt [33] that the cross-attention maps within diffusion models can effectively act as a pseudo-segmentation for a given text query - in other words, it contains semantic information. This is further supported by the fact that intermediate features of diffusion models can be used for semantic segmentation [65]. In this work, with these observations, and with the help of textual inversion [34], we show that we can utilize Stable Diffusion [30] for not just generative tasks, but for the task of finding semantic correspondences, a completely different task from what these models are trained for, all without any training by repurposing the attention maps, similarly as in [66] but for an entirely different application and framework.

Concurrently to our work, work utilizing feature representations within Stable Diffusion for correspondences has been proposed [67, 68, 69]. These methods look into how to use the deep features within the Stable Diffusion Network effectively, similar to how VGG19 features [70] are widely used for various tasks. Our method, on the other hand, looks into how we can alter the attention maps within Stable Diffusion to our will, in other words taking into account how these features are supposed to be used within Stable Diffusion - we optimize word embeddings. By doing so we show that one can perform alternative tasks than simple image creation such as the semantic correspondence task we demonstrated. However, this is not the end of what our framework can do. For example, a recent followup to our preprint demonstrated an extension of our method to segmentation [71]

3 Method

Our method identifies semantic correspondences between pairs of images by leveraging the attention maps induced by latent diffusion models. Given a pair of images (respectively source and target), and a query point in the source image domain, we seek to compute an attention map that highlights areas in the target image that are in semantic correspondence with the query. While classical diffusion models work on images directly, we employ latent diffusion models [30], which operate on encoded images. Further, rather than being interested in the generated image, we use the attention maps generated as a by-product of the synthesis process. Our process involves two stages; see Figure 3. In the first stage (optimization), we seek to find an embedding that represents the semantics of the query region in the source image by investigating the activation map of the denoising step at time t . In the second stage (inference), the embeddings from the source image are kept fixed, and attention maps for a given target image again at time t are computed. The location attaining the highest value of attention in the generated map provides the desired semantic correspondence. We start by reviewing the basics of diffusion models in Section 3.1, detail our two-stage algorithm in Section 3.2, and augmentation techniques to boost its performance in Section 3.3.

3.1 Preliminaries

Diffusion models are a class of generative models that approximate the data distribution by denoising a base (Gaussian) distribution [59]. In the forward diffusion process, the input image I is gradually transformed into Gaussian noise over a series of T timesteps. Then, a sequence of denoising iterations ϵ θ ( I t , t ) , parameterized by θ , and t = 1 , . . . , T take as input the noisy image I t at each timestep and

Image Encoder

Source Image

t-1

z

Cross

Attention

Layers

Source Attention Map

Denoise

Target Image

Target Attention Map

Text Embedding

z

0

Image Encoder

Cross

Attention

Layers

z

Denoise

0

Argmax

t

z

Optimization

Inference

z

t-1

Prediction

Query

t

z

Diffuse

Diffuse

predict the noise added for that iteration ϵ . The diffusion objective is given by:

Rather than directly operating on images I , latent diffusion models (LDM) [30] execute instead on a latent representation z , where an encoder maps the image I into a latent z , and a decoder maps the latent representation back into an image. The model can additionally be made conditional on some text y , by providing an embedding e = τ θ ( y ) using a text encoder τ θ to the denoiser:

The denoiser for a text-conditional LDM is implemented by a transformer architecture [59, 30] involving a combination of self-attention and cross-attention layers.

3.2 Optimizing for correspondences

In what follows, we detail how we compute attention masks given an image/embedding pair, how to optimize for an embedding that activates a desired position in the source image, and how to employ this optimized embedding to identify the corresponding point in the target image.

Attention masks . Given an image I , let z ( t ) be its latent representation within the diffusion model at the t -th diffusion step. We first compute query Q l = Φ l ( z ( t =8)) 1 , and key K l = Ψ l ( e ) , where Φ l ( · ) , and Ψ l ( · ) are the l -th linear layers of the U-Net [72] that denoises in the latent space. The cross-attention at these layers are then defined as:

where the attention map M ′ ∈ R C × ( h × w ) × P , and P , C , h , and w respectively represent the number of tokens, the number of attention heads in the transformer layer, the height, and width of the image

(a) Source image

(b) Layer 7

(c) Layer 8

(d) Layer 9

(e) Layer 10

(f) Average

(g) Target image

(h) Layer 7

(i) Layer 8

(j) Layer 9

(k) Layer 10

(l) Average

at the particular layer in the U-Net. Here, Q ∈ R ( h × w ) × d l , and K ∈ R P × d l where d l represents the dimensionality of this layer, and softmax denotes the softmax operation along the P dimension.

As mentioned earlier in Section 1, and observed in [65], different layers of the U-Net exhibit different 'level' of semantic knowledge; see Figure 4. Thus, to take advantage of the different characteristics of each layer, we average along both the channel axis and across a subset of U-Net layers M ′′ ∈ R ( h × w ) × P = average c =1 ...C,l =7 .. 10 ( M ′ l ) . Note that the size of the attention maps M ′ l differ according to each layer, hence we utilize bilinear interpolation when averaging. Hence, with M ( u ; e , I ) we denote indexing a pixel u of the attention map M ′′ [1] ∈ R ( h × w ) via bilinear interpolation, where [1] extracts the first of the P available attention maps 2 , that is, we use the first token of the embedding to represent our query. 3 Examples of these attention maps for embeddings e derived from natural text are visualized in Figure 2.

Optimization . Given a source image I i and a query pixel location u i ∈ [0 , 1] 2 , we are interested in finding the corresponding pixel u j in the target image I j . We emulate the source attention map for the query as a Gaussian of standard deviation σ centered at the query location u i :

The Gaussian map represents the desired region of focus of the attention mechanism. We then optimize for the (text) embedding e that reproduces the desired attention map as:

Inference . With the optimized embedding e ∗ , we can then identify the target point (the point u j in I j most semantically similar to u i in I i ) by computing the attention map for the target image, and finding the spatial location that maximizes it. We write:

Successful

Mixed

Failure

Spair-71k

CUB-200

PF-Willow

3.3 Regularizations

As discussed in Section 1, optimizing text embeddings on a single image makes it susceptible to overfitting. Furthermore, the process of finding embeddings via optimization, textual inversion, has recently been shown to be sensitive to initialization [35, 36]. To overcome these issues, we apply various forms of regularization. Note that while we describe these one at a time, these two augmentations are simultaneously enabled.

Averaging across crops . Let C c ( I ) be a cropping operation with parameters c , and U c ( I ) the operation placing the crop back to its original location; that is, C c ( U c ( x c )) = x c for some crop x c . For the c we reduce the image dimensions to 93% (found via hyperparameter sweep) and sample a uniformly random translation within the image - we will denote this sampling as c ∼ D . We augment the optimization in Equation 5 by averaging across cropping augmentations as:

similarly, at inference time, we average the attention masks generated by different crops:

Averaging across optimization rounds . We empirically found that doing multiple rounds of optimization is one of the keys to obtaining good performance; see Figure 9. To detail this process, let us abstract the optimization in Equation 7 as e ∗ = O (¯ e , I i ) , where ¯ e is its initialization. We then average the attention masks induced by multiple optimization runs as:

4 Results

We evaluate semantic correspondence search on three standard benchmarks: SPair-71k [14] is the largest standard dataset for evaluating semantic correspondences composed of 70 , 958 image pairs of 18 different classes. Since we do not perform any training, we only use the 12 , 234 correspondences of the test set for evaluation; PF-Willow [13] comprises four classes - wine bottle, duck, motorcycle,

and car - with 900 correspondences in the test set; CUB-200 [37] includes 200 different classes of birds. Following ASIC [46] we select the first three classes, yielding a total of 1 , 248 correspondences in the test set.

Metrics . We measure the performance of each method via the standard metric [13, 14, 23] of Percentage of Correct Keypoints (PCK) under selected thresholds - we report how many semantic correspondence estimates are within the error threshold from human-annotation. Following the standard protocol [13, 14, 23], for SPair-71k and PF-Willow thresholds are determined with respect to the bounding boxes of the object of interest, whereas for CUB-200, the thresholds are set relative to the overall image dimensions. We report results for thresholds 0 . 05 and 0 . 1 , as annotations themselves are not pixel-perfect due to the nature of semantic correspondences i.e. , there is no exact geometric correspondence.

Implementation details . We use Stable Diffusion version 1.4 [30]. In our ablations, we find that more augmentation help, but with the caveat of (linearly) requiring more computation. Hence, on CUB-200 and PF-Willow datasets we use 10 optimization rounds for the embeddings and 30 random crops for the inference. For the larger SPair-71k dataset we use less 5 embeddings and 20 crops. We choose our hyperparameters based on the validation subset of SPair-71k and PCK @0 . 05 via a fully-randomized search and applied them to all datasets. We use the Adam [73] optimizer to find the prompts. For detailed hyperparameter setup see Supplementary Material.

Qualitative highlights . We show qualitative examples of the semantic correspondences estimated by our method in Figure 8. Interestingly, our method, even when it disagrees with the human annotation, provides results that can arguably be interpreted as plausible. For example, as shown in the wine bottle example at the bottom-right of Figure 8, source points occluded by the wine glasses are correctly mapped to another wine glass in the target image, which disagrees with the human annotator's label which points to the wine bottle.

Quantitative results . We report the performance of our method in Table 1. We compare our method against weakly-supervised baselines [21, 52, 46], as well as baselines from ASIC [46] that are based on general-purpose deep features - using VGG [24] features with Moving Least Squares (MLS)[56] and DINO [25] features with Moving Least Squares (MLS) or Nearest Neighbor (NN) matching.

Our method outperforms all compared weakly supervised baselines. Note that the performance gap in terms of average PCK @0 . 1 compared to the second best method, ASIC [46], is large - 20.9% relative. Note also that in the case of the PF-Willow dataset, our method is on par with the current strongly supervised state-of-the-art. Even in the case of the SPair-71k dataset, our results are comparable to a very recent method VAT [15] besides the few problematic classes - bottle, bus, plant, train, tv. These problematic classes are typically those that exhibit strong symmetry, which we do not explicitly deal with. For detailed per-class results, see our Supplementary Material. Considering that our method is

PCK@0.05

50

45

40

35

30

25

7

8

9

10

Layer within the network

(a) Individual layer performance

1

2

3

4

5

6

7

8

9

10

52

50

48

46

44

Number of embeddings optimized

1

6

11

16

21

26

Number of crops

49

50

51

52

53

PCK@0.05

(c) #crops vs performance

fully unsupervised when it comes to the task of semantic correspondences, these results are quite surprising.

Ablations . To ablate our model, we use the PF-Willow [13] dataset and report PCK @0 . 05 . In Figure 9a, we show that using individual layers leads to significantly worse results than using multiple layers together; this demonstrates the importance of capturing semantic knowledge across multiple receptive fields. In Figure 9b, we show that using multiple optimized embeddings significantly boosts performance, and in Figure 9c, we see how using more crops further leads to improvements during inference. Finally, besides crops during inference, if we disable random crops during embedding optimization PCK @0 . 05 drops to 45 . 5 (vs. 53 . 0 in our full model).

Beyond semantic correspondences of the same class. . We further experiment with correspondences across different classes e.g. , correlating different animals. As shown in Figure 10 in many cases our method provides correspondences that are arguably correct even for objects that are of different classes. This includes being from similar classes (sheep and dog) to more different classes (bird and airplane), and to wildly different classes (chair and horse).

5 Conclusions

We have demonstrated the remarkable potential of leveraging diffusion models, specifically Stable Diffusion [30], for the task of semantic correspondence estimation. We have shown that by simply optimizing and finding the embeddings for a location of interest, one can find its semantically corresponding locations in other images, although the diffusion model was never trained for this task. We further introduced design choices that significantly impact the accuracy of the estimated correspondences. Ultimately, our approach significantly outperforms existing weakly supervised methods on SPair-71k [14], PF-Willow [13], and CUB-200 datasets [37] (20.9% relative for SPair-71k), and even achieves performance on par with strongly supervised methods in PF-Willow dataset [13].

Limitations and future work . This study highlights the emergent properties of training large models on vast amounts of data, revealing that a model primarily designed for image generation can also be utilized for other tasks, specifically semantic correspondences. The success of our method

PCK@0.05

underscores the importance of exploring the hidden capabilities of these large text-to-image generative models and suggests that there may be other novel applications and tasks that can benefit from the vast knowledge encoded within these models. For example, as an immediate application, our method can be used to scale up training of 3D generative models such as FigNeRF [49] with images from the web without human supervision.

We note that our method does have its limitations. Many of the failure modes include dealing with symmetric objects. An explicit way to deal with these cases with more refined techniques to extract correspondences may help solve this problem. It also requires significant compute. On an NVIDIA RTX 3090 GPU, finding a single prompt for a single keypoint takes 30 seconds. We suspect that one could potentially train a semantic correspondence network based on our estimates to achieve both scalability in terms of training data and fast inference time.

Acknowledgments and Disclosure of Funding

This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google, Digital Research Alliance of Canada, and by Advanced Research Computing at the University of British Columbia.

References

B.B. Hansen and B.S. Morse. Proceedings of the ieee conf. on computer vision and pattern recognition. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 1999.

Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape context: A new descriptor for shape matching and object recognition. Advances in Neural Information Processing Systems , 13, 2000.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition , 2016.

David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision , 2004.

Alex Fisher, Ricardo Cannizzaro, Madeleine Cochrane, Chatura Nagahawatte, and Jennifer L Palmer. Colmap: A memory-efficient occupancy grid mapping framework. Robotics and Autonomous Systems , 2021.

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision , 2016.

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2018.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2020.

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2018.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision , 2020.

Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2010.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 2021.

Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. In arXiv Preprint , 2017.

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv Preprint , 2019.

Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation. In European Conference on Computer Vision , 2022.

Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: Correspondence Transformer for Matching Across Images. In International Conference on Computer Vision , 2021.

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detectorfree local feature matching with transformers. Conference on Computer Vision and Pattern Recognition , 2021.

Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Conference on Computer Vision and Pattern Recognition , 2018.

Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. International Journal of Computer Vision , 2021.

Xin Li, Deng-Ping Fan, Fan Yang, Ao Luo, Hong Cheng, and Zicheng Liu. Probabilistic model distillation for semantic correspondence. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2021.

Jiwon Kim, Kwangrok Ryoo, Junyoung Seo, Gyuseong Lee, Daehwan Kim, Hansang Cho, and Seungryong Kim. Semi-Supervised Learning of Semantic Correspondence with Pseudo-Labels. In Conference on Computer Vision and Pattern Recognition , 2022.

Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. arXiv Preprint , 2023.

Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR) , 2015.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv Preprint , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv Preprint , 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv Preprint , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , 2020.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2022.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv Preprint , 2022.

Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion. arXiv Preprint , 2023.

Anton Voronov, Mikhail Khoroshikh, Artem Babenko, and Max Ryabinin. Is This Loss Informative? Speeding Up Textual Inversion with Deterministic Objective Evaluation. arXiv Preprint , 2023.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical report, California Institute of Technology, 2011.

Bruce D Lucas and Takeo Kanade. An Iterative Image Registration Technique With an Application to Stereo Vision. In International Joint Conference on Artificial Intelligence , 1981.

Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Rick Szeliski. Building rome in a day. Communications of the ACM , 2011.

Jan-Michael Frahm, Pierre Fite-Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, and Marc Pollefeys. Building rome on a cloudless day. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, European Conference on Computer Vision , 2010.

Ondrej Chum, Tomas Werner, and Jiri Matas. Two-View Geometry Estimation Unaffected by a Dominant Plane. In Conference on Computer Vision and Pattern Recognition , 2005.

Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to Find Good Correspondences. In Conference on Computer Vision and Pattern Recognition , 2018.

Weiwei Sun, Wei Jiang, Andrea Tagliasacchi, Eduard Trulls, and Kwang Moo Yi. Attentive context normalization for robust permutation-equivariant learning. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2020.

Prune Truong, Martin Danelljan, and Radu Timofte. GLU-Net: Global-local universal network for dense flow and correspondences. In Conference on Computer Vision and Pattern Recognition , 2020.

Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. GOCor: Bringing globally optimized correspondence volumes into your neural network. In Advances in Neural Information Processing Systems , 2020.

Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. arXiv Preprint , 2023.

Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision , 2020.

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Conference on Computer Vision and Pattern Recognition , 2019.

Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown. Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling. In International Conference on 3D Vision , 2021.

Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition , 2016.

Gopal Sharma, Kangxue Yin, Subhransu Maji, Evangelos Kalogerakis, Or Litany, and Sanja Fidler. Mvdecor: Multi-view dense correspondence learning for fine-grained 3d segmentation. In European Conference on Computer Vision , 2022.

Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. Pyramidal semantic correspondence networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech, and Time Series . 1998.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision , 2012.

Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics , 2018.

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv Preprint , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , 2021.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 2021.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation. In Conference on Computer Vision and Pattern Recognition , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv Preprint , 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv Preprint , 2022.

Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-Efficient Semantic Segmentation with Diffusion Models. In International Conference on Learning Representations , 2022.

Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. In International Conference on Robotics and Automation , 2022.

Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems , 2023.

Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems , 2023.

Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems , 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv Preprint , 2014.

Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamarneh. Slime: Segment like me. arXiv Preprint , 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Conference on Medical Image Computing and Computer Assisted Intervention , 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv Preprint , 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach to establish semantic correspondences, ie correspondences between different instances of the same object, using a pre-trained stable diffusion models, and without any task-specific finetuning. In particular, they leverage the fact that intermediate attention layers of unet diffusion model respond to the semantic of the text prompt. For a source image and a particular query pixel location for which a match should be computed, they optimize the prompt embedding that leads to a cross-attention map with a high activation at the query location. Once the prompt is recovered, they can apply the reverse attention process on a target image conditioned on the optimized prompt. The intermediate cross-attention layers of the diffusion model highlight the matching location. They also introduce technical ‘tricks’ to make it work, for example averaging over multiple crops of an image or starting from different random embeddings. This process must be repeated for each query pixel location.\n\nSTRENGTHS:\n- The insight that stable diffusion models inherently learn semantic correspondences without task-specific training is interesting. How to extract and use such knowledge has not been explored in the past. \n\n- The results are convincing - the approach obtains near state-of-the-art in semantic matching without any task-specific knowledge or funetuning. \n\n- The paper reads well overall\n\nWEAKNESSES:\n1) Missing comparisons and citations in results: PWarpC [Truong et al. CVPR 2022]: \n- the paper only includes comparisons to two weakly-supervised semantic matching approaches, with relatively weak results, which allows the authors to claim that their approach is 20% better than the state-of-the-art in weakly-supervised semantic matching. \n- The authors did not include results of PWarpC, a weakly-supervised approach for semantic matching presented at CVPR 2022. The results of PWarpC significantly outperform the baseline results of this paper for SPair-71K (38.0 on SPair-71K for PWarpC-NC-Net), making the above claim invalid. \n- on PF-Willow: to double check since the two papers use different metrics but PWarpC (weakly-supervised) outperforms CATs, which obtains better metrics than the proposed approach. It might therefore outperform the proposed approach. \n- please include this comparisons and revise the claims/analysis accordingly\n\n2) while the paper presents an interesting analysis, the proposed approach is currently very impractical and almost unusable, since as the authors say, it takes 30s to generate a single match. I believe it would be interesting to have even a small experiment training a semantic matching model based on pseudo ground-truth matches obtained from the diffusion model.\n\nQUESTIONS:\nIn addition to the weakness above:\n\nA) What is the resolution of the extracted feature/attention maps and at what resolution is the matching done? for example are the features interpolated to higer resolution before computing the cross-attention weights? \nIs a similar resolution used in competitor works like DINO? \nThis is important because the resolution of the feature maps is of crucial importance for obtaining fine-grained/accurate correspondences. It has been the limiting factor of many previous approaches that needed to compute all-to-all correspondences, because of memory constraint. Just using a higher resolution feature map for the cross-attention would lead to an improvement in results, not necessarily related to the features themselves. \n\n\nB) L.230 the authors mention it takes ‘30s’ to optimize the prompt, does it include only finding the prompt? or finding the match? A real run-time analysis of time versus performance for different settings would be interesting here (depending on how many crops are used, how many initializations ect).\n\nC) clarification needed: \nC.1) L.178: M′ has dimension C×(h×w)×P, where is the channel coming from? According to equation, M’ is computed with the standard attention mechanism, which is just a matrix multiplication followed by a softmax. As a result, the dimensions should be (hxw)xP. \n\nC.2) L.185, 187, it would be helpful to specify the shape of M′′ and M, to better follow the different steps.\nWhat is the dimension of P for the learnt embedding?\n\nC.3) L213, is it equation 5 or equation 7? The authors say they average the optimization over multiple crops and over multiple initialization of the embeddings. Is this happening at the same time, i.e. for different initialization, the optimization is done over multiple crops? If so, then it should be equation (7). \n\nC.4) in Tab1, what is meant by ‘weakly-supervised (test-time optimization)’? DINO with NN is for me unsupervised since it only requires single images for the contrastive learning part. \n\n\nsome typos:\nL.207 hyperparemter\nL.53-54 Note that while finding actual prompt thatcorrespond to words is a discrete problem, hence difficult to solve. -> remove while\n\nLIMITATIONS:\nthey have adequately addressed the limitations an negative impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-13"}

comments: SUMMARY:
This paper proposes a model that finds semantic correspondence between a pair of images, with a pretrained diffusion model. By optimizing the prompt embeddings, such correspondence can be read out via attention maps in the UNet. Empirical study shows the good performance of this proposed model over exising baselines.

STRENGTHS:
1. The proposed method has a simple form but is effective. 
2. The presentation is clear and easy to follow.
2. The qualitative results are impressive, especially the results between different classes.
3. Some quantitative results are significantly better than baselines, or comparable to supervised results.

WEAKNESSES:
1. More information should be provided for the optimization set up and implementation, e.g. the optimization algorithm for Eq (5) and Eq (9).
2. The complexity and efficiency are needed for better assessment of the algorithm, especially in comparison to baselines.

QUESTIONS:
1. I'm wondering how would the trained prompt compare to text embeddings, e.g. how would a prompt optimized from a bird's eye compare to the text embedding of "a bird's left eye"?
2. One textual baseline that doesn't appear in the experiments is the attention map of an actual text prompt, i.e. would the prompt of "a bird's left eye" have its highest attention to the eye's location?

LIMITATIONS:
The authors has addressed valid limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses unsupervised semantic correspondence and proposes to leverage the semantic knowledge within fashionable text-to-image diffusion model to accomplish this task. The method optimizes a learnable text prompt to maximize source image attention value of query location. Then, the optimized text prompt is used to find corresponding location of target image by finding the max value of target image attention value. Related experiments show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.The idea of leveraging semantic knowledge of text-to-image diffusion model sounds interesting. This paper builds a new type of unsupervised semantic correspondence method based on text-to-image diffusion model.\n\n2.The designed method is effective and achieve promising correspondence results on common datasets.\n\n3.The design choices are reasonable, e.g., random crop for preventing overfitting when finding the text prompts.\n\nWEAKNESSES:\n1.The usage of text-to-image (t2i) diffusion model is not well-motivated. The proposed method mainly leverages the cross-modal ability of t2i diffusion model but underuses the generative ability of t2i diffusion model. The role of t2i diffusion model in proposed method can be replaced by other cross-modal pretraining model, e.g., CLIP, BLIP, GLIP. Please claim the unique significance for adopting t2i diffusion model rather than other cross-modal pretraining model.\n\n2.The proposed method has to execute dozens of denoising steps for each query location. As diffusion model is notorious for its inference speed and cost, the total overhead is expensive. Are there any mitigation measures? For example, grouping the query locations or modifying the diffusion model architecture?\n\nQUESTIONS:\nPlease see “Weakness” part.\n\nLIMITATIONS:\nThe authors objectively discuss the limitations of the proposed method, e.g., the slow inference time caused by the introduction of diffusion model. This paper doesn’t have negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper explores unsupervised semantic correspondence tasks with stable diffusion model. Specifically, the authors proposed to first optimize the prompt embeddings of stable diffusion model, to maximize attention on the region of interest, then the optimized prompts are used for semantic correspondence.
Experimental results on PF-Willow, CUB-200 and SPair-71k datasets show that proposed method significantly outperforms weakly / unsupervised methods.

STRENGTHS:
- This paper shows that one could explore diffusion models for semantic correspondence tasks.


- the writing is clear and easy to follow.

- The experiments show that the proposed method could even outperform supervised method on PF-Willow.

WEAKNESSES:
1. How important is the optimization for prompts? Could the authors show results if the prompts are a vanilla sentence, e.g. "an image of a cat"?
2. Concurrent works: there are multiple papers presenting correspondence ability of diffusion models:

[1*] Luo, Grace, et al. "Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence." arXiv preprint arXiv:2305.14334 (2023).

[2*] Tang, Luming, et al. "Emergent Correspondence from Image Diffusion." arXiv preprint arXiv:2306.03881 (2023).

[3*] Zhang, Junyi, et al. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv preprint arXiv:2305.15347 (2023).

Could the authors explain the differences if possible?

QUESTIONS:
please refer them to the weakness session

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for unsupervised semantic correspondence by employing a pre-trained Stable Diffusion generator. The basic idea is that since text-to-image diffusion models are capable of generating realistic images, they must understand the semantics of the objects they generate and thus be capable of finding semantic similarities in images.\n\nTechnically, the approach feeds a real image (or, better, its latent vector) into a diffusion pipeline, recover attention matrices between pixel values and the textual prompt, and optimizes the textual prompt embedding so that the cross-attention focuses on a query pixel location. Once optimization is over, the same text embedding can be employed to find semantic correspondences in a target image. Cross-attention matrices are averaged across channels and a subset of U-Net layers, using bilinear interpolation. Further, the target attention map is simulated with a Gaussian distribution centered on the query location, and a L2 loss is employed for optimization.\n\nAuthors also propose two regularization strategies to prevent over-fitting during optimization, i.e. averaging across crops of the query image, and averaging across multiple rounds of optimizations/initialization choices.\n\nExperimental results are conducted over three standard benchmarks for the task, i.e. SPair-71k, PF-Willow and CUB-200. The proposed approach achieves superior results when compared to unsupervised or weakly-supervised approaches, and is also competitive when compared to models employing strong supervision.\n\nSTRENGTHS:\n- The main idea of the paper (extracting/exploiting the semantics embedded in a pre-trained a SD network) is novel and interesting. It is in-line with some recent pre-prints working on related tasks (e.g. image segmentation), but to my knowledge the proposed approach is novel wrt published papers.\n- The technique for extracting semantics is simple and effective, and also are the two regularization techniques.\n- Experimental results are solid and confirm the appropriateness of the approach.\n- The paper is well written, clear and a pleasure to read. The supplementary material is also interesting and comprehensive.\n\nWEAKNESSES:\n- While the idea behind the paper is very general and could have been applied to a variety of tasks involving semantics (e.g. image segmentation, panoptic segmentation, classification), the paper only focuses on finding semantic correspondences. This limits the overall impact of the work to a single task of Computer Vision. I wonder if authors have tried to extend to other tasks, and how difficult that would be.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors have successfully discussed the limitations of the work in the appropriate section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-30"}