Emergent Correspondence from Image Diffusion
Authors: Luming Tang
Menglin Jia
Qianqian Wang
Cheng Perng Phoo
Bharath Hariharan
Venue: NeurIPS 2023 poster (2023)
Keywords: Correspondence
Diffusion Model
TL;DR: Visual correspondences emerge from image diffusion models without any explicit supervision.

Abstract: Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io.

Emergent Correspondence from Image Diffusion
Luming Tang"
Menglin Jia'
Qianqian Wang"
Cheng Perng Phoo
Bharath Hariharan
Comell University
Abstract
Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively the challenging SPair-7IK Ik benchmark. It even outperforms the state-of-the-art supervised methods on out 18 categories while remaining on par for the overall performance. Project page: https: http:::/dfffffffoo eatures. github.
Introduction
Drawing correspondences between images a critical primitive in 3D reconstruction [73], object tracking [22, 90], video segmentation [92]. image and video editing [102, 58, 98]. This problem of drawing correspondence is easy for humans: we can match object parts not only across different viewpoints, articulations and lighting changes, but
across drastically different categories (e.g.. between cats and horses) or different modalities (e.g.. between photos and cartoons). Yet, we rarely ever get explicit correspondence labels for training. The question is, can computer vision systems similarly learn accurate correspondences without any labeled data at all?
There is indeed some evidence that contrastive self-supervised learning techniques produce good correspondences as a side product of learning on unlabeled data [10, 28J. However, in this paper, look to class of self-supervised models that has been attracting attention: diffusion-based generative models [32, 79]. While diffusion models are primarily models for image synthesis, a key observation is that these models produce good results for image-to-image translation [53, 85] and image editing [8, 80]. For instance, they can convert a dog to a cat without changing its pose context [61]. It would appear that perform such editing, the model must implicitly reason about correspondence between the two categories (e.g., the model needs to know where the dog's s eye is in order to replace with the cat's eye). We therefore ask, do image diffusion models learn correspondences?
We answer the question in the affirmative by construction: we provide simple way of extracting correspondences on real images using pre-trained diffusion models. These diffusion models [41] have the core a U-Net [71, 17, 70] that takes noisy images input and produces clean images as output. As such they already extract features from the input image that can be used for correspondence. Unfortunately, the U-Net is trained de-noise, and has been trained noisy images. Our strategy
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Source Point
DIFT Predicted Target Points
cross-instance
cross-calegory
cross-domain
Figure Given red source point in an image (far left), we would like to develop model that automatically finds the corresponding point in the images on the right. Without any fine-tuning or correspondence supervision, our proposed diffusion features (DIFT) could establish semantic correspondence across instances, categories and even domains, e.g., from duck to penguin, from photo to an oil-painting. More results are Figs. and 16 of Appendix
for handling this issue is simple but effective: we add noise to the input image (thus simulating the forward diffusion process) before passing it into the U-Net to extract feature maps. We call these feature maps (and through a slight abuse of notation, our approach) DIffusion FeaTures (DIFT). DIFT can then be used to find matching pixel locations the two images by doing simple nearest neighbor lookup using cosine distance. We find the resulting correspondences are surprisingly robust and accurate (Fig. 1), even across multiple categories and image modalities.
We evaluate DIFT with two different types of diffusion models, on three groups of visual cor- respondence tasks including semantic correspondence, geometric correspondence, and temporal correspondence. We compare DIFT with other baselines, including task-specific methods, and other self-supervised models trained with similar datasets and similar amount of supervision (DINO [10] and OpenCLIP [36]). Although simple, DIFT demonstrates strong performance on all tasks without any additional fine-tuning or supervision, outperforms both weakly-supervised methods and other self-supervised features, and even remains par with the state-of-the-ard supervised methods on semantic correspondence.
2
Related Work
Visual Correspondence. Establishing visual correspondences between different images is crucial
for various computer vision tasks such as Seuucture-from-Moiiee / 3D reconstruction [2. 73, 60, 74], object tracking [22, 97], image recognition [63, 81, 9] and segmentation [50, 47, 72, 28). Traditionally, correspondences are established using hand-designed features, such as SIFT [51] and SURF [6]. With the advent of deep learning, methods that leamn find correspondences in a supervised-learnin regime have shown promising results [46, 14, 42, 35]. However, these approaches are difficult to scale due to the reliance on ground-truth correspondence annotations. To overcome difficulties in collecting a large number of image pairs with annotated correspondences, recent works have started looking into how to build visual correspondence models with weak supervision [91] or self-supervision [92, 37]. Meanwhile, recent works on self superviedd representation learning [10] has yielded strong per-pixel features that could be used to identify visual correspondence [84, 3, 10, 28]. In particular, recent work has also found that the internal representation of Generative Adversarial Networks (GAN) [23] could be used for identifying visual correspondence [99, 62, 57] within certain image categories. Our work shares similar spirits with these works: we show that diffusion models could generate features that are useful for identifying visual correspondence on general images. addition, we show that features generated at different timesteps and different layers of the de-noising process

---

encode different information that could be used for determining correspondences needed for different downstream tasks.
Diffusion Model [78, 32, 79, 41] is a powerful family of generative models. Ablated Diffusion Model [17] first showed that diffusion could surpass GAN's ss image generation quality Ima- geNet [15]. Subsequently, the introduction of classifier-free guidance [33] and latent diffusion model [70] made it scale up to billions of text-image pairs [75], leading to the popular open-sourced text-to-image diffusion model, i.e., Stable Diffusion. With its superior generation ability, recently people also start investigating the internal representation of diffusion models. For example, previous works [85, 31] found that the intermediate-layer features and attention maps of diffusion models are crucial for controllable generations; other works [5, 94, 101] explored adapting pre-trained diffusion models for various downstream visual recognition tasks. Different from these works, we are the first directly evaluate the efficacy of features inherent pre-trained diffusion models on various visual correspondence tasks.
Problem Setup
Given two images 41, 1a and pixel location pi in 41, we are interested in finding its corresponding pixel location P2 in Iz. Relationships between P1 and P2 could be semantic correspondence (i.e., pixels of different objects that share similar semantic meanings), geometric correspondence (i.e., pixels of the same object captured from different viewpoints), or temporal correspondence (i.e., pixels of the same object in a video that may deform over time).
The most straightforward approach to obtaining pixel correspondences is to first extract dense image features in both images and then match them. Specifically, given feature map F; for image 7,, can extract feature vector (p) for pixel location p through bilinear interpolation. Then given pixel p1 in image IT, we can obtain the corresponding pixel in image 12 as:
P2 = arg min d (FF())))
(1)
where d is distance metric and cosine distance by default in this work.
4 Diffusion Features (DIFT)
In this section, we first review what diffusion models are and then explain how we extract dense features on real images using pre-trained diffusion models.
4.1 Image Diffusion Model
Diffusion models [32, 79] are generative models that aim to transform Normal distribution to an arbitrary data distribution. In our case, we use image diffusion models, thus the data distribution and the Gaussian prior are both over the space of 2D images.
During training, Gaussian noise of different magnitudes is added to clean data points to obtain noisy data points. This is typically thought of as a "diffusion" process, where the starting point of the diffusion Bg is clean image from the training dataset and F, is a noisy image obtained by "mixing" To with noise:
27 Or + (vl a
where I) is the randomly samolee noise, and t [0, indexes "time" in the diffusion process with larger time steps involving more noise. The amount of noise is determined by {a+)? which is pre-defined noise schedule. We call this the diffusion forward process.
neural network fo is trained to take 2; and time step t as input and predict the input noise c. For image generation, fo is usually parametrized as U-Net [71, 17, 70]. Once trained, fo can be used to "reverse" the diffusion process. Starting from pure noise xT sampled from a Normal distribution, fo can be iteratively used to estimate noise from the noisy data Z, and remove this noise to get cleaner data 1, eventually leading to a sample zg from the original data distribution. We call this the diffusion backward process.
3

---

Source Point
Predicsed Target Point
Source Point
Predictid Tirget Point
Figure 2: Given Stable Diffusion generated image, we extract its intermediate layer activations at a certain time step t during its backward process, and use them as the feature map to predict the corresponding points. Although simple, this method produces correct correspondences images already not only within category, but also cross cotserr even in cross- cross-domain situations, e.g., from photo an oil painting.
generated
4.2 Extracting Diffusion Features on Real Images
We hypothesize that diffusion models learn correspondence implicitly [85, 61] in Sec. 1, but how we extract this correspondence? Consider first generated images, where have access the complete internal state of the network throughout the entire backward process. Given a generated image from Stable Diffusion [70], , extract the feature maps of its intermediate layers specific time step during the backward process, which we then utilize to establish correspondences between two different generated images as described in Sec. As illustrated in Fig. this straightforward approach allows us to find correct correspondences between generated images, even when they belong different categories domains.
Replicating this approach for real images is challenging because of the fact that the real image itself does not belong to the training distribution of the U-Net (which trained noisy images), and we do not have access to the intermediate noisy images that would have been produced during the generation of this image. Fortunately, found simple approximation using the forward diffusion process to be effective enough. Specifically, we first add noise of time step f to the real image (Eq. (2)) move it to the x, distribution, and then feed it to network fe together with to extract the intermediate layer activations as our DIffusion FeaTures, namely DIFT. As shown in Figs. and 3, this approach yields surprisingly good correspondences for real images.
Moving forward, a crucial consideration is the selection of the time step and the network layer from which extract features. Intuitively we find that larger t and earlier network layer tend to yield more semantically-awy aware features, while smaller and a later layer focus more on low-level details. The optimal choices of and layer depend the specific correspondence task at hand, as different tasks may require varying trade-offs between semantic and low-level features. For example, semantic correspondence likely benefits from more semantic-level features, whereas geometric correspondence between two views of the same instance may perform well with low-level features. We therefore use 2D grid search to determine these two syper-parametees for each correspondence task. For comprehensive list of the hyper-paramett values used in this paper, please refer to Appendix C.
Lastly, to enhance the stability of the representation in the presence of random noise added to the input image, we extract features from multiple noisy versions with different samples of noise, and average them to form the final representation.
5 Semantic Correspondence
In this section, we investigate how to use DIFT to identify pixels that share similar semantic meanings across images, e.g., the eyes of two different cats in two different images.
5.1 Model Variants and Baselines
We extract DIFT from two commonly used, open-sourced image diffusion models: Stable Diffusion 2-1 (SD) [70] and Ablated Diffusion Model (ADM) [17]. SD is trained on the LAION [75] whereas ADM is trained on ImageNet [15] without labels. We call these two features DIFT aa and DIFT respectively.
To separate the impact of training data on the performance of DIFT, we also evaluate two other commonly used self-supervised features baselines that share basically the same training data:

---

Source Image
DINO
OpenCLIP
DIFT adm
DIFT
Target Image
Figure 3: Visualization of semantic correspondence prediction on SPair-71k using different features. The leftmost image is the source image with set of keypoints; the rightmost image contains the ground-trute correspondence for target image whereas any images in between contain keypoints found using feature matching with various features. Different colors indicate different keypoints. We use circles indicate correctly-predicte incorrect matches. DIFT is able to establish correct correspondences under clustered scenes (row 3), viewpoint changes (row 2 and 4), and occlusions (row 5). See Fig. 17 in Appendix E for more results.
points under the threshold alabo 0.1 and crosses for
OpenCLIP [36] with ViT-H/14 [18] trained on LAION, well DINO [10] with ViT-B/8 trained on ImageNet [15] without labels. Note that for both DIFT and other self-supervised features, we do not fine-tune or re-train the models with any additional data supervision.
5.2 Benchmark Evaluation
Datasets. We conduct evaluation on three popular benchmarks: SPair-7Ik [55], PF-WILLOW [27] and CUB-200-2011 [89]. SPair-71k is the most challenging semantic correspondence dataset, containing diverse variations viewpoint and scale with 12,234 image pairs on 18 categories for testing. PF-Willow is subset of PASCAL VOC dataset [20] with 900 image pairs for testing. For CUB, following [58], we evaluate 14 different splits of CUB (each containing 25 images) and report the average performance across all splits.
Evaluation Metric. Following prior work, report the percentage of correct keypoints (PCK).
The predicted keypoint is considered to be correct if they lie within a max(h, w) pixels from the ground-truth keypoint for a [0, 1]. where and w are the height and width of either the image (Aimg) or the bounding box (Atsox)) To find a suitable time step and layer feature to use for DIFT and other self- supervised features, we grid search the hyper- parameters using SPair-71k and use the same
settings for PF-WILLOW and CUB.
We observed inconsistencies in PCK measurements across prior literature'. Some works [35, 42, 14] use the total number of correctly-predictedd points in the whole dataset (or each category split) divided
ScN GANgealing [62]'s evaluation code snippets, which calculate PCK per inage PCK per point respectively.
5

---

Table 1: PCK(amerr = 0.1) per image on SPair-7lk. All the DIFT results have gray background for easy lookups. Methods are grouped into 3 groups: (a) fully supervised with correspondence
annotations, (b) weakly supervised with in-domain image collections, (c) no supervision. Best numbers in group (a) are bolded. Among groups (b) and (c) taken together, we annotate best and second-best results. Without any supervision, both DIFT,d ad and DIFT adm outperform previous reekkl--uuperviedd methods and self-supervised techniques by a large margin.
Sup.
Method
SPaic-IKK Catepory
Arm
1ike
Bind
Boot
Hotlle
Buus
Car
C2
Cherr
Cow
13gg
Hors
Motre
Pereon
Piant
Sheep
Train
TV
MMNNe
55.9
57.0
35.4
50.0
28.7
34.5
36.3
77.7
MranteMMaccher (42)
59.2
730
41.2
$2.5
677
56.6
53,3
45.0
42.1
15.3
68.6
51-7
SCoeSAN [25)
$7.1
48.3
78.3
38.1
51.8
57.8
47.1
67.9
25.2
71.3
63.9
49.3
45,3
49.8
48.31
40.3
77.7
69.7
55.3
NCNa [60]
17.9
12.2
32.4
11.7
29.0
19.9
16.1
39.2
9.9
23.9
18.8
15.7
17.4
15.9
14.8
9.6
24.3
31.1
20.1
CNNNGGO (67)
23-4
10-7
40.2
16.3
16.4
21-3
26-0
12.7
12.7
27.6
22.8
13.7
20.9
21.0
12.5
10.3
90.8
18.1
20/6
WeatAlion [65]
41.9
15.1
12.5
22.2
17.9
10.4
32.2
SENee (45]
26.9
45.5
14.7
38.0
13.5
21.1
16.6
15.6
32.2
25.9
PMD [18)
16.2
18.5
48.6
15.5
18.0
21.3
13.3
51.6
13.7
34.5
25.4
18.0
20.0
24.0
15.2
16.3
51.4
18.1
26.5
PSCNer [38]
28.3
17.3
45.1
15.1
37.5
30.1
21.5
47.4
14.5
32.5
26.4
17.7
24.9
24.5
19.9
16.9
34.2
27.9
27.0
Pillage (83)
$7.4
25.5
60.8
22.9
40.5
20.4
22.8
60.1
19.5
37.8
58.4
25.0
32.1
20.7
20.2
20.2
44.5
50.0
35.3
DINO (10)
64.9
34.0
10.5
16.8
93.9
41.1
32.0
96.8
96.4
10
DIPT cours)
49.7
29.3
77.5
20.3
40.9
36.1
30.5
25.5
25.7
63.7
52.8
09.3
34.1
53.3
39.3
37.3
59.5
45.4
OpcaCLIP 196)
28.4
31.5
21.1
51.8
21.7
28.6
46.3
20.7
DIFT
83.2
79.8
31.3
16.3
97.8
34.7
70.1
87.3
03.4
$1.9
46.0
67.6
59.5
52.9
Table 2: PCK(otsoz = 0.1) per point of various methods on SPair-71k. The groups and colors follow Tab. "Mean" denotes the PCK averaged over categories. Same as in Tab. 1, without any supervision, both DIFT,a sa and DIFT outperform previous weakly-supervised methods with large margin, and also outperform their contrastive-leaming counterparts by over 14 points.
Sep.
Method
SPair-TIK
Cideggsy
Acto
Bile
Bird
Bost
Botte
Bu
Cluir
Coe
Dog
House
Moux
Pesson
Flur.
Steep
Tran
Mean
NBB 20]
206
23-7
14.2
27.6
300
27.4
19.1
19.3
26.6
22.6
27.4
06
Newr PowCongeal (58)
GiNNeaaling (62]
37.5
7M
51.3
67.0
25.1
352
$1.9
57.9
2533
06.1
267
354
284
30.9
543
216
450
472
398
26.3
488
14.5
26.5
49.0
26.6
36.9
DENO [10]
55N
559
OpenC [99]
552
15.4
694
760
53.3
61
41.
58.
21.3
470
65.9
46.1
415
55.
35.6
11.7
478
218
41.
41.
INEFE (Dems)
46.2
76.0
56.9
460
59.8
593
Table 3: Comparison with state-of-the art methods on PF- WF-WWLLOWW PCK per image (left) and CUB PCK per point (right). The groups follow Tab. 1. Colors of numbers indicate the best, second-best results. All the DIFT results have gray background for better reference. DIFT sa achieves the best results without any fine-tuning or supervision with in-domain annotations or data.
PCK Ogbox
Sup.
Method
0.05
0.10
SCNet [29]
38.6
70.4
DHPF [56]
49.5
77.6
PMD [48]
75.6
Sup.
Method
PCK@auma 0.1
(a)
CHM [54]
52.7
79.4
CAT's [14]
TransforMatcher [42]
50.3
76.0
79,2
(b)
GANgealing [62]
NeuCongeal [58]
65.6
56.8
SCorSSA [35]
54.1
80.0
DINO [10]
66.4
DIFT adw (ours)
78.0
WarpC [82]
49.0
75.1
(b)
PWarpC [83]
45.0
75.9
OpenCLIP [36]
67.5
GSF [39]
49.1
78.7
DIFT,a oa (ours)
83.5
DINO [10]
30.8
51.1
(c) DIFT.dm (ours)
46.9
67.0
OpenCLIP [36]
34.4
61.3
DIFT,d (ours)
58.1
81.2
by the total number of predicted points the final PCK, while some works [62, 58, 26] first calculate a PCK value for each image and then average it across the dataset (or each category split). We denote the first metric as PCK per point and the second as PCK per image. We calculate both metrics for DIFT and self-supervised features, and compare them to methods using that metric respectively.
Quantitative Results. We report our results in Tabs. to 3. In addition to feature matching using DINO and OpenCLIP, we also report state-of-the-art fully-supervised and weakly-supervised methods in the respective tables for completeness. Across the three datasets, we observe that features learned via diffusion yield more accurate correspondences compared to features learned using contrastive approaches (DIFT,d sd vs. OpenCLIP, DIFT adm vs. DINO).

---

Source patch
Top-5 nearest neighbur cruss-calegory largel patches predicted by DIFT
Figure 4: Given image patch specified in the leftmost image (red rectangle), the top-5 nearest patches images from different categories in the SPair-71k test set. DIFT is able to find correct correspondence for different objects sharing similar semantic parts, e.g., the wheel of an airplane vs. the wheel of a bus. More results are in Fig. 18 of Appendix E.
DIFT sd to retrieve
DIFT with different time step t
SPair-71k all PCX
60
OpenCLIP
40
DINO
200
400
600
800
Time step t
Figure 5: PCK per point of DIFT, sa SPair-71k. It maintains high accuracy with wide range of t, outperforming other off-the-shelf self-supervised features.
Furthermore, even without any supervision (be explicit correspondence or in-domain data), DIFT outperforms all the weakly--uperrvied baselines on all benchmarks by large margin. It even outperforms the state-of-the-an supervised methods on PF-WILLOW, and for out of 18 categories on SPair-7Ik.
Qualitative Results. To get better understanding of DIFT's performance, visualize few correspondences on SPair-71k using various off-the-shelf features in Fig. 3. We observe that DIFT is able to identify correct correspondences under cluttered scenes, viewpoint changes, and instance-level appearance changes.
In addition to visualizing correspondence within the same categories in SPair-7Ik, also visualize the correspondence established using DIFT.a across various categories in Fig. 4. Specifically, we select image patch from random image and query the image patches with the nearest DIFT embedding in the rest of the test split but from different categories. DIFT is able to identify correct correspondence across various categories.
Sensitivity to the choice time step 4. For DIFT,a, we plot how its PCK per point varies with different choices of on SPair-71k in Fig. 5. DIFT is robust to the choice of L on semantic correspon- dence, as wide range of outperforms other off-the-shelf self-supervised features. Appendix includes more discussion on how and why does f affect the nature of correspondence.
7

---

DIFT.2
OpenCLIP
OpeCLIP DIFT
Figure 6: Edit propagation. The first column shows the source image with edits (i.e., the overlaid stickers), and the rest columns are the propagated results on new images from different instances, categories, and domains, respectively. Compared to OpenCLIP, DIFT ad propagates edits much more accurately. More results are in Fig. 20 of Appendix
5.3 Application: Edit Propagation
One application of DIFT is image editing: we can propagate edits in one image to others that share semantic correspondences. This capability demonstrated Fig. 6, where we showcase DIFT's ability to reliably propagate edits across different instances, categories, and domains, without any correspondence supervision.
To achieve this propagation, we simply compute a homography transformation between the source and target images using only matches found in the regions of the intended edits. By applying this transformation to the source image edits (e.g., an overlaid sticker), we can integrate them into the corresponding regions of the target image. Figure shows the results for both OpenCLIP and DIFT, sd using the same propagation techniques. OpenCLIP fails to compute reasonable transformation due the lack of reliable correspondences. In contrast, DIFT, ed achieves much better results, further justifying the effectiveness of DIFT in finding semantic correspondences.
6 Other Correspondence Tasks
We also evaluate DIFT on geometric correspondence and temporal correspondence. in Sec. 5, we compare DIFT to other off-the-shelf self-supervised features well as task-specific methods.
6.1 Geometric Correspondence
Intuitively, we find when small, DIFT focuses more on low-level details, which makes useful as geometric feature descriptor.
Setup. We evaluate DIFT for homography estimation the HPatches benchmark [4]. It con- tains 116 sequences, where 57 sequences have illumination changes and 59 have viewpoint changes. Following [91], extract maximum of 1,000 keypoints from each image, and use cv2. 122.finde wet.ioooooogrrphyy () to estimate the homography from mutual nearest neighbor matches. For DIFT, we use the same of keypoints detected by SuperPoint [16], as in CAPS [91].
For cvaluation, we follow the corner correctness metric used in [91]: the four corners of one image transformed into the other image using the estimated homography and are then compared with those computed using the ground -truh homography. We deem the estimation correct if the average error of the four corners is less than pixels. Note that we do this evaluation the original image resolution following [91], unlike [16].
Results. We report the accuracy comparison between DIFT and other methods in Tab. 4. Visualization of the matched points can be found in Fig. 7. Though not trained using any explicit geometry supervision, DIFT is still on par with the methods that utilize explicit geometric supervision signals

---

Table 4: Homography estimation accuracy [%] at I, 3, pixels on HPatches. Colors of numbers indicate the best, second best results. All the DIFT results have gray background for better reference. DIFT with SuperPoint keypoints achieves competitive performance.
Method
Geometric
All
Viewpoint Change
Illmiinaionn Change
Supervision
SIFT [51]
None
40.2
68.0
79.3
26.8
55.4
72.1
34.6
81.5
86.9
LF-Not [59]
34.4
62.2
73.7
16.8
43.9
60.7
53.5
81.9
87.7
SuperPoint [16]
72.7
56.1
D22-Net Net [19]
16.7
61.0
75.9
3.7
38.0
56.6
30.2
84.9
95.8
DISK [86]
Strong
40.2
70.6
23.2
51.4
91.2
ContexiDes: [52]
40.9
73.0
82.2
29.6
60.7
72.5
53.1
86.2
92.7
R2D2 [66]
40.0
74.4
84.3
26.4
60.4
73.9
34.6
89.6
95.4
w/ Super/Puiar Ap.
CAPS 191]
Weak
44.8
76.3
85.2
35.7
62.9
74.3
54.6
90.8
96.9
DINO [00]
38.9
70.0
81.7
21.4
50.7
67.1
57.7
90.8
97.3
DIFT (ours)
43.7
73.1
84.8
26.4
57.5
74.3
62.3
90.0
96.2
OpenCLIP [36]
33.3
67.2
78.0
18.6
45.0
59.6
49.2
91.2
97.7
DIFT (ouns)
45.6
73.9
56.8
92.3
98.1
Viewpoint Change
Illumination Change
Figure 7: Sparse feature matching using DIFT,d sd on HPatches after removing outliers with cv2. inddoooggaaphy() () Left are image pairs under viewpoint change, and right are ones un- der illumination change. Although never trained with correspondence labels, it works well under both challenging changes. More results are in Fig. 21 of Appendix E.
designed specifically for this task, such as correspondences obtained from Structure- fronee-ffo-----eeeeee Mottooottion [73] pipelines. This shows that not only semantic-level correspondence, but also geometric correspondence emerges from image diffusion models.
6.2 Temporal Correspondence
DIFT also demonstrates strong performance on temporal correspondence tasks, including video object segmentation and pose tracking, although never trained or fine-tuned on such video data.
Setup. We evaluate DIFT on two challenging video label propagation tasks: (1) DAVIS-2017 video instance segmentation benchmark [65]: (2) JHMDB keypoint estimation benchmark [40].
Following evaluation setups in [49, 37, 10, 95], representations are used as a similarity function: segment scenes with nearest neighbors between consecutive video frames. Note that there is no training involved this label propagation process. We report region-based similarity J and contour-based accuracy F [64] for DAVIS, and PCK for JHMDB.
experimental results, comparing DIFT with other self-supervised features (pre-)trained with or without video data. DIFT outperforms all the other self-supervised learning methods on both benchmarks, even surpassing models specifically trained on video data by significant margin. DIFT also yields the best results within the same pre- pre-training dataset.
Results. Table 5 reports

---

Table Video label propagation results on DAVIS- 2017 and JHMDB. Colors of numbers indicate the best, second-bes, results. All the DIFT results have gray background for better reference. DIFT even outperforms other seff-supeviiee learning methods specifically trained with video data.
ipre-)Irained on Videos
DAVIS Fm
JHMDB
Method
Dataset
F&FFm
Jm
P.K@0.1
PCK@0.2
InstDis [93]
66.4
63.9
80.2
MoCo [30]
SimCLR [12]
ImageNet [15]
64.4
80.8
BYOL [25]
66.5
64.0
58.8
80.9
w/ labels
X
SimSiam [13]
67.2
64.8
68.8
59.9
81.6
DINO [10]
71.4
67.9
74.9
57.2
81.2
DIFTadm (ours)
75.7
72.7
78.6
63.4
84.3
OpenCIIP [36]
60.6
64.4
41.7
71.7
LAION [75]
DIFT wn (ours)
70.0
67.4
72.5
81.8
VINCE [24]
65.2
62.5
67.8
58.8
80.4
VFS [95]
68.9
66.5
71.3
60.9
80.7
UVC [49]
Kinetic [11]
60.9
59.3
62.7
58.6
79.6
CRW [37]
67.6
64.8
70.2
58.8
80.3
Colorization [88]
34.0
34.6
32.7
45.2
69.6
CorPFlow [44]
OxUvA [87]
48.4
52.2
78.8
TymeCycle [92]
VLOG [2]]
78.1
MAST [43]
65.5
63.3
67.6
YT-VOS [96]
SFC [34]
71.2
68.3
74.0
61.9
83.0
DINO
DIFT ouiee
DINO
DIFT alds
Figure 8: Video label propagation results DAVIS-2011 Colors indicate segmentation masks for different instances. Blue rectangles show the first frames. Compared to DINO, DIFT adm produces masks with more accurate and sharper boundaries. More results in Fig. 22 of Appendix E.
We also show qualitative results in Fig. 8, presenting predictions of video instance segmentation in DAVIS, comparing DIFT with DINO. DIFT adm produces masks with clearer boundaries when single or multiple objects are presented in the scene, even attends well to objects in the presence of occlusion.
Conclusion
This paper demonstrates that correspondence emerges from image diffusion models without explicit supervision. We propose simple technique to extract this implicit knowledge as feature extractor named DIFT. Through extensive experiments, we show that although without any explicit supervision, DIFT outperforms both weakly-supervin methods and other off-the- off-the-shelf self-supervised features identifying semantic, geometric and temporal correspondences, and even remains on par with state-of-the-are supervised methods semantic correspondence. We hope our work inspires future research on how to better utilize these emergent correspondence from image diffusion, as well as rethinking diffusion models self-supervised learners.
10

---

This work was partially funded by NSF N144114 and the DARPA Learning with Less Labels program (HR001 11800044). We would like to thank Zeya Peng for her help on the edit propagation section and the project page, thank Kamal Gupta for sharing the evaluation details in the ASIC paper, thank Aaron Gokaslan, Utkarsh Mall, Jonathan Moon, Boyang Deng, and all the anonymous reviewers for valuable discussion and feedback.
Acknowledgement.
References
[1] K. Aberman, J. Liao, M. Shi, D. Lischinski, B. Chen, and D. Cohen-Or. Neural bast-buddies: Sparse cross-domain correspondence. ACM Transactions Graphics (TOG), 37(4):1-14, 2018. 6
[2] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Scitz, and R. Szeliski. Building rome in a day. Communications of the ACM, 54(10):105-122, 2011.
[3] Amir, Gandelsman, Bagon, and T. Dekel. Deep features as dense visual descriptors. arXiv
preprint arXiv:2112.05811 2(3):4 2021. 2
[4] Balntas, K. Lenc. A. Vedaldi, and K. Mikolajczyk. Hpatches: A benchmark and evaluation of handerafted and learned local descriptors. In CVPR, 2017. 8,
[5] D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko. Label-efficient semantic segmenta- tion with diffusion models. arXiv preprint 12.00:2122.03266, 2021. 3, 16
[6] H. Bay. T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. Lecture notes in computer
science, 3951:404 41744447444444444111444444444114 2006. 2
[7] A. Birhane, V. Prabhu, and Kahembwe, Multimodal datasets: misogyny, pornography, and malignant
stereotypes. arXiv preprint
2021. 15
[8] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning follow image editing instructions.
arXiv preprint arXiv:221 1.09800, 2022.
[9] K. Cao, M. Brbic, and J. Leskovec. Concept learners for few-shot learning. arXiv preprint
XXXi:2007.003755 2020. 2
[10] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties
in self-supervised vision transformers. In ICCV, pages 9650-9660, 2021. 1. 2, 5, 6, 9, 10, 17
[11] J. Carreira A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017. 10
[12] T. Chen, S. Kornhlith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMIR, 2020. 10 [13] X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, pages 15750-15758, 2021. 10
[14] S. Cho, S. Hong, S. Jeon, Y. Lee, K. Sohn, and S. Kim. Cats: Cost aggregation transformers for visual correspondence. NeurlPS, 34:901 1-9023, 2021. 2,5,
[15] J. Deng, W. Dong, R. Sncher, I..J. Li, Li, and I.. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. Ieee, 2009. 3,
[16] D. DeTone, Malisicwiez, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPRW, pages 224-236, 2018. 8, 9, 18
[17] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. Neur/PS, 34:8780-888 8794, 2021. 3,4
[18] A. Dasovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020. 5
[19] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler. D2-net: A trainable cnn for joint detection and description of local features. arXiv preprint arXiv:1905.08:: 03561, 2019. 9, 18
[20] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88:303-338, 2010.
[21] D. F. Fouhey, W.-c Kuo, A. A. Efros, and J. Malik. From lifestyle vlogs everyday interactions. In CVPR, pages 4991-50000 2018. 10
[22] S. Gao, C. Zhou, C. Ma, X. Wang, and J. Yuan. Aiatrack: Attention in attention for transformer visual tracking. In ECCV, pages 146-164, Springer, 2022. 1,2
[23] I. Goodfellow, J. Pouget-Abual M. Mirza, B. Xu. D. Warde- Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(1)) 144, 2020. 2
11

---

[24] D. Gordon, K. Ehsani, D. Fox, and A. Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.07990 2020. 10
[25] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.0777:
2020. 10
[26] K. Gupta, V. Jammain,, C. Esteves, A. Shrivastava, A. Makadia, N. Snavely, and A. Kar. Asic: Aligning
sparse in-the-wild image collections. arXiv preprint arXiv:2303. /6201, 2023. 6
[27] B. Ham, M. Cho, C. Schmid, and Ponce. Proposal flow. In CVPR, 2016. 5, 17
[28] M. Hamilton, Z. Zhang, B. Hariharan, N. Snavely, and W. T. Freeman. Unsupervised semantic segmenta-
tion by distilling feature correspondences. In ICLR, 2022, 1, 2
[29] K. Han, R. S. Rezende, B. Ham, K.-Y K. Wong, M. Cho, C. Schmid, and J. Ponce. Scnet: Learning
semantic correspondence. In ICCV, 2017. 6
[30] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation
learning. In CVPR, pages 9729-9738, 2020. 10
[31] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022. 3
image
[32] J. Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-663 6851, 2020.
1,3
[33] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint urXiv:2207.12598 2022.
[34] Y. Hu, R. Wang, K. Zhang, and Y. Gao. Semantic-aware fine-grained correspondence. In ECCV, pages 97-115, Springer, 2022. 10
[35] S. Huang, L. Yang. B. He, S. Zhang. X. He, and Shrivastava, Learning semantic correspondence with sparse annotations. arXiv preprint arXiv:2208.069 2022. 2,5,
[36] G. Ilharco, Wortsman, R. Wightman, Gordon, N. Carlini, R. Taori. A. Dave. V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openelip, July 2021. 2, 5, 6,
9, 10, 17
[37] Jabri, A. Owens, and A. A. Efros. Space-time correspondence as contrastive random walk. NeurlPS, 2020. 2, 9, 10, 17
[38] Jeon, S. Kim, D. Min, and K. Sohn. Pyramidal semantic correspondence networks. IEEE TPAMI, 44(1))91102- 9118, 2021.
[39] S. Jeon, D. Min, S. Kim, J. Choc, and K. Sohn. Guided semantic flow. In European Conference on Computer Vision, 2020.
[40] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition. In ICCV, pages 3192-3199, 2013. 9
[41] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206. 00364, 2022. 1,
[42] Kim, Min, M. Cho. Transformatcher: Match-to-match attention semantic correspondence. CVPR, pages 8697-8707. 2022. 2, 5,
[43] Z. Lai, E. Lu, and W. Xie. Mast: A
self-supervised tracker. In CVPR, pages
6479-6488, 2020. 10
[44] 7. Lai and W. Xie. Self-supervised learning for video correspondence flow. arXiv preprint arXiv: 1/Xiv:905.00085 2019. 10
[45] J. Lec, D. Kim, J. Ponce, and B. Ham. Sfnet: Learning object-aware semantic correspondence. In CVPR, pages 1278-2287. 2019.
[46] J. Y. Lec, DeGol, V. Fragoso, and S. N. Sinha. Entchmatch-baser neighborhood consensus for semantic correspondence. In CVPR, pages 13153 13163, 2021. 2
[47] W. Li,, Hosseini Jafari, and C. Rother. Deep object co-segmentation. ACCV, pages 638-653.
Springer, 2019. 2
[48] X. Li, D.-P. Fan, F. Yang, Lao, Cheng, and Z. Liu. Probabilistic model distillation for semantic correspondence. CVPR, pages 7505-7514, 2021.
[49] X. Li, S. Liu, S. De Mello, X. Wang, J. Kautz, and M.-H. Yang. Joint-task self-supervised learning for temporal correspondence. NeurIPS, 32, 2019. 9, 10
[50] S. Liu, L. Zhang, X. Yang. H. Su, and J. Zhu. Unsupervised part segmentation through disentangling appearance and shape. In CVPR, pages 8355 8364, 2021. 2
12

---

[51] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJC,, 60:91-110, 2004. 2,9 [52] Z. Luo, T. Shen, L. Zhou, J. Zhang. Y. Yao, S. Li, T. Fang, and L. Quan. Contextdesc: Local deseriptor augmentation with cross-modality context. In CVPR, pages 2527-2536, 2019. 9
[53] Meng. Y. He, Y. Song. Song, Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. ICLRR, 2021.
[54] J. Min and M. Cho. Convolutional hough matching networks. In CVPR, 2021. 6
[55] J. Min, J. Lee, J. Ponce, and M. Cho. Spair-7lk: A large-scale benchmark for semantic conrespondence. arXiv prepreint arXiv:1908, 10543, 2019. 5
[56] J. Min, J. Lee, Ponce, and M. Cho. Learning to compose hypercolumns for visual correspondence. In ECCV, 2020.
[57] J. Mu, De Mello, Z. Yu, N. Vasconcelos, X. Wang. Kautz, and S. Liu. Coordgan: Self-supervised dense correspondences emerge from gans. CVPR, pages 10011-10020, 2022. 2
[58] D. Ofri-Amar, M. Geyer, Y. Kasten, and T. Dekel. Neural congealing: Aligning images a joint semantic atlas. CVPR, 2023. 1,5,6
[59] Y. Ono, E. Trulls, P. Fua, and K. M. Yi. Lf-net: Learning local features from images. NeurIPS, 31, 2018.
9
[60] O. Özyeşil, V. Voroninski, R. Basri, and A. Singer. A survey of structure from motion. Acta Numerica, (6:305-364, 2017.
[61] G. Parmar, K. K. Singh, R. Zhang. Y. Li, J. Lu, and J.-Y. Zhu, Zero-shot image-to-ime to--tgo translation. arXiv preprint arXiv:2302.0r8 03027, 2023. 1,4
[62] W. Peebles, J.-Y. Zhu, R. Zhang, A. Torralb,, A. Efros, and E. Shechtman. Gan-supervised dense visual alignment. In CVPR, 2022. 5,
[63] Y. Peng. X. He, and Zhao. Object-part attention model for fine-grained image classification. IEEE TIP.
1703)11887-1500 2017. 2
[64] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016. 9
[65] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint I 1004.000000000 7704000655 2017.
[66] J. Revaud, Weinzaepfel, C. De Souza, N. Pion, G. Csurka, Y. Cabon, and M. Humenberger. R2d2: repeatable and reliable detector and descriptor. arXiv preprint arXiv:/906.00 1995,, 2019. 9
[67] I. Rocco, Arandjelovic, and Sivic. Convolutional neural network architecture for geometric matching. CVPR, pages 6148 6157, 2017.
[68] I. Rocco, R. Arandjelovié, and J. Sivic. End-to-end weakly-supervised semantic alignment. In CVPR, pages 6917-69199 6925, 2018.
[69] I. Rocco, M. Cimpoi, R. Arandjelovié, A. Torii, T. Pajdla, and J. Sivie. Neighbourhood consensus networks. NewrlPS, 31, 2018. 6
[70] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021. 1, 3.4
[71] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Computer Assister-Assitted Intervention MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015. 1,3
[72] J. C. Rubio, J. Serrat, A. Lópcz, and N. Paragios. Unsupervised co-segmentation through region matching.
In CVPR, pages 749-755. IEEE, 2012. 2
[73] J. L. Schönberger and J.-M. Frahm. Structure-fiod from-motion revisited. CVPR, 2016. 1, 2,
[74] J. L. Schönhergerr Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. ECCV, 2016.
[75] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta,
C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.0840) 2022. 3, 4. 10,
[76] P. H. Sco, J. Lee, D. Jung, B. Han, and M. Cho. Attentive semantic alignment with offsct-aware correlation kernels. In ECCV, pages 349-364, 2018.
[77] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint a2X::2000.02250.. 2020. 15
13

---

[78] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS.
32, 2019. 3
[79] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint (rXiv:2011.33555 2020. 1,3
[80] Tang, N. Ruiz, Q. Chu, Y. Li. A. Holynski, D. E. Jacobs, B. Hariharan, Pritch, N. Wadhwa,
K. Aberman, et al. Realfill: Refereence-rriven generation for authentic image completion. arXiv preprint arXiv:2309. /6668, 2023. 1
[81] Tang, D. Wertheimer, and B. Hariharan, Revisiting pose-normalization for fine-grainee fow-shot recognition. In CVPR, pages 11521, 2020. 2
[82] P. Truong, M. Danelljan, F. Yu, and I. Van Gool. Warp consistency for unsupervised learning of dense correspondences. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
10346-10356, 2021. 6
[83] P. Truong. M. Danelljan, F. Yu, and L. Van Gool. Probabilistic warp consistency for weakky-suuperisee semantic correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recogmition,, pages 8708-8718, 2022. 6,
[84] N. Tumanyan, O. Tal. S. Bagon, and T. Dekel. Splicing vit features for semantic appearance transfer. In CVPR, pages 10748 10757, 2022. 2
[85] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211. 12572, 2022. 1, 1, 4
[86] M. Tyszkicwiez, Fua, and E. Trulls. Disk: Learning local features with policy gradient. Advances in Neural Information Processing Systems, 33: 14254-244422 1225--14265,, 2020, 9
[87] J. Valmadre, I.. Bertinetto, J. F. Henriques, R. Tao, A. Vedaldi, A. W. Smeulders, P. H. Torr, and E. Gavves.
I Long-term tracking in the wild: A benchmark. In ECCV, pages 670-685, 2018. 10
[88] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy. Tracking emerges by colorizing videos. In ECCV, pages 391-408, 2018. 10
[89] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd b0rds-200-2011 dataset.
2011. 5
[90] Q. Wang, Y.-Y. Chang. R. Cai, Z. Li, B. Hariharan, A. Holynski. and N. Snavely. Tracking everything everywhere all at once. arXiv preprint arXiv:2306.085 054000055422222 2023.
[91] Q. Wang. X. Zhou, B. Hariharan, and N. Snavely. Learning feature descriptors using camera pose supervision. In ECCV, pages 757-774. Springer, 2020. 2, 8, 9, 18
[92] X. Wang, A. Jabri, and A. A. Efros. Learning correspondence from the cycle-consistency of time. In CVPR, pages 2566 2576-25766222 2019. 1, 10
[93] Wu, Xiong, Yu, and D. Lin. Unsupervised feature learning via non-parametric instance-level discrimination. arXiv preprint 1
2018, 10
[94] J. Xu, S. Liu, A. Vahdat, W. Bycon, X. Wang, and S. De Mello. Open-vocabulan panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303. 04803, 2023, 3, 16
[95] J. Xu and X. Wang- Rethinking self-supervised correspondence learning: A video frame-level similarity
perspective. arXiv preprint arXiv:210: 203. /7263, 2021. 9, 10
[96] N. Xu, L. Yang. Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang. Youtube-vos: A large-scale vidco object
segmentation benchmark. arXiv preprint arXiv: 018880099332... 2018. 10
[97] B. Yan, Y. Jiang, Sun, D. Wang, Z. Yuan, P. Luo, and H. Lu. Towards grand unification of object tracking. In ECCV, pages 733-751. Springer, 2022. 2
[98] E. Yu, K. Blackburn-Mann Maizen, C. Nguyen, O. Wang. R. Habib Kazi, and A. Bousseau. Videodoodles: Hand-drawn animations videos with scene-aware canvases. ACM Transactions Graphics (TOG). 42(4):1-12, 2023.
[99] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. 2
[100] D. Zhao, Z. Song, Z. Ji, G. Zhao, W. Ge, and Y. Yu. Multi-scale matching networks for semantic correspondence. In ICCV, pages 3354-3364, 2021. 6
[101] W. Zhao, Y. Rao, Z. Liu, B. Liu, Zhou, and Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint vrXiv:2303.02/55 2023. 3, 16
[102] Y. Zhou, C. Barnes, F. Shechtman, and S. Amirghodsi. Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2266-2276, 2021.
14

---

A Societal Impact
Although DIFT can be used with any diffusion model parameterized with U-Net, the dominant publicly available model is the one trained on LAION [75]. The LAION dataset has been identified as having several issues including racial bias and stereotypes [7]. Diffusion models trained on these datasets inherit these issues. While these issues may a priori seem less important for estimating correspondences, it might lead to differing accuracies for different kinds of images. One could obtain the benefit of good correspondences without the associated issues if one could trained a diffusion model on a curated dataset. Unfortunately. the huge computational cost also prohibits the training of diffusion models in academic settings on cleaner datasets. We hope that our results encourage efforts to build more carefully trained diffusion models.
B Discussion
Why does correspondence emerge from image diffusion? One conjecture is that the diffusion training objective (i.e., coarse-to-fing reconstruction loss) requires the model to produce good. informative features for every pixel. This is in contrast to DINO and OpenCLIP that use image-level contrastive learning objectives. our experiments, we have attempted to evaluate the importance of the training objective by specifically comparing DIFT and DINO in all our evaluations: two models that share exactly the same training data, i.e., ImageNet- k without labels.
How and why does affect the nature of correspondence? In Fig. 9, for the same clean image, we first add different amount of noise get different xe following Eq. (2), then feed it into SD's de- noising network eg together with time step i to get the predicted clean image L #--lyI =--(vT-ooccutrr..t
va: We that, with the increase of t. more and more details removed and only semantic- features are preserved, and when t becomes too large, even the object structure is distorted. Intuitively, this explains why need small for correspondences that requires details and relatively large t for semantic correspondence.
input clean image
predicted clean images at diffcrent time step
51
261
[= 501
Figure 9: Within a reasonable range, when gets larger, the predicted clean images remain the overall structure but have less details, suggesting DIFT contains more semantic-level information and less low-level features with the increase of t.
How long does it take run DIFT? Since we only perform a single inference step when extracting DIFT, it actually takes similar running time compared to competing self-supervised features with the same input image size. For example, when extracting features for semantic correspondence as single NVIDIA A6000 GPU, DIFT ed takes 203 ms vs. OpenCLIP's 231 ms
in Sec. 5, one single 768x 768 image; DIFT adm takes 110 ms vs. DINO's 154 ms on one single 512x512 image. In practice, as mentioned in the last paragraph of Sec. 4.2, since there is randomness when extracting DIFT, we actually use a batch of random noise to get an averaged feature map for each image to slightly boost stability and performance, which would increase the running time shown above. But if computation is bottleneck, one can remove this optimization at the cost of a tiny loss in performance: e.g., on SPair-7 kk, DIFT .a: PCK 59.5 -5.5-577.9. 99; DIFT adm: PCK 52.0
Would diffusion inversion help? Another way to get x, from real input image is diffusion inversion. Using DDIM inversion [77] to recover input image's 's corresponding 2; and then feeding into fo to
15

---

get diffusion feature yielded similar results. At the same time, inversion makes the inference process several times slower. We leave how to utilize diffusion inversion get better correspondence to future work.
Does correspondence information exist in SD's encoder? We also evaluated SD's VAE encoder's performance on all benchmarks and found that its performance was lower by an order of magnitude. So ad correspondence only emerges inside its U-Net and requires diffusion- based training.
Would task-specific adaptation lead DIFT to better results? More sophisticated mechanisms could be applied to further enhance the diffusion features, e.g., concatenating and re-weighting features from different time step t and different network layers, or even fine-tuning the network with task-specific supervision. Some recent works [5, 94, 101] fine-tune either the U-Net or the attached head for dense prediction tasks and yield better performance. However, task-specific adaptation entangles the quality of the features themselves with the efficacy of the fine-tuning procedure. To keep the focus on the representation, we chose to avoid any fine-tuning to demonstrate the quality of the off-the-shelf DIFT. Nevertheless, our preliminary experiments suggest that such fine-tuning would indeed further improve performance on correspondence. We'll leave how to better adapt DIFT downstream tasks future work.
C Implementation Details
The total time step T for both diffusion models (ADM and SD) is 1000. U-Net consists of downsam- pling blocks, middle blocks and upsampling blocks. We only extract features from the upsampling blocks. ADM's U-Net has 18 upsampling blocks and SD's U-Net has upsampling blocks (the definition of blocks are different between these two models). Feature maps from the n-th upsampling block output are used as the final diffusion feature. For a fair comparison, we also grid-search which layer to extract feature for DINO and OpenCLIP for each task, and report the best results among the choices.
As mentioned in the last paragraph of Sec. 4.2, when extracting features for one single image using DIFT, we use a batch of random noise to get an averaged feature map. The batch size is 8 by default. We shrink it to 4 when encountering GPU memory constraints.
The input image resolution varies across different tasks but we always keep it the same within the comparison vs. other off-the-shelf self-supervised features (i.e., DIFT adm vs. DINO, DIFT, sd vs. OpenCLIP) thus the comparisons are fair. For DIFT, feature map size and dimension also depend on which U-Net layer features are extracted from.
The following sections list the time step i and upsampling block index n (n starts from we used for each DIFT variant on different tasks as well as input image resolution and output feature map tensor shape.
C.1 Semantic Correspondence
We use / = 101 and n = 4 for DIFT on input image resolution 512x512 so feature map size is 1/16 of input and dimension is 1024; we use = 261 and n = 1 for DIFT.a on input image resolution 768 768 so feature map size is 1/16 of input and dimension is 1280. These hyper-paramen shared on all semantic correspondence tasks including SPair-7Ik, PF-WILLOW, and CUB, as well as the visualizations in Figs. 15 and 16.
We don't t use image-specifs prompts for DIFT ad- Instead, we use general prompt "a photo of [class]" where [class] denotes the string of the input images' category, which is given by the dataset. For example, for the images of SPair-71k under cat class, the prompt would be "a photo of a cat". For CUB, the same prompt is used for all images: "a photo of bird". Changing per-class prompt to null prompt (empty string "") will only lead very small performance drop, DIFT. ad 's PCK per point on SPair- 59.5-47.66
16

---

C.2 Geometric Correspondence
On HPatches, the input images are resized to x 768 to extract features for both DIFT adm and DIFT sd- We use t 41, 11 for DIFT adm feature map size is 1/2 of imput and dimension is 512; we use = 0, n = for DIFT sd so feature map size is 1/8 of imput and dimension is 640.
In addition, for DIFT each image's prompt is null prompt, i.e., empty string ""
For all the methods listed in Tab. 4, when doing homography estimation, tried both sine and L. distance for mutual nearest neighbor matching, and both RANSAC and LMEDS for cv2. f indHomography(), (), and eventually report the best number among these choices for each method.
C.3 Temporal Correspondence
The configurations we use for DIFT add and DIFT are:
Time step
Block index
Temperature for softmax
Propagation radius
k for top-k
Number of prev. frames
Dataset
Method
π
DAVIS-2017
DIFT
51
0.1
15
10
28
DAVIS-2017
DIFT.d
51
0.2
15
15
28
JHMDB
DIFT.dm
101
0.2
5
15
28
JHMDB
DIFT
51
2
0.1
5
15
14
For experiments on DAVIS, we use the same original video frame size (480p version of DAVIS, specific size varies across different videos) in DINO's implementation [10]. for both DIFT adm and DIFT n=7 for DIFT adm. so feature map size is 1/8 of imput and dimension is 512. n=2 for DIFT feature map size is 1/8 of input and dimension is 640. For experiments CRW's implementation [37], we resize each video frame's smaller side to 320 and keep the original aspect ratio. n=5 for DIFT adm feature map size is 1/8 of input and dimension is 1024. n=2 for DIFT,d sd so feature map size is 1/8 of input and dimension is 640.
JHMDB, following
In addition, for DIFT each image's prompt is null prompt, i.e., empty string ""
D Additional Quantitative Results
D.1 Semantic Correspondence on PF-PASCAL
We didn't do evaluation PF-PASCAL [27] in the main paper because found over half of the test images (i.e., 302 out of 506) actually also appear in the training set, which makes the benchmark numbers much less convincing, and also partially explains why the previous supervised methods tend to have much higher test accuracy on PF-PASCAL vs. PF-WILLOW (e.g., over 90 vs. around 70) even using exactly the same trained model. And this duplication issue of train/test images also gives huge unfair disadvantage to the methods that are never adapted (either supervised or unsupervised) on the training set before evaluation.
However, even in this case, as shown in Tab. 6, DIFT still demonstrates competitive performance compared to the state-of-the-arr weakly-supervine method PWarpC [83], well huge gains vs. other off-the-shelf self-supervised features.
Table 6: PCK per image on PF-PASCAL. The groupings and colors follow Tab. 1.
PCK (rag
Sup. Method
0.05
0.10
PWarpC [83]
64.2
84.4
DINO [10]
36.9
$3.6
DIFT ood (ours)
56.5
72.5
(0)
OpenCLIP [36]
39,8
61.1
DIFT (ours)
69.4
84.6
17

---

1.0
Overall
Humination
Viewpoint
0.8
Methals
Feature
Match
HeAAll Roat SIFT
6.7k
2.5
SuperPoit
1.7K
BSK
DELLF
4AK
1.%
0.4
SIFT + Conraloo
44k
1.%
DE-Na
8:3
288
0.2
10K
230we1 16
EARS
17K
T0K
0.0
1022
SuperPaux DIFT.
1.7K
0.9
2
4
5
9011
5 9 1011
5
1011
threshold (psi
Figure 10: Mean Matching Accuracy (MMA) on HPatches [4]. For each method, we show the MMA with varying pixel error thresholds. We also report the mean number of detected features and mutual nearest neighbor matches. Although not trained with any explicit geometric correspondence labels, DIFT,a is able to achieves competitive performance compared to other feature descriptors that specifically design or trained for this task.
DIFT with different step
epsilon=5
80
epsilon=3
optilon=1
70-
Homogrsphy estimation
200
300
time step
Figure 11: Homography estimation accuracy [%] at 1, 3, 5 pixels on HPatches using DIFT ad with different time step t. Intuitively, gets larger, DIFT contains more semantic information and less low-level details, so the accuracy decreases when using larger t.
D.2 Feature Matching HPatches
Following CAPS [91], for the given image pair, extract SuperPoint [16] keypoints in both images and match them using our proposed feature descriptor, DIFT ad- We follow the evaluation protocol as in [19, 91] and use Mean Matching Accuracy (MMA) as the evaluation metric, where only mutual nearest neighbors are considered as matched points. The MMA score defined as the average percentage of correct matches per image pair under certain pixel error threshold. Fig. 10 shows the comparison between DIFT and other feature descriptors that are specially designed or trained for geometric correspondence. We report the average results for the whole dataset, well subsets on illumination and viewpoint changes respectively. For each method, we also present the mean number of detected features per image and mutual nearest neighbor matches per image pair. We can see that, although not trained with any explicit geometry supervision, DIFT is still able to achieve competitative performance.
D.3 Analysis on Hyper-parameters
Here we analyze how the choice time step and U-Net layer would affect DIFT's performance on different correspondence tasks.
Ablation on time step. Similar to Fig. 5, we plot how HPatches homography estimation accuracy and DAVIS video label propagation accuracy vary with different choices of t, in Figs. 11 and 12
18

---

DIFT son with different time step?
76
Video label propagation
74
accuracy
72-
70-
68
150
200
time step t
Figure 12: Video label propagation accuracy (I&Fm) on DAVIS using DIFT adm with different time step L. There's s wide range of where DIFT maintains a stable and competitive performance.
DiFT,3 features from different U layer
60
SPain 1kk
30
20
10
SD U-Net layer
Figure 13: PCK per point on SPair-7Ik using DIFT,d sd with different layer inside U-Net's 15 upsampling layers in total. The transition from block index n to layer index is 0/1/2/3 to 3/7/11/14 respectively. We can see there are multiple choices of leading to good performance.
respectively. Both curves have smooth transitions and there's 's large range of where DIFT gives competitive performance.
Ablation on U-Net layer. Compared to the definition of 4 block choices in Appendix C, here we make a more fine-grained sweep over SD's 15 layers inside U-Net upsampling blocks. The transition from block index n to layer index i is 0/1/2/3 to 3/7/1 1/14 respectively and both start from O. We evaluate PCK per point on SPair-7Ik lk using DIFT.d with different layer index As shown Fig. 13, the accuracy varies but there are still multiple choices of that lead to good performance.
E
Additional Qualitative Results
PCA visualization of DIFT. In Fig. 14, for each pair of images, we extract DIFT from the segmented instances, then compute PCA and visualize the first 3 components, where each component serves as a color channel. We can see the same object parts share similar embeddings, which also demonstrates the emergent correspondence.
Correspondence on diverse internet images. Same as Fig. 1, in Figs. 15 and 16 we show more correspondence prediction on various image groups that share similar semantics. For each target image, the DIFT.d predicted point will be displayed as red circle, together with a heatmap showing the per-pixel cosine distance calculated using DIFT sel We categories, and even image domains, e.g., from an umbrella photo to an umbrella logo.
it works well across instances,
Semantic correspondence comparison among off-the- features on SPair-71k. Same as Fig. 3, we show more comparison in Fig. 17, where we can see DIFT works well under challenging occlusion, viewpoint change and intra-class appearance variation.
19

---

Same Instance
Cross Instance
Cross Caregory
Figure 14: Visualization of the first three PCA components of DIFT,d on the segmented instance pairs (same instance, cross instance, cross category). Each component matches color channel. We can see the same object parts share similar DIFT embeddings.
Crossrcategory semantic correspondence. Same as Fig. 4, Fig. 18 we select an interesting image patch from a random source image and query the image patches with the nearest DIFT features in rest of the test split but with different categories. We see that DIFT able to identify reasonable correspondence across various categories.
Failure Cases on SPair-71k. In Fig. 19, we select four examples with low PCK accuracy and visualize DIFT.d predictions along with ground-truths. We can see that, when the semantic definition of key points ambiguous, the appearance/viewpoinn change between source and target images is too dramatic, DIFT fails to give correct predictions.
Image editing propagation. Similar Fig. 6, Fig. 20 shows examples edit propagation using our proposed DIFT. .d- It further demonstrates the effectiveness DIFT on finding semantic correspondence, even when source image and target image from different categories or domains.
Geometric correspondence. Same as Fig. 7, Fig. we show the sparse feature matching results using DIFT ad HPatches. Though not trained using any explicit geometry supervision, DIFT still works well under large viewpoint change and challenging illumination change.
Temporal correspondence. Similar to Fig. 8, Fig. 22 presents additional examples of video instance segmentation results on DAVIS-201 comparing DINO, DIFT and Ground-truth (GT). We can see DIFT adm could create instance masks that closely follow the silhouette of instances.
20

---

Point
DIFT, Predicted Target
SPIEE-NIS
the weats
prance
Coffee
Figure 15: DIFT can find correspondences on real images across instances, categories, and even
domains, e.g., from a photo of statue of liberty a logo.
21

---

Source Point
DIFT Predicted Target Points
Figure 16: DIFT can find correspondences on real images across instances, categories, and even domains, e.g., from a photo of an aeroplane to sketch.
22

---

DINO
OpenCLIP
DIFT
DIFT
tarpet image
Figure 17: Semantic correspondence using various off-the-shelf features SPair-71k. Circles indicates correct predictions while crosses incorrect ones.
23

---

Source patch
Top-S nearest neighbor cross-category target patches predicted by DIFT,,
Figure 18: Given the image patch specified in the leftmost image (red dot), we use DIFT to query the top-5 nearest image patches from different categories the SPair-7Ik test set. DIFT still able to find correct correspondence for object parts with different overall appearance but sharing the same semantic meaning, e.g., the leg bird vs. the leg of a dog.
source image
DIFT Af
farget image
source image
DIFT wt
larget image
Figure 19: Failure cases of semantic correspondence on SPair-71k. Circle denotes correct predictions
while cross for wrong ones. When the semantic definition of key points ambiguous, or appearance/viewpoint change between source and target images is too dramatic, DIFT fails to predict the correct corresponding points.
24

---

Figure 20: Edit propagation using DIFT sd Far left column: edited source images. Right columns: target images with the propagated edits. Note that despite the large domain gap in the last row, DIFT, ed still manages to establish reliable correspondences for correct propagation.
25

---

Figure 21: Sparse feature matching using DIFT,a sa on HPatches after removing outliers. Left are image pairs under viewpoint change, and right are ones under illumination change. Although never trained with geometric correspondence labels, it works well under both challenging changes.
26

---

DINO
DIFT adn
GT
DINO
DIFT adin
GT
DINO
DIFT adne
GT
Figure 22: Additional video label propagation results on DAVIS-2017. Colors indicate segmentation
masks for different instances. Blue rectangles show the first frames. GT is short for "Ground-Trun
27


Reviews:
[{"review_id": "5m0BUjNztv", "reviewer": "Reviewer_4Vxk", "summary": "The authors proposed a method for semantic correspondence using pretrained diffusion model as a feature extractor of the images. Without explicitly training on the additional data/annotations, a simple feature matching based on winner-take-all strategy with cosine distance metric surpasses the previous works on three different tasks; semantic/geometric/temporal correspondence.", "questions": "It would be interesting to see how the performances can be pushed further when sophisticated optimization techniques are applied to the raw matching similarities d(f^1,f^2).", "limitations": "The limitations and potential societal impact of this work are appropriately discussed", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Exploration on the usage of diffusion model for correspondence tasks\n- Good performances just with frozen pretrained model & simple matching pipeline", "weaknesses": "1. Limited novelty\n- Merely replacing the backbone network for feature extraction with the latest model in a straightforward manner does not captivate my interest. It also fails to provide new insights to the readers. It would be beneficial to delve deeper into why these pretrained diffusion models outperform previous backbones like DINO, CLIP, or ResNet.\n\n2. Limited demonstrations\n- The paper lacks several essential ablation studies that explore architectural design choices. These studies could include comparing models with and without a decoder, evaluating different options for Q/K/V selection, and assessing the impact of finetuning on correspondence datasets.\n- Given the relatively low originality, the paper would benefit from showcasing more diverse pixel-level prediction tasks. This could involve demonstrating the model's performance in object detection, image/ video segmentation, as well as other correspondence tasks such as depth estimation and optical flow. These additional demonstrations would significantly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback! Please find our response as below:\n\n**Limited novelty**.\n\nWe would like to point out that the \"backbone\" we are proposing to use comes from a diffusion model which is trained with a generative modeling objective that has prima facie little to do with learning good features. Thus the fact that correspondence emerges from this generative training and can even outperform techniques specifically designed for representation learning (DINO, OpenCLIP) is a surprising finding. Our simple approach for extracting this correspondence suggests that this is something intrinsic to the diffusion training objective. We think it's valuable to share this evidence with the community so that more interesting research directions could be explored, such as understanding why diffusion objectives are more suitable for capturing correspondence, and perhaps rethinking the diffusion model as a self-supervised learner.\n\n**Explain why pre-trained diffusion models outperforms previous backbones like DINO, OpenCLIP**.\n\nWe agree that more insights would be useful. As mentioned in the global rebuttal (point 1), we have added two new visualizations that we hope can shed some light on the features. We also have a conjecture that the diffusion model training objective (i.e., coarse-to-fine reconstruction loss) requires the model to produce good, informative features for every pixel compared to contrastive learning objectives (i.e., loss on image-level global feature).  In our experiments, we have attempted to evaluate the importance of the training objective by specifically comparing DIFT$\\_{adm}$ and DINO in all our evaluations, where they share exactly the same training data, i.e., ImageNet-1k without labels. We will add this discussion to the camera ready.\n\n**Evaluating more pixel-level prediction tasks**.\n\nWe indeed evaluate on video segmentation (DAVIS) and human keypoint tracking (JHMDB) in Sec. 6.2 and DIFT shows competitive performance although never being trained on such supervision or even video data. We have now also evaluated on image segmentation, where we freeze the backbone and only train a linear head on top of different feature extractors.\nThe mIOU results on ADE20k val set (single-scale evaluation)  are as follows:\n\n| Method | mIOU |\n| --------- | -------- |\n| DIFT$\\_{adm}$ | 29.2 |\n| DIFT$\\_{sd}$ | 34.5 |\n| DINO | 31.1 |\n| OpenCLIP | 40.9 | \n\nNote that this task is different from correspondence tasks with feature matching because it requires training a head with downstream supervision thus introduces additional parameters and hyperparameters. We performed an additional experiment where we evaluated the off-the-shelf feature correspondence using COCO image segmentation labels (as a precursor to object discovery; see Fig. 3 in STEGO [20]):\n\n| Method | AP |\n| --------- | -------- |\n| DIFT$\\_{adm}$ | 80 |\n| DIFT$\\_{sd}$ | 78 |\n| DINO | 79 |\n| OpenCLIP | 65 | \n \nDIFT achieves competitive performance on both these tasks. \n\nUnfortunately for depth estimation and optical flow, current pipelines (e.g., RAFT) usually involve a complex global optimization that is beyond simple local feature matching.\n\n**Exploring more sophisticated design choices or optimization techniques to further improve the correspondence performance**.\n\nIn our preliminary experiments, we do find that more sophisticated task-specific adaptation mechanism could further enhance the diffusion features, e.g., finetuning the network with specific downstream correspondence supervision. However, this would inevitably entangle the quality of the features themselves with the efficacy/strategy/hyperparameters of the adaptation procedure. To keep the focus on the evaluation of the representation itself, we chose to avoid any additional adaptation to demonstrate the raw quality of the off-the-shelf DIFT. Also see the discussion (Line 458-464) in Appendix B.\n\n“Comparing models with and without a decoder”: if \"decoder\" means the VAE decoder of the diffusion model, we indeed compared two types of diffusion models in all the evaluations: SD is a latent diffusion model with VAE decoder while ADM is directly on pixel space without en/de-coders. But note that for DIFT$\\_{sd}$, the decoder is actually not used during inference; if “decoder” means whether to attach a task-specific decoder for downstream correspondence tasks, we chose not to do that because it will require finetuning with supervision, see the discussion above. \n\n“Evaluating different options for Q/K/V selection”: our current implementation follows the most straight-forward way to get the intermediate feature map after each resnet-attention upsampling blocks inside U-Net (see the class `MyUNet2DConditionModel` of `sd_featurizer.py` in our submitted code). There contains multiple self-attention and cross-attention layers within each SD U-Net attention block. Delving into it and exploring key/value/query tokens as representation could potentially give performance boost, but also complicates the design choice and usage (different tasks require different set of hyperparameters)."}, {"review_id": "OkDgXsEfKZ", "reviewer": "Reviewer_u3f5", "summary": "This paper introduces DIFT, a method to yield emergent correspondence from image diffusion models without training or additional fine-tuning.\nThe method is simple - given an image (or an image pair), DIFT adds noise to the image to simulate the forward diffusion process, and pass it to the U-Net of a pretrained diffusion model to extract feature maps.\nThe authors discover that by simply computing the cosine similarity between the emergent feature maps, one can establish strong semantic, geometric and temporal correspondences without training.", "questions": "Please refer to the weaknesses section. The idea, simplicity, and novelty of the paper is strong, but I believe the paper lacks the experimental details and analyses to be accepted as-is.", "limitations": "The authors have included the ethical considerations in the paper, but not the limitations.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "* The discovery of emergent correspondences from image diffusion is novel. The proposed method is surprisingly simple as well, and this opens new possibilities and research directions for future work.\n\n* Strong performances on standard benchmarks of semantic correspondence, geometric correspondence and temporal correspondence.\n\n* The writing is clear and easy to follow.", "weaknesses": "* Missing evaluation of PF-PASCAL for semantic correspondence. This is not a critical drawback, as results on PF-PASCAL tend to be saturated.\n\n* Incomplete implementation details. What was the image size / feature map size used to establish the correspondences? Image size is a critical factor in many image correspondence methods.\n\n* Lack of latency and computation analysis. This is crucial to identify the applicability of DIFT to real world scenarios.\n\n* Lack of rationale or analysis on how exactly the image diffusion models can yield such feature maps easily. While some motivation is provided in the introduction, it seems insufficient for the readers to fully understand how the correspondences are emerging from image diffusion. Analyzing some failure cases could be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback! Please find our response as below:\n\n**Missing evaluation on PF-PASCAL**: \n\nHere’re the comparison on PF-PASCAL: \n| Method | PCK@$\\alpha_{img}$=0.1 |\n| ----- | ----- |\n| PWarpC [Truong et al. CVPR 2022] (see review by rXbs) |  87.6 |\n| DIFT$\\_{sd}$ | 84.6|\n| OpenCLIP | 61.1 |\n| DIFT$\\_{adm}$ | 72.5 | \n| DINO | 53.6 |\n\n\n PWarpC is a state-of-the-art weakly-supervised method mentioned by Reviewer rXbs, trained on training images with only image-level class labels. We didn’t do evaluation on PF-PASCAL at the beginning because we found over half of (i.e., 302 out of 506) the test images actually also appear in the training set, which makes the benchmark numbers much less convincing, and also partially explains why the previous supervised methods tend to have much higher test accuracy on PF-PASCAL vs. PF-WILLOW (e.g., over 90 vs. around 70) even using exactly the same trained model. And this duplication issue of train/test images also gives huge disadvantage to the methods that are never adapted (either supervised or unsupervised) on the training set before evaluation. However, even in this case, DIFT still demonstrates competitive performance as well as huge gains vs. other off-the-shelf self-supervised features. We’ll add this clarification in the future version of the paper.\n\n**Latency of DIFT**:\n\nPlease refer to point 3 of the global rebuttal above. \n\nBreifly speaking, DIFT takes similar running time as its competitor self-supervised features, e.g., on one single A6000, for each image, DIFT$\\_{sd}$ takes 203 ms vs. OpenCLIP’s 231 ms, DIFT$\\_{adm}$ takes 110 ms vs. DINO’s 154 ms. We'll include these details in the future version of the paper.\n\n**Lack of experiment details, e.g., image size, feature map size**.\n\nThe input image size varies across different tasks but we always keep it the same within the comparison vs. other off-the-shelf self-supervised features (i.e., DIFT$\\_{adm}$ vs. DINO, DIFT$\\_{sd}$ vs. OpenCLIP) thus the comparison should be fair. For every correspondence tasks, we include the details of image size and feature map size/dimension in the point 2 of global rebuttal above. We also included implementation details in Appendix C and the code of DIFT (see `sd_featurizer.py` and `adm_featurizer.py`) in the submitted supplementary material. We'll include above details either in the main paper or add a pointer to the supplementary.\n\n**Lack of analysis on why correspondence emerges from image diffusion**.\n\nPlease refer to point 1 of the global rebuttal above. \n\nWe provide two more visualizations in the attached pdf to help audience better understand how DIFT works. In Fig. 1, we visualize the first three PCA components of DIFT$\\_{sd}$ on the segmented instance pairs and we can see that the same kind of object parts share similar features. In Fig. 2, we visualize the predicted clean images at different time step $t$ and we can see that, within a reasonable range, when $t$ gets larger, the predicted clean images remain the overall structure but have less details, suggesting DIFT contains more semantic-level information and less low-level features with the increase of $t$. It also explains that we need a small $t$ for correspondence tasks that require details and relatively large $t$ for semantic correspondence.\n\nWe believe that the diffusion training objective (i.e., coarse-to-fine reconstruction loss) requires the model to produce good, informative features for every pixel. This is in contrast to DIFT outperforms DINO and OpenCLIP that use image-level contrastive learning objectives. In our experiments, we have attempted to evaluate the importance of the training objective by specifically comparing DIFT$\\_{adm}$ and DINO in all our evaluations: two models that share exactly the same training data, i.e., ImageNet-1k without labels.\n\nWe'll include these figures and analysis in the future version of the paper. \n\n**Analyzing failure cases**.\n\nPlease refer to Fig. 3 in the attached rebuttal pdf file for the failure case visualization, where we randomly select two examples from the SPair-71k category with the lowest per-class accuracy (i.e., boat and bottle). We can see that, when the semantic definition of key points are ambiguous or the appearance change between source and target images are too dramatic, DIFT$_{sd}$ fails to predict the ground-truth labels. We'll include more visualizations and analysis on different failure cases in the future version of the paper."}, {"review_id": "9JdIRJYlYt", "reviewer": "Reviewer_4nEG", "summary": "The paper addresses a classical computer vision problem, ie, points correspondence. The authors show that the feature maps of the decoder of a diffusion model U-Net enable robust feature matching with a simple nearest neighbor search. Semantic or geometric correspondence can be achieved, by selecting the appropriate denoiser time step. The authors report results, quantitative analysis and comparison on several benchmarks, chosen for each specific task.", "questions": "1) In figure 3,  it seems that the semantic information is such that the right ear feature is distinct from the left ear one. Is it a general case or does this happen in this specific example only due to a very similar pose in the source and the target image?\n\n2) Did the authors try to apply clustering based on U-Net features? \n \n3) Could the authors elaborate on the  computational time. What is the dimension of a DIFT feature?", "limitations": "the authors rightly mentioned the ethical issues related to generative models .", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors observe empirically that the features contained in the U-Net decoder are powerful features  for image correspondence. They also show that those features embed different levels of semantic information, depending on the time step of the denoiser. \n\nIt is particularly relevant given the fact that diffusion models used by the authors are pretrained (the authors only add noise to the input image), and that the drawn conclusions are consistent over different DMs. \n\nThe set of experiments reported in the paper are thorough and results are convincing. The variety of the results, quantitative and qualitative, demonstrate the reliability and flexibility of such features.\n\nI particularly enjoyed reading a paper presented with such simplicity and clarity.", "weaknesses": "The drawback of the approach is probably the computational complexity that is intrinsic to any point matching approach relying on nearest neighbor search based on high dimensional features. \n\nIt is not clear from the paper (nor the supplementary material), if choosing the optimal time denoising time step is critical or not. A corollary of this question would be: do we expect the semantic level contained in the features to degrade smoothly from large time step to small ones?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback! Please find our response as below:\n\n**Clustering on U-Net features**.\n\nPlease refer to point 1 of the global rebuttal above and Fig.1 of the attached pdf, where we we visualize the first three PCA components of DIFT$\\_{sd}$ on the segmented instance pairs and we can see that the same kind of object parts share similar feature embeddings. We'll include more such visualization in the future version.\n\n**Computation time and feature dimensions**.\n\nPlease refer to point 2 and 3 of the global rebuttal above.\n\nBreifly speaking, DIFT takes similar running time as its competitor self-supervised features, e.g., on one single A6000, for each image, DIFT$\\_{sd}$ takes 203 ms vs. OpenCLIP’s 231 ms, DIFT$\\_{adm}$ takes 110 ms vs. DINO’s 154 ms. The feature dimension depends on which U-Net layer is extracted from, and we list the details for every correspondence tasks above. \n\nWe'll also include these details in the future version of our paper.\n\n**Is the choice of time step t critial?**.\n\nIn the main paper, we briefly discussed the influence of time step t in Sec. 5.2 Line 176-178 and Fig. 5 shows how the semantic correspondence performance varies across different t. It can be seen that curve is pretty smooth and there’s a large range of t where DIFT gives competitive performance. But the accuracy also decreases significantly when t becomes too large or too small. \n\nAs suggested by reviewer zLC3, we have added new experiments that ablate the choice of $t$ on the geometric correspondence (HPatches) and temporal correspondence (DAVIS) tasks in Fig. 5 and 6 of the rebuttal document respectively, where they share the similar observation as above. \n\nGiven these experiments,  we would say the choice of t is definitely an important hyperparameter that user need to tune for specific downstream correspondence tasks, but the trend is quite smooth and the change is also not sharp, so it’s not hard to find a good enough t within a reasonable number of trials. Based on our experiments, intuitively we find that within a reasonable range (e.g., t cannot be too large, otherwise $x_t$ is too noisy), larger t usually leads to more semantic-level features and smaller t leads to features that contain more low-level image details. Users can also use this heuristics to help narrow down the search.\n\nIn addition to these new ablations and visualizations, Figure 2 in the rebuttal document attempts to give an intuition for the effect of $t$. Please see point 1 of the global rebuttal for further explanations of this figure.\n\n**Do we expect the semantic level contained in the features to degrade smoothly from large time step to small ones?**\n\nAs mentioned in the above comments, based on the ablations and experiments we have, we think this is a good heuristics to have when tuning the time step t for downstream tasks. The smoothness could also be verified in the low-sensitivity of semantic correspondence performance to the choice of t, as in Fig. 5 of the main paper. In Fig. 2 of the attached pdf and point 1 of the global rebuttal above, we also give an intuitive explanation on why this would happen by visualizing the predicted clean image at different time step t.\n\n**Distinction between symmetric object parts using DIFT**.\n\nThere’re a lot of such symmetric keypoints In SPair-71k/CUB/WILLOW (e.g., left vs. right ear/handler/wing, fore-wheel vs. back-wheel) and the prediction has to distinguish them from each other in order to be counted as correct. So the high accuracy numbers on these benchmarks actually indicate that DIFT is able to handle these cases pretty well in general. In addition to Fig. 3, the last two rows of Fig. 4 also demonstrate a few cross-category cases on distinguishing left/right paws/eyes. Especially in the last row, we can see even the pose between source and target are quite different, DIFT with simple cosine similarities is still able to give correct predictions. \n\n**Large computational complexity that is intrinsic to any point matching approach relying on nearest neighbor search on high dimensional features**.\n\nThe main focus of our paper is to demonstrate that correspondence emerges from image diffusion models without explicit supervision. In order to show that, we use simple technique to extract the feature representation inside U-Net, paired with the most straightforward feature matching strategy (i.e., nearest neighbor using cosine distance) and it achieves competitive performance on several correspondence tasks. The simplicity of the matching mechanism actually further proves the good quality of diffusion features themselves. \n\nBut we agree that this nearest neighbor matching strategy is definitely not ideal, and more sophisticated designs from the feature matching literature could further boost DIFT’s performance and efficiency, e.g., a coarse-to-fine matching strategy. Meanwhile, in terms of efficiency in practice, note that DIFT is basically as fast as other off-the-shelf self-supervised features (e.g., DIFT$\\_{sd}$ takes 203 ms running on a single input image, see more details in point 2 of the global rebuttal above). And when calculating cosine distance, we could L2-normalize each feature independently and then dot product each other, which is also very efficient in terms of time and memory, compared to other correspondence methods [27, 31] that use the attention mechanism to calculate the correlation. In addition, since DIFT feature is image-independent so we can pre-calculate it for each image and then save them in advance before doing the matching, whereas some methods [27, 31] that require a pair of images as network input cannot take advantage of this pre-processing, especially when the same image need to be queried multiple times.\n\nWe'll add this clarification into the future version of our paper."}, {"review_id": "KeG0reUWcM", "reviewer": "Reviewer_zLC3", "summary": "The paper proposes to use off-the-shelf generative networks based on denoising diffusion models to find local correspondences. The paper is extremely simple: instead of generating samples purely from random noise or doing some kind of image-based conditioning, the method just adds random noise to the input image and takes some intermediate level in the denoising U-Net as a dense feature map, from which it can extract sparse features. Feature matching is done via cosine distances. Keypoints may be provided (e.g. for semantic matching) or taken with an off-the-shelf keypoint detector, such as SuperPoint. The authors show that their approach works for high-level semantics (e.g. an eye across different animals or even species) and geometric correspondences, and can also track points across time in video sequences. Parameters such as the stage in the denoising process or which intermediate layers to use are chosen per dataset. There are some tricks (e.g. featuremaps are averaged over batches with different noise inputs), but that's about it.", "questions": "The paper is cool and novel, but given the issues above I'll mark it as borderline (BA, since I only have BA and BR) and wait on the rebuttal and reviewer discussion.\n\n- How do you deal with the loss in resolution, especially for geometric matching? The supplementary material says the method uses the 11th upsampling block out of 18 for ADM, and the 2nd block out of 4 for SD. Do you simply upscale the feature maps? And if so how do you achieve accurately localized correspondences?\n- Do you choose per-dataset hyperparameters on a validation set or the test set?\n- What is the computational cost and how does it compare with other methods?\n- What does \"after removing outliers\" mean in fig 10 in the supplementary material? Why not show the raw matches and color-code them according to correct/incorrect?\n- What happens without per-class prompts?\n\nNotes:\n\n- It's nice to see that the method is sensitive to the choice of t but varies smoothly (Fig. 5). It would be nice to see this in other datasets, though. And what about n?\n- The images in Fig. 10 in the supplementary material have the wrong colors (I guess BGR, from loading/saving with OpenCV).\n- Please add the dataset and metric to the headers in Tables 1 and 2 (one's missing the metric, the other two the dataset names).\n\nCitations:\n\n[DISK] https://arxiv.org/abs/2006.13566 (https://github.com/cvlab-epfl/disk)\n[PoSFeat] https://openaccess.thecvf.com/content/CVPR2022/papers/Li_Decoupling_Makes_Weakly_Supervised_Local_Feature_Better_CVPR_2022_paper.pdf (https://github.com/The-Learning-And-Vision-Atelier-LAVA/PoSFeat)\n[ALIKE] https://arxiv.org/pdf/2112.02906.pdf (https://github.com/Shiaoming/ALIKE)\n[SILK] https://arxiv.org/abs/2304.06194 (https://github.com/facebookresearch/silk)", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Cool idea.\n2. Very simple. The actual method section is a single paragraph and doesn't require any math. The paper is well written and easy to understand.\n3. Very good results on multiple tasks, including semantic matching and tracking in video sequences, without training or fine-tuning.", "weaknesses": "1. My main complaint is that I find it hard to believe this approach will work well across arbitrary geometric changes. I have experience in this field and conclusions drawn from HPatches rarely translate to real scenarios. The paper gives very few results, which makes these experiments unreliable and difficult to trust (my guess is that this gap will disappear when the method is properly evaluated). For instance:\n- What is the image size?\n- How many points do you use? \n- Do you use both the illumination and viewpoint splits for HPatches (the supplementary material suggests so)? If so, why not split the results by sequence type?\n- Why not use the more standard MMA metric from D2-Net (which isn't perfect but is easier to understand), instead of estimating homographies? Why not both?\n- Why not use modern baselines? The most recent method is from 2019 (R2D2): see e.g. DISK, PoSFeat, ALIKE, or SILK (citations below). (Note: CAPS is weakly supervised and works worse than the DISK variant supervised only with epipolar geometry, as far as I know.)\n- Could you provide a simple precision/recall curve and compare it against traditional local feature methods? My intuition is that it would be worse.\n- What happens if you add RANSAC?\n\nI generally discount claims on this dataset and steer people towards evaluation benchmarks focusing on downstream tasks, such as visualocalization.net or the image matching challenge (https://image-matching-workshop.github.io/). I understand that this is likely beyond the scope of this paper, and I think it would be acceptable that it just shows these features work reasonably well for rigid matching, but if you want to claim that \"though not trained using any explicit geometry supervision, DIFT outperforms prior state-of-the-art methods that utilize explicit geometric supervision signals designed specifically for this task, such as correspondences obtained from Structure from Motion pipelines\" (L210-213), then you're going to have to substantiate that much better.\n\n2. Given that the paper is so simple and the concept itself is very easy to explain, it would be nice to see more introspection. For instance, showing qualitative results using PCA or t-SNE to cluster the features (see for instance the videos from DINO v2) would help understand what's going on.\n\n3. Important details are relegated to the supplementary material:\n- The amount of noise used per dataset. It's surprising that, for instance, for geometric correspondence with Stable Diffusion the best results are with t=0, which means basically no noise (if I understand the formulation correctly; I looked at the code).\n- The fact that for semantic correspondence per-class prompts are used. I think this is a reasonable assumption, but it should be mentioned. It also suggests the method may not work so well in non-object-centric images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback! Please see our response below:\n\n**Implementation details and evaluations on HPatches**.\n- Image size: all images are resized to 768x768 then fed into the network to extract feature maps.\n- Number of points: following CAPS, we use SuperPoint to extract keypoints and it has 1.7k points per image on average.\n- MMA metric: we plot MMA curves in Fig. 4 of the attached pdf.\n- Modern baselines: We'll include the suggested methods in the revised comparison table. We include the DISK variant trained with epipolar supervision only (DISK$\\_{epi}$) in the MMA figure above and report their homography numbers below.\n- Dataset splits: Tab. 3 shows the overall accuracy. Here’re the per-split numbers (also for DISK$\\_{epi}$):\n| Method | $\\epsilon=1$ | $\\epsilon=3$ | $\\epsilon=5$ |\n| ----- | ------ | ----- | ----- |\n| Illumination |  | | |\n| DIFT$\\_{sd}$ | 64.2 |  93.5 |  97.3 | \n| DISK$\\_{epi}$ | 66.9 |  93.1 | 96.9 |\n| Viewpoint | | | |\n| DIFT$\\_{sd}$ | 28.9 |  63.2 | 72.9 |\n| DISK$\\_{epi}$ | 28.6 | 62.1|  75 |\n- PR curve comparison with traditional local feature methods: using the same SIFT keypoints, we compare the mean Average Precision (mAP) of all image pairs using different feature descriptors with a threshold of 5 pixels: SIFT=74.9, DIFT$\\_{sd}$=69.7, DINO=45.6. Although both DIFT and SIFT have high precision at high recall, the mAP for DIFT is lower because DIFT's scoring function is not optimized. DIFT achieves better MMA and homography accuracy than SIFT.\n- Add RANSAC: given a pair of images, we first use cosine distance to find mutual nearest neighbor matches, then use `cv2.findHomography()` with `method=cv2.LMEDS` to remove outliers and calculate the homography transformation. Empirically, we find LMEDS works better than RANSAC.\n- Loss of resolution: after getting the dense (but also lower-resolution) feature map, pixel-level features are extracted via bilinear interpolation. We use SuperPoint to localize the keypoints then extract their DIFT features to do matching. More details about image/feature map resolution are in point 2 of the global rebuttal.\n\nTo address the reviewer's concern, we will modify our claim: \"though not trained with any explicit geometry supervision, DIFT still achieves competitive performance on HPatches compared to the methods explicitly trained for geometric correspondence with weak/epipolar supervision\".\n\n**More introspection of DIFT**.\n\nPlease refer to point 1 of the global rebuttal above, and Fig. 1 and 2 of the attached pdf.\n- PCA visualization: we visualize the first 3 PCA components of DIFT$\\_{sd}$ on segmented object pairs. As shown in Fig. 1, the object parts that share the same semantic meaning tend to have similar DIFT embeddings. \n- Different t for different correspondence tasks: we visualize the predicted clean images with different time step t in Fig. 2 as an intuitive way to demonstrate that features contains more semantic-level information and less low-level details with the increase of t, which partially validate the usage of very small t for geometric correspondence.\n\n**The usage of per-class prompt in semantic correspondence**.\n\nChanging per-class prompt to a null prompt (empty string) will only lead a very small performance drop for DIFT$\\_{sd}$ on SPair-71k: 59.5&rarr;57.6 as in Tab. 1 of the main paper.\n\nWe'll mention the usage of per-class prompt in the revised main paper. Note that only DIFT$\\_{sd}$ needs a prompt as part of the input while DIFT$\\_{adm}$ doesn’t. This per-class prompt is only used in semantic correspondence tasks, while null prompt is used for the others. The prompt design is simple and not image specific, only using class name, i.e., “a photo of a {class}”. For CUB, we actually use the same prompt for all images: “a photo of a bird”. \n\n**Wrong color format in Fig. 10; Missing headers in Tab. 1 and 2; Important details in the supp**.\n\nWe apologize for the error. We'll fix this and table headers in the revision. We'll also add these implementation details in the main text plus a pointer to the appendix.\n\n**Per-dataset hyperparameters**.\n\nMost of the datasets used in the paper don't have a validation set, so we chose the two hyperparameters of DIFT (i.e., time step t and block index n) based on the test performance. But since our hyperparameter search on DIFT is pretty coarse, we do not believe our results are due to overfitting. Fig. 5 in the main paper and Fig. 5, 6 and 7 in the attached pdf show the performance for all possible $t$ and $n$ values; we will show similar plots for all tasks in the camera ready supplementary.  Note that, for fair comparison, we also densely grid search which layer to extract feature for DINO and OpenCLIP for each dataset. Also as mentioned in Line 154-156, for semantic correspondence, the hyperparameter tuning is only done on SPair and fixed afterwards on other datasets.\n\n**Computation cost**.\n\nPlease refer to point 2 of global rebuttal above.\n\nBriefly speaking, DIFT takes similar running time as its competitor self-supervised features, e.g., on one single A6000, for each image, DIFT$\\_{sd}$ takes 203 ms vs. OpenCLIP’s 231 ms. We'll also include these details in the revision.\n\n**\"after removing outliers\" in Fig. 10**.\n\nWe only visualize the matches that are marked as \"inliner\" after applying `cv2.findHomography()`. We'll add figures with colored raw matches in the revision.\n\n**Ablation on t and n**.\n\nWe ablate DIFT$\\_{sd}$ with different t on HPatches and DAVIS, as in Fig. 5 and 6 of the attached pdf, where similarly to Fig. 5 of the paper, performance varies across t smoothly. \n\nWe also ablate which layer to extract DIFT$\\_{sd}$ for SPair in Fig. 7 of the attached pdf, where layer index i is more fine-grained than block index n, i.e., SD U-Net has 4 upsampling blocks with 15 layers in total. The transition from n to i is 0/1/2/3 to 3/7/11/14 (both start from 0). We can see the performance varies but there's a wide range choice of i having good performance."}, {"review_id": "iQ9IZQ0TQB", "reviewer": "Reviewer_rXbs", "summary": "The authors find that intra and cross-category correspondences are implicitly learnt by diffusion models trained self-supervised on large datasets. The paper proposes an approach to extract this knowledge as features from pre-trained Unet-based diffusion models. In particular, to compute the features of a particular image, noise is added to the image to simulate the diffusion process, and then input to the pre-trained diffusion model. The intermediate layer activations from the Unet at a particular timestep are used as features. They showcase the performance of these features on semantic-matching, outperforming other unsupervised baselines and strongly-supervised approaches specifically designed for semantic matching. They also show that without any task specific finetuning, the features can be used for geometric matching and temporal matching ( video-object-segmentation throught label propagation), with competitive performance compared to state-of-the-art. ", "questions": "A) What is the run-time of the approach to extract feature on a single image? \n\nB) I think details on the ‘architecture’ of the extracted features are missing. It is only said L.113-114 that “intermediate layer activations” are extracted from the unet, and that each depends on the task. I would appreciate to have some details there - what layers are extracted (for the different tasks), what resolution and channel dimension do they have, are they interpolated? Are features from multiple levels used? If so how are they aggregated?\n\nC) What is the resolution of the extracted feature and at what resolution is the matching done? Is a similar resolution used in competitor works like DINO? \nThe resolution of the feature maps is of crucial importance for obtaining fine-grained/accurate correspondences. Since matching is done by computing all-to-all similarities between the two images, It has often limited previous approaches  to small resolution because of memory constraint. Just using a higher resolution feature map would lead to a significant improvement in results, not necessarily related to the features themselves. \n\nD) Some misclassification and missing comparisons in semantic matching:\n\nD.1) CNNGeo and A2Net are unsupervised - they only use single images as supervision, as opposed to NCNet for example which requires pairs of similar images \n\nD.2) Comparison to recent approches are missing. In particular PWarpC [Truong et al. CVPR 2022] obtained state-of-the-art results in weakly-supervised semantic matching (with extension to supervised)\n\nE) Would it be possible to extract features from multiple layers of the diffusion model such as to build a feature pyramid (similar to VGG or ResNet)? This could potentially be used as a plug-in replacement for backbones in correspondence approaches with a decoder. \n\n\nF) How many different passes through the network (ie different random noise) are used for an image? \n\n", "limitations": "- ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea is simple and effective\n\n- The results are convincing. Having a single model applicable to many different correspondence tasks without specific architecture/training and outperform task-specific methods would be a useful contribution. The new perspectives that it opens are exciting. \n\n- The paper overall reads well. \n", "weaknesses": "1) The authors do not comment on the run-time of the proposed approach. I expect it will be quite slow since the image needs to go through the reverse diffusion process.  \n\n2) Related to the above, relying on the diffusion model at inference time makes the method impractical in many applications. To remove the reliance on diffusion models at inference, would it be possible to train a feature predictor, using as ground-truth the features extracted from a pre-trained models? Have you done any experiments in this direction?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback! Please see our response below:\n\n**Run-time of DIFT**.\n\nPlease refer to point 2 of global rebuttal above.\n\nBriefly speaking, DIFT is actually fast because it doesn’t need to run diffusion inversion thus only one network inference is involved (see the second last paragraph of Sec. 4.2). Also, since we only need the intermediate U-Net features, the inference process can stop immediately when reaching the desired block and doesn't need to run the following layers. For example, it only takes 203 ms for DIFT$\\_{sd}$ vs. 231 ms for OpenCLIP on one single image. In addition, we also discuss the usage of diffusion inversion in the first paragraph of Appendix B.\n\n**Train a feature predictor network to have faster diffusion features**.\n\nGiven the speed of DIFT is already pretty fast as mentioned above, we haven’t explored the direction to train a feature predictor network such that it can make DIFT even faster. But we believe the techniques in the field of knowledge distillation (e.g., train a smaller student network to distill knowledge from a much larger teacher network) could be generally applied to diffusion features, especially when latency of DIFT becomes a main bottleneck in practical usage.\n\n**Implementation details of DIFT including hyperparameters and resolutions**.\n\nWe include the implementation details in Appendix C including the time step $t$ and network block index $n$ used in each task. We’ll add a pointer in the main paper in the future version. For input image size, feature map size and feature dimensions, see point 2 of global rebuttal above. Note that the image size used in DIFT and its competitor self-supervised feature is the same for each task, i.e., DIFT$\\_{sd}$ vs. OpenCLIP, DIFT$\\_{adm}$ vs. DINO, so the comparison should be fair. The per-pixel feature is extracted through bilinear interpolation on the feature map. We also include the code of DIFT (see `sd_featurizer.py` and `adm_featurizer.py`) in the submitted supplementary material.\n\n**Misclassification and missing comparisons in semantic matching**.\n\nThanks for the correction and we will revise the tables accordingly and also include PWarpC as baseline.\n\n**Aggregation of multi-layer diffusion features**.\n\nFor all the results in the paper, only single layer diffusion features are used. In our preliminary experiments, we also found that aggregating features from multiple layers could slightly improve the performance. However, this also inevitably introduces many design choices (e.g., how to do the aggregation) and hyper-parameters (e.g., the layer and time step t to aggregate and the weight on each of them), which could vary across different downstream tasks. Optimizing them could further improve the performance, but also entangles the quality of the features with the tuning of these design choices and hyper-parameters. Since the main focus of our paper is to demonstrate correspondence emerges from image diffusion without explicit supervision, we focus on the most simple technique and the raw off-the-shelf single timestep/layer diffusion features. We’ll leave how to improve the performance by constructing a multi-layer/timestep diffusion feature pyramid to future work.\n\n**Number of network inference passes used for each image**.\n\nIt only takes one single network pass when extracting DIFT for each image. As mentioned in Line 128-130, to enhance the stability of the representation in the presence of random noise added to the input image, we extract features from multiple noisy versions with different samples of noise, and average them to form the final representation. As mentioned in the first paragraph (Line 470-472) of Appendix C, this is usually being done in a batch manner so only one network inference is needed per image (see the `forward()` function in `sd_featurizer.py` and `adm_featurizer.py` of the code in our submitted supplementary material). We usually sample 8 noise per image but sometimes shrink it to 4 when having memory issues. Changing from 8 to 1 noise per image only lead to very small performance drop, e.g., on SPair-71k, DIFT$\\_{sd}$: PCK 59.5&rarr;57.9; DIFT$\\_{adm}$: PCK 52.0&rarr;51.1."}]

Author rebuttal:
We thank all the reviewers for their time and valuable feedbacks! We are encouraged that reviewers find that our paper is well-written and easy to follow (u3f5, 4nEG, zLC3, rXbs), that our approach achieves good performance with a simple technique (all 5 reviewers!) along with cool and novel ideas (u3f5, zLC3). Here, we answer a few global questions below:

**1. Understand why correspondence emerges from image diffusion, and why it outperforms previous backbones like DINO, OpenCLIP**.

We agree that understanding why correspondence emerges is an intriguing question. Inspired by reviewers' suggestions, we add two more visualizations in the attached pdf. In Fig. 1, for each pair of images, we extract DIFT$\_{sd}$ from the segmented instances, then compute PCA and visualize the first 3 components, where each component serves as a color channel. We can see the same object parts share similar embeddings, which also demonstrates the emergent correspondence. To further explore how this correspondence depends on $t$, in Fig. 2, for the same clean image, we first add different amount of noise to get different $x_t$ following Eq. 1, then feed it into network $\epsilon_\theta$ together with time step $t$ to get the predicted clean image $\hat{x}_0^t=\frac{x_t - (\sqrt{1 - \alpha_t}) \epsilon
_\theta (x_t, t)}{\sqrt{\alpha_t}}$. We can see that, with the increase of $t$, more and more details are removed and only semantic-level features are preserved, and when $t$ becomes too large, even the object structure is distorted. Intuitively, this explains why we need a small $t$ for correspondences that requires details and a relatively large $t$ for semantic correspondence. We'll include more examples of both  visualizations in the future version.

As a more general explanation for the emergent correspondence, we have the following conjecture. We believe that the diffusion training objective (i.e., coarse-to-fine reconstruction loss) requires the model to produce good, informative features for every pixel. This is in contrast to DIFT outperforms DINO and OpenCLIP that use image-level contrastive learning objectives. In our experiments, we have attempted to evaluate the importance of the training objective by specifically comparing DIFT$\_{adm}$ and DINO in all our evaluations: two models that share exactly the same training data, i.e., ImageNet-1k without labels. A rigorous evaluation of our conjecture would be very interesting, but we lack the tools to perform such rigorous analysis: how to understand and explain why certain properties emerge from deep neural networks trained on large-scale data remains an unsolved research problem. We will add this discussion to the camera ready.

**2. Input image size, feature map size and feature dimension used in each correspondence task**.

The input image size varies across different tasks but we always keep it the same within the comparison vs. other off-the-shelf self-supervised features (i.e., DIFT$\_{adm}$ vs. DINO, DIFT$\_{sd}$ vs. OpenCLIP) thus it should be fair. For DIFT, feature map size and dimension also depend on which U-Net layer features are extracted from. ADM’s U-Net has 18 upsampling blocks and SD has 4 upsamping blocks (the definition of blocks varies). Below for each tasks, we list all the image sizes, block index $n$ ($n$ starts from 0), the size and dimension of feature maps. The per-pixel feature is extracted through bilinear interpolation on the feature map.  The code of DIFT (see `sd_featurizer.py` and `adm_featurizer.py`) is in the supplementary.

For semantic correspondence tasks, we use the input image size of 512x512 for DIFT$\_{adm}$, 768x768 for DIFT$\_{sd}$. n=4 for DIFT$
_{adm}$ so feature map size is 1/16 of input and dimension is 1024. n=1 for DIFT$\_{sd}$ so feature map size is 1/16 of input and dimension is 1280.

For experiments on HPatches, input image size is 768x768 for both DIFT$\_{adm}$ and DIFT$\_{sd}$. n=11 for DIFT$\_{adm}$ so feature map size is 1/2 of input and dimension is 512. n=2 for DIFT$\_{sd}$ so feature map size is 1/8 and dimension is 640.

For experiments on DAVIS, we use the same original video frame size (480p version of DAVIS, specific size varies across different videos) as in DINO [5]’s implementation, for both DIFT$\_{adm}$ and DIFT$\_{sd}$. n=7 for DIFT$\_{adm}$ so feature map size is 1⁄8 of input and dimension is 512. n=2 for DIFT$\_{sd}$ so feature map size is 1⁄8 of input and dimension is 640. For experiments on JHMDB, following CRW [29]’s implementation, we resize each video frame’s smaller size to 320 and keep the original aspect ratio. n=5 for DIFT$\_{adm}$ so feature map size is 1⁄8 of input and dimension is 1024. n=2 for DIFT$\_{sd}$ so feature map size is 1⁄8 of input and dimension is 640.

We'll add the above details to the future version of the paper.

**3. Latency of DIFT vs. its competitor self-supervised features**.

Since we only perform a single inference step when extracting DIFT features, it actually takes similar running time compared to its competitor self-supervised features with the same input image size. Taking semantic correspondence with the above configuration for example, on one single A6000, for each image, DIFT$\_{sd}$ takes 203 ms vs. OpenCLIP’s 231 ms, DIFT$\_{adm}$ takes 110 ms vs. DINO’s 154 ms. 

In practice, as mentioned in the Line 128-130 of Sec. 4.2, since there is randomness when extracting DIFT, we actually use a batch of random noise to get an averaged feature map for each image to slightly boost stability and performance, which would increase the running time shown above. But if computation is a bottleneck, one can remove this optimization at the cost of a tiny loss in performance: e.g., on SPair-71k, DIFT$\_{sd}$: PCK 59.5&rarr;57.9; DIFT$\_{adm}$: PCK 52.0&rarr;51.1.

We'll add the above clarification to the future version of the paper.

Meta-review: The majority of the reviews were positive and there was one review that was negative initially. After the rebuttal phase, most reviewers became more optimistic about the paper with the provided clarifications and the newly added results. It is thus strongly recommended that the rebuttal is integrated within the camera-ready. The negative reviewer never returned, but the comment on novelty is subjective and weak considering how other reviewers were convinced of the significance of the finding. The criticism of the experimental results is also weak as it is based on the weak-novelty argument, and asks for re-training of various architectures that is tangent to the main paper's argument. The AC hence is more in line with the other reviewers and recommends accepting the paper.
Decision: Accept (poster)