SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Luming Tang* Menglin Jia* Qianqian Wang* Cheng Perng Phoo Bharath Hariharan Cornell University
Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIFusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io.
Drawing correspondences between images is a critical primitive in 3D reconstruction [73], object tracking [22, 90], video segmentation [92], image and video editing [102, 58, 98]. This problem of drawing correspondence is easy for humans: we can match object parts not only across different viewpoints, articulations and lighting changes, but even across drastically different categories (e.g., between cats and horses) or different modalities (e.g., between photos and cartoons). Yet, we rarely if ever get explicit correspondence labels for training. The question is, can computer vision systems similarly learn accurate correspondences without any labeled data at all?
There is indeed some evidence that contrastive self-supervised learning techniques produce good correspondences as a side product of learning on unlabeled data [10, 28]. However, in this paper, we look to a new class of self-supervised models that has been attracting attention: diffusion-based generative models [32, 79]. While diffusion models are primarily models for image synthesis, a key observation is that these models produce good results for image-to-image translation [53, 85] and image editing [8, 80]. For instance, they can convert a dog to a cat without changing its pose or context [61]. It would appear that to perform such editing, the model must implicitly reason about correspondence between the two categories (e.g., the model needs to know where the dog's eye is in order to replace it with the cat's eye). We therefore ask, do image diffusion models learn correspondences?
We answer the question in the affirmative by construction: we provide a simple way of extracting correspondences on real images using pre-trained diffusion models. These diffusion models [41] have at the core a U-Net [71, 17, 70] that takes noisy images as input and produces clean images as output. As such they already extract features from the input image that can be used for correspondence. Unfortunately, the U-Net is trained to de-noise, and so has been trained on noisy images. Our strategy
Figure 1: Given a red source point in an image (far left), we would like to develop a model that automatically finds the corresponding point in the images on the right. Without any fine-tuning or correspondence supervision, our proposed diffusion features (DIFT) could establish semantic correspondence across instances, categories and even domains, e.g., from a duck to a penguin, from a photo to an oil-painting. More results are in Figs. 15 and 16 of Appendix E.
for handling this issue is simple but effective: we add noise to the input image (thus simulating the forward diffusion process) before passing it into the U-Net to extract feature maps. We call these feature maps (and through a slight abuse of notation, our approach) DIfusion FeaTures (DIFT). DIFT can then be used to find matching pixel locations in the two images by doing simple nearest neighbor lookup using cosine distance. We find the resulting correspondences are surprisingly robust and accurate (Fig. 1), even across multiple categories and image modalities.
We evaluate DIFT with two different types of diffusion models, on three groups of visual correspondence tasks including semantic correspondence, geometric correspondence, and temporal correspondence. We compare DIFT with other baselines, including task-specific methods, and other self-supervised models trained with similar datasets and similar amount of supervision (DINO [10] and OpenCLIP [36]). Although simple, DIFT demonstrates strong performance on all tasks without any additional fine-tuning or supervision, outperforms both weakly-supervised methods and other self-supervised features, and even remains on par with the state-of-the-art supervised methods on semantic correspondence.
Visual Correspondence. Establishing visual correspondences between different images is crucial for various computer vision tasks such as Structure-from-Motion / 3D reconstruction [2, 73, 60, 74], object tracking [22, 97], image recognition [63, 81, 9] and segmentation [50, 47, 72, 28]. Traditionally, correspondences are established using hand-designed features, such as SIFT [51] and SURF [6]. With the advent of deep learning, methods that learn to find correspondences in a supervised-learning regime have shown promising results [46, 14, 42, 35]. However, these approaches are difficult to scale due to the reliance on ground-truth correspondence annotations. To overcome difficulties in collecting a large number of image pairs with annotated correspondences, recent works have started looking into how to build visual correspondence models with weak supervision [91] or self-supervision [92, 37]. Meanwhile, recent works on self-supervised representation learning [10] has yielded strong per-pixel features that could be used to identify visual correspondence [84, 3, 10, 28]. In particular, recent work has also found that the internal representation of Generative Adversarial Networks (GAN) [23] could be used for identifying visual correspondence [99, 62, 57] within certain image categories. Our work shares similar spirits with these works: we show that diffusion models could generate features that are useful for identifying visual correspondence on general images. In addition, we show that features generated at different timesteps and different layers of the de-noising process
encode different information that could be used for determining correspondences needed for different downstream tasks.
Diffusion Model [78, 32, 79, 41] is a powerful family of generative models. Ablated Diffusion Model [17] first showed that diffusion could surpass GAN's image generation quality on ImageNet [15]. Subsequently, the introduction of classifier-free guidance [33] and latent diffusion model [70] made it scale up to billions of text-image pairs [75], leading to the popular open-sourced text-to-image diffusion model, i.e., Stable Diffusion. With its superior generation ability, recently people also start investigating the internal representation of diffusion models. For example, previous works [85, 31] found that the intermediate-layer features and attention maps of diffusion models are crucial for controllable generations; other works [5, 94, 101] explored adapting pre-trained diffusion models for various downstream visual recognition tasks. Different from these works, we are the first to directly evaluate the efficacy of features inherent to pre-trained diffusion models on various visual correspondence tasks.
Given two images  $I_1, I_2$  and a pixel location  $p_1$  in  $I_1$ , we are interested in finding its corresponding pixel location  $p_2$  in  $I_2$ . Relationships between  $p_1$  and  $p_2$  could be semantic correspondence (i.e., pixels of different objects that share similar semantic meanings), geometric correspondence (i.e., pixels of the same object captured from different viewpoints), or temporal correspondence (i.e., pixels of the same object in a video that may deform over time).
The most straightforward approach to obtaining pixel correspondences is to first extract dense image features in both images and then match them. Specifically, given a feature map  $F_{i}$  for image  $I_{i}$ , we can extract a feature vector  $F_{i}(p)$  for pixel location  $p$  through bilinear interpolation. Then given a pixel  $p_{1}$  in image  $I_{1}$ , we can obtain the corresponding pixel in image  $I_{2}$  as:
$$
p _ {2} = \underset {p} {\arg \min } d \left(F _ {1} \left(p _ {1}\right), F _ {2} (p)\right) \tag {1}
$$
where  $d$  is a distance metric and we use cosine distance by default in this work.
4 Diffusion Features (DIFT)
In this section, we first review what diffusion models are and then explain how we extract dense features on real images using pre-trained diffusion models.
4.1 Image Diffusion Model
Diffusion models [32, 79] are generative models that aim to transform a Normal distribution to an arbitrary data distribution. In our case, we use image diffusion models, thus the data distribution and the Gaussian prior are both over the space of 2D images.
During training, Gaussian noise of different magnitudes is added to clean data points to obtain noisy data points. This is typically thought of as a "diffusion" process, where the starting point of the diffusion  $x_0$  is a clean image from the training dataset and  $x_{t}$  is a noisy image obtained by "mixing"  $x_0$  with noise:
$$
x _ {t} = \sqrt {\alpha_ {t}} x _ {0} + (\sqrt {1 - \alpha_ {t}}) \epsilon \tag {2}
$$
where  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$  is the randomly-sampled noise, and  $t\in [0,T]$  indexes "time" in the diffusion process with larger time steps involving more noise. The amount of noise is determined by  $\{\alpha_t\}_1^T$  which is a pre-defined noise schedule. We call this the diffusion forward process.
A neural network  $f_{\theta}$  is trained to take  $x_{t}$  and time step  $t$  as input and predict the input noise  $\epsilon$ . For image generation,  $f_{\theta}$  is usually parametrized as a U-Net [71, 17, 70]. Once trained,  $f_{\theta}$  can be used to "reverse" the diffusion process. Starting from pure noise  $x_{T}$  sampled from a Normal distribution,  $f_{\theta}$  can be iteratively used to estimate noise  $\epsilon$  from the noisy data  $x_{t}$  and remove this noise to get a cleaner data  $x_{t - 1}$ , eventually leading to a sample  $x_{0}$  from the original data distribution. We call this the diffusion backward process.
Source Point
Predicted Target Point
Figure 2: Given a Stable Diffusion generated image, we extract its intermediate layer activations at a certain time step  $t$  during its backward process, and use them as the feature map to predict the corresponding points. Although simple, this method produces correct correspondences on generated images already not only within category, but also cross-category, even in cross-domain situations, e.g., from a photo to an oil painting.
4.2 Extracting Diffusion Features on Real Images
We hypothesize that diffusion models learn correspondence implicitly [85, 61] in Sec. 1, but how can we extract this correspondence? Consider first generated images, where we have access to the complete internal state of the network throughout the entire backward process. Given a generated image from Stable Diffusion [70], we extract the feature maps of its intermediate layers at a specific time step  $t$  during the backward process, which we then utilize to establish correspondences between two different generated images as described in Sec. 3. As illustrated in Fig. 2, this straightforward approach allows us to find correct correspondences between generated images, even when they belong to different categories or domains.
Replicating this approach for real images is challenging because of the fact that the real image itself does not belong to the training distribution of the U-Net (which was trained on noisy images), and we do not have access to the intermediate noisy images that would have been produced during the generation of this image. Fortunately, we found a simple approximation using the forward diffusion process to be effective enough. Specifically, we first add noise of time step  $t$  to the real image (Eq. (2)) to move it to the  $x_{t}$  distribution, and then feed it to network  $f_{\theta}$  together with  $t$  to extract the intermediate layer activations as our DIFusion FeaTures, namely DIFT. As shown in Figs. 1 and 3, this approach yields surprisingly good correspondences for real images.
Moving forward, a crucial consideration is the selection of the time step  $t$  and the network layer from which we extract features. Intuitively we find that a larger  $t$  and an earlier network layer tend to yield more semantically-aware features, while a smaller  $t$  and a later layer focus more on low-level details. The optimal choices of  $t$  and layer depend on the specific correspondence task at hand, as different tasks may require varying trade-offs between semantic and low-level features. For example, semantic correspondence likely benefits from more semantic-level features, whereas geometric correspondence between two views of the same instance may perform well with low-level features. We therefore use a 2D grid search to determine these two hyper-parameters for each correspondence task. For a comprehensive list of the hyper-parameter values used in this paper, please refer to Appendix C.
Lastly, to enhance the stability of the representation in the presence of random noise added to the input image, we extract features from multiple noisy versions with different samples of noise, and average them to form the final representation.
5 Semantic Correspondence
In this section, we investigate how to use DIFT to identify pixels that share similar semantic meanings across images, e.g., the eyes of two different cats in two different images.
5.1 Model Variants and Baselines
We extract DIFT from two commonly used, open-sourced image diffusion models: Stable Diffusion 2-1 (SD) [70] and Ablated Diffusion Model (ADM) [17]. SD is trained on the LAION [75] whereas ADM is trained on ImageNet [15] without labels. We call these two features  $\mathrm{DIF}_{sd}$  and  $\mathrm{DIF}_{adm}$  respectively.
To separate the impact of training data on the performance of DIFT, we also evaluate two other commonly used self-supervised features as baselines that share basically the same training data:
Figure 3: Visualization of semantic correspondence prediction on SPair-71k using different features. The leftmost image is the source image with a set of keypoints; the rightmost image contains the ground-truth correspondence for a target image whereas any images in between contain keypoints found using feature matching with various features. Different colors indicate different keypoints. We use circles to indicate correctly-predicted points under the threshold  $\alpha_{bbox} = 0.1$  and crosses for incorrect matches. DIFT is able to establish correct correspondences under clustered scenes (row 3), viewpoint changes (row 2 and 4), and occlusions (row 5). See Fig. 17 in Appendix E for more results.
OpenCLIP [36] with ViT-H/14 [18] trained on LAION, as well as DINO [10] with ViT-B/8 trained on ImageNet [15] without labels. Note that for both DIFT and other self-supervised features, we do not fine-tune or re-train the models with any additional data or supervision.
5.2 Benchmark Evaluation
Datasets. We conduct evaluation on three popular benchmarks: SPair-71k [55], PF-WILLOW [27] and CUB-200-2011 [89]. SPair-71k is the most challenging semantic correspondence dataset, containing diverse variations in viewpoint and scale with 12,234 image pairs on 18 categories for testing. PF-Willow is a subset of PASCAL VOC dataset [20] with 900 image pairs for testing. For CUB, following [58], we evaluate 14 different splits of CUB (each containing 25 images) and report the average performance across all splits.
Evaluation Metric. Following prior work, we report the percentage of correct keypoints (PCK). The predicted keypoint is considered to be correct if they lie within  $\alpha \cdot \max(h, w)$  pixels from the ground-truth keypoint for  $\alpha \in [0, 1]$ , where  $h$  and  $w$  are the height and width of either the image  $(\alpha_{img})$  or the bounding box  $(\alpha_{bbox})$ . To find a suitable time step and layer feature to use for DIFT and other self-supervised features, we grid search the hyper-parameters using SPair-71k and use the same hyper-parameter settings for PF-WILLOW and CUB.
We observed inconsistencies in PCK measurements across prior literature<sup>1</sup>. Some works [35, 42, 14] use the total number of correctly-predicted points in the whole dataset (or each category split) divided
1 ScorrSAN [35] and GANgealing [62]'s evaluation code snippets, which calculate PCK per image and PCK per point respectively.
Table 1: PCK(αbbox = 0.1) per image on SPair-71k. All the DIFT results have gray background for easy lookups. Methods are grouped into 3 groups: (a) fully supervised with correspondence annotations, (b) weakly supervised with in-domain image collections, (c) no supervision. Best numbers in group (a) are bolded. Among groups (b) and (c) taken together, we annotate best and second-best results. Without any supervision, both  $\mathrm{DIFT}_{sd}$  and  $\mathrm{DIFT}_{adm}$  outperform previous weakly-supervised methods and self-supervised techniques by a large margin.
Sup.	Method	SPair-71K Category
Aero	Bike	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow	Dog	Horse	Motor	Person	Plant	Sheep	Train	TV	All
(a)	CATs [14]	52.0	34.7	72.2	34.3	49.9	57.5	43.6	66.5	24.4	63.2	56.5	52.0	42.6	41.7	43.0	33.6	72.6	58.0	49.9
MMNet [100]	55.9	37.0	65.0	35.4	50.0	63.9	45.7	62.8	28.7	65.0	54.7	51.6	38.5	34.6	41.7	36.3	77.7	62.5	50.4
TransforMatcher [42]	59.2	39.3	73.0	41.2	52.5	66.3	55.4	67.1	26.1	67.1	56.6	53.2	45.0	39.9	42.1	35.3	75.2	68.6	53.7
SCorrSAN [35]	57.1	40.3	78.3	38.1	51.8	57.8	47.1	67.9	25.2	71.3	63.9	49.3	45.3	49.8	48.8	40.3	77.7	69.7	55.3
(b)	NCNet [69]	17.9	12.2	32.1	11.7	29.0	19.9	16.1	39.2	9.9	23.9	18.8	15.7	17.4	15.9	14.8	9.6	24.2	31.1	20.1
CNGeo [67]	23.4	16.7	40.2	14.3	36.4	27.7	26.0	32.7	12.7	27.4	22.8	13.7	20.9	21.0	17.5	10.2	30.8	34.1	20.6
WeakAlign [68]	22.2	17.6	41.9	15.1	38.1	27.4	27.2	31.8	12.8	26.8	22.6	14.2	20.0	22.2	17.9	10.4	32.2	35.1	20.9
A2Net [76]	22.6	18.5	42.0	16.4	37.9	30.8	26.5	35.6	13.3	29.6	24.3	16.0	21.6	22.8	20.5	13.5	31.4	36.5	22.3
SFNet [45]	26.9	17.2	45.5	14.7	38.0	22.2	16.4	55.3	13.5	33.4	27.5	17.7	20.8	21.1	16.6	15.6	32.2	35.9	26.3
PMD [48]	26.2	18.5	48.6	15.3	38.0	21.7	17.3	51.6	13.7	34.3	25.4	18.0	20.0	24.9	15.7	16.3	31.4	38.1	26.5
PSCNet [38]	28.3	17.7	45.1	15.1	37.5	30.1	27.5	47.4	14.6	32.5	26.4	17.7	24.9	24.5	19.9	16.9	34.2	37.9	27.0
PwarpC [83]	37.4	28.8	60.8	22.9	40.5	29.4	22.8	60.1	19.5	37.8	38.4	27.9	32.1	29.7	29.2	20.2	44.5	50.0	35.3
(c)	DINO [10]	43.6	27.2	64.9	24.0	30.5	31.4	28.3	55.2	16.8	40.2	37.1	32.9	29.1	41.1	22.0	26.8	36.4	26.9	33.9
\( \text{DIFF}_{adm} \) (ours)	49.7	39.2	77.5	29.3	40.9	36.1	30.5	75.5	23.7	63.7	52.8	49.3	34.1	52.3	39.3	37.3	59.6	45.4	46.3
OpenCLIP [36]	51.7	31.4	68.7	28.4	31.5	34.9	36.1	56.4	21.1	44.5	41.5	41.2	41.2	51.8	21.7	28.6	46.3	20.7	38.4
\( \text{DIFF}_{sd} \) (ours)	61.2	53.2	79.5	31.2	45.3	39.8	33.3	77.8	34.7	70.1	51.5	57.2	50.6	41.4	51.9	46.0	67.6	59.5	52.9
Table 2: PCK  $(\alpha_{bbox} = 0.1)$  per point of various methods on SPair-71k. The groups and colors follow Tab. 1. "Mean" denotes the PCK averaged over categories. Same as in Tab. 1, without any supervision, both  $\mathrm{DIFT}_{sd}$  and  $\mathrm{DIFT}_{adm}$  outperform previous weakly-supervised methods with a large margin, and also outperform their contrastive-learning counterparts by over 14 points.
Sup.	Method	SPair-71K Category
Aero	Bike	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow	Dog	Horse	Motor	Person	Plant	Sheep	Train	TV	Mean	All
(b)	NBB [1, 26]	29.5	22.7	61.9	26.5	20.6	25.4	14.1	23.7	14.2	27.6	30.0	29.1	24.7	27.4	19.1	19.3	24.4	22.6	27.4	-
GANgealing [62]	-	37.5	-	-	-	-	-	67.0	-	-	23.1	-	-	-	-	-	-	57.9	-	-
NeuCongeal [58]	-	29.1	-	-	-	-	-	53.3	-	-	35.2	-	-	-	-	-	-	-	-	-
ASIC [26]	57.9	25.2	68.1	24.7	35.4	28.4	30.9	54.8	21.6	45.0	47.2	39.9	26.2	48.8	14.5	24.5	49.0	24.6	36.9	-
(c)	DINO [10]	45.0	29.5	66.3	22.8	32.1	36.3	31.7	54.8	18.7	43.1	39.2	34.9	31.0	44.3	23.1	29.4	38.4	27.1	36.0	36.7
\( \text{DIFT}_{adm} \) (ours)	51.6	40.4	77.6	30.7	43.0	47.2	42.1	74.9	26.6	67.3	55.8	52.7	36.0	55.9	46.3	45.7	62.7	47.4	50.2	52.0
OpenCLIP [36]	53.2	33.4	69.4	28.0	33.3	41.0	41.8	55.8	23.3	47.0	43.9	44.1	43.5	55.1	23.6	31.7	47.8	21.8	41.0	41.4
\( \text{DIFT}_{sd} \) (ours)	63.5	54.5	80.8	34.5	46.2	48.3	77.7	39.0	76.0	54.9	61.3	53.3	46.0	57.8	57.1	71.1	63.4	57.7	59.5
Table 3: Comparison with state-of-the-art methods on PF-WILLOW PCK per image (left) and CUB PCK per point (right). The groups follow Tab. 1. Colors of numbers indicate the best, second-best results. All the DIFT results have gray background for better reference.  $\mathrm{DIF}_{sd}$  achieves the best results without any fine-tuning or supervision with in-domain annotations or data.
Sup.	Method	PCK@αbbox
α = 0.05	α = 0.10
(a)	SCNet [29]	38.6	70.4
DHPF [56]	49.5	77.6
PMD [48]	-	75.6
CHM [54]	52.7	79.4
CATs [14]	50.3	79.2
TransforMatcher [42]	-	76.0
SCorrSAN [35]	54.1	80.0
(b)	WarpC [82]	49.0	75.1
PWarpC [83]	45.0	75.9
GSF [39]	49.1	78.7
(c)	DINO [10]	30.8	51.1
DIFTadm (ours)	46.9	67.0
OpenCLIP [36]	34.4	61.3
DIFTsd (ours)	58.1	81.2
Sup.	Method	PCK@αimg=0.1
(b)	GANgealing [62]	56.8
NeuCongeal [58]	65.6
(c)	DINO [10]	66.4
DIFTadm (ours)	78.0
OpenCLIP [36]	67.5
DIFtsd (ours)	83.5
by the total number of predicted points as the final PCK, while some works [62, 58, 26] first calculate a PCK value for each image and then average it across the dataset (or each category split). We denote the first metric as PCK per point and the second as PCK per image. We calculate both metrics for DIFT and self-supervised features, and compare them to methods using that metric respectively.
Quantitative Results. We report our results in Tabs. 1 to 3. In addition to feature matching using DINO and OpenCLIP, we also report state-of-the-art fully-supervised and weakly-supervised methods in the respective tables for completeness. Across the three datasets, we observe that features learned via diffusion yield more accurate correspondences compared to features learned using contrastive approaches (DIFTsd vs. OpenCLIP,  $\mathrm{DIFT}_{adm}$  vs. DINO).
Source patch
Figure 4: Given image patch specified in the leftmost image (red rectangle), we use  $\mathrm{DIFT}_{sd}$  to retrieve the top-5 nearest patches in images from different categories in the SPair-71k test set. DIFT is able to find correct correspondence for different objects sharing similar semantic parts, e.g., the wheel of an airplane vs. the wheel of a bus. More results are in Fig. 18 of Appendix E.
Top-5 nearest neighbor cross-category target patches predicted by  $\mathrm{DIFT}_{sd}$
Figure 5: PCK per point of  $\mathrm{DIFT}_{sd}$  on SPair-71k. It maintains high accuracy with a wide range of  $t$ , outperforming other off-the-shelf self-supervised features.
Furthermore, even without any supervision (be it explicit correspondence or in-domain data), DIFT outperforms all the weakly-supervised baselines on all benchmarks by a large margin. It even outperforms the state-of-the-art supervised methods on PF-WILLOW, and for 9 out of 18 categories on SPair-71k.
Qualitative Results. To get a better understanding of DIFT's performance, we visualize a few correspondences on SPair-71k using various off-the-shelf features in Fig. 3. We observe that DIFT is able to identify correct correspondences under cluttered scenes, viewpoint changes, and instance-level appearance changes.
In addition to visualizing correspondence within the same categories in SPair-71k, we also visualize the correspondence established using  $\mathrm{DIFT}_{sd}$  across various categories in Fig. 4. Specifically, we select an image patch from a random image and query the image patches with the nearest DIFT embedding in the rest of the test split but from different categories. DIFT is able to identify correct correspondence across various categories.