SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sensitivity to the choice of time step  $t$ . For  $\mathrm{DIFT}_{sd}$ , we plot how its PCK per point varies with different choices of  $t$  on SPair-71k in Fig. 5. DIFT is robust to the choice of  $t$  on semantic correspondence, as a wide range of  $t$  outperforms other off-the-shelf self-supervised features. Appendix B includes more discussion on how and why does  $t$  affect the nature of correspondence.
Figure 6: Edit propagation. The first column shows the source image with edits (i.e., the overlaid stickers), and the rest columns are the propagated results on new images from different instances, categories, and domains, respectively. Compared to OpenCLIP,  $\mathrm{DIFT}_{sd}$  propagates edits much more accurately. More results are in Fig. 20 of Appendix E.
5.3 Application: Edit Propagation
One application of DIFT is image editing: we can propagate edits in one image to others that share semantic correspondences. This capability is demonstrated in Fig. 6, where we showcase DIFT's ability to reliably propagate edits across different instances, categories, and domains, without any correspondence supervision.
To achieve this propagation, we simply compute a homography transformation between the source and target images using only matches found in the regions of the intended edits. By applying this transformation to the source image edits (e.g., an overlaid sticker), we can integrate them into the corresponding regions of the target image. Figure 6 shows the results for both OpenCLIP and  $\mathrm{DIFT}_{sd}$  using the same propagation techniques. OpenCLIP fails to compute reasonable transformation due to the lack of reliable correspondences. In contrast,  $\mathrm{DIFT}_{sd}$  achieves much better results, further justifying the effectiveness of DIFT in finding semantic correspondences.
6 Other Correspondence Tasks
We also evaluate DIFT on geometric correspondence and temporal correspondence. As in Sec. 5, we compare DIFT to other off-the-shelf self-supervised features as well as task-specific methods.
6.1 Geometric Correspondence
Intuitively, we find when  $t$  is small, DIFT focuses more on low-level details, which makes it useful as a geometric feature descriptor.
Setup. We evaluate DIFT for homography estimation on the HPatches benchmark [4]. It contains 116 sequences, where 57 sequences have illumination changes and 59 have viewpoint changes. Following [91], we extract a maximum of 1,000 keypoints from each image, and use cv2.findHomography() to estimate the homography from mutual nearest neighbor matches. For DIFT, we use the same set of keypoints detected by SuperPoint [16], as in CAPS [91].
For evaluation, we follow the corner correctness metric used in [91]: the four corners of one image are transformed into the other image using the estimated homography and are then compared with those computed using the ground-truth homography. We deem the estimation correct if the average error of the four corners is less than  $\epsilon$  pixels. Note that we do this evaluation on the original image resolution following [91], unlike [16].
Results. We report the accuracy comparison between DIFT and other methods in Tab. 4. Visualization of the matched points can be found in Fig. 7. Though not trained using any explicit geometry supervision, DIFT is still on par with the methods that utilize explicit geometric supervision signals
Table 4: Homography estimation accuracy [%] at 1, 3, 5 pixels on HPatches. Colors of numbers indicate the best, second-best results. All the DIFT results have gray background for better reference. DIFT with SuperPoint keypoints achieves competitive performance.
Method	Geometric Supervision	All	Viewpoint Change	Illumination Change
ε = 1	ε = 3	ε = 5	ε = 1	ε = 3	ε = 5	ε = 1	ε = 3	ε = 5
SIFT [51]	None	40.2	68.0	79.3	26.8	55.4	72.1	54.6	81.5	86.9
LF-Net [59]	34.4	62.2	73.7	16.8	43.9	60.7	53.5	81.9	87.7
SuperPoint [16]	36.4	72.7	82.6	22.1	56.1	68.2	51.9	90.8	98.1
D2-Net [19]	Strong	16.7	61.0	75.9	3.7	38.0	56.6	30.2	84.9	95.8
DISK [86]	40.2	70.6	81.5	23.2	51.4	67.9	58.5	91.2	96.2
ContextDesc [52]	40.9	73.0	82.2	29.6	60.7	72.5	53.1	86.2	92.7
R2D2 [66]	40.0	74.4	84.3	26.4	60.4	73.9	54.6	89.6	95.4
w/ SuperPoint kp.
CAPS [91]	Weak	44.8	76.3	85.2	35.7	62.9	74.3	54.6	90.8	96.9
DINO [10]	38.9	70.0	81.7	21.4	50.7	67.1	57.7	90.8	97.3
DIFTadm(ours)	None	43.7	73.1	84.8	26.4	57.5	74.3	62.3	90.0	96.2
OpenCLIP [36]	33.3	67.2	78.0	18.6	45.0	59.6	49.2	91.2	97.7
DIFTsd(ours)	45.6	73.9	83.1	30.4	56.8	69.3	61.9	92.3	98.1
Figure 7: Sparse feature matching using  $\mathrm{DIFT}_{sd}$  on HPatches after removing outliers with cv2.findHomography(). Left are image pairs under viewpoint change, and right are ones under illumination change. Although never trained with correspondence labels, it works well under both challenging changes. More results are in Fig. 21 of Appendix E.
designed specifically for this task, such as correspondences obtained from Structure-from-Motion [73] pipelines. This shows that not only semantic-level correspondence, but also geometric correspondence emerges from image diffusion models.
6.2 Temporal Correspondence
DIFT also demonstrates strong performance on temporal correspondence tasks, including video object segmentation and pose tracking, although never trained or fine-tuned on such video data.
Setup. We evaluate DIFT on two challenging video label propagation tasks: (1) DAVIS-2017 video instance segmentation benchmark [65]; (2) JHMDB keypoint estimation benchmark [40].
Following evaluation setups in [49, 37, 10, 95], representations are used as a similarity function: we segment scenes with nearest neighbors between consecutive video frames. Note that there is no training involved in this label propagation process. We report region-based similarity  $\mathcal{I}$  and contour-based accuracy  $\mathcal{F}$  [64] for DAVIS, and PCK for JHMDB.
Results. Table 5 reports the experimental results, comparing DIFT with other self-supervised features (pre-)trained with or without video data.  $\mathrm{DIFT}_{adm}$  outperforms all the other self-supervised learning methods on both benchmarks, even surpassing models specifically trained on video data by a significant margin. DIFT also yields the best results within the same pre-training dataset.
Table 5: Video label propagation results on DAVIS-2017 and JHMDB. Colors of numbers indicate the best, second-best results. All the DIFT results have gray background for better reference. DIFT even outperforms other self-supervised learning methods specifically trained with video data.
(pre-)Trained on Videos	Method	Dataset	DAVIS	JHMDB
J&Fm	Jm	Fm	PCK@0.1	PCK@0.2
X	InstDis [93]	66.4	63.9	68.9	58.5	80.2
MoCo [30]	65.9	63.4	68.4	59.4	80.9
SimCLR [12]	ImageNet [15]	66.9	64.4	69.4	59.0	80.8
BYOL [25]	w/o labels	66.5	64.0	69.0	58.8	80.9
SimSiam [13]	67.2	64.8	68.8	59.9	81.6
DINO [10]	71.4	67.9	74.9	57.2	81.2
DIFTadm (ours)	75.7	72.7	78.6	63.4	84.3
OpenCLIP [36]	62.5	60.6	64.4	41.7	71.7
DIFTsd (ours)	LAION [75]	70.0	67.4	72.5	61.1	81.8
✓	VINCE [24]	65.2	62.5	67.8	58.8	80.4
VFS [95]	68.9	66.5	71.3	60.9	80.7
UVC [49]	Kinetic [11]	60.9	59.3	62.7	58.6	79.6
CRW [37]	67.6	64.8	70.2	58.8	80.3
Colorization [88]	34.0	34.6	32.7	45.2	69.6
CorrFlow [44]	OxUvA [87]	50.3	48.4	52.2	58.5	78.8
TimeCycle [92]	VLOG [21]	48.7	46.4	50.0	57.3	78.1
MAST [43]	YT-VOS [96]	65.5	63.3	67.6	-	-
SFC [34]	71.2	68.3	74.0	61.9	83.0
Figure 8: Video label propagation results on DAVIS-2017. Colors indicate segmentation masks for different instances. Blue rectangles show the first frames. Compared to DINO,  $\mathrm{DIFT}_{adm}$  produces masks with more accurate and sharper boundaries. More results are in Fig. 22 of Appendix E.
We also show qualitative results in Fig. 8, presenting predictions of video instance segmentation in DAVIS, comparing  $\mathrm{DIFT}_{adm}$  with DINO.  $\mathrm{DIFT}_{adm}$  produces masks with clearer boundaries when single or multiple objects are presented in the scene, even attends well to objects in the presence of occlusion.
This paper demonstrates that correspondence emerges from image diffusion models without explicit supervision. We propose a simple technique to extract this implicit knowledge as a feature extractor named DIFT. Through extensive experiments, we show that although without any explicit supervision, DIFT outperforms both weakly-supervised methods and other off-the-shelf self-supervised features in identifying semantic, geometric and temporal correspondences, and even remains on par with the state-of-the-art supervised methods on semantic correspondence. We hope our work inspires future research on how to better utilize these emergent correspondence from image diffusion, as well as rethinking diffusion models as self-supervised learners.
Acknowledgement. This work was partially funded by NSF 2144117 and the DARPA Learning with Less Labels program (HR001118S0044). We would like to thank Zeya Peng for her help on the edit propagation section and the project page, thank Kamal Gupta for sharing the evaluation details in the ASIC paper, thank Aaron Gokaslan, Utkarsh Mall, Jonathan Moon, Boyang Deng, and all the anonymous reviewers for valuable discussion and feedback.
A Societal Impact
Although DIFT can be used with any diffusion model parameterized with a U-Net, the dominant publicly available model is the one trained on LAION [75]. The LAION dataset has been identified as having several issues including racial bias and stereotypes [7]. Diffusion models trained on these datasets inherit these issues. While these issues may a priori seem less important for estimating correspondences, it might lead to differing accuracies for different kinds of images. One could obtain the benefit of good correspondences without the associated issues if one could trained a diffusion model on a curated dataset. Unfortunately, the huge computational cost also prohibits the training of diffusion models in academic settings on cleaner datasets. We hope that our results encourage efforts to build more carefully trained diffusion models.
B Discussion
Why does correspondence emerge from image diffusion? One conjecture is that the diffusion training objective (i.e., coarse-to-fine reconstruction loss) requires the model to produce good, informative features for every pixel. This is in contrast to DINO and OpenCLIP that use image-level contrastive learning objectives. In our experiments, we have attempted to evaluate the importance of the training objective by specifically comparing  $\mathrm{DIFT}_{adm}$  and DINO in all our evaluations: two models that share exactly the same training data, i.e., ImageNet-1k without labels.
How and why does  $t$  affect the nature of correspondence? In Fig. 9, for the same clean image, we first add different amounts of noise to get different  $x_{t}$  following Eq. (2), then feed it into SD's denoising network  $\epsilon_{\theta}$  together with time step  $t$  to get the predicted clean image  $\hat{x}_0^t = \frac{x_t - (\sqrt{1 - \alpha_t})\epsilon_\theta(x_t,t)}{\sqrt{\alpha_t}}$ . We can see that, with the increase of  $t$ , more and more details are removed and only semantic-level features are preserved, and when  $t$  becomes too large, even the object structure is distorted. Intuitively, this explains why we need a small  $t$  for correspondences that requires details and a relatively large  $t$  for semantic correspondence.
input clean image
Figure 9: Within a reasonable range, when  $t$  gets larger, the predicted clean images remain the overall structure but have less details, suggesting DIFT contains more semantic-level information and less low-level features with the increase of  $t$ .
predicted clean images at different time step  $t$
How long does it take to run DIFT? Since we only perform a single inference step when extracting DIFT, it actually takes similar running time compared to competing self-supervised features with the same input image size. For example, when extracting features for semantic correspondence as in Sec. 5, on one single NVIDIA A6000 GPU,  $\mathrm{DIFT}_{sd}$  takes  $203\mathrm{ms}$  vs. OpenCLIP's  $231\mathrm{ms}$  on one single  $768\times 768$  image;  $\mathrm{DIFT}_{adm}$  takes  $110\mathrm{ms}$  vs. DINO's  $154\mathrm{ms}$  on one single  $512\times 512$  image. In practice, as mentioned in the last paragraph of Sec. 4.2, since there is randomness when extracting DIFT, we actually use a batch of random noise to get an averaged feature map for each image to slightly boost stability and performance, which would increase the running time shown above. But if computation is a bottleneck, one can remove this optimization at the cost of a tiny loss in performance: e.g., on SPair-71k,  $\mathrm{DIFT}_{sd}$ : PCK  $59.5\rightarrow 57.9$ ;  $\mathrm{DIFT}_{adm}$ : PCK  $52.0\rightarrow 51.1$ .
Would diffusion inversion help? Another way to get  $x_{t}$  from a real input image is diffusion inversion. Using DDIM inversion [77] to recover input image's corresponding  $x_{t}$  and then feeding into  $f_{\theta}$  to
get diffusion feature yielded similar results. At the same time, inversion makes the inference process several times slower. We leave how to utilize diffusion inversion to get better correspondence to future work.
Does correspondence information exist in SD's encoder? We also evaluated SD's VAE encoder's performance on all benchmarks and found that its performance was lower by an order of magnitude. So  $\mathrm{DIFT}_{sd}$  's correspondence only emerges inside its U-Net and requires diffusion-based training.
Would task-specific adaptation lead DIFT to better results? More sophisticated mechanisms could be applied to further enhance the diffusion features, e.g., concatenating and re-weighting features from different time step  $t$  and different network layers, or even fine-tuning the network with task-specific supervision. Some recent works [5, 94, 101] fine-tune either the U-Net or the attached head for dense prediction tasks and yield better performance. However, task-specific adaptation entangles the quality of the features themselves with the efficacy of the fine-tuning procedure. To keep the focus on the representation, we chose to avoid any fine-tuning to demonstrate the quality of the off-the-shelf DIFT. Nevertheless, our preliminary experiments suggest that such fine-tuning would indeed further improve performance on correspondence. We'll leave how to better adapt DIFT to downstream tasks to future work.
The total time step  $T$  for both diffusion models (ADM and SD) is 1000. U-Net consists of downsampling blocks, middle blocks and upsampling blocks. We only extract features from the upsampling blocks. ADM's U-Net has 18 upsampling blocks and SD's U-Net has 4 upsampling blocks (the definition of blocks are different between these two models). Feature maps from the  $n$ -th upsampling block output are used as the final diffusion feature. For a fair comparison, we also grid-search which layer to extract feature for DINO and OpenCLIP for each task, and report the best results among the choices.
As mentioned in the last paragraph of Sec. 4.2, when extracting features for one single image using DIFT, we use a batch of random noise to get an averaged feature map. The batch size is 8 by default. We shrink it to 4 when encountering GPU memory constraints.
The input image resolution varies across different tasks but we always keep it the same within the comparison vs. other off-the-shelf self-supervised features (i.e.,  $\mathrm{DIFT}_{adm}$  vs. DINO,  $\mathrm{DIFT}_{sd}$  vs. OpenCLIP) thus the comparisons are fair. For DIFT, feature map size and dimension also depend on which U-Net layer features are extracted from.
The following sections list the time step  $t$  and upsampling block index  $n$  ( $n$  starts from 0) we used for each DIFT variant on different tasks as well as input image resolution and output feature map tensor shape.
C.1 Semantic Correspondence
We use  $t = 101$  and  $n = 4$  for  $\mathrm{DIFT}_{adm}$  on input image resolution  $512 \times 512$  so feature map size is  $1/16$  of input and dimension is  $1024$ ; we use  $t = 261$  and  $n = 1$  for  $\mathrm{DIFT}_{sd}$  on input image resolution  $768 \times 768$  so feature map size is  $1/16$  of input and dimension is  $1280$ . These hyper-parameters are shared on all semantic correspondence tasks including SPair-71k, PF-WILLOW, and CUB, as well as the visualizations in Figs. 1, 15 and 16.
We don't use image-specific prompts for  $\mathrm{DIFT}_{sd}$ . Instead, we use a general prompt "a photo of a [class]" where [class] denotes the string of the input images' category, which is given by the dataset. For example, for the images of SPair-71k under cat class, the prompt would be "a photo of a cat". For CUB, the same prompt is used for all images: "a photo of a bird". Changing per-class prompt to a null prompt (empty string "") will only lead a very small performance drop, e.g.,  $\mathrm{DIFT}_{sd}$ 's PCK per point on SPair-71k:  $59.5 \rightarrow 57.6$ .
C.2 Geometric Correspondence
On HPatches, the input images are resized to  $768 \times 768$  to extract features for both  $\mathrm{DIFT}_{adm}$  and  $\mathrm{DIFT}_{sd}$ . We use  $t = 41$ ,  $n = 11$  for  $\mathrm{DIFT}_{adm}$  so feature map size is  $1/2$  of input and dimension is 512; we use  $t = 0$ ,  $n = 2$  for  $\mathrm{DIFT}_{sd}$  so feature map size is  $1/8$  of input and dimension is 640.
In addition, for  $\mathrm{DIFT}_{sd}$ , each image's prompt is a null prompt, i.e., an empty string ""
For all the methods listed in Tab. 4, when doing homography estimation, we tried both cosine and L2 distance for mutual nearest neighbor matching, and both RANSAC and LMEDS for cv2.findHomography(), and eventually we report the best number among these choices for each method.
C.3 Temporal Correspondence
The configurations we use for  $\mathrm{DIFT}_{adm}$  and  $\mathrm{DIFT}_{sd}$  are:
Dataset	Method	Time step t	Block index n	Temperature for softmax	Propagation radius	k for top-k	Number of prev. frames
DAVIS-2017	DIFTadm	51	7	0.1	15	10	28
DAVIS-2017	DIFTsd	51	2	0.2	15	15	28
JHMDB	DIFTadm	101	5	0.2	5	15	28
JHMDB	DIFTsd	51	2	0.1	5	15	14
For experiments on DAVIS, we use the same original video frame size (480p version of DAVIS, specific size varies across different videos) as in DINO's implementation [10], for both  $\mathrm{DIFT}_{adm}$  and  $\mathrm{DIFT}_{sd}$ .  $n = 7$  for  $\mathrm{DIFT}_{adm}$  so feature map size is  $1/8$  of input and dimension is 512.  $n = 2$  for  $\mathrm{DIFT}_{sd}$  so feature map size is  $1/8$  of input and dimension is 640. For experiments on JHMDB, following CRW's implementation [37], we resize each video frame's smaller side to 320 and keep the original aspect ratio.  $n = 5$  for  $\mathrm{DIFT}_{adm}$  so feature map size is  $1/8$  of input and dimension is 1024.  $n = 2$  for  $\mathrm{DIFT}_{sd}$  so feature map size is  $1/8$  of input and dimension is 640.
In addition, for  $\mathrm{DIFT}_{sd}$ , each image's prompt is a null prompt, i.e., an empty string "".
D.1 Semantic Correspondence on PF-PASCAL
We didn't do evaluation on PF-PASCAL [27] in the main paper because we found over half of the test images (i.e., 302 out of 506) actually also appear in the training set, which makes the benchmark numbers much less convincing, and also partially explains why the previous supervised methods tend to have much higher test accuracy on PF-PASCAL vs. PF-WILLOW (e.g., over 90 vs. around 70) even using exactly the same trained model. And this duplication issue of train/test images also gives huge unfair disadvantage to the methods that are never adapted (either supervised or unsupervised) on the training set before evaluation.
However, even in this case, as shown in Tab. 6, DIFT still demonstrates competitive performance compared to the state-of-the-art weakly-supervised method PWarpC [83], as well as huge gains vs. other off-the-shelf self-supervised features.
Table 6: PCK per image on PF-PASCAL. The groupings and colors follow Tab. 1.
Sup.	Method	PCK@αimg
α = 0.05	α = 0.10
(b)	PWarpC [83]	64.2	84.4
(c)	DINO [10]	36.9	53.6
DIFTadm (ours)	56.5	72.5
OpenCLIP [36]	39.8	61.1
DIFTsd (ours)	69.4	84.6
Figure 10: Mean Matching Accuracy (MMA) on HPatches [4]. For each method, we show the MMA with varying pixel error thresholds. We also report the mean number of detected features and mutual nearest neighbor matches. Although not trained with any explicit geometric correspondence labels,  $\mathrm{DIFT}_{sd}$  is able to achieve competitive performance compared to other feature descriptors that are specifically designed or trained for this task.
Methods	# Feature	# Match
HesAff + Root-SIFT	6.7K	2.8K
\( HAN + HN++ \)	3.9K	2.0K
LF-Net	0.5K	0.2K
SuperPoint	1.7K	0.9K
DELF	4.6K	1.9K
SIFT + ContextDesc	4.4K	1.9K
D2-Net MS	8.3K	2.8K
R2D2	5.0K	1.8K
DISK (2k)	2.0K	1.0K
SuperPoint + CAPS	1.7K	1.0K
SuperPoint + \( DIFT_{sd} \)	1.7K	0.9K
Figure 11: Homography estimation accuracy  $[\%]$  at 1, 3, 5 pixels on HPatches using  $\mathrm{DIFT}_{sd}$  with different time step  $t$ . Intuitively, as  $t$  gets larger, DIFT contains more semantic information and less low-level details, so the accuracy decreases when using larger  $t$ .
D.2 Feature Matching on HPatches
Following CAPS [91], for the given image pair, we extract SuperPoint [16] keypoints in both images and match them using our proposed feature descriptor,  $\mathrm{DIFT}_{sd}$ . We follow the evaluation protocol as in [19, 91] and use Mean Matching Accuracy (MMA) as the evaluation metric, where only mutual nearest neighbors are considered as matched points. The MMA score is defined as the average percentage of correct matches per image pair under a certain pixel error threshold. Fig. 10 shows the comparison between DIFT and other feature descriptors that are specially designed or trained for geometric correspondence. We report the average results for the whole dataset, as well as subsets on illumination and viewpoint changes respectively. For each method, we also present the mean number of detected features per image and mutual nearest neighbor matches per image pair. We can see that, although not trained with any explicit geometry supervision, DIFT is still able to achieve competitive performance.
D.3 Analysis on Hyper-parameters
Here we analyze how the choice of time step and U-Net layer would affect DIFT's performance on different correspondence tasks.
Ablation on time step. Similar to Fig. 5, we plot how HPatches homography estimation accuracy and DAVIS video label propagation accuracy vary with different choices of  $t$ , in Figs. 11 and 12
Figure 12: Video label propagation accuracy  $(\mathcal{J}\& \mathcal{F}_{\mathrm{m}})$  on DAVIS using  $\mathrm{DIFT}_{adm}$  with different time step  $t$ . There's a wide range of  $t$ , where DIFT maintains a stable and competitive performance.
Figure 13: PCK per point on SPair-71k using  $\mathrm{DIFT}_{sd}$  with different layer  $i$  inside U-Net's 15 upsampling layers in total. The transition from block index  $n$  to layer index  $i$  is  $0/1/2/3$  to  $3/7/11/14$  respectively. We can see there are multiple choices of  $i$  leading to good performance.
respectively. Both curves have smooth transitions and there's a large range of  $t$  where DIFT gives competitive performance.
Ablation on U-Net layer. Compared to the definition of 4 block choices in Appendix C, here we make a more fine-grained sweep over SD's 15 layers inside U-Net upsampling blocks. The transition from block index  $n$  to layer index  $i$  is  $0/1/2/3$  to  $3/7/11/14$  respectively and both start from 0. We evaluate PCK per point on SPair-71k using  $\mathrm{DIFT}_{sd}$  with different layer index  $i$ . As shown in Fig. 13, the accuracy varies but there are still multiple choices of  $i$  that lead to good performance.
E Additional Qualitative Results
PCA visualization of DIFT. In Fig. 14, for each pair of images, we extract  $\mathrm{DIFT}_{sd}$  from the segmented instances, then compute PCA and visualize the first 3 components, where each component serves as a color channel. We can see the same object parts share similar embeddings, which also demonstrates the emergent correspondence.