A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Authors: Junyi Zhang
Charles Herrmann
Junhwa Hur
Luisa Polania Cabrera
Varun Jampani
Deqing Sun
Ming-Hsuan Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Semantic Correspondence
Diffusion Models
Vision Transformer
Representation

Abstract: Text-to-image diffusion models have made significant advances in generating and editing high-quality images.  As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects.  In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS.  We also show that these correspondences can enable interesting applications such as instance swapping in two images. Project page: https://sd-complements-dino.github.io/.

A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Junyi Zhang'
Charles Herrmann
Junhwa Hur
Luisa F. Polanía"
Varun Jampani?
Deqing Sun?
Ming-Hsuan Yang2 2.3
Shanghai Jiao Tong University
2 Google Research
UC Merced
https://sd-comg: http:/dd-ccommlmme---deeeiee wwi.ddiiinnniiii-.o github.uh..
Abstract
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multi- ple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post- processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, our analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches, We demonstrate that simple fusion of the features works surprisingly well, and a zero-shot evaluation via nearest neighbor search on the fused features provides a significant performance gain over ttate-offie of-the-art methods on benchmark datasets, e.g., SPair-7Ik, PF-Pascal, and TSS. We also show that these correspondences enable high-quality object swapping without task -asc-ipec fine-tuning.
Introduction
Recent findings from text-to- tot-too-mmag diffusion models [44, 48, 50) have attracted increasing attention to analyze and understand their internal representations for single image. Due the -ot-tto-image diffusion model's 's ability to effectively connect a text prompt to the content in the images, they should be able to understand what image contains and effective classifiers. Furthermore, since these models can generate specific scenes and have some degree of control over the layout and placement of objects [72]. they should be able to localize objects and were shown in [68] to be effective for semantic segmentation. In addition, several other methods [12. 17, 27, 65] focus on accessing and editing features for downstream tasks such as stylization and image editing.
recently shown in [11, 31] to be
However, these approaches have almost exclusively examined the properties of text-to-image diffusion features on a single image; significantly less is known about how these features relate across multiple, different images and objects. Simply put, in Stable Diffusion, how similar are the features of a cat to the features of a dog? In this paper, we focus on understanding how features in different images relate to another by examining Stable Diffusion (SD) features through the lens of semantic correspondences, a classical vision task that aims to connect similar pixels in two or more images.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Accurate Semantic Correspondence
Robust Semantic Comespondence
Target
Source
Target
Target
Object Swapphing
Figure Semantic correspondence with fused Stable Diffusion and DINO features. On the left, we demonstrate the accuracy of our correspondences and demonstrate the instance swapping process. From top to bottom:
Starting with pairs of images (source image in orange box), we fuse Stable Diffusion and DINO features construct robust representations and build high-quality dense correspondence. This facilitates pixel-level instance swapping, and a subsequent saable-iiffssion instance. On the right, we demonstrate the robustness of our approach by matching dog. horses, cows, and even motorcycles to the cat in the source image. Our approach is capable of huilding reasonable correspondence even when the paired instances exhibit significant differences in categories, shapes, and poses.
refinement process yields a plausible swapped
For the correspondence task, we show that simply ensembling and reducing these features through basic techniques can lead to effective representation that quantitatively does as well as other state-of- the-art representations for dense and semantic correspondences. However, a close analysis of the qualitative results leads to interesting discovery. We observe that SD features have strong sense of spatial layout and generate smooth correspondences (front and back of bus are represented differently as shown in Fig. 3 left bottom), its pixel level matching between two objects can often be inaccurate (a bus front might match to another bus back). In fact, compared to existing features like DINOv1 [2], SD features appear to have very different strengths and weaknesses.
While prior work [2] showed that DINOv1 can an effective dense visual descriptor for semantic correspondence and part estimation, to the best of our knowledge, this has not yet been thoroughly analyzed for DINOv2, which demonstrated improved downstream performance for multiple vision tasks but not correspondence. For correspondence. we show that DINOy2 does outperform DINOv1 but requires a different post post-processing. DINOv2 generates sparse but accurate matches, which surprisingly, form a natural complement to the higher spatial information from SD features.
In this work, we propose a simple yet effective strategy for aligning and fusing these features. The fused representation utilizes the strengths of both feature types. To establish correspondence, we evaluate the zero-shot (no training for the correspondence task) performance of these features using nearest neighbors. Despite this very simple setup. our fused features outperform previous methods on the SPair-71k [36], PF-Pascal [20], and TSS [59] benchmark datasets. In addition, the fused representation facilitates other novel tasks such as instance swapping, where the objects in two images can be naturally swapped using estimated dense correspondence, while successfully maintaining its identity. The main contributions of this work are:
We demonstrate the potential of the internal representation from a text-to-image generative model for semantic and dense correspondence.
We analyze the characteristics of SD features, which produce spatially aware but inaccurate correspondences, and standard representation learning features, i.e., DINOv2, which generate accurate but sparse correspondence and show that they complement each other.
We design a simple strategy to align and ensemble SD and DINOv2 features and demonstrate that these features with a zero-shot evaluation (only nearest neighbors, no specialized training) can outperform many SOTA methods for semantic and dense correspondence. Experiments show that

---

the fused features boost the performance of the two and outperform previous methods significantly on the challenging SPair-71k dataset (+13%), as well as PF-Pascal (+15%) and TSS (+5%). We present an instance swapping application using high-quality correspondence from our methods.
2 Related work
Feature descriptor for dense correspondence. Deep models [15, 40, 45, 52, 66, 71] learn effective features for dense correspondence that can be robust to photometric and geometric changes such as rotation, scaling, perspective transformation. However, most of the methods focus on the image matching task for outdoor scenes to account for rigid transformation. Amir et al. [2] demonstrate that features extracted from self-supervised Vision Transformer (DINOv1 [6]) serve effective dense visual descriptors with localized semantic information by applying them to a variety of applications, e.g., (part) co-segmentation and semantic correspondences. Recently, DINOv2 [41] introduces scaled-up version of DINOv1 [6] by increasing the model size and combining a large quantity of curated data. DINOv2 shows strong generalization to downstream tasks such classification, segmentation, and monocular depth, but has not been extensively studied on correspondence tasks. Our work shows that DINOv2 also leads to significantly better correspondence results than DINOv1, but those DINO features in general produce sparse and noisy correspondence field.
Semantic correspondence. Semantic correspondence [33] aims to estimate dense correspondence between objects that belong to the same object class with different appearance, viewpoint, or non- rigid deformation. Conventional methods consist of three steps [64]: feature extraction, cost volume construction, and displacement field [60-63] or parameterized transformation regression [25, 28, 46, 47]. Those methods, however, cannot effectively determine dense correspondence for complex articulated deformation due to their smooth displacement fields or locally varying parameterized (affine) transformations. Motivated by a classical congealing method [29], couple of recent methods introduce to learn to align multiple objects in the same class by using pretrained DINOv] feature [18, 39] or GAN-generated data [42]. The methods show the possibility that knowledge learned for different tasks can also be utilized for the dense correspondence task. However, those models are effective in handling severe topological changes probably due to their strong rigidity assumption in the object alignment. In contrasts, we discover that the Stable Diffusion features for the image generation task exhibit great capability for the zero-shot dense correspondence task as well, where challenging articulate, non-rigid deformation exists.
Diffusion models. Diffusion probabilistic models (diffusion models) [55] and denoising diffusion probabilistic models [23, 38] have demonstrated state-of-the- performance for image generation task [13, 37, 48, 50, 51, 57]. In addition, the diffusion models have been applied to numerous vision tasks, e.g., image segmentation [3, 4, 8, 26, 58, 67], object detection [7]. and monocular depth estimation [14,53] 53]. Recently, much attention has been made analyze what pre-trained diffusion models understand and extract its knowledge for single-image tasks, e.g., panoptic segmentation [68], semantic segmentation and depth [73]. In this paper, we unveil the potential of diffusion features for zero-shot dense correspondence between images.
3 Semantic correspondences via Stable Diffusion and Vision Transformer
We first examine properties of Stable Diffusion (SD) features for semantic correspondence, then study the complementary nature between DINO and SD features, and finally introduce a simple and effective fusion strategy to leverage the strengths of both features.
3.1 Are Stable Diffussin features good for semantic correspondence?
Stable Diffusion (SD) [48] demonstrates remarkable ability to synthesize high-quality images conditioned on text input, suggesting that it has powerful internal representation for images and can capture both an image's content and layout. Several recent works [4, 68, 73] have used the pre-trained SD features for single-image tasks, such as semantic segmentation and depth perception. As features play an important role for visual correspondence, we are interested in investigating whether SD features can help establish semantic correspondences between images.
3

---

PCA Visualization
Clustering Match
Visualization
Input image
Layer
Layer 5
Layer
Layer
Layer 2+5+8
Figure 2: Analysis of features from different decoder layers in SD. Top: Visualization of PCA-computed features from early (layer 2), intermediate (layers 5 and 8) and final (layer 11) layers. The first three components of PCA, computed across a pair of segmented instances, semanties, while later layers concentrate on textures. Botton: K-Means clustering of these features. K-Means clusters are computed for each image individually, followed by an application of the Hungarian method to find the optimal match between clusters. The color in each columa represents pair of matched clusters.
color channels. Early layers focus
Next, we briefly explain how we extract SD features. The architecture of Stable Diffusion consists of three parts: an encoder E, a decoder ID that is derived from VQGAN [16] and facilitates the conversion between the pixel and latent spaces, and a denoising U-Net UA that operates in the latent space. We first project an input image Tg into the latent space via the encoder & to produce latent code zg. Next, we add a Gaussian noise e to the latent code according to a pre-defined time step t. Then, taking the latent code 2 at the time step i with the text embedding C' as inputs, extract the features FsD from the denoising U-Net. The entire process can be formally represented as follows:
Zg = Exr),
21 = Vāl=o + -gc, FSD = U(E,, C,C,
(1)
where the coefficient ag controls the noise schedule, as defined in [23]. For the text embedding C, we follow [68] to use an implicit captioner, which empirically is superior to explicit text prompts..
While prior work [65] has reported that the intermediate U-Net layers have more semantic information for image-to-image translation task, it is unclear whether these features are suitable for semantic correspondence. To examine their properties, on the features at various layers of the U-Net decoder. As shown in the top of Fig. 2, early layers tend semantics and structures (e.g. wheels and seats) but coarse level; the later layers tends to contain detailed information about texture and appearance. Furthermore, we apply K-Means clustering (k = 5) on the features paired instances and visually examine whether the clustered features contain consistent semantic information across intra-class examples. As shown in the bottom of Fig. different parts of the objects are clustered and matched across the instance pairs. Both PCA and K-means analysis show that earlier layer features capture coarse yet consistent semantics, while later layers focus low-level textural details. These observations motivate us combine the features at different levels (2+5+8) to capture both semantic and local details, through concatenation.
first perform principal component analysis (PCA)
focus
simple concatenation, however, results in unnecessarily high-dimensional feature (1280- 960 480 2720). To reduce the high dimension, we compute PCA across the pair of images for each feature layer, and then aggregate them together the same resolution. Specifically, we first extract the jth layer's 's features for source and target images, f:" and f. Next, we concatenate each layer's source feature and target feature and compute PCA together:
f = PCA( (I,f), iE (2,588).
Then we gather each layer's dimension-reducee features ft and f/ and upsample them to the same resolution to form the final SD feature f" and ft. As shown in the last column of Fig. 2, the

---

Table Evaluation of correspondence SPair-7Ik subsets. We sample 20 pairs for each category and report PCK@0. score for different settings. We underline the best performances achieved by each model. S/8: small model with a patch size of 8, 14: Base model with patch size of 14.
Layer 1lf
Layer 9f
Model
Token
Key
Query
Value
Token
Key
Query
Value
DINOv 1- 1-VIT-8/8
28.8
30.4
26.9
25.8
30.9 45.5
31.4
29.9
27.7
4
52.7
30.3
30.6
47.1
12.7
13.2
40.6
DINOV2-ViT-B/LE
55.7
42.6
40.7
53.4
50.8
25.2
25.3
46.0
Inpat image
Stable Diffusion
DINOv2
Fused
Input image
Stable Diffusion DINOv2
Fused
Figure 3: Analysis of different features for correspondence. We present visualization of PCA for the inputs from DAVIS [43] (left) and dense correspondence for SPair-71k [36] (right). The figures show the performance of SD and DINO features under different inputs: identical instance (top lcft), pure object masks (bottom left), challenging imputs requiring semantic understanding (right top) and spatial information (right bottom). Please refer to Supplemental B.1 for more results.
ensembled features strike a balance between the two different properties and focus on both semantics and textures at sufficient resolution.
In addition, we find that the specific location within the decoder layer to extract the feature also plays important role. U-Net's decoder layer inputs the feature of the previous decoder layer as well as the skip-connected encoder layer feature, and outputs the decoder feature after passing through sequence of layers: a convolutional layer, self-attention layer, and cross-attention layer. We empirically find that using only the decoder feature achieves better performance than combining with the encoder feature, and also outperforms the features of a single sub-layer (e.g., convolutional layer, self-attention layer). Please refer to appendix for more details.
One natural question is whether SD features provide useful and complementary semantic correspon- dences with respect to more widely studied discriminative features. Next, we discuss the semantic correspondence properties of the widely used DINOv1 [6] features in conjunction with the successor DINOv2 [41] features.
3.2 Evolution of DINO features for semantic correspondence
Caron al. [6] show that self-supervised ViT features (DINO), "contain explicit information for semantic segmentation and are excellent k-NN classifiers." " Amir et al. [2] further demonstrate that DINO features have several interesting properties that allow zero-shot semantic correspondence across images. Most recently, Oquab et al. [41] introduce DINOv2 by scaling up their training data and the model size and observe significant performance boost numerous single-image tasks, such as segmentation and depth. this work, we examine whether DINOv2 can significantly improve semantic correspondence across images.
We test different internal features from DINOv2 on the standard correspondence benchmark SPair- 7 71k [36] (please refer Sec. 4 for details). As shown in Tab. 1 PCK results (higher the better),
5

---

Table Smoothness (first-order differencc) of the valid estimated flow field the TSS dataset. We report the results three techniques and include the ground truth as a reference (lower is smoother).
Method
FG3DCar
JODS
Pascal
Avg.J
DINOv2- DINOv2-ViT-BB11
6.99
10.09
15.14
10.15
Stable Diffusion
3.48
7.87
8.44
5.90
Fuse- Fuse-ViT--BLL
3.52
7.55
8.75
5.96
Ground Truth
2.22
5.20
4.06
3,40
Stable Diffusion
DINOv2
Fused
Stable Diffusion
DINOv2
Fused
Figure 4: Semantic flow maps using different features. White mask indicates valid pixels and orange mask separates background flow. SD features yield smoother flow fields versus DINOv2's isolated outliers.
DINOv2 features achieve significant improvement over DINOv1. However, we observe that the best performance is achieved by the "token" facet from the last (1 Ith) layer of DINOv2, while Amir et al. [2] find that, for DINOv1, the best performance is achieved by the "key" facet earlier layers. We refer the token features from layer 11 of DINOv2 FiIn DINO.
3.3 Comparing the strengths and weaknesses SD and DINO features
We now discuss the properties and potential complimentary nature of SD and DINO features. Aside from the PCA visualization above, we compute dense correspondences between image pairs using both features via nearest neighbor search and examine their behaviors under different input conditions.
Correspondence for the same object instance. For simpler cases where instances are the same in paired images (Fig. top left), both SD and DINO features perform well. However, their performance differs on textureless examples. When giving an object mask as input (Fig, 3 bottom left), we observe that DINOv2 features cannot establish valid correspondence for this textureless example, while SD features work robustly they have strong sense of spatial layout.
Correspondence between different object instances. On examples with intra-class variation, both SD and DINO features perform well although different regions. DINO yields more accurate matches (e.g., in the thigh region of Fig. top right case), but the correspondence field tends to be noisy, as demonstrated in both cases on the right side of Fig. 3. In contrast, SD excels at constructing smoother correspondence, shown in both cases the right side of Fig. 3, and provides crucial spatial information, notably in the bottom right case.
Spatial coherence of the correspondence. To analyze the spatial coherence of the estimated correspondence, we can extract the semantic flow, i.e., the 2D motion vector for every pixel from the source image the corresponding pixel in the target image. We then compute the smoothness, Le. first-order difference, the estimated semantic flow on the TSS dataset [59], as shown in Tab. 2. This analysis reveals the inherent smoothness in the flow maps generated by different methods. The Stable Diffusion features yield smoother results compared to MINOv2-ViT-B// suggesting that Stable Diffusion features exhibit better spatial understanding than the DINO features.
Discussion. Tab. evaluates the spatial smoothness of the flow fields estimated by different methods on the TSS dataset. The features of Stable Diffusion lead to smoother dense correspondence than DINO features. Fig. 4 visually shows that the correspondence by DINO features contains moderate amounts of isolated outliers, while that by Stable Diffusion is more spatially coherent.
Summary. The studies above reveal the complementary nature of DINO and Stable Diffusion features. DINO features can capture high-level semantic information and excel at obtaining sparse but accurate matches. In contrast, Stable Diffusion features focus on low-level spatial information and ensure the spatial coherence of the correspondence, particularly in the absence of strong texture signals. The complementary properties of DINOv2 and diffusion features offer promising potential for enhancing the performance of semantic correspondence tasks. A natural question arises:

---

3.4 How to fuse SD and DINO features?
Based on the discussions above, we propose a simple yet effective fusion strategy to exploit SD and DINO features. The core idea is to independently normalize both features to align their scales and distributions, and then concatenate them together:
(2) where || l|2 denotes the 1.2 normalization, a is a hyperparameter that controls the relative weight of the two features. An important aspect of this fusion approach is the selection of the weight a. Empirically, we find that a = 0.5 offers a good balance between the two features, effectively leveraging their complementary strengths. See the supplementary material for the ablation studies.
FFUSE = (all l2, (1 (1-a) FDINoll))
Observing the surprising effectiveness of our simple fusion strategy in Fig. 3, we see notable improvements in challenging cases from the SPair-71k dataset. The combined features noticeably outperforms either features alone in all the cases. It not only demonstrates enhanced precision and reduced noise in correspondences but also retains the inherently smoother transitions and spatial information unique to the SD feature. Furthermore, as shown both quantitatively in Tab. 2 and qualitatively in Fig. 4, the performance of the fused features aligns remarkably closely with that of the SD features in retaining smoother transitions and spatial information.
In the later section, we further show the remarkable effectiveness of our simple fusion strategy through extensive comparison with existing methods on various public benchmark datasets.
Experiments and analysis
We first provide in-depth amalyses of both sparse and dense correspondence tasks public benchmark datasets. Then as a novel application of our approach, we present a simple, efficacious instance swapping method between paired images, utilizing dense correspondence from our method.
Evaluation of features for correspondence. We evaluate the extracted features in two settings. First, in zero-shot setting, we follow [2] by searching the nearest neighbors directly on the feature maps of pair images. Second, for datasets with a training split, we add bottleneck layer [22, 68] on top of the extracted features and finetune it in a supervised manner. It is guided by the same objective as in [35]: a CLIP--tyle symmetric cross entropy loss with respect to corresponding keypoints.
Implementation details. We employ the Stable Diffusion v1-5 model as our feature extractor, with the timestep for the diffusion model be 100 by default. We use the input resolution 960 x 960 for the diffusion model and 840 for the DINO model, which results in a feature map with a 60 resolution. For PCA, we adopt nearly optimal approximation SVD algorithm [19] for better efficiency and reduce the feature dimension to 256 when fusing with DINO-ViTY MINO-ViT-B/14 When visualizing dense correspondence, we apply a segmentation mask obtained from [68] as a preprocessing step. All experiments are conducted on a single NVIDIA RTX3090 GPU.
Datasets. For the sparse correspondence task, evaluate on two standard datasets, SPair-71k [36] and PF-Pascal [20], which consists of 70 958 and 1341 image pairs respectively, sampled from 18 and 20 categories. We evaluate the dense correspondence TSS [59], the only dataset that provides dense correspondence annotations on 400 image pairs derived from FG3DCAR [32], JODS [49], and PASCAL [21] datasets.
Metrics. To evaluate the correspondence accuracy. we use the standard Percentage of Correct Keypoints (PCK) [70] metric with threshold of x- k-max(h, w) where is positive integer. and (h, w) denotes the dimensions of the bounding box of an instance SPair- and the dimensions of the images in PF-Pascal and TSS respectively, by following the same protocol in prior work [18. 39, 42].
4.1 Sparse correspondence
Tab. 3 provides our zero-shot and supervised evaluation on the SPair-71k dataset. Our zero-shot method (Fuse-ViT ViT-B/14) significantly outperforms all methods including unsupervised methods as well as even previous supervised methods by a large margin, achieving a leading average PCK score of 64.0. We also observe that our fusion strategy significantly improves the performance of the DINOv2 baseline (DINOv2-ViT-B/114) resulting PCK score improvement of 8.4p (from 55.6 to 64.0). The Stable Diffusion features, not only being competitive to DINOv2 on the average PCK metric,
7

---

Table 3: Evaluation on SPair-7Ik. Per-class and average PCK @0.10 test split. The methods are categorized into four types: strong supervised (S), GAN supervised (G), unsupervised with task-specific design (U"), ), and unsupervised with only nearest neighboring (U"). *: fine-tuned backbone. i: a trained bottleneck layer is applied on top of the features, We report per image PCK result for the (S) methods and per point result for other methods. The highest PCK among supervised methods and all other methods are highlighted in bold, while the second highest are underlined. Our NN-based method surpasses all previous unsupervised methods significantly.
Method
Aeto Bike Bird Bottle Cur Chuir Cuw Dug Horse Motor Person Plant Sheep Train TV
S SCOT [34]
34.9 20.7 63.8 21.1
43.5
27.3
21.3
63.1 20.0
42,9
42.5
31.1
29.8
35.0
27.7
24.4
48.4
40.8
35.6
CATs" [9]
52.0 34.7 72.2 34,3
49,9
57.5
43.6
66.5
24.4
63.2
56.5
52.0
42.6
41.7
43.0
33.6
72.6
58.0 49.9
PMNC" [30]
54.1 35,9 74,9 36.5
42.1 48.8
40.0
72.6
21.1 67.6
58.1
50.5
40.1
54.1
43.3
35.7
74.5
59.9 50.4
SComSAN" [24]
57.1 40.3 78.3 38.1
51.8
57.8
67.9
25.2 71.3 63.9
49.3
45.3
49.8
48.8
40.3
77.7
69.7 55.3
CATs++ [10]
60.6 46.9 82.5 41.6
56.8
64.9
50.4
72.8
29.2 75.8
65.4
62.5
50.9
56.1
54.8
48.2
80.9
74.9 59.9
DINOV2-VTTBB144
80.4 60.2 88.1 59.5
54.9
83.0
89.1 53.3 85.5
73.6
73.8
65.2
72.3
43.6
65.6
91.4 60.3 69.9
Stable Diffusion' (Ours) 75.6 60.3 87.3
41.5
50.8
68.4
81.4 44.3
79.4
62.8
67.7
64,9
71.6
57.8
53.3
89.2
65.1 66.3
Fuxe Fuse-VIT-B// VIT-B/14" (Ours)
81.2 66.9 91.6
61.4
57.4
85.3
83.1
90.8
54.5
88.5
75,1
80.2
71.9
77.9
60.7
68.9
92.4 65.8 74.6
G. GANgcaling [42]
37,5
67.0
57.9-
U VGG+MLS [1]
29.5 22.7
61.9
26.5
20.6 14.1
23.7
14.2
27.6
30.0
29.1
24.7
27.4
19.1
19.3
24.4
27.4
DINO-MLS [1
49.7 20,9
63.9
19.1
32.5 27.6
22.4
48.9
14.0
36,9
39.0
30.1
21.7
41.1
17.1
18.1
35.9
31.1
NeuCongeal [39]
29.1
53.3
35.2
ASIC [18]
57.9 25.2 68.1
24.7 35.4 28.4
30.9
21.6
45.0
47.2
39.9
26.2
48.8
14.5
24.5
49.0
24.6 36,9
U DINOVI-VET-S/ (2)
57.2 24.1 67.4 24.5
26.8 29.0
52.1
15.7
42.4
43.3
30.1
23.2
40.7
16.6
24.1
310
24.9 33.3
DINOv2-VET-B/14
72.7 62.0 85.2 41.3
40.4 52.3515.51.
71.1 36.2 67.1 64.6
67.6
61.0
68.2
30.7
62.0
54.3
24.2 55.6
Stable Diffusion (Ours)
63.1 55.6 80,2 33.8
44.9
49.3
74.4 38.4 20.8
53.7
61.1
54.4
55.0
$4.8
53.5
65.0
57.2
Fuxe VIT B/14 (Ours)
73.0 64.1 86.4 40.7 52.9 55.0
53.8
78.6 45.5 77.3 64.7
69.7
63.3
69.2
58.4
67.6
66.2
53.5 64.0
but also notably excel in certain categories where spatial information is critical (e.g., "potted plant", "train", and "TV"). The same observation holds for supervised setting, notably the improvement in PCK score of 14.7p (from 59.9 to 74.6) compared to previous method.
In Tab. 4, we further validate approach on the PF-Pascal dataset, which is less challenging dataset that presents low variations on appearance, pose, or shape between instance pairs. Our method consistently outperforms all unsupervised methods, achieving the highest average PCK scores across all thresholds. Our fusion approach (Fuse- (Fuse-ViT-B/1k ViT-B/14), again, substantially improves the performance over the DINOv2 baseline that benefits from both features.
highlighting the effectiveness of our fusion strategy
4.2 Dense correspondence
Tab. 4 further provides the dense correspondence evaluation on the TSS dataset [59]. Our fusion approach (Fuse- ViT-B/14) outperforms all unsupervised neighbo-eggh on the TSS dataset. The result again confirms the effectiveness of our fusion strategy, demonstrating a substantial gain of 7.7p over the DINOv2- DINOV2-ViT-B/I ViT-B/14 baseline. Note that the TSS dataset contains less challenging examples with low variations on appearance, viewpoint, and deformation (e.g. car, bus, train, plane, bicycle, etc.), where having a strong spatial smoothness prior can be an advantage for achieving good metries on the benchmark.
methods (U*)
4.3 Instance swapping
Given pair of images with instances of the same or similar categories, instance swapping is defined as the task of transposing an instance from a target image (target instance) onto the instance(s) in a source image (source instance), while preserving the identity of the target instance and the pose of the source instance. This creates a novel image where the target instance appears naturally integrated into the source environment.
By leveraging the high-quality dense correspondence obtained by our method, we can achieve plausible instance swapping through a straightforward process: 1) Initially, we upsample the low- resolution feature map to match the size of the target image; 2) Based on the upsampled feature map, we perform pixel-level swapping the segmented instances according to the nearest neighbor patch, yielding a preliminary swapped image; 3) To enhance the quality of the swapped image, we refine the preliminary result with cillleddffuioo--bbeee
image-to-image translation process [65].
8

---

Table Evaluation on PF-Paseal and TSS. The highest PCK among nearest-neighhoring based unsupervised are highlighted bold, while second highest are underlined. fusion results result large improvement over the DINO baselines, and are comparable to other task-specific methods. S: Supervised methods, U Task-specifie unsupervised methods, U Nearest-neighhorie neighboring based unsupervised methods. s; fine-tuned backbone. t: trained bottleneck layer is applied top of the features.
PF-Pascal, P
TSS, PCK @0.05
Method
0,05
0.10
0.15
PG3DCar
JODS
Pascal
Avg.
SCOT [34]
63.1
85.4
92.7
95.3
81.3
57.7
78.1
CATs [9]
76.8
92.7
96.5
92.1
78.9
64.2
78.4
PWarpC-CATs [64]
79.8
92.6
96.4
95.5
85.0
85.5
88.7
CATs++ [10]
84.9
93.8
96.8
DINOv2-VIT-B/14*
74.2
90.8
95.4
Stable Diffusion! (Ours)
77.4
89.7
93.9
BTT--VYT-B/14' (Ours)
80.9
93.6
96.9
CNNGeo [46]
76.4
74.4
PARN [25]
89.5
75.9
71.2
78.8
GL.U-Net [60]
42.2
69.1
83.1
93.2
73.3
71.1
79.2
Semantic- GLU--ntee--U--ettttee (63)
48.3
72.5
85.1
95.3
82.2
78.2
85.2
UN
DINOvI-ViT [2]
41.5
62.4
72.5
64.7
51.2
36.7
53.3
DINOv2-VIT-BBL4
56.2
77.3
83.3
82.8
73,9
53.9
72.0
Stable Diffusion (Ours)
61.0
86.1
93.9
69.4
57.7
77.7
Fuse-VYT-B/14 (Ours)
73.0
86.1
91.1
94.3
73.2
60.9
79.7
Source
Target
Stable Diffusion
DINOv2
Fused
Figure Qualitative comparison of instance swapping with different features. SD fcatures deliver smoother swapped results, DINOv2 reveals greater details, and fused approach takes the strengths of both. Notably, the fused features generate more faithful results to the reference image, as highlighted by the preserved stripes on the cat instance in the top-right example. Please refer to Supplemental B.2 for more results.
To be more specific with the refinement process, we begin by inverting the swapped image the initail noise with DDIM inversion [56], followed by denoising DDIM sampling process where we extract the features from the diffusion model. Finally, merging this with prompt that includes the instance category, we generate refned image. This image exhibits swapped instance with an appearance that visually more coherent and plausible. Please refer to Supplemental A.8 and B.2 respectively for the quantitative and qualitative analysis of the refinement process.
Fig. 5 provides visual comparison of instance swapping using different features. SD leads smooth results, whereas DINOv2 accentuates fine details. The fusion both methods showcases a balance between spatial smoothness and details. More examples are available in the supplementary material.
We also compare different features quantitatively on an 800-pair subset of [69] benchmark. The quantitative evaluation in Tab. 5 reveals consistent performance gains of the fused approach across all three metrics: FID score, quality score, and CLIP score. We follow the same metrics from [69]: FID score is measured using the CLIP features between the generated images and the COCO 2017 testing images, and CLIP score measures the cosine similarity between CLIP features extracted from the edited region of the source image and the target image.
These results show the strength of the dense correspondences with fused features not only maintain- ing the instance's appearance but also generating high-quality images. The CLIP score is higher for

---

Table 5: Quantitative comparison for instance swapping. The best performance approach is in bold.
Method
FID Score(CLIP-basell))
Quality scoret
CLIP scoret
DINOv2-ViT-B/14
12.49
63.18
72.63
Stable Diffusion
13.72
61.38
71.48
Fuse-ViT-B/14
12.47
64.84
73.21
Table 6: Distribution outcomes under different datasets and PCK levels. Under most settings, one feature succeeds while the other fails 20-30% total cases (see row 2 & 3), which suggests that they have a substantial amount of non-redundant information.
SPair- @
PF- PF-Pascal, PCK@*
Cases
0.15
0.10
0.05
0.15
0.10
0.05
SD, DINO fails
21.7
29.2
44.5
5.6
10.0
27.1
SD fails, DINO correct
15.7
15.8
14.2
8.3
9.7
12.0
SI correct, DINO fails
14.0
15.3
15.8
11.1
12.7
16.8
SD, DINO correct
48.6
39.7
25.5
75.0
67.6
44.2
the fused approach, indicating that the fused feature representation is more faithful to the reference image. Meanwhile, the higher quality scores and lower FID scores point the superior perceptual quality and semantic coherence of the images generated by the fused approach.
4.4 Limitations
Although our approach shows promising correspondence performance, it does come with certain limitations. The relatively low resolution of the fused features impedes the construction of precise matches, which particularly required in dense correspondence tasks as in the TSS dataset. Detailed exploration of failure cases can be found in the supplementary material. Additionally, the integration of the Stable Diffusion model, while only requiring single inference, considerably increases the cost of the correspondences computation in comparison to using DINO features only.
4.5
Quantitative analysis of feature complementary
We present additional quantitative analysis on the non-redundancy of SD and DINOv2 features in Tab. 6, which details the error distribution for these two features on SPair- 1 and PF-Pascal benchmarks at three different PCK levels. In 20-30% total cases under most settings, one feature succeeds where the other fails, suggesting that they have a substantial amount of non-redundant information. For a more detailed analysis on this non-redundancy, please refer to Supplemental A. 1.
5 Discussion on feature behavior
The distinct behaviors observed in Stable Diffusion (SD) and DINOv2 features have spurred questions regarding the underlying causes. Though it is challenging to validate due to resource limitations, we suggest some possible explanations: Training paradigms, DINO's self-supervised learning approach could induce an invariance to spatial information because of its training augmentations, This contrasts with SD, which, being trained for text-to-image synthesis, inherently demands awareness of both global and local visual cues, possibly leading to its heightened spatial sensitivity. Architectural differences, DINOv2, built upon the ViT model, interprets images as sequences of patches. Even with positional encoding, it might prioritize local structures as much. On the other hand, the convolution layers in SD's UNet may retain more details and enhance the retention of spatial information. In general, identifying the causes of these variances remains an intriguing topic and is great direction for future exploration.
6 Conclusion
In this work, we explore the potential of leveraging internal representations from a text-to-image generative model, namely Stable Diffusion, for the semantic correspondence task. We reveal the com- plementary nature of the Stable Diffusion features and self- a deeper understanding
DINO-VIT features, providing their individual strengths and how they can be effectively combined. Our proposed simple strategy of aligning and fusing these two types of features has empirically proven to significantly enhance performance on the challenging SPair-71k kk dataset and outperform existing methods, suggesting the benefits of pursuing better features for visual correspondence.
10

---

References
[1] Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or Neural best-buddies: Sparse cross-domain correspondence. ACM Transitions on Graphics, 37 (4):1-14, 2018. 8
[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. In European Conference on Computer Vision Workshops, 2022. 3, 5, 6, 8,
[3] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. SegDiff: Image segmentation with diffusion probabilistic models. arXiv:2112.00390 2021.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference Learning Representations, 2022. 3
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, pages 9912-9924, 2020. 8
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In International Conference Computer Vision, pages 9650-9660, 2021. 3,5
[7] Shoufa Chen, Peize Sun, Yibing Song. and Ping Luo. DiffusionDet: Diffusion model for object detection. In International Conference Computer Vision, 2023. 3
[8] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleet. A generalist framework for panoptic segmentation of images and videos. In International Conference Computer Vision, 2023.
[9] Scokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. Advances in Neural Information Processing Systems, 34:9011 100011 2021. 8,9
[10] Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. arXiv preprint arXiv:2202.06817, 2022. 8,
[11] Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. arXiv preprint arXiv:2303. 15233, 2023. 1
[12] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. DiffEdit: Diffusion- based semantic image editing with mask guidance. In International Conference on Learning Representations, 2023.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusson models beat GANs on image synthesis. Advances in Neural Information Processing Systems, pages 8780-8794, 2021. 3
[14] Yiqun Duan, Xianda Guo, and Zheng Zhu. DiffusionDepthh: Diffusion denoising approach for monocular depth estimation. arXiv:2303.0502/, 2023.
[15) Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-Net: A trainable CNN for joint description and detection of local features. In IEEE Conference on Computer Vision and Puttern Recogition, pages 8092-8101, 2019.
[16] Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recogition, pages 12873-12883, 2021.
[17] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. PAIR-Diffusion: Object-level image editing with structure-and-appeaaane paired diffusion models. arXiv preprint arXiv:2303. 2023.

---

[18] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. ASIC: Aligning sparse in-the-wild image collections. arXiv preprint arXiv:2303. 16201, 2023. 3, 7,8
[19] Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53 (2):217-288, 2011.
[20] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal Flow: Semantic correspondences from object proposals. IEEE Transactions on Pattern Analysis and Machine Intelligence, (07)):711111255 2017. 2,
[21] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference Computer Vision, pages 991-998, 2011.7
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[23] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages (840-6851, 2020. 3,
[24] Shuaiyi Huang, Luyu Yang. Bo He, Songyang Zhang, Xuming He, and Abhinav Shrivastava. Learning semantic correspondence with sparse annotations. In European Conference Computer Vision, pages 267-284, 2022. 8
[25] Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. PARN: regression networks for dense semantic correspondence. In European Conference Computer Vision, pages 351-366 2018. 3,9
Pyramidal affine
[26] Peng Jiang, Fanglin Gu, Yunhai Wang, Changhe Tu, and Baoquan Chen. DifNet: Semantic segmentation by diffusion networks. In Advances in Neural Information Processing Systems, 2018.
[27] Babjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri. and Michal Irani. Imagic: Text-based real image editing with diffusion models. In IEEE Conference on Computer Vision and Pattern Recogition, 2023. 1
[28] Seungryong Kim, Stephen Lin, Sang Ryul Jeon, Dongho Min, and Kwanghoon Sohn. Recur- rent transformer networks for semantic correspondence. In Advances in Neural Information Processing Systems, 2018.
[29] Erik G. Learned-Miller, Data driven image models through continuous joint alignment. JEEE Transactions on Puttern Analysis and Machine Intelligence, 28(2):236-2555 2005.
[30] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N. Sinha. PatchMatch-baase neighborhood consensus for semantic correspondence. IEEE Conference on Computer Vision and Pattern Recogition, pages 13 13153-13163, 2021.
[31] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303. 2.20::3303..623, 2023. 1
[32] Yen-Liang Lin, Vlad I Morariu, Winston Hsu, and Larry S. Davis. Jointly optimizing 3D model fitting and fine-grained classification. In European Conference Computer Vision, pages 466-480, 2014.7 7
[33] Ce Liu, Jenny Yuen, and Antonio Torralba. SIFT Flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5): 978-994 99499944222 2010. 3
[34] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In IEEE Conference on Computer Vision and Pattern Recogition, pages 4463-4472 2020. 8,
12

---

[35] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell, Diffusion hyperfeatures: Searching through time and space for semantic correspondence. arXiv preprint arXiv:2305. 14334, 2023. 7
[36] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. SPair-7Ik: A large-scale benchmark for semantic correspondence. arXiv: 1908. 10543, 2019. 2, 5,7
[37] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Confernece on Machine Learning, 2022.
[38] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Confernece on Machine Learning, pages 8162-8171, 2021.
[39] Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel. Neural Congealing: Aligning images to joint semantic atlas. In JEEE Conference on Computer Vision and Pattern Recogition, 2023. 3, 7,
[40] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moe Yi. LF-Net: Learning local features from images. In Advances in Neural Information Processing Systems, 2018. 3
[41] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. 3,5
[42] William Peebles, Jun- Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A. Efros, and Eli Shechtman. GAN-supervised dense visual alignment. In IEEE Conference Computer Vision and Puttern Recogition, pages 13470-13481, 2022. 7,8
[43] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliass, Luc Van Gool, Markus Gross, and Alexander Hornung A benchmark dataset and evaluation methodology for video object segmentation. IEEE Conference on Computer Vision and Pattern Recogition, pages 724-732, 2016. 5
[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXhv:220..06125, 2022.
[45] Jerome Revaud, César De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: Reliable and repeatable detector and descriptor. In Advances in Neural Information Processing Systems, 2019.
[46] Ignacio Rocco, Relja Arandjelovié, and Josef Sivic. Convolutional neural network architecture for geometric matching. In IEEE Conference on Computer Vision and Pattern Recogition, pages 6148-6157, 2017. 3.9
[47] Ignacio Rocco, Relja Arandjelović, and Josef Sivie. End-to-end weakly-supervised semantic alignment. IEEE Conference on Computer Vision and Pattern Recogition, pages 6917-6925, 2018. 3
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. JEEE Conference on Computer Vision and Pattern Recogition, pages 10684-10695, 2022. 1,
[49] Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation in internet images. In IEEE Conference Computer Vision and Pattern Recogition, pages 1939-1946, 2013.
[50] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, pages 1-10, 2022. 1,3
[51] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, pages 36479-36494, 2022. 3
13

---

[52] Faul-Edouaad Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In IEEE Conference on Computer Vision and Puttern Recogition, pages 4938 -1884497,.5 2020. 3
[53] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J. Fleet. Monocular depth estimation using diffusion models. arXiv:2302. 2023. 3
[54] Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic alignment with offset-aware correlation kernels. In European Conference Computer Vision, pages 349-364, 2018. 3
[55] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Confernece on Machine Learning, pages 2256-2265, 2015. 3
[56] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021. 9
[57] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, pages 12438-12448, 2020. 3
[58] Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. In Advances in Neural Information Processing Systems, pages 8702-8716, 2022. 3
[59] Tatsumori Taniai, Sudipta N Sinha, and Yoichi Sato. Joint recovery of dense correspondence and cosegmentation in two images. In IEEE Conference Computer Vision and Pattern Recogition, 2016. 2, 6, 8
pages
[60] Prune Truong, Martin Danelljan, and Radu Timofte. GLU-Net: Global-local universal network for dense flow and correspondences. In IEEE Conference on Computer Vision and Pattern Recogition, pages 2258-6266 2020. 3, 9
[61] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. GOCor: Bringing globally optimized correspondence volumes into your neural network. In Advances in Neural Information Processing Systems, pages 14278-14290, 2020.
[62] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning accurate dense correspondences and when to trust them. In IEEE Conference on Computer Vision and Pattern Recogition, pages 57 1714-57 2021.
[63] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Warp consistency for unsuper- vised learning of dense correspondences. International Conference Computer Vision, pages 00346-10356, 2021. 3,9
[64] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Probabilistic warp consistency for weakly-superriin semantic correspondences. In IEEE Conference on Computer Vision and Pattern Recogition, pages
2022. 3,
[65] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and- ala-aa diffusion features for text-driven image-to tomeeagee- translation. In IEEE Conference on Computer Vision and Pattern Recogition, 2023. 1, 4, 8
[66] Michat J. Tyszkiewicz, Pascal Fua, and Eduard Trulls. DISK: Learning local features with policy gradient. In Advances in Neural Information Processing Systems, pages 14254-14265, 2020.
[67] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C. Cattin. Diffusion models for implicit image segmentation ensembles. 1 2021. 3
[68] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE Confer- ence Computer Vision and Pattern Recogition, 2023. 1, 4,
14

---

[69] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by Example: Exemplar-bas based image editing with diffusion models. D.Xiv:2111.13227, 2022. 9
[70] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12)28878-2 2890, 2012.
[71] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned invariant feature transform. In European Conference Computer Vision, pages 467-483, 2016. 3
[72] Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, arXiv preprint 23X.::23020055433 2023.
[73] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception, wwXiv:2303.02/53 2023. 3
15


Reviews:
[{"review_id": "A3lcLJkDs3", "reviewer": "Reviewer_Pbqb", "summary": "The paper proposes using features extracted from a stable diffusion model for dense image correspondence tasks. The paper further proposes using DINO features along with stable diffusion features for the task and empirically shows that the combination has a complementary effect--specifically SD features have good spatial localization and are smooth, whereas DINO features are sparse but accurate. Experiments are done on several datasets and the proposed feature extraction technique improves drastically over supervised, unsupervised, and weakly supervised baselines.", "questions": "1. Though the motivation for using these features is clear, the paper doesn't provide reasons for the complementary properties of the features from DINO and SD. It would be interesting to analyze what specifically gives rise to these complementary properties. Is it because of the training objective, architecture, datasets, etc?\n2. Second and fourth rows of the right side of Figure 3 can be explained more for clarity. Specifically, if the figure also includes the original images, it will make more sense.\n3. Did the authors experiment with giving different textual inputs to SD? Does including object categories in the textual categories improve the performance of correspondences?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, adequately inspired, and well-executed.\n2. Experiments depict the importance of these large pre-trained models for the task of correspondence learning in images. Specifically, the large gains over supervised methods are quite surprising.\n3. Instance swapping application clearly demonstrates the ability to do accurate correspondences between different instances of the same category of objects.", "weaknesses": "1. The use of pre-trained features from large models is a well-explored area of research [1,2,3]. These pre-trained models are known to perform well on downstream tasks such as semantic segmentation, detection, and classification. It is unsurprising that these features are also useful for correspondence learning.\n2. Though the authors show limitations of the features. It should have been focussed more on the paper. Specifically categorizing the mistakes made by the approach and possible ways to alleviate the problems. This will be a good contribution to the community as it informs when to avoid using the proposed approach.\n\n\n[1] _Unleashing text-to-image diffusion models for visual perception._ Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. 2023\n\n[2] _Emerging properties in self-supervised vision transformers._ Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021\n\n[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models. Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Leveraging SD / DINO features for correspondence learning.\n\n**A:** While [1,3] demonstrate that SD features are useful for depth and semantic segmentation and [2] that DINO features are useful for image retrieval and semantic segmentation, less work has focused on the tasks of semantic and dense correspondences: [4] explores using DINOv1 for these tasks but we are unaware of any published work which explores SD or DINOv2 features for correspondences.\n\nUnlike semantic segmentation or detection which are defined on a single image, **the correspondence task requires that different objects have similar representations across different images** with potentially different lighting or camera intrinsics. In addition, dense correspondence quality tends to improve when the features accurately encodes lower level details like textures, unlike segmentation or detection which benefits from features with more high-level semantics. Also different from [1], we show that SD features are surprisingly useful for correspondences in a zero-shot setting, meaning that these properties are already present in the features and do not require additional fine-tuning or processing.\n\nIn addition to demonstrating that Stable Diffusion features are useful for correspondences, **our other main contribution is an in-depth analysis of the strengths and weaknesses of SD versus DINO features for correspondence**, and demonstrating that these features are surprisingly complementary. While these two features perform roughly equally, their fusion can lead to SOTA results on several correspondence benchmarks, even in a zero-shot setting with no training.\n\n------\n\n**W2:** Limitations of the approach and potential solutions.\n\n**A:** Fig. 7 of the Supplementals show representative failures of the fusion feature, especially on matching tiny objects due to the low resolution of feature maps. We have explored different techniques for improving the resolution of the features (e.g. resizing the input or different projection layers). Our method also sometimes struggles on symmetric objects, such as Airplane image in the Pixel Warping tab in index.html, included in the Appendix. This can be overcome by exploiting geometric priors during the matching stage.\n\nOur paper primarily focuses on the unsupervised, zero shot setting at test time, which allows us to better understand the information present in the features as is. For optimal performance, we show in the rebuttal that these results can be improved through training a projection layer on top of these features (please refer to Table 1 in R2-fsqe's W1 response). We expect that even better performance can be achieved by finetuning a larger post processing network after the fusion features or just fine-tuning the existing backbone feature networks (DINOv2 and SD) in a supervised setup.\n\nWe will include this discussion in the main paper.\n\n------\n\n**Q1:** Causes of the complementary properties.\n\n**A:** Please refer to the General Response Q&A.\n\n------\n**Q2:** Clarity on Figure 3.\n\n**A:** For the right side of Figure 3, the original image is in *the first column of the first and third rows*, and the target image is in the first column of the second and fourth rows. In the current paper, the original image has color coding to indicate the colormap for correspondences. We will also include the original image without the color coding in the updated version.\n\n------\n**Q3:** Effect of explicit textual descriptions.\n\n**A:** The table below reports the PCK@0.10 performance of Diffusion features and Fused features in the SPair-71k 20-samples subset, when implicit and explicit textural (specifically, “a photo of {category}”) inputs are given. Overall, there are only marginal differences. The explicit textual inputs help in earlier steps (200 steps), while implicit captioner helps in denoised images. We conjecture that this is due to the implicit captioner from ODISE [1] being trained with timestep=0.\n\n\n|Method|Captioner|0|50|100|150|200|\n|-|-|:-:|:-:|:-:|:-:|:-:|\n|SD|Implicit|**54.93**|**55.67**|***56.18***|55.11|55.11|\n||Explicit|53.58|55.63|*55.90*|**55.45**|**55.15**|\n||||||||\n|Fuse|Implicit|**63.25**|**63.10**|***63.28***|62.46|62.50|\n||Explicit|62.20|62.50|62.61|**62.72**|***63.32***|\n\nTable 1.  The PCK performance on SPair-71k for both implicit and explicit captioner under different timesteps. Best results between captioner are **bold**, best results among different timesteps are *italicized*.\n\n-----\n**References**\n\n[1] Unleashing text-to-image diffusion models for visual perception, Zhao et al. 2023\n\n[2] Emerging properties in self-supervised vision transformers, Caron et al. 2021\n\n[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models, Xu et al. 2023\n\n[4] Deep ViT Features as Dense Visual Descriptors, Amir et al. 2021"}, {"review_id": "Bl153S34Sq", "reviewer": "Reviewer_fsqe", "summary": "This paper proposes to study the effectiveness of features extracted from Stable Diffusion for dense correspondences. The extracted features are compared to that of DINOv2 and shown to be complementary. A very simple fusion scheme is then proposed and evaluation on datasets for sparse and dense correspondences as well as on instance swapping with convincing results.\n\n---\n\nThe rebuttal is convincing, I maintain my initial rating leaning towards acceptance.", "questions": "- The timestep at which features are extracted is not clearly mentioned (is it t=100 at line 214, in that case over a total time length of how long?). What influence influence does it have? Are the features more meaning closer to the denoised image ?\n- The swapping are impressive, but they are refined by running the diffusion process after wise. It would be great to see compositing before the diffusion process to evaluate the quality of the correspondences.", "limitations": "The paper discusses 2 limiations: namely the low resolution of the features and the computational budget to get features from a big diffusion model. The first one is probably the bigger concern for correspondences.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "With all the effort that has been poured recently in diffusion models, this empirical study regarding their use outside of generative nice images is refreshing, especially since it tackle a very low level problem (keypoint correspondences) that is somewhat very remote from image generation and yet surprisingly close since the model has to generate coherent local structures. The study shows that indeed, diffusion model features are very useful for such problems and thus sheds light on what one can expect from the latent space structure of SD.", "weaknesses": "- All the results are performed in zero-shot, which is a bit limiting. I am not familiar with the recent literature on correspondence problems, but I assume that a fine-tuning of the features (at the very least of the projection) is possible to see how much we can get from these models.\n- All the experiments were made with stable diffusion 1.5. It would be interesting to test another model to see if the same results hold across architecture and training change and thus if it is a generic property of diffusion models.\n- The fact that the best results are obtained using 2 really big models is a bit annoying. It means it is difficult to know if the results come from the nature of the methods employed (SSL and diffusion) or just from the sheer capacity of the models employed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Effect of fine-tuning the features with the projection layer.\n\n**A:** We briefly explored a supervised adaptation by training a projection layer [1] on top of the extracted features, guided by the CLIP-style symmetric cross entropy loss with respect to corresponding keypoints. As in Table 1, this approach yields marked improvements across both SPair-71k and PF-Pascal datasets. Notably, the fused features consistently outperform individual SD or DINOv2 features, even under the supervised setting. \n\n|Technique|Method||SPair-71k|||PF-Pascal||\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|||PCK@0.01|PCK@0.05|PCK@0.1|PCK@0.05|PCK@0.1|PCK@0.15|\n|NN-based|DINOv2|6.1|39.7|55.2|61.1|77.3|83.3|\n||SD|6.8|42.8|56.2|63.2|77.7|84.3|\n||Fuse|8.3|47.6|63.3|72.1|86.0|90.6|\n||\n|Projection Layer|DINOv2|8.6|56.4|74.1|74.2|90.8|95.4|\n||SD|10.0|56.0|71.1|77.4|89.7|93.9|\n||Fuse|**10.7**|**62.5**|**78.2**|**80.9**|**93.6**|**96.9**|\n||\n|Supervised Baselines|PWarpC-NC-Net|-|31.6|52.0|79.2|92.1|95.6|\n|(*: fine-tuned backbone)|CATs++*|-|-|*59.8*|*84.9*|*93.8*|*96.8*|\n||IFCAT*|-|-|*64.4*|*88.0*|*94.8*|*97.9*|\n\nTable 1. Comparison between NN-based methods, fine-tuning a projection layer, and other supervised baselines.\n\nWe will include these findings in the paper, which underscores the potential of our simple fusion strategy that already yields SOTA results in an unsupervised setup.\n\n------\n**W2:** Test other SD models with varying architectures and training settings.\n\n**A:** We further tested two alternative variations of the SD model with the same architecture but different training settings, SD-1-3 and SD-2-1-base. We also explored two work-in-progress distilled SD architectures, SD-tiny and SD-small [2] (models released by Segmind Inc.), which have 45% and 65% fewer parameters than the base model.\n\nTable 2 reports each model’s PCK metric on the SPair-71k dataset. All SD base models exhibit similar performances for both individual and fused features. \nDespite their slight performance drops, the distilled SD-tiny and SD-small variants yield noticeable improvements when fused with DINOv2. We hope to expand this analysis further to other models in future work, e.g. Pixel diffusion (Imagen [3]), token-based models (Muse [4]), once they are publicly available.\n\n|Method|PCK@0.10|PCK@0.05|PCK@0.01|\n|-|:-:|:-:|:-:|\n|SD-tiny|41.07|28.67|5.21|\n|SD-small|51.05|38.33|6.28|\n|SD-1-3|55.30|42.72|**7.72**|\n|SD-1-5|**55.90**|**42.76**|7.01|\n|SD-2-1-base|54.43|41.68|7.19|\n||\n|DINOv2-vitb14|55.15|39.66|6.12|\n|Fuse-vitb-tiny|56.96|42.35|7.27|\n|Fuse-vitb-small|60.36|45.08|7.95|\n|Fuse-vitb-1-3|**62.69**|**47.09**|8.25|\n|Fuse-vitb-1-5|62.61|46.60|**8.47**|\n|Fuse-vitb-2-1-base|62.22|46.10|8.40|\n\nTable 2. Comparison of different variants of SD models on SPair-71k, we use explicit captioner for fair comparison.\n\n------\n**W3:** Ablate on the capacity of the models employed.\n\n**A:** To ablate the model capacity, we explore the use of smaller, distilled versions. In particular:\n\n- We explored DINOv2-vits14, which has about 25% network parameters of the base DINOv2 model, As in Table 3, this substantially smaller variant still delivers comparable results to the base model. This suggests that while capacity plays a role, the core techniques remain effective even with a significantly smaller model.\n\n|Method|PCK@0.10|PCK@0.05|PCK@0.01|\n|-|:-:|:-:|:-:|\n|DINOv2-vits14|53.28|37.20|5.86|\n|DINOv2-vitb14|55.15|39.66|6.12|\n|Fuse-DINOv2-vits14|61.34|46.57|7.84|\n|Fuse-DINOv2-vitb14|63.28|47.61|8.32|\n\nTable 3. Comparison of DINOv2-vit small and base model on SPair-71k.\n- In Table 2, we explored the smaller SD-tiny and SD-small variants. While these models maintain similar properties as the base, they do perform worse overall (Table 2). The performance drop can be attributed to the \"in progress\" nature of these distilled models as well as a decrease in capacity. In general, the fusion results indicate that the DINO and SD features are complementary even when the capacity of the individual networks is changed.\n\n------\n**Q1:** Clarity and effect of the timesteps.\n\n**A:** Our method extracts features at the time step 100 of 1000. As shown in the Table 1 of the R1-Pbqb's Q3 response, feature extraction at different time steps doesn’t significantly affect the accuracy overall. We find the time step 100 to be optimal by searching it on the validation set. \n\nODISE finds that t=0 yields optimal results. This would be the case for semantic segmentation where a denoised image with clear object boundaries is critical for the accuracy. However, for semantic correspondence where semantic information is also important, feature maps with more structural information at a little bit earlier denoising step may help better.\n\n------\n**Q2:** Ablate on the refinement process for swapping.\n\n**A:** Section A.5 and Table 4 in Supplementals provide quantitative comparison on the diffusion-based refinement process. The refinement step both improves the Quality score and the CLIP score but hurts the FID scores, which is possibly due to certain artifacts that amplify the discrepancy with the distribution of real images during the refinement stage.\n\nSection B.2 in Supplementals further provides qualitative examples that the refinement process visually smooths local textures from warping results. \n\n-----\n\n**References**\n\n[1] ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models, Xu et al, 2023\n\n[2] On Architectural Compression of Text-to-Image Diffusion Models, Kim et al. 2023\n\n[3] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, Saharia et al. 2022\n\n[4] Muse: Text-To-Image Generation via Masked Generative Transformers, Chang et al. 2023"}, {"review_id": "U0LfPK5cnA", "reviewer": "Reviewer_fh35", "summary": "The paper explores the use of Stable Diffusion (SD) features for dense correspondence. The authors investigate the potential of SD and DINOv2 features and show some complementarity. SD features provide high-quality spatial information but sometimes inaccurate semantic matches while DINOv2 features offer sparse but accurate matches. The authors show that averaging the features from the two models might achieve strong performances for dense correspondence. The fused features are evaluated using a zero-shot approach, where no specialized training is performed for the correspondence task, and nearest neighbors are used for evaluation. Surprisingly, the fused features outperform state-of-the-art methods on benchmark datasets like SPair-71k, PF-Pascal, and TSS. The authors also demonstrate that these correspondences enable interesting applications such as instance swapping between images while preserving object identities.", "questions": "To provide a fair evaluation, it would be interesting to have the performances of other fused features based on DINO varaitions such as DINOv1 + SD or EsViT[1] + SD.\n\n[1] Chunyuan Li et al. “Efficient Self-supervised Vision Transformers for Representation Learning”, ICLR22", "limitations": "The authors underline the technical limitation of their work.", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "**In-depth Qualitative Analysis** The paper conducts a detailed qualitative analysis of the Stable Diffusion (SD) features and DINOv2 features, shedding light on their respective strengths and weaknesses. This analysis provides valuable insights into the semantic and texture relevance of these features, highlighting their distinct properties.\n\n**Extensive Experiments** The paper presents extensive experimental results on benchmark datasets, including SPair-71k, PF-Pascal, and TSS. The authors report significant gains compared to SOTA methods.\n\n**Application of Instance Swapping** The paper well illustrate the complementarity of the feature on this task.", "weaknesses": "**Lack of Strong Quantitative Assessments** While the paper provides an in-depth qualitative analysis of the complementarity between SD and DINOv2 features, it lacks strong quantitative assessments to support these claims. The true added value of the paper should have been a robust quantitative evaluation showcasing in particular the non-redundancy of the SD and DINOv2 features. This absence limits the overall impact and credibility of the proposed approach.\n\n**Incomplete Figure 2 ?** There appears to be an issue with Figure 2, as the bottom illustration is missing despite being mentioned in the legend.\n\n**Lack of Clarity in PCA Aggregation** The authors mention the use of PCA for aggregating SD features, but the explanation provided is not clear enough. The authors mention computing PCA across the pair of images for each layer of the feature map, followed by upsampling features from different layers to the same resolution to form the ensembled feature. I however have trouble understanding clearly what is the workflow here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**W1**: Quantitative assessments on the non-redundancy of SD and DINOv2 features.\n\n**A**: We include several quantitative evaluations that underscore the non-redundancy and distinctiveness of SD and DINOv2 features. Specifically, we provide: *1) quantitative error analysis on fused and individual features; 2) evidence of non-redundancy across datasets; 3) smoothness analysis on the TSS flow fields; and 4) the correspondence performance of the fused and independent features.* Details from these evaluations are provided as follows:\n\n1. **Quantitative error analysis on fused and individual features:** In **Supplemental A.7**, we provide an in-depth quantitative analysis on feature fusion. It includes: \n    - Distribution of outcomes, considering the performance of SD, DINOv2, and fused features -- showing the considerable proportion of cases where only one features succeeds; \n    - A detailed discussion on the conditional probability of the performance of fused features under different scenarios of SD and DINOv2 -- showing fusion can helps rectify when one or both features fail; \n    - A relative distance analysis in certain senarios -- further sheding light on how the fused features helps under certain circumstances.\n\n2. **Non-redundancy across datasets and PCK levels:** We present additional evidence of the non-redundancy of SD and DINOv2 features in Table 1, which details the error distribution for these two features on SPair-71k and PF-Pascal benchmarks at three different PCK levels. As shown in this table, in 20~30% of total cases under most settings, one feature succeeds where the other fails; this suggests that they have a substantial amount of non-redundant information.\n\n|||SD, DINO fails|SD fails, DINO correct|SD correct, DINO fails|SD, DINO correct|\n|-|-|:-:|:-:|:-:|:-:|\n|SPair-71k|PCK@0.15|21.68|15.70|13.95|48.67|\n||PCK@0.10|29.20|15.76|15.33|39.66|\n||PCK@0.05|44.50|14.20|15.81|25.50|\n||\n|PF-Pascal|PCK@0.15|5.60|8.27|11.12|75.01|\n||PCK@0.10|9.99|9.68|12.72|67.62|\n||PCK@0.05|27.07|11.98|16.78|44.17|\n\nTable 1. Distribution of outcomes under different datasets and PCK levels.\n\n3. **Smoothness analysis on the TSS flow fields:** Table 2 in the main paper offers a quantitative analysis of smoothness on computed flow fields and suggests that SD features produce dense correspondences which are much smoother than those from DINOv2 features; this strongly suggests that SD features contain more spatial information than DINO features.\n4. **Correspondence performance of fused and individual features:** In Table 3 of the main paper, the improved quantitative performance of our fusion results shows that these features have non-redundant elements (otherwise fusing them would not provide a quantitative improvement). Our fusion result leads to a 13% improvement on SPair over either individual feature. Additionally, Table 3 reports the performance of different features on per-category subsets, with accompanying analysis in Section 4.1. In particular, several categories demonstrate that DINO and SD have drastically different performance. Aero, Bike, Boat, Car, Dog, Horse, Motor, Person, Sheep all perform better with DINO features, while Cow, Plant, Train, TV perform better with SD features. Specifically, the categories that SD performs better on tend to have less texture signal (e.g. TV, Plant).\n\nWe also provide qualitative analysis to support our claim in: 1) Section 3.3 which showcases extensive qualitative experiments highlighting multiple distinctions between the two features; 2) Supplemental B.1 where we provide additional visual results to showcase the complementary.\n\nWe will highlight the quantitative analysis more clearly in the updated main paper.\n\n------\n**W2**: Clarity on the grouping of Figure 2.\n\n**A:** We divide Figure 2 into two parts with a dotted line. The top two rows show the visualization of PCA-computed features and the bottom two rows show K-Mean clustered features. We will make the figure more clear in an updated version.\n\n------\n**W3**: Clarity on the details of PCA aggregation.\n\n**A:** Here is a more detailed explanation on how to aggregate SD features using PCA. \n\nWe first extract the $i^{th}$ layer’s features for source and target images, $\\{f^s\\_i\\}\\_{i\\in[2,5,8]}$ and $\\{f^t\\_i\\}\\_{i\\in[2,5,8]}$. Next, we concatenate each layer’s source feature and target feature and compute PCA together: $\\{\\tilde f^s\\_i, \\tilde f^t\\_i = PCA(f^s\\_i || f^t\\_i)\\}\\_{i\\in[2,5,8]}$. Then we gather each layer’s dimension-reduced features $\\{\\tilde f^s\\_i\\}\\_{i\\in[2,5,8]}$ and $\\{\\tilde f^t\\_i\\}\\_{i\\in[2,5,8]}$, and upsample them to the same resolution to form the final SD feature $\\tilde f^s$ and $\\tilde f^t$.\n\nWe will include these details in the main paper.\n\n------\n**Q1**: Performance of other DINO variations.\n\n**A:** The pre-trained EsViT model checkpoint is currently inaccessible due to public access restrictions (issue #27 in the EsViT GitHub repo). Thus, we tried other DINO variations - iBOT [1], and DINOv1. As shown in Table 2, both DINOv1 and iBOT perform significantly worse than DINOv2. Surprisingly, a zero-shot fusion with SD slightly decreases the overall performance. We hypothesize that this may be due to the relatively weak performance of DINOv1 and iBOT; if these features are strictly worse than SD features, they may only contribute noise to the zero-shot fusion results. A learned projection would enable fusion to ignore features that decrease the overall performance and at least perform similarly to SD by itself.\n\n|Method|PCK@0.10|PCK@0.05|PCK@0.01|\n|-|:-:|:-:|:-:|\n|DINOv1-vitb16|33.17|19.93|2.43|\n|iBOT-vitb16|38.85|23.90|2.63|\n|DINOv2-vitb14|55.15|39.66|6.12|\n||\n|Stable Diffusion|56.18|42.80|6.79|\n|Fuse-DINOv1-vitb16|51.79|37.50|5.34|\n|Fuse-iBOT-vitb16|55.11|38.99|4.85|\n|Fuse-DINOv2-vitb14|63.28|47.61|8.32|\n\nTable 2. Comparison with DINOv2 and two other variants on SPair-71k.\n\n------\n**References**\n\n[1] iBOT: Image BERT Pre-Training with Online Tokenizer, Zhou et al. 2022"}, {"review_id": "wiAXXq0SmV", "reviewer": "Reviewer_NMVR", "summary": "This paper, for the first time, proposes a novel method to fuse Stable Diffusion features and DINOv2 features to obtain robust feature representation that readily surpasses the SOTA semantic correspondence work without further training, rather simply adopting Winner Takes All yields SOTA correspondence performance. Interestingly, with such correspondences, high-quality object swapping is also enabled without task-specific fine-tuning.  ", "questions": "See the weaknesses above. If they are adequately addressed in the rebuttal, I think this paper will be a very strong paper. My rating is solely based on the performance part, which  if clarified, then I am happy to increase my rating to accept this paper.", "limitations": "Limitations are adequately discussed. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel idea, approach and thorough analysis.\n\n2. Motivation is really good and the paper was easy to read and understand. \n\n3. Impressive performance. \n\n4. Good application (swapping)", "weaknesses": "1. This is more like a question rather than a weakness. In figure 1, the visualization shows that a cat given as a source image, the proposed approach finds dense \"correspondence\" to dog, horse and motorcycle. Is this really a correspondence? To me, it feels like this is more like  foreground segmentation.  From the colors of the features, iI could tell that the facial part of cat is matched to the frontal part of motorcycle and facial part of other animals. This is very interesting. Why do you think this visualization is obtained even though strictly speaking, there should not be correspondences?  \n\n2. Why is it that on SPair-71k, the proposed method yields leading results, but not for  PF-PASCAL and TSS? As the authors state, PF-PASCAL and TSS are less challenging datasets, but results seem to show the opposite.\n\n3. Missing baselines : for supervised ones, CATs++ (extension of CATs) and IFCAT from the same author achieve better performance. Also, PWarpC is a work that adopts unsupervised learning for semantic correspondence. I would suggest including this as well. It is better to compare with them as well. I don't think it would be a weakness even if the proposed method does not attain SOTA, since the contribution lies more on the fusion of features and their analysis. \n\n4. In implementation detail, it says the input resolution is 960 x 960. What was the evaluation keypoint annotation resolution? The input resolution, evaluation resolution and many other factors related to resolutions have high influence on performance in this task. This is addressed by PwarpC (CVPR'22) and CATs++ (TPAMI). So if the evaluation is performed at higher resolution than other works, the comparison may not be fair at all. This needs to be clarified. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Clarity on the visualization of Fig. 1.\n\n**A**: We provide the challenging cross-category semantic correspondence that matches semantically related or geometrically similar parts across different object categories or even domains. The Neural Best-Buddies [1] and DINOv2 paper also visualizes those examples. \n\nAs in Fig. 1, our method matches objects even at different poses, suggesting that diffusion features contain information related to semantic/pose/shape. This is more information than needed for foreground segmentation. We hypothesize that diffusion features contain high-level semantic and structural information which can relate conceptually different objects eg, cats and motorcycles.\n\n---\n**W2**: Clarity on the performance of PF-Pascal and TSS datasets.\n\n**A:** Our method achieves leading results on PF-Pascal and TSS datasets, specifically:\n\n- **The best results on PF-Pascal dataset among the *unsupervised* methods**, with 49.17%, 18.62%, and 6.46% relative improvement on PCK@0.05, 0.10, and 0.15, respectively (see Table 1 below). Additionally, if we use a simple form of supervision for our fused features (by learning a single projection layer with the PF-Pascal training set), **our method performs similarly to other supervised methods on PF-Pascal**. If we were to fine-tune the backbone networks (SD or DINOv2) like other supervised approaches, we would expect to see even bigger improvements.\n- **The best results on TSS among *unsupervised nearest neighbors* methods** and is near SOTA for *unsupervised* methods including those with task specific networks and losses (see Table 2 below). While Semantic-GLU-Net performs better on TSS, it is important to note that Semantic-GLU-Net significantly underperforms our fused features on SPair-71k (23.5 vs our 62.9 for PCK@0.1) and PF-Pascal (72.5 vs our 86.0 for PCK@0.1).\n\n|Technique|Method|PCK@0.15|PCK@0.10|PCK@0.05|\n|-|:-|:-:|:-:|:-:|\n|Unsupervised|CNN-Geo [43]|41.0|69.5|80.4|\n||Glu-Net [57]|42.2|69.1|83.1|\n||Semantic-Glu-Net [60]|48.3|72.5|85.1|\n||Stable Diffusion (**Ours**)|*63.2*|*77.7*|*86.3*|\n||Fuse-ViT-B/14 (**Ours**)|**72.1**|**86.0**|**90.6**|\n||\n|Supervised|PWarpC-NC-Net|79.2|92.1|95.6|\n||CATs++* |*84.9*|*93.8*|96.8|\n||IFCAT*|**88.0**|**94.8**|**97.9**|\n||Fuse-ViT-B/14 trained projection layer (**Ours**)|80.9|93.6|*96.9*|\n\nTable 1. PF-Pascal Performance. * denotes finetuned backbones. In a category, **bold** is best, *italics* is second best.\n\n|Technique|Method|FG3DCar|JODS|Pascal|Avg.|\n|-|-|:-:|:-:|:-:|:-:|\n|Unsupervised TS|CNN-Geo [43]|90.1|*76.4*|56.3|74.4|\n||PARN [23]|89.5|75.9|*71.2*|78.8|\n||GLU-Net [57]|93.2|73.3|71.1|79.2|\n||Semantic-GLU-Net [60]|**95.3**|**82.2**|**78.2**|**85.2**|\n||\n|Unsupervised NN|DINOv2|82.8|73.9|53.9|72.0|\n||Stable Diffusion (**Ours**)|93.9|69.4|57.7|77.7|\n||Fuse-ViT-B/14 (**Ours**)|*94.3*|73.2|60.9|*79.7*|\n\nTable 2. TSS Performance.  TS is task-specific, NN is nearest-neighbors. **Bold** is best, *italics* is second best. Fuse-ViT-B/14 (**Ours**) performs best among Unsupervised NN methods and second best among all Unsupervised methods (see note above regarding Semantic-GLU-Net).\n\n**With regards to the dataset difficulty**, weaker methods can often perform better on easier datasets by exploiting specific biases or limitations. Specifically, \n- **TSS** contains a limited number of object classes mostly under rigid transforms. Having a template-based parametric approach [2] can achieve good performance on TSS. \n- **PF-Pascal** contains image pairs with the same viewpoint, and does not require an intrinsic understanding of the instance's structure and orientation.\n- **SPair-71k**, in contrast, contains image pairs with more diverse viewpoint, scale, occlusion, and truncation than PF-PASCAL and TSS, is known to be more challenging and thus more informative as an evaluation. \n\nThat said, **achieving strong performance on SPair-71k is particularly commendable**. Our zero-shot setting already achieves comparable performance with the supervised SOTA on SPair-71k. With a simple form of supervision to our fused features, our method surpasses supervised SOTA (ours 78.2 vs. IFCAT* 64.4).\n\n---\n**W3**: Additional baselines.\n\n**A:** We include PWarpC-CATs in Table 4 in the main paper but didn’t include its weakly-supervised variations due to space limit. Please see Tab. 1 in R2-fsqe's W1 response, for a comparison with CATs++, IFCAT, and PWarpC; this show that we still attain SOTA in the challenging SPair-71k datasets under supervised setting. We will include these baselines in the revised version.\n\n---\n**W4**: Effect of the keypoint annotation resolution and the input image resolution.\n\n**A:** Our evaluation keypoint annotation resolution is 840, employed for all NN-based methods. Table 3 provides the PCK results for SPair-71k 20-samples subsets at different input resolutions and annotation resolutions (the min input resolution is restricted by the SD model). Similar to PWarpC's Table 1, using different annotation resolutions only marginally affects the accuracy. On the other hand, as in CATs++, we also observe that input image's resolution matters more, especially under stricter constraints (PCK@0.05 and 0.01).\n\n|Input Image Reso.|Annotation Reso.|0.10|0.05|0.01|\n|-|-|:-:|:-:|:-:|\n|960|840|63.28|47.61|8.32|\n|(feat map 60*60)|ori.|62.80|47.36|8.11|\n||256|62.54|47.58|8.36|\n||\n|512|840|61.66|40.73|4.58|\n|(feat map 32*32)|ori.|61.58|40.43|4.28|\n||256|61.51|40.35|4.40|\n\nTable 3. PCK performance of fuse features under different input image resolution and keypoint annotation resolution on SPair-71k.\n\nComparing with CATs++ under the same settings (512 input and 256 annotation resolution), our method still performs better, notably in PCK@0.10, with ours 61.51 vs theirs 59.9 (number taken from their Fig. 12a).\n\n---\n**References**\n\n[1] Neural Best-Buddies: Sparse Cross-Domain Correspondence, Aberman et al. 2018\n\n[2] GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences, Truong et al. 2020\n"}, {"review_id": "4HnEAeXt7T", "reviewer": "Reviewer_YopU", "summary": "This paper exploits Stable Diffusion (SD) features for semantic and dense correspondence tasks. The authors first conduct evaluations of SD features and found that SD features provide high-quality spatial information but sometimes inaccurate semantic matches. This paper further shows that such SD features are complementary to DINOv2 features which provide sparse but accurate matches. By fusing SD and DINOv2 features with a simple weighted concatenation strategy, this paper achieves significant performance gains over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS.", "questions": "- I am wondering what might lead to the different behaviors of Stable Diffusion features and DINOv2 features? A more in-depth discussion might make this paper stronger. Could the authors further comment on this?\n- As also discussed in the limitation section, the spatial resolutions of Stable Diffusion features and DINOv2 features are both relative low, which might hurt the performance for fine-grained details. Besides Stable Diffusion, there are also other diffusion models that directly work on the original pixel space instead of the downsampled latent space, would these kind of diffusion models potentially alleviate this issue? I am not asking for such experiments, but just wondering the authors' opinion on this.", "limitations": "Yes, the limitations are discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Extensive evaluation and analysis.** This paper conducts lots of experiments and visualizations to analyze the behavior of Stable Diffusion and DINOv2 features. The experiments are also well-organized, with a smooth logic flow, which makes it easy for readers to follow.\n- **Strong zero-shot performance.** By simply combining the Stable Diffusion and DINOv2 features, this paper shows strong zero-shot performance of the semantic correspondence task, outperforming previous task-specific methods by a large margin.\n- **The message shown in this paper could be interesting to the community.** Unlike previous methods that mostly focus on studying single image diffusion models, this paper studies the properties of diffusion model features across different images in the context of semantic correspondence. This paper shows that the publicly available Stable Diffusion and DINOv2 models contain rich information to solve the semantic correspondence task, even outperforming task-specific models in the zero-shot setting. This would be a strong message and might stimulate future work along this direction.\n\n", "weaknesses": "It's unclear how the proposed method handles outliers in the correspondences. Since the correspondences are obtained with nearest neighbor search, which might be less robust for occlusion or out-of-frame points. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**W1:** Handle outliers in the correspondences.\n\n**A:** Our method primarily focuses on analyzing properties of each feature and their fusion, and thus we adopt a very simple matching method (i.e., NN search) without using any matching priors or templates. Despite this, our method achieves outperforming/competitive accuracy on benchmark datasets (Table 3 and 4 in the main paper). However, we believe certain methods that are designed specifically for outliers in the correspondence task (OmniMotion [1] for occlusion, and PWarpC [2] for unmatchable points) can benefit from our proposed fused features. We consider integrating these features into those systems or a novel fusion architecture which identifies occluded points or outliers to be an important future direction for our work.\n\n------\n\n**Q1:** Causes of the complementary properties of SD and DINOv2.\n\n**A:** Please refer to the \"General Response Q&A section\".\n\n------\n\n**Q2:** Potential of pixel-space diffusion models for overcoming limitations in feature resolution.\n\n**A:** While we agree that this seems like a promising direction, there may be a few challenges. Pixel-space diffusion models typically operate through a coarse-to-fine methodology. They start with an initial low-resolution, often 64x64, text-to-image diffusion process, followed by (several) subsequent upsample diffusion process(es). Extracting a feature map larger than 64x64 would necessitate relying on the upsample diffusion model. However, while the upsample diffusion models are crucial for image clarity, they do not necessarily need to comprehend the inherent structure of the image. As such, these models might contain less semantic information than the text-to-image diffusion models, potentially limiting its efficacy in semantic correspondence tasks.\n\nAdditionally, from our investigation of competitive diffusion models in the current landscape, it appears that none of them simultaneously meet the conditions of 1) being open-sourced, 2) operating within the pixel-space, and 3) offering feature maps with a resolution higher than 64x64. This makes it challenging for us to test the above idea, though it certainly is interesting.\n\n|Model|FID on I-Net↓|Open-sourced|Pixel-space|Large f_map|\n|-|:-:|:-:|:-:|:-:|\n|GLIDE|12.24|||❌|\n|DALLE2|10.39|❌|||\n|SD|8.32||❌||\n|Imagen|7.27|❌|||\n|eDiff-I|6.95|❌|||\n|ERNIE-ViLG2.0|6.75|❌|❌||\n|RAPHAEL|6.61|❌|❌||\n\nTable 1. Survey of competitive diffusion models.\n\nAs a potential fix for the spatial resolution limitation, one could train an additional projection layer. This layer, when trained on top of both the input image and its corresponding low-resolution features, might offer a bridge between spatial granularity and semantic depth. We consider this a promising direction for future exploration.\n\n-----\n**References**\n\n[1] Tracking Everything Everywhere All at Once, Wang et al. 2023\n\n[2] Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences, Truong et al. 2022\n"}]

Author rebuttal:
## Acknowledgements

We thank the reviewers for the comments, extensive feedback, and recognition of the strengths of our work:

- **Comprehensive Analysis:** Our "extensive evaluation and analysis" of Stable Diffusion (SD) features and DINOv2 features provide "valuable insights" into their distinct properties and potential uses, as recognized by the reviewers (R3-fh35, R5-YopU).
  
- **Exceptional Performance:** The "impressive performance" of our method in the zero-shot setting, as well as its substantial gains over existing supervised methods, were highlighted by the reviewers (R1-Pbqb, R3-fh35, R4-NMVR, R5-YopU).
  
- **Practical Applications:** Our successful demonstration of instance swapping illustrated the real-world applicability of our method, as noted by the reviewers (R1-Pbqb, R3-fh35, R4-NMVR).
  
- **Clarity of Presentation:** The reviewers praised the "smooth logic flow" of our paper and the clarity of our explanations, noting that it was well-written and "easy to read and understand" (R1-Pbqb, R4-NMVR, R5-YopU).
  
- **Inspiration:** The reviewers commended our work as well-inspired, reflecting the successful combination of Stable Diffusion and DINOv2 features for semantic correspondence tasks, providing a refreshing perspective on the applications of diffusion models outside of image generation (R1-Pbqb, R2-fsqe, R3-fh35, R5-YopU).

We first address common questions by the reviewers, and then respond to individual inquiries.

------

## Questions & Responses

**Q:** (R1-Pbqb, R5-YopU) Why do Stable Diffusion and DINOv2 features behave differently? 

**A:**  Due to resource limitations (e.g., time, data, and computation resources), we focus primarily on the empirical study of existing models. Unfortunately, experimentally determining the causes for feature properties often requires clean ablations with respect to data, training schemes, and architecture, and we do not have the resources to do these ablations. That said, we have several conceptual hypotheses to explain the observed differences that SD has better spatial information but worse accuracy than DINOv2:

- **Training Paradigms:** DINO's self-supervised learning training works by taking an image and producing global and local views; the global views are passed to a teacher network and used to distill information into a student's representation of both the global and local views. While this encourages "local-to-global" features [1], it also has the side effect of encouraging the features to be invariant to the training augmentations used to generate the different views (e.g., color jittering, Gaussian blur, solarization, multi-crop, and horizontal flips). **This invariance in DINO decreases the network's ability to differentiate spatial information**, e.g., the left side of an object to its right side, **particularly for symmetrical objects or objects without texture signals** (such as the bus in the bottom of Figure 3).
  **On the other hand, the SD model has been trained for text-to-image generation, which requires the model to be aware of both global object structure as well as local shape and texture cues**; for example, since the model has been trained to generate both photos and sketches of dogs, the model should be able to understand where a dog's tail is in relation to its head, even with minimal texture signal.

- **Architecture Differences:** The ViT employed by DINOv2 processes the image as a stack of patches; while the positional encoding preserves a sense of spatial awareness, attention is computed globally and less emphasis may be placed on local structure. In contrast, the convolution layers in SD’s UNet may retain more details and enhance the retention of spatial information.

Despite the lack of clean ablations, there are a few experimental data points that we can consider:
- **Model Capacity:** In this rebuttal (R2-fsqe-W2 and R2-fsqe-W3), we experiment with lower capacity versions of SD and DINOv2. The smaller distilled versions of SD have the same properties as base SD and fusing these smaller models with DINOv2 still result in improved performance on SPair-71k. The same is true of lower capacity versions of DINOv2. This suggests that capacity is not the sole reason for these features.

- **Training schemes:** We investigated models with identical architectures but diverse training protocols (R2-fsqe-W2, different variants of SD). Specifically, the SD-1-5 model is fine-tuned using different steps and datasets in comparison to SD-1-3 (195,000 steps on "laion-aesthetics v2 5+" versus 595,000 steps on "laion-improved-aesthetics"). This is also distinct from the SD-2-1-base model, which is trained on the filtered LAION-5B dataset. Despite these variations, all three models demonstrate comparable performance, indicating that these properties are robust to small variations in training schemes and datasets.

In general, we think that trying to identify specific causes and provide evidence is a very interesting question and a great direction for future work.

[1] Emerging Properties in Self-Supervised Vision Transformers, Caron et al. 2021.

Meta-review: After the rebuttal and discussion phase, all reviewers recommend acceptance. The paper shows that SD features are very good for establishing dense correspondences when combined with DINO features. Before diffusion models, generative features have not been very strong for other tasks. This seems to be changing now and this paper provides insights into one area of applications: semantic correspondence and is thus valuable to the community.
Decision: Accept (poster)