SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation of features for correspondence. We evaluate the extracted features in two settings. First, in a zero-shot setting, we follow [2] by searching the nearest neighbors directly on the feature maps of pair images. Second, for datasets with a training split, we add a bottleneck layer [22, 68] on top of the extracted features and finetune it in a supervised manner. It is guided by the same objective as in [35]: a CLIP-style symmetric cross entropy loss with respect to corresponding keypoints.
Implementation details. We employ the Stable Diffusion v1-5 model as our feature extractor, with the timestep for the diffusion model to be  $t = 100$  by default. We use the input resolution  $960 \times 960$  for the diffusion model and 840 for the DINO model, which results in a feature map with a 60 resolution. For PCA, we adopt a nearly optimal approximation SVD algorithm [19] for better efficiency and reduce the feature dimension to 256 when fusing with DINO-ViT-B/14. When visualizing dense correspondence, we apply a segmentation mask obtained from [68] as a preprocessing step. All experiments are conducted on a single NVIDIA RTX3090 GPU.
Datasets. For the sparse correspondence task, we evaluate on two standard datasets, SPair-71k [36] and PF-Pascal [20], which consists of 70958 and 1341 image pairs respectively, sampled from 18 and 20 categories. We evaluate the dense correspondence on TSS [59], the only dataset that provides dense correspondence annotations on 400 image pairs derived from FG3DCAR [32], JODS [49], and PASCAL [21] datasets.
Metrics. To evaluate the correspondence accuracy, we use the standard Percentage of Correct Keypoints (PCK) [70] metric with a threshold of  $\kappa \cdot \max(h, w)$  where  $\kappa$  is a positive integer. and  $(h, w)$  denotes the dimensions of the bounding box of an instance in SPair-71k, and the dimensions of the images in PF-Pascal and TSS respectively, by following the same protocol in prior work [18, 39, 42].
4.1 Sparse correspondence
Tab. 3 provides our zero-shot and supervised evaluation on the SPair-71k dataset. Our zero-shot method (Fuse-ViT-B/14) significantly outperforms all methods including unsupervised methods as well as even previous supervised methods by a large margin, achieving a leading average PCK score of 64.0. We also observe that our fusion strategy significantly improves the performance of the DINOv2 baseline (DINOv2-ViT-B/14), resulting in a PCK score improvement of 8.4p (from 55.6 to 64.0). The Stable Diffusion features, not only being competitive to DINOv2 on the average PCK metric,
Table 3: Evaluation on SPair-71k. Per-class and average PCK@0.10 on test split. The methods are categorized into four types: strong supervised (S), GAN supervised (G), unsupervised with task-specific design  $(\mathbf{U}^{\mathrm{T}})$ , and unsupervised with only nearest neighboring  $(\mathbf{U}^{\mathrm{N}})$ . *: fine-tuned backbone. †: a trained bottleneck layer is applied on top of the features. We report per image PCK result for the (S) methods and per point result for other methods. The highest PCK among supervised methods and all other methods are highlighted in bold, while the second highest are underlined. Our NN-based method surpasses all previous unsupervised methods significantly.
Method	Aero Bike Bird Boat	Bottle	Bus	Car	Cat	Chair	Cow Dog	Horse	Motor	Person	Plant	Sheep	Train	TV	All
S	SCOT [34]	34.9	20.7	63.8	21.1	43.5	27.3	21.3	63.1	20.0	42.9	42.5	31.1	29.8	35.0	27.7	24.4	48.4	40.8	35.6
CATs* [9]	52.0	34.7	72.2	34.3	49.9	57.5	43.6	66.5	24.4	63.2	56.5	52.0	42.6	41.7	43.0	33.6	72.6	58.0	49.9
PMNC* [30]	54.1	35.9	74.9	36.5	42.1	48.8	40.0	72.6	21.1	67.6	58.1	50.5	40.1	54.1	43.3	35.7	74.5	59.9	50.4
SCorrSAN* [24]	57.1	40.3	78.3	38.1	51.8	57.8	47.1	67.9	25.2	71.3	63.9	49.3	45.3	49.8	48.8	40.3	77.7	69.7	55.3
CATs++* [10]	60.6	46.9	82.5	41.6	56.8	64.9	50.4	72.8	29.2	75.8	65.4	62.5	50.9	56.1	54.8	48.2	80.9	74.9	59.9
DINOv2-ViT-B/14†	80.4	60.2	88.1	59.5	54.9	82.0	73.5	89.1	53.3	85.5	73.6	73.8	65.2	72.3	43.6	65.6	91.4	60.3	69.9
Stable Diffusion† (Ours)	75.6	60.3	87.3	41.5	50.8	68.4	77.2	81.4	44.3	79.4	62.8	67.7	64.9	71.6	57.8	53.3	89.2	65.1	66.3
Fuse-ViT-B/14† (Ours)	81.2	66.9	91.6	61.4	57.4	85.3	83.1	90.8	54.5	88.5	75.1	80.2	71.9	77.9	60.7	68.9	92.4	65.8	74.6
G	GANgealing [42]	-	37.5	-	-	-	-	67.0	-	-	23.1	-	-	-	-	-	-	57.9	-
UT	VGG+MLS [1]	29.5	22.7	61.9	26.5	20.6	25.4	14.1	23.7	14.2	27.6	30.0	29.1	24.7	27.4	19.1	19.3	24.4	22.6	27.4
DINO+MLS [1, 5]	49.7	20.9	63.9	19.1	32.5	27.6	22.4	48.9	14.0	36.9	39.0	30.1	21.7	41.1	17.1	18.1	35.9	21.4	31.1
NeuCongeal [39]	-	29.1	-	-	-	-	53.3	-	-	35.2	-	-	-	-	-	-	-	-	-
ASIC [18]	57.9	25.2	68.1	24.7	35.4	28.4	30.9	54.8	21.6	45.0	47.2	39.9	26.2	48.8	14.5	24.5	49.0	24.6	36.9
UN	DINOv1-ViT-S/8 [2]	57.2	24.1	67.4	24.5	26.8	29.0	27.1	52.1	15.7	42.4	43.3	30.1	23.2	40.7	16.6	24.1	31.0	24.9	33.3
DINOv2-ViT-B/14	72.7	62.0	85.2	41.3	40.4	52.3	51.5	71.1	36.2	67.1	64.6	67.6	61.0	68.2	30.7	62.0	54.3	24.2	55.6
Stable Diffusion (Ours)	63.1	55.6	80.2	33.8	44.9	49.3	47.8	74.4	38.4	70.8	53.7	61.1	54.4	55.0	54.8	53.5	65.0	53.3	57.2
Fuse-ViT-B/14 (Ours)	73.0	64.1	86.4	40.7	52.9	55.0	53.8	78.6	45.5	77.3	64.7	69.7	63.3	69.2	58.4	67.6	66.2	53.5	64.0
but also notably excel in certain categories where spatial information is critical (e.g., "potted plant", "train", and "TV"). The same observation holds for supervised setting, notably the improvement in PCK score of 14.7p (from 59.9 to 74.6) compared to previous method.
In Tab. 4, we further validate our approach on the PF-Pascal dataset, which is a less challenging dataset that presents low variations on appearance, pose, or shape between instance pairs. Our method consistently outperforms all unsupervised methods, achieving the highest average PCK scores across all thresholds. Our fusion approach (Fuse-ViT-B/14), again, substantially improves the performance over the DINOv2 baseline (DINOv2-VIT-B/14), highlighting the effectiveness of our fusion strategy that benefits from both features.
4.2 Dense correspondence
Tab. 4 further provides the dense correspondence evaluation on the TSS dataset [59]. Our fusion approach (Fuse-ViT-B/14) outperforms all unsupervised nearest-neighbor-search-based methods  $(\mathrm{U}^{\mathrm{N}})$  on the TSS dataset. The result again confirms the effectiveness of our fusion strategy, demonstrating a substantial gain of  $7.7\mathbf{p}$  over the DINOv2-ViT-B/14 baseline. Note that the TSS dataset contains less challenging examples with low variations on appearance, viewpoint, and deformation (e.g. car, bus, train, plane, bicycle, etc.), where having a strong spatial smoothness prior can be an advantage for achieving good metrics on the benchmark.
4.3 Instance swapping
Given a pair of images with instances of the same or similar categories, instance swapping is defined as the task of transposing an instance from a target image (target instance) onto the instance(s) in a source image (source instance), while preserving the identity of the target instance and the pose of the source instance. This creates a novel image where the target instance appears naturally integrated into the source environment.
By leveraging the high-quality dense correspondence obtained by our method, we can achieve plausible instance swapping through a straightforward process: 1) Initially, we upsample the low-resolution feature map to match the size of the target image; 2) Based on the upsampled feature map, we perform pixel-level swapping on the segmented instances according to the nearest neighbor patch, yielding a preliminary swapped image; 3) To enhance the quality of the swapped image, we refine the preliminary result with a stable-diffusion-based image-to-image translation process [65].
Table 4: Evaluation on PF-Pascal and TSS. The highest PCK among nearest-neighboring based unsupervised are highlighted in bold, while the second highest are underlined. Our fusion results result in a large improvement over the DINO baselines, and are comparable to other task-specific methods. S: Supervised methods,  $\mathbf{U}^{\mathrm{T}}$ : Task-specific unsupervised methods,  $\mathbf{U}^{\mathrm{N}}$ : Nearest-neighboring based unsupervised methods. *: fine-tuned backbone. †: a trained bottleneck layer is applied on top of the features.
Method	PF-Pascal, PCK@κ	TSS, PCK@0.05
0.05	0.10	0.15	FG3DCar	JODS	Pascal	Avg.
S	SCOT [34]	63.1	85.4	92.7	95.3	81.3	57.7	78.1
CATs* [9]	76.8	92.7	96.5	92.1	78.9	64.2	78.4
PWarpC-CATs* [64]	79.8	92.6	96.4	95.5	85.0	85.5	88.7
CATs++* [10]	84.9	93.8	96.8	-	-	-	-
DINOv2-ViT-B/14†	74.2	90.8	95.4	-	-	-	-
Stable Diffusion† (Ours)	77.4	89.7	93.9	-	-	-	-
Fuse-ViT-B/14† (Ours)	80.9	93.6	96.9	-	-	-	-
UT	CNNGeo [46]	41.0	69.5	80.4	90.1	76.4	56.3	74.4
PARN [25]	-	-	-	89.5	75.9	71.2	78.8
GLU-Net [60]	42.2	69.1	83.1	93.2	73.3	71.1	79.2
Semantic-GLU-Net [63]	48.3	72.5	85.1	95.3	82.2	78.2	85.2
UN	DINOv1-ViT-S/8 [2]	41.5	62.4	72.5	64.7	51.2	36.7	53.3
DINOv2-ViT-B/14	56.2	77.3	83.3	82.8	73.9	53.9	72.0
Stable Diffusion (Ours)	61.0	80.3	86.1	93.9	69.4	57.7	77.7
Fuse-ViT-B/14 (Ours)	73.0	86.1	91.1	94.3	73.2	60.9	79.7
Stable Diffusion
Figure 5: Qualitative comparison of instance swapping with different features. SD features deliver smoother swapped results, DINOv2 reveals greater details, and the fused approach takes the strengths of both. Notably, the fused features generate more faithful results to the reference image, as highlighted by the preserved stripes on the cat instance in the top-right example. Please refer to Supplemental B.2 for more results.
DINOv2
To be more specific with the refinement process, we begin by inverting the swapped image to the initail noise with DDIM inversion [56], followed by a denoising DDIM sampling process where we extract the features from the diffusion model. Finally, merging this with a prompt that includes the instance category, we generate a refined image. This image exhibits a swapped instance with an appearance that is visually more coherent and plausible. Please refer to Supplemental A.8 and B.2 respectively for the quantitative and qualitative analysis of the refinement process.
Fig. 5 provides a visual comparison of instance swapping using different features. SD leads to smooth results, whereas DINOv2 accentuates fine details. The fusion of both methods showcases a balance between spatial smoothness and details. More examples are available in the supplementary material.
We also compare different features quantitatively on an 800-pair subset of [69] benchmark. The quantitative evaluation in Tab. 5 reveals consistent performance gains of the fused approach across all three metrics: FID score, quality score, and CLIP score. We follow the same metrics from [69]: FID score is measured using the CLIP features between the generated images and the COCO 2017 testing images, and CLIP score measures the cosine similarity between CLIP features extracted from the edited region of the source image and the target image.
These results show the strength of the dense correspondences with fused features in not only maintaining the instance's appearance but also generating high-quality images. The CLIP score is higher for
Table 5: Quantitative comparison for instance swapping. The best performance approach is in bold.
Method	FID score(CLIP-based)↓	Quality score↑	CLIP score↑
DINOv2-ViT-B/14	12.49	63.18	72.63
Stable Diffusion	13.72	61.38	71.48
Fuse-ViT-B/14	12.47	64.84	73.21
Table 6: Distribution of outcomes under different datasets and PCK levels. Under most settings, one feature succeeds while the other fails in  $20\sim 30\%$  of total cases (see row 2 & 3), which suggests that they have a substantial amount of non-redundant information.
Cases	SPair-71k, PCK@κ	PF-Pascal, PCK@κ
0.15	0.10	0.05	0.15	0.10	0.05
SD, DINO fails	21.7	29.2	44.5	5.6	10.0	27.1
SD fails, DINO correct	15.7	15.8	14.2	8.3	9.7	12.0
SD correct, DINO fails	14.0	15.3	15.8	11.1	12.7	16.8
SD, DINO correct	48.6	39.7	25.5	75.0	67.6	44.2
the fused approach, indicating that the fused feature representation is more faithful to the reference image. Meanwhile, the higher quality scores and lower FID scores point to the superior perceptual quality and semantic coherence of the images generated by the fused approach.
4.4 Limitations
Although our approach shows promising correspondence performance, it does come with certain limitations. The relatively low resolution of the fused features impedes the construction of precise matches, which particularly required in dense correspondence tasks as in the TSS dataset. Detailed exploration of failure cases can be found in the supplementary material. Additionally, the integration of the Stable Diffusion model, while only requiring a single inference, considerably increases the cost of the correspondences computation in comparison to using DINO features only.
4.5 Quantitative analysis of feature complementary
We present additional quantitative analysis on the non-redundancy of SD and DINOv2 features in Tab. 6, which details the error distribution for these two features on SPair-71k and PF-Pascal benchmarks at three different PCK levels. In  $20\sim 30\%$  of total cases under most settings, one feature succeeds where the other fails, suggesting that they have a substantial amount of non-redundant information. For a more detailed analysis on this non-redundancy, please refer to Supplemental A.10.
5 Discussion on feature behavior
The distinct behaviors observed in Stable Diffusion (SD) and DINOv2 features have spurred questions regarding the underlying causes. Though it is challenging to validate due to resource limitations, we suggest some possible explanations: Training paradigms, DINO's self-supervised learning approach could induce an invariance to spatial information because of its training augmentations. This contrasts with SD, which, being trained for text-to-image synthesis, inherently demands awareness of both global and local visual cues, possibly leading to its heightened spatial sensitivity. Architectural differences, DINOv2, built upon the ViT model, interprets images as sequences of patches. Even with positional encoding, it might not prioritize local structures as much. On the other hand, the convolution layers in SD's UNet may retain more details and enhance the retention of spatial information. In general, identifying the causes of these variances remains an intriguing topic and is a great direction for future exploration.
In this work, we explore the potential of leveraging internal representations from a text-to-image generative model, namely Stable Diffusion, for the semantic correspondence task. We reveal the complementary nature of the Stable Diffusion features and self-supervised DINO-ViT features, providing a deeper understanding of their individual strengths and how they can be effectively combined. Our proposed simple strategy of aligning and fusing these two types of features has empirically proven to significantly enhance performance on the challenging SPair-71k dataset and outperform existing methods, suggesting the benefits of pursuing better features for visual correspondence.
[69] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by Example: Exemplar-based image editing with diffusion models. arXiv:2211.13227, 2022. 9
[70] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2878-2890, 2012. 7
[71] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned invariant feature transform. In European Conference Computer Vision, pages 467-483, 2016. 3
[72] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023. 1
[73] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv:2303.02153, 2023. 3