SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junyi Zhang<sup>1</sup> Charles Herrmann<sup>2</sup> Junhwa Hur<sup>2</sup> Luisa F. Polania<sup>2</sup>  
Varun Jampani<sup>2</sup> Deqing Sun<sup>2</sup> Ming-Hsuan Yang<sup>2,3</sup>
$^{1}$  Shanghai Jiao Tong University  $^{2}$  Google Research  $^{3}$  UC Merced
https://sd-complements-dino.github.io
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, our analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of the two features works surprisingly well, and a zero-shot evaluation via nearest neighbor search on the fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences enable high-quality object swapping without task-specific fine-tuning.
Recent findings from text-to-image diffusion models [44, 48, 50] have attracted increasing attention to analyze and understand their internal representations for a single image. Due to the text-to-image diffusion model's ability to effectively connect a text prompt to the content in the images, they should be able to understand what an image contains and were recently shown in [11, 31] to be effective classifiers. Furthermore, since these models can generate specific scenes and have some degree of control over the layout and placement of objects [72], they should be able to localize objects and were shown in [68] to be effective for semantic segmentation. In addition, several other methods [12, 17, 27, 65] focus on accessing and editing features for downstream tasks such as stylization and image editing.
However, these approaches have almost exclusively examined the properties of text-to-image diffusion features on a single image; significantly less is known about how these features relate across multiple, different images and objects. Simply put, in Stable Diffusion, how similar are the features of a cat to the features of a dog? In this paper, we focus on understanding how features in different images relate to one another by examining Stable Diffusion (SD) features through the lens of semantic correspondences, a classical vision task that aims to connect similar pixels in two or more images.
Figure 1: Semantic correspondence with fused Stable Diffusion and DINO features. On the left, we demonstrate the accuracy of our correspondences and demonstrate the instance swapping process. From top to bottom: Starting with pairs of images (source image in orange box), we fuse Stable Diffusion and DINO features to construct robust representations and build high-quality dense correspondence. This facilitates pixel-level instance swapping, and a subsequent stable-diffusion-based refinement process yields a plausible swapped instance. On the right, we demonstrate the robustness of our approach by matching dog, horses, cows, and even motorcycles to the cat in the source image. Our approach is capable of building reasonable correspondence even when the paired instances exhibit significant differences in categories, shapes, and poses.
For the correspondence task, we show that simply ensembling and reducing these features through basic techniques can lead to an effective representation that quantitatively does as well as other state-of-the-art representations for dense and semantic correspondences. However, a close analysis of the qualitative results leads to an interesting discovery. We observe that SD features have a strong sense of spatial layout and generate smooth correspondences (front and back of a bus are represented differently as shown in Fig. 3 left bottom), its pixel level matching between two objects can often be inaccurate (a bus front might match to another bus back). In fact, compared to existing features like DINOv1 [2], SD features appear to have very different strengths and weaknesses.
While prior work [2] showed that DINOv1 can be an effective dense visual descriptor for semantic correspondence and part estimation, to the best of our knowledge, this has not yet been thoroughly analyzed for DINOv2, which demonstrated improved downstream performance for multiple vision tasks but not correspondence. For correspondence, we show that DINOv2 does outperform DINOv1 but requires a different post-processing. DINOv2 generates sparse but accurate matches, which surprisingly, form a natural complement to the higher spatial information from SD features.
In this work, we propose a simple yet effective strategy for aligning and fusing these features. The fused representation utilizes the strengths of both feature types. To establish correspondence, we evaluate the zero-shot (no training for the correspondence task) performance of these features using nearest neighbors. Despite this very simple setup, our fused features outperform previous methods on the SPair-71k [36], PF-Pascal [20], and TSS [59] benchmark datasets. In addition, the fused representation facilitates other novel tasks such as instance swapping, where the objects in two images can be naturally swapped using estimated dense correspondence, while successfully maintaining its identity. The main contributions of this work are:
- We demonstrate the potential of the internal representation from a text-to-image generative model for semantic and dense correspondence.
- We analyze the characteristics of SD features, which produce spatially aware but inaccurate correspondences, and standard representation learning features, i.e., DINOv2, which generate accurate but sparse correspondence and show that they complement each other.
- We design a simple strategy to align and ensemble SD and DINOv2 features and demonstrate that these features with a zero-shot evaluation (only nearest neighbors, no specialized training) can outperform many SOTA methods for semantic and dense correspondence. Experiments show that
the fused features boost the performance of the two and outperform previous methods significantly on the challenging SPair-71k dataset  $(+13\%)$ , as well as PF-Pascal  $(+15\%)$  and TSS  $(+5\%)$ .
- We present an instance swapping application using high-quality correspondence from our methods.
Feature descriptor for dense correspondence. Deep models [15, 40, 45, 52, 66, 71] learn effective features for dense correspondence that can be robust to photometric and geometric changes such as rotation, scaling, or perspective transformation. However, most of the methods focus on the image matching task for outdoor scenes to account for rigid transformation. Amir et al. [2] demonstrate that features extracted from self-supervised Vision Transformer (DINOv1 [6]) serve as effective dense visual descriptors with localized semantic information by applying them to a variety of applications, e.g., (part) co-segmentation and semantic correspondences. Recently, DINOv2 [41] introduces a scaled-up version of DINOv1 [6] by increasing the model size and combining a large quantity of curated data. DINOv2 shows strong generalization to downstream tasks such as classification, segmentation, and monocular depth, but has not been extensively studied on correspondence tasks. Our work shows that DINOv2 also leads to significantly better correspondence results than DINOv1, but those DINO features in general produce sparse and noisy correspondence field.
Semantic correspondence. Semantic correspondence [33] aims to estimate dense correspondence between objects that belong to the same object class with different appearance, viewpoint, or non-rigid deformation. Conventional methods consist of three steps [64]: feature extraction, cost volume construction, and displacement field [60-63] or parameterized transformation regression [25, 28, 46, 47, 54]. Those methods, however, cannot effectively determine dense correspondence for complex articulated deformation due to their smooth displacement fields or locally varying parameterized (affine) transformations. Motivated by a classical congealing method [29], a couple of recent methods introduce to learn to align multiple objects in the same class by using pretrained DINOv1 feature [18, 39] or GAN-generated data [42]. The methods show the possibility that knowledge learned for different tasks can also be utilized for the dense correspondence task. However, those models are not effective in handling severe topological changes probably due to their strong rigidity assumption in the object alignment. In contrasts, we discover that the Stable Diffusion features for the image generation task exhibit great capability for the zero-shot dense correspondence task as well, where challenging articulate, non-rigid deformation exists.
Diffusion models. Diffusion probabilistic models (diffusion models) [55] and denoising diffusion probabilistic models [23, 38] have demonstrated state-of-the-art performance for image generation task [13, 37, 48, 50, 51, 57]. In addition, the diffusion models have been applied to numerous vision tasks, e.g., image segmentation [3, 4, 8, 26, 58, 67], object detection [7], and monocular depth estimation [14, 53]. Recently, much attention has been made to analyze what pre-trained diffusion models understand and extract its knowledge for single-image tasks, e.g., panoptic segmentation [68], semantic segmentation and depth [73]. In this paper, we unveil the potential of diffusion features for zero-shot dense correspondence between images.
3 Semantic correspondences via Stable Diffusion and Vision Transformer
We first examine properties of Stable Diffusion (SD) features for semantic correspondence, then study the complementary nature between DINO and SD features, and finally introduce a simple and effective fusion strategy to leverage the strengths of both features.
3.1 Are Stable Diffusion features good for semantic correspondence?
Stable Diffusion (SD) [48] demonstrates a remarkable ability to synthesize high-quality images conditioned on a text input, suggesting that it has a powerful internal representation for images and can capture both an image's content and layout. Several recent works [4, 68, 73] have used the pre-trained SD features for single-image tasks, such as semantic segmentation and depth perception. As features play an important role for visual correspondence, we are interested in investigating whether SD features can help establish semantic correspondences between images.
PCA Visualization
Input image
Layer 5
Figure 2: Analysis of features from different decoder layers in SD. Top: Visualization of PCA-computed features from early (layer 2), intermediate (layers 5 and 8) and final (layer 11) layers. The first three components of PCA, computed across a pair of segmented instances, serve as color channels. Early layers focus more on semantics, while later layers concentrate on textures. Bottom: K-Means clustering of these features. K-Means clusters are computed for each image individually, followed by an application of the Hungarian method to find the optimal match between clusters. The color in each column represents a pair of matched clusters.
Layer 11
Layer  $2 + 5 + 8$
Next, we briefly explain how we extract SD features. The architecture of Stable Diffusion consists of three parts: an encoder  $\mathcal{E}$ , a decoder  $\mathcal{D}$  that is derived from VQGAN [16] and facilitates the conversion between the pixel and latent spaces, and a denoising U-Net  $\mathcal{U}$  that operates in the latent space. We first project an input image  $x_0$  into the latent space via the encoder  $\mathcal{E}$  to produce a latent code  $z_0$ . Next, we add a Gaussian noise  $\epsilon$  to the latent code according to a pre-defined time step  $t$ . Then, taking the latent code  $z_t$  at the time step  $t$  with the text embedding  $C$  as inputs, we extract the features  $\mathcal{F}_{\mathrm{SD}}$  from the denoising U-Net. The entire process can be formally represented as follows:
$$
z _ {0} = \mathcal {E} (x _ {0}), \quad z _ {t} = \sqrt {\bar {\alpha} _ {t}} z _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon , \quad \mathcal {F} _ {\mathrm {S D}} = \mathcal {U} (z _ {t}, t, C), \tag {1}
$$
where the coefficient  $\bar{\alpha}_t$  controls the noise schedule, as defined in [23]. For the text embedding  $C$ , we follow [68] to use an implicit captioner, which empirically is superior to explicit text prompts.
While prior work [65] has reported that the intermediate U-Net layers have more semantic information for image-to-image translation task, it is unclear whether these features are suitable for semantic correspondence. To examine their properties, we first perform principal component analysis (PCA) on the features at various layers of the U-Net decoder. As shown in the top of Fig. 2, early layers tend to focus more on semantics and structures (e.g. wheels and seats) but at a coarse level; the later layers tends to contain detailed information about texture and appearance. Furthermore, we apply K-Means clustering  $(k = 5)$  on the features of paired instances and visually examine whether the clustered features contain consistent semantic information across intra-class examples. As shown in the bottom of Fig. 2, different parts of the objects are clustered and matched across the instance pairs. Both PCA and K-means analysis show that earlier layer features capture coarse yet consistent semantics, while later layers focus on low-level textural details. These observations motivate us to combine the features at different levels  $(2 + 5 + 8)$  to capture both semantic and local details, through concatenation.
A simple concatenation, however, results in an unnecessarily high-dimensional feature  $(1280 + 960 + 480 = 2720)$ . To reduce the high dimension, we compute PCA across the pair of images for each feature layer, and then aggregate them together to the same resolution. Specifically, we first extract the  $i^{\text{th}}$  layer's features for source and target images,  $f_{i}^{s}$  and  $f_{i}^{t}$ . Next, we concatenate each layer's source feature and target feature and compute PCA together:
$$
\tilde {f} _ {i} ^ {s}, \tilde {f} _ {i} ^ {t} = \operatorname {P C A} \left(f _ {i} ^ {s} | | f _ {i} ^ {t}\right), \quad i \in \{2, 5, 8 \}.
$$
Then we gather each layer's dimension-reduced features  $\tilde{f}_i^s$  and  $\tilde{f}_i^t$ , and upsample them to the same resolution to form the final SD feature  $\tilde{f}^s$  and  $\tilde{f}^t$ . As shown in the last column of Fig. 2, the
Table 1: Evaluation of correspondence on SPair-71k subsets. We sample 20 pairs for each category and report the PCK@0.10 score for different settings. We underline the best performances achieved by each model. S/8: A small model with a patch size of 8, B/14: Base model with a patch size of 14.
Model	Layer 11↑	Layer 9↑
Token	Key	Query	Value	Token	Key	Query	Value
DINOv1-ViT-S/8	28.8	30.4	26.9	25.8	30.9	31.4	29.9	27.7
DINOv2-ViT-S/14	52.7	30.3	30.6	47.1	45.5	12.7	13.2	40.6
DINOv2-ViT-B/14	55.7	42.6	40.7	53.4	50.8	25.2	25.3	46.0
Figure 3: Analysis of different features for correspondence. We present visualization of PCA for the inputs from DAVIS [43] (left) and dense correspondence for SPair-71k [36] (right). The figures show the performance of SD and DINO features under different inputs: identical instance (top left), pure object masks (bottom left), challenging inputs requiring semantic understanding (right top) and spatial information (right bottom). Please refer to Supplemental B.1 for more results.
ensembled features strike a balance between the two different properties and focus on both semantics and textures at a sufficient resolution.
In addition, we find that the specific location within the decoder layer to extract the feature also plays a important role. U-Net's decoder layer inputs the feature of the previous decoder layer as well as the skip-connected encoder layer feature, and outputs the decoder feature after passing through a sequence of layers: a convolutional layer, a self-attention layer, and a cross-attention layer. We empirically find that using only the decoder feature achieves better performance than combining it with the encoder feature, and also outperforms the features of a single sub-layer (e.g., convolutional layer, self-attention layer). Please refer to appendix for more details.
One natural question is whether SD features provide useful and complementary semantic correspondences with respect to more widely studied discriminative features. Next, we discuss the semantic correspondence properties of the widely used DINOv1 [6] features in conjunction with the successor DINOv2 [41] features.
3.2 Evolution of DINO features for semantic correspondence
Caron et al. [6] show that self-supervised ViT features (DINO), "contain explicit information for semantic segmentation and are excellent k-NN classifiers." Amir et al. [2] further demonstrate that DINO features have several interesting properties that allow zero-shot semantic correspondence across images. Most recently, Oquab et al. [41] introduce DINOv2 by scaling up their training data and the model size and observe significant performance boost on numerous single-image tasks, such as segmentation and depth. In this work, we examine whether DINOv2 can significantly improve semantic correspondence across images.
We test different internal features from DINOv2 on the standard correspondence benchmark SPair-71k [36] (please refer to Sec. 4 for details). As shown in Tab. 1 PCK results (higher the better),
Table 2: Smoothness (first-order difference) of the valid estimated flow field on the TSS dataset. We report the results of three techniques and include the ground truth as a reference (lower is smoother).
Method	FG3DCar↓	JODS↓	Pascal↓	Avg.↓
DINOv2-ViT-B/14	6.99	10.09	15.14	10.15
Stable Diffusion	3.48	7.87	8.44	5.90
Fuse-ViT-B/14	3.52	7.55	8.75	5.96
Ground Truth	2.22	5.20	4.06	3.40
Figure 4: Semantic flow maps using different features. White mask indicates valid pixels and orange mask separates the background flow. SD features yield smoother flow fields versus DINOv2's isolated outliers.
DINOv2 features achieve a significant improvement over DINOv1. However, we observe that the best performance is achieved by the "token" facet from the last (11th) layer of DINOv2, while Amir et al. [2] find that, for DINOv1, the best performance is achieved by the "key" facet in earlier layers. We refer to the token features from layer 11 of DINOv2 as  $\mathcal{F}_{\mathrm{DINO}}$  or DINO.
3.3 Comparing the strengths and weaknesses of SD and DINO features
We now discuss the properties and potential complimentary nature of SD and DINO features. Aside from the PCA visualization above, we compute dense correspondences between image pairs using both features via nearest neighbor search and examine their behaviors under different input conditions.
Correspondence for the same object instance. For simpler cases where instances are the same in paired images (Fig. 3 top left), both SD and DINO features perform well. However, their performance differs on textureless examples. When giving an object mask as input (Fig. 3 bottom left), we observe that DINOv2 features cannot establish valid correspondence for this textureless example, while SD features work robustly as they have a strong sense of spatial layout.
Correspondence between different object instances. On examples with intra-class variation, both SD and DINO features perform well although in different regions. DINO yields more accurate matches (e.g., in the thigh region of Fig. 3 top right case), but the correspondence field tends to be noisy, as demonstrated in both cases on the right side of Fig. 3. In contrast, SD excels at constructing smoother correspondence, as shown in both cases on the right side of Fig. 3, and provides crucial spatial information, notably in the bottom right case.
Spatial coherence of the correspondence. To analyze the spatial coherence of the estimated correspondence, we can extract the semantic flow, i.e., the 2D motion vector for every pixel from the source image to the corresponding pixel in the target image. We then compute the smoothness, i.e. first-order difference, of the estimated semantic flow on the TSS dataset [59], as shown in Tab. 2. This analysis reveals the inherent smoothness in the flow maps generated by different methods. The Stable Diffusion features yield smoother results compared to DINOv2-ViT-B/14, suggesting that Stable Diffusion features exhibit better spatial understanding than the DINO features.
Discussion. Tab. 2 evaluates the spatial smoothness of the flow fields estimated by different methods on the TSS dataset. The features of Stable Diffusion lead to smoother dense correspondence than DINO features. Fig. 4 visually shows that the correspondence by DINO features contains moderate amounts of isolated outliers, while that by Stable Diffusion is more spatially coherent.
Summary. The studies above reveal the complementary nature of DINO and Stable Diffusion features. DINO features can capture high-level semantic information and excel at obtaining sparse but accurate matches. In contrast, Stable Diffusion features focus on low-level spatial information and ensure the spatial coherence of the correspondence, particularly in the absence of strong texture signals. The complementary properties of DINOv2 and diffusion features offer promising potential for enhancing the performance of semantic correspondence tasks. A natural question arises:
3.4 How to fuse SD and DINO features?
Based on the discussions above, we propose a simple yet effective fusion strategy to exploit SD and DINO features. The core idea is to independently normalize both features to align their scales and distributions, and then concatenate them together:
$$
\mathcal {F} _ {\text {F U S E}} = (\alpha | | \mathcal {F} _ {\mathrm {S D}} | | _ {2}, (1 - \alpha) | | \mathcal {F} _ {\mathrm {D I N O}} | | _ {2}) \tag {2}
$$
where  $||\cdot ||_2$  denotes the L2 normalization,  $\alpha$  is a hyperparameter that controls the relative weight of the two features. An important aspect of this fusion approach is the selection of the weight  $\alpha$ . Empirically, we find that  $\alpha = 0.5$  offers a good balance between the two features, effectively leveraging their complementary strengths. See the supplementary material for the ablation studies.
Observing the surprising effectiveness of our simple fusion strategy in Fig. 3, we see notable improvements in challenging cases from the SPair-71k dataset. The combined features noticeably outperforms either features alone in all the cases. It not only demonstrates enhanced precision and reduced noise in correspondences but also retains the inherently smoother transitions and spatial information unique to the SD feature. Furthermore, as shown both quantitatively in Tab. 2 and qualitatively in Fig. 4, the performance of the fused features aligns remarkably closely with that of the SD features in retaining smoother transitions and spatial information.
In the later section, we further show the remarkable effectiveness of our simple fusion strategy through extensive comparison with existing methods on various public benchmark datasets.
4 Experiments and analysis
We first provide in-depth analyses of both sparse and dense correspondence tasks on public benchmark datasets. Then as a novel application of our approach, we present a simple, efficacious instance swapping method between paired images, utilizing dense correspondence from our method.