SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As discussed in Section 1, optimizing text embeddings on a single image makes it susceptible to overfitting. Furthermore, the process of finding embeddings via optimization, textual inversion, has recently been shown to be sensitive to initialization [35, 36]. To overcome these issues, we apply various forms of regularization. Note that while we describe these one at a time, these two augmentations are simultaneously enabled.
Averaging across crops. Let  $\mathcal{C}_{\mathbf{c}}(\mathbf{I})$  be a cropping operation with parameters  $\mathbf{c}$ , and  $\mathcal{U}_{\mathbf{c}}(\mathbf{I})$  the operation placing the crop back to its original location; that is,  $\mathcal{C}_{\mathbf{c}}(\mathcal{U}_{\mathbf{c}}(\mathbf{x}_{\mathbf{c}})) = \mathbf{x}_{\mathbf{c}}$  for some crop  $\mathbf{x}_{\mathbf{c}}$ . For the  $\mathbf{c}$  we reduce the image dimensions to  $93\%$  (found via hyperparameter sweep) and sample a uniformly random translation within the image - we will denote this sampling as  $\mathbf{c} \sim \mathbf{D}$ . We augment the optimization in Equation 5 by averaging across cropping augmentations as:
$$
\mathbf {e} ^ {*} = \arg \min  _ {\mathbf {e}} \mathbb {E} _ {\mathbf {c} \sim \mathbf {D}} \sum_ {\mathbf {u}} \| \mathcal {C} _ {\mathbf {c}} (\mathbf {M} (\mathbf {u}; \mathbf {e}, \mathbf {I} _ {i})) - \mathcal {C} _ {\mathbf {c}} (\mathbf {M} _ {s} (\mathbf {u})) \| _ {2} ^ {2}, \tag {7}
$$
similarly, at inference time, we average the attention masks generated by different crops:
$$
\mathbf {u} _ {j} = \underset {\mathbf {u}} {\arg \max } \mathbb {E} _ {\mathbf {c} \sim \mathbf {D}} \mathcal {U} _ {\mathbf {c}} \left(\mathbf {M} \left(\mathbf {u}; \mathbf {e} ^ {*}, \mathcal {C} _ {\mathbf {c}} \left(\mathbf {I} _ {j}\right)\right)\right). \tag {8}
$$
Averaging across optimization rounds. We empirically found that doing multiple rounds of optimization is one of the keys to obtaining good performance; see Figure 9. To detail this process, let us abstract the optimization in Equation 7 as  $\mathbf{e}^{*} = \mathcal{O}(\bar{\mathbf{e}},\mathbf{I}_{i})$  , where  $\bar{\mathbf{e}}$  is its initialization. We then average the attention masks induced by multiple optimization runs as:
$$
\mathbf {u} _ {j} = \underset {\mathbf {u}} {\arg \max } \mathbb {E} _ {\bar {\mathbf {e}} \sim \mathbf {D}} \mathbf {M} (\mathbf {u}; \mathcal {O} (\bar {\mathbf {e}}, \mathbf {I} _ {i}), \mathbf {I} _ {j}). \tag {9}
$$
We evaluate semantic correspondence search on three standard benchmarks: SPair-71k [14] is the largest standard dataset for evaluating semantic correspondences composed of 70,958 image pairs of 18 different classes. Since we do not perform any training, we only use the 12,234 correspondences of the test set for evaluation; PF-Willow [13] comprises four classes - wine bottle, duck, motorcycle,
Table 1: Quantitative results - We report the Percentage of Correct Keypoints (PCK), where bold numbers are the best results amongst weakly- or un-supervised methods. Our method outperforms all weakly supervised baselines (we use the numbers reported in the literature). Note also that for PF-Willow, our method outperforms even the strongly-supervised state of the art in terms of  $\mathrm{PCK}_{@0.1}$
CUB-200	PF-Willow	SPair-71k
PCK@0.05	PCK@0.1	PCK@0.05	PCK@0.1	PCK@0.05	PCK@0.1
Strong supervision	PWarpC-NC-Net*res101	-	-	48.0	76.2	21.5	37.1
CHM	-	-	52.7	79.4	27.2	46.3
VAT	-	-	52.8	81.6	35.0	55.5
CATs++	-	-	56.7	81.2	-	59.8
Weak supervision	PMD	-	-	40.3	74.7	-	26.5
PSCNet-SE	-	-	42.6	75.1	-	27.0
VGG+MLS	18.3	25.8	41.2	63.2	-	27.4
DINO+MLS	52.0	67.0	45.0	66.5	-	31.1
PWarpC-NC-Netres101	-	-	45.0	75.9	18.2	35.3
ASIC	57.9	75.9	53.0	76.3	-	36.9
Unsupervised	DINO+NN	52.8	68.3	40.1	60.1	-	33.3
Our method	61.6	77.5	53.0	84.3	28.9	45.4
and car – with 900 correspondences in the test set; CUB-200 [37] includes 200 different classes of birds. Following ASIC [46] we select the first three classes, yielding a total of 1, 248 correspondences in the test set.
Metrics. We measure the performance of each method via the standard metric [13, 14, 23] of Percentage of Correct Keypoints (PCK) under selected thresholds – we report how many semantic correspondence estimates are within the error threshold from human-annotation. Following the standard protocol [13, 14, 23], for SPair-71k and PF-Willow thresholds are determined with respect to the bounding boxes of the object of interest, whereas for CUB-200, the thresholds are set relative to the overall image dimensions. We report results for thresholds 0.05 and 0.1, as annotations themselves are not pixel-perfect due to the nature of semantic correspondences – i.e., there is no exact geometric correspondence.
Implementation details. We use Stable Diffusion version 1.4 [30]. In our ablations, we find that more augmentation help, but with the caveat of (linearly) requiring more computation. Hence, on CUB-200 and PF-Willow datasets we use 10 optimization rounds for the embeddings and 30 random crops for the inference. For the larger SPair-71k dataset we use less - 5 embeddings and 20 crops. We choose our hyperparameters based on the validation subset of SPair-71k and  $\mathrm{PCK}_{\text{@0.05}}$  via a fully-randomized search and applied them to all datasets. We use the Adam [73] optimizer to find the prompts. For detailed hyperparameter setup see Supplementary Material.
Qualitative highlights. We show qualitative examples of the semantic correspondences estimated by our method in Figure 8. Interestingly, our method, even when it disagrees with the human annotation, provides results that can arguably be interpreted as plausible. For example, as shown in the wine bottle example at the bottom-right of Figure 8, source points occluded by the wine glasses are correctly mapped to another wine glass in the target image, which disagrees with the human annotator's label which points to the wine bottle.
Quantitative results. We report the performance of our method in Table 1. We compare our method against weakly-supervised baselines [21, 52, 46], as well as baselines from ASIC [46] that are based on general-purpose deep features - using VGG [24] features with Moving Least Squares (MLS)[56] and DINO [25] features with Moving Least Squares (MLS) or Nearest Neighbor (NN) matching.
Our method outperforms all compared weakly supervised baselines. Note that the performance gap in terms of average  $\mathrm{PCK}_{\text{@0.1}}$  compared to the second best method, ASIC [46], is large -  $20.9\%$  relative. Note also that in the case of the PF-Willow dataset, our method is on par with the current strongly supervised state-of-the-art. Even in the case of the SPair-71k dataset, our results are comparable to a very recent method VAT [15] besides the few problematic classes - bottle, bus, plant, train, tv. These problematic classes are typically those that exhibit strong symmetry, which we do not explicitly deal with. For detailed per-class results, see our Supplementary Material. Considering that our method is
(a) Individual layer performance
(b) # embeddings vs performance
(c) #crops vs performance
Figure 9: Ablations - We ablate performance as measured by the PCK@0.05 metric on the PFWillow [13] dataset. (a) Using multiple layers; (b) Using optimization augmentations; (c) Using crop augmentations. Dashed-line denotes the performance of our full method. Note that in (a) individual layer performance is significantly worse, showing that the information within layers is complimentary. In (b) and (c) using more embeddings and crops leads to improved performance.
Figure 10: Semantic correspondence between different classes - We show examples of applying our method to pairs of image from different classes. We manually mark those that seem arguably correct with blue. Our method in many cases generalizes well.
fully unsupervised when it comes to the task of semantic correspondences, these results are quite surprising.
Ablations. To ablate our model, we use the PF-Willow [13] dataset and report  $\mathrm{PCK}_{\mathbb{Q}0.05}$ . In Figure 9a, we show that using individual layers leads to significantly worse results than using multiple layers together; this demonstrates the importance of capturing semantic knowledge across multiple receptive fields. In Figure 9b, we show that using multiple optimized embeddings significantly boosts performance, and in Figure 9c, we see how using more crops further leads to improvements during inference. Finally, besides crops during inference, if we disable random crops during embedding optimization  $\mathrm{PCK}_{\mathbb{Q}0.05}$  drops to 45.5 (vs. 53.0 in our full model).
Beyond semantic correspondences of the same class.. We further experiment with correspondences across different classes - e.g., correlating different animals. As shown in Figure 10 in many cases our method provides correspondences that are arguably correct even for objects that are of different classes. This includes being from similar classes (sheep and dog) to more different classes (bird and airplane), and to wildly different classes (chair and horse).
We have demonstrated the remarkable potential of leveraging diffusion models, specifically Stable Diffusion [30], for the task of semantic correspondence estimation. We have shown that by simply optimizing and finding the embeddings for a location of interest, one can find its semantically corresponding locations in other images, although the diffusion model was never trained for this task. We further introduced design choices that significantly impact the accuracy of the estimated correspondences. Ultimately, our approach significantly outperforms existing weakly supervised methods on SPair-71k [14], PF-Willow [13], and CUB-200 datasets [37] (20.9% relative for SPair-71k), and even achieves performance on par with strongly supervised methods in PF-Willow dataset [13].
Limitations and future work. This study highlights the emergent properties of training large models on vast amounts of data, revealing that a model primarily designed for image generation can also be utilized for other tasks, specifically semantic correspondences. The success of our method
underscores the importance of exploring the hidden capabilities of these large text-to-image generative models and suggests that there may be other novel applications and tasks that can benefit from the vast knowledge encoded within these models. For example, as an immediate application, our method can be used to scale up training of 3D generative models such as FigNeRF [49] with images from the web without human supervision.
We note that our method does have its limitations. Many of the failure modes include dealing with symmetric objects. An explicit way to deal with these cases with more refined techniques to extract correspondences may help solve this problem. It also requires significant compute. On an NVIDIA RTX 3090 GPU, finding a single prompt for a single keypoint takes 30 seconds. We suspect that one could potentially train a semantic correspondence network based on our estimates to achieve both scalability in terms of training data and fast inference time.
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google, Digital Research Alliance of Canada, and by Advanced Research Computing at the University of British Columbia.
[1] B.B. Hansen and B.S. Morse. Proceedings of the IEEE conf. on computer vision and pattern recognition. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 1999.
[2] Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape context: A new descriptor for shape matching and object recognition. Advances in Neural Information Processing Systems, 13, 2000.
[3] Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition, 2016.
[4] David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004.
[5] Alex Fisher, Ricardo Cannizzaro, Madeleine Cochrane, Chatura Nagahawatte, and Jennifer L Palmer. Colmap: A memory-efficient occupancy grid mapping framework. Robotics and Autonomous Systems, 2021.
[6] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision, 2016.
[7] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2018.
[8] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2020.
[9] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2018.
[10] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision, 2020.
[11] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.
[64] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv Preprint, 2022.
[65] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-Efficient Semantic Segmentation with Diffusion Models. In International Conference on Learning Representations, 2022.
[66] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. In International Conference on Robotics and Automation, 2022.
[67] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 2023.
[68] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 2023.
[69] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 2023.
[70] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv Preprint, 2014.
[71] Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamarneh. Slime: Segment like me. arXiv Preprint, 2023.
[72] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Conference on Medical Image Computing and Computer Assisted Intervention, 2015.
[73] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv Preprint, 2014.