SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Klemen Kotar*, Stephen Tian*, Hong-Xing Yu, Daniel L. K. Yamins, Jiajun Wu  
Stanford University  
*Equal contribution  
{klemenk, tians, koven, yamins, jiajunw}@stanford.edu
The human visual system can effortlessly recognize an object under different extrinsic factors such as lighting, object poses, and background, yet current computer vision systems often struggle with these variations. An important step to understanding and improving artificial vision systems is to measure image similarity purely based on intrinsic object properties that define object identity. This problem has been studied in the computer vision literature as re-identification, though mostly restricted to specific object categories such as people and cars. We propose to extend it to general object categories, exploring an image similarity metric based on object intrinsics. To benchmark such measurements, we collect the Common paired objects Under different Extrinsic's (CUTE) dataset of 18,000 images of 180 objects under different extrinsic factors such as lighting, poses, and imaging conditions. While existing methods such as LPIPS and CLIP scores do not measure object intrinsics well, we find that combining deep features learned from contrastive self-supervised learning with foreground filtering is a simple yet effective approach to approximating the similarity. We conduct an extensive survey of pre-trained features and foreground extraction methods to arrive at a strong baseline that best measures intrinsic object-centric image similarity among current methods. Finally, we demonstrate that our approach can aid in downstream applications such as acting as an analog for human subjects and improving generalizable re-identification. Please see our project website at https://s-tian.github.io/projects/cute/ for visualizations of the data and demos of our metric.
Human vision is extraordinarily robust against extrinsic factors that affect object appearances. When driving on sunny or snowy days, in familiar streets or new cities, we can recognize and track traffic lights, other cars, and pedestrians effortlessly. Such an ability to recognize an object based purely on its intrinsic properties, irrespective of any extrinsic factors such as lighting and imaging conditions, object pose, and background, is a key aspect of intelligent vision systems.
Existing computer vision systems often struggle with variations in lighting conditions, object poses, and backgrounds [Hendrycks and Dietterich, 2019, Drenkow et al., 2021, Lee and Kim, 2023]. For example, visual perception models for autonomous driving can fail during extreme weather conditions [Michaelis et al., 2019], and current video generation models suffer from temporal consistency of object identities, producing flickering artifacts that are easily noticeable by humans [Ho et al., 2022]. To understand and improve these artificial vision systems, an important step is to measure the similarity between visual contents (such as images) purely based on the intrinsic object properties that correspond to object identities.
Figure 1: For each reference image, green checkmarks indicate which image a given metric scores as closer to the reference image. Our metric scores images containing the same object as closer than similar but ultimately different ones.
This idea has been studied in the context of re-identification (Re-ID), where the task is to judge whether a pair of images contain an object with the same identity. However, prior approaches to Re-ID are limited to specific categories such as humans [Zheng et al., 2016] or vehicles [Liu et al., 2016]. In this work, we generalize this problem by measuring visual similarities between the intrinsics of general objects that define their identities.
While many prior visual similarity metrics have been proposed, we find empirically that they do not serve our goal well, as exemplified in Figure 1. For instance, image-level similarity metrics like PSNR, SSIM [Wang et al., 2004], and LPIPS [Zhang et al., 2018] consider the appearance of the entire image without distinguishing between intrinsic and extrinsic factors. CLIPScore [Radford et al., 2021] focuses on high-level semantic concepts and does not fully account for low-level intrinsic properties of object instances such as shape and texture.
Benchmarking intrinsic object similarity metrics in a systematic way requires images of slightly different objects with the same semantics, combined with varying extrinsics. Existing object-centric datasets that contain similar objects do not capture the same objects with varying poses or lighting conditions. On the other hand, datasets that contain objects with these factors of variation do not include sufficiently similar objects within a single category. This motivates us to collect a new dataset, Common paired objects Under different Extrinsics (CUTE), to evaluate these metrics. CUTE contains 18,000 images of 180 objects from semantically similar groups in varying extrinsic conditions such as lighting and orientation as well as in-the-wild images, as shown in Figure 3.
Using the CUTE dataset, we analyze existing similarity metrics and a simple but surprisingly effective framework for approximately measuring intrinsic object similarity. This approach combines foreground filtering and visual features of deep networks trained by self-supervised learning. We conduct an in-depth investigation into which pre-trained features and foreground extraction method provide the best measure of this similarity. From our results, we propose a strong baseline that incorporates two key ingredients that lead to a compact object-centric measurement: using pre-trained representations from DINOv2 [Oquab et al., 2023] and patch-level foreground feature pooling.
Our work represents a step forward in the foundational understanding of how we can model and compute visual similarities based on intrinsic object identity. On one hand, it contributes to the broader scientific objective of creating AI systems that understand the visual world as we do. We show that our measurement approach is able to better approximate the visual similarity perceived by humans than previous metrics [Bonnen et al., 2021]. On the other hand, it holds promise in helping artificial vision systems generalize across different environments and settings. We show that in a visual re-identification task, our metric helps domain-specific vision models generalize better in unseen environments.
In summary, our contributions are as follows: Firstly, we collect a dataset called CUTE containing 18,000 images of 180 grouped objects in varying pose, lighting, and in-the-wild settings to benchmark approaches on measuring intrinsic object similarity. We then perform an empirical study of metrics based on pre-trained visual feature extractors to determine which is most effective at measuring this similarity. From this, we propose a simple but strong baseline to measure the similarity called foreground feature averaging (FFA) that combines foreground filtering with features learned by DINOv2 [Oquab et al., 2023], a self-supervised deep model. Lastly, we find that our approach can aid in downstream applications such as acting as an analog for human subjects and improving generalizable re-identification.
Image similarity metrics. Many prior methods have been proposed for evaluating the similarity between images, from classical metrics such as peak signal-to-noise ratio to perceptual metrics like the structural similarity index measure (SSIM) [Wang et al., 2004]. Recently, metrics based on deep neural network features have been used to measure image similarity [Salimans et al., 2016, Heusel et al., 2017, Zhang et al., 2018, Sylvain et al., 2021] for training and evaluating generative models. Our metric seeks to compare images based on the intrinsic of objects present in the scene, rather than provide a pixel-by-pixel comparison or image quality assessment.
Concurrent work introduced a text-conditioned metric of image similarity called GeneCIS [Vaze et al., 2023]. It measures a model's ability to adapt to a range of similarity conditions. Our metric on the other hand focuses on a single dimension of similarity - the similarity between the intrinsics of objects. While GeneCIS mines its data from pre-existing datasets and annotates it post-hoc, we collect the data for our benchmark systematically to ensure the ground truth similarity values. DreamSim [Fu et al., 2023] attempts to capture the human notion of visual similarity using a deep neural network trained on a dataset of synthetically generated image triplets with human-annotated similarity preferences. Our metric instead focuses on the ground truth identity of objects, not human preferences.
Re-identification. A relevant problem in computer vision is person/vehicle re-identification [Zheng et al., 2016, Liu et al., 2016], which focuses on judging if a pair of person/vehicle images capture the same identity. Early work in this area typically uses hand-crafted visual features [Liao et al., 2015, Liu et al., 2012] with learned classifiers [Prosser et al., 2010] or distance metrics [Zheng et al., 2012], and recent work exploits deep models [Xiao et al., 2016, Sun et al., 2018, Luo et al., 2019, Ming et al., 2022] learned from pair-wise person/vehicle image datasets [Zheng et al., 2015, Wei et al., 2018, Liu et al., 2017, Lou et al., 2019]. In the deep learning era, re-identification has been studied with a supervised learning setup supported by annotated datasets [Li et al., 2014, Zheng et al., 2015, Liu et al., 2017, Bai et al., 2018], and later extended to unsupervised learning [Yu et al., 2017, Fan et al., 2018, Yu et al., 2019, Ge et al., 2020] and a generalizable setup where the testing datasets are not seen during model training [Jin et al., 2020, Zhang et al., 2022]. The intrinsic object-centric image similarity problem that we study can be considered as a generalization of the re-identification problem, where we aim at quantitatively measuring the instance-level similarity between a pair of images of general objects. Our quantitative measurement can be directly applied to the re-identification problem and we evaluate our final metric along with baselines on this problem in Section 5.2.
Object-centric vision datasets. Prior efforts have collected visual datasets consisting of different instances of objects from common categories. CO3D (v2) [Reizenstein et al., 2021], Objectron [Ahmadyan et al., 2021], and the Object Scans dataset [Choi et al., 2016] consist of real-world videos of different object instances from typical categories. However, they are not an ideal testbed for our metric as extrinsic factors such as lighting do not vary within a particular video, making it impossible to separate object identity from these cues. The Amazon-Berkeley Objects Dataset [Collins et al., 2022] contains 3D models and catalog images for many items, but these images are not guaranteed to be taken from the real world or even to include the object itself. On the other hand, our dataset contains not only images taken from real-world objects of various categories, but also contains stimuli obtained by varying the lighting conditions and object poses.
Self-supervised image representation learning methods. We find that an approach that leverages features learned by deep models trained in a self-supervised manner on large-scale datasets is surprisingly effective on this task. While self-supervised methods continue to be developed [He et al., 2019, Chen et al., 2020b, Caron et al., 2020, Chen et al., 2020a, Grill et al., 2020], we find that DINOv2 [Oquab et al., 2023] yields the best results on our task. While Amir et al. [2021] have explored the use of DINOv1 [Caron et al., 2021] features as dense visual descriptors for co-segmentation and correspondences, in this work we explore how these features can be used to compute intrinsic object-centric similarities.
3 Measuring Intrinsic Object-Centric Distances
Practical measurement. We aim at a metric such that for a given image of an object, no other object should have a lower distance than any image of the same object. To this end, we have conducted a thorough evaluation of approaches based on deep features learned from self-supervised training,
Figure 2: The Foreground Feature Averaging (FFA) (Crop-Feat) pipeline. Each input image is resized to a 336px square, then passed through the DINOv2 [Oquab et al., 2023] model as well as a foreground segmentation model. The foreground features are averaged and compared to another image via cosine similarity. The segmentation is computed at a higher resolution for this visualization.
which we present in Section 4.2. Informed by this, we propose a simple yet effective approach based on deep features learned from self-supervised training, called foreground feature averaging (FFA), as a strong baseline for measuring intrinsic object similarity. Our approach computes embeddings for each image using pre-trained deep models and then computes a similarity score from those embeddings. We show an illustration of our approach in Figure 2.
Generalizable pre-trained deep feature backbone. What types of embeddings will enable our metric to exhibit invariances to extrinsics, and generalize to arbitrary objects? Our preliminary exploration showed that features obtained from text-aligned encoders trained on internet scale data such as CLIP [Radford et al., 2021], which have become common backbones in many modern vision pipelines, oftentimes perform poorly at measuring the similarity between highly similar objects. We hypothesize this is because it can be difficult to describe the difference between two similar objects semantically, even when humans can differentiate between them based on visual features.
On the other hand, we find that contrastively-trained models such as DINOv2 [Oquab et al., 2023] produce features that contain a large amount of instance-level information, which is useful for measuring intrinsic object similarity, so we utilize it as the backbone of our metric. DINOv2 encodes input images into patch-level features as well as a global feature. We experiment using both the global feature and the patch-level features. We do not fine-tune the DINOv2 model – our new dataset is used purely as a test set to avoid any possible overfitting brought by fine-tuning.
Foreground filtering. Another challenge in computing object-centric similarity is the influence of the image's background, especially when computing image-level features. For our metric, we implement a background cropping procedure that makes our metric more robust for small objects and images with complex backgrounds (such as grass or benches). We analyze two practical instantiations of foreground filtering: (1) cropping the image with off-the-shelf segmentation models before computing features (Crop-Img) and (2) computing deep features on the entire image and then discarding the patch features associated with the background (Crop-Feat). We perform foreground cropping with an off-the-shelf foreground segmentation model, CarveKit [Selin, 2022].
Finally, we compute the cosine similarity between these embeddings for each image as a inverted proxy for the intrinsic object-centric image similarity. This enables non-parametric classification by simply considering image pairs with the highest cosine similarity as representing images of the same object. We implement our metric in PyTorch [Paszke et al., 2019] and release an easy-to-use implementation of it at https://github.com/s-tian/CUTE.
4 Dataset and Benchmarking
4.1 The CUTE Dataset
In this section, we describe the composition and capture of our new dataset, CUTE, that allows for systematic evaluation of intrinsic object-centric similarity metrics. The dataset can be downloaded at https://purl.stanford.edu/gj714cj0414.
Figure 3: Example images from the CUTE dataset. The top four rows show images captured for paired objects in different illumination conditions and object poses. The last row shows random samples of object instances from across the dataset.
Category configuration. Our dataset is composed of object instances from each of 50 semantic object categories. Objects belonging to a particular semantic category have differences such as shape or texture that identify them as distinct objects. For example, the category "apples" contains apples with distinct shapes and textures. The dataset contains a total of 180 objects. These categories range from fruits like "apples" and "oranges", to household items like "plates" and "forks". Examples of the objects in the dataset are shown in Figure 3, and a full list of object categories can be found in the appendix. Of the 50 object categories, 10 categories contain ten object instances for additional intra-class variation, while all other categories contain two object instances.
Extrinsic variations. To test the ability of metrics to isolate object intrinsics from visual stimuli, which are composed of both intrinsics and extrinsic, we collect images of each object with varying extrinsic parameters. In particular, for each of the 180 objects, we consider three types of extrinsic configurations: different illumination, different object poses, and in-the-wild captures. For different illumination and object poses, we consider a studio setup, where we have 4 lighting conditions and 24 object poses, leading to a total of  $4 \times 24 = 96$  images of the same object. We capture in-the-wild images under 4 different environments using different cameras (i.e., different imaging systems). Therefore, our CUTE dataset consists of a total of 18,000 images. In the following, we provide details on our capture setup. More imaging details are in Appendix A.
- Different illumination. For the studio capture configurations, we place each object at the center of a turntable in a controlled environment. We capture images using a Sony  $\alpha$  7IV mirrorless camera equipped with a FE 24-70mm F2.8 GM lens. We vary the illumination of the object by turning on each of three LED light panels placed to the left, right, and back of the stage in turn, or turning all of them off, for a total of four lighting settings.
- Different object poses. We vary the orientation of each object by rotating it on an automatic turntable at a constant velocity and capturing an image approximately every 15 degrees of rotation.
- In-the-wild capture. Finally, for each object, we collect "in-the-wild" images in both indoor and outdoor settings. We collect four images for each object, each in a different location, and with varying imaging conditions: a combination of mobile phones and the aforementioned mirrorless camera. This represents a more realistic distribution of images.
Figure 4: Example images from the apple, banana, and mouse categories. Our dataset contains 10 object instances each for 10 specific categories.
In-the-wild
4.2 Evaluation
In this section, we profile the performance of popular image similarity metrics and similarities computed using several deep self-supervised learning backbones, including our proposed FFA metric, using the CUTE dataset. We use a subset of CUTE with two object instances per category to ensure a balanced evaluation.
We perform a group-based evaluation, measured by image retrieval (quantified by top-1 accuracy and mean average precision (mAP)). The retrieval candidate with the maximum score is predicted as containing the same object. For CLIPScore and ours which are embedding-based, we also use K-means (with  $K = 2$ ) to cluster embeddings to measure how well the embedding space gets separated according to object identity (quantified by the adjusted Rand index [Rand, 1971] (ARI)). We use separate groups to evaluate the sensitivity of metrics for each of the following scenarios:
- Illumination: Each group contains 8 images, 4 containing one object in 4 different lighting conditions, and 4 containing the paired object in 4 different lighting conditions. The pair of objects have similar poses. Thus, 24 groups are formed for each object pair, one for each object pose. In each group, we iterate over every image as the query, and the other 7 as retrieval candidates.
- **Pose:** Each group contains 48 images and is formed by 24 images of one object from each of 24 poses, and 24 of the paired object from 24 poses. Four groups are formed per object pair, corresponding to each lighting condition. In each group, we iterate over every image as the query, and the other 47 as retrieval candidates.
- In-the-wild: Each group contains 8 images, with 4 images from different scenes for each of the paired objects. So we have 1 group for each pair. In each group, we iterate over every image as the query, and the other 7 as retrieval candidates.
- All: Each group contains all 200 images for a given object pair. We iterate over every image as the query and use all other 199 images as retrieval candidates.
Results. As Table 1 shows, our FFA approach compares favorably to prior image similarity methods at most of the evaluation groups. In particular, in the in-the-wild group which is particularly difficult, SSIM and LPIPS are no better than random guess (their top-1 accuracy is around  $50\%$ ), and CLIPScore suffers from different object in the same environment. For controlled groups (illumination and pose), although the top-1 accuracies are above  $60\%$  and  $90\%$  respectively for both FFA and CLIPScore, the retrieval results are far from optimal reflected by imperfect mAP. The clustering results show low ARI scores for both CLIPScore and ours, suggesting that both cannot separate the embedding space very well, leaving clear room for improvement. Nevertheless, our simple approach performs better at most evaluation tasks.
In Figure 5, we show confusion matrices for a few representative examples from each of the first three groups. In these examples, we show that our metric is able to score images that contain the same object more highly than images containing subtly different objects, even when conditions such as lighting and object pose vary. In contrast, LPIPS is highly sensitive to lighting and background, as
Foreground feature avg. (ours)
LPIPS
CLIPScore
Foreground feature avg.(ours)
Figure 5: Qualitative visualizations of evaluations on our dataset. Each grid depicts the pairwise similarity between sets of four images from each group. Images in green frames contain the same objects. An ideal metric would score all values in the upper left and lower right quadrants of each grid, which are comparisons between the same objects, most highly (closer to 1). Our metric displays this behavior, while LPIPS and CLIPScore do not. Please zoom in to see the visual differences between each pair of objects. The displayed images are cropped for better visibility in this visualization only.
1.0 0.0
Setting	In-the-wild	Illumination	Pose	All
mAP↑	top-1↑	ARI↑	mAP↑	top-1↑	ARI↑	mAP↑	top-1↑	ARI↑	mAP↑	top-1↑	ARI↑
LPIPS	38.7	49.1	-	41.6	53.9	-	38.2	49.3	-	39.6	49.8	-
SSIM	40.1	49.5	-	39.4	52.1	-	40.3	50.6	-	39.9	50.0	-
CLIPScore	54.4	13.3	-8.7	62.0	38.8	-1.0	82.8	98.2	48.6	65.2	94.9	-0.1
CLIPScore + Crop	69.3	59.5	15.4	69.3	75.0	11.5	77.7	94.7	38.7	68.1	94.5	18.6
DreamSim	56.9	11.8	-6.3	60.7	38.5	-11.0	82.7	96.5	48.2	64.3	93.2	3.6
DreamSim + Crop	66.8	47.0	6.0	69.4	72.0	-4.0	80.6	96.4	39.9	66.8	95.0	3.7
DinoV2	70.9	47.3	19.8	86.0	82.3	54.0	85.2	97.2	56.1	78.4	95.7	40.3
DinoV2 + Crop	79.1	69.0	41.0	84.0	85.5	39.8	83.1	96.4	53.0	76.2	96.1	36.6
DinoV1	52.6	3.8	-8.9	58.6	37.0	-10.8	81.8	97.2	47.2	63.6	93.5	1.8
DinoV1 + Crop	64.4	37.8	7.1	67.0	63.0	-4.7	79.6	96.5	38.0	66.4	94.8	4.0
Unicom	63.4	37.0	4.8	64.9	46.5	6.7	81.9	97.1	48.5	66.6	94.8	9.7
Unicom + Crop	73.9	66.3	25.5	72.2	72.3	11.3	78.0	93.8	39.5	67.9	93.5	15.1
FFA Crop-Img (ours)	76.7	68.3	35.5	81.7	83.0	48.2	79.1	95.2	41.5	72.2	94.9	26.8
FFA Crop-Feat (ours)	75.1	61.8	27.2	88.8	89.0	62.9	81.9	96.6	46.8	77.1	96.2	40.5