SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Performance of image similarity metrics on differentiating object instances from the paired object section of the dataset. We evaluate them by image retrieval, measured by top-1 accuracy (\%) and mean average precision (mAP, %). For embedding-based approach including ours and CLIPScore, we also evaluate it by clustering measured by the adjusted Rand index (ARI) which is multiplied by 100 in the table with 100 indicating perfect clustering, 0 indicating a random clustering and -50 indicating worst-case clustering.
can be seen in the "illumination" and "in-the-wild" groups. CLIPScore scores the highest at matching objects to different poses (though not by a wide margin) while performing worse than our metric - often significantly - in all other categories.
We also evaluate the DreamSim [Fu et al., 2023] metric on our dataset and find that it performs similarly to CLIP. To ablate the effect of foreground cropping, we also test both CLIPScore and DreamSim in a pipeline with the CarveKit [Selin, 2022] as in our FFA pipeline. We find that adding cropping decreases the performance of CLIPScore and DreamSim in the pose category and increases the performance in the other categories. This suggests that cropping the background tends to decrease performance in the pose category - perhaps due to the high similarity of the images and imperfect background removal.
5 Applications
5.1 Human Subject Analog
We also evaluate foreground feature averaging (FFA) on a dataset of stimuli used in prior studies of human perception mechanisms [Bonnen et al., 2021]. In the study, human subjects are presented with a panel of four images of similar objects and asked to pick the odd one out. The objects are divided along two axes as high and low similarity (to other objects in the panel) and as familiar or unfamiliar (familiar objects are similar to those seen in everyday life, while unfamiliar objects are more abstract). Figure 6 contains a sample of these images. This task was designed to study the effects of the Medial Temporal Lobe (MTL) in image processing, so results were collected for both humans with normal MTL functions and humans with lesioned MTLs.
For neural networks, we encode each image and measure which embedding lies farthest from the others and select it as the odd image out.
Findings. We report the results in Table 2. We find that our model outperforms the previously studied best baseline (ResNet-50) [Bonnen et al., 2021], and approaches the performance of human subjects with a lesioned MTL, who rely only on the ventral visual stream (VVS). This forces them to operate closer to a single forward pass in a neural network. Human subjects with a limited  $200\mathrm{ms}$  exposure to the stimulus exhibit similar behavior, in line with the findings that the low similarity object discrimination task requires only the VVS. We also evaluate the performance of directly using the DINOv2 class token, and find that it outperforms forward feature pooling in two of the subsets.
These findings suggest that our model approaches human-level performance on tasks that require only the VVS, but falls far short on tasks that require the MTL. We also demonstrate that a large gap
Similarity	Familiar	Unfamiliar
High	Low	High	Low
ResNet-50	0.28	0.90	0.18	0.72
FFA Crop-Feat (ours)	0.40	0.93	0.27	0.80
DINOv2	0.42	0.91	0.32	0.74
Human (lesioned MTL)	0.50	0.95	0.20	0.95
Human	0.90	0.95	0.80	0.98
Table 2: Classification accuracy of deep neural networks (ResNet-50 is used as the best baseline from Bonnen et al. [2021]) and human subjects on oddity visual discrimination tasks.
Metric	VeRi	CityFlow
top-1↑	top-5↑	top-1↑	top-5↑
SpCL	43.0	57.8	26.7	35.2
DINOv2 + Crop	44.6	59.5	33.5	40.1
FFA (Crop-Img)	41.0	55.7	28.6	36.1
SpCL+LPIPS	41.6	56.5	24.0	33.1
SpCL+DINOv2+Crop	46.1	60.5	32.6	39.6
SpCL+FFA (Crop-Img)	44.0	58.2	27.7	35.8
Table 3: Generalizable vehicle Re-ID results using different distance metrics including a strong Re-ID model SpCL [Ge et al., 2020].
Similarity ↓ Familiarity ↓
Figure 6: Examples of stimuli presented to humans and our metric. Tasks vary across two dimensions: similarity and familiarity. In each task, the human or model is presented with four objects and asked to determine which object is different from the others.
Similarity  $\downarrow$  Familiarity  $\uparrow$
Similarity  $\uparrow$  Familiarity  $\downarrow$
Similarity  $\uparrow$  Familiarity  $\uparrow$
in difficulty exists between the high and low similarity discrimination tasks, even with cutting-edge neural networks trained on hundreds of millions of images, just as they do with humans. Additionally, our approach outperforms humans with lesioned MTL on the Unfamiliar High similarity test case, suggesting that the model is less sensitive to familiarity of objects than human subjects. Overall, our approach is a step towards improved artificial analogs of human subjects, encouraging further studies to eventually create AI systems that understand the visual world as humans do.
5.2 Vehicle Re-Identification
In this section, we use our similarity measurement based on FFA to augment the performance of existing models on a generalizable re-identification (Re-ID) task. Specifically, we show how FFA can be used to achieve stronger transfer performance on a vehicle Re-ID task. We investigate transfer scenarios between two datasets: VeRi [Liu et al., 2016] and CityFlow-ReID [Naphade et al., 2021]. This transfer setup is referred to as generalizable Re-ID [Jin et al., 2020]. For the CityFlow dataset, as test labels are not provided, we construct a validation set from 100 randomly selected vehicle IDs and randomly sample 1000 query images from that subset.
Results. We report the results in Table 3. While the two datasets both contain images of vehicles tracked across multiple cameras, many highly-performant models like self-paced contrastive learning (SpCL) [Ge et al., 2020] do not generalize very well to different datasets due to domain gaps such as different lighting conditions, camera view angles, and background clutter [Song et al., 2020]. To improve the generalization of the model, we augment SpCL with our FFA model measurement via a simple late fusion. Specifically, we compute an  $\alpha$ -weighted average of the distances computed by the SpCL model and cosine similarity distances computed by our metric. We set the  $\alpha$  value to 0.6 for VeRi and 0.9 for CityFlow after tuning over  $\alpha = [0.1, 0.2, \dots, 0.9]$  on a validation set. As shown in Table 3 and qualitatively in Figure 7, our distance metric aids SpCL in generalization on the VeRi dataset and directly outperforms it on CityFlow.
In this work, we study the problem of measuring object image similarity based solely on the intrinsic properties of the object. We find that a simple approach based on deep features from self-supervised learning can effectively capture the object intrinsics without being distracted much by extrinsic
Figure 7: Using our foreground feature averaging (FFA) model directly yields improved vehicle re-identification performance compared to a SpCL model transferred to this task on the challenging CityFlow dataset. In this example, FFA retrieves the correct vehicle in the top four candidates while SpCL only re-identifies it correctly at the third best prediction.
variations such as lighting, object pose, and backgrounds, to approximately measure the similarity. Evaluation on our CUTE dataset validates this finding, yet it also shows that there is room for future improvements. We demonstrate that our approach, with the goal to measure such intrinsic object-centric image similarity, can better approximate human visual similarity understanding behavior, showing the promise to contribute to creating artificial agents that perceive the visual worlds as we do. We hope our findings encourage further explorations in exploiting such similarity for other applications, such as consistent video generation.
Limitations. Our approach is focused and evaluated on images containing single objects. The CUTE dataset does not contain images depicting occluded objects or with object poses varied relative to the turntable. Their capture conditions are limited in terms of geographic and outdoor lighting conditions, and while we attempt to vary the capture devices themselves, only four total devices are used.
We thank Kyle Sargent for helpful discussions. This work is in part supported by Air Force Office of Scientific Research (AFOSR) YIP FA9550-23-1-0127, ONR MURI N00014-22-1-2740, NSF CCRI #2120095, and Amazon, Bosch, Ford, and Google. ST is supported by NSF GRFP Grant No. DGE-1656518.
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. arXiv:2112.05814, 2021.
Yan Bai, Yihang Lou, Feng Gao, Shiqi Wang, Yuwei Wu, and Ling-Yu Duan. Group-sensitive triplet embedding for vehicle reidentification. IEEE Transactions on Multimedia, 2018.
Tyler Bonnen, Daniel L.K. Yamins, and Anthony D. Wagner. When the ventral visual stream is not enough: A deep learning account of medial temporal lobe involvement in perception. *Neuron*, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020a.
Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object-centric image generation from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Sagar Vaze, Nicolas Carion, and Ishan Misra. GeneCIS: A benchmark for general conditional image similarity. arXiv:2306.07969, 2023.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer GAN to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 79–88, 2018.
Tong Xiao, Hongsheng Li, Wanli Ouyang, and Xiaogang Wang. Learning deep feature representations with domain guided dropout for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Hong-Xing Yu, Ancong Wu, and Wei-Shi Zheng. Cross-view asymmetric metric learning for unsupervised person re-identification. In Proceedings of the IEEE international conference on computer vision, 2017.
Hong-Xing Yu, Wei-Shi Zheng, Ancong Wu, Xiaowei Guo, Shaogang Gong, and Jian-Huang Lai. Unsupervised person re-identification by soft multilabel learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Pengyi Zhang, Huanzhang Dou, Yunlong Yu, and Xi Li. Adaptive cross-domain learning for generalizable person re-identification. In Proceedings of the European Conference on Computer Vision, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identification: Past, present and future. arXiv:1610.02984, 2016.
Wei-Shi Zheng, Shaogang Gong, and Tao Xiang. Reidentification by relative distance comparison. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012.
Our dataset consists of a total of 50 categories with 2 or 10 instances each, for a total of 180 objects. The specific categories are listed below:
1. Apple 18. Bell pepper (orange) 35. Bowl
2. Strawberry 19. Bell pepper (red) 36. Notebook
3. Orange 20. Can of tomatoes 37. Book
4. Pear 21. Eggplant 38. Diet Coke
5. Apricot 22. Green water bottle 39 Red soda
6. Banana 23. Marker 40 Rose
7. Mango 24. Pencil (blue) 41. Cake
8.Broccoli 25.Pen 43.
9.Carrot 26.Mug 42.Coll.
10. Potato 27. Ceramic Mug 45. Cardboard box
11. Yellow onion 28. Fork 44. Mouse
12.Shallot 29.Spoon 43.Keybo
13. Tomato 30. Butter Knife 46. Cable
14. Grapes 31. Spatula (wood) 47. Screwdriver
15.Lettuce 32.Cups 48.Pliers
16. Avocado 33. Ceramic pot 49. Hammer
17. Bell pepper (yellow) 34. Plate 50. Screw
Additionally, ten categories contain ten object instances. These categories are enumerated below:
1. Apple 5. Fork 9. Screw
2. Banana 6. Spoon 10. Screwdriver
3. Pen 7.Book
4. Ceramic Mug 8. Mouse
Light setting	Shutter speed	F-number	ISO	Focal length
Left, back, right	1/640s	f/2.8	2000	39mm
Low light	1/20s	f/2.8	2000	39mm
Table 4: Camera settings for our dataset.
We capture images using a Sony  $\alpha$  7IV mirrorless camera equipped with a FE 24-70mm F2.8 GM lens. The camera settings used in our different illumination and different object poses data capture configurations are enumerated in Table 4. We show our capture setup in Figure A1.
For the different object poses data capture configuration, we use the turntable to perform a full 360 degree rotation at a constant velocity with a period of approximately 24 seconds. We then configure the camera to capture 24 images shooting at an interval of 1 second.
We capture images in the controlled setting at a resolution of  $3168 \times 3168$ .
For "in-the-wild" settings, we capture images using cell phones (iPhone 13, iPhone 14 Pro, iPhone 12 Mini) at varying resolutions.
B Data
Dataset description. We provide a dataset description in a dataset sheet: https://github.com/s-tian/CUTE/blob/main/datasheet.md
Figure A1: Capture setting for our controlled different illumination and different object poses configurations.
Link and license. The dataset is uploaded for public download under the CC-BY-4.0 license: https://purl.stanford.edu/gj714cj0414.
Maintenance. Our dataset is hosted on the Stanford Research Data digital repository which will provide long-term support for hosting the dataset. It also provides structured metadata (schema.org standards) It has the following DOI: https://doi.org/10.25740/gj714cj0414.
Author statement. The authors bear all responsibility in case of violation of rights. All dataset images were collected by the authors and we are releasing the dataset under CC-BY-4.0.
Format. The data is uploaded in a simple zip format. Upon decompressing the archive, a directory is provided for each object category. In each directory, the data is further split into "instance_1" and "instance_2" which represent the two object instances for each category. Under these directories, JPEG images are stored for each lighting condition, with file name descriptors indicating the relative angle in degrees of rotation under which the object was taken, as well as in-the-wild images, which are labeled only with an arbitrary index 0 through 4.
C Potential societal impacts
Our paper introduces a metric that may find a range of downstream applications, including improving the temporal consistency of generative models, augmenting vehicle and human re-identification, to aiding perception for embodied agents. As with any improvements, e.g., in consistent video generation, nefarious actors may seek to use these technologies for harmful purposes. We recognize that there may be additional future applications that we cannot currently foresee.
We see opportunities for this research and our findings in Section 5.1 to provide some inspiration for additional research in neuroscience on understanding the human medial temporal lobe.
By introducing a new evaluation dataset, we are also providing a benchmark that may impact the direction of future work. For example, our dataset consists of common objects largely found around North American homes and laboratories, and images are captured in outdoor settings in a limited set of geographical locations. While we select items that we believe are commonly occurring around the world, performing future evaluation on just the CUTE dataset may create a bias towards particular types of objects or scenes. Our metric also largely inherits any biases present in the original DINOv2 model.
D Computational resources
The computational resources used were a personal workstation and computing nodes from the Stanford SC computational cluster. We used a personal workstation with an NVIDIA RTX 3090 GPU for the main experiments, and on the SC cluster, we used around 30 jobs lasting at most 2 hours each to perform the Re-ID experiments, including the sweeps on the values of  $\alpha$ . We use 1 NVIDIA Titan RTX GPU for each job. We also ran additional backbone ablations on the SC cluster with around 30 jobs lasting at most 4 hours each using one NVIDIA A40 GPU each.
E Qualitative Analysis
One characteristic of the proposed metric is its continuous nature. Different from recognition tasks such as Re-ID, it does not simply classify two objects as being the same instance but rather measures how similar they are. Since a measure of distance in the space of all objects is highly subjective and a definitive ground truth is exceedingly difficult to establish, we evaluate our metric using human preference.
Our study design is as follows: one of the authors generated 10 sets of 5 images each, consisting of photos taken by the author and photos from the internet. Then 2 other authors each ranked the image sets, considering both personal preference and the demonstration quality of the set. Their votes were averaged out and the top 5 image sets were selected. These image sets were then scored by LPIPS, CLIPScore, and foreground feature averaging (FFA) and ordered from most similar to least similar to a query object in each set. 34 participants then chose which ordering they prefer according to their personal subjective opinion. The specific prompt they were given was:
"This is a quick, anonymous survey about ordering preference. Please carefully look at each set of images. 3 Orderings are presented for each set. Select the ordering that makes the most sense to you. The images are ordered from most similar to least similar to the first image (left to right). There is no one correct answer, we seek your subjective opinion."
In Figure A2 we see that FFA was the top choice on 4 out of the 5 sets and a close second choice on the fifth. While the participants showed strong agreement on certain sets, they generally displayed pretty mixed opinions, highlighting the subjective nature of this type of classification. The given prompt was intentionally vague, allowing the participants to focus on various aspects of the images such as the pose of the objects, the background, the class, etc. Despite this, the results of this limited study suggest that our proposed metric is reasonably aligned with the intuitive human definition of similarity.
F Code and instructions for experiments
The code can be found at: https://github.com/s-tian/CUTE.
Experimental details and hyperparameters. To strike a balance between performance and speed we chose the DINOv2 ViT-B/14 distilled backbone (dinov2_vitb14) for FFA, consisting of 86M parameters. DINOv2 models are capable of accepting inputs at various resolutions, but we select a fixed input size of  $336 \times 336$  for the same reason as above, and also to provide a fair comparison to the CLIPScore metric. Our CLIPScore is based on the ViT-L/14@336px model from OpenAI.
In order to obtain the foreground mask we pass the input image through the Tracer-B7 model provided by CarveKit. We then downsample the foreground mask by a factor of 14 (the DINOv2 patch size) in order to obtain a mask of the same size as the DINOv2 feature grid. We then superimpose the two and average the unmasked patches.
G Re-ID Experimental Details
We select the hyperparameter  $\alpha$  for the weighted sum between SpCL and each considered model by sweeping over the values [0.1, 0.2, ..., 0.9] on a validation set and picking the  $\alpha$  value with the best top-1 accuracy on the validation set. The score is always computed by summing  $\alpha$  times the model
Figure A2: Human survey results. Each of the five groups of images represents an image set. For each image set, we generated three orderings, based on LPIPS, CLIPScore, and foreground feature averaging (FFA) respectively. For each image set, participants were asked to determine which of the three orderings they preferred, where the ordering represented a ranking of the similarity of the first image to the others. We see that in four out of five cases, participants preferred orderings scored as in FFA.
in question and  $1 - \alpha$  times the SpCL score. The selected values of  $\alpha$  for each model are reported below:
Model	VeRi	CityScapes
SpCL+LPIPS	0.1	0.1
SpCL+DINOv2+Crop	0.5	0.9
SpCL+DINOv2 (Global)	0.1	0.2
SpCL+FFA DINOv1 (Crop-Img)	0.7	0.9
SpCL+FFA (Crop-Img)	0.6	0.9
We also conducted comparisons to additional models and ablations which were not included in the main text due to space. We report the full results in Table 6.
Table 5: Selected  $\alpha$  values for ReID experiments combining SpCL with various metrics.
Metric	VeRi	CityFlow
top-1↑	top-5↑	top-1↑	top-5↑
SpCL	43.0	57.8	26.7	35.2
LPIPS	22.5	36.1	3.5	8.0
SSIM	6.1	14.0	4.2	9.1
DINOv2 (Global)	16.0	26.5	15.3	23.0
DINOv2 + Crop	44.6	59.5	33.5	40.1
FFA (Crop-Img)	41.0	55.7	28.6	36.1
SpCL+LPIPS	41.6	56.5	24.0	33.1
SpCL+DINOv2+Crop	46.1	60.5	32.6	39.6
SpCL+DINOv2 (Global)	42.3	56.5	24.3	33.5
SpCL+FFA DINOv1 (Crop-Img)	43.6	58.1	23.9	32.7
SpCL+FFA (Crop-Img)	44.0	58.2	27.7	35.8
Table 6: Full version of Table 3, with additional baselines and ablations.
H Relation to Other Metrics of Similarity
There are many aspects to object similarity. One can measure the visual similarity - such as the shape, color or texture of objects or functional similarity such as the purpose or affordance of an object. Often these measures are entirely orthogonal to each other, and further influenced by the context of the comparison. Because of this, many measurements of object similarity lack proper ground truth. Our aim is to define one particular dimension of similarity where we can obtain at least partial ground truth labels. We can obtain strong binary labels for this particular metric based on the identity of the objects themselves - different images of the same object should ideally have a perfect similarity.
I Evaluations on Extended Dataset
We evaluate all of our pipelines on the extended section of our dataset which contains 10,000 images - 100 image of 10 instances of objects across 10 categories. Since the result of this 10 way re-identification task is not directly comparable to the 2-way task presented in the paper, we study it here separately. We compute all of the metric as before, except this time we use just the categories containing 10 instances each. As shown in Table 7, the mAP and top1 performance of all of the pipelines is on average higher on this task for deep learning pipelines. We hypothesize this is due to the greater diversity among the 10 object instances (the initial pair of instances was deliberately chosen to be as similar as possible). Furthermore, the ARI scores tend to be much lower because the clustering is performed over ten clusters instead of two. Still, our method obtains the best score on more metrics than any other pipeline.