Are These the Same Apple? Comparing Images Based on Object Intrinsics
Authors: Klemen Kotar
Stephen Tian
Hong-Xing Yu
Daniel LK Yamins
Jiajun Wu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Object-centric
intrinsic
dataset
metric
image similarity
TL;DR: A dataset of similar objects under different extrinsic conditions used to benchmark object intrinsic — based image similarity.

Abstract: The human visual system can effortlessly recognize an object under different extrinsic factors such as lighting, object poses, and background, yet current computer vision systems often struggle with these variations. An important step to understanding and improving artificial vision systems is to measure image similarity purely based on intrinsic object properties that define object identity. This problem has been studied in the computer vision literature as re-identification, though mostly restricted to specific object categories such as people and cars. We propose to extend it to general object categories, exploring an image similarity metric based on object intrinsics. To benchmark such measurements, we collect the Common paired objects Under differenT Extrinsics (CUTE) dataset of 18, 000 images of 180 objects under different extrinsic factors such as lighting, poses, and imaging conditions. While existing methods such as LPIPS and CLIP scores do not measure object intrinsics well, we find that combining deep features learned from contrastive self-supervised learning with foreground filtering is a simple yet effective approach to approximating the similarity. We conduct an extensive survey of pre-trained features and foreground extraction methods to arrive at a strong baseline that best measures intrinsic object-centric image similarity among current methods. Finally, we demonstrate that our approach can aid in downstream applications such as acting as an analog for human subjects and improving generalizable re-identification. Please see our project website at https://s-tian.github.io/projects/cute/ for visualizations of the data and demos of our metric.

Are These the Same Apple? Comparing Images Based on Object Intrinsics
Klemen Kotar", Stephen Tian', Hong-Xing Yu, Daniel L. K. Yamins, Jiajun Wu
Stanford University
Equua contribution
{klemenk, tians, koven, yamins, jiajuns)@stanford.cm
Abstract
The human visual system can effortlessly recognize an object under different extrinsic factors such as lighting, object poses, and background, yet current com- puter vision systems often struggle with these variations. An important step understanding and improving artificial vision systems to measure image simi- larity purely based intrinsic object properties that define object identity. This problem has been studied in the computer vision literature as re-identification, though mostly restricted to specific object categories such people and cars. We propose to extend it to general object categories, exploring an image simi- larity metric based object intrinsics. To benchmark such measurements, we collect the Common paired objects Under differen'T Extrinsics (CUTE) dataset of 18, 000 images of 180 objects under different extrinsie factors such light- ing, poses, and imaging conditions. While existing methods such as LPIPS and CLIP scores do not measure object intrinsies well, we find that combining deep features learned from contrastive self-supervised learning with foreground filtering is a simple yet effective approach to approximating the similarity. We conduct extensive survey of pre-trained features and foreground extraction methods to arrive strong baseline that best measures intrinsic object-centric image similarity among current methods. Finally, we demonstrate that our approach can aid in downstream applications such as acting as an analog for human subjects and improving generalizable re-identification. Please see our project website at https: //s- tian. github. io/projects/cute// for visualizations of data and demos of our metric.
Introduction
Human vision is extraordinarily robust against extrinsic factors that affect object appearances. When driving on sunny or snowy days, in familiar streets or new citics, we can recognize and track traffic lights, other cars, and pedestrians effortlessly. Such an ability to recognize object based purely on its intrinsic properties, irrespective any extrinsic factors such as lighting and imaging conditions, object pose, and background, is key aspect of intelligent vision systems.
Existing computer vision systems often struggle with variations in lighting conditions, object poses, and backgrounds (Hendrycks and Dietterich) 2019 Drenkow et al. 2021 Lee and Kim 2023]. For example, visual perception models autonomous driving can fail during extreme weather conditions [Michaelis et al.) 2019]. and current video generation models suffer from temporal consistency of object identities, producing flickering artifacts that are easily noticeable by humans (Ho) al. 2022 To understand and improve these artificial vision systems, important step is to measure the similarity between visual contents (such as images) purely based on the intrinsic object properties that correspond to object identities.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Reference
Reference
Ours CLIPScore LPIPS
Figure I: For each reference image, green checkmarks indicate which image given metric scores as closer to the reference image. Our metric scores images containing the same object as closer than similar but ultimately different ones.
This idea has been studied in the context of re-identification (Re-ID), where the task is to judge whether a pair of images contain an object with the same identity. However, prior approaches to Re-ID are limited specific categories such humans [Zheng et al. [016]] vehicles (Liu et al.. 2016). In this work, we generalize this problem by measuring visual similarities between the intrinsics of general objects that define their identities.
While many prior visual similarity metrics have been proposed, we find empirically that they do not serve our goal well, exemplified in Figure [ For instance, image- similarity metrics like PSNR, SSIM [Wang et al. [204],, and LPIPS [Zhang et al., [2018) consider the appearance of the entire image without distinguishing between intrinsic and extrinsic factors. CLIPScore [Radford ct al. 2021 focuses on high-level semantic concepts and does not fully account for low-level intrinsic properties of object instances such as shape and texture.
Benchmarking intrinsic object similarity metries in a systematic way requires images of slightly different objects with the same semantics, combined with varying extrinsics. Existing object-centric datasets that contain similar objects do not capture the same objects with varying poses or lighting conditions. On the other hand, datasets that contain objects with these factors of variation do not include sufficiently similar objects within a single category. This motivates us to collect a new dataset, Common paired objects Under differen'T Extrinsics (CUTE), to evaluate these metrics. CUTE contains 18, 000 images of 180 objects from semantically similar groups in varying extrinsic conditions such lighting and orientation well as in-the-wild images, shown in Figure 3
Using the CUTE dataset, we analyze existing similarity metrics and simple but surprisingly effective framework for approximately measuring intrinsic object similarity. This approach combines foreground filtering and visual features of deep networks trained by self-supervised learning. We conduct an in-depth investigation into which pre-trained features and foreground extraction method provide the best measure of this similarity. From our results, we propose a strong baseline that incorporates two key ingredients that lead to compact object-centric measurement: using pre-trained representations from DINOv2 [Qquab et al. 2023 and patch-level foreground feature pooling.
Our work represents a step forward in the foundational understanding of how we can model and compute visual similarities based on intrinsic object identity. On one hand, contributes to the broader scientific objective of creating AI systems that understand the visual world as we do. We show that our measurement approach is able to better approximate the visual similarity perceived by humans than previous metrics [Bonnen et al. [2021]. On the other hand, it holds promise in helping artificial vision systems generalize across different environments and settings. We show that in visual re-identification task, our metric helps domain-specific vision models generalize better in unseen environments.
In summary, our contributions are as follows: Firstly, we collect a dataset called CUTE containing 18, 000 images of 180 grouped objects varying pose, lighting, and in-the-wild settings benchmark approaches on measuring intrinsic object similarity. We then perform an empirical study of metrics based on pre-trained visual feature extractors to determine which most effective measuring this similarity. From this, we propose a simple but strong baseline to measure the similarity called foreground feature averaging (FFA) that combines foreground filtering with features learned by DINOv2 (Qquab et al.) 2023]. a self-supervised deep model. Lastly, we find that our approach can aid in downstream applications such acting as an analog for human subjects and improving generalizable re-identification.

---

2
Related Work
Image similarity metries. Many prior methods have been proposed for evaluating the similarity between images, from classical metrics such as peak signal-to-nois ratio to perceptual metrics like the structural similarity index measure (SSIM) [Wang et al.) 2004]. Recently, metrics based on deep neural network features have been used to measure image similarity [Salimans et al. 2016 Heusel et al., 2017 Zhang et al., 2018 Sylvain et al. 2021] for training and evaluating generative models. Our metric seeks to compare images based on the intrinsics of objects present in the scene, rather than provide a suuel-bb-piidd comparison image quality assessment.
Concurrent work introduced text-conditioned metric of image similarity called GeneCIS [Vaze et al. 2023). It measures model's ability to adapt to range of similarity conditions. Our metric on the other hand focuses on a single dimension of similarity the similarity between the intrinsics of objects. While GeneCIS mines its data from pre-existing datasets and annotates it post-hoc, we collect the data for our benchmark systematically to ensure the ground truth similarity values. DreamSim [Fu et al. 2023] attempts to capture the human notion of visual similarity using a deep neural network trained on a dataset of synthetically generated image triplets with auman-annotated similarity preferences. Our metric instead focuses on the ground truth identity of objects, not human preferences.
Re-identification. A relevant problem in computer vision person/vehicle re-identification ZZhngg et al., 2016 [iu et al.. [016]] which focuses on judging if pair of person/vehicle images capture the same identity. Early work in this area typically uses cand-crafted visual features [Liao et al. 2015
Liu et al. 2012 with learned classifiers [Prosser et al.] 2010) or distance metrics (Zheng et al. 2012]
and recent work exploits deep models [Xiao et al. 2016 Sun et al. 2018 Luo et al. |019 Ming et al.
2022) learned from pair-wise person/vehicle image datasets (Zheng et al. 2015 Wei et al. 2018
Liu et al. 2017 [ou et al. 2019]. In the deep learning era, re-identification has been studied with
supervised learning setup supported by annotated datasets (Li et al.) [014 Zheng et al. 2015 Liu
cct al. 2017 [Bii et al. 2018), and later extended to unsupervised learning Yu et al. 2017 Fan et al. Yu ct al.) 2019 Ge et al. 2020 and a generalizable setup where the testing datasets are not seen during model training [Jin et al. 2020 Zhang et al., 2022]. The intrinsic object-centre image similarity problem that we study can be considered as a generalization of the re-ideniification problem, where we aim at quantitatively measuring the instance- similarity between a pair of images of general objects. Our quantitative measurement can be directly applied the re-identification problem and we evaluate our final metric along with baselines on this problem in Section 5.2
2018
Object-centric vision datasets. Prior efforts have collected visual datasets consisting of different instances of objects from common categories. CO3D (v2) [Reizenstein et al. [021]. Objectron [Ah madyan et al. [.22]. and the Object Scans dataset (Choi et al. 2016) consist of real-world videos of different object instances from typical categories. However, they are not an ideal testbed for our metric extrinsic factors such as lighting do not vary within a particular video, making it impossible to separate object identity from these cues. The Amazon Berkeley Objects Dataset [Collins et al. 2022] contains 3D models and catalog images for many items, but these images not guaranteed to be taken from the real world or even to include the object itself. On the other hand, our dataset contains not only images taken from real-world objects of various categories, but also contains stimuli obtained by varying the lighting conditions and object poses.
Self-supervised image representation learning methods. We find that approach that leverages features learned by deep models trained in a self-supervised manner on large-scale datasets is surprisingly effective on this task. While self-supervised methods continue to be developed (He et al. 2019 Chen et al.) 2020b Caron et al.) 2020 Chen et al.] 2020a Grill et al. 2020). we find
that DINOv2 |Oquab et al.) 2023] yields the best results
task. While Amir et al. [021]
have explored the use of DINOv1 [Caron et al., 202T] features as dense visual descriptors for co-segmentation and correspondences, in this work we explore how these features can be used to compute intrinsic object-centric similarities.
3 Measuring Intrinsic Object-Centric Distances
Practical measurement. We aim at metric such that for given image of an object, no other object should have a lower distance than any image of the same object. To this end, we have conducted thorough evaluation of approaches based on deep features learned from self-supervised training.
3

---

Foreground Segmenter
Foreground segmentation
Average foreground feature
Filtered features
DINOv2
Feature grid
The Foreground Feature Averaging (FFA) (Crop-Feat pipeline. Each input image is resized to a 336px square, then passed through the DINOv2 (Oquab al. (2023) model well foreground segmentation model. The foreground features are averaged and compared to another image cosine similarity. The segmentation computed at a higher resolution for this visualization.
Figure 2:
which present in Section 4.2 Informed by this, propose simple yet effective approach based on deep features learned from self-supervieed training, called foreground feature averaging (FFA), strong baseline for measuring intrinsic object similarity. Our approach computes embeddings for each image using pre-trained deep models and then computes a similarity score from those embeddings. We show an illustration of approach in Figure
Generalizable pre-trained deep feature backbone. What types of embeddings will enable our metric to exhibit invariances to extrinsics, and generalize to arbitrary objects? Our preliminary exploration showed that features obtained from text-aligned encoders trained on internet scale data such CLIP [Radford al. 2021]. which have become common backbones in many modern vision pipelines, oftentimes perform poorly at measuring the similarity between highly similar objects. We hypothesize this is because it can be difficult to describe the difference between two similar objects semantically, even when humans can differentiate between them based on visual features.
On the other hand, we find that contrastively -rained models such as DINOv2 [Oquab et al. 2023 produce features that contain large amount of instance-level information, which is usetul for measuring intrinsic object similarity so we utilize it as the backhone of our metric. DINOv2 encodes input images into patch-level features as well as a global feature. We experiment using both the global feature and the patch-level features. We do not fine-tune the DINOv2 model - our new dataset is used purely as a test set to avoid any possible overfitting brought by fine-tuning.
Foreground filtering, Another challenge in computing object-centric similarity is the influence of the image's 's background, especially when computing image-level features. For our metric, we implement background cropping procedure that makes our metric more robust for small objects and images with complex backgrounds (such as grass or benches). We analyze two practical instantiations of foreground filtering: (1) cropping the image with off- off-the-shelf segmentation models before computing features (Crop-Img) and (2) computing deep features on the entire image and then discarding the patch features associated with the background (Crop-Feat). We perform foreground cropping with an off-the-shelf foreground segmentation model, CarveKit (Selin. 2022).
Finally, we compute the cosine similarity between these embeddings for each image as a inverted proxy for the intrinsic object-centric image similarity. This enables non-parametric classification by simply considering image pairs with the highest cosine similarity as representing images of the same object. We implement our metric in Py PyTorch [Paszke et al. 2019] and release an easy-to-us! implementation of it at /ttps::/gtti tigu........ com/ s- tian/Cann CUTE
Dataset and Benchmarking
4.1 The CUTE Dataset
In this section, we describe the composition and capture of our new dataset, CUTE, that allows for systematic evaluation of intrinsic object-centric similarity metrics. The dataset can downloaded at https://purl stanford. edu/
wtw.oodddddd.d/..../.....1 gj714cj0414

---

Illumination
Object pose
In-the-wild
Figure 3:
Example images CUTE dataset. The top four rows show images captured for paired objects different illumination conditions and object poses. The last row shows random samples of object instances from across the dataset.
Category configuration. Our dataset composed of object instances from each of 50 semantic object categories. Objects belonging partieular semantic category have differences such as shape or texture that identify them as distinct objects. For example, the category "apples" contains apples with distinct shapes and textures. The dataset contains total of 180 objects. These categories range from fruits like "apples" and "oranges", to household items like "plates" and "forks". Examples of the objects the dataset are shown in Figure 3 and a full list of object categories can be found in the appendix. Of the 50 object categories, 10 categories contain ten object instances for additional intra-class variation, while all other categories contain two object instances.
Extrinsic variations. To test the ability metrics to isolate object intrinsics from visual stimuli, which are composed of both intrinsics and extrinsics, collect images of each object with varying extrinsic parameters. In particular, for each of the 180 objects, we consider three types of extrinsic configurations: different illumination, different object poses, and in-the-wild captures. For different illumination and object poses, we consider a studio setup, where we have 4 lighting conditions and 24 object poses, leading images under 4 different environments using different cameras (i.e., different imaging systems). Therefore, our CUTE dataset consists of total of 18,000 images. In the following, we provide details on our capture setup. More imaging details are in Appendix A
total of 4 24 96 images of the same object. We capture in-the-wild
Different illumination. For the studio capture configurations, place each object at the center of a turntable in a controlled environment. We capture images using a Sony a 7IV mirrorless camera equipped with FE 24-2242mmm F2.8 GM lens. We vary the illumination of the object by turning each of three LED light panels placed to the left, right, and back the stage in turn, or turning all of them off, for total of four lighting settings.
Different object poses. We vary the orientation of each object by rotating it an automatic turntable at a constant velocity and capturing an image approximately every 15 degrees of rotation.
In-the-wild capture. Finally, for each object, we collect "in-the-wild" images in both indoor and outdoor settings. We collect four images for each object, each in different location, and with varying imaging conditions: combination of mobile phones and the aforementioned mirrorless camera. This represents more realistic distribution of images.
5

---

In-the-wild
Figure 4: Example images from the apple, banana, mouse categories. dataset contains 10 object instances each for specific categories.
4.2 Evaluation
In this section, we profile the performance of popular image similarity metrics and similarities computed using several deep self-supervised learning backbones, including proposed FFA metric, using the CUTE dataset. We use a subset CUTE with two object instances per category to ensure a balanced evaluation.
We perform group- based evaluation, measured by image retrieval (quantified by top-1 accuracy and mean average precision (mAP)). The retrieval candidate with the maximum score is predicted as containing the same object. For CLIPScore and ours which are embedding- based, we also use K-means (with K 2) to cluster embeddings to measure how well the embedding space gets separated according to object identity (quantified by the adjusted Rand index [Rand] 1971] (ARI)). We use separate groups evaluate the sensitivity of metrics for each of the following scenarios:
Illumination: Each group contains 8 images, 4 containing one object in 4 different lighting conditions, and 4 containing the paired object in 4 different lighting conditions. The pair of objects have similar poses. Thus, 24 groups are formed for each object pair, one for each object pose. In each group, we iterate
every image the query, and the other
retrieval candidates.
Each group contains 48 images and is formed by 24 images of one object from each of 24 poses, and 24 of the paired object from 24 poses. Four groups are formed per object pair, corresponding each lighting condition. In each group, we iterate over every image as the query, and the other 47 as retrieval candidates.
Pose:
Each group contains 8 images, with 4 images from different scenes for each of the paired objects. So we have 1 group for each pair. each group, we iterate over every image the query, and the other as retrieval candidates.
In-the-wild:
All: Each group contains all 200 images for given object pair. We iterate over every image as the query and use all other 199 images as retrieval candidates.
Results. As Table shows,, our FFA approach compares favorably to prior image similarity methods most of the evaluation groups. particular, in the in-the-wild group which is particularly difficult, SSIM and I.IPS are no better than random guess (their top-1 accuracy is around 50%), and CLIPScore suffers from different object the same environment. For controlled groups (illumination and pose), although the top-1 accuracies are above 60% and 90% respectively for both FFA and CLIPScore, the retrieval results are far from optimal reflected by imperfect mAP. The clustering results show low ARI scores for both CLIPScore and ours, suggesting that both cannot separate the embedding space very well, leaving clear room for improvement. Nevertheless, our simple approach performs better at most evaluation tasks.
In Figure 5 we show confusion matrices for a few representative examples from each of first three groups. In these examples, we show that our metric is able to score images that contain the same object more highly than images containing subtly different objects, even when conditions such as lighting and object pose vary. In contrast, LPIPS is highly sensitive to lighting and background, as

---

Illumination
1.0
Foreground feature avg. (ours)
I.PISS
CLiPScore
1.00
0.92
0.91
0.83
1.00
0.62
0.91
0.61
1.00
0.93
0.93
0.86
0.92
0.90
0.86
1.00
0.93
0.93
1.00
0.93
0.91
0.93
0.62
0.93
0.83
0.93
1.00
0.86
0.93
1.00
Pose
Foreground feature avg. (ours)
LPIPS
CLIPScore
1.00
0.90
0.83
0.81
1.00
0.91
0.92
0.93
1.00
0.97
0.90
0.90
1.00
0.85
0.82
0.91
1.00
0.93
0.95
0.97
1.00
0.98
0.93
0.83
0.85
1.00
0.95
0.92
0.93
1.00
0.95
0.97
0.98
1.00
0.94
0.81
0.82
0.95
1.00
0.93
0.95
0.95
1.00
0.90
0.93
0.94
1.00
In-the-wild
Foreground feature avg. (ours)
LPIPS
CLIPScore
1.00
0.87
0.83
0.76
1.00
0.05
0.75
0.07
1.00
0.83
0.91
0.72
0.87
1.00
0.73
0.83
0.05
1.00
-0.01
0.81
0.83
1.00
0.78
0.84
0.83
0.73
1.00
0.85
0.75
-0.01
1.00
0.002
0.91
0.78
1.00
0.80
0.76
0.83
0.85
1.00
0.07
0.81
-0.02
1.00
0.72
0.84
0.80
1.00
0.0
Qualitative visualizations of evaluations on our dataset. Each grid depicts the pairwise similarity between sets of four images from each group. Images in green frames contain the same objects. An ideal metric would score all values in the upper left and lower right quadrants of each grid, which comparisons between the objects, most highly (closer 1). Our metric displays this behavior, while I.PISS and CLIPSSore do not. Please zoom to see visual differences between each pair of objects. The displayed images are cropped for better visibility in this visualization only.
Figure 5:

---

In-the-wild
Illumination
Pose
All
Setting
MAPt
top-If If ARII
MAPt
top-11
ARIT
MAPt
top-11
ARIT
MAP
top-11
ARIT
LPIPS
38.7
49.1
41.6
53.9
38.2
49.3
39.6
49.8
SSIM
40.1
49.5
39.4
52.1
40.3
50.6
39.9
50.0
CLIPScore
54.4
13.3
-8.7
62.0
38.8
-1.0
82.8
98.2
48.6
65.2
94,9
-0.1
CLIPScore Crop
69.3
59.5
15.4
69.3
75.0
11.5
77.7
94.7
38.7
68.1
94.5
18.6
DieamSim
56.9
11.8
-6.3
60.7
38.5
-11.0
82.7
96.5
48.2
64.3
93.2
3.6
DreamSim + Crop
66.8
47.0
6.0
69.4
72.0
.4.0
80.6
96.4
39.9
66.8
95.0
3.7
D2
70.9
47.3
19.8
86.0
82.3
54.0
85.2
97.2
56,1
78.4
95.7
40.3
Dino + Crop
79.1
41.0
84.0
85.5
39.8
83.1
96.4
76.2
96.1
36.6
Di
52.6
-8.9
58.6
37.0
81.8
97.2
63.6
93.5
DinoVI+ Crop
64.4
37.8
67.0
63.0
79.6
96.5
38.0
66.4
94.8
4.0
Unicom
63.4
37.0
4.8
64.9
46.5
6.7
81.9
97.1
48.5
66.6
94.8
9.7
Unicom Crop
73.9
66.3
25,5
72.2
72.3
11.3
78.0
93.8
39.5
67.9
93.5
15.1
FFA Crop-Img (ours)
76.7
68.3
35.5
81.7
83.0
48.2
79.1
95.2
41.5
72.2
94.9
26.8
FFA Crop-Feat (ours)
75.1
61.8
27.2
88.8
89.0
62.9
81.9
96.6
46.8
77.1
96.2
40.5
Table I: Performance of image similarity metrics on diffcrentiating object instances from the paired object
section of the datasct. Wc evaluate them by image retrieval, measured by top-1 accuracy (%) and mean average precision (mAP, %). For embedding-basee based approach including ours and CLIPScore, clustering measured by the adjusted Rand index (ARI) which is multiplied by 100 in the table with 100 indicating perfect clustering, indicating random clustering and -50 indicating worst-case clustering.
also evaluate it by
can be seen in the "illumination" and "in-the-wild" groups. CLIPScore scores the highest at matching objects to different poses (though not by a wide margin) while performing worse than metric often significantly
all other categories.
We also evaluate the DreamSim [Fu et al. 2023 metric on our dataset and find that it performs similarly CLIP. To ablate the effect of foreground cropping, we also test both CLIPScore and DreamSim in a pipeline with the CarveKit [Selin] [2022] as in our FFA pipeline. We find that adding cropping decreases the performance of CLIPScore
DreamSim the pose category and increases the performance in the other categories. This suggests that cropping the background tends to decrease performance in the pose category perhaps due the high similarity of the images and imperfect background removal.
Applications
5.1 Human Subject Analog
We also evaluate foreground feature averaging (FFA) on dataset of stimuli used in prior studies of human perception mechanisms [Bonnen et 2022 In the study, human subjects are presented with panel of four images similar objects and asked to pick the odd one out. The objects are divided along two axes as high and low similarity (to other objects in the panel) and as familiar or unfamiliar (familiar objects are similar to those seen in everyday life, while unfamiliar objects are more abstract). Figure T contains a sample of these images. This task was designed to study the effects of the Medial Temporal Lobe (MTL) in image processing, so results were collected for both humans with normal MTL functions and humans with lesioned MTLs.
For neural networks, encode each image and measure which embedding lies farthest from the others and select as the odd image out.
Findings. We report the results in Table 2 We find that model outperforms the previously studied best baseline (ResNet-50) [Bonnen et al. 2021], and approaches the performance human subjects with lesioned MTL, who rely only the ventral visual stream (VVS). This forces them to operate closer to a single forward pass in a neural network. Human subjects with a limited 200ms exposure the stimulus exhibit similar behavior, in line with the findings that the low similarity object discrimination task requires only the VVS. We also evaluate the performance of directly using the DINOv2 class token, and find that it outperforms forward feature pooling in two of the subsets.
These findings suggest that our model approaches human-level performance on tasks that require only the VVS, but falls far short tasks that require the MTL. We also demonstrate that large gap

---

VeRi
CityFlow
Similarity
Familiar
Unfamiliar
Metric
High
Low
High
Low
top-1f
top-5t
top-11 If
top-51
SpCL
43.0
57.8
26.7
35.2
ResNet-30
0.28
0.90
0.18
0.72
DINOv2 Crop
44.6
59.5
33.5
FFA Crop-Feat (ours)
0.40
0.93
0.27
0.80
FFA (Crop-Img)
41.0
55.7
28.6
36.1
DINOv2
0.42
0.91
0.32
0.74
Human (lesioned MTL)
0.50
0.95
0.20
0.95
SpCL+LPIPS
41.6
56.5
24.0
33.1
SpCL+DINOv2<Crop
46.1
60.5
32.6
39.6
Human
0.90
0.95
0.80
0.98
SpCI,+FFA (Crop-Img)
44.0
58.2
27.7
35.8
Table 2: Classification accuracy of deep neural net- works (ResNet-500 used the hest baseline from Bon- nen (02TT) and human subjects on oddity visual discrimination tasks.
Table 3: Generalizable vehicle Re-ID results using dif- ferent distance metrics including strong Re-ID model SpCL [G et al. 2020
Similarity
Familiarity
Similarity
Familiarity
Similarity
Familiarity
Similarity Familiarity
Examples of stimuli presented to humans and our metric. Tasks vary across two dimensions: similarity and familiarity. In cach task, the human model is presented with four objects and asked to determine which object is different from the others.
Figure 6:
in difficulty exists between the high and low similarity discrimination tasks, even with cutting-edge neural networks trained hundreds of millions of images, just as they do with humans. Additionally, our approach outperforms humans with lesioned MTT. on the Unfamiliar High similarity test case, suggesting that the model is less sensitive familiarity of objects than human subjects. Overall, approach is step towards improved artificial analogs of human subjects, encouraging further studies eventually create AI systems that understand the visual world as humans do.
5.2 Vehicle Re-Identiffiation
In this section, we use our similarity measurement based on FFA to augment the performance of existing models on generalizable re-identification (Re-ID) task. Specifically, we show how FFA can be used to achieve stronger transfer performance on vehicle Re-ID) task. We investigate transfer scenarios between two datasets: VeRi [Liu al.) 2016) and SityFlow-Rell [Naphade et al.) [021]. This transfer setup is referred as test labels are not provided, we construct a validation set from 100 randomly selected vehicle IDs and randomly sample 1000 query images from that subset.
generalizable Re-ID JJin et al. 2020]. For the CityFlow dataset,
While the two datasets both contain images of vehicles tracked across multiple cameras, many highly-performan (SpCL) [Ge et al.) 2020) do not generalize very well to different datasets due to domain gaps such as different lighting conditions, camera view angles, and background clutter (Song et al., 2020). To improve the generalization of the model, we augment SpCL. with our FFA model measurement via simple late fusion. Specifically, we compute a w-weighted average of the distances computed by the SpCL model and cosine similarity distances computed by our metric. We set the a value to 0.6 for VeRi and 0.9 for CityFlow after tuning over [0.1, 0.2, ..., 0.9) on validation set. As shown in Table 3 and qualitatively Figure 7 our distance metric aids SpCL generalization on the VeRi dataset and directly outperforms it on CityFlow.
Results. We report the results in Table 3
models like self-paced contrastive learning
6 Conclusion
In this work, we study the problem of measuring object image similarity based solely on the intrinsic properties of the object. We find that a simple approach based on deep features from self-supervised learning effectively capture the object intrinsics without being distracted much by extrinsic

---

SpCL (transfer)
FFA
Query
Ranking
Figure Using our foreground feature averaging (FFA) model directly yields improved vehicle re-identification performance compared to SpCL model transferred to this task on the challenging CityFlow dataset. In this example, FFA retrieves the correct vehicle in the top four candidates while SpCL only re- re-identifies it correctly at the third best prediction.
variations such as lighting, object pose, and backgrounds, to approximately measure the similarity. Evaluation CUTE dataset validates this finding. yet it also shows that there is room for future improvements. We demonstrate that our approach, with the goal to measure such intrinsic object- centric image similarity, better approximate human visual similarity understanding behavior, showing the promise to contribute to creating artificial agents that perceive the visual worlds as do. We hope findings encourage further explorations in exploiting such similarity for other applications, such as consistent video generation.
Limitations. Our approach is focused and evaluated on images containing single objects. The CUTE dataset does not contain images depicting occluded objects or with object poses varied relative to the turntable. Their capture conditions are limited in terms of geographic and outdoor lighting conditions, and while we attempt to vary the capture devices themselves, only four total devices are used.
Acknowledgments and Disclosure of Funding
We thank Kyle Sargent for helpful discussions. This work is in part supported by Air Force Office of Scientific Research (AFOSR) YIP FA9550-23-1-0127, ONR MURI N00014-22-1-2740,, NSF CCRI #21 #0025 and Amazon, Bosch, Ford, and Google. ST is supported by NSF GRFP Grant No. DGE-1656518.
References
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep VIT features dense visual descriptors. arXiv:2112.05814, 2021.
Yan Bai, Yihang Lou, Feng Gao, Shiqi Wang. Yuwei Wu, and Ling. Yu Duan. Group-sensitive triplet embedding for vehicle reidentification. IEEE Transactions on Multimedia, 2018.
Tyler Bonnen, Daniel L.K. Yamins, and Anthony D. Wagner. When the ventral visual stream is not enough: A deep learning account of medial temporal lobe involvement in perception. Neuron, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, 2020a.
10

---

Xinlei Chen, Haoqi Fan. Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv:2003.0429) 040900 2020b.
Sungjoon Choi, Qian- Yi Zhou, Stephen Miller, and Vladlen Koltun. A large dataset object scans. arXiv:1602.02481, 2016.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. ABO: Dataset and benchmarks for real-world 3D object understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Nathan Drenkow, Numair Sani, Ilya Shpitser, and Mathias Unberath. A systematic review of robustness in deep learning for computer vision: Mind the gap? arXiv:21 2021.
Hehe Fan, Liang Zheng, Chenggang Yan, and Yi Yang. Unsupervised person re-identification: Clustering and fine-tuning. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang. Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions of human visual similarity using synthetic data. 1:Xiv:2306.09344 2023.
Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, et al. Self-paced contrastive learning with hybrid memory for domain adaptive object re-id. Advances in Neural Information Processing Systems, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv:191..0572. 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv: 12261, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. AXXv:2204.03455, 2022.
Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang. Style normalization and restitution for generalizable person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Youngseok Lee and Jongweon Kim. Robustness of deep learning models for vision tasks. Applied Sciences, 2023.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. DeepRelD: Deep filter pairing neural network for person re-identification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2014.
Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z Li. Person re-identification by local maximal occurrence representation and metric learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2015.
Chunxiao Liu, Shaogang Gong, Chen Change Loy, and Xinggang Lin. Person re-identification: What features are important? In Proceedings of the European Conference on Computer Vision, 2012.
Xinchen Liu, Wu Liu, Huadong Ma, and Huiyuan Fu. Large-scale vehicle re-identification in urban surveillance videos. In IEEE International Conference Multimedia and Expo, 2016.

---

Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma. PROVID: Progressive and multimodal vehicle reidentification for large-scale urban surveillance. IEEE Transactions on Multimedia, 2017.
Yihang Lou. Yan Bai, Jun Liu, Shiqi Wang, and Lingyu Duan. PeRi-Wild: A large dataset and a new method for vehicle re-identification the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
Hao Luo, Youzhi Gu, Xingyu Liao. Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re- re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. /901./99000075 2019.
Zhangqiang Ming, Min Zhu, Xiangkun Wang, Jiamin Zhu, Junlong Cheng, Chengrui Gao, Yong Yang. and Xiaoyong Wei. Deep learning-based person re- re-identification methods: A survey and outlook of recent works. Image and Vision Computing, 2022.
Milind Naphade, Shuo Wang, David C. Anastasiu, Zheng Tang, Ming-Ching Chang, Xiaodong Yang. Yue Yao, Liang Zheng, Pranamesh Chakraborty, Christian E. Lopez, Anuj Sharma, Qi Feng. Vitaly Ablavsky, and Stan Sclaroff. The 5th AI City Challenge. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv:2304.07793, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Bryan James Prosser, Wei-Shi Zheng, Shaogang Gong, Tao Xiang, Q Mary, et al. Person re- identification by support vector ranking. In British Machine Vision Conference, 2010.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 1971.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In Proceedings of the International Conference on Computer Vision, 2021.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in Neural Information Processing Systems, 2016.
Nikita Selin. http:///w.....tt...c...eeeeee//r 2022.
I. Liangehen Song, Cheng Wang, Lefei Zhang, Bo Du, Qian Zhang, Chang Huang, and Xinggang Wang. Unsupervised domain adaptive re-identification: Theory and practice. Puttern Recognition, 2020.
Yifan Sun, Liang Zheng. Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the European Conference Computer Vision (ECCV), 2018.
12

---

Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object- centric image generation from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Sagar Vaze, Nicolas Carion, and Ishan Misra. GeneCIS: A benchmark for general conditional image similarity. arXiv:2306.07969, 2023.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer GAN bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 79-88, 2018
Tong Xiao, Hongsheng Li, Wanli Ouyang, and Xiaogang Wang. Learning deep feature representations with domain guided dropout for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Hong-Xing Yu, Ancong Wu, and Wei-Shi Zheng. Cross-view asymmetric metric learning for unsupervised person re-identification. In Proceedings of the IEEE international conference computer vision, 2017.
Hong-Xing Xing Yu, Wei-Shi Zheng, Ancong Wu, Xiaowei Guo, Shaogang Gong, and Jian-Huang Lai. Unsupervised person re-identification by soft multilabel learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Putlern Recognition, 2019.
Pengyi Zhang, Huanzhang Dou, Yunlong Yu, and Xi Li. Adaptive cross- learning for generalizable person re-identification. In Proceedings of the European Conference on Computer Vision, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identification: Past, present and future. 161iv:1610.000
2016.
Wei-Shi Zheng, Shaogang Gong, and Tao Xiang. Reidentification by relative distance comparison. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2012.
13

---

A Dataset Details
Our dataset consists of total 50 categories with 10 instances each, for a total of 180 objects.
The specific categories are listed below:
1. Apple
18. Bell pepper (orange)
35. Bowl
Strawberry
19. Bell pepper (red)
36. Notebook
Orange
20. Can of tomatoes
37. Book
4. Pear
21. Eggplant
38. Diet Coke
Apricot
22. Green water bottle
39. Red soda
Banana
23. Marker
40. Rose
7. Mango
24. Pencil (blue)
41. Cookie
8. Broccoli
25. Pen
42. Octopus
Carrot
26. Mug
43. Cardboard box
10. Potato
27. Ceramic Mug
44. Mouse
11. Yellow onion
28. Fork
45. Keyboard
12. Shallot
29. Spoon
13. Tomato
30. Butter Knife
46. Cable
14. Grapes
31. Spatula (wood)
47. Screwdriver
15. Lettuce
32. Cups
48. Pliers
16. Avocado
33. Ceramic pot
49. Hammer
17. Bell pepper (yellow)
34. Plate
50. Screw
Additionally, ten categories contain ten object instances. These categories are enumerated below:
1. Apple
5. Fork
9. Screw
Banana
6. Spoon
10. Screwdriver
3. Pen
7. Book
Ceramic Mug
8. Mouse
Light setting
Shutter speed
F-number
ISO
Focal length
Left, back, right
1/640s
f/2.8
2000
39mm
Low light
1/20s
f/2.8
2000
39mm
Table Camera settings for our dataset.
We capture images using a Sony a 7IV mirrorless camera equipped with a FE 24-70mm F2.8 GM lens. The camera settings used in different illumination and different object poses data capture configurations are enumerated Table
We show our capture setup Figure
For the different object poses data capture configuration, we use the turntable to perform full 360 degree rotation constant velocity with period of approximately 24 seconds. We then configure the camera to capture 24 images shooting at an interval of second.
We capture images in the controlled setting at resolution of 3168 3168.
For "in-the-wild" settings, we capture images using cell phones (iPhone 13, iPhone 14 Pro, iPhone 12 Mini) varying resolutions.
B Data
Dataset description. We provide dataset description in dataset sheet: https: //gg://///ttb//////// com/ http:///ww./..olooaaaaaaaaaaaaaa
14

---

Format. The data uploaded in a simple zip format. Upon decompressing the archive, a directory provided for each object category. In each directory, the data is further split into "instance 1" and "instance_ 2" which represent the two object instances for each category. Under these directories, JPEG images are stored for each lighting condition, with file name descriptors indicating the relative angle in degrees rotation under which the object was taken, as well as in-the- wild images, which labeled only with arbitrary index 0 through 4. Back light
Right light
Left light
Camera
Turntable
Figure A1: Capture setting for our controlled different illumination and different object poses configurations.
Link and license. The dataset is uploaded for public download under the CC-BY-4.0 license: http:://purr..ssafffooo edw.eduu/uo. htt.mforrd.dd///71/044...1 gj714cj0414
Maintenance, Our dataset is hosted on the Stanford Research Data digital repository which will provide long-term support for hosting the dataset. It also provides structured metadata (schema. org standards) has the following DOI: httpt:://////// /ttp::////o/..o//.///////// org/10 257 gj 714440444
Author statement. The authors bear all responsibility in case of violation of rights. All dataset images were collected by the authors and we are releasing the dataset under CC-BY-4.0.
C Potential societal impacts
Our paper introduces a metric that may find a range downstream applications, including improving the temporal consistency of generative models, augmenting vehicle and human re-identification. to aiding perception for embodied agents. As with any improvements, e.g., in consistent video generation, nefarious actors may seek use these technologies for harmful purposes. We recognize that there may be additional future applications that we cannot currently foresee.
We opportunities for this research and findings in Section 5.1 to provide some inspiration for additional research in neuroscience on understanding the human medial temporal lobe.
By introducing a new evaluation dataset, we are also providing a benchmark that may impact the direction of future work. For example, our dataset consists of common objects largely found around North American homes and laboratories, and images are captured in outdoor settings in limited set of geographical locations. While we select items that we believe are commonly occurring around world, performing future evaluation just the CUTE dataset may create bias towards particular types of objects or scenes. Our metric also largely inherits any biases present in the original DINOv2 model.
15

---

D Computational resources
The computational resources used were a personal workstation and computing nodes from the Stanford SC computational cluster. We used a personal workstation with an NVIDIA RTX 3090 GPU for the main experiments, and on the SC cluster, we used around 30 jobs lasting at most 2 hours each to perform the Re-ID experiments, including the sweeps on the values of a. We use NVIDIA TITAN RTX GPU for each job. We also ran additional backbone ablations on the SC cluster with around 30 jobs lasting at most 4 hours each using one NVIDIA A40 GPU each.
E
Qualitative Analysis
One characteristic of the proposed metric is its continuous nature. Different from recognition tasks such as Re-ID, it does not simply classify two objects being the same instance but rather measures how similar they are. Since a measure of distance in the space of all objects is highly subjective and a definitive ground truth is exceedingly difficult to establish, we evaluate our metric using human preference.
Our study design is follows: one of the authors generated 10 sets of images each, consisting of photos taken by the author and photos from the internet. Then 2 other authors each ranked the image sets, considering both personal preference and the demonstration quality of the set. Their votes were averaged out and the top 5 image sets were selected. These image sets were then scored by LPIPS, CLIPScore, and foreground feature averaging (FFA) and ordered from most similar least similar to query object in each set. 34 participants then chose which ordering they prefer according to their personal subjective opinion. The specific prompt they were given was:
"This is a quick, anonymous survey about ordering preference. Please carefully look at each set of images. 3 Orderings are presented for each set. Select the ordering that makes the most sense you. The images are ordered from most similar to least similar to the first image (left to right). There is no one correct answer, we seek your subjective opinion."
In Figure A2 we see that FFA was the top choice on 4 out of the 5 sets and close second choice on the fifth. While the participants showed strong agreement on certain sets, they generally displayed pretty mixed opinions. highlighting the subjective nature of this type of classification. The given prompt was intentionally vague, allowing the participants focus on various aspects of the images such as the pose of the objects, the background, the class, etc. Despite this, the results of this limited study suggest that our proposed metric reasonably aligned with the intuitive human definition of similarity.
F
Code and instructions for experiments
The code can be found at: https://github. cow.so CUTE
Experimental details and hyperparameters. To strike balance between performance and speed we chose the DINOv2 ViT-B/14 distilled backbone (dinov2_vity (ittb_ 14) for FFA, consisting of 86M parameters. DINOv2 models are capable of accepting inputs various resolutions, but select fixed input size 336x 336 for the same reason as above, and also to provide a fair comparison to the CLIPScore metric. Our CLIPScore is based on the ViT-L/14 336px model from OpenAI.
In order to obtain the foreground mask we pass the input image through the Tracer-B7 model provided by CarveKit. We then downsample the foreground mask by a factor of 14 (the DINOv2 patch size) in order to obtain a mask of the same size as the DINOv2 feature grid. We then superimpose the two and average the unmasked patches.
G Re-ID Experimental Details
We select the hyperparameter a for the weighted sum between SpCL and each considered model by sweeping over the values [0.1, 0.,...,, ........... 0.9) on validation set and picking the a value with the best
1 accuracy on the validation set. The score is always computed by summing a times the model
16

---

CLIPScore 11.8%
CLIPScore 26.5%
LPIPS 5.9%
FFA 55.9%
FFA 82.4%
LPIPS 17.6%
LPIPS 0.0%
CLIPScore 0.0%
67.8%
FFA 52.9%
CLIPScore
CLiPScare 2.9%
50.0%
LPIPS 4711
Figure A2: Human survey results. Each of the five groups of images represents an image set. For each image set, we generated three orderings, based on LPIPS, CLIPScore, and foreground feature averaging (FFA) respectively. For each image set, participants were asked to determine which of the three orderings they preferred, where the ordering represented ranking of the similarity of the first image to the others. We see that in four out of five cases, participants preferred orderings scored as in FFA.
17

---

in question and 1 a times the SpCL score. The selected values of for each model are reported below:
Model
VeRi
CityScapes
SpCL+LPIPS SpCL+DDNOO2++rrpp
0.1
0.1
0.5
0.9
SpCL+DINOv2 (Global)
0.1
0.2
SpCL+FFA DINOv1 (mg)
0.7
0.9
SpCL+FFA (Crop-Img)
0.6
0.9
Table 5: Selected a values for RelD experiments combining SpCL. with various metrics.
We also conducted comparisons to additional models and ablations which were not included in the main text due to space, We report the full results in Table
VeRi
CityFlow
Metric
top-11 1
top-51
top-11
top-51
SpCL
43.0
57.8
26.7
35.2
LPIPS
22.5
36.1
3.5
8.0
SSIM
6.1
14.0
4.2
9.1
DINOv2 (Global)
16.0
26.5
15.3
23.0
DINOv2 + Crop
44.6
59.5
33.5
40.1
FFA (Crop-Img)
41.0
55.7
28.6
36.1
SpCL+LPIPS
41.6
56.5
24.0
33.1
SpCL+DDNNVV+Crrop
46.1
60.5
32.6
39.6
SpCL+DINOv2 (Global)
42.3
56.5
24.3
33.5
SpCL+FFA DINOv1 Cro)-Img)
43.6
58.1
23.9
32.7
SpCL+FFA (Crop-Img)
44.0
58.2
27.7
35.8
Table 6: Full version of Table 3, with additional baselines and ablations.
H
Relation to Other Metrics of Similarity
There are many aspects to object similarity. One can measure the visual similarity such as the shape, color texture of objects functional similarity such the purpose affordance of object. Often these measures are entirely orthogonal to each other, and further influenced by the context of the comparison. Because of this, many measurements of object similarity lack proper ground truth. Our aim is to define one particular dimension of similarity where we can obtain at least partial ground truth labels. We can obtain strong binary labels for this particular metric based the identity of the objects themselves - different images of the same object should ideally have a perfect similarity.
I Evaluations on Extended Dataset
evaluate all of our pipelines on the extended section of our dataset which contains 10, 000 images 100 image of 10 instances of objects across 10 categories. Since the result of this 10 way re-identification task is not directly comparable to the 2-way task presented in the paper, we study here separately. We compute all of the metric as before, except this time we use just the categories containing 10 instances each. As shown in Table 7 the mAP and topl performance of all of the pipelines is average higher on this task for deep learning pipelines. We hypothesize this is due to the greater diversity among the 10 object instances (the initial pair of instances was deliberately chosen to be similar possible). Furthermore, the ARI scores tend to be much lower because the clustering is performed over ten clusters instead of two. Still, our method obtains the best score on more metrics than any other pipeline.
18

---

In-the-wild
Illumination
Pose
All
Setting
mAPf
top-f
ARTI
MAPt
top-1t
ARIT
MAPt
top-1f
ARII
mAP
top-11
ARTT
LPIPS
9.0
0.0
11.5
0.0
47.1
57.5
39.9
52.0
SSIM
14.8
0.1
11.7
0.0
57.6
72.0
40.3
60.2
CLIPScore
84.0
82.5
-0.3
85.0
85.9
-0.9
87.1
97.4
8.6
84.9
96.9
3.5
CLIPScore Crop
86.0
86,6
2.8
86.5
89.9
2.8
86.2
94.3
5.43
84.9
94.3
3.3
DreamSim
84.2
81.7
-0.9
84.2
85.3
-2.1
87.6
96.0
6.4
84.3
95.5
-0.1
DreamSim + Crop
86.4
86.2
86.7
90.7
86.6
95.3
84.6
95.4
2.1
D2
84.5
88.4
91.8
87.5
97.5
85.8
97.1
6.5
Dino Crup
93.0
87.6
96.4
86.4
DI
83.1
80.4
-1.0
84.2
86.6
87.6
84.3
2.9
DinoV Crop
86.1
85.0
5.0
86.3
90.5
86.7
95.9
84.8
95.7
4.0
Unicom
85.0
84.4
2.3
84,9
86.0
-0.6
87.8
96.3
7.5
66.6
94.8
9.7
Unicom Crop
86.5
88.0
4.2
86.6
89.7
3,9
86.1
93.4
5.9
84.7
93.6
2.5
FFA Crop-Img (ours)
87.1
88.5
3.3
88.1
92.1
5,4
86,8
95.5
7.6
85.6
95.6
6.6
FFA Crop-Feat (ours)
86.6
86.7
3.0
89.0
93.1
5.6
87.9
96.9
7.6
86.7
96.9
7.4
Table 7: Performance image similarity metrics on differentiating object instances from our extended 10 instance object dataset. We evaluate them by image retrieval, measured by top-1 accuracy (%) and mean average precision (mAP, %). For embedding-basee approach including ours and CLIPScore, we also evaluate it by clustering measured by the adjusted Rand index (ART) which is multiplied by 100 in the table with 100 indicating perfect clustering, indicating random clustering and -50 indicating worst-case clustering.
19


Reviews:
[{"review_id": "j9FmMISmnI", "reviewer": "Reviewer_Ewhk", "summary": "", "questions": "", "limitations": "I would like to see a more nuanced discussion of the fundamental limitations of the dataset or the similarity metric. Not considering multi-object images is really not a limitation because similarity between images containing multiple objects could theoretically be reduced to similarity between image regions containing single objects. Expansion to animate objects would be nice but this is not a limitation, this is future work. An example of an actual dataset limitation could be a lack of images in the dataset depicting occluded objects.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Novel Dataset: The authors present a useful and novel dataset structure that is ideally positioned to study object instance similarity. Controlled variation in illumination and pose help indicate exactly which extrinsic image attributes different similarity metrics struggle with. The addition of in-the-wild images helps test the generalizability of similarity metrics to real-world problems. I think a dataset of this structure would be relevant and helpful to the community for studying object similarity.\n2. Human Interpretability: The authors present a nice evaluation of how PD performs relative to humans in terms of outlier detection and image similarity ranking. Increasingly relevant to AI is the design of algorithms that understand human preferences for a variety of tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HI6eE3h17o", "reviewer": "Reviewer_ieDf", "summary": "", "questions": "", "limitations": "The authors have addressed the limitations of their work well.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The Platonic Distance pipeline is straightforward to implement. Additionally, it is intuitive and should indeed provide a good measure of similarity between objects as it is known that DINO or other self-supervised methods obtain robust feature representations. This should enable a good measurement of object similarity in terms of intrinsics, while excluding factors such as lighting, pose or background.\n\nThe paper describes the approach in detail and conduct extensive experiments to evaluate its effectiveness. The experiment with human subjects is particularly interesting (humans can classify objects without a high bias towards extrinsic factors, and therefore its ) and might provide further research directions (\"large gap between proposed approach and humans regarding regarding high vs. low similarity discrimination tasks\"). The other experiment about the influence of the extrinsics (table 1) provides interesting insights as well. Additionally, the authors provide code, data, and instructions needed to reproduce the main experimental results to enhance the reproducibility.\n\nThe paper shows that the proposed distance metric and the CUTE benchmark are relevant for other research areas/downstream tasks: a comparison between machine-based und human recognition; showing that the metric can be used to augment existing highly-performant models to achieve stronger transfer performance on a vehicle Re-ID task.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "oMTm093ga7", "reviewer": "Reviewer_xype", "summary": "", "questions": "", "limitations": "The proposed dataset focuses on cases where there is a single object, relatively close to the camera, and the proposed approach depends on being able to reliably segment that object from the foreground. All of these are conditions that mean the proposed dataset is very different from real-world, \"objects in context\" conditions where object re-id or similarity metrics are relevant.\n\nThe \"extrinsic\" variations that vary for the turntable images are illumination (the brightness of light is varied, but not the color or quality of the light) and position on the turntable (the object is placed on a turntable and the turntable is moved). While the dataset does include \"In-the-wild\" images in addition to the images where the object is placed on a blank turntable, the example images in Figure 3 show that these images are still close up views of single objects on fairly non-differentiated backgrounds. This does not truly capture the complexity of real world extrinsics or the impact that complex real-world context (compared to the simple \"in-the-wild\" contexts included in the dataset) has on being able to assess object similarity. To be fair, the authors are aware of these limitations (as mentioned in the brief limitations section at the end). However, they are very significant because they limit how much work using this dataset would generalize to real world cases where object re-id and similarity measures might be needed.\n\nThere is no discussion of potential negative societal impact, but that does not seem unreasonable given the context of this work.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "While there are a number of datasets that support recognizing the same class of object in different contexts, there are very few, especially outside of the face re-id domain, that support recognizing instances of objects or objects with very high semantic similarity in different real world contexts. The proposed dataset is a meaningful step in filling this gap.\n\nWhile it's not a significant algorithmic contribution, the proposed approach (segmenting the object and then pooling features from just the foreground region from a DINOv2 backbone) is simple, intuitive, and provides a reasonable baseline.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "H2IT5VutrI", "reviewer": "Reviewer_iSG4", "summary": "", "questions": "", "limitations": "The limitations mentioned are specific and reasonable.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1: The CUTE dataset provides unique data that will enable future work on lighting and pose invariant computer vision. The paired objects will enable future work on object re-identification.\n\nS2: The proposed Platonic Distance model is an original combination of existing techniques that can easily be applied to existing models. The loose requirements on the architecture of the feature extractor and foreground segmenter models makes the technique widely applicable.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Three reviewers support the acceptance of this paper. One reviewer Ewhk remains marginally below the acceptance threshold at 5 after improving the score from the original score. The authors have provided extensive responses and discussions to address the issues raised. The remaining two issues Ewhk highlighted are the small size of the dataset especially from in-the-wild, and weak justification on the novelty of the proposed similarity metric. Overall, all four reviewers acknowledged that the authors' extensive responses and clarifications have improved the paper notably. I also share the view that the similarity metric per se is not central to the contribution of this paper and appreciate that the authors have moderated their claims on this novelty. I also share the view that the dataset is indeed small as it stands, and in particularly on in-the-wild setting, in order for it to be more widely useful in practice. Having said that, the authors are aware of this weakness and are making a proactive effort to continuously increase the scale of the dataset. On balance, I recommend accept.
Decision: Accept (Poster)