SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(MAE, CLIP, DINO). The best-performing configurations of existing base models are DINO B/16, MAE B/16, CLIP B/32, and OpenCLIP B/32 (additional settings are in SM Section B.1). However, existing metrics have significant misalignments with humans; for example, DINO has approximately  $10\%$  disagreement. To improve performance, we finetune with LoRA (as described in Section 4.2), denoted as Tuned - LoRA, achieving significant alignment improvements over pre-trained baselines (Figure 4) and training an MLP head (Tuned - MLP). Ensembling models by concatenating OpenCLIP, CLIP, and DINO features achieves the highest score across baselines  $(90.8\%)$ , MLP-tuned models  $(93.4\%)$ , and LoRA-tuned models  $(96.2\%)$ . We refer to the best LoRA-tuned ensemble model as DreamSim, and use it for subsequent analysis and applications.
Do different ways of measuring human perception agree? We corroborate our results on our 2AFC test set by evaluating the models against JND perceptual judgments. The high correlation between 2AFC and JND scores indicates that our 2AFC judgments capture a general sense of similarity that can replicated across different settings (Figure 5).
Does improved mid-level perception lead to improved low-level and high-level perception? We use the same models as in Figure 4 to examine how performance on our dataset corresponds to performance on two others: BAPPS, containing image triplets with low-level augmentations, and THINGS, containing triplets with categorical variations. Training on our dataset improves human alignment on BAPPS but decreases alignment on THINGS, suggesting that humans, when making an automatic judgment, tend to focus on appearance-based visual differences (captured in BAPPS and our dataset) rather than high-level object category. We additionally evaluate on the TID2013 [64] and KADID-10k [49] image quality assessment (IQA) benchmarks as alternative low-level datasets. Similar to BAPPS, we find that our metric outperforms base models and is competitive with low-level metrics despite not training for low-level similarity. For full results, see SM Section B.1.
5.2 What image attributes affect similarity decisions?
Sensitivity to image ablations. We study which image properties affect our model's decisions by ablating properties of the image triplets, such as orientation, color, shading, structure, and foreground or background components. To do this, we modify the triplets to change or discard each attribute and compute the agreement between the model decisions on the modified and unmodified triplets. This operation measures the model's robustness when a given attribute is missing. Note that this analysis aims to provide insight into how our model determines similarity between two images, rather than to compare to human judgments; it is possible that humans may be more sensitive to different image attributes.
As shown in Figure 6, our model is least affected by orientation (tested by flipping the reference image), but ablating color or shading (the L or AB channels in LAB color space), structure [28], or both color and shading together using contours [12] has larger impacts. We conclude that color, shading, and structure contain critical information for our model, but our model can tolerate
Figure 7: Alignment with semantic and low-level metrics. We select random triplets from the COCO dataset and assess how well low-level and semantic metrics predict model judgments. Compared to LPIPS & DISTS, our model is more sensitive to objects that appear in individual instances ("things") and the presence of people, and less sensitive to "stuff" categories, such as sky and road. Compared to OpenCLIP & DINO, color explains more of our model's decisions. We mark our model's results with a dashed pink line for comparison.
differences in orientation. To ablate foreground and background components, we segment the image [1] and replace the corresponding pixels with random uniform noise. This operation removes the color, shading, and texture of the region, even though the same outline remains in both. Removing the background has a smaller impact on model alignment compared to removing the foreground, suggesting that the foreground color and texture is more important for similarity recognition in our model compared to the background.
Alignment with semantic and low-level metrics. Given that some image attributes, like semantics, are difficult to ablate, we complement our ablation analysis by examining how well semantic and handcrafted low-level features align with model judgments. We use 10,000 random image triplets from MS-COCO [50] annotated with individual object instances ("things") and background categories such as grass and sky ("stuff"). We compare our metric to LPIPS, DISTS, DINO, and OpenCLIP (Figure 7). Our model aligns less strongly with RGB-color histogram similarity (calculated using histogram intersection [80]) compared to LPIPS and DISTS, however is more sensitive to color compared to other large vision models, while none align with depth map similarity [27, 69]. Next, we compute alignment with the things-category and stuff-category histograms, which summarize the area occupied by each semantic category. Our model aligns better with things-histogram similarity, whereas other models are comparatively more sensitive to stuff. This result suggests that our model is relatively more sensitive to foreground object instances. Finally, we compute how often metrics align with the per-category intersection of categorical histograms. Our model aligns best with the presence of people (additional results in SM Section B.1).
Image retrieval. We use our metric for image retrieval on two datasets: ImageNet-R [37], which contains 30K images of various renditions of ImageNet categories (e.g., sketches, cartoons), and a random subset of 30K images from MS-COCO [50], which contains diverse natural photos. Given a query image, we compute its similarity to the entire dataset and present the top-3 nearest images under each metric in Figure 8. As can be seen, our metric finds meaningful neighbors, demonstrating that it can generalize to diverse image domains. We compare to the best performing ViT embedding methods (OpenCLIP, DINO) and prior learned metrics (LPIPS, DISTS), as evaluated in Section 5.1. Compared to the baselines, which are either dominated by low-level similarity (e.g., background color in LPIPS/DISTS), or higher-level similarity (e.g., semantic class in OpenCLIP), our retrieved images exhibit similarity across a continuum of visual traits. See Section B.1 for more results and analysis on sketch-to-photo and photo-to-sketch domains.
We confirm our retrieval quality with a user study. For each metric, we retrieve the top-10 neighbors for a set of query images. For ImageNet-R, users prefer our metric's retrievals  $36.8\%$  of the time, followed by OpenCLIP  $(28.7\%)$  and DINO  $(19.5\%)$ . We observe similar trends in a study with COCO images (see Figure 9).
Feature inversion. To better understand the information captured by our metric, we apply existing feature inversion techniques, where the aim is to optimize for an image that is similar to a target image under some metric. We compare our metric to the DINO and OpenCLIP embeddings, as well as to an ensemble embedding that simply concatenates all features without finetuning.
Figure 8: Nearest-neighbor retrieval across different metrics. Nearest neighbor retrieval on ImageNet-R (top) and COCO (bottom), comparing different metrics. Although the datasets include images outside of our training domain, our model consistently retrieves neighbors with similar appearance and class to that of the query image.
Figure 9: User preferences for image retrieval results by metric. We conduct a user study that collects preferences for retrieval results from LPIPS, DISTS, DINO, OpenCLIP, and DreamSim. We visualize one standard deviation above and below each bar. On ImageNet-R (top, orange), our metric is preferred by users  $36.8\%$  of the time, followed by OpenCLIP  $(28.7\%)$ . Similarly, on COCO (bottom, blue), users prefer our metric  $33.6\%$  of the time, with DINO being the second choice  $(28.1\%)$ .
Figure 10 shows the inversion results under a range of image priors. Under vanilla optimization, i.e., not using any image prior [52, 61], our metric's results exhibit higher fidelity to the original colors, shapes, and semantics. Using Deep Image Prior [83, 84], our metric reveals better global layout preservation and semantics. This is also evident when using our metric to guide recent generative models [22], resulting in higher quality image samples that resemble the semantic concepts and appearance of the target image.
Consistent with Section 5.2 and the retrieval results, these visualizations demonstrate our metric's sensitivity to color, layout, and semantics. The increased quality of our fine-tuned embeddings compared to just ensembling shows the effectiveness of tuning on our human-aligned dataset. See SM for more examples, and Section A.4 for details on feature inversion methodology.
Figure 10: Feature inversion across different metrics and image priors. Given a target image, we optimize for an image, where the objective is to match the target image embedding of a given backbone. Without any image prior (Optimization), our metric better recovers the color, shape and semantics of the target image. With a weak image prior [83, 84] (DIP Inversion), our metric is able to reproduce scene structure and semantics. Using a diffusion model [22] as a strong prior, our metric better captures overall semantics and scene appearance.
k-NN Classification. We evaluate our metric as a  $k$ -Nearest Neighbors classifier, which requires the retrieval of images that are both visually and semantically relevant to a given query. We find that our metric outperforms all baselines on the ObjectNet [7] dataset and performs competitively with DINO on the ImageNet100 [81] dataset. For full details and results, refer to SM Section B.1.
We expand the fundamental task of measuring perceptual image similarity to encompass factors beyond mere low-level similarity, spanning multiple notions of similarities (e.g., object poses, colors, shapes, and camera angles). By harnessing a state-of-the-art generative model, we overcome the lack of suitable data and introduce a new dataset consisting of human-judged synthetic image triplets. We evaluate candidate embeddings and propose a new similarity metric that better aligns with human judgments, demonstrating its generalization and effectiveness compared to existing metrics. We note that by using Stable Diffusion (SD) [71], our benchmark is exposed to potential preexisting biases and sensitive content in the model. Our perceptual model is also finetuned from existing pre-trained backbones, and thus may also inherit prior errors and biases. However, our work is a step in aligning representations to human preferences, and we believe our work can inspire further research in leveraging advances in generative image models for perceptual studies.
Acknowledgments. We thank Jim DiCarlo, Liad Mudrik, Nitzan Censor, Michelle Li, and Narek Tumanyan for fruitful discussions throughout the project. This work was supported by a Packard Fellowship to P.I., Israeli Science Foundation (grant 2303/20) to T.D., Meta PhD Fellowship to L.C., and NSF GRFP Fellowship to S.S.
object sketching. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022. 3
[88] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004. 2
[89] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural Language Processing. pages 38-45. Association for Computational Linguistics, Oct. 2020. 19
[90] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872, 2021. 3
[91] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010. 4
[92] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378-2386, 2011. 2
[93] Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pages 7324-7334. PMLR, 2019. 3
[94] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018. 2, 3, 4, 5, 6
In the supplemental materials, Sec. A contains additional methodological details on dataset collection and model training, Sec. B provides more analyses of our model, and Sec. C discusses the broader impact of our work, limitations, and licensing. Additional qualitative results, such as additional triplets from our dataset and visualizations of image retrieval and image reconstruction, are available on our project page.
A Method
A.1 AMT Details
User interface. During the 2AFC study, users are instructed with the following prompt:
You will see three images: one labeled "Reference",
one labeled "Image A", and one labeled "Image B".
Select whether Image A or B is more similar to the Reference.
For each task, users see an image triplet with the reference in the center and distortions on either side (randomized). Each user completes 2 practice tasks, 50 real tasks, and 10 sentinels (randomly placed), averaging 3 minutes for the entire assignment. We discard responses from users who do not respond with  $100\%$  sentinel accuracy. See Figure 11 (left) for an example of the user interface.
Instructions for JND are as follows:
You will see four images one after the other.
Determine whether the first and third images are identical, then whether
the second and fourth images are identical.
Each correct answer earns 1 point.
Users are shown four images for 500 ms each with a 1 second gap in between, and are then prompted to answer the two questions in Figure 11 (right). They are given feedback and a score, though these have no bearing on the JND results themselves. Each user completes 2 practice tasks, 24 tasks with "different" pairs, and 12 tasks with "same" pairs, averaging 10 minutes for the entire assignment.
The object retrieval study instructs users with the following text:
You will see a reference image and five other images under it.
Pick the image that is most similar to the reference.
See Figure 12 for an example of an object retrieval task. Each user completes 2 practice tasks, 40 real tasks, and 5 sentinels (randomly placed), averaging 3 minutes for the entire assignment. We discard responses from users who do not respond with  $100\%$  sentinel accuracy.
Figure 11: User interface for AMT studies. (Left) One image triplet shown to a user in 2AFC, who is prompted to pick "Image A" or "Image B". (Right) In each JND task, users are shown a sequence of images and asked whether the image pairs were identical. Upon answering, they are given the correct answers.
Figure 12: User interface for image retrieval study. To evaluate object retrieval performance, users are asked to pick the image (A-E) most similar to the reference.
Round	# unanimous	# additional sentinel failures	# kept
1	100,000	0	100,000
2	74,346	6,750	81,096
3	63,423	2,411	65,834
4	51,097	592	51,689
5	43,615	289	43,904
6	37,696	113	37,809
7	30,420	16	30,436
8	25,079	0	25,079
9	22,098	0	22,098
10	20,019	0	20,019
Table 2: Filtering for cognitively impenetrable triplets. We start with 100K triplets, and advance triplets to the subsequent round if the human vote remains unanimous, or if the added vote came from a user who did not pass the sentinels and thus the vote is inconclusive (the vote from this user is discarded).
Cost breakdown. Across 10 rounds of 2AFC studies, we show users 477,964 triplet instances (see Table 2 for the full breakdown). Each user is paid  $0.50 for one assignment consisting of 50 triplets, averaging$ 10.00/hr. In total, we pay users $4779.64 to collect 2AFC judgments on our dataset.
We run JND on image triplets in our test set that received  $\geq 6$  2AFC judgments. Each user is paid  $2.00 for one assignment consisting of 48 image pairs, averaging$ 12.00/hr. In total, we pay users $156.00 to collect JND judgments on our test set.
Our object retrieval user study is conducted over 200 query images with 10 nearest neighbors. Users are paid  $0.50 for one assignment consisting of 40 tasks, averaging$ 10.00/hr. In total, we pay users $40.50 to collect object retrieval judgments.
A.2 Additional Dataset Details
Filtering text prompts. As described in Sec. 3.1 we sample images using image labels drawn from popular image classification datasets. We perform basic filtering to avoid human categories that typically result in malformed generated images, such as "man" and "baby". Thus, human faces only feature in a small percentage of our dataset ( $<0.5\%$ ), typically in categories such as "lab coat" and "bonnet" that are still associated with humans. We note that even with some human categories
excluded, our model is still sensitive to the presence of people, as shown in Fig. 7 and Fig. 8. For a full list of categories used, refer to our GitHub page.
2AFC task. Following Sec. 3.2 in the main text, we filter the images over 10 rounds to obtain cognitively impenetrable triplets where humans tend to vote the same way despite various differences between the images. Statistics for each round of filtering is reported in Table 2, which leaves us with roughly  $20\%$  of the original triplets containing unanimous votes. We discard all votes from any worker who fails the sentinel task. As a result, not all triplets have the same number of votes. The resulting sizes of the train, validation, and test splits and additional statistics on each split are reported in Table 3 (top). In practice, we further discard triplets with five or fewer unanimous votes.
JND task. The JND triplets are meant to capture the decision boundary where two different images are similar enough to be confused as identical, in the presence of a masking image. Each triplet is divided into two pairs: Ref vs. A and Ref vs. B. These pairs are presented to different workers in different interleaving sequences. We collect three judgments for each pair and take the majority vote among the three judgments, thus collecting six judgments in total per triplet (Table 3 (bottom)).
Dataset	Split	# Samples	Avg # Votes	Consensus Type
2AFC	Train	15941	7.11	Unanimous
Validation	1958	7.07	Unanimous
Test	2120	7.04	Unanimous
JND	Test	411	6	Majority
Table 3: Dataset Statistics. For each dataset, we report the number of samples and the average number of votes for each triplet after filtering for sentinel failures. Labels for the 2AFC dataset are based on a unanimous vote, while for JND we take the majority vote over three trials per pair (six trials per triplet).
A.3 Model Training.
For all fine-tuned models (both Tuned - MLP and Tuned - LoRA) we use the NIGHTS training dataset, with an  $80\% - 10\% - 10\%$  split as described in Table 3, and images of size  $768 \times 768$  resized to  $224 \times 224$ . We train on a single NVIDIA GeForce RTX 3090 or NVIDIA Titan RTX GPU with an Adam optimizer, learning rate of  $3e - 4$ , weight decay of 0, and batch size of 512 (non-ensemble models) and 16 (ensemble models). In MLP-tuned training, we use a width of 512. We tune the number of training epochs using the validation set; for the Tuned - LoRA ensemble model (DreamSim) we train for 6 epochs. For Tuned - LoRA models we use rank  $r = 16$ , scaling  $\alpha = 0.5$ , and dropout  $p = 0.3$ . Training time is approximately  $30\mathrm{min}$ /epoch for LoRA-tuned models, and 15 min/epoch for MLP-tuned models.
A.4 Feature Inversion
In Fig. 10 we show examples of using our metric to optimize for an image generation that is similar to some target image. We do so in three experiments, with increasingly strong generative priors.
Optimization. In this setup, we initialize an image randomly and update it iteratively using gradient descent with respect to the distance between it and the target image. Given a random crop of our working image and the target image we compute the distance between the two using a specific metric (DINO/OpenCLIP/Ensemble/DreamSim) and their corresponding embeddings. Note that in this setup there is no generator neural network.
Deep Image Prior. This technique involves beginning with a frozen random noise which serves as an input to a trainable U-Net. The U-Net is optimized iteratively by comparing the output of the U-Net with the target image using different metrics and their embeddings. This introduces a CNN prior on the generated image as described in the Deep Image Prior paper [84].
Guided Diffusion. This technique involves using classifier guidance to guide the generation of a 256x256 unconditional ImageNet-trained diffusion model [23] using an input prompt image. The guidance is done by computing the gradient of some loss function, given the estimated clean image
Figure 13: Alignment on ImageNet and non-ImageNet triplets. We split the test set into triplets generated from ImageNet categories and Non-ImageNet categories, as some model backbones are trained only on ImageNet images. For all models, alignment is highly correlated between the two splits.
and the target image, with respect to the current iteration. At every training step we compute the spherical distance loss between our model's embeddings of 64 random crops of the input image and the synthesized image. We combine this distance with a total variation loss, following [18, 58]. The gradient of this loss is then added to the mean of the current iteration, which pushes the generated image to be closer to the target image.
B.1 Additional Evaluations
Full Evaluation on Large Vision Models. In Section 5.1 and Figure 4 in the main text we report results using the best-performing setting of various large vision model backbones. Table 4 shows the experimental variation over multiple runs, in which the LoRA variation consistently outperforms the MLP variation. Table 5 evaluates additional model settings, spanning different ViT model sizes, patch sizes, and strides.
Ensemble tuning	Independent training seeds	Avg.	Stdev.
Trial 1	Trial 2	Trial 3	Trial 4	Trial 5
MLP	93.4	93.1	93.1	93.9	92.9	93.3	0.326
LoRA	96.2	96.3	95.1	96.0	95.8	95.9	0.416
Table 4: Experimental variation for ensemble models. We train the Tuned - MLP and Tuned - LoRA ensemble models on 5 seeds each and record their test accuracies at the same epoch as was recorded in the main paper.
As some models are adapted from backbones trained on ImageNet [21] (including the prior learned metrics and DINO), we split our dataset into categories contained in ImageNet and those not in ImageNet, and evaluate alignment on each split. Performance on both splits is highly correlated (Figure 13), suggesting that the notions of visual similarity are related regardless of whether or not the triplet was generated from an ImageNet category, and whether or not the model was trained only on ImageNet.
We note that the MAE ViT B/16 checkpoint was taken from Hugging Face transformers repository [89], while the others were taken from the official Facebook repository [31].
Alignment on Alternative Datasets. As depicted in the left plot of Figure 14, training on our dataset (with either tuning method) indeed improves BAPPS metric-human alignment in nearly every model, suggesting that some of these patch-based distortions are implicitly still captured in our dataset. We