SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	Alignment
Model Class	Model Name	Model Type	Feature	Overall	ImageNet	Non-ImageNet
Base	PSNR	-	-	57.1	57.3	57.0
Models	SSIM	-	-	57.3	58.5	55.8
Prior-Learned	LPIPS	AlexNet-Linear	-	70.7	69.3	72.7
Metrics	DISTS	VGG16	-	86.0	87.1	84.5
Base	CLIP	ViT B/16	Embedding	82.2	82.6	81.7
ViT B/32	Embedding	83.1	83.8	82.1
ViT L/14	Embedding	81.8	83.3	79.8
DINO	ViT S/8	CLS	89.0	89.7	88.0
ViT S/16	CLS	89.6	90.2	88.8
ViT B/8	CLS	88.6	88.6	88.5
ViT B/16	CLS	90.1	90.6	89.5
MAE	ViT B/16	CLS	82.1	81.0	83.5
ViT L/16	CLS	82.7	82.4	83.0
ViT H/14	CLS	83.5	83.2	83.8
OpenCLIP	ViT B/16	Embedding	87.1	87.8	86.2
ViT B/32	Embedding	87.6	87.5	87.6
ViT L/14	Embedding	85.9	86.7	84.9
Ensemble	ViT B/16	Mixed	90.8	91.6	89.8
Tuned MLP	CLIP	ViT B/32	Embedding	87.3	88.2	86.2
DINO	ViT B/16	CLS	91.2	91.8	90.3
MAE	ViT B/16	CLS	87.6	87.3	88.1
OpenCLIP	ViT B/32	Embedding	89.9	91.0	88.5
Ensemble	ViT B/16	Mixed	93.4	94.2	92.2
Tuned LoRA	CLIP	ViT B/32	Embedding	93.9	94.0	93.6
DINO	ViT B/16	CLS	94.6	94.6	94.5
MAE	ViT B/16	CLS	90.5	90.6	90.3
OpenCLIP	ViT B/32	Embedding	95.5	96.5	94.1
Ensemble	ViT B/16	Mixed	96.2	96.6	95.5
Table 5: Alignment on NIGHT test set. We evaluate alignment on additional model settings, and separate the test set into ImageNet categories and non-ImageNet categories.
observe that MAE exhibits the best out-of-the-box performance, indicating a greater sensitivity to lower-level image distortions (e.g. color and shape) than DINO, CLIP or OpenCLIP. Surprisingly however, it is the only model whose performance decreases on BAPPS as it is further tuned. DINO, CLIP and OpenCLIP are not as sensitive to the image distortions in BAPPS, suggesting that before tuning, they are more attuned to higher-level image attributes that the dataset does not capture.
On THINGS, further training actually diminishes alignment with humans (see right plot of Figure 14). CLIP and OpenCLIP's superior performance on this dataset supports our hypothesis that they are more well-adjusted to higher-level image attributes, which THINGS aims to capture, rather than appearance-level variations.
Our evaluations across these three datasets show that, as we train perceptual metrics that align more closely with human perceptual similarity, we also improve on low-level similarity but perform slightly worse on high-level image distortions. These results suggest that humans, when making an automatic judgment, are more inclined to focus on immediate visual differences (captured in BAPPS and our dataset) rather than the image's category, context, or related words.
Figure 14: Evaluation on existing low-level and high-level similarity datasets. (Left) Despite never being trained for low-level similarity, LoRA-finetuned models on OpenCLIP, DINO, and Ensemble achieve similar human alignment to LPIPS, which was directly trained on the BAPPS dataset. (Right) The THINGS dataset measures high-level conceptual similarity, rather than appearance similarity. As such, we find that LoRA finetuning on our dataset degrades performance, as our triplets contain appearance similarity, by design.
Metric	Image Part	Ours	DINO	OpenCLIP	DISTS	LPIPS
Color (RGB)	Foreground	58.3	53.7	51.2	59.4	60.5
Color (RGB)	Background	58.7	57.2	55.5	64.1	64.7
Color (RGB)	Total	58.4	55.8	54.1	62.9	65.3
Luminance	Foreground	54.5	51.9	49.9	56.8	55.6
Luminance	Background	55.5	54.3	54.1	57.7	54.4
Luminance	Total	54.1	52.9	51.5	56.9	55.6
Depth	Total	54.2	53.6	53.3	54.7	55.8
Category Histogram	Things	58.7	58.3	55.1	55.3	53.8
Category Histogram	Stuff	59.5	58.8	57.9	62.5	61.3
Presence of Person	-	55.3	52.6	55.2	51.8	53.8
Presence of Furniture	-	53.1	52.2	54.2	53.4	53.6
Presence of Textiles	-	52.8	52.4	53.1	51.6	53.6
Table 6: Automated Metrics on COCO. Alignment of hand-crafted metrics with model decisions on the COCO dataset, which provides ground-truth semantic labels.
Metric	Image Part	Ours	DINO	OpenCLIP	DISTS	LPIPS
Color (RGB)	Foreground	71.7	70.0	69.3	68.7	60.6
Color (RGB)	Background	65.4	66.2	64.7	66.4	62.0
Color (RGB)	Total	69.8	67.9	67.6	66.0	64.3
Luminance	Foreground	63.1	62.6	62.2	61.4	56.1
Luminance	Background	59.3	59.5	58.8	60.3	56.8
Luminance	Total	59.4	60.6	59.8	59.2	56.5
Depth	Total	54.2	53.6	53.3	54.7	55.8
Table 7: Automated Metrics on NIGHTS. Alignment of hand-crafted metrics with model decisions on our dataset.
Alignment with low-level features. In Section 5.2 and Figure 7 of the main text we report results on the alignment between our metric, OpenCLIP, DINO, LPIPS, and DISTS with low-level and semantic metrics for the COCO dataset. In Table 6 we also report additional, fine-grained results for COCO triplets. We use CarveKit [1] to segment out the foreground and background of each image, and then break down how well each metric agrees with RGB color histogram similarity, luminance histogram similarity, and depth map distance for foreground, background, and the full image. For color histograms we use 32 bins for each channel, and for luminance histograms we use 10 bins.
We also examine semantic features. For each image, we find the percentage of area that each semantic category occupies, and then compute the alignment between the difference in area for each category and perceptual metrics. Note that when the difference in area is the same for both pairs in a triplet, it is counted as  $50\%$  alignment. When the difference in area is smaller for the pair chosen by the perceptual metric, it is counted as  $100\%$  alignment (and  $0\%$  in the case of disagreement). In Table 6 we show the five semantic categories most aligned with our metric. Our metric has a  $55\%$  alignment score with the "people" category, though does not align well above chance for other categories.
In Table 7 we show alignment with low-level metrics for our dataset (which does not have semantic annotations). On our dataset there is higher alignment with color, luminance, and depth across all metrics, as compared to COCO triplets. This is likely because the images in each of our dataset's triplets all share the same semantic category, making lower-level features more important than for the randomly-chosen COCO triplets. Our model aligns significantly better with foreground metrics – particularly foreground color – whereas LPIPS aligns slightly better with background.
Sketch-photo image retrieval. To further analyze perceptual metric performance on out-of-domain data, we evaluate sketch-to-photo and photo-to-sketch image retrieval on the Sketchy database [74]. First, we assign sketches as queries and photos as the search space. As seen in Figure 16 (left), DINO, DISTS, and LPIPS perform poorly on this task compared to OpenCLIP and DreamSim because they focus on color similarity, retrieving photos with predominantly white/gray backgrounds. This sometimes appears in DreamSim's results as well, hindering performance (Figure 15, top-left). Without this failure case, we return impressive results (Fig. 15, top-right) focusing on the subject appearance. We also evaluate retrieval in the other direction, using photos as queries and sketches as the search space. As shown in Figure 15 (bottom), DreamSim outperforms all other metrics in retrieving the main object while remaining sensitive to its size and location in the image.
Figure 15: Sketch-photo nearest-neighbor retrieval. Image retrieval on the Sketchy database [74], with a sketch as query (top) and photo as query (bottom). OpenCLIP and DreamSim best preserve image content, returning photos of the subject itself (e.g. church, top right) rather than focusing on the black-and-white sketch style in the query. Our metric also best localizes the subject, returning a small boat in the corner while OpenCLIP and DINO both miss this spatial detail (bottom right).
Figure 16: Sketch-photo retrieval recall scores. When using real photos as the queries, our model outperforms all methods across # nearest neighbors, obtaining a Recall@10 score of 0.32, compared to the next best model, OpenCLIP, with Recall@10 = 0.20. When using sketches as the queries, we find that our model is competitive with OpenCLIP and outperforms the remaining models.
Image Quality Assessment (IQA) benchmarks. We evaluate various metrics on the TID2013 [64] and KADID-10k [49] image quality assessment (IQA) benchmarks and report results in Table 8. Following standard IQA literature, we measure the Spearman Rank Order Correlation Coefficient (SROCC) between metric similarities and Mean Opinion Score (MOS). While LPIPS and FSIM remain best-performing overall, DreamSim is still competitive in IQA, outperforming most low-level metrics and base ViT models. Notably, our metric improves SROCC by  $\sim 0.05$  after training on NIGHTs despite not having seen these low-level distortions. These results are consistent with those in Figure 14, where tuning on NIGHTs consistently improves human alignment on BAPPS.
TID2013	KADID-10k
Low-level	PSNR*	0.687	0.676
SSIM	0.720	0.724
MS-SSIM	0.798	0.802
FSIM*	0.851	0.854
Prior learned	LPIPS	0.838	0.883
DISTS	0.794	0.867
Base models	DINO	0.722	0.753
OpenCLIP	0.764	0.841
Ensemble	0.757	0.812
Ours	DreamSim	0.814	0.868
Table 8: Evaluation on IQA datasets. Alignment of various models on TID2013 [64] and KADID-10k [49] While LPIPS and FSIM are best-performing overall, DreamSim is still competitive in IQA despite not having been tuned on low-level distortions. *While in some IQA literature PSNR/FSIM refer to computation on the luminance channel and PSNRc/FSIMc on color channels, we keep the metric names as-is and compute on color channels to be consistent with our PSNR labels elsewhere in the paper.
$k$ -Nearest Neighbors Classification. We evaluate our model as a  $k$ -Nearest Neighbor (k-NN) classifier on the ObjectNet [7] and ImageNet100 [82] datasets, both standard classification benchmarks. Results are shown in Figure 17. Similarly to our other applications, we also evaluate OpenCLIP, DINO, and LPIPS as baselines (we were unable to evaluate DISTS due to computational constraints).
For ImageNet100 (a 100-category subset of ImageNet), we evaluate on the validation set and use the training set as the candidate search space. We note that DINO was trained using self-supervision on ImageNet, and has been previously shown to achieve strong k-NN results on ImageNet. DINO performs best, however our model still achieves competitive performance. For ObjectNet, we use an 80:20 split to randomly partition the dataset into a search-space and test set. Our model outperforms both OpenCLIP and DINO on top-1 classification across different values of  $k \in \{1,3,5,11,21,101,201\}$ .
Figure 17: k-NN classification results. When evaluated as a k-NN classifier on the ObjectNet dataset, our model outperforms all methods across different values of  $k$ , with a maximum accuracy of  $30.92\%$ ; the next best method, OpenCLIP, achieves  $22.48\%$ . When evaluating on ImageNet100, our model is competitive (1-2% difference) with DINO and outperforms other baselines.
Figure 18: Ablating Feature Dimension. We apply a PCA decomposition to the output features of our model and vary the number of dimensions kept.
# PCA Components	2AFC Score
1	63.4
512	95.7
768	96.1
1792	96.2
Table 9: Feature PCA Decomposition. We list 2AFC scores as a function of the number of PCA components kept, beating both the CLIP/OpenCLIP dimensionality (512) and DINO (768).
A common failure case for all three models is when the retrieved neighbors are visually similar to the query image but of a different category. We also note that ObjectNet is designed to differ from the ImageNet distribution, indicating that DreamSim may generalize well across different image distributions.
Dimensionality reduction with PCA. Our model consists of the concatenation of the DINO, OpenCLIP, and CLIP backbones, and therefore uses a higher-dimensional feature space to compute similarity compared to each of these models independently. To investigate whether the increased dimensionality is critical for improving human alignment, we ablate feature dimensions by applying PCA, taking a certain number of the top components, as seen in Figure 18 and Table 9. We can achieve comparable performance using just 500 of the top components, similar to the 512 dimensions of the CLIP and OpenCLIP embedding outputs, suggesting that the improved alignment is not just due to the higher-dimensional feature space used to compute similarity, but rather the additional capacity and model priors obtained from assembling different models.
B.2 Additional Visualizations
Qualitative metric comparisons. In Section 5.2 and Figure 6-7 of the main text we quantitatively analyze the differences between metrics. Here, we also provide a qualitative analysis by comparing image pairs that achieve high and low similarity scores for each metric.
In Figure 19 we plot, for each image pair in our dataset, the similarity scores from our metric against similarity scores from DINO, OpenCLIP, DISTS, and LPIPS. We show the same for image pairs drawn from the COCO dataset. In Figure 20 and Figure 21 we show the pairs where our metric most agrees and most disagrees with DINO, OpenCLIP, DISTS, and LPIPS, for our test set and the COCO dataset. Note that for COCO we draw random pairs of images that share at least instance category so
Figure 19: Correlation between metric scores. For image pairs from our dataset (above), and the COCO dataset (below), we plot similarity scores from our metric against similarity scores from DINO, OpenCLIP, LPIPS, and DISTS. Our metric's scores are most correlated with other ViT-based metrics, and correlate better with DISTS than LPIPS.
Figure 20: Visualizing differences between our model and other metrics for NIGHTS. We show the examples where our metric most agrees and disagrees with other metrics. Primary differences are that our metric is more sensitive to major changes in pose, semantics, and color. It is less sensitive to granular changes in structure when overall appearance is preserved, such as the honeycomb example in the DISTS quadrant and the forest example in the DINO quadrant.
that pairs have some semantic commonality, enabling better visualization of qualitative differences between metrics.
Comparing DISTS, the top-performing prior learned similarity metric, to our model, we find that DISTS is sensitive to structural changes despite similar overall appearance (such as the width of the honeycomb or the position of similar objects), while our model rates these pairs as nearby. On the other hand, pairs that are far in our feature space but close in DISTS feature space have less appearance similarity (e.g. the houses and rooms of different colors). Comparing to deep ViT features (DINO, OpenCLIP) our model is more likely to rate pairs with similar foreground color/appearance as similar, and less likely for pairs that are similar semantically but not appearance-wise. For COCO
Figure 21: Visualizing differences between our model and other metrics for COCO. We show examples where our metric most agrees and disagrees with other metrics for pairs drawn from the COCO dataset. These pairs share fewer appearance similarities than pairs drawn from our dataset. Our metric seems particularly sensitive to foreground semantic similarities, such as the horse pair in the OpenCLIP quadrant and the snowboarders in the LPIPS quadrant.
pairs, where there are fewer appearance similarities than in our dataset, our model chooses pairs that are similar semantically first, and only then appearance-wise.
Attention Map Visualizations. As an alternate way of understanding our model's similarity decisions, we visualize the transformer attention maps following Chefer et al. [15]. Our model consists of finetuned versions of DINO, CLIP, and OpenCLIP backbones, and the resulting attention map places the largest activations on the finetuned DINO backbone (Figure 22 (left)). Accordingly, the overall attention map looks largely similar to the attention map constructed from only the finetuned DINO branch. Compared to the pretrained versions of each model backbone, the finetuned model better captures full object identity (such as over the entire body of the fish), while also minimizing spurious attention values in the background (Figure 22 (right)). Consistent with earlier analysis, this supports the fact that the foreground plays a larger role in the DreamSim similarly judgment than the background.
Figure 22: Visualizing attention maps. The finetuned DINO branch of our model has the largest contribution in the attention map [15], with the largest median activation compared to the CLIP and OpenCLIP branches (computed over 400 test set images). As such, in our model, the overall attention map is similar to the attention map from only the DINO branch. Compared to the pretrained backbones, our model better captures the entire object of interest (the fish body) while reducing spurious attention in background regions.
As the DINO model has the largest contribution in the attention maps, Figure 23 shows additional examples focusing on the difference between the finetuned DINO backbone within our model, and
Figure 23: Comparing our model and baseline attention maps from the DINO branch. Focusing on the DINO branch, which has the largest contribution in our model's attention map, we compare the attention maps of the pretrained and finetuning models. The finetuned attention map in our model better captures the foreground object or relevant regions of interest. In all examples, the DINO baseline selects the A image, while humans and our model select the B image.
the pretrained DINO backbone prior to finetuning. This visualizes how the DINO backbone changes as it is finetuned on our dataset. Our finetuned model better captures the foreground object, while attention maps from the pretrained DINO backbone may only focus on small portions of the object. In the lobster example (Figure 23 (top-right)), our model places attention on the relevant parts of the object, such as the lobster body rather than the claws in the A image as the claws do not appear in the other two images.
C Discussion
Broader Impacts and Limitations. Our model and dataset are developed from pretrained Stable Diffusion, CLIP, OpenCLIP and DINO backbones. As such, our model can inherit and propagate biases existing in these models for decisions on downstream tasks. Our dataset is generated using prompts that describe a single category, and we manually filter out categories that may generate sensitive content, such as violence or malformed human faces. As a result, the dataset has a large focus on object-centric domains, and content containing humans is considered out-of-domain. The resulting dataset and model does not capture the full range of human similarity judgements, but only the variations that we can capture in our synthetically-generated dataset.
Licenses. The icons for Figure 1 as well as the images for the inversion experiments are licensed by Adobe Stock, under the Adobe Stock Standard License, and by Pixabay, under their content license.
IRB Disclosure. We received IRB approvals for our AMT experiments from all of the institutions involved. Accordingly, we took measures to ensure participant anonymity and refrained from showing them potentially offensive content.