DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Authors: Stephanie Fu
Netanel Yakir Tamir
Shobhita Sundaram
Lucy Chai
Richard Zhang
Tali Dekel
Phillip Isola
Venue: NeurIPS 2023 spotlight (2023)
Keywords: perceptual similarity
foundation model
perception
computer vision
image metric
TL;DR: We fit a model of human perception on a synthetic dataset, and show that it aligns better with image attributes that humans are sensitive to.

Abstract: Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks. Our project page: https://dreamsim-nights.github.io/

DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Stephanie Fu
Netanel Y. Tamir
Shobhita Sundaram
Lucy Chai
Richard Zhang
Tali Dekel
Phillip Isola'
MIT
2 Weizmann Institute of Science
3 Adobe Research
Reference
8
Reference
Reference
Reference
Reference
Reference
Reference
Reference
LPIPS
DINO
DreamSim
Humans
What makes two images look similar? We generate benchmark of synthetic image triplets that span wide range mid-level variations and gather human judgments, rating whether image A/B more similar to the reference. Our benchmark spans various notions of similarity such as pose (top-left), perspective (top-mid), foreground color (mid-left), number of items (mid-right), and object shape (bottom-left) This allows to leam metric (DreamSim) that better coincides with human judgments metrics (LPIPS) embedding-basee metries extracted from recent large vision models (DINO & CLIPP).
Figure 1:
existing similarity
Abstract
Current perceptual similarity metrics operate at the level of pixels and patches. These metries compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop perceptual metric that assesses images holistically. Our first step is to collect judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short explaining our new data, and we introduce a new metric. DreamSim, tuned to better align with human perception. We analyze how metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks. Our project page: https: // /ttp//dremmmaa/ee wiw.iniiiiii.iiiie gi githhu.iii ii/
dataset of human similarity
Equal contribution, corresponding authors. Order decided by random seed.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
"A sense of sameness is the very keel and backbone of our thinking" William James, 1890
Our understanding of the visual world hinges crucially on our ability to perceive the similarities between different images. Moreover, humans can reason about many notions of similarity, ranging from low-level perceptual properties such as color and texture, to higher-level concepts such as an object's category or a scene's ' emotional valence [76]. This capacity to conduct meaningful visual comparisons underlies our ability to effortlessly transfer our knowledge to new environments, e.g..
recognizing unseen or unfamiliar objects based on their relatedness to familiar ones [42, 70].
Computer vision has tried to capture this sense of similarity with low-level metrics like PSNR and SSIM [88], as well as learned perceptual metrics such as LPIPS [94] and DISTS [24]] Despite their utility, these metrics are limited in that they focus on the pixel or patch level and fail to capture higher- level structures. These limitations have motivated researchers to reach for image-level embeddings from large vision models such as DINO or CLIP to measure image-to-image distances in large variety of applications [12, 40, 41, 44, 62]. Recent studies have shown that these embeddings do well at capturing certain high-level similarity judgments, in particular, predicting which semantic categories will be considered alike by humans [59]. It remains unclear, however, how well these models align with human perception of richer and more fine-grained visual structure.
In this paper, we introduce a new perceptual metric, which bridges the gap between lower-level patch-based metrics and broad categorical comparisons. We collect a new dataset named NIGHTS Novel Image Generations with Juman--estedd Similarity containing human similarity judgments over image triplets. Each triplet consists of reference image and two perturbed versions, along with human judgments as to which version is most similar to the reference (Fig. 1). We use iterative filtering together with recent diffusion models to collect our dataset, which is designed to capture image sets that are cognitively impenetrable (i.e. result in consistent decisions across different individuals) yet showcase rich variations. For example, our data contains images of similar object appearance, viewing angles, camera poses, overall layout, etc. This dataset differs qualitatively from prior low-level datasets [94], which focused on perturbations like blurring and adding noise, and from previous high-level datasets [34, 35], which showed variation just at the level of categories (e.g. "is an image of kitchen more like image of giraffe or image of beach").
On the mid-level similarity task presented by our data, we find that features from recent large pre-trained vision models [10, 39, 66] outperform the current set of standard perceptual metrics [24, 94]. We further show that these large vision models can be tuned on our data to substantially more human-aligned. Our resulting metric, DreamSim, can be dropped into existing pipelines and demonstrates high agreement with human visual perception in both quantitative assessments and qualitative comparisons using out-of-domain real images (c.g., image retrieval, image synthesis). We also analyze which features our metric is most sensitive to and find that, compared to previous perceptual metrics, it focuses relatively heavily foreground objects, while compared to modern image embeddings, it does not neglect color and layout.
In summary, our contributions are the following:
A new image similarity dataset, consisting of 20k synthetic image triplets designed to be cognitively impenetrable with human judgments as labels.
A tuned metric that captures how humans naturally perceive image similarity, achieving 96.16% 16% accuracy in predicting human judgments on our dataset.
Analysis of the image properties that affect our model's decisions.
Demonstration of downstream applications to image retrieval and synthesis.
2 Related work
Perceptual similarity. Classical metrics such as Manhattan &,, Euclidean bg, MSE, and PSNR use point-wise difference to measure similarity, thus failing to capture important joint image statistics. Patch-based metrics, including SSIM [88], FSIM [92], and HDR-VDP-2 [55] tackle this issue and are widely used in applications involving photometric distortions such as image quality assessment. However, they do not capture the nuances of human vision when more structural ambiguity present [73] and are not suited for more complex image generation tasks.

---

With the deep learning revolution, classical metrics have been replaced by learning- Gaarning-based metrics [26, 29, 43]. These metrics are defined in the space of deep features extracted from pre-trained networks, such as VGG [78] or AlexNet [47]. Amir and Weiss [4] demonstrate that even untrained networks can be adapted as perceptual metrics. Zhang et al. [94] observe that feature-based metries outperform classical metrics across different convolutional architectures and learning paradigms, suggesting that perceptual similarity is an emergent property in deep representations. Further tuning on the perceptual data yields improvements, such as in LPIPS [94], PIE-APP [65], or DPAM in the audio domain [53, 54]. Further improvements include ensembling for robustness [45], antialiasing for stability [24, 36, 93], and global descriptors for texture [24]. Muttenthaler et [l.[59] provide insight into high-leve! human similarity by training on a subset of the THINGS [35] dataset, focusing on concept similarity and omitting visual cues for images within category.
While strong computer vision features make for strong perceptual metrics, counterintuitively, they eventually become decorrelated with perceptual similarity [20, 48). Today, the predominantly-used perceptual metric is L.IPS, operating on 64x64 patches.
Recent foundation models as metries. Foundation models provide strong pre-trained backbones for a variety of downstream tasks. These models primarily leverage the Vision Transformer (ViT) [25] architecture and are trained through self-supervised learning [16, 32, 81]. CLIP [66] learns to map images and text captions into a shared embedding space, proving useful for many (often zero-shot) tasks [41, 67, 90]. CLIP has been employed as a perceptual metric to train models for semantic consistency [12, 87J. Another self-supervised ViT-based model, DINO [10], extracts disentangled appearance and structure descriptors that can be employed in image generation pipelines. Amir et al.[5] show that DINO encodes valuable semantic information about object parts. Our work aims to systematically evaluate such representations for perceptual similarity, also including OpenCLIP (an open-source implementation of CLIP) [39] and pre-trained masked autoencoders (MAE) [31].
Perceptual tests. The two alternative forced choice (2AFC) test has historically been used by behavioral psychologists to study decision- making [51, 56]. Humans judge the similarity between two images by choosing to consider certain dimensions of similarity more than others [85]. Gathering judgments on ambiguous sets of images can be cognitively penctrable, calling upon a subject's cognitive processes rather than more automatic, "wired-in" sense that is stable across humans and over time [11, 19, 79]. Previous studies have raised concerns about cognitive penetrability [57, 94]. On the other hand, as a psychophysical measure, just noticeable difference experiments (JND) are thought to be independent of subjective biases [3]. We follow best practices [94] and collect judgments on both of these complementary perceptual tests.
Synthetic data. GANs [30] have been used widely for dataset generation on tasks such as visual
alignment [63], face manipulation [86], and adversarial training for image synthesis [77]. In recent years, text-driven generative models (e.g., Stable Diffusion [71], Imagen [72], DALLE-2 [68], MUSE [13]) have emerged as powerful tools for image synthesis. They have also been used to generate training data for a variety of downstream tasks [6. 9, 33, 75].
3 Perceptual dataset collection
While previous datasets focus on low-level, patch-based distortions [2, 94] or high-level, categorical [35] changes, we aim to close the gap, capturing distortions including mid-level variations. We aim to produce images with an underlying semantic commonality, but variations in a diversity of factors, such as style, color, pose, and other details, so that a human can assess their visual relationship. In Section 3.1, we describe our data generation pipeline we prompt Stable Diffusion for related images of a given category, leveraging its natural image prior for variations within the category. We then describe our mechanism for collecting cognitively impenetrable perceptual judgments in Section 3.2.
3.1 Generating images with varied distortions
We leverage Stable Diffusion v1.4 [71], which generates diverse and high-quality images that adhere to given text prompt. We sample images with a prompt of the same category, using the structure "An image of a <category>"' The <category> is drawn from image labels in popular datascts: ImageNet [21], CIFAR-10 [46], CIFAR-100 [46], Oxford 102 Flower [60], Food-101 [8],
3

---

Same? no
Sane?
yes
Ref
Which Inage is sore sinilar?
10
time 500 ms
1114
2AFC Data Collection
IND Data Collection
Figure 2: Overview of 2AFC and JND data collection. (Left) We collect up 10 similarity judgments for each triplet our dataset, filter for unanimous examples. (Right) We validate our 2AFC results with a JND study. We flash two interleaved image pairs, and ask if the first & third, and second & fourth were the same. Interleaving the pairs ensures that users decide solely based on their initial reaction to each image from standardized (500ms) viewing time.
Dataset
Perturbation Type
Data Source
Tnpet type
Images / Patches
# Samples
per sample
Judge
Judgment type
Examples
low-level (traditional CNN-based)
MIT-AdobeSk.
64x64 patches
DAPPS [94]
19.2k
9.6k
3.0
JND
high-level conceptual
Google, Bing. Ebay, Flickr
THINGS [34]
Images
1.8k
4.7M
Old--oneoout
NIGHTS (Ours)
low mid-lavel syntheric
Diffusion- synthesized
20.0k
7.1
2AFC
Images
JND
Table 1: Perceptual dataset comparison. While previous work targets low-level and high-level variations, we leverage synthetic data to generate diverse perturbations. In addition, while previous data gathers fewer judgments per sample, we filter for unanimous judgments, curating cleaner samples.
and SUN397 [91]. While the ImageNet labels make up 55% of the categories, the superset of categories across these datasets encompass wide range of objects and scenes.
As illustrated in Figure the resulting images display mid-level variations, such as pose, perspective, and shape, falling in-between pixel/patch-bat based distortions and categorical differences. Additional triplets and the full list of categories are in the Supplementary Materials (SM). We generate an initial set of 100,000 image triplets, which are later filtered during the labeling process. Each triplet consists of a reference image and two "distortions", generated from the same prompt.
3.2 Human perceptual judgments
We strive to gather perceptual judgments that are automatic (requiring little to cognition). stable (invariant to changes in mental representation), and shared across humans, which requires a cog- nitively impenetrable judging process. Our main dataset consists of two alternative forced choice (2AFC) judgments on a triplet. Additionally, we collect just noticeable difference (JND) judgments on image pairs, corroborating our findings. Table 1 shows overview of our dataset, compared to prior perceptual datasets.
Two alternative forced choice (2AFC). Figure 2 (left) illustrates data gathering setup. We collect judgments on the commonly-used Amazon Mechanical Turk (AMT) platform [12, 83, 94]. We show participants triplets of images (Ir T., T1) and ask whether To or ã ("distortions" of 2) is more similar to the reference image x. We collect judgments y E (0. 1} denoting the more perceptually similar distortion. This choice does not necessarily need be easily articulated but should be cognitively impenetrable, that is, instinctive and consistent across humans.
Additionally, we use sentinels to ensure high-quality responses. Sentinels are designed as (r, xr v), where one "distortion" the image itself and the other is from completely different prompt. Approximately 20% of participants failed at least one sentinel, and we discard all their responses.

---

To collect cognitively impenetrable triplets, we design our 2AFC experiments to ensure that each triplet included the dataset obtains a unanimous vote for either žg or 21. We divide our experiments into 10 rounds, starting with 100,000 triplets, advancing instances that maintain unanimous votes. Triplets retained through all 10 rounds can earn a maximum of 10 votes, but may have less as a result of the aforementioned sentinels. To maintain dataset quality, we only include triplets with 26 unanimous judgments in the final dataset.
Ultimately, our 2AFC dataset D = {(x, To, T1), y) consists of 20,019 triplets with an average of 7 unanimous votes each. We partition our resulting dataset into train, validation, and test components with random 80/10/10 split. Our dataset is publicly available on our project page.
Just noticeable differences (JND). JND aims characterize the boundary when a distortion becomes just noticeable. Below a threshold, a small perturbation (e.g., a I-pixel shift) appears identical to human observer. This perceptual test provides complementary signal for perceptual similarity and allows us to exploit uniform timing and the presence of a correct answer.
Intuitively, an image pair that falls below the JND threshold should be more likely to be selected as similar in the 2AFC test; thus a high correlation between 2AFC and JND scores indicates that our judgments are reliable under multiple experimental settings. To assess how well our perceptual tests agree, we collect judgments on the triplets from the 2AFC test set, filtering for triplets that "straddle" the JND threshold. Specifically, given triplet (x, To. .1) D2afe independently collect JND judgments on pairs (z, To) and (ær, T keeping triplets where humans find one and only one to be identical. In total, our JND dataset contains 411 triplets Dind {xr, Zo. Ti), 8), where users chose one of E {0, 1} to be identical.
We show our JND procedure in Figure 2. We interleave two images and their respective distortions into sequence x + v + ã - Ü, and ask whether the images in each pair were identical. This exploits masking, perceptual phenomenon that occurs when an image is obscured by another. By masking with a time gap, we standardize viewing time across users and prompt for a decision based on an initial reaction and memory of the image. For each user, we show 48 distorted pairs, along with 24 identical pairs, to balance the expected responses. We collect judgments per pair, with no person seeing a repeat, and take the majority vote the label. Users mark 20.4% of distorted images being identical, indicating that some of our distortions indeed fall below the noticeable threshold.
4 Perceptual metric learning
LoRA
A
Ensembie
Cosine distance
de
We next evaluate how well pretrained embed- dings and learned metrics align with our data, and we investigate if alignment can be improved by fine-tuning.
Ref
Ainge Loss
Cosine
d
distance
8
4.1 Embeddings distance metric
Figure 3: Training overview. Our perceptual metric an ensemble of backbones. concatenating the representa- tions together and fine-tuning with LoRA. We evaluate how well cosine distance on candidate feature spaces aligns with perceptual judgments y.
We denote distance between two images as D(-,-: fo), where fo is a feature extractor. We
evaluate a variety of -fate-of-the-arr candidate embeddings and metrics. LPIPS [94] and DISTS
[24] use CNN backbones, whereas DINO [10],
CLIP [66], OpenCLII [17], and MAE [31] use transformer-based backbones [25]. Following standard practice, distance (xx, ã; fo) = cos So (a), fa)) is taken as the cosine distance between the CLS tokens taken from the last layer for DINO and MAE (before and after the layer normalization, respectively), and the embedding vector for CLIP and OpenCLIP, For LPIPS and DISTS, D(x, ã; fe) is simply the distance metric itself. Given triplet (z, To, T1), and feature extractor fo. the model vote is calculated as:
11 d1 do where do Dar, No; So) and d1 D(x, (1;,
(1)
$,0 0, do dj
5

---

Metric Human Alignment
Alignment Humans (2AFC)
100
Ours
80
JND-2AFC Correlation
mastir
Lake
Low-leeel
90
mocrl
learned metrics
Ours
80
JND Score
MAG
DINO
70
cLin
Opennuu
70
0,536
60
LPIPS
60
50
PSHR
SSIM
LPIPS
DISTS
MAE
CLIP
OpenCLIP
DINO
Ensemble
60
70
2AFC Score
Metric
Figure 4: Metrics performance on our benchmark. Large vi- sion models OpenCLIP and DINO outperform prior learned metries LPIPS and DISTS (orange) (chance 50%). Further tuning perceptual data with MLP (blue) improves performance over out- of-the-box features (green); we find Low-rank LoRA [38] tuning boosts performance significantly (pink). Ensembling CLIP, Open- CLIP, and DINO models together improves performance as well. Our final model, DreamSim, combines these insights and achieves high agreement with humans (96.16%) 16%). Error bars represent 95% confidence interval.
Figure 5: Correlation between 2AFC & JND. We plot the agreement between human judgments and existing similarity metries two tasks, 2AFC and JND. The strong correlation between the met- rics' agreement with both tasks suggests that our dataset captures a general notion of similarity that can be replicated in two separate, independent human studies.
Evaluating human- metric agreement. Recall that for triplet, we collect indicating which image human selects as more similar. We evaluate how often each metric agrees with human judges as Score2xrc(fo Epaa g=y= and Score Ep [[lg=]]-
4.2 Learning an improved metric
Objective. To better align perceptual metric with human judgments, given triplet (z, To Ø1) we maximize the difference between the perceptual distances do, d1, with smaller distance associated with the distortion y humans voted be most similar. To accomplish this, map y E {0, 1} + z {-1, 1} and use a hinge loss, equivalent to triplet loss [14] between the embeddings, with a margin of 0.05:
Cly. 0) (0 m Ad 7), where Ad = do d1.
Tuning. Next, we investigate the best method tune or modify feature extractor fo. A challenge is the large number of parameters (billions, for VIT backbones), compared to the relatively small amount of perceptual data. Inspired by [94], we first try tuning via MLP head, adding a 1-hidden- layer MLP with a residual connection on top of the pre-trained embeddings. We compare this to fine-tuning through the pre-trained backbones using Low-Rank Adaptation (LoRA) [38], which we find to achieve better results than full fine-tuning. Using LoRA (r 16, dropout p 0.3, a 0.5) we tune approximately 0.67% of each model's parameters. all experiments, LoRA significantly outperformed using the MLP head.
Feature concatenation. As multiple models may provide complementary features, we try combining them to boost performance. We concatenate features from the best- rest-perrorming models on our dataset (DINO [10], CLIP [66], and OpenCIIP [17]) into one ensemble metric. By training just a handful of parameters in this ensemble (through LoRA [38]), we gain the benefits of each large model's s embedding space without sacrificing much computational load while fine-tuning.
Experiments
5.1 How well do existing metrics align with human judgments?
In Figure 4, we show how well metrics align with humans on our dataset, across low-level metrics (PSNR, SSIM), prior learned metries (LPIPS, DISTS) and recent large vision model embeddings

---

Ref
Ref
100
Unmoolfeo
Remove
Structure
A ignment
80
Refenince
Shawling
Colore
Riveooe
Background
Pap Referanca
Remove Color
Remove St Strading
Pamtev Structure
Am Color Stading
Background
Rm For earegouund
Romove
Remove
Shading
Foreground
Figure 6: Sensitivity of our similarity metric to different image ablations. (Left) We ablate elements of the image, assessing performance dropoff by changing the orientation, color, shading, structure, and foreground vs. background content. The model is fairly rohust to orientation, and more impacted by changes in color, shading, structure. Foreground content is more important for similarity recognition than background content. (Right) For example triplet (unmodified), we show visualizations of removing elements of the images.
(MAE, CLIP, DINO). The best- best-performing configurations of existing base models are DINO B/16, MAE B/16, CLIP B/32, and OpenCLIP B/32 (additional settings are in SM Section B. However, existing metrics have significant misalignments with humans; for example, DINO has approximately 10% disagreement. To improve performance, we finetune with LoRA (as described in Section 4.2), denoted as Tuned LoRA, achieving significant alignment improvements over pre-trained baselines (Figure 4) and training an MLP head (Tuned - MLP). Ensembling models by concatenating Open- CLIP, CLIP, and DINO features achieves the highest score across baselines (90.8%), MLP-tuned models (93,4%), ), and LoRA-tuned models (96.2%). We refer to the best LoRA-tuned ensemble model DreamSim, and use it for subsequent analysis and applications.
Do different ways of measuring human perception agree? We corroborate our results on our 2AFC test set by evaluating the models against JND perceptual judgments. The high correlation between 2AFC and JND scores indicates that our 2AFC judgments capture a general sense of similarity that can replicated across different settings (Figure 5).
Does improved mid-level perception lead to improved low-level and high-level perception? We use the same models as in Figure 4 to examine how performance our dataset corresponds to performance on two others: BAPPS, containing image triplets with low-level augmentations, and THINGS, containing triplets with categorical variations. Training alignment on BAPPS but decreases alignment on THINGS, suggesting that humans, when making an automatic judgment, tend to focus on aopearance-based visual differences (captured in BAPPS and our dataset) rather than high-level object category. We additionally evaluate on the TID2013 [64] and KADID- [49] image quality assessment (IQA) benchmarks as alternative low-level datasets. Similar to BAPPS, we find that our metric outperforms base models and is competitive with low-level metrics despite not training for low-level similarity. For full results, see SM Section B.
dataset improves human
5.2 What image attributes affect similarity decisions?
Sensitivity to image ablations. We study which image properties affect our model's decisions by ablating properties of the image triplets, such as orientation, color, shading, structure, and foreground or background components. To do this, we modify the triplets to change or discard each attribute and compute the agreement between the model decisions the modified and unmodified triplcts. This operation measures the model's robustness when given attribute is missing. Note that this analysis aims provide insight into how model determines similarity between two images, rather than to compare to human judgments; it is possible that humans may be more sensitive to different image attributes.
As shown Figure 6, our model least affected by orientation (tested by flipping the reference image), but ablating color or shading (the L or AB channels in LAB color space), structure [28], or both color and shading together using contours [12] has larger impacts. We conclude that color, shading, and structure contain critical information for model, but our model can tolerate
7

---

Ours
LPIPS Things
DISTS Things
NO
OpenCLIP Things
Things
Things
Depth
Depth
Stuff
Stuff
Stuff
Stuff
Stuff
Color
Color
Color
Color
Color
Peop a
People
People
Peop
People
Figure 7: Alignment with semantic and low-level metrics. We select random triplcts from the COCO dataset and assess how well low-level and semantic metrics predict model judgments. Compared to LPIPS & DISTS, our model is more sensitive to objects that appear in individual instances ("things") and the presence of people. and less sensitive to "stuff" categories, such as sky and road. Compared to OpenCLIP & DINO, color explains of our model's decisions. We mark model'ss results with dashed pink line for comparison.
differences in orientation. To ablate foreground and background components, we segment the image [1] and replace the corresponding pixels with random uniform noise. This operation removes the color, shading, and texture of the region, even though the same outline remains in both. Removing the background has a smaller impact on model alignment compared to removing the foreground, suggesting that the foreground color and texture is more important for similarity recognition in our model compared to the background.
Alignment with semantic and low-level metrics. Given that some image attributes, like semantics, difficult ablate, complement our ablation analysis by examining how well semantic and handcrafted low-level features align with model judgments. We use 10,000 random image triplets from MS- -SS-COCO [50] annotated with individual object instances ("things") and background categories such as grass and sky ("stuff""). We compare our metric to LPIPS, DISTS, DINO, and OpenCLIP (Figure 7). Our model aligns less strongly with RGB-color histogram similarity (calculated using histogram intersection [80]) compared to LPIPS and DISTS, however is more sensitive to color compared other large vision models, while none align with depth map similarity [27, 69]. Next, we compute alignment with the things-category and stuff- stuff-category histograms, which summarize the area occupied by each semantic category. Our model aligns better with things-histogram similarity, whereas other models are comparatively more sensitive to stuff. This result suggests that our model is relatively more sensitive to foreground object instances. Finally, we compute how often metrics align with the per-category intersection of categorical histograms. Our model aligns best with the presence of people (additional results SM Section B.
Applications
Image retrieval. We use our metric for image retrieval on two datasets: ImageNet-R [37], which contains 30K images of various renditions of ImageNet categories (e.g., sketches, cartoons), and a random subset of 30K images from MS-COCO [50], which contains diverse natural photos. Given a query image, we compute its similarity to the entire dataset and present the top-3 nearest images under each metric in Figure 8. As can be seen, our metric finds meaningful neighbors, demonstrating that it can generalize to diverse image domains. We compare to the best performing ViT embedding methods (OpenCLIP, DINO) and prior learned metries (LPIPS, DISTS), evaluated in Section 5.1. Compared to the basclines, which are either dominated by low-level similarity (c.g., background color in L LPIPS/DISTS).
higher-level similarity (e.g., semantic class in OpenCLIP), our retrieved images exhibit similarity across a continuum of visual traits. See Section B. for more results and analysis on sketch-to- to-photo and photo-to-Se ghoto-to-sketch domains.
We confirm our retrieval quality with a user study. For each metric, we retrieve the top-10 10 neighbors for a set of query images. For ImageNet-R, users prefer our metric's retrievals 36.8% of the time, followed by OpenCLIP (2) and DINO (19.5%). We observe similar trends a study with COCO images (see Figure 9).
Feature inversion. To better understand the information captured by our metric, we apply existing feature inversion techniques, where the aim is optimize for image that is similar to target image under some metric. We compare our metric to the DINO and OpenCLIP embeddings, as well as to ensemble embedding that simply concatenates all features without finetuning.

---

LPIPS
DNSTS
OpenCUP
APVP
DISTS
OpenCTiP
Ours
Impot
Input
Nearess Neighbors
Noorestt Neiphbors
LPIPS
DISTS
OpenCUP
DINO
LRVP
DNSTS
OpenCT/P
Input
Input
Nearest Neighbers
Nearest Neighbors
Figure 8: Nearest-neighhor retrieval across different metrics. Nearest neighhor retrieval ImageNet-R (top) and COCO (bottom), comparing different metrics. Although the datasets include images outside our training domain, our model consistently retrieves neighbors with similar appearance and class to that of the query image.
Figure 9: User preferences for image retrieval results by metric. We conduct user study that collects preferences retrieval results from LPIPS, DISTS, DINO, OpenCLIP. and DreamSim. We visualize one standard deviation above and below each bar. On ImageNet-R (top. orange), our metric is preferred by users 36.8% of the time, followed by OpenCLIP (28.7%) Similarly, on COCO (bottom, blue). users prefer our metric 33.6% of the time, with DINO heing the second choice (28.155 1%).
Figure 10 shows the inversion results under range of image priors. Under vanilla optimization, i.c., not using any image prior [52, 61], metric's results exhibit higher fidelity to the original colors, shapes, and semantics. Using Deep Image Prior [83, 84], our metric reveals better global layout preservation and semantics. This is also evident when using metric to guide recent generative models [22], resulting in higher quality image samples that resemble the semantic concepts and appearance of the target image.
Consistent with Section 5.2 and the retrieval results, these visualizations demonstrate our metric's sensitivity to color, layout, and semantics. The increased quality of our fine-tuned embeddings compared just ensembling shows the effectiveness of tuning on our human-aligned dataset. See SM for more examples, and Section A.4 for details feature inversion methodology.

---

lorget
OpenCLiP
Ensemble
Torger
OpenC1
Opem2olion
Gulded Diffusion DIP Inversion
Ef
Figure 10: Feature inversion across different metries and image priors. Given a target image, we optimize for an image, where the objective is to match the target image embedding of given backbone. Without any image prior (Optimization), weak image prior [83, 84] (DIP Inversion). our metric able to reproduce scene structure and semantics. Using a diffusion model [22] as a strong prior, our metric better captures overall semantics and scene appearance.
metric better recovers the color, shape and semantics of the target image. With
k-NN Classification. We evaluate metric k-Nearest Neighbors classifier, which requires the retrieval of images that are both visually and semantically relevant to given query. We find that our metric outperforms all baselines the ObjectNet [7] dataset and performs competitively with DINO on the ImageNet100 [81] dataset. For full details and results, refer to SM Section B.I.
7 Discussion
We expand the fundamental task of measuring perceptual image similarity to encompass factors beyond mere low-level similarity, spanning multiple notions of similarities (e.g., object poses, colors, shapes, and camera angles). By harnessing state-of-the-art generative model, we overcome lack of suitable data and introduce new dataset consisting of human-judged synthetic image triplets. We evaluate candidate embeddings and propose a new similarity metric that better aligns with human judgments, demonstrating its generalization and effectiveness compared to existing metrics. We note that by using Stable Diffusion (SD) [71] [],, our benchmark is exposed to potential preexisting biases and sensitive content in the model. Our perceptual model is also finetuned from existing pre-trained backbones, and thus may also inherit prior errors and biases. However, our work is a step in aligning representations to human preferences, and we believe work can inspire further research in leveraging advances in generative image models for perceptual studies.
Acknowledgments. We thank Jim DiCarlo, Liad Mudrik, Nitzan Censor, Michelle Li, and Narek Tumanyan for fruitful discussions throughout the project. This work was supported by a Packard Fellowship P.I., Israeli Science Foundation (grant 2303/20) to T.D.,, Meta PhD Fellowship to L.C., and NSF GRFP Fellowship to S.S.
10

---

References
[1] CarveKit.
8,
21
[2] Image database tid2013: Peculiarities, results and perspectives. Signal Processing: Image
Communication, 30:57-77, 2015.
[3] Daniel E. Acuna, Max Berniker, Hugo L. Fernandes, and Konrad P. Kording. Using psy- chophysics to ask if the brain samples or maximizes. Journal of Vision, 15(3):7-7, 03 2015.
3
[4] Dan Amir and Yair Weiss. Understanding and simplifying perceptual distances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 12226-12235,
2021. 3
[5] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.0588 12.08814,, 2(3);4, 2021.
[6] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. ArXiv, abs/2304.08466
2023. 3
[7] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019. 10, 23
[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Springer,
Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages
2014. 3
[9] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions, 2023.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021. 2, 3, 5,
[11] Patrick Cavanagh. The cognitive impenetrability of cognition. Behavioral and Brain Sciences,
22(3):370-371, 1999. 3
[12] Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7915-7925, 2022. 2, 4,
[13] Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301. 1070040 2023. 3
[14] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image
similarity through ranking. Journal of Machine Learning Research, 11(3), 2010. 6
[15] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition,
pages 782-791, 2021. 26
[16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 3
[17] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. ArXiv, abx/2212.07145, 2022. 5, 6
[18]
Katherine Crowson. K-diffusion. https:/ httg/gittuu....//// com/ wttp:/wwwww//ccccccoococco wiikk--fffff.oooooo 2022.
19
[19]
Michael R.W. Dawson. Cognitive Impenetrability. pages 1-3. Springer International Publishing.
Cham, 2017. 3
[20]
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabculmohsin, al. Scaling vision transformers to 22 billion parameters. arXiv preprint 2xXv:2302.005422 2023.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009. 3, 19

---

[22]
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances in Neural Information Processing Systems, 87984, 2021. 9, 10
[23] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan. editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates,
Inc., 2021. 18
[24] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence,
44(5):2567-258) 2020. 2, 3,5
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins
wxXv:2010.11929, 2020. 3.5
[26] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics
based on deep networks. Advances in neural information processing systems, 29, 2016. 3 [27] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. In NIPS. 2014. 8
[28] Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances neural information processing systems, 28, 2015.
[29] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint
2015. 3
[30] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications
of the ACM, 63(11):139-144, 2020.
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked
autoencoders are scalable vision learners, 2021. 3, 5, 19
[32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[33] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? ArXiv, abs/2210.07574, 2022.
[34] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam
and Chris I Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and
behavior. Elife, 12:882580, 2023. 2, 4
[35] Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10):00233792 2019. 2, 3
[36] Olivier J Hénaff and Eero P Simoncelli. Geodesics of learned representations. arXiv preprint
1xXv::511.0633/ 2015.
[37] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of disttii-iitiittoonn generalization.
ICCV, 2021.
[38] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen ZZuu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09 00165. 2021. 6
[39] Gabriel Ilharco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi,
Ali Farhadi. and Ludwig Schmidt. Openclip, July 2021. 2,
[40] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot
text-guided object generation with dream fields. 2022. 2
[41] Ajay Jain, Matthew Tancik, and Picter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885-5894, 2021. 2.
[42] William James. The Principles of Psychology, volume 1. Henry Holt, New York, 1890. 2
[43] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Eiion-EECVv 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part 11 14, pages 694-711 Springer, 2016.
3
12

---

[44] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019. 2
[45] Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: robust perceptual image similarity via random transformation ensembles. arXiv preprint arXiv: 0e0/999660977,.. 2019. [46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
[47] Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep
convolutional neural networks. Neural Information Processing Systems, 25, 01 2012. 3
[48] Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. Do better imagenet classifiers assess perceptual similarity better? Transactions of Machine Learning Research,
2022.
[49] Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In 2019 Eleventh International Conference Quality of Multimedia Experience (QoMEX), pages 1-3, 2019. 7.
[50] Ysung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer. 2014.
[51] S. W. Link and R. A. Heath. A sequential theory of psychological discrimination. Psychometrika,
40:77-105, 1975. 3
[52] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 5188-5196, 2015. 9
[53] Pranay Manocha, Adam Finkelstein, Richard Zhang, Nicholas J Bryan, Gautham J Mysore, and Zeyu Jin. A differentiable perceptual audio metric learned from just noticeable differences. arXiv preprint arXiv:200I. 04460, 2020. 3
[54] Pranay Manocha. Zeyu Jin, Richard Zhang. and Adam Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 196-200. IEEE, 2021. 3
[55] Rafal Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on graphics (TOG), 30(4):1-14, 2011.
[56] Brian McElree and Barbara A. Dosher. Serial retrieval processes in the recovery of order information. Journal of Experimental Psychology: General, 122:291-315, 1993, 3
[57] Doug Medin, Robert Goldstone, and Dedre Gentner. Respects for similarity. Psychological
Review, 100:254-278, 04 1993. 3
[58] Clay Mullis. Haap-guided-oe diffusion, 2022. 19
Clip guided diffusion.
https: /ttps://githbbb com/ coomaffinn /
[59] Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Korn- blith. Human alignment of neural network representations. arXiv preprint arXiv:2211.01200 2022. 3,
[60]
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008. 3
[61] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. https://distill.ubb2000 wttpa/iiii...ui////////aerrriaaa 7/feauureeiisualizztinn.. 9
[62] Taesung Park. Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis normalization. In Proceedings of the IEEE Conference on Computer with spatially-adaptio Vision and Pattern Recognition, 2019. 2
[63] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A Efros, and Eli Shechtman. Gan-supervised dense visual alignment. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 13470-13481, 2022. 3
[64] Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C.-C. Jay Kuo. Image
database tid2013. Image Commun., 30(C):57-77, jan 2015. 7, 23
[65] Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 1808-1817, 2018.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
13

---

models from natural language supervision. In International conference on machine learning.
pages 8748-88 -763. PMLR, 2021. 2. 6.6
[67]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical
text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 2022.
[68]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical
text-conditional image generation with clip latents. ArXiv, (los/2204.06122 2022. 3
[69] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020. 8
[70]
Philippe Rochat. What like be a newborn? In Shaun Gallagher, editor, The Oxford
Handbook of the Self. Oxford University Press, 2011. 2
[71] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF
Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 3, 10 [72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffuusinn models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages
14479-36494. Curran Associates, Inc.. 2022. 3
[73] Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing, (1(11))2385-2))
2009. 2
[74] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: Learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (proceedings of
SIGGRAPH). 2016. 22
[75] Mert Bulent Sariyildiz, Alahari Karteek, Diane Larlus, and Yannis Kalantidis. Fake it till you
make it: Learning transferable representations from synthetic imagenet clones. 2022. 3
[76] Robert Schreuder, Giovanni Flores d' Arcais, and Ge Glazenborg. Effects of perceptual and conceptual similarity in semantic priming. Psychological Research, 45(4):333-
1984. 2 [77] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2107- 2116, 2017. 3
[78] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale
image recognition, arXiv preprint arXiv:1409 /556, 2014.
[79]
Dustin Stokes. Cognitive penetrability of perception. Philosophy Compass, 8(7):646 663, 2013.
3
[80] Michael J Swain and Dana H Ballard, Color indexing. International journal of computer vision,
1-32, 1991. 8
[81] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Ssion-EECYV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer. 2020. 3, 10
[82] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI. page 776-794, Berlin, Heidelberg, 2020. Springer-Verre Verlag. 23 [83] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10748-10755 10757, 2022. 4, 4, 10
[84] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446 0454,,95, 2018. 9, 10, 18
[85] Andreas Veit, Serge Belongie, and Theofanis Karaletsos. Conditional similarity networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages $30-838,
2017.
[86] Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feed- forward image manipulation. In Computer Eisi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pages 170-186. Springer, 2020.
3
[87] Yacl Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Orh Amir Zamir, and Ariel Shamir. Clipasso: Semantically-awan
14

---

object sketching. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022. 3
[88] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-
612, 2004. 2
[89] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- Art Natural Language Processing. pages 48.45. Association for Computational Linguistics,
Oct. 2020. 19
[90] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings the IEEE/CVV Conference on Computer Vision
and Pattern Recognition, pages 12863-12872, 2021. 3
[91] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference
on computer vision and pattern recognition, pages 14485-3492 IEEE, 2010. 4
[92] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, (188)2378-23886 2011. [93] Richard Zhang. Making convolutional networks shift-invariant again. In International confer-
ence on machine learning, pages 7324-7323 -334.. PMLR, 2019. 3
[94] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018. 2, 3, 4, 5, 6
15

---

Appendix
In the supplemental materials, Sec. A contains additional methodological details on dataset collection and model training, Sec. B provides more analyses of our model, and Sec. C discusses the broader impact of our work, limitations, and licensing. Additional qualitative results, such as additional triplets from our dataset and visualizations of image retrieval and image reconstruction, available on our project page.
A
Method
A.1
AMT Details
User interface. During the 2AFC study, users instructed with the following prompt:
You will three images: labeled "Reference", one labeled "Image A", and one labeled "Image B". Select whether Image A or is more similar to the Reference.
For each task, users see an image triplet with the reference in the center and distortions on either side (randomized), Each user completes practice tasks, 50 real tasks, and sentinels (randomly placed), averaging minutes for the entire assignment. We discard responses from users who do not respond with 100% sentinel accuracy. See Figure (left) for an example the user interface.
Instructions for JND are as follows:
You will four images after the other. Determine whether the first and third images are identical, then whether the second and fourth images are identical. Each correct answer earns 1 point.
Users are shown four images for 500 ms each with a second gap inbetween, and are then prompted to answer the two questions in Figure 11 (right). They are given feedback and a score, though these have no bearing the JND results themselves. Each user completes 2 practice tasks, 24 tasks with "different" pairs, and 12 tasks with "same" pairs, averaging 10 minutes for the entire assignment.
The object retrieval study instructs users with the following text:
You will Pick the image that is most similar to the reference.
reference image and five other images under it.
See Figure 12 for example of object retrieval task. Each user completes practice tasks, 40 real tasks, and sentinels (randomly placed), averaging minutes for the entire assignment. We discard responses from users who do not respond with 100% sentinel accuracy.
Image Siilarity
Taak
Image Pairs
Task 90 of
Soore:
inagas was contial
Weee third deetical Wev fourth alwericah?
Figure 11: User interface for AMT studies. (Left) One image triplct shown to user in 2AFC, who is prompted pick "Image A" "Image B". (Right) In each JND task, users shown sequence of images and asked whether image pairs were identical. Upon answering, they are given correct answers.
16

---

Most Similar Image
Task 45
Instrctions: Refeennnn, choose which images beltow mone siriiier
Refenence
Image
Inage
Image
Image
Figure 12: User interface for image retrieval study. to pick the image (A-E) most similar to the reference.
To evaluate object retrieval performance, users are asked
Round
# unanimous
additional sentinel failures
# kept
1
100,000
0
100,000
2
74,346
6,750
81,096
3
63,423
2,411
65,834
4 51,097
592
51,689
43,615 5
289
43,904
6
37,696
113
37,809
30,420 7
16
30,436
8
25,079
25,079
9
22,098
0
22,098
10
20,019
20,019
Table Filtering for cognitively impenetrable triplets. We start with 100K triplets, and advance triplets to the subsequent round if the human vote remains unanimous, or if the added vote came from user who did not pass the sentinels and thus the vote is inconclusive (the vote from this user is discarded).
Cost breakdown. Across 10 rounds of 2AFC studies, we show users 477,964 triplet instances (see Table 2 for the full breakdown). Each user paid $0.50 for assignment consisting of 50 triplets, averaging $10.00/hr. In total, we pay users $4779.60 64 to collect 2AFC judgments on our dataset.
We run JND image triplets in our test set that received 2 2AFC judgments. Each user is paid $2.00 for one assignment consisting of 48 image pairs, averaging $12.00/hr. In total, we pay users $1.6.00 to collect JND judgments
test set.
Our object retrieval user study is conducted over 200 query images with 10 nearest neighbors. Users are paid $0.50 for one assignment consisting of 40 tasks, averaging $10.00/hr. In total, we pay users $40.50 to collect object retrieval judgments.
A.2 Additional Dataset Details
Filtering text prompts. As described in Sec. 3.1 we sample images using image labels drawn from popular image classification datasets. We perform basic filtering to avoid human categories that typically result in malformed generated images, such "man" and "baby". Thus, human faces only feature in a small percentage of our dataset (40.5%), typically categories such as "lab coat" and "bonnet" that are still associated with humans. We note that with some human categories
17

---

excluded, our model is still sensitive to the presence of people, as shown in Fig. 7 and Fig. 8. For full list of categories used, refer our GitHub page.
2AFC task. Following Sec. 3.2 in the main text, we filter the images over 10 rounds to obtain cognitively impenetrable triplets where humans tend vote the same way despite various differences between the images. Statistics for each round of filtering is reported in Table 2, which leaves us with roughly 20% of the original triplets containing unanimous votes. We discard all votes from any worker who fails the sentinel task. As result, not all triplets have the same number of votes. The resulting sizes of the train, validation, and test splits and additional statisties on each split are reported in Table 3 (top). In practice, we further discard triplets with five or fewer unanimous votes.
JND task. The JND triplets are meant to capture the decision boundary where two different images are similar enough to be confused as identical, in the presence of a masking image. Each triplet is divided into two pairs: Ref vs. A and Ref vs. B. These pairs are presented to different workers in different interleaving sequences. We collect three judgments for each pair and take the majority vote among the three judgments, thus collecting six judgments in total per triplet (Table (bottom)).
Dataset
Split
# Samples
Avg # Votes
Consensus Type
Train
15941
7.11
Unanimous
2AFC
Validation
1958
7.07
Unanimous
Test
2120
7.04
Unanimous
JND
Test
411
6
Majority
Table 3: Dataset Statistics. For each dataset, we report the number of samples and the average number votes for each triplet after filtering for sentinel failures. Labels for the 2AFC dataset are based on a unanimous vote, while for JND we take the majority vote over three trials pair (six trials per triplet).
A.3 Model Training.
For all fine-tuned models (both Tuned MLP and Tuned- LoRA) we use the NIGHTS training dataset, with an 80%-10%--10% split as described in Table 3, and images of size 768 x 768 resized to 224 x 224. We train on single NVIDIA GeForce RTX 3090 or NVIDIA TITAN RTX GPU with an Adam optimizer, learning rate of ensemble models) and 16 (ensemble models). In MLP-tuned training, we use width of 512. We tune the number training epochs using the validation set; for the Tuned (DreamSim) we train for 6 epochs. For Tuned LoRA models we use rank = 16, scaling 0.5. and dropout = 0.3. Training time is approximately 30 min/epoch for LoRA-tuned models, and 15 min/epoch for MLP-tuned models.
4, weight decay of 0, and batch size of 512 (non-
LoRA ensemble model
A.4 Feature Inversion
In Fig. 10 we show examples of using our metric to optimize for an image generation that is similar to some target image. We do in three experiments, with increasingly strong generative priors.
Optimization. this setup, we initialize an image randomly and update it iteratively using gradient descent with respect to the distance between it and the target image. Given a random crop of our working image and the target image we compute the distance between the two using a specific metric http/OppnnC///Eeeeel/DDeeee
and their corresponding embeddings. Note that in this setup there is no generator neural network.
Deep Image Prior. This technique involves beginning with frozen random noise which serves an input to a trainable U-Net. The U-Net is optimized iteratively by comparing the output the U-Net with the target image using different metrics and their embeddings. This introduces a CNN prior on the generated image as described in the Deep Image Prior paper [84].
Guided Diffusion.
This technique involves using classifier guidance to guide the generation of 256x256 unconditional ImageNet-trained diffusion model [23] using an input prompt image. The guidance is done by computing the gradient of some loss function, given the estimated clean image
18

---

category alignment
95
Lowe-avel
Con-ImageNet categories
90
Rose
Taned MLP
85
Tuned LoRA.
80
75
70
F=
65
60
55
55
60
65
70 75 80 85 90 95
ImageNet categories
Figure 13: Alignment on ImageNet and non-ImageNett triplets. We split the test set into triplets generated from ImageNet categories and Non-Imane Im---megeeet categories, as some model backbones are trained only on ImageNet images. For all models, alignment is highly correlated between the two splits.
and the target image, with respect to the current iteration. At every training step we compute the spherical distance loss between our model's embeddings of 64 random crops of the input image and the synthesized image. We combine this distance with total variation loss, following [18, 58J. The gradient of this loss is then added the mean of the current iteration, which pushes the generated image to be closer to the target image.
B Experiments
B.1 Additional Evaluations
Full Evaluation Large Vision Models. In Section 5.1 and Figure 4 in the main text report results using the best-performing setting of various large vision model backbones. Table shows the experimental variation multiple runs, in which the LoRA variation consistently outperforms the MLP variation. Table evaluates additional model settings, spanning different ViT model sizes, patch sizes, and strides.
Independent training seeds
Ensemble tuning
Trial 1
Trial 2
Trial 3
Trial
Trial 5
Avg.
Stdev.
MLP
93.1
93.9
92.9
93.3
0.326
LoRA
96.2
96.3
95.1
96.0
95.8
95.9
0.416
Table 4: Experimental variation for ensemble models. We train the Tuned MLP and Tuned LoRA ensemble models S seeds each and record their test accuracies the epoch recorded in the main paper.
As some models adapted from backbones trained on ImageNet [21] (including the prior learned metrics and DINO), we split our dataset into categories contained ImageNet and those not in ImageNet, and evaluate alignment (Figure 13), suggesting that the notions of visual similarity are related regardless of whether or not the triplet was generated from an ImageNet category, and whether or not the model was trained only on ImageNet.
each split. Performance both splits is highly correlated
We note that the MAE VIT B/16 checkpoint was taken from Hugging Face transformers repository [89], while the others were taken from the official Facebook repository [31].
Alignment on Alternative Datasets. As depicted in the left plot of Figure 14, training on our dataset (with either tuning method) indeed improves BAPPS metric-human alignment nearly every model, suggesting that some of these batch-based distortions implicitly still captured in dataset. We
19

---

Model
Alignment
Model Class
Model Name
Model Type
Feature
Overall
ImageNet
Non-ImageNet
Base
PSNR SSIM
57.1
57.3
57.0
Models
57.3
58.5
55.8
Prior-Learned Metries
LPIPS DISTS
AlexNet-Linear VGG16
70.7
69.3
72.7
86.0
87.1
84.5
VIT B/16
Embedding Embedding
82.2
82.6
81.7
CLIP
VIT B/32
83.1
83,8
82.1
VIT L/14
Embedding
81.8
83.3
79.8
S/8
89.0
89.7
88.0
VIT S/16
CLS
89.6
90.2
88.8
DINO
VIT B/8
CLS
88.6
88.6
88.5
VIT B/16
CLS
90.1
90.6
89.5
Models
VIT B/16
CLS
82.1
81.0
83.5
L/16
82.7
82.4
83.0
H/14
CLSS
83.5
83.2
83.8
ViT B/16
Embedding
87.1
87.8
86.2
OpenCLIP
ViT B/32
Embedding
87.6
87.5
87.6
VIT L/14
Embedding
85.9
86.7
84.9
Ensemble
VIT B/16
Mixed
90.8
91.6
89.8
CLIP
B/32
Embedding
87.3
88.2
86.2
DINO
VIT B/16
CLS
91.2
91.8
90.3
Tuned
MLP
MAE
VIT B/16
CLS
87.6
87.3
88.1
OpenCLIP
ViT B/32
Embedding
89.9
91.0
88.5
Ensemble
VIT B/16
Mixed
93.4
94.2
92.2
CLIP
B/32
Embedding
93.9
94.0
93.6
B/16
94.6
94.6
94.5
Tuned LoRA
MAE
ViT B/16
CLS
90.5
90.6
90.3
OpenCLIP
ViT B/32
Embedding Mixed
95.5
96.5
94.1
Ensemble
VIT B/16
96.2
96.6
95.5
Table 5:
Alignment on NIGHT test set. We evaluate alignment additional model settings, and separate the test set into ImageNet categories and Imn-eeeeeett categories.
observe that MAE exhibits the best out-of-the- tit-theeooxo box performance, indicating a greater sensitivity to lower-level image distortions (e.g. color and shape) than DINO, CLIP OpenCLIP. Surprisingly however, it the only model whose performance decreases on BAPPS as it further tuned. DINO, CLIP and OpenCIIP tuning, they are more attuned to higher-level image attributes that the dataset does not capture.
sensitive to the image distortions in BAPPS, suggesting that before
On THINGS, further training actually diminishes alignment with humans (see right plot of Figure 14). CLIP and OpenCLIP's superior performance on this dataset supports our hypothesis that they more well-adjusted to higher-level image attributes, which THINGS aims capture, rather than appearance-leved variations.
Our evaluations across these three datasets show that, as we train perceptual metrics that align more closely with human perceptual similarity, we also improve on low-level similarity but perform slightly worse on high-level image distortions. These results suggests that humans, when making an automatic judgment, are more inclined to focus on immediate visual differences (captured in BAPPS and our dataset) rather than the image's category, context, or related words.
20

---

% Alggmmett with Humans (LAFC)
Metric Alignment
(3AFC)
Human Algnment (THINGS)
Texr
Low-trel Essa moces
Tims: RP
Ems nodeo
% Alint
50
PSNR
SSIN
UNIS
DI5T5
MAE
CUP
OpanCUp
DINO
Ensemble
PSHR
SSIM
LPPS
DISTS
MAE
CLP
OpenCLi uP
DIÑO
Ensamble
Metric
Figure 14: Evaluation existing low-level and high-level similarity datasets. (Left) Despite never being trained for low-level similarity, LoRA-finetune models on OpenCLIP, DINO, and Ensemble achieve similar human alignment to LPIPS, which was directly trained on the BAPPS dataset. (Right) The THINGS dataset measures high-level conceptual similarity, rather than appearance similarity. As such, we find that LoRA finetuning dataset degrades performance, triplets contain appearance similarity, by design.
Metric
Image Part
Ours
DINO
OpenCLIP
DISTS
LPIPS
Color (RGB)
Foreground
58.3
53.7
51.2
59.4
60.5
Color (RGB)
Background
58.7
57.2
55,5
64.1
64.7
Color (RGB)
Total
58.4
55.8
54.1
62.9
65.3
Luminance Luminance
Foreground
54.5
51.9
49.9
56.8
55.6
Background
55.5
54.3
54.1
57.7
54.4
Luminance
Total
54.1
52.9
51.5
56.9
55.6
Depth
Total
54.2
53.6
53.3
54.7
55.8
Category Histogram
Things
58.7
58.3
55.1
55.3
53.8
Category Histogram
Stuff
59.5
58.8
57.9
62.5
61.3
Presence of Person
55.3
52.6
55.2
51.8
53.8
Presence of Furniture
53.1
52.2
54.2
53.4
53.6
Presence Textiles
52.8
52.4
53.1
51.6
53.6
Table 6:
Automated Metries on COCO. Alignment hand-crafted metrics model decisions on
COCO dataset, which provides ground-truth semantic labels.
Metric
Image Part
Ours
DINO
OpenCLIP
DISTS
LPIPS
Color (RGB)
Foreground Background
71.7
70.0
69.3
68.7
60.6
Color (RGB)
65.4
66.2
64.7
66.4
62.0
Color (RGB)
Total
69.8
67.9
67.6
66.0
64.3
Luminance
Foreground
63.1
62.6
62.2
61.4
56.1
Luminance Background
59.3
59.5
60.3
56.8
Luminance
Total
59.4
60.6
59.8
59.2
56.5
Depth
Total
54.2
53.6
53.3
54.7
55.8
Table Automated Metrics on NIGHTS. Alignment hand-crafted metrics with model decisions on our dataset.
Alignment with low-level features. In Section 5.2 and Figure 7 of the main text we report results on the alignment between our metric, OpenCLIP, DINO. LPIPS, and DISTS with low-level and semantic metrics for the COCO dataset. In Table we also report additional, fine- fine-grained results for COCO triplets. We use CarveKit [1] to segment out the foreground and background of each image, and then break down how well each metric agrees with RGB color histogram similarity, luminance histogram similarity, and depth map distance for foreground, background, and the full image. For color histograms 32 bins for each channel, and for luminance histograms
10 bins.
21

---

We also examine semantic features. For each image, we find the percentage of area that each semantic category occupies, and then compute the alignment between the difference in area for each category and perceptual metrics. Note that when the difference in area is the same for both pairs in triplet, is counted as 50% alignment. When the difference in area is smaller for the pair chosen by the perceptual metric, it is counted as 100% alignment (and 0% in the case of disagreement). In Table 6 we show the five semantic categories most aligned with our metric. Our metric has 55% alignment score with the "people" category, though does not align well above chance for other categories.
In Table we show alignment with low-level metries for our dataset (which does not have semantic annotations). On our dataset there is higher alignment with color, luminance, and depth across all metrics, as compared COCO triplets. This likely because the images in each of our dataset's triplets all share the same semantic category, making lower-level features more important than for the randomly-chosen COCO triplets. Our model aligns significantly better with foreground metries particularly foreground color- whereas LPIPS aligns slightly better with background.
Sketch-photo image retrieval. To further analyze perceptual metric performance on oft-of-domain data, we evaluate sketch-to-phote and photo-to-sketch image retrieval on the Sketchy database [74]. First, we assign sketches as queries and photos as the search space. As seen in Figure 16 (left), DINO, DISTS, and LPIPS perform poorly on this task compared to OpenCLIP and DreamSim because they focus on color similarity, retrieving photos with predominantly white/gray backgrounds. This sometimes appears in DreamSim's 's results as well, hindering performance (Figure 15, top-left). Without this failure case, we return impressive results (Fig. 15, top-right) focusing on the subject appearance. We also evaluate retrieval in the other direction, using photos queries and sketches as the search space. As shown Figure 15 (bottom). DreamSim outperforms all other metries in retrieving the main object while remaining sensitive to its size and location in the image.
1PIPS
DISTS
OptnCC/PP
DINO
Ours
APIP
DNSTS
OpenCLuP
DINO
Ours
Inpust
Noorestt Neiphbors
Nesorest Webioa Mobolbbio
IPIPS
DISTS
OpeaCU/P
DINO
Ours
JPPP
DNSTS
OpenCL/P
DINO
Ours
Input
hoor
Noorest Neighborr
Nesress
Figure 15: Sketch-photo nearest-neighbor retrieval. Image retrieval on the Sketchy datahase [74], with sketch as query (top) and photo query (hottom). OpenCLIP and DreamSim best preserve image content, returning photos the subject itself (e.g. church. top right) rather than focusing on the Mack-and-wlin sketch style in the query. Our metric also best localizes the subject. returning a small boat in the corner while OpenCLIP and DINO both miss this spatial detail (bottom right).
22

---

Sketch-based Photo Retrieval
Photo-based Sketch Retrieval
0.12
0.30
0.10
DISTS
0.25
Recall@K
0.08
LPPS
Recall@
0.20
0.06
0.15
0.04
0.10
0.02
0.05
0.00
10
3
5
10
Number of neighbors (K)
Number of neighbors (K)
Figure 16: Sketch-photo retrieval recall scores. When using real photos the queries, model outperforms all methods across nearest neighbors. obtaining a Recall @ 10 score of 0.32, compared the next best model. OpenCLIP, with Recall@ =0= 0.20. When using sketches as the queries, we find that our model is competitive with OpenCLIP and outperforms the remaining models.
Image Quality Assessment (IQA) benchmarks. We evaluate various metries on the TID2013 [64] and KADID-1OK [49] image quality assessment (IQA) benchmarks and report results in Table 8. Following standard IQA literature, we measure the Spearman Rank Order Correlation Coefficient (SROCC) between metric similarities and Mean Opinion Score (MOS). While LPIPS and FSIM remain best-performing overall, DreamSim is still competitive in IQA, outperforming most low-level metrics and base ViT models. Notably, our metric improves SROCC by ~ 0.05 after training on NIGHTS despite not having seen these low-level distortions. These results are consistent with those in Figure 14, where tuning on NIGHTS consistently improves human alignment on BAPPS.
TID2013
KADID-10K
PSNR*
0.687
0.676
SSIM
0.720
0.724
Low-level
MS-SSIM
0.798
0.802
FSIM
0,851
0.854
LPIPS
0.838
0.883
Prior learned
DISTS
0.794
0.867
DINO
0.722
0.753
Base models
OpenCLIP
0.764
0.841
Ensemble
0.757
0,812
Ours
DreamSim 0.814
0.868
Table 8: Evaluation on IQA datasets. Alignment of various models on TID2013 [64] and KADID-1OK [49] While LPIPS and FSIM are best-performing overall, DreamSim is still competitive in IQA despite not having been tuned on low-level distortions. "While in some IQA literature PSNR/FSIM refer computation the luminance channel and PSNR</FSIMc on color channels, we keep the metric names as-is and compute on color channels to be consistent with our PSNR labels elsewhere in the paper.
k-Nearest Neighbors Classification. We evaluate our model as k-Nearest Neighbor (k-NN) classi- fier on the ObjectNet [7] and ImageNet100 [82] datasets, both standard classification benchmarks. Results are shown in Figure 17. Similarly to our other applications, we also evaluate OpenCLIP, DINO, and LPIPS as baselines (we were unable to evaluate DISTS due to computational constraints).
For ImageNNtt100 10 (a 100-category subset of ImageNet), we evaluate on the validation set and use the training set as the candidate search space. We note that DINO was trained using superviirnn on ImageNet, and has been previously shown to achieve strong k-NN results on ImageNet. DINO performs best, however our model still achieves competitive performance. For ObjectNet, we use an 80:20 split to randomly partition the dataset into a search-space and test set. Our model outperforms both OpenCLIP and DINO on top-1 classification across different values of E (1,3,5, 5, 11, 21, 101, 201).
23

---

ObjectNet k-NN Classification
ImageNet100 K-NN Classification
Top-1 Accuracy (%
30
Top-1 Accuracy (%)
25
20
Nodel
Durs
DIVO
15
SpenCLIP
Ours DIO OpenCUP LPISS
(PIPS
10
5
30
13511 21
101
201
13 511 21
101
201
Number of neighbors (K)
Number of neighbors (K)
Figure 17: k-NN classification results. When evaluated as k-NN classifier on ObjectNet dataset, our model outperforms all methods across different values of k with a maximum accuracy of 30.92%; the next best method, OpenCLIP, achieves 22.48%. When evaluating on ImageNet100, our model is competitive (1-2% difference) with DINO and outperforms other baselines.
PCA Components
2AFC Score
63.4
512
95.7
768
96.1 1
1792
96.2
250
1250 Number PCA Components
500
750
1850
1520 1750
Table 9: Feature PCA Decomposition. We list 2AFC scores function of the number of PCA components kept, beating both the CLIP/OpenCLIP dimensionality (512) and DINO (768).
Figure 18: Ablating Feature Dimension. We apply PCA decomposition to output features of our model and vary the number of dimensions kept.
common failure case for all three models is when the retrieved neighbors are visually similar to the query image but of different category. We also note that ObjectNet is designed to differ from the ImageNet distribution, indicating that DreamSim may generalize well across different image distributions.
Dimensionality reduction with PCA. Our model consists of the concatenation the DINO, Open- CLIP, and CL.IP backbones, and therefore uses a higher-dimensional feature space to compute similarity compared to each of these models independently. To investigate whether the increased dimensionality is critical for improving human alignment, we ablate feature dimensions by applying PCA, taking a certain number the top components, as seen Figure 18 and Table 9. We can achieve comparable performance using just 500 of the top components, similar to the 512 dimensions of the CLIP and OpenCLIP embedding outputs, suggesting that the improved alignment is not just due the higher-dimensional feature space used to compute similarity, but rather the additional capacity and model priors obtained from ensembling different models.
B.2 Additional Visualizations
Qualitative metric comparisons. In Section 5.2 and Figure 6-7 the main text we quantitatively analyze the differences between metrics. Here, also provide qualitative analysis by comparing image pairs that achieve high and low similarity scores for each metric.
In Figure 19 we plot, for each image pair in our dataset, the similarity scores from our metric against similarity scores from DINO, OpenCLIP, DISTS, and LPIPS. We show the same for image pairs drawn from the COCO dataset. In Figure 20 and Figure 21 we show the pairs where our metric most agrees and most disagrees with DINO, OpenCLIP, DISTS, and LPIPS, for our test set and the COCO datasct. Note that for COCO draw random pairs of images that share at least instance category so
24

---

NIGHTS
UITS emients
lerty
Gur cimbbrity
OuTS errlantyy
COCO
1.4
1.4
1.04
LYTS orrdento
09
02 66
10
Our arrebents
Dura SITE berty
OUTS Frrerty
Ouma porpkntty
Figure 19: Correlation between metric scores. For image pairs from dataset (above), and the COCO dataset (below), we plot similarity scores from our metric against similarity scores from DINO, OpenCLIP, LPIPS, and DISTS. Our metric's scores most correlated with other ViT-based metrics, and correlate better with DISTS than LPIPS.
OpenCUP
DINO
cboce
Ous
C
Ours
20r
close
Ours
Visualizing differences between model and other metrics for NIGHTS. We show the examples where our metric most agrees and disagrees with other metrics. Primary differences are that our metric more sensitive to major changes in pose, semantics, and color. is less sensitive to granular changes in structure when overall appearance is preserved, such as the honeycomb example in the DISTS quadrant and the forest example in the DINO quadrant.
Figure 20:
that pairs have some semantic commonality, enabling better visualization of qualitative differences between metrics.
Comparing DISTS, the top-performing prior learned similarity metric, to our model, we find that DISTS sensitive to structural changes despite similar overall appearance (such as the width of the honeycomb or the position of similar objects), while our model rates these pairs nearby. On other hand, pairs that are
our feature space close DISTS feature space have less appearance similarity (e.g. the houses and rooms of different colors). Comparing to deep VIT features (DINO, OpenCLIPP) our model is more likely rate pairs with similar foreground color/appearance similar, and less likely for pairs that are similar semantically but not appearance-wist For COCO
25

---

DINO
cboc
Ours
CO5C
dose
Durs
Figure 21: Visualizing differences between our model and other metries for COCO. We show examples where our metric most agrees and disagrees with other metrics for pains drawn from the COCO dataset. These pairs share fewer appearance similarities than pairs drawn from our dataset. Our metric seems particularly sensitive to foreground semantic similarities, such as the horse pair in in the OpenCLIP quadrant and the snowboarders the LPIPS quadrant.
pairs, where there are fewer appearance similarities than in our dataset, our model chooses pairs that are similar semantically first, and only then appearance-wirg
Attention Map Visualizations. As an alternate way of understanding our model's similarity decisions, visualize the transformer attention maps following Chefer al. [15]. Our model consists of finetuned versions of DINO, CLIP, and OpenCLIP backbones, and the resulting attention map places the largest activations on the finetuned DINO backbone (Figure 22 (left)). Accordingly, the overall attention map looks largely similar to the attention map constructed from only the finetuned DINO branch. Compared to the pretrained versions of each model backbone, the finetuned model better captures full object identity (such as over the entire body of the fish), while also minimizing spurious attention values in the background (Figure 22 (right)). Consistent with earlier analysis, this supports the fact that the foreground plays a larger role in the DreamSim similarily judgment than the background.
Oriainal
Ours
Ours: DINO
DINO
CUP
OpenCLiP
120
OpenCLPP
CLIP
100
Frequency
80
60
Ref
40
20
1.25 B
0.00
Median Activation
Figure 22: Visualizing attention maps. The finetuned DINO branch of model has the largest contribution the attention map [15]. with the largest median activation compared to CLIP and OpenCLIP branches (computed over 400 test set images). As such, in our model, the overall attention map is similar to the attention map from only the DINO branch. Compared to the pretrained backbones, our model better captures the entire object of interest (the fish body) while reducing spurious attention in background regions.
As the DINO model has the largest contribution in the attention maps, Figure 23 shows additional examples focusing on the difference between the finetuned DINO backbone within our model, and
26

---

Original
Ours DINO
DINO
Onginol
Ours DINO
DINO
Original
Ours DINO
DINO
Ref
Ref
Originai
Ours-DINo
DINO
Originol
Ours-DiVO
DING
Original
Qurs-DINO
DINO
Ref
Ref
Ref
Comparing our model and baseline attention maps from the DINO branch. Focusing the DINO branch, which has the largest contribution in model'ss attention map. we compare the attention maps the pretrained and finetuning models. The finetuned attention map in our model better captures foreground object or relevant regions of interest. In all examples, the DINO baseline selects the image, while humans and our model select the B image.
Figure 23:
the pretrained DINO backbone prior to finetuning. This visualizes how the DINO backbone changes it is finetuned our dataset. Our finetuned model better captures the foreground object, while attention maps from the pretrained DINO backhone may only focus on small portions of the object. In the lobster example (Figure 23 (top-right)), our model places attention on the relevant parts the object, such as the lobster body rather than the claws in the A image as the claws do not appear in the other two images.
C Discussion
Broader Impacts and Limitations. Our model and dataset are developed from pretrained Stable Diffissoo, CLIP, OpenCLIP and DINO backbones. As such, our model can inherit and propagate biases existing in these models for decisions on downstream tasks. Our dataset is generated using prompts that describe a single category, and we manually filter out categories that may generate sensitive content, such violence malformed human faces. As result, the dataset has large focus on object-centric domains, and content containing humans is considered -ft-of-domain. The resulting dataset and model does not capture the full range of human similarity judgements, but only the variations that we can capture in our synthetically-generaae
dataset.
Licenses. The icons for Figure well the images for the inversion experiments licensed by Adobe Stock, under the Adobe Stock Standard License, and by Pixabay, under their content license.
IRB Disclosure. We received IRB approvals for our AMT experiments from all of the institutions involved. Accordingly, we took measures to ensure participant anonymity and refrained from showing them potentially offensive content.
27


Reviews:
[{"review_id": "wipt8xDffi", "reviewer": "Reviewer_23q7", "summary": "This paper leverages recent advances in synthetic image generation (namely text-to-image) to explore the space of image similarity assessment. They generate a novel dataset based on human forced choice input on pairs of synthetic images generated with respect to a reference image as well as interleaved visual memory tasks to assess when images are noticeably different on first assessment. The metric they learn over this novel dataset appears to better capture mid-level features such as pose and viewing angle.\n", "questions": "- How are the categories drawn from the labels? Is it a random sampling (and if so, are duplicate categories controlled for?), a subset (and if so, how was this selected?), or another arrangement?\n\n- Although it is mentioned that 7 votes is the average for retained triplets (L148), was there a minimum number required to be retained?\n\n- How do humans react to the image ablations tested in Section 5.2? It seems unclear to me whether the model should be impervious to some of the changes made.", "limitations": "The limitations seem clearly articulated.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall clearly written. The experiments are well-described and compelling, with multiple complementary facets supporting each other. The paper provides both a dataset and metric which may be of interest to the image similarity domain.", "weaknesses": "I apologize for a somewhat unclear criticism, but the JND task appears to be a visual working memory task, not a perception task. As we know from investigations of change blindness, there can be quite distinct differences between images that humans will only notice given sufficient time to explore the image, but if they happen to be attending the location that is different, they will immediately notice the difference. This seems to be getting at a different aspect of similarity than the perceptual choice task, which makes me want more of an explanation for why these two tasks are paired, or how they interrelate and why they are paired in this way. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We address questions and concerns from the reviewer below.\n\n**Reasoning for JND study**\n\nTo clarify our purpose for JND, we pair it with 2AFC following prior works like Zhang et al. [1] and Nazeri et al. [2] to verify if our 2AFC annotations generalize to a separate, independent way of measuring similarity. A high correlation between 2AFC and JND scores indicates that our judgments are reliable under multiple experimental settings. As shown in Figure 5, we achieve $r^2=0.922$ across metrics, suggesting that despite the differences in the two tasks, there is a core notion of similarity shared between the tasks – two images that appear more similar when observed simultaneously are more likely to be confused with one another when observed briefly at different times. \n\nAdditionally, JND is complementary as it is an objective measure of perceptual similarity; unlike 2AFC, which relies on subjective human ratings, it has a ground truth answer (images are either identical or different). However, JND is less data efficient than 2AFC as it relies on human mispredictions, whereas for 2AFC we can iteratively filter for a consensus agreement – thus it is easier to collect more data for 2AFC than JND. \n\n**How are categories drawn from labels?**\n\nWe begin by collecting a list of all of the labels taken from the ImageNet, CIFAR-10, CIFAR-100, Oxford 102 Flower, Food-101, and SUN397 datasets. Then, we remove duplicates by transforming our list into a set. Finally, we remove labels that produce low-quality images (corresponding to obscure classes or complex objects).  This leaves us with a filtered set of 1449 labels, with ~55% contained in ImageNet. These labels are included in the file classes.txt in our supplementary material.  Our categories for our prompts are then simply randomly sampled from our filtered set to obtain 100,000 total triplets. \n\n**Minimum number of votes**\n\nOur NIGHTS dataset contains an average of 7 votes per triplet. For training, validation, and testing, we filtered for only those triplets with at least 6 votes to reduce noise.\n\n**Humans ratings on image ablations?**\n\nThe ablation analysis in Section 5.2 does not claim that our model should be impervious to certain ablations; rather, we intend to understand what image attributes our model is sensitive to. This experiment gives us insight into how our trained model determines similarity between two images. Studying whether humans are sensitive to certain ablations and how models align to those judgments would be an interesting analysis of what models “should” be robust to, but we note that it is a separate experiment from our experiments in Section 5.2.\n\n\n**References**\n\n[1] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effec- tiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 586–595 (2018)\n\n[2] Nazeri, Kamyar, et al. \"Edgeconnect: Generative image inpainting with adversarial edge learning.\" arXiv preprint arXiv:1901.00212 (2019)."}, {"review_id": "O5rCca1ze9", "reviewer": "Reviewer_Mq9Y", "summary": "The paper expands the fundamental task of measuring perceptual image similarity to encompass factors\n beyond low-level and high-level similarity. To achieve this, it leverages the recent advances in generative visual AI and generate synthetic images with text prompts. These synthetic images are expected to contain multi-faceted notions of similarity, which are then handed over to humans for A/B similarity test. The proposed model then learns from these human similarity ratings and is shown to uncover the emergent hidden factors defining visual similarity. In extensive evaluations, it is shown to surpass a number of baselines, including DINO and CLIP.\n\n", "questions": "- Would be great if the paper shows more practical use cases that benefit from such a new visual similarity judge. For example, as a plug-n-play module in diverse image retrieval tasks? Or as an essential component for better evaluating a generative visual agent?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-- Research angle on visual similarity is timely, given its rather narrow scope of examinations in the past. Breakthrough on this angle can benefit vision community at large,.\n\n-- Using synthetic images for visual similarity learning is a promising technical path forward given our increasing controllability on generating target visual contents. Instead of harnessing human ratings on the similarity between any two real images and letting models to learn as a black-box, using synthetic images has the advantage of leaving the discretion to human designers on what constitutes visual similarity -- for example in this paper, although only superficially, manipulates on mid-level visual variations.\n\n-- Using LoRA to efficiently tune pre-trained visual ensembles towards a visual similarity judge is also reasonable.\n\n-- Extensive analysis is extensive.", "weaknesses": "The paper is itself is quite intact by itself. So what I propose here is not necessarily a weakness, but rather with such a new tool on visual similarity, I keep wondering what can it, immediately, benefit vision tasks at my hand. The paper showcases a pilot study on the application of image retrieval, but in my understanding, the paper's contribution is best leveraged as a universal (differentiable) plug-n-play module to provide a complementary power to many computer vision tasks.\n\nFor example, it would be lovely to see how this paper opens new opportunities to image quality assessment community, where the intrinsic ambiguity of ground-truth human ratings is painstaking for generalisable modelling. Can DreamSim help alleviate this problem as a regularisation force by requiring similar visual contents to be of similar quality score? Same for the problem of sketch-based image retrieval community, where sketch-image ground-truth paired data is ill-defined. In the work of [A], they have shown how leveraging a visual similarity metric like LPIPS can greatly advance the understanding of the problem. Can DreamSim bring even further breakthrough?\n\n[A] Photo Pre-Training, But for Sketch. CVPR 2023\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful suggestions, and are glad that the reviewer finds the analysis extensive and our contribution beneficial to the vision community. We address specific suggestions below. \n\n**Image Quality Assessment**\n\nThe reviewer mentions that our model may be useful for varied vision tasks, including as a differentiable module and for image retrieval. To assess its potential for the domains cited by the reviewer, we evaluated on standard benchmarks. We first ran our model on the TID2013 [1] and KADID-10k [2] datasets for image quality assessment. We discovered that DreamSim is competitive on this task, which aligns with our previous results on the BAPPS dataset. Additional details can be found in the global response. \n\n**Sketch-Image Retrieval**\n\nWe also ran our model on the Sketchy Database [3], which contains images and a set of corresponding sketches for each image. When using real photos as the queries, we find that our model outperforms all methods, obtaining a **Recall@10 score of 0.32, compared to the next best model, OpenCLIP, with a score of 0.20**. When using sketches as the queries, we find that our model is competitive with OpenCLIP and outperforms the remaining models. We find that a common failure case for our model when using a sketch query is that it finds images of unrelated small objects on a white background to be more similar to the sketch, as our model relies on appearance similarity. Additional details can be found in the global response. Quantitative results and qualitative examples are provided in the attached PDF. \n\n**Practical Use Cases**\n\nOur sketch-image retrieval and IQA results demonstrate the potential for our model to be directly plugged into other retrieval pipelines, including images that are greatly different from the training domain. In our global response, we also investigate the applicability of our model to ObjectNet and ImageNet100 classification. We hope that these additional evaluations demonstrate that our model can be readily adopted into a diverse set of downstream tasks, and we will include these additional results in our revision. \n\n**References**\n\n[1] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, C.-C. Jay Kuo, Image database TID2013: Peculiarities, results and perspectives, Signal Processing: Image Communication, vol. 30, Jan. 2015, pp. 57-77.\n\n[2] H. Lin, V. Hosu and D. Saupe, \"KADID-10k: A Large-scale Artificially Distorted IQA Database,\" 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), Berlin, Germany, 2019, pp. 1-3, doi: 10.1109/QoMEX.2019.8743252.\n\n[3] Sangkloy, Patsorn, et al. \"The sketchy database: learning to retrieve badly drawn bunnies.\" ACM Transactions on Graphics (TOG) 35.4 (2016): 1-12."}, {"review_id": "0wPdpX4J2h", "reviewer": "Reviewer_71tx", "summary": "In this paper, the authors develop a perceptual metric that assesses  images similarity holistically. They first collect a new dataset of human similarity  judgments over image pairs that are alike in diverse ways. During the dataset construction, the authors utilized Stable Diffusion to create synthetic data, aiming to model mid-level similarity. Based on the dataset, the authors observe that popular perceptual metrics fall short of explaining the new data, so they introduce a new metric, DreamSim, tuned to better align with human perception. Finally, authors applied this metric in image retrieval and showed that it can generalize to real images. ", "questions": "1.\tWhy do the authors not compare all the metrics used in Figure 4 during the image retrieval task?\n2.\tWho are the users who rated the preference on the image retrieval results?\n3.\tCan the authors give a more comprehensive evaluation of the generalizability of the proposed dataset?  \n", "limitations": "A more comprehensive evaluation of the generalizability of the dataset and metric is needed. Furthermore, humans are sensitive to faces from birth. Visual similarity on images with human faces is important.  ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Human visual similarity is an important and practical topic in the domain of computer vision. It has many potential applications such as image retrieval. One of the major contributions of this paper is the new dataset (NIGHTS), which contains human similarity judgments of 20k synthetic image triplets. I believe this dataset will be useful for fellow researchers in the related domain. Another strength of this paper is the proposed similarity metric that captures how humans naturally perceive image similarity. I also appreciate the authors detailed analyses and comparison of the metric and its performance. ", "weaknesses": "Though the paper is well written, and the topic is interesting and practical, the paper has several weaknesses. First of all, as the authors have mentioned, due to the potential biases and other limitations of Stable Diffusion, the authors deliberately avoided human faces during dataset construction. However, human faces are a common subject in general visual scenes and human perception of images containing human faces should be studied. Second, though the authors have applied the metric in image retrieval and used Figure 8 as a vivid proof to demonstrate that the proposed metric can be generalized to real images, I still have several concerns: first, is it possible to test the proposed metric on another benchmark dataset of real images, instead of the image retrieval task? Second, who are the users who rated the preference on the image retrieval results? More details of performance evaluation are needed. Third, not all metrics in Figure 4 are evaluated in the image retrieval task, why? Finally, while the authors avoided human faces in the proposed dataset, the models retrieved all images with human faces in the bottom right block in Fig 8. Does it show that the model focused on features other than human faces? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments, and address their questions and concerns below.\n\n**Inclusion of human faces**\n\nAs the reviewer notes, humans are uniquely sensitive to faces. We believe that face similarity is important but merits its own specialized studies, and a general-purpose diffusion model is not the best face generator. There is a rich history of metrics that focus on face similarity/recognition [1-4], apart from existing literature on general perceptual similarity.\n\nWe additionally clarify that we did not exclude all human faces. Although this was not the focus of our dataset, several of our categories lead Stable Diffusion to generate images with humans, such as “lab coat” and “bonnet”.  We will revise our supplementary material to clarify this. We specifically exclude categories such as man and baby which often result in malformed generated images. Thus, human faces only feature in a small percentage of our dataset (<0.5%). \t\n\nFinally, we note that even with some human categories excluded, our model is still sensitive to the presence of people. In Fig. 7 we evaluate DreamSim on image triplets drawn from the COCO dataset, and find that it is more sensitive to the presence of humans than other semantic categories. Additionally, in Fig. 8, as the reviewer mentions, our model retrieves images with humans wearing similarly-colored clothing. \n\n**Benchmarking on real images**\n\nIn Fig. 4 of the SM we evaluate our model on BAPPS and THINGS, which are benchmark perceptual similarity datasets with real images. For analysis of these results please refer to section B.1. \n\nAs a further benchmark of generalization to real images, we evaluate our model as a k-NN classifier on ObjectNet [5] and ImageNet100 [6]. Note that ObjectNet is designed to capture a different image distribution from ImageNet. On ObjectNet our model outperforms other methods, with a **Top-1 accuracy of 30.2% with k=201, compared to OpenCLIP (next best) with 22.48%**. On ImageNet DINO performs best (accuracy of 90.52% with k=5) however our model is still competitive with 89.54%. We provide additional details in the global response, and full quantitative results in the attached PDF.\n\nFinally, in response to reviewers Mq9Y and hsYj, we further evaluate our model on Image Quality Assessment datasets and show competitive performance against our baselines in the global response. \n\nWe hope that our additional evaluations on benchmark datasets demonstrate the generalizability of our model to real images on diverse applications. We will include these additional results in our revision. \n\n**Image Retrieval Evaluation**\n\nLike our 2AFC and JND judgments, image retrieval preferences were collected on Amazon Mechanical Turk. Please refer to A.1. in the supplement for more image retrieval user study details, and B.3 for the results.\n\n**Models for Retrieval Evaluation**\n\nFor our investigations of applications in retrieval and feature inversion we evaluate the best-performing models from Fig. 4. We exclude SSIM and PSNR because they were the worst-performing “low-level” (non-ViT) metrics, and we exclude CLIP and MAE because they were the lowest-performing transformer-based metrics. \n\n\n**References**\n\n[1] Deng, Jiankang, et al. \"Arcface: Additive angular margin loss for deep face recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[2] Meng, Qiang, et al. \"Magface: A universal representation for face recognition and quality assessment.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[3] Parkhi, Omkar, Andrea Vedaldi, and Andrew Zisserman. \"Deep face recognition.\" BMVC 2015-Proceedings of the British Machine Vision Conference 2015. British Machine Vision Association, 2015.\n\n[4] Schroff, Florian, Dmitry Kalenichenko, and James Philbin. \"Facenet: A unified embedding for face recognition and clustering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\n[5] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems 32, pages 9448–9458. 2019.\n\n[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database. IEEE Computer Vision and Pattern Recognition (CVPR), 2009."}, {"review_id": "c24q9swtwm", "reviewer": "Reviewer_hsYj", "summary": "This paper introduces a new dataset consisting of images generated by prompting the Stable Diffusion models, aiming for studying the mid-level image similarities. Human perceptual judgments are collected for image triplets via both 2AFC and JND tests. A perceptual metric is built upon large pretrained vision models, which is empirically shown to be highly consistent with human judgments.\n\n", "questions": "Does the proposed metric perform well on classical image quality assessment datasets?", "limitations": "Not applicable.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ It makes one of the first attempts to study the evaluation of AIGC images, the contribution of the new dataset is significant and the pipeline to collect human perceptual data is reasonable.\n+ Experiments are well organized, which manifests the shortages of current perceptual metrics in evaluating AIGC images and verifies the effectiveness of the proposed metric.\n+ The feature inversion part is interesting, aligning well with the Analysis-by-Synthesis model evaluation methodology. ", "weaknesses": "- The definition of distortion seems a bit confusing. In the literature, distortions are generally associated with degradation of image quality. In this work, distortions refer to mid-level changes, which do not necessarily lead to the loss of visual quality.\n- It is unclear how the JND experiment is conducted. When the participants are expected to give an answer of yes? Two images without any noticeable pixel changes, or perceptually the same in terms of the mid-level change?\n- More details of the feature inversion part are expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We are glad that the reviewer finds the contribution of our dataset to be significant, and the experiments well organized. Below we address specific questions and suggestions.  \n\n**Definition of distortion**\n\nWe use the term “distortions” to refer to variations between two images. We do so to be consistent with Zhang et. al [1], where distortions refer to visual changes relative to a reference image, particularly in the context of 2AFC experiments.  \n\nIn general we divide the spectrum of possible variations into low, mid, and high-level.\n- Low-level:, Pixel-level changes such as blurring or color shifts (As seen in BAPPS [1])\n- Mid-level: Variations in pose, spatial layout, style, numbers, etc. (As seen in NIGHTS)\n- High-level: Deals with conceptual information as opposed to perceptual in the low and mid levels, i.e knowledge that isn’t apparent in the pixels themselves. For example, a toaster and a skillet are conceptually similar as cooking appliances, though they don’t share much visual similarity (As seen in THINGS [2]).\n\nWe will make these divisions more apparent in the final version of our paper.\n\n**JND Experiment**\n\nIn the JND experiment, participants are shown 2 interleaved image pairs and asked whether the images in each pair were identical (i.e pixelwise) or not. We construct image pairs as $(x, \\tilde{x})$ or $(x, x)$ so there is indeed a correct answer. Notably, this task is not simple due to the time between images, where participants see a blank screen for 1 second, and the interleaving of pairs designed to overcome the use of visual working memory.\n\nJND image pairs are created by splitting 2AFC image triplets $(x, \\tilde{x}_0, \\tilde{x}_1)$ into $(x, \\tilde{x}_0)$ and $(x, \\tilde{x}_1)$. Each pair receives either a “same” or “different” vote, and we take triplets receiving one of each to conclude that the $\\tilde{x}$ associated with the “same” vote is more perceptually similar to $x$ than the $\\tilde{x}$ associated with the “different” vote. We omit JND data from triplets that receive two “same” votes or two “different” votes as they provide no perceptual signal as to which distortion is more similar. We will clarify in the revision.\n\n**Feature Inversion**\n\nWe perform the inversion experiment to better understand the information encoded in embeddings across different models. We do this with 3 separate experiments, with increasing prior strength.\n\n- Optimization - In this setup, we initialize an image randomly and update it iteratively using gradient descent with respect to the distance between it and the target image. Given a random crop of our working image and the target image we compute the distance between the two using a specific metric (DINO/CLIP/DreamSim) and their corresponding embeddings. Note that in this setup there is no generator neural network.\n\n- Deep Image Prior - This technique involves beginning with a frozen random noise which serves as an input to a trainable U-Net. The U-Net is optimized iteratively by comparing the output of the U-Net with the target image using different metrics and their embeddings. This introduces a CNN prior on the generated image as described in the Deep Image Prior paper [3]. \n\n- Guided Diffusion  - This method involves using classifier guidance to guide the generation of an unconditioned diffusion model. The guidance is done by computing the gradient of some loss function, given the estimated clean image and the target image, with respect to the current iteration $x_t$. At every training step we take the spherical distance loss between our model's embeddings of 64 random crops of the input image and the synthesized image. We combine this distance with a total variation loss, following [4, 5]. The gradient of this loss is then added to the mean of the current iteration, which pushes the generated image to be closer to the target image. \n\nWe will include these details in the revision.\n\n**Image Quality Assessment Benchmarks**\n\nWe ran our model on TID2013 [6] and KADID-10k [7] datasets for image quality assessment. We discovered that DreamSim is competitive on this task, which aligns with our previous results on the BAPPS dataset. Additional details can be found in the global response. \n\n**References**\n\n[1] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[2] Hebart MN, Dickter AH, Kidder A, Kwok WY, Corriveau A, Van Wicklin C, et al. (2019) THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS ONE 14(10): e0223792. https://doi.org/10.1371/journal.pone.0223792\n\n[3] Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. \"Deep image prior.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[4] Mullis, Clay. “Clip Guided Diffusion.” https://github.com/afiaka87/clip-guided-diffusion\n\n[5] Crowson, Katherine. “K-Diffusion.” https://github.com/crowsonkb/k-diffusion \n\n[6] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, C.-C. Jay Kuo, Image database TID2013: Peculiarities, results and perspectives, Signal Processing: Image Communication, vol. 30, Jan. 2015, pp. 57-77.\n\n[7] H. Lin, V. Hosu and D. Saupe, \"KADID-10k: A Large-scale Artificially Distorted IQA Database,\" 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), Berlin, Germany, 2019, pp. 1-3, doi: 10.1109/QoMEX.2019.8743252."}]

Author rebuttal:
We thank all reviewers for their feedback. We are glad they found our research well-motivated [Mq9y, 71tx].  Additionally, we appreciate their acknowledgment of our synthetic dataset and metric as novel [23q7, 71tx, hsYj], and of our experiments and analyses as well-organized and rigorous [23q7, Mq9Y, 71tx, hsYj]. 

We provide results on three additional applications based on suggestions from reviewers.

**Image Quality Assessment**

We ran our model on TID2013 [1] and KADID-10k [2] datasets for IQA. We note that these image distortions are low-level, structure-preserving changes, while our NIGHTS dataset does not preserve structure entirely but maintains appearance similarities. Following standard IQA literature, we measure the Spearman Rank Order Correlation Coefficient (SROCC) between metric similarities and Mean Opinion Score (MOS) [1,2,3] and obtain the results below, with best and second-best results bolded and italicized respectively:

|  |  | **TID2013** | **KADID-10k** |
|---:|---|:---:|---|
| **Low-level** | **PSNR*** | 0.687 | 0.676 |
|  | **SSIM** | 0.72 | 0.724 |
|  | **MS-SSIM** | 0.798 | 0.802 |
|  | **FSIM*** | **0.851** | 0.854 |
| **Prior learned** | **LPIPS** | _0.838_ | **0.883** |
|  | **DISTS** | 0.794 | 0.867 |
| **Base models** | **DINO** | 0.722 | 0.753 |
|  | **OpenCLIP** | 0.764 | 0.841 |
|  | **Ensemble** | 0.757 | 0.812 |
| **Ours** | **DreamSim** | 0.814 | _0.868_ |

While LPIPS and FSIM remain best-performing overall, DreamSim is still competitive in IQA. Additionally, our metric improves SROCC by ~0.05 after training on NIGHTS despite not having seen these low-level distortions. These results are consistent with those in Figure 4 of the supplement, where tuning on NIGHTS consistently improves human alignment on BAPPS.

*note that while in some IQA literature PSNR/FSIM refer to computation on the luminance channel and PSNRc/FSIMc on color channels, we keep the metric names as PSNR/FSIM and compute on color channels to be consistent with our PSNR labels elsewhere in the paper.

**KNN**

We evaluate our model as a k-NN classifier on the ObjectNet [4] and ImageNet100 [5] datasets, both standard classification benchmarks. k-NN classification requires retrieval of images that are both visually and semantically relevant to a query. Results are shown in Fig. 2 of the attached PDF. We also evaluate OpenCLIP and DINO as baselines. Due to time and computational constraints we were unable to complete evaluations for LPIPS and DISTS, however plan to include them in our revision.

For ImageNet100 (a 100-category subset of ImageNet), we evaluate on the validation set and use the training set as the candidate search space. We note that DINO was trained using self-supervision on ImageNet, and has been previously shown to achieve strong k-NN results on ImageNet. DINO performs best, however our model still achieves competitive performance. For ObjectNet, we use an 80:20 split to randomly partition the dataset into a search-space and test set. Our model outperforms both OpenCLIP and DINO on top-1 classification across different values of k. 

A common failure case for all three models is when the retrieved neighbors are visually similar to the query image but of a different category. We also note that ObjectNet was designed to differ from the ImageNet distribution, indicating that DreamSim may generalize well across different image distributions. 

**Sketch-Based & Photo-based Image Retrieval**

We ran two image retrieval experiments on the Sketchy database [6], which consists of photos and corresponding sketches. 

In the first experiment, we assign sketches as queries and photos as the search space. As seen from Fig. 2a (attached), DINO, DISTS, and LPIPS perform poorly on this task compared to OpenCLIP and DreamSim (ours) because they focus on color similarity, retrieving photos with predominantly white/gray backgrounds. This sometimes appears in DreamSim’s results as well, hindering performance (Fig. 1 attached, top-left). Without this failure case, we can return impressive results (Fig. 1 attached, top-right). We thus hypothesize that our performance is impacted by generic photos in the search space that resemble sketches.

In our second experiment, we use photos as queries and sketches as our search space. Results are shown in Fig. 2b attached, showing that DreamSim outperforms all other metrics. Qualitatively, we handle complex photos well (Fig. 1 attached, bottom-left), and are sensitive to location and size of the foreground (Fig. 1 attached, bottom-right). This observation is consistent with the analyses in our main paper.

The results of these experiments together with those in the paper, show that DreamSim is competitive or performs better than all other benchmarks compared against, across a wide variety of tasks, ranging from low-level to high-level. 

**References**

[1] Ponomarenko, Nikolay, et al. "Image database TID2013: Peculiarities, results and perspectives." Signal processing: Image communication 30 (2015): 57-77.

[2] Lin, Hanhe, Vlad Hosu, and Dietmar Saupe. "KADID-10k: A large-scale artificially distorted IQA database." 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019.

[3] Sheikh, Hamid R., Muhammad F. Sabir, and Alan C. Bovik. "A statistical evaluation of recent full reference image quality assessment algorithms." IEEE Transactions on image processing 15.11 (2006).

[4] Barbu, Andrei, et al. "Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models." Advances in neural information processing systems 32 (2019).

[5] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[6] Sangkloy, Patsorn, et al. "The sketchy database: learning to retrieve badly drawn bunnies." ACM Transactions on Graphics (2016).

Meta-review: The paper proposed a new syntactic dataset and perceptual metric that better capture mid-level features such as pose and viewing angle. In addition, the proposed model can learn from human similarity ratings to uncover possible hidden factors that define visual similarity, as shown via extensive evaluations.
Decision: Accept (spotlight)