Title: DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: perceptual similarity

foundation model

perception

computer vision

image metric
Abstract: Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks. Our project page: https://dreamsim-nights.github.io/

DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

Abstract

Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim , tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.

Our project page: https://dreamsim-nights.github.io/

1 Introduction

'A sense of sameness is the very keel and backbone of our thinking' - William James, 1890

Our understanding of the visual world hinges crucially on our ability to perceive the similarities between different images. Moreover, humans can reason about many notions of similarity, ranging from low-level perceptual properties such as color and texture, to higher-level concepts such as an object's category or a scene's emotional valence [76]. This capacity to conduct meaningful visual comparisons underlies our ability to effortlessly transfer our knowledge to new environments, e.g., recognizing unseen or unfamiliar objects based on their relatedness to familiar ones [42, 70] .

Computer vision has tried to capture this sense of similarity with low-level metrics like PSNR and SSIM [88], as well as learned perceptual metrics such as LPIPS [94] and DISTS [24]. Despite their utility, these metrics are limited in that they focus on the pixel or patch level and fail to capture higherlevel structures. These limitations have motivated researchers to reach for image-level embeddings from large vision models such as DINO or CLIP to measure image-to-image distances in a large variety of applications [12, 40, 41, 44, 62]. Recent studies have shown that these embeddings do well at capturing certain high-level similarity judgments, in particular, predicting which semantic categories will be considered alike by humans [59]. It remains unclear, however, how well these models align with human perception of richer and more fine-grained visual structure.

In this paper, we introduce a new perceptual metric, which bridges the gap between lower-level patch-based metrics and broad categorical comparisons. We collect a new dataset named NIGHTS Novel Image Generations with Human-Tested Similarity - containing human similarity judgments over image triplets. Each triplet consists of a reference image and two perturbed versions, along with human judgments as to which version is most similar to the reference (Fig. 1). We use iterative filtering together with recent diffusion models to collect our dataset, which is designed to capture image sets that are cognitively impenetrable (i.e. result in consistent decisions across different individuals) yet showcase rich variations. For example, our data contains images of similar object appearance, viewing angles, camera poses, overall layout, etc. This dataset differs qualitatively from prior low-level datasets [94], which focused on perturbations like blurring and adding noise, and from previous high-level datasets [34, 35], which showed variation just at the level of categories (e.g. 'is an image of a kitchen more like an image of a giraffe or an image of a beach').

On the mid-level similarity task presented by our data, we find that features from recent large pre-trained vision models [10, 39, 66] outperform the current set of standard perceptual metrics [24, 59, 94]. We further show that these large vision models can be tuned on our data to be substantially more human-aligned. Our resulting metric, DreamSim, can be dropped into existing pipelines and demonstrates high agreement with human visual perception in both quantitative assessments and qualitative comparisons using out-of-domain real images (e.g., image retrieval, image synthesis). We also analyze which features our metric is most sensitive to and find that, compared to previous perceptual metrics, it focuses relatively heavily on foreground objects, while compared to modern image embeddings, it does not neglect color and layout.

In summary, our contributions are the following:

A new image similarity dataset, consisting of 20k synthetic image triplets designed to be cognitively impenetrable with human judgments as labels.

A tuned metric that captures how humans naturally perceive image similarity, achieving 96.16% accuracy in predicting human judgments on our dataset.

Analysis of the image properties that affect our model's decisions.

Demonstration of downstream applications to image retrieval and synthesis.

2 Related work

Perceptual similarity. Classical metrics such as Manhattan ℓ 1 , Euclidean ℓ 2 , MSE, and PSNR use point-wise difference to measure similarity, thus failing to capture important joint image statistics. Patch-based metrics, including SSIM [88], FSIM [92], and HDR-VDP-2 [55] tackle this issue and are widely used in applications involving photometric distortions such as image quality assessment. However, they do not capture the nuances of human vision when more structural ambiguity is present [73] and are not suited for more complex image generation tasks.

With the deep learning revolution, classical metrics have been replaced by learning-based metrics [26, 29, 43]. These metrics are defined in the space of deep features extracted from pre-trained networks, such as VGG [78] or AlexNet [47]. Amir and Weiss [4] demonstrate that even untrained networks can be adapted as perceptual metrics. Zhang et al . [94] observe that feature-based metrics outperform classical metrics across different convolutional architectures and learning paradigms, suggesting that perceptual similarity is an emergent property in deep representations. Further tuning on the perceptual data yields improvements, such as in LPIPS [94], PIE-APP [65], or DPAM in the audio domain [53, 54]. Further improvements include ensembling for robustness [45], antialiasing for stability [24, 36, 93], and global descriptors for texture [24]. Muttenthaler et al .[59] provide insight into high-level human similarity by training on a subset of the THINGS [35] dataset, focusing on concept similarity and omitting visual cues for images within a category.

While strong computer vision features make for strong perceptual metrics, counterintuitively, they eventually become decorrelated with perceptual similarity [20, 48]. Today, the predominantly-used perceptual metric is LPIPS, operating on 64 × 64 patches.

Recent foundation models as metrics. Foundation models provide strong pre-trained backbones for a variety of downstream tasks. These models primarily leverage the Vision Transformer (ViT) [25] architecture and are trained through self-supervised learning [16, 32, 81]. CLIP [66] learns to map images and text captions into a shared embedding space, proving useful for many (often zero-shot) tasks [41, 67, 90]. CLIP has been employed as a perceptual metric to train models for semantic consistency [12, 87]. Another self-supervised ViT-based model, DINO [10], extracts disentangled appearance and structure descriptors that can be employed in image generation pipelines. Amir et al .[5] show that DINO encodes valuable semantic information about object parts. Our work aims to systematically evaluate such representations for perceptual similarity, also including OpenCLIP (an open-source implementation of CLIP) [39] and pre-trained masked autoencoders (MAE) [31].

Perceptual tests. The two alternative forced choice (2AFC) test has historically been used by behavioral psychologists to study decision-making [51, 56]. Humans judge the similarity between two images by choosing to consider certain dimensions of similarity more than others [85]. Gathering judgments on ambiguous sets of images can be cognitively penetrable, calling upon a subject's cognitive processes rather than a more automatic, 'wired-in' sense that is stable across humans and over time [11, 19, 79]. Previous studies have raised concerns about cognitive penetrability [57, 94]. On the other hand, as a psychophysical measure, just noticeable difference experiments (JND) are thought to be independent of subjective biases [3]. We follow best practices [94] and collect judgments on both of these complementary perceptual tests.

Synthetic data. GANs [30] have been used widely for dataset generation on tasks such as visual alignment [63], face manipulation [86], and adversarial training for image synthesis [77]. In recent years, text-driven generative models (e.g., Stable Diffusion [71], Imagen [72], DALLE-2 [68], MUSE [13]) have emerged as powerful tools for image synthesis. They have also been used to generate training data for a variety of downstream tasks [6, 9, 33, 75].

3 Perceptual dataset collection

While previous datasets focus on low-level , patch-based distortions [2, 94] or high-level , categorical [35] changes, we aim to close the gap, capturing distortions including mid-level variations. We aim to produce images with an underlying semantic commonality, but variations in a diversity of factors, such as style, color, pose, and other details, so that a human can assess their visual relationship. In Section 3.1, we describe our data generation pipeline - we prompt Stable Diffusion for related images of a given category, leveraging its natural image prior for variations within the category. We then describe our mechanism for collecting cognitively impenetrable perceptual judgments in Section 3.2.

3.1 Generating images with varied distortions

We leverage Stable Diffusion v1.4 [71], which generates diverse and high-quality images that adhere to a given text prompt. We sample images with a prompt of the same category, using the structure 'An image of a <category>' . The <category> is drawn from image labels in popular datasets: ImageNet [21], CIFAR-10 [46], CIFAR-100 [46], Oxford 102 Flower [60], Food-101 [8],

Ref

A

B

x 0

x 10

2AFC Data Collection

JND Data Collection

time

500 ms

1s

same?

yes | no

same?

yes | no

Which image is

more similar?

Dataset

Perturbation

Type

Data Source

Input

type

# Images

/ Patches

# Judge

per

sample

Judgment

type

Examples

BAPPS [94]

low-level

(traditional

&

CNN-based)

MIT-Adobe5k,

RAISE1k

64

×

64

patches

563.1k

187.7k

2.6

2AFC

19.2k

9.6k

3.0

JND

THINGS [34]

high-level

conceptual

Ebay, Flickr

scrape

Images

1.8k

4.7M

1

Odd-one-out

NIGHTS

(Ours)

low &

mid-level

synthetic

Diffusion-

synthesized

Images

60.0k

20.0k

7.1

2AFC

411

137

3.0

JND

and SUN397 [91]. While the ImageNet labels make up ∼ 55% of the categories, the superset of categories across these datasets encompass a wide range of objects and scenes.

As illustrated in Figure 1, the resulting images display mid-level variations, such as pose, perspective, and shape, falling in-between pixel/patch-based distortions and categorical differences. Additional triplets and the full list of categories are in the Supplementary Materials (SM). We generate an initial set of 100,000 image triplets, which are later filtered during the labeling process. Each triplet consists of a reference image and two 'distortions', generated from the same prompt.

3.2 Human perceptual judgments

We strive to gather perceptual judgments that are automatic (requiring little to no cognition), stable (invariant to changes in mental representation), and shared across humans, which requires a cognitively impenetrable judging process. Our main dataset consists of two alternative forced choice (2AFC) judgments on a triplet. Additionally, we collect just noticeable difference (JND) judgments on image pairs, corroborating our findings. Table 1 shows an overview of our dataset, compared to prior perceptual datasets.

Two alternative forced choice (2AFC). Figure 2 (left) illustrates our data gathering setup. We collect judgments on the commonly-used Amazon Mechanical Turk (AMT) platform [12, 83, 94]. We show participants triplets of images ( x, ˜ x 0 , ˜ x 1 ) and ask whether ˜ x 0 or ˜ x 1 ('distortions' of x ) is more similar to the reference image x . We collect judgments y ∈ { 0 , 1 } denoting the more perceptually similar distortion. This choice does not necessarily need to be easily articulated but should be cognitively impenetrable, that is, instinctive and consistent across humans.

Additionally, we use sentinels to ensure high-quality responses. Sentinels are designed as ( x, x, v ) , where one 'distortion' is the image itself and the other is from a completely different prompt. Approximately 20% of participants failed at least one sentinel, and we discard all their responses.

To collect cognitively impenetrable triplets, we design our 2AFC experiments to ensure that each triplet included in the dataset obtains a unanimous vote for either ˜ x 0 or ˜ x 1 . We divide our experiments into 10 rounds, starting with 100,000 triplets, advancing instances that maintain unanimous votes. Triplets retained through all 10 rounds can earn a maximum of 10 votes, but may have less as a result of the aforementioned sentinels. To maintain dataset quality, we only include triplets with ≥ 6 unanimous judgments in the final dataset.

Ultimately, our 2AFC dataset D 2afc = { ( x, ˜ x 0 , ˜ x 1 ) , y } consists of 20,019 triplets with an average of 7 unanimous votes each. We partition our resulting dataset into train, validation, and test components with a random 80/10/10 split. Our dataset is publicly available on our project page.

Just noticeable differences (JND). JND aims to characterize the boundary when a distortion becomes just noticeable. Below a threshold, a small perturbation (e.g., a 1-pixel shift) appears identical to a human observer. This perceptual test provides a complementary signal for perceptual similarity and allows us to exploit uniform timing and the presence of a correct answer.

Intuitively, an image pair that falls below the JND threshold should be more likely to be selected as similar in the 2AFC test; thus a high correlation between 2AFC and JND scores indicates that our judgments are reliable under multiple experimental settings. To assess how well our perceptual tests agree, we collect judgments on the triplets from the 2AFC test set, filtering for triplets that 'straddle' the JND threshold. Specifically, given triplet ( x, ˜ x 0 , ˜ x 1 ) ∈ D 2afc , we independently collect JND judgments on pairs ( x, ˜ x 0 ) and ( x, ˜ x 1 ) , keeping triplets where humans find one and only one to be identical. In total, our JND dataset contains 411 triplets D jnd = { ( x, ˜ x 0 , ˜ x 1 ) , s } , where users chose one of s ∈ { 0 , 1 } to be identical.

We show our JND procedure in Figure 2. We interleave two images and their respective distortions into sequence x → v → ˜ x → ˜ v , and ask whether the images in each pair were identical. This exploits masking, a perceptual phenomenon that occurs when an image is obscured by another. By masking with a time gap, we standardize viewing time across users and prompt for a decision based on an initial reaction and memory of the image. For each user, we show 48 distorted pairs, along with 24 identical pairs, to balance the expected responses. We collect 3 judgments per pair, with no person seeing a repeat, and take the majority vote as the label. Users mark 20.4 % of distorted images being identical, indicating that some of our distortions indeed fall below the noticeable threshold.

4 Perceptual metric learning

We next evaluate how well pretrained embeddings and learned metrics align with our data, and we investigate if alignment can be improved by fine-tuning.

4.1 Embeddings as a distance metric

We denote a distance between two images as D ( · , · ; f θ ) , where f θ is a feature extractor. We evaluate a variety of state-of-the-art candidate embeddings and metrics. LPIPS [94] and DISTS [24] use CNN backbones, whereas DINO [10],

Cosine

distance

Ref

Hinge

Loss

A

𝑑

!

𝑦

𝑑

#

Cosine

distance

B

Ensemble

LoRA

CLIP [66], OpenCLIP [17], and MAE [31] use transformer-based backbones [25]. Following standard practice, distance D ( x, ˜ x ; f θ ) = 1 -cos ( f θ ( x ) , f θ (˜ x ) ) is taken as the cosine distance between the CLS tokens taken from the last layer for DINO and MAE (before and after the layer normalization, respectively), and the embedding vector for CLIP and OpenCLIP. For LPIPS and DISTS, D ( x, ˜ x ; f θ ) is simply the distance metric itself. Given a triplet ( x, ˜ x 0 , ˜ x 1 ) , and a feature extractor f θ , the model vote is calculated as:

Ours

Evaluating human-metric agreement. Recall that for a triplet, we collect y , indicating which image a human selects as more similar. We evaluate how often each metric agrees with human judges as Score 2AFC ( f θ ) = E D 2afc [ 1 ˆ y = y ] and Score JND = E D jnd [ 1 ˆ y = s ] .

4.2 Learning an improved metric

Objective. To better align a perceptual metric with human judgments, given triplet ( x, ˜ x 0 , ˜ x 1 ) we maximize the difference between the perceptual distances d 0 , d 1 , with smaller distance associated with the distortion y humans voted to be most similar. To accomplish this, we map y ∈ { 0 , 1 } → ¯ y ∈ {-1 , 1 } and use a hinge loss, equivalent to triplet loss [14] between the embeddings, with a margin of m = 0 . 05 :

Tuning. Next, we investigate the best method to tune or modify feature extractor f θ . A challenge is the large number of parameters (billions, for ViT backbones), compared to the relatively small amount of perceptual data. Inspired by [94], we first try tuning via an MLP head, adding a 1-hiddenlayer MLP with a residual connection on top of the pre-trained embeddings. We compare this to fine-tuning through the pre-trained backbones using Low-Rank Adaptation (LoRA) [38], which we find to achieve better results than full fine-tuning. Using LoRA ( r = 16 , dropout p = 0 . 3 , α = 0 . 5 ) we tune approximately 0.67% of each model's parameters. In all experiments, LoRA significantly outperformed using the MLP head.

Feature concatenation. As multiple models may provide complementary features, we try combining them to boost performance. We concatenate features from the best-performing models on our dataset (DINO [10], CLIP [66], and OpenCLIP [17]) into one ensemble metric. By training just a handful of parameters in this ensemble (through LoRA [38]), we gain the benefits of each large model's embedding space without sacrificing much computational load while fine-tuning.

5 Experiments

5.1 How well do existing metrics align with human judgments?

In Figure 4, we show how well metrics align with humans on our dataset, across low-level metrics (PSNR, SSIM), prior learned metrics (LPIPS, DISTS) and recent large vision model embeddings

Unmodified

A

Ref

B

Remove

Background

Remove

Foreground

Flip

Reference

Remove

Color

Remove

Color+

Shading

A

Ref

B

Remove

Structure

Remove

Shading

(MAE, CLIP, DINO). The best-performing configurations of existing base models are DINO B/16, MAE B/16, CLIP B/32, and OpenCLIP B/32 (additional settings are in SM Section B.1). However, existing metrics have significant misalignments with humans; for example, DINO has approximately 10% disagreement. To improve performance, we finetune with LoRA (as described in Section 4.2), denoted as Tuned -LoRA , achieving significant alignment improvements over pre-trained baselines (Figure 4) and training an MLP head ( Tuned -MLP ). Ensembling models by concatenating OpenCLIP, CLIP, and DINO features achieves the highest score across baselines (90.8%), MLP-tuned models (93.4%), and LoRA-tuned models (96.2%). We refer to the best LoRA-tuned ensemble model as DreamSim , and use it for subsequent analysis and applications.

Do different ways of measuring human perception agree? We corroborate our results on our 2AFC test set by evaluating the models against JND perceptual judgments. The high correlation between 2AFC and JND scores indicates that our 2AFC judgments capture a general sense of similarity that can replicated across different settings (Figure 5).

Does improved mid-level perception lead to improved low-level and high-level perception? We use the same models as in Figure 4 to examine how performance on our dataset corresponds to performance on two others: BAPPS, containing image triplets with low-level augmentations, and THINGS, containing triplets with categorical variations. Training on our dataset improves human alignment on BAPPS but decreases alignment on THINGS, suggesting that humans, when making an automatic judgment, tend to focus on appearance-based visual differences (captured in BAPPS and our dataset) rather than high-level object category. We additionally evaluate on the TID2013 [64] and KADID-10k [49] image quality assessment (IQA) benchmarks as alternative low-level datasets. Similar to BAPPS, we find that our metric outperforms base models and is competitive with low-level metrics despite not training for low-level similarity. For full results, see SM Section B.1.

5.2 What image attributes affect similarity decisions?

Sensitivity to image ablations. We study which image properties affect our model's decisions by ablating properties of the image triplets, such as orientation, color, shading, structure, and foreground or background components. To do this, we modify the triplets to change or discard each attribute and compute the agreement between the model decisions on the modified and unmodified triplets. This operation measures the model's robustness when a given attribute is missing. Note that this analysis aims to provide insight into how our model determines similarity between two images, rather than to compare to human judgments; it is possible that humans may be more sensitive to different image attributes.

As shown in Figure 6, our model is least affected by orientation (tested by flipping the reference image), but ablating color or shading (the L or AB channels in LAB color space), structure [28], or both color and shading together using contours [12] has larger impacts. We conclude that color, shading, and structure contain critical information for our model, but our model can tolerate

differences in orientation. To ablate foreground and background components, we segment the image [1] and replace the corresponding pixels with random uniform noise. This operation removes the color, shading, and texture of the region, even though the same outline remains in both. Removing the background has a smaller impact on model alignment compared to removing the foreground, suggesting that the foreground color and texture is more important for similarity recognition in our model compared to the background.

Alignment with semantic and low-level metrics. Given that some image attributes, like semantics, are difficult to ablate, we complement our ablation analysis by examining how well semantic and handcrafted low-level features align with model judgments. We use 10,000 random image triplets from MS-COCO [50] annotated with individual object instances ('things') and background categories such as grass and sky ('stuff'). We compare our metric to LPIPS, DISTS, DINO, and OpenCLIP (Figure 7). Our model aligns less strongly with RGB-color histogram similarity (calculated using histogram intersection [80]) compared to LPIPS and DISTS, however is more sensitive to color compared to other large vision models, while none align with depth map similarity [27, 69]. Next, we compute alignment with the things-category and stuff-category histograms, which summarize the area occupied by each semantic category. Our model aligns better with things -histogram similarity, whereas other models are comparatively more sensitive to stuff . This result suggests that our model is relatively more sensitive to foreground object instances. Finally, we compute how often metrics align with the per-category intersection of categorical histograms. Our model aligns best with the presence of people (additional results in SM Section B.1).

6 Applications

Image retrieval. We use our metric for image retrieval on two datasets: ImageNet-R [37], which contains 30K images of various renditions of ImageNet categories (e.g., sketches, cartoons), and a random subset of 30K images from MS-COCO [50], which contains diverse natural photos. Given a query image, we compute its similarity to the entire dataset and present the top-3 nearest images under each metric in Figure 8. As can be seen, our metric finds meaningful neighbors, demonstrating that it can generalize to diverse image domains. We compare to the best performing ViT embedding methods (OpenCLIP, DINO) and prior learned metrics (LPIPS, DISTS), as evaluated in Section 5.1. Compared to the baselines, which are either dominated by low-level similarity (e.g., background color in LPIPS/DISTS), or higher-level similarity (e.g., semantic class in OpenCLIP), our retrieved images exhibit similarity across a continuum of visual traits. See Section B.1 for more results and analysis on sketch-to-photo and photo-to-sketch domains.

We confirm our retrieval quality with a user study. For each metric, we retrieve the top-10 neighbors for a set of query images. For ImageNet-R, users prefer our metric's retrievals 36.8 % of the time, followed by OpenCLIP (28.7 % ) and DINO (19.5 % ). We observe similar trends in a study with COCO images (see Figure 9).

Feature inversion. To better understand the information captured by our metric, we apply existing feature inversion techniques, where the aim is to optimize for an image that is similar to a target image under some metric. We compare our metric to the DINO and OpenCLIP embeddings, as well as to an ensemble embedding that simply concatenates all features without finetuning.

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Ours

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Ours

Ours

Ours

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

Neighbor: 1

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 2

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 3

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

Neighbor: 4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 5

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 6

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 7

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 8

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

Neighbor: 9

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

Neighbor: 10

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

0.4

LPIPS

DISTS

DINO

OpenCLIP

Ours

0.0

0.2

Figure 10 shows the inversion results under a range of image priors. Under vanilla optimization, i.e., not using any image prior [52, 61], our metric's results exhibit higher fidelity to the original colors, shapes, and semantics. Using Deep Image Prior [83, 84], our metric reveals better global layout preservation and semantics. This is also evident when using our metric to guide recent generative models [22], resulting in higher quality image samples that resemble the semantic concepts and appearance of the target image.

Consistent with Section 5.2 and the retrieval results, these visualizations demonstrate our metric's sensitivity to color, layout, and semantics. The increased quality of our fine-tuned embeddings compared to just ensembling shows the effectiveness of tuning on our human-aligned dataset. See SM for more examples, and Section A.4 for details on feature inversion methodology.

Target

Optimization

DINO

OpenCLIP

Ensemble

Ours

DIP Inversion

Guided Diffusion

Target

DINO

OpenCLIP

Ensemble

Ours

k-NN Classification. We evaluate our metric as a k -Nearest Neighbors classifier, which requires the retrieval of images that are both visually and semantically relevant to a given query. We find that our metric outperforms all baselines on the ObjectNet [7] dataset and performs competitively with DINO on the ImageNet100 [81] dataset. For full details and results, refer to SM Section B.1.

7 Discussion

We expand the fundamental task of measuring perceptual image similarity to encompass factors beyond mere low-level similarity, spanning multiple notions of similarities (e.g., object poses, colors, shapes, and camera angles). By harnessing a state-of-the-art generative model, we overcome the lack of suitable data and introduce a new dataset consisting of human-judged synthetic image triplets. We evaluate candidate embeddings and propose a new similarity metric that better aligns with human judgments, demonstrating its generalization and effectiveness compared to existing metrics. We note that by using Stable Diffusion (SD) [71], our benchmark is exposed to potential preexisting biases and sensitive content in the model. Our perceptual model is also finetuned from existing pre-trained backbones, and thus may also inherit prior errors and biases. However, our work is a step in aligning representations to human preferences, and we believe our work can inspire further research in leveraging advances in generative image models for perceptual studies.

Acknowledgments. We thank Jim DiCarlo, Liad Mudrik, Nitzan Censor, Michelle Li, and Narek Tumanyan for fruitful discussions throughout the project. This work was supported by a Packard Fellowship to P.I., Israeli Science Foundation (grant 2303/20) to T.D., Meta PhD Fellowship to L.C., and NSF GRFP Fellowship to S.S.

References

CarveKit. https://github.com/OPHoperHPO/image-background-remove-tool/ . 8, 21

Image database tid2013: Peculiarities, results and perspectives. Signal Processing: Image Communication , 30:57-77, 2015. 3

Daniel E. Acuna, Max Berniker, Hugo L. Fernandes, and Konrad P. Kording. Using psychophysics to ask if the brain samples or maximizes. Journal of Vision , 15(3):7-7, 03 2015. 3

Dan Amir and Yair Weiss. Understanding and simplifying perceptual distances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12226-12235, 2021. 3

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021. 3

Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. ArXiv , abs/2304.08466, 2023. 3

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. 10, 23

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014. 3

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions, 2023. 3

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021. 2, 3, 5, 6

Patrick Cavanagh. The cognitive impenetrability of cognition. Behavioral and Brain Sciences , 22(3):370-371, 1999. 3

Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7915-7925, 2022. 2, 3, 4, 7

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023. 3

Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research , 11(3), 2010. 6

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 782-791, 2021. 26

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. 3

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. ArXiv , abs/2212.07143, 2022. 5, 6

Katherine Crowson. k-diffusion. https://github.com/crowsonkb/k-diffusion , 2022. 19

Michael R.W. Dawson. Cognitive Impenetrability , pages 1-3. Springer International Publishing, Cham, 2017. 3

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 , 2023. 3

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 3, 19

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021. 9, 10

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8780-8794. Curran Associates, Inc., 2021. 18

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence , 44(5):2567-2581, 2020. 2, 3, 5

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 3, 5

Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems , 29, 2016. 3

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS , 2014. 8

Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances in neural information processing systems , 28, 2015. 7

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 , 2015. 3

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020. 3

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 3, 5, 19

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020. 3

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? ArXiv , abs/2210.07574, 2022. 3

Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife , 12:e82580, 2023. 2, 4

Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one , 14(10):e0223792, 2019. 2, 3

Olivier J Hénaff and Eero P Simoncelli. Geodesics of learned representations. arXiv preprint arXiv:1511.06394 , 2015. 3

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV , 2021. 8

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. 6

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. 2, 3

Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022. 2

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5885-5894, 2021. 2, 3

William James. The Principles of Psychology , volume 1. Henry Holt, New York, 1890. 2

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 694-711. Springer, 2016. 3

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019. 2

Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: robust perceptual image similarity via random transformation ensembles. arXiv preprint arXiv:1906.03973 , 2019. 3

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Neural Information Processing Systems , 25, 01 2012. 3

Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. Do better imagenet classifiers assess perceptual similarity better? Transactions of Machine Learning Research , 2022. 3

Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX) , pages 1-3, 2019. 7, 23

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 8

S. W. Link and R. A. Heath. A sequential theory of psychological discrimination. Psychometrika , 40:77-105, 1975. 3

Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5188-5196, 2015. 9

Pranay Manocha, Adam Finkelstein, Richard Zhang, Nicholas J Bryan, Gautham J Mysore, and Zeyu Jin. A differentiable perceptual audio metric learned from just noticeable differences. arXiv preprint arXiv:2001.04460 , 2020. 3

Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 196-200. IEEE, 2021. 3

Rafał Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on graphics (TOG) , 30(4):1-14, 2011. 2

Brian McElree and Barbara A. Dosher. Serial retrieval processes in the recovery of order information. Journal of Experimental Psychology: General , 122:291-315, 1993. 3

Doug Medin, Robert Goldstone, and Dedre Gentner. Respects for similarity. Psychological Review , 100:254-278, 04 1993. 3

Clay Mullis. Clip guided diffusion. https://github.com/afiaka87/ clip-guided-diffusion , 2022. 19

Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. arXiv preprint arXiv:2211.01201 , 2022. 2, 3

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008. 3

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill , 2017. https://distill.pub/2017/feature-visualization. 9

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019. 2

William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A Efros, and Eli Shechtman. Gan-supervised dense visual alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13470-13481, 2022. 3

Nikolay Ponomarenko, Lina Jin, Oleg Ieremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, and C.-C. Jay Kuo. Image database tid2013. Image Commun. , 30(C):57-77, jan 2015. 7, 23

Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1808-1817, 2018. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual

models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 2, 3, 5, 6

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 3

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv , abs/2204.06125, 2022. 3

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2020. 8

Philippe Rochat. What is it like to be a newborn? In Shaun Gallagher, editor, The Oxford Handbook of the Self . Oxford University Press, 2011. 2

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 3, 10

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 36479-36494. Curran Associates, Inc., 2022. 3

Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing , 18(11):2385-2401, 2009. 2

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: Learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (proceedings of SIGGRAPH) , 2016. 22

Mert Bulent Sariyildiz, Alahari Karteek, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. 2022. 3

Robert Schreuder, Giovanni B Flores d'Arcais, and Ge Glazenborg. Effects of perceptual and conceptual similarity in semantic priming. Psychological Research , 45(4):339-354, 1984. 2

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 21072116, 2017. 3

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 3

Dustin Stokes. Cognitive penetrability of perception. Philosophy Compass , 8(7):646-663, 2013. 3

Michael J Swain and Dana H Ballard. Color indexing. International journal of computer vision , 7(1):11-32, 1991. 8

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020. 3, 10

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI , page 776-794, Berlin, Heidelberg, 2020. Springer-Verlag. 23

Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10748-10757, 2022. 4, 9, 10

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9446-9454, 2018. 9, 10, 18

Andreas Veit, Serge Belongie, and Theofanis Karaletsos. Conditional similarity networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 830-838, 2017. 3

Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feedforward image manipulation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16 , pages 170-186. Springer, 2020. 3

Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware

object sketching. ACM Transactions on Graphics (TOG) , 41(4):1-11, 2022. 3

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004. 2

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theArt Natural Language Processing. pages 38-45. Association for Computational Linguistics, Oct. 2020. 19

Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12863-12872, 2021. 3

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010. 4

Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing , 20(8):2378-2386, 2011. 2

Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning , pages 7324-7334. PMLR, 2019. 3

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018. 2, 3, 4, 5, 6

Appendix

In the supplemental materials, Sec. A contains additional methodological details on dataset collection and model training, Sec. B provides more analyses of our model, and Sec. C discusses the broader impact of our work, limitations, and licensing. Additional qualitative results, such as additional triplets from our dataset and visualizations of image retrieval and image reconstruction, are available on our project page.

A Method

A.1 AMTDetails

User interface. During the 2AFC study, users are instructed with the following prompt:

For each task, users see an image triplet with the reference in the center and distortions on either side (randomized). Each user completes 2 practice tasks, 50 real tasks, and 10 sentinels (randomly placed), averaging 3 minutes for the entire assignment. We discard responses from users who do not respond with 100 % sentinel accuracy. See Figure 11 (left) for an example of the user interface.

Instructions for JND are as follows:

Users are shown four images for 500 ms each with a 1 second gap inbetween, and are then prompted to answer the two questions in Figure 11 (right). They are given feedback and a score, though these have no bearing on the JND results themselves. Each user completes 2 practice tasks, 24 tasks with 'different' pairs, and 12 tasks with 'same' pairs, averaging 10 minutes for the entire assignment.

The object retrieval study instructs users with the following text:

See Figure 12 for an example of an object retrieval task. Each user completes 2 practice tasks, 40 real tasks, and 5 sentinels (randomly placed), averaging 3 minutes for the entire assignment. We discard responses from users who do not respond with 100 % sentinel accuracy.

ImageSimilarity

Task9of60

ImagePairs

Task10of36

Score:11

Instructions: Based on the Reference, choose whether Image A or Image B is more similar.

Instructions: Determine whether the first and third images were identical, and whether the second and fourth

imageswere identical.

Were the first and third images identical?Were the second and fourth images identical?

Yes

No

Yes

No

Correct!

Incorrect

Image A

Reference

Image B

Ready

Most Similar Image

Task18of 45

Reference

Image C

Image B

Image A

Image D

Image E

Cost breakdown. Across 10 rounds of 2AFC studies, we show users 477,964 triplet instances (see Table 2 for the full breakdown). Each user is paid $0.50 for one assignment consisting of 50 triplets, averaging $10.00/hr. In total, we pay users $4779.64 to collect 2AFC judgments on our dataset.

We run JND on image triplets in our test set that received ≥ 6 2AFC judgments. Each user is paid $2.00 for one assignment consisting of 48 image pairs, averaging $12.00/hr. In total, we pay users $156.00 to collect JND judgments on our test set.

Our object retrieval user study is conducted over 200 query images with 10 nearest neighbors. Users are paid $0.50 for one assignment consisting of 40 tasks, averaging $10.00/hr. In total, we pay users $40.50 to collect object retrieval judgments.

A.2 Additional Dataset Details

Filtering text prompts. As described in Sec. 3.1 we sample images using image labels drawn from popular image classification datasets. We perform basic filtering to avoid human categories that typically result in malformed generated images, such as 'man' and 'baby'. Thus, human faces only feature in a small percentage of our dataset (<0.5%), typically in categories such as 'lab coat' and 'bonnet' that are still associated with humans. We note that even with some human categories

excluded, our model is still sensitive to the presence of people, as shown in Fig. 7 and Fig. 8. For a full list of categories used, refer to our GitHub page.

2AFC task. Following Sec. 3.2 in the main text, we filter the images over 10 rounds to obtain cognitively impenetrable triplets where humans tend to vote the same way despite various differences between the images. Statistics for each round of filtering is reported in Table 2, which leaves us with roughly 20% of the original triplets containing unanimous votes. We discard all votes from any worker who fails the sentinel task. As a result, not all triplets have the same number of votes. The resulting sizes of the train, validation, and test splits and additional statistics on each split are reported in Table 3 (top). In practice, we further discard triplets with five or fewer unanimous votes.

JND task. The JND triplets are meant to capture the decision boundary where two different images are similar enough to be confused as identical, in the presence of a masking image. Each triplet is divided into two pairs: Ref vs. A and Ref vs. B. These pairs are presented to different workers in different interleaving sequences. We collect three judgments for each pair and take the majority vote among the three judgments, thus collecting six judgments in total per triplet (Table 3 (bottom)).

A.3 Model Training.

For all fine-tuned models (both Tuned -MLP and Tuned -LoRA ) we use the NIGHTS training dataset, with an 80%-10%-10% split as described in Table 3, and images of size 768 × 768 resized to 224 × 224 . We train on a single NVIDIA GeForce RTX 3090 or NVIDIA TITAN RTX GPU with an Adam optimizer, learning rate of 3 e -4 , weight decay of 0, and batch size of 512 (nonensemble models) and 16 (ensemble models). In MLP-tuned training, we use a width of 512. We tune the number of training epochs using the validation set; for the Tuned -LoRA ensemble model (DreamSim) we train for 6 epochs. For Tuned -LoRA models we use rank r = 16 , scaling α = 0 . 5 , and dropout p = 0 . 3 . Training time is approximately 30 min/epoch for LoRA-tuned models, and 15 min/epoch for MLP-tuned models.

A.4 Feature Inversion

In Fig. 10 we show examples of using our metric to optimize for an image generation that is similar to some target image. We do so in three experiments, with increasingly strong generative priors.

Optimization. In this setup, we initialize an image randomly and update it iteratively using gradient descent with respect to the distance between it and the target image. Given a random crop of our working image and the target image we compute the distance between the two using a specific metric (DINO/OpenCLIP/Ensemble/DreamSim) and their corresponding embeddings. Note that in this setup there is no generator neural network.

Deep Image Prior. This technique involves beginning with a frozen random noise which serves as an input to a trainable U-Net. The U-Net is optimized iteratively by comparing the output of the U-Net with the target image using different metrics and their embeddings. This introduces a CNN prior on the generated image as described in the Deep Image Prior paper [84].

Guided Diffusion. This technique involves using classifier guidance to guide the generation of a 256x256 unconditional ImageNet-trained diffusion model [23] using an input prompt image. The guidance is done by computing the gradient of some loss function, given the estimated clean image

and the target image, with respect to the current iteration. At every training step we compute the spherical distance loss between our model's embeddings of 64 random crops of the input image and the synthesized image. We combine this distance with a total variation loss, following [18, 58]. The gradient of this loss is then added to the mean of the current iteration, which pushes the generated image to be closer to the target image.

B Experiments

B.1 Additional Evaluations

Full Evaluation on Large Vision Models. In Section 5.1 and Figure 4 in the main text we report results using the best-performing setting of various large vision model backbones. Table 4 shows the experimental variation over multiple runs, in which the LoRA variation consistently outperforms the MLP variation. Table 5 evaluates additional model settings, spanning different ViT model sizes, patch sizes, and strides.

As some models are adapted from backbones trained on ImageNet [21] (including the prior learned metrics and DINO), we split our dataset into categories contained in ImageNet and those not in ImageNet, and evaluate alignment on each split. Performance on both splits is highly correlated (Figure 13), suggesting that the notions of visual similarity are related regardless of whether or not the triplet was generated from an ImageNet category, and whether or not the model was trained only on ImageNet.

We note that the MAE ViT B/16 checkpoint was taken from Hugging Face transformers repository [89], while the others were taken from the official Facebook repository [31].

Alignment on Alternative Datasets. As depicted in the left plot of Figure 14, training on our dataset (with either tuning method) indeed improves BAPPS metric-human alignment in nearly every model, suggesting that some of these patch-based distortions are implicitly still captured in our dataset. We

observe that MAE exhibits the best out-of-the-box performance, indicating a greater sensitivity to lower-level image distortions (e.g. color and shape) than DINO, CLIP or OpenCLIP. Surprisingly however, it is the only model whose performance decreases on BAPPS as it is further tuned. DINO, CLIP and OpenCLIP are not as sensitive to the image distortions in BAPPS, suggesting that before tuning, they are more attuned to higher-level image attributes that the dataset does not capture.

On THINGS, further training actually diminishes alignment with humans (see right plot of Figure 14). CLIP and OpenCLIP's superior performance on this dataset supports our hypothesis that they are more well-adjusted to higher-level image attributes, which THINGS aims to capture, rather than appearance-level variations.

Our evaluations across these three datasets show that, as we train perceptual metrics that align more closely with human perceptual similarity, we also improve on low-level similarity but perform slightly worse on high-level image distortions. These results suggests that humans, when making an automatic judgment, are more inclined to focus on immediate visual differences (captured in BAPPS and our dataset) rather than the image's category, context, or related words.

PSNR

SSIM

LPIPS

DISTS

MAE

CLIP

OpenCLIP

DINO

Ensemble

Metric

50

55

60

65

70

75

% Alignment with Humans (2AFC)

Metric-Human Alignment (BAPPS)

Low-level

Prior learned metrics

Base models

Tuned - MLP

Tuned - LoRA

Alignment with low-level features. In Section 5.2 and Figure 7 of the main text we report results on the alignment between our metric, OpenCLIP, DINO, LPIPS, and DISTS with low-level and semantic metrics for the COCO dataset. In Table 6 we also report additional, fine-grained results for COCO triplets. We use CarveKit [1] to segment out the foreground and background of each image, and then break down how well each metric agrees with RGB color histogram similarity, luminance histogram similarity, and depth map distance for foreground, background, and the full image. For color histograms we use 32 bins for each channel, and for luminance histograms we use 10 bins.

We also examine semantic features. For each image, we find the percentage of area that each semantic category occupies, and then compute the alignment between the difference in area for each category and perceptual metrics. Note that when the difference in area is the same for both pairs in a triplet, it is counted as 50% alignment. When the difference in area is smaller for the pair chosen by the perceptual metric, it is counted as 100% alignment (and 0% in the case of disagreement). In Table 6 we show the five semantic categories most aligned with our metric. Our metric has a 55% alignment score with the 'people' category, though does not align well above chance for other categories.

In Table 7 we show alignment with low-level metrics for our dataset (which does not have semantic annotations). On our dataset there is higher alignment with color, luminance, and depth across all metrics, as compared to COCO triplets. This is likely because the images in each of our dataset's triplets all share the same semantic category, making lower-level features more important than for the randomly-chosen COCO triplets. Our model aligns significantly better with foreground metrics particularly foreground color - whereas LPIPS aligns slightly better with background.

Sketch-photo image retrieval. To further analyze perceptual metric performance on out-of-domain data, we evaluate sketch-to-photo and photo-to-sketch image retrieval on the Sketchy database [74]. First, we assign sketches as queries and photos as the search space. As seen in Figure 16 (left), DINO, DISTS, and LPIPS perform poorly on this task compared to OpenCLIP and DreamSim because they focus on color similarity, retrieving photos with predominantly white/gray backgrounds. This sometimes appears in DreamSim's results as well, hindering performance (Figure 15, top-left). Without this failure case, we return impressive results (Fig. 15, top-right) focusing on the subject appearance. We also evaluate retrieval in the other direction, using photos as queries and sketches as the search space. As shown in Figure 15 (bottom), DreamSim outperforms all other metrics in retrieving the main object while remaining sensitive to its size and location in the image.

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Ours

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Input

LPIPS

Nearest Neighbors

DISTS

OpenCLIP

DINO

Ours

Ours

Ours

+

①

④

1

3

5

10

Number of neighbors (K)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Recall@K

Sketch-based Photo Retrieval

Ours

DINO

OpenCLIP

DISTS

LPIPS

1

3

5

10

Number of neighbors (K)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Recall@K

Ours

DINO

OpenCLIP

DISTS

LPIPS

Image Quality Assessment (IQA) benchmarks. We evaluate various metrics on the TID2013 [64] and KADID-10k [49] image quality assessment (IQA) benchmarks and report results in Table 8. Following standard IQA literature, we measure the Spearman Rank Order Correlation Coefficient (SROCC) between metric similarities and Mean Opinion Score (MOS). While LPIPS and FSIM remain best-performing overall, DreamSim is still competitive in IQA, outperforming most low-level metrics and base ViT models. Notably, our metric improves SROCC by ∼ 0 . 05 after training on NIGHTS despite not having seen these low-level distortions. These results are consistent with those in Figure 14, where tuning on NIGHTS consistently improves human alignment on BAPPS.

k -Nearest Neighbors Classification. We evaluate our model as a k -Nearest Neighbor (k-NN) classifier on the ObjectNet [7] and ImageNet100 [82] datasets, both standard classification benchmarks. Results are shown in Figure 17. Similarly to our other applications, we also evaluate OpenCLIP, DINO, and LPIPS as baselines (we were unable to evaluate DISTS due to computational constraints).

For ImageNet100 (a 100-category subset of ImageNet), we evaluate on the validation set and use the training set as the candidate search space. We note that DINO was trained using self-supervision on ImageNet, and has been previously shown to achieve strong k-NN results on ImageNet. DINO performs best, however our model still achieves competitive performance. For ObjectNet, we use an 80:20 split to randomly partition the dataset into a search-space and test set. Our model outperforms both OpenCLIP and DINO on top-1 classification across different values of k ∈ { 1 , 3 , 5 , 11 , 21 , 101 , 201 } .

Top-1 Accuracy (%)

30

25

20

15

10

5

ObjectNet k-NN Classification

Ours

DINO

OpenCLIP

LPIPS

201

13511 21

101

201

Number of neighbors (K)

30

40

50

60

70

80

90

Top-1 Accuracy (%)

ImageNet100 k-NN Classification

Model

Ours

DINO

OpenCLIP

LPIPS

13511 21

101

Number of neighbors (K)

0

250

500

750

1000

1250

1500

1750

Number of PCA Components

50

60

70

80

90

% Alignment with Humans (2AFC)

A common failure case for all three models is when the retrieved neighbors are visually similar to the query image but of a different category. We also note that ObjectNet is designed to differ from the ImageNet distribution, indicating that DreamSim may generalize well across different image distributions.

Dimensionality reduction with PCA. Our model consists of the concatenation of the DINO, OpenCLIP, and CLIP backbones, and therefore uses a higher-dimensional feature space to compute similarity compared to each of these models independently. To investigate whether the increased dimensionality is critical for improving human alignment, we ablate feature dimensions by applying PCA, taking a certain number of the top components, as seen in Figure 18 and Table 9. We can achieve comparable performance using just 500 of the top components, similar to the 512 dimensions of the CLIP and OpenCLIP embedding outputs, suggesting that the improved alignment is not just due to the higher-dimensional feature space used to compute similarity, but rather the additional capacity and model priors obtained from ensembling different models.

B.2 Additional Visualizations

Qualitative metric comparisons. In Section 5.2 and Figure 6-7 of the main text we quantitatively analyze the differences between metrics. Here, we also provide a qualitative analysis by comparing image pairs that achieve high and low similarity scores for each metric.

In Figure 19 we plot, for each image pair in our dataset, the similarity scores from our metric against similarity scores from DINO, OpenCLIP, DISTS, and LPIPS. We show the same for image pairs drawn from the COCO dataset. In Figure 20 and Figure 21 we show the pairs where our metric most agrees and most disagrees with DINO, OpenCLIP, DISTS, and LPIPS, for our test set and the COCO dataset. Note that for COCO we draw random pairs of images that share at least instance category so

NIGHTS

COCO

F0'T

1.0

1.0

1.0

DINO similarity

(normalized scores)

0.8

OpenCLIP similarity

(normalized scores)

0.8-

LPIPS similarity

(normalized scores)

DISTS similarity

(normalized scores)

0.6

0.6

0.4 -

0.4

t0

z0

z0

00

0.0

0.2

Ours similarity

0.4

0.6

0.8

T

1.0

00

0.0

0.2

Ours similarity

0.4

0.6 0.8

1.0

00

0.0

0.2

Ours similarity

0.40.6

0.8

1.0

00

0.0

0.2

Ours similarity

0.4

0.6

0.8

T

1.0

(normalized scores)

(normalized scores)

(normalized scores)

(normalized scores)

1.0

1.0

1.0

1.0

DINO similarity

(normalized scores)

0.8

OpenCLIP similarity

(normalized scores)

0.8

LPIPS similarity

(normalized scores)

(normalized scores)

0.8

0.6

o.6-

0.6-

z0

z'0

00

00

10°0

00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4  0.6

0.8

1.0

0.0

0.2

0.4 0.6

0.8

1.0

Ours similarity

Ours similarity

Ours similarity

Ours similarity

(normalized scores)

(normalized scores)

(normalized scores)

(normalized scores)

Ours

far

close

LPIPS

close

Ours

far

close

OpenCLIP

far

close

Ours

far

close

DINO

far

close

Ours

far

close

DISTS

far

close

+

CVEtle

that pairs have some semantic commonality, enabling better visualization of qualitative differences between metrics.

Comparing DISTS, the top-performing prior learned similarity metric, to our model, we find that DISTS is sensitive to structural changes despite similar overall appearance (such as the width of the honeycomb or the position of similar objects), while our model rates these pairs as nearby. On the other hand, pairs that are far in our feature space but close in DISTS feature space have less appearance similarity ( e . g . the houses and rooms of different colors). Comparing to deep ViT features (DINO, OpenCLIP) our model is more likely to rate pairs with similar foreground color/appearance as similar, and less likely for pairs that are similar semantically but not appearance-wise. For COCO

Ours

far

close

DINO

far

close

Ours

far

close

OpenCLIP

far

close

Ours

far

close

DISTS

far

close

Ours

far

close

LPIPS

far

close

pairs, where there are fewer appearance similarities than in our dataset, our model chooses pairs that are similar semantically first, and only then appearance-wise.

Attention Map Visualizations. As an alternate way of understanding our model's similarity decisions, we visualize the transformer attention maps following Chefer et al. [15]. Our model consists of finetuned versions of DINO, CLIP, and OpenCLIP backbones, and the resulting attention map places the largest activations on the finetuned DINO backbone (Figure 22 (left)). Accordingly, the overall attention map looks largely similar to the attention map constructed from only the finetuned DINO branch. Compared to the pretrained versions of each model backbone, the finetuned model better captures full object identity (such as over the entire body of the fish), while also minimizing spurious attention values in the background (Figure 22 (right)). Consistent with earlier analysis, this supports the fact that the foreground plays a larger role in the DreamSim similarily judgment than the background.

Original

A

Ref

B

DINO

OpenCLIP

Ours

Ours-DINO

CLIP

As the DINO model has the largest contribution in the attention maps, Figure 23 shows additional examples focusing on the difference between the finetuned DINO backbone within our model, and

A

Ref

B

Original

DINO

Ours-DINO

Ref

A

B

Original

DINO

Ours-DINO

Ref

A

B

Original

DINO

Ours-DINO

Ref

A

B

Original

DINO

Ours-DINO

Ref

A

B

Original

DINO

Ours-DINO

Ref

A

B

Original

DINO

Ours-DINO

E

+

the pretrained DINO backbone prior to finetuning. This visualizes how the DINO backbone changes as it is finetuned on our dataset. Our finetuned model better captures the foreground object, while attention maps from the pretrained DINO backbone may only focus on small portions of the object. In the lobster example (Figure 23 (top-right)), our model places attention on the relevant parts of the object, such as the lobster body rather than the claws in the A image as the claws do not appear in the other two images.

C Discussion

Broader Impacts and Limitations. Our model and dataset are developed from pretrained Stable Diffusion, CLIP, OpenCLIP and DINO backbones. As such, our model can inherit and propagate biases existing in these models for decisions on downstream tasks. Our dataset is generated using prompts that describe a single category, and we manually filter out categories that may generate sensitive content, such as violence or malformed human faces. As a result, the dataset has a large focus on object-centric domains, and content containing humans is considered out-of-domain. The resulting dataset and model does not capture the full range of human similarity judgements, but only the variations that we can capture in our synthetically-generated dataset.

Licenses. The icons for Figure 1 as well as the images for the inversion experiments are licensed by Adobe Stock, under the Adobe Stock Standard License, and by Pixabay, under their content license.

IRB Disclosure. We received IRB approvals for our AMT experiments from all of the institutions involved. Accordingly, we took measures to ensure participant anonymity and refrained from showing them potentially offensive content.

Peer reviews:
comments: SUMMARY:
This paper introduces a new dataset consisting of images generated by prompting the Stable Diffusion models, aiming for studying the mid-level image similarities. Human perceptual judgments are collected for image triplets via both 2AFC and JND tests. A perceptual metric is built upon large pretrained vision models, which is empirically shown to be highly consistent with human judgments.

STRENGTHS:
+ It makes one of the first attempts to study the evaluation of AIGC images, the contribution of the new dataset is significant and the pipeline to collect human perceptual data is reasonable.
+ Experiments are well organized, which manifests the shortages of current perceptual metrics in evaluating AIGC images and verifies the effectiveness of the proposed metric.
+ The feature inversion part is interesting, aligning well with the Analysis-by-Synthesis model evaluation methodology.

WEAKNESSES:
- The definition of distortion seems a bit confusing. In the literature, distortions are generally associated with degradation of image quality. In this work, distortions refer to mid-level changes, which do not necessarily lead to the loss of visual quality.
- It is unclear how the JND experiment is conducted. When the participants are expected to give an answer of yes? Two images without any noticeable pixel changes, or perceptually the same in terms of the mid-level change?
- More details of the feature inversion part are expected.

QUESTIONS:
Does the proposed metric perform well on classical image quality assessment datasets?

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors develop a perceptual metric that assesses  images similarity holistically. They first collect a new dataset of human similarity  judgments over image pairs that are alike in diverse ways. During the dataset construction, the authors utilized Stable Diffusion to create synthetic data, aiming to model mid-level similarity. Based on the dataset, the authors observe that popular perceptual metrics fall short of explaining the new data, so they introduce a new metric, DreamSim, tuned to better align with human perception. Finally, authors applied this metric in image retrieval and showed that it can generalize to real images.\n\nSTRENGTHS:\nHuman visual similarity is an important and practical topic in the domain of computer vision. It has many potential applications such as image retrieval. One of the major contributions of this paper is the new dataset (NIGHTS), which contains human similarity judgments of 20k synthetic image triplets. I believe this dataset will be useful for fellow researchers in the related domain. Another strength of this paper is the proposed similarity metric that captures how humans naturally perceive image similarity. I also appreciate the authors detailed analyses and comparison of the metric and its performance.\n\nWEAKNESSES:\nThough the paper is well written, and the topic is interesting and practical, the paper has several weaknesses. First of all, as the authors have mentioned, due to the potential biases and other limitations of Stable Diffusion, the authors deliberately avoided human faces during dataset construction. However, human faces are a common subject in general visual scenes and human perception of images containing human faces should be studied. Second, though the authors have applied the metric in image retrieval and used Figure 8 as a vivid proof to demonstrate that the proposed metric can be generalized to real images, I still have several concerns: first, is it possible to test the proposed metric on another benchmark dataset of real images, instead of the image retrieval task? Second, who are the users who rated the preference on the image retrieval results? More details of performance evaluation are needed. Third, not all metrics in Figure 4 are evaluated in the image retrieval task, why? Finally, while the authors avoided human faces in the proposed dataset, the models retrieved all images with human faces in the bottom right block in Fig 8. Does it show that the model focused on features other than human faces?\n\nQUESTIONS:\n1.\tWhy do the authors not compare all the metrics used in Figure 4 during the image retrieval task?\n2.\tWho are the users who rated the preference on the image retrieval results?\n3.\tCan the authors give a more comprehensive evaluation of the generalizability of the proposed dataset?\n\nLIMITATIONS:\nA more comprehensive evaluation of the generalizability of the dataset and metric is needed. Furthermore, humans are sensitive to faces from birth. Visual similarity on images with human faces is important.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper expands the fundamental task of measuring perceptual image similarity to encompass factors\n beyond low-level and high-level similarity. To achieve this, it leverages the recent advances in generative visual AI and generate synthetic images with text prompts. These synthetic images are expected to contain multi-faceted notions of similarity, which are then handed over to humans for A/B similarity test. The proposed model then learns from these human similarity ratings and is shown to uncover the emergent hidden factors defining visual similarity. In extensive evaluations, it is shown to surpass a number of baselines, including DINO and CLIP.\n\nSTRENGTHS:\n-- Research angle on visual similarity is timely, given its rather narrow scope of examinations in the past. Breakthrough on this angle can benefit vision community at large,.\n\n-- Using synthetic images for visual similarity learning is a promising technical path forward given our increasing controllability on generating target visual contents. Instead of harnessing human ratings on the similarity between any two real images and letting models to learn as a black-box, using synthetic images has the advantage of leaving the discretion to human designers on what constitutes visual similarity -- for example in this paper, although only superficially, manipulates on mid-level visual variations.\n\n-- Using LoRA to efficiently tune pre-trained visual ensembles towards a visual similarity judge is also reasonable.\n\n-- Extensive analysis is extensive.\n\nWEAKNESSES:\nThe paper is itself is quite intact by itself. So what I propose here is not necessarily a weakness, but rather with such a new tool on visual similarity, I keep wondering what can it, immediately, benefit vision tasks at my hand. The paper showcases a pilot study on the application of image retrieval, but in my understanding, the paper's contribution is best leveraged as a universal (differentiable) plug-n-play module to provide a complementary power to many computer vision tasks.\n\nFor example, it would be lovely to see how this paper opens new opportunities to image quality assessment community, where the intrinsic ambiguity of ground-truth human ratings is painstaking for generalisable modelling. Can DreamSim help alleviate this problem as a regularisation force by requiring similar visual contents to be of similar quality score? Same for the problem of sketch-based image retrieval community, where sketch-image ground-truth paired data is ill-defined. In the work of [A], they have shown how leveraging a visual similarity metric like LPIPS can greatly advance the understanding of the problem. Can DreamSim bring even further breakthrough?\n\n[A] Photo Pre-Training, But for Sketch. CVPR 2023\n\nQUESTIONS:\n- Would be great if the paper shows more practical use cases that benefit from such a new visual similarity judge. For example, as a plug-n-play module in diverse image retrieval tasks? Or as an essential component for better evaluating a generative visual agent?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages recent advances in synthetic image generation (namely text-to-image) to explore the space of image similarity assessment. They generate a novel dataset based on human forced choice input on pairs of synthetic images generated with respect to a reference image as well as interleaved visual memory tasks to assess when images are noticeably different on first assessment. The metric they learn over this novel dataset appears to better capture mid-level features such as pose and viewing angle.\n\nSTRENGTHS:\nThe paper is overall clearly written. The experiments are well-described and compelling, with multiple complementary facets supporting each other. The paper provides both a dataset and metric which may be of interest to the image similarity domain.\n\nWEAKNESSES:\nI apologize for a somewhat unclear criticism, but the JND task appears to be a visual working memory task, not a perception task. As we know from investigations of change blindness, there can be quite distinct differences between images that humans will only notice given sufficient time to explore the image, but if they happen to be attending the location that is different, they will immediately notice the difference. This seems to be getting at a different aspect of similarity than the perceptual choice task, which makes me want more of an explanation for why these two tasks are paired, or how they interrelate and why they are paired in this way.\n\nQUESTIONS:\n- How are the categories drawn from the labels? Is it a random sampling (and if so, are duplicate categories controlled for?), a subset (and if so, how was this selected?), or another arrangement?\n\n- Although it is mentioned that 7 votes is the average for retained triplets (L148), was there a minimum number required to be retained?\n\n- How do humans react to the image ablations tested in Section 5.2? It seems unclear to me whether the model should be impervious to some of the changes made.\n\nLIMITATIONS:\nThe limitations seem clearly articulated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}