SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stephanie  $\mathbf{F}\mathbf{u}^{*1}$
Netanel Y. Tamir*2
Shobhita Sundaram*1
Lucy Chai
Richard Zhang<sup>3</sup>
Tali Dekel
Phillip Isola
$^{1}$ MIT  $^{2}$ Weizmann Institute of Science  $^{3}$ Adobe Research
Figure 1: What makes two images look similar? We generate a new benchmark of synthetic image triplets that span a wide range of mid-level variations and gather human judgments, rating whether image A/B is more similar to the reference. Our benchmark spans various notions of similarity such as pose (top-left), perspective (top-mid), foreground color (mid-left), number of items (mid-right), and object shape (bottom-left). This allows us to learn a new metric (DreamSim) that better coincides with human judgments w.r.t. existing similarity metrics (LPIPS) or embedding-based metrics extracted from recent large vision models (DINO & CLIP).
Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.
Our project page: https://dreamsim-nights.github.io/
* Equal contribution, corresponding authors. Order decided by random seed.
"A sense of sameness is the very keel and backbone of our thinking" - William James, 1890
Our understanding of the visual world hinges crucially on our ability to perceive the similarities between different images. Moreover, humans can reason about many notions of similarity, ranging from low-level perceptual properties such as color and texture, to higher-level concepts such as an object's category or a scene's emotional valence [76]. This capacity to conduct meaningful visual comparisons underlies our ability to effortlessly transfer our knowledge to new environments, e.g., recognizing unseen or unfamiliar objects based on their relatedness to familiar ones [42, 70].
Computer vision has tried to capture this sense of similarity with low-level metrics like PSNR and SSIM [88], as well as learned perceptual metrics such as LPIPS [94] and DISTS [24]. Despite their utility, these metrics are limited in that they focus on the pixel or patch level and fail to capture higher-level structures. These limitations have motivated researchers to reach for image-level embeddings from large vision models such as DINO or CLIP to measure image-to-image distances in a large variety of applications [12, 40, 41, 44, 62]. Recent studies have shown that these embeddings do well at capturing certain high-level similarity judgments, in particular, predicting which semantic categories will be considered alike by humans [59]. It remains unclear, however, how well these models align with human perception of richer and more fine-grained visual structure.
In this paper, we introduce a new perceptual metric, which bridges the gap between lower-level patch-based metrics and broad categorical comparisons. We collect a new dataset named NIGHTS – Novel Image Generations with Human-Tested Similarity – containing human similarity judgments over image triplets. Each triplet consists of a reference image and two perturbed versions, along with human judgments as to which version is most similar to the reference (Fig. 1). We use iterative filtering together with recent diffusion models to collect our dataset, which is designed to capture image sets that are cognitively impenetrable (i.e. result in consistent decisions across different individuals) yet showcase rich variations. For example, our data contains images of similar object appearance, viewing angles, camera poses, overall layout, etc. This dataset differs qualitatively from prior low-level datasets [94], which focused on perturbations like blurring and adding noise, and from previous high-level datasets [34, 35], which showed variation just at the level of categories (e.g. “is an image of a kitchen more like an image of a giraffe or an image of a beach”).
On the mid-level similarity task presented by our data, we find that features from recent large pre-trained vision models [10, 39, 66] outperform the current set of standard perceptual metrics [24, 59, 94]. We further show that these large vision models can be tuned on our data to be substantially more human-aligned. Our resulting metric, DreamSim, can be dropped into existing pipelines and demonstrates high agreement with human visual perception in both quantitative assessments and qualitative comparisons using out-of-domain real images (e.g., image retrieval, image synthesis). We also analyze which features our metric is most sensitive to and find that, compared to previous perceptual metrics, it focuses relatively heavily on foreground objects, while compared to modern image embeddings, it does not neglect color and layout.
In summary, our contributions are the following:
- A new image similarity dataset, consisting of 20k synthetic image triplets designed to be cognitively impenetrable with human judgments as labels.
- A tuned metric that captures how humans naturally perceive image similarity, achieving  $96.16\%$  accuracy in predicting human judgments on our dataset.
- Analysis of the image properties that affect our model's decisions.
- Demonstration of downstream applications to image retrieval and synthesis.
Perceptual similarity. Classical metrics such as Manhattan  $\ell_1$ , Euclidean  $\ell_2$ , MSE, and PSNR use point-wise difference to measure similarity, thus failing to capture important joint image statistics. Patch-based metrics, including SSIM [88], FSIM [92], and HDR-VDP-2 [55] tackle this issue and are widely used in applications involving photometric distortions such as image quality assessment. However, they do not capture the nuances of human vision when more structural ambiguity is present [73] and are not suited for more complex image generation tasks.
With the deep learning revolution, classical metrics have been replaced by learning-based metrics [26, 29, 43]. These metrics are defined in the space of deep features extracted from pre-trained networks, such as VGG [78] or AlexNet [47]. Amir and Weiss [4] demonstrate that even untrained networks can be adapted as perceptual metrics. Zhang et al. [94] observe that feature-based metrics outperform classical metrics across different convolutional architectures and learning paradigms, suggesting that perceptual similarity is an emergent property in deep representations. Further tuning on the perceptual data yields improvements, such as in LPIPS [94], PIE-APP [65], or DPAM in the audio domain [53, 54]. Further improvements include ensembling for robustness [45], antialIASing for stability [24, 36, 93], and global descriptors for texture [24]. Muttenthaler et al.[59] provide insight into high-level human similarity by training on a subset of the THINGS [35] dataset, focusing on concept similarity and omitting visual cues for images within a category.
While strong computer vision features make for strong perceptual metrics, counterintuitively, they eventually become decorrelated with perceptual similarity [20, 48]. Today, the predominantly-used perceptual metric is LPIPS, operating on  $64 \times 64$  patches.
Recent foundation models as metrics. Foundation models provide strong pre-trained backbones for a variety of downstream tasks. These models primarily leverage the Vision Transformer (ViT) [25] architecture and are trained through self-supervised learning [16, 32, 81]. CLIP [66] learns to map images and text captions into a shared embedding space, proving useful for many (often zero-shot) tasks [41, 67, 90]. CLIP has been employed as a perceptual metric to train models for semantic consistency [12, 87]. Another self-supervised ViT-based model, DINO [10], extracts disentangled appearance and structure descriptors that can be employed in image generation pipelines. Amir et al.[5] show that DINO encodes valuable semantic information about object parts. Our work aims to systematically evaluate such representations for perceptual similarity, also including OpenCLIP (an open-source implementation of CLIP) [39] and pre-trained masked autoencoders (MAE) [31].
Perceptual tests. The two alternative forced choice (2AFC) test has historically been used by behavioral psychologists to study decision-making [51, 56]. Humans judge the similarity between two images by choosing to consider certain dimensions of similarity more than others [85]. Gathering judgments on ambiguous sets of images can be cognitively penetrable, calling upon a subject's cognitive processes rather than a more automatic, "wired-in" sense that is stable across humans and over time [11, 19, 79]. Previous studies have raised concerns about cognitive penetrability [57, 94]. On the other hand, as a psychophysical measure, just noticeable difference experiments (JND) are thought to be independent of subjective biases [3]. We follow best practices [94] and collect judgments on both of these complementary perceptual tests.
Synthetic data. GANs [30] have been used widely for dataset generation on tasks such as visual alignment [63], face manipulation [86], and adversarial training for image synthesis [77]. In recent years, text-driven generative models (e.g., Stable Diffusion [71],Imagen [72], DALLE-2 [68], MUSE [13]) have emerged as powerful tools for image synthesis. They have also been used to generate training data for a variety of downstream tasks [6, 9, 33, 75].
3 Perceptual dataset collection
While previous datasets focus on low-level, patch-based distortions [2, 94] or high-level, categorical [35] changes, we aim to close the gap, capturing distortions including mid-level variations. We aim to produce images with an underlying semantic commonality, but variations in a diversity of factors, such as style, color, pose, and other details, so that a human can assess their visual relationship. In Section 3.1, we describe our data generation pipeline – we prompt Stable Diffusion for related images of a given category, leveraging its natural image prior for variations within the category. We then describe our mechanism for collecting cognitively impenetrable perceptual judgments in Section 3.2.
3.1 Generating images with varied distortions
We leverage Stable Diffusion v1.4 [71], which generates diverse and high-quality images that adhere to a given text prompt. We sample images with a prompt of the same category, using the structure "An image of a <category>". The <category> is drawn from image labels in popular datasets: ImageNet [21], CIFAR-10 [46], CIFAR-100 [46], Oxford 102 Flower [60], Food-101 [8],
2AFC Data Collection
JND Data Collection
Figure 2: Overview of 2AFC and JND data collection. (Left) We collect up to 10 similarity judgments for each triplet in our dataset, and filter for unanimous examples. (Right) We validate our 2AFC results with a JND study. We flash two interleaved image pairs, and ask if the first & third, and second & fourth were the same. Interleaving the pairs ensures that users decide solely based on their initial reaction to each image from a standardized (500ms) viewing time.
Dataset	Perturbation Type	Data Source	Input type	# Images / Patches	# Samples	# Judge per sample	Judgment type	Examples
BAPPS [94]	low-level (traditional & CNN-based)	MIT-Adobe5k, RAISE1k	64×64 patches	563.1k	187.7k	2.6	2AFC
19.2k	9.6k	3.0	JND
THINGS [34]	high-level conceptual	Google, Bing, Ebay, Flickr scrape	Images	1.8k	4.7M	1	Odd-one-out
NIGHTS (Ours)	low & mid-level synthetic	Diffusion-synthesized	Images	60.0k	20.0k	7.1	2AFC
411	137	3.0	JND
Table 1: Perceptual dataset comparison. While previous work targets low-level and high-level variations, we leverage synthetic data to generate diverse perturbations. In addition, while previous data gathers fewer judgments per sample, we filter for unanimous judgments, curating cleaner samples.
and SUN397 [91]. While the ImageNet labels make up  $\sim 55\%$  of the categories, the superset of categories across these datasets encompasses a wide range of objects and scenes.
As illustrated in Figure 1, the resulting images display mid-level variations, such as pose, perspective, and shape, falling in-between pixel/patch-based distortions and categorical differences. Additional triplets and the full list of categories are in the Supplementary Materials (SM). We generate an initial set of 100,000 image triplets, which are later filtered during the labeling process. Each triplet consists of a reference image and two "distortions", generated from the same prompt.
3.2 Human perceptual judgments
We strive to gather perceptual judgments that are automatic (requiring little to no cognition), stable (invariant to changes in mental representation), and shared across humans, which requires a cognitively impenetrable judging process. Our main dataset consists of two alternative forced choice (2AFC) judgments on a triplet. Additionally, we collect just noticeable difference (JND) judgments on image pairs, corroborating our findings. Table 1 shows an overview of our dataset, compared to prior perceptual datasets.
Two alternative forced choice (2AFC). Figure 2 (left) illustrates our data gathering setup. We collect judgments on the commonly-used Amazon Mechanical Turk (AMT) platform [12, 83, 94]. We show participants triplets of images  $(x, \tilde{x}_0, \tilde{x}_1)$  and ask whether  $\tilde{x}_0$  or  $\tilde{x}_1$  ("distortions" of  $x$ ) is more similar to the reference image  $x$ . We collect judgments  $y \in \{0, 1\}$  denoting the more perceptually similar distortion. This choice does not necessarily need to be easily articulated but should be cognitively impenetrable, that is, instinctive and consistent across humans.
Additionally, we use sentinels to ensure high-quality responses. Sentinels are designed as  $(x,x,v)$  where one "distortion" is the image itself and the other is from a completely different prompt. Approximately  $20\%$  of participants failed at least one sentinel, and we discard all their responses.
To collect cognitively impenetrable triplets, we design our 2AFC experiments to ensure that each triplet included in the dataset obtains a unanimous vote for either  $\tilde{x}_0$  or  $\tilde{x}_1$ . We divide our experiments into 10 rounds, starting with 100,000 triplets, advancing instances that maintain unanimous votes. Triplets retained through all 10 rounds can earn a maximum of 10 votes, but may have less as a result of the aforementioned sentinels. To maintain dataset quality, we only include triplets with  $\geq 6$  unanimous judgments in the final dataset.
Ultimately, our 2AFC dataset  $\mathcal{D}^{2\mathrm{afc}} = \{(x,\tilde{x}_0,\tilde{x}_1),y\}$  consists of 20,019 triplets with an average of 7 unanimous votes each. We partition our resulting dataset into train, validation, and test components with a random  $80 / 10 / 10$  split. Our dataset is publicly available on our project page.
Just noticeable differences (JND). JND aims to characterize the boundary when a distortion becomes just noticeable. Below a threshold, a small perturbation (e.g., a 1-pixel shift) appears identical to a human observer. This perceptual test provides a complementary signal for perceptual similarity and allows us to exploit uniform timing and the presence of a correct answer.
Intuitively, an image pair that falls below the JND threshold should be more likely to be selected as similar in the 2AFC test; thus a high correlation between 2AFC and JND scores indicates that our judgments are reliable under multiple experimental settings. To assess how well our perceptual tests agree, we collect judgments on the triplets from the 2AFC test set, filtering for triplets that "straddle" the JND threshold. Specifically, given triplet  $(x,\tilde{x}_0,\tilde{x}_1)\in \mathcal{D}^{2\mathrm{afc}}$ , we independently collect JND judgments on pairs  $(x,\tilde{x}_0)$  and  $(x,\tilde{x}_1)$ , keeping triplets where humans find one and only one to be identical. In total, our JND dataset contains 411 triplets  $\mathcal{D}^{\mathrm{jnd}} = \{(x,\tilde{x}_0,\tilde{x}_1),s\}$ , where users chose one of  $s\in \{0,1\}$  to be identical.
We show our JND procedure in Figure 2. We interleave two images and their respective distortions into sequence  $x \to v \to \tilde{x} \to \tilde{v}$ , and ask whether the images in each pair were identical. This exploits masking, a perceptual phenomenon that occurs when an image is obscured by another. By masking with a time gap, we standardize viewing time across users and prompt for a decision based on an initial reaction and memory of the image. For each user, we show 48 distorted pairs, along with 24 identical pairs, to balance the expected responses. We collect 3 judgments per pair, with no person seeing a repeat, and take the majority vote as the label. Users mark  $20.4\%$  of distorted images being identical, indicating that some of our distortions indeed fall below the noticeable threshold.
4 Perceptual metric learning
We next evaluate how well pretrained embeddings and learned metrics align with our data, and we investigate if alignment can be improved by fine-tuning.
4.1 Embeddings as a distance metric
We denote a distance between two images as  $D(\cdot, \cdot; f_{\theta})$ , where  $f_{\theta}$  is a feature extractor. We evaluate a variety of state-of-the-art candidate embeddings and metrics. LPIPS [94] and DISTS [24] use CNN backbones, whereas DINO [10],
CLIP [66], OpenCLIP [17], and MAE [31] use transformer-based backbones [25]. Following standard practice, distance  $D(x, \tilde{x}; f_{\theta}) = 1 - \cos(f_{\theta}(x), f_{\theta}(\tilde{x}))$  is taken as the cosine distance between the CLS tokens taken from the last layer for DINO and MAE (before and after the layer normalization, respectively), and the embedding vector for CLIP and OpenCLIP. For LPIPS and DISTS,  $D(x, \tilde{x}; f_{\theta})$  is simply the distance metric itself. Given a triplet  $(x, \tilde{x}_0, \tilde{x}_1)$ , and a feature extractor  $f_{\theta}$ , the model vote is calculated as:
$$
\hat {y} = \left\{ \begin{array}{l l} 1, & d _ {1} <   d _ {0} \\ 0, & d _ {0} <   d _ {1} \end{array} , \text {w h e r e} d _ {0} = D \left(x, \tilde {x} _ {0}; f _ {\theta}\right) \text {a n d} d _ {1} = D \left(x, \tilde {x} _ {1}; f _ {\theta}\right). \right. \tag {1}
$$
Figure 3: Training overview. Our perceptual metric is an ensemble of backbones, concatenating the representations together and fine-tuning with LoRA. We evaluate how well cosine distance on candidate feature spaces aligns with perceptual judgments  $y$ .
Figure 4: Metrics performance on our benchmark. Large vision models OpenCLIP and DINO outperform prior learned metrics LPIPS and DISTS (orange) (chance = 50%). Further tuning on our perceptual data with an MLP (blue) improves performance over out-of-the-box features (green); we find Low-rank LoRA [38] tuning boosts performance significantly (pink). Ensembling CLIP, OpenCLIP, and DINO models together improves performance as well. Our final model, DreamSim, combines these insights and achieves high agreement with humans (96.16%). Error bars represent a 95% confidence interval.
Figure 5: Correlation between 2AFC & JND. We plot the agreement between human judgments and existing similarity metrics on two tasks, 2AFC and JND. The strong correlation between the metrics' agreement with both tasks suggests that our dataset captures a general notion of similarity that can be replicated in two separate, independent human studies.
Evaluating human-metric agreement. Recall that for a triplet, we collect  $y$ , indicating which image a human selects as more similar. We evaluate how often each metric agrees with human judges as  $\mathsf{Score}_{2\mathsf{AFC}}(f_{\theta}) = \mathbb{E}_{\mathcal{D}^{2\mathsf{a}\mathsf{c}}}[\mathbb{1}_{\hat{y} = y}]$  and  $\mathsf{Score}_{\mathsf{JND}} = \mathbb{E}_{\mathcal{D}^{\mathsf{j}\mathsf{nd}}}[\mathbb{1}_{\hat{y} = s}]$ .
4.2 Learning an improved metric
Objective. To better align a perceptual metric with human judgments, given triplet  $(x,\tilde{x}_0,\tilde{x}_1)$  we maximize the difference between the perceptual distances  $d_0,d_1$ , with smaller distance associated with the distortion  $y$  humans voted to be most similar. To accomplish this, we map  $y\in \{0,1\} \to \bar{y}\in \{-1,1\}$  and use a hinge loss, equivalent to triplet loss [14] between the embeddings, with a margin of  $m = 0.05$ :
$$
\mathcal {L} (y, \hat {y}) = \max  (0, m - \Delta d \cdot \bar {y}), \text {w h e r e} \Delta d = d _ {0} - d _ {1}. \tag {2}
$$
Tuning. Next, we investigate the best method to tune or modify feature extractor  $f_{\theta}$ . A challenge is the large number of parameters (billions, for ViT backbones), compared to the relatively small amount of perceptual data. Inspired by [94], we first try tuning via an MLP head, adding a 1-hidden-layer MLP with a residual connection on top of the pre-trained embeddings. We compare this to fine-tuning through the pre-trained backbones using Low-Rank Adaptation (LoRA) [38], which we find to achieve better results than full fine-tuning. Using LoRA ( $r = 16$ , dropout  $p = 0.3$ ,  $\alpha = 0.5$ ) we tune approximately  $0.67\%$  of each model's parameters. In all experiments, LoRA significantly outperformed using the MLP head.
Feature concatenation. As multiple models may provide complementary features, we try combining them to boost performance. We concatenate features from the best-performing models on our dataset (DINO [10], CLIP [66], and OpenCLIP [17]) into one ensemble metric. By training just a handful of parameters in this ensemble (through LoRA [38]), we gain the benefits of each large model's embedding space without sacrificing much computational load while fine-tuning.
5.1 How well do existing metrics align with human judgments?
In Figure 4, we show how well metrics align with humans on our dataset, across low-level metrics (PSNR, SSIM), prior learned metrics (LPIPS, DISTS) and recent large vision model embeddings
Figure 6: Sensitivity of our similarity metric to different image ablations. (Left) We ablate elements of the image, assessing performance dropoff by changing the orientation, color, shading, structure, and foreground vs. background content. The model is fairly robust to orientation, and more impacted by changes in color, shading, or structure. Foreground content is more important for similarity recognition than background content. (Right) For an example triplet (unmodified), we show visualizations of removing elements of the images.