When does perceptual alignment benefit vision representations?
Authors: Shobhita Sundaram
Stephanie Fu
Lukas Muttenthaler
Netanel Yakir Tamir
Lucy Chai
Simon Kornblith
Trevor Darrell
Phillip Isola
Venue: NeurIPS 2024 poster (2024)
Keywords: representation learning
alignment
perception
transfer learning
computer vision
foundation model
Primary area: machine_vision
TL;DR: We find that aligning the representations of large vision models to human perceptual judgements improves downstream performance on a variety of tasks.

Abstract: Humans judge perceptual similarity according to diverse visual attributes, including scene layout, subject location, and camera pose. Existing vision models understand a wide range of semantic abstractions but improperly weigh these attributes and thus make inferences misaligned with human perception. 
While vision representations have previously benefited from human preference alignment in contexts like image generation, the utility of perceptually aligned representations in more general-purpose settings remains unclear. Here, we investigate how aligning vision model representations to human perceptual judgments impacts their usability in standard computer vision tasks. We finetune state-of-the-art models on a dataset of human similarity judgments for synthetic image triplets and evaluate them across diverse computer vision tasks. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks, including counting, semantic segmentation, depth estimation, instance retrieval, and retrieval-augmented generation. In addition, we find that performance is widely preserved on other tasks, including specialized out-of-distribution domains such as in medical imaging and 3D environment frames. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can make them better representation learners.

When Does Perceptual Alignment Benefit Vision Representations?
Shobhita Sundaram
Stephanie Fu
Lukas Muttenthaler
Netanel Y. Tamir
Lucy Chail
Simon Kornblith
Trevor Darrell2
Phillip Isola'
MIT
U.C. Berkeley
TTU Berlin
BIFOLD
Weizmann Institute of Science
Anthropic
Segmentation
Depth Estimation
Pretrained vision model
Counting
Fine-tune on perceptual similarity judgments
RAG
VLM
Figure 1: Does human perceptual alignment improve vision representations? Vision models have been shown to learn useful image representations through large-scale pretraining (c.g., CLIP, DINO). We find that additionally aligning these models to human perceptual judgments yields representations that improve upon the original backbones across many downstream tasks, including counting, segmentation, depth estimation, instance retrieval, retrieval-ane -augmetedd generation, while degrading performance in natural classification tasks. Our blog post and code are available at peppppreaaaaaa github.io
Abstract
Humans judge perceptual similarity according to diverse visual attributes, in- cluding scene layout, subject location, and camera pose. Existing vision models understand a wide range of semantic abstractions but improperly weigh these at- tributes and thus make inferences misaligned with human perception. While vision representations have previously benefited from alignment in contexts like image generation, utility of perceptually aligned representations general-purposs settings remains unclear. Here, investigate how aligning vision representations to human perceptual judgments impacts their usability across diverse vision tasks. We finetune of-the--arrrrr models on human similarity judgments for image triplets and evaluate them across standard benchmarks. We find that perceptual alignment yields representations that improve upon the original backbones across many tasks, including counting, segmentation, depth estimation, instance retrieval, and retrieval-augmenter generation, while deteriorating performance on natural classification. Performance is widely preserved on other tasks, including special- ized distiiibbittiiiinnnnn domains such in medical imaging and 3D environment frames. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
Equal contribution.
Work partly done while Student Researcher Google DeepMind.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Our sense of similarity is crucial to how we perceive and act in the world. Consider the range of factors that might influence your judgment of visual similarity: layout, color, perspective, semantics, and more. These characteristics shape our local inferences about object relationships, enabling us to build comprehensive understanding of the visual world.
Given the importance of similarity judgments in our visual perception, it follows that aligning vision models to these judgments could help them develop more human-like visual capabilities. In the language domain, we have seen the power of aligning LLMs to human feedback [RLHF; 2 resulting in safer and more useful models 3 Similar trends are emerging in vision, where there is growing interest in alignment with human perceptual judgments to improve vision representations [e.g..
61
42 18 43). Human feedback has also been used to improve the aesthetic quality of diffusion- generated images [58] 16, 30). increase retrieval capabilities [18], and improve downstream task performance of image/text models 42 While acknowledging this wide space of human preference data used for alignment in vision and language, we focus our study on image similarity judgments, which give more stable and shared measure of human visual perception [8, 54 43).
Although there consensus that alignment to perceptual judgements can enable specific goals (e.g.. similarity [18 61 414 their utility in creating
representations is less clear. Does human alignment improve model performance by leveraging buman-providedd labels as an inductive bias, or compromise the model's original representational power by diverting it towards a separate task? Recent work suggests that the conclusions are nuanced: naively incorporating human percep- tual knowledge into model finetuming can distort representations, requiring strong regularization to maintain downstream performance while increasing the representations' interpretability and align- ment 42). Furthermore, objective function and training data appear to matter more than architecture or model size for aligning to perceptual judgments [41). The nature of human preference labels also plays a crucial role, different types of labels produce distinet learning signals. For instance, fine-tuning vision backbones with mid-level perceptual similarity judgments yields representations well-suited for image retrieval tasks [8]. while low-level similarity labels are more effective for image reconstruction losses [61].
It is clear from this body of work that careful perceptual alignment helps with specific tasks, however the effect of these adjustments on the models' representation spaces is less well understood. That is, models aligned to perceptual judgments only better at specific tasks such predicting image similarity - or do they, like humans, actually have better general-purpos Here, we evaluate the usefulness of human-aligned representations not just in predicting perceptual judgments but also on standard vision benchmarks requiring diverse notions of visual understanding. We finetune several trate-of-the-ar art models- including CLIP, DINO, DINOv2, and SynCLR on NIGHTS, a dataset of human similarity judgments over synthetic image triplets [8], and evaluate the finetuned models on standard tasks. Our experiments suggest that these numan-aligned representations demonstrate strong gains compared to the original model backbones, even on tasks requiring skills beyond those needed during training or perceptual alignment. We also identify limitations of human perceptual alignment, and find that finetuning can sacrifice performance some natural data tasks in which models had strong prior performance. In summary, our contributions are the following:
representations?
We investigate the effects of aligning pretrained vision models to human perceptual judg- ments on various downstream visual recognition tasks.
We find that propagating global image-level human similarity annotations to ViT patch tokens benefits downstream dense prediction tasks such depth prediction and segmentation.
We show that human-aligned representations also beneficial in retrieval-based tasks requiring global image understanding, including retrieval -uugeetne generation for recent lisionllagguae models, counting counting-based retrieval, and instance-busee retrieval.
We ablate the effect of various mid-level image similarity, as opposed to low-level pixel-level variations or high-level semantic associations, offers the largest improvements in generalization capabilities.
visual similarity datasets, and find that
2 Related Works
Vision backhones as learned feature extractors. Using the intermediate activations of deep net- works has been a long- long-standing strategy, originally used as a way of harnessing data-driven priors

---

from models trained on fewer large datasets and repurposing them for downstream tasks where such expensive supervision is less readily available [2 50 For instance, ImageNet pretrained models
have proven useful for tasks such as texture synthesis 19 style-transfer [20], and super-resolution [27]. But given the challenges of collecting large-scale labelled datasets using human annotators, other approaches such as self-superviser learning [6] 7 59 4 44 and contrastive learning on im- age/alt text pairs [46, 8 (26] have since superseded supervised learning as the standard approach for training vision models. Such models have been shown to yield rich multipurpose representations that can generalize to a variety of tasks involving visual perception, scene understanding, and image
generation [51 38
52
31 39 557.
View selection for self-supervised learning. The supervisdd, contrastiveleeaning objective aims to maximize the feature similarity between similar views of the data and minimize the similarity between different (negative) views. While originally the negative views are selected randomly from a pool of images [6] [2]. this often results in requiring large batch sizes to learn useful representations. The process of selecting these positive and negative learning examples remains an active research area. Recent approaches have suggested that alternative strategies including hard-negative mining [47), nearest neighbors for positive pairs [1], and supervised labels when available (28) can provide useful learning signals. Other avenues include training the contrastive framework synthetic data from generative models (25]55), (5). which provides an infinite data source of image variations rather than the traditional data augmentations typically used to generate alternative views.
Learning directly from human feedback can provide models with more targeted supervision using fewer examples 99. and, thus, has been beneficial for fine-tuning Ding et al. [1] show that image similarity metrics, trained on human feedback, can be subsequently used for evaluating the diversity of tex--to-image model. Several datasets aim to annotate human visual preferences in image similarity, including low-level [61] and high-level [23] image variations. In particular, Zhang et al. [6]] and Fu et al. [8] use these datasets to learn a more human-aligned perceptual metric that improves the image retrieval abilities of vision models, while Muttenthaler et al. 42 used THINGS for learning a linear transform on top of vision representations to increase downstream task performance and improve alignment with human similarity judgments. These annotated human preference datasets serve as useful signals for contrastive objectives, providing direct supervision for positive and negative pairings that are aligned with human decisions; here, we investigate how fine-tuning self-supervised models using these human annotations impacts model performance on various downstream tasks.
Learning with human alignment.
large models towards specific human preferences (33 62)
3 Learning from perceptual judgments
We propose to use the method described below as a "second pretraining stage", which aligns the feature representations from large vision models with human perceptual judgments before applying them to downstream tasks. We note that prior work on this dataset aimed to develop a model for measuring image similarity based on human judgments. Here, we investigate if pretraining on this dataset leads to a better general-purpose representation, as measured by performance on different downstream tasks.
3.1 Human Similarity Annotations
We use the NIGHTS dataset to produce human-aligned variations of several large vision models [18] The NIGHTS dataset consists of 20k synthetically generated image triplets, annotated with two alternative forced-choice human similarity judgments. These triplets are collected so that each has 6-10 unanimous human ratings, thus eliminating ambiguous cases where humans are likely to disagree.
NIGHTS consists of image triplets varying in mid-level information. Images in triplet roughly share the same semantic content; however, they vary in pose, layout, shape, color, and the number of objects (see Fig. 1.5 in the Appendix for examples). Thus, the perceptual judgments indicate the shared visual appearance properties, as opposed to requiring higher-level semantic knowledge about the image content.
3.2 Image-level objective
Given pre- pre-trained backbone fe. we fine-tune its parameters 0 on dataset of triplets D {(x, zo. d1), y), where I denotes a reference image, and E. and 2 denote two variation images. The judgement y (0, indicates which of do and oi is more similar to r. We measure distance
3

---

(dissimilarity) between two images (a, Æb) using the cosine distance between their respective image features fo( which is defined as:
fo(x) fo(
d(z, σo) = 1
[oo
We use an alignment loss to encourage the model to match human preferences:
Lalignment
max(0, Ad.9)
where Ad= d(z, _0) dzz, 1.). y maps y E {0.1] {-1, 1}. and is the margin, which we set
to 0.05 following [8]. Note that this loss is equivalent to the triplet loss [ and thus minimizes the cosine distance between the representations of the more similar pair and maximizes the distance
between the representations of the other pair.
3.3 Patch-level objective
CLS
svgparr
patch enbeds
Figure 2: Diagram of our feature extraction method when training with patch-level objective. Left: We extract the CLS and patch embeddings from DINO and DINOv2, perform spatial average-pool on the patch embeddings, and concatenate [CLS, patch] vectors. Right: We train these concatenated features with hinge loss, identical to the image-level objective.
The NIGHTS dataset contains global annotations of similarity at the image level, but the holistic label is the result of several local attributes, such perspective, layout, foreground appearance etc. In addition the global CT.S token from the Vision Transformer model backbones, each model also contains a set of spatial patch embeddings. Propagating this global human annotation to individual patch tokens allows for spatial representations that are aligned with human similarity preferences. Thus, we formulate patch alignment objective to optimize these local patch features jointly with the global image label.
The local objective only differs from the global objective in how the features are extracted. Instead of computing & (CLSA, CLSB), we compute
L (cat(CLLS,, (PATCHA))))]] cat[CLSp, CAO.
CLS is of dimension (1, d) and PATCH is (s, d) where is the number of patches along each spatial dimension. We spatially average the patch tokens to get dimension (1, d). We then concatenate the CLS and pooled patch tokens to get dimension (1, 2d). We fine-tune the same alignment loss (see Eq. 2 on concatenated CLS and seeraged-pooles patch tokens, and train heads for semantic segmentation and depth estimation on the resulting patch embeddings. Note that only experiments reported Sec. 4.1 use this objective, as they require local features. For all other evaluations, we exclusively use CLS tokens (see Eq. 2 if not mentioned otherwise.
3.4 Implementation details
Vision Model Backbones. We fine-tune several state-of-the-art Vision Transformer [VIT: [3 backbones including DINO [4, DINOv2 [44) CLIP [46), OpenCLIP (8), and SynCLR [55]. For DINO, DINOv2, MAE, and SynCLR, we use the CLS token the final layer. We extract the CLS token before layer norm for MAE and after it for the other backbones. For CLIP and OpenCLIP, we use the representations of the image encoder. We also experiment with concatenated features from DINO, CLIP, and OpenCLIP, used to train the DreamSim Ensemble in [8 (referred as "ensemble" in our results). For each model, we use its base size, ViT-B.
We implicitly ablate the usage of synthetic triplets in NIGHTS by including SynCLR in our exper- iments; as that backbone was contrastively trained on generated images, additional performance changes can be attributed to human perceptual alignment.

---

Finetuning with human preference labels. We finetune each backbone using Low-Rank Adaptation (LoRA), which was found to achieve better alignment performance and efficiency than full fine-tuning in [8] For more training and technical details see Sections C. and C.6 in the Appendix.
4 Experiments
In this section, we evaluate perceptually-aligned backbones against base models on common vision tasks. We study global representations through instance retrieval, object-counting, and retrieval- augmented generation experiments. Additionally, we find that local patch-level representations can be improved by tuning on image-level perceptual judgments, and show performance increases on semantic segmentation and depth estimation.
4.1 Dense Prediction
Semantic segmentation. Following the procedure detailed in Section 3.3 and Fig. 2 we LoRA- tune new backbones with perceptually-alignee CLS and patch tokens. To evaluate segmentation performance, we freeze these backbones and train a single linear layer transforming patch tokens to segmentation map. We evaluate DINO and DINOv2 standard segmentation benchmarks in Table 1 and show that numan-aligned models boost performance in 16 out of 20 cases. Across all datasets and metrics, human-aligned DINO (denoted as DINO-HA) outperforms the base model and often achieves the highest mloU and Pixel Accuracy (P.A.) overall. DINOv2-HA also outperforms its nonaligned counterpart COCO and DAVIS2017.
We flag datasets already seen in the DINOv2 retrieval pretraining 44 such as Paseal VOC, ADE20k, and Cityscapes. If a dataset is already in-distribution for backbone, fine-tuning different data may be more likely to change the feature space such that that dataset is more out-of-distribution. Thus this is potential confounding factor.
Pascal VOC
ADE20K
Cityscapes
COCO
DAVIS2017
mloU
P.A.
mIoU
P.A.
mloU
P.A.
mloU
P.A.
mloU
P.A.
DINO
0.729
0.800
0.342
0.548
0.562
0.749
0.810
0.901
0.809
0.886
DINO-HA
0.745
0.840
0.391
0.602
0.573
0.7691
0.816
0.914
0.818
0.913
DINOv2
0.686
0.803
0.441
0.645
0.535
0.737
0.759
0.877
0.794
0.906
DINOv2-HA
0.635
0.771
0.418
0.700t
0.528
0.739
0.761
0.891
0.810
0.908
Table I: Base and aligned model performance on semantic segmentation. Aligned models largely outperform baselines, with DINO-HA achieving the highest performance across models for 4 out of datasets. Note that Pascal VOC, ADE20K, and Cityscapes were included in DINOv2's retrieval pretraining. indicates best score in the column.
and train a single linear layer on frozen patch tokens to output depth map with values mapped into 256 mifformly-distribun bins. This head is trained with a scale-invariant log loss introduced in (I5) and a inverainva maddeettmmtthhnng term described in [36]. In Table estimation and show that numan-aligned models outperform base models in 27 out of 36 cases. Consistent with segmentation performance, human-aligned DINO outperforms the base model on all metrics across all datasets, and is often the gighest-performinn model overall (denoted by t). We also evaluate -uttt--diittibttion generalization by training depth head on NYUv2 and evaluating on the 4D Light Field dataset; combined with the performance boost even on datasets that DINOv2 was trained on (NYUv2 and SUN-RGBD), these results demonstrate that human-aligned models have strong generalization capabilities prior to any training for downstream tasks.
Depth estimation. We follow the evaluation protocol of 44
report performance
monocular depth
4.2
Retrieval-augmentee generation
First introduced for text generation with non- parametric memory +. eetrieval-augmente generation (RAG) has become a popular method for selecting relevant few-shot examples when prompting large vision-language models (VLMs) 2 33.. RAG evaluations go beyond conventional retrieval benchmarks on top-k recall, offering a more informative indicator of downstream large-model performance and utility the multimodal domain. We evaluate OpenFlamingo 2 few-shot classification accuracy by using vision backbone to retrieve a query image's ' 3 nearest neighbors and prepending the query image with those examples along with their class labels. Following OpenFlamingo's image classification evaluation framework [2], we extract the model's logits per
5

---

NYUv2
RMSE ())
AbsRel ())
log10 (.)
8 1.25 (1)
8
1.25 (1)
6 1.25 (1)
DINO
1.034
3.517
0.173
0,415
0.746
0.895
DINO-HA
1.032
3.759
0.1691
0.4451
0.761
0,900
DINOv2
1.003'
3.188
0.178
0.4451
0.7851
0.907!
DINOv2-HA
1.062
3.1671
0.185
0,419
0.749
0.891
NYUv2
4D LF
RMSE ())
AbsRel ())
log (.)
8 1.25 (1)
8
1.25 (t)
6 1.25 (f)
DINO
3.817
0.543
0.380
0.160
0.335
0.454
DINO-HA
3.757
0.538
0.362
0.187
0.346
0.463
DINOv2
3.460
0.497
0.3371
0.194
0.337
0.464
DINOv2-HA
3,720
0.4871
0.356
0.203
0.3561
0.455
SUN-RGBD
RMSE (L)
AbsRel ())
log10 (c)
1.25
6 1.25
6 1,25 (f)
4.900
2.350
0,205
0.524
DINO-HA
4.788|
2.300
0.526
0.230
0.418!
0.531
DINOv2
5.200
3.023
0.615
0.173
0.309
0.444
DINOv2-HA
5.082
2.904
0.599
0.237!
0.409
0.487
Table 2: Human-aligned DINO and DINOv2 performance on monocular depth estimation benchmarks. Note that NYUv2 and SUN- BUN-RGBD were included in DINOv2's retrieval pretraining set, yet human-alig -alaand DINOv2 still outperforms the base model on (train NYUv2 test 4D Light Field), these results demonstrate strong generalization performance of models aligned to human perceptual judgments. indicates best score the column.
Along with the results unseen test data domain
In-context examples retrieved by model
Disbetic Betinopathy
dSprites: x position
Base model munen-agged Rasdon
844
0.20
<class>
0.15
<class>
<class>
Proenslee
SnicLe
DM
Street View
0:25
Query
0.20
0.35
VLM
<class>
0.30
0.00
0.
Spcck
0000
CUP
Syctls
Figure 3: Left: Diagram of evaluation setup for retreetal-augmenteed generation. We retrieve the top-3 nearest image-prompt examples for each datasets and prompt OpenFlamingo with them before inputring the query image. Right: Classification accuracy on VTAB 35 from wide-varying domains. Error bars indicate 95% confidence interval over 5 random seeds.
object class and determine the model's decision by selecting the class with the largest log probability. See the Appendix for full details on the RAG experimental setup.
As illustrated in Fig. 3l classification accuracy on a wide variety of data domains improves with prompts retrieved by numan-aligned models, compared to the original model backbones. Even in outtof-distribution domains such as medical imagery and renders of game scenes, human-aligned models retrieve in-context examples that boost OpenFlamingo classification accuracy. These results suggest that human-aligned models can select more informative examples for in-context learning, thereby boosting the few-shot generalization abilities of downstream multimodal VLM.

---

4.3 Counting
well-documented limitation of large vision backbones is their performance on compositional tasks: in particular, on object counting [45]. We investigate how aligning to perceptual judgments affects performance counting tasks via the FSC147, CARPK, and Clevr-Count (adapted from the original Clevr dataset by (60)) benchmarks by computer k-Nearest Neighbors accuracy on frozen vision representations. We report results in Table and retrieval visualizations for few (n 3) and many (n 8, 10) objects in Fig. and find that, across 6 different models, the human-aligned versions outperform their counterparts in 35 out of 36 cases. See the Appendix for full details on the counting experimental setup. Given that the perceptual similarity dataset we use for finetuning
Base CLIP
Base DINO
Ensomble
10
Query: 3
Query:
Query: 8
10
Human-Aligned CLIP
Human-Aligned DINO
Human-Aligned Ensemble
Visualizations of nearest-neighhor examples retrieved by CLIP, DINO, and Ensemble models as well their human-aligned versions. Overall, see retrieved images with more accurate object counts in CLIP-HA, DINO-HA, and Ensemble-HA across multiple nearest neighbors.
Figure 4:
contains image-level similarity judgments, the consistent improvements that we observe on counting tasks, which requires local object awareness, is somewhat surprising. We hypothesize that the sensitivity of our human-aa aoman-aligned models to object counts may be a byproduct of NIGHTS examples themselves, many of which include image triplets with varying numbers of objects (see Fig. [5 in the Appendix). In terms of human perception, we note that humans consider object count when evaluating image similarity soon they develop counting profiency [40]. Thus, possible that given the prevalence of triplets with object-count variations NIGHTS, the human annotations naturally capture this counting aware effect in the global image-level labels and propagate this information to the numan-aligned models.
Counting Performance
74
Dare model
FSC147
CARPK
Clever-Cound
70
MAE
RMSE
MAE
RMSE
MAE
RMSE
66
DINO
44.1
118.7
51.4
56.8
1.25
L.70
64
DINO-HA
41.31
119.3
48.7
54.57
1.08
1,50
60
DINOv2
$7.5
128.3
52.4
59.5
1.18
L.60
DINOv2-HA
44.9
113.61
52.1
58.2
0.84
1.20
CLIP
59.1
158.4
52.5
60.3
1.09
1.52
37
CLIP-HA
53.2
156.0
52.3
58.8
0.81
OpenCLIP
54.4
153.5
54.1
60.3
0.97
4
33
OpenCIJP-HA
50.2
139.1
49.9
55.8
0.801
1.15t
32
SyaCLR
50.6
139.6
54.3
60.3
1.05
1.45
SynCLR-HA
46.4
128.1
51.3
58.2
0.98
1.37
CUP
OpenCLIP
SynCUR
Ensemble
DiOOv
Diko
Ensemble
48.4
132.4
49.6
60.3
1.10
1.51
Ensemble-HA
45.4
130.3
48.4
55.2
0.83
1.19
Models
Table 3: Error comparisons for base and human human-aligned models standard counting benchmarks. Though FSC147 and CARPK have examples with extreme object counts (tens and hundreds) unseen in the NIGHTS data, human-aligned models still achieve higher performance in each pair. indicates best score in the column, lower is better.
Figure 5: Performance improve-
ments Clevr-Count visualized by backhone RMSE (top) and MAE (bottom), averaged across all datasets. Lower is better.
4.4 Instance retrieval

---

DeepFashion2
In this section, we evaluate how aligning models to perceptual judgments affects their performance on the instance retrieval task. This task aims to retrieve images from a gallery contain- ing shared subject or object with the query image. At test time, retrieval is performed by computing the cosine similarity between the extracted features of the query and gallery images. Succeeding at this task requires that representation be ro- bust to recognizing instance identities under different lighting, backgrounds, poses, and other such shifts.
Top-1
Top-3
Top-5
DINO
8.02
12.15
14.44
DINO-HA
17.55
20.84
DINO<2
5.95
8.47
9.98
DINOv2-HA
13.57
CLIPP
4.59
6.89
8.23
CLIP-HA
8.62
13/02
15.60
OpenCLIP
14.88
22.74
27.36
OpenCI.P-HH IIP-HA
16.85
24.58
28.64
SyaCLR
4.88
7.34
9.02
SynCLR-HA
8.35
12:31
14.86
We evaluate all base models and their numan-aligned counter- parts on the Consumer-to- Shop benchmark of the DeepFash- ion2 dataset [21] The benchmark consists of 10990 consumer "in the wild images" as the query set, and 21438 gallery images with matching clothing items consumer images. Following the evaluation protocol from [2I]. we report Top-1, 3, and accuracy in Table Human aligned models outperform base models by significant margin across all metrics and backbones (visualized in Fig. 6 In Fig. 7 we provide qualitative retrieval results. These results agree with prior work showing that training on NIGHTS improves performance in retrieving similar images to queries [18]
Ensemble
13.54
20.01
23.54
Ensemble-HA
Table Top-1, -3, and -5 recall scores for instance retricval on DeepFashion 2. i indicates best score in the column, higher is better.
Base Model
DeepFashion2 Performance
30
Base model
Query
Quary
Human
Accuracy
20
15
Allinned
Human Human-Aligned
Buse Model
10
Query
CLIP
OpenCLiP
SynCLR
DINOv2
Ensemble
DINO
Models
Figure 6: Performance improvements on the DecpFashion2 instance retrieval, task visual- ized by backhone and averaged across all & for top-k recall. Higher better.
HkmannAligned
Haman-Alineed
Figure 7: Examples top-3 retrievals given query image on DeepFashion2. Overall, the human-aligned models return matching clothing items more frequently.
4.5
What type of human similarity annotation most beneficial?
As evidenced by the current widespread use of large vision models 4 46). trained on massive datasets, model performance largely correlates with data scale. This raises the question: are performance gains purely due to training the base models on additional data, or as a result of the perceptual qualities embedded in NIGHTS? To investigate this, ablate the training dataset- rather than tune on 13,900 NIGHTS triplets, we train on three other image triplet datasets of the same size:
1. BAPPS 61
Originally the training set for the LPIPS perceptual similarity metric [61], BAPPS consists image patch triplets with various low-level distortions applied (c.g. color jitter, gaussian blur, JPEG compression artifacts).
2. THINGS 24 This dataset contains image triplets with each image encoding different concept (c.g. triplet of images categorized as (airplane, elephant, football)) )), labeled by humans tasked to determine which concept the odd-one-out.
3. ImageNet 48 To ablate whether perceptual judgments at any level are needed, we construct an image triplet dataset by randomly selecting two images from one category and one image from another, labeling the first image pair more similar each other.

---

For all three datasets, we apply the same training settings with the original LoRA -tuning on NIGHTS. See Fig. for dataset ablations on object counting and instance retrieval.
Tuning on NIGHTS indeed provides the largest improvements across these tasks, with THINGS worsening performance overall and BAPPS/ImageNet having minimal effect. These trends may appear because BAPPS' photometric distortions are too low-level to impart any perceptual signal onto the backbones, THINGS encodes higher-level conceptual similarity irrelevant these mid-level vision tasks, and pre-trained vision models already perform quite well at discriminating ImageNet categories. Indeed, previous work [8] has found that similarity judgments by perceptual metrics trained on BAPPS correlate better to low-level metrics such as color than to semantic attributes.
In Section A.2 of the Appendix find that the performance boost from NIGHTS other datasets also consistent across semantic segmentation and depth estimation. Conversely, tuning on NIGHTS fails improve other datasets classification datasets; further discuss this finding in Sections and A.1
Dataset Ablation: Counting RMSE (4)
Dataset Adlation: Retrieval Recal (1)
85
1.5
NIGHTS
magetet
60
1.
THINES
BAMP5
20152
15
10
CarPK
Clevr-count
Tap-1 Tap
Tap-5
Figure 8: Evaluations comparing dataset utility counting tasks (lower RMSE is better) and DeepFashion2 instance retrieval (higher recall better). Across each task, tuning NIGHTS yields largest improvements while THINGS worsens performance and BAPPS/ImageNet makes minimal changes.
Discussion
Recently, the vision research community has converged on the idea that using human perception to improve machine perception can bolster the transfer of vision representations to downstream tasks [6]
However, remained unclear which tasks benefit most from alignment with human perceptual judgments; different tasks may demand representations that encode different levels of granularity and semantics. Here, we fine-tune modern backbones, pretrained on different tasks, on human perceptual judgments [18] and subsequently investigate which downstream tasks benefit. We develop a better understanding of how perceptual alignment affects performance on important tasks e.g., segmentation, depth estimation, RAG which may in turn inform future model training and data curation decisions across different applications. By evaluating competency at downstream tasks, we quantify what these representations capture and make decodable, enabling better understanding the human-aligned feature space. While we probe representations in terms of competency, understanding them in terms of their mechanism is rich direction for future work.
We find widespread benefits from perceptual alignment across both image- and patch-level tasks. At the global level, performance consistently improves for aetrieval-augment based and instance- based retrieval. Moreover, propagating the supervision from image-level similarity judgments to VIT patch tokens improves performance dense prediction tasks (semantic segmenta- tion and depth prediction). Since model performance closely linked data scale, we also ablate the choice of perceptual dataset. While fine-tuning NIGHTS dataset of mid-level perceptual judgments - leads to improvements across various tasks, fine-tuning on triplets from ImageNet [10), BAPPS [61). and THINGS (23 preserves deteriorates transfer performance.
generation, counting-
Why does fine-tuning on NIGHTS in particular lead improvements? We hypothesize that the variations found in BAPPS and THINGS are solely high- or low-level, whereas the mid-level distortions in NIGHTS cover salient features that humans use when making inferences about what they see; these characteristics include style. pose, color, and count (see Fig and largely correlate with the characteristics model must successfully extract for many computer vision tasks. Previous work [18) found that models fine-tuned on NIGHTS seem to attend to both low-level and semantic attributes. Aligning feature space to these concepts may be useful for visual tasks requiring both visual and semantic knowledge, such retrieval, counting, segmentation, etc. This hypothesis may

---

also explain why tuning on NIGHTS hurts performance on fine-grained tasks, in which perceptually similar images may belong different categories.
Limitations. Perceptual alignment does not appear to improve performance for standard image classification tasks such natural image datasets in the VTAB benchmark [60] (see Tables 5
in the Appendix). This is surprising in light of recent findings that demonstrate downstream task improvements in image classification tasks for human-aligned representations [42). Although is hard to pinpoint the exact cause, we hypothesize two reasons: First, perceptual judgments at different levels of abstraction may be helpful for different downstream tasks. While the mid-level perceptual judgments in NIGHTS boost performance for retrieval basse and dense prediction tasks, they may not impart a useful inductive bias for standard image classification tasks; high-level semantic associations could simply be better suited for these kinds of tasks. Alternatively, the visual features that humans use to judge similarity may not be appropriately captured classification accuracy metrics. Some VTAB datasets have numerical ground truth in which the distance from the correct answer is meaningful; thus, it may be ill-suited for classification, and better suited to continuous evaluations, which we report in sections of the paper.
Additionally, key insight from our dataset ablations is that not all human preferences improve performance. Finetuning on perceptual datasets with solely high-level (THINGS) or low-level (BAPPS) variations hurts performance for many downstream tasks (Section 4.5 and A.2). Similarly, Muttenthaler et al. [43] recently discovered that finetuning vision models on THINGS can hurt downstream task transfer. Due to our evaluation of the attribute the drop in performance perceptual alignment. Furthermore, by ablating the number of fine-tuning steps in Section A.2] we show that overfitting to perceptual judgments may also harm downstream performance.
SynCLR [55]. we
Societal impacts. Beyond our findings, there are other possibilities for harm outside the scope the type of perceptual annotations we have studied: Human preferences may reflect unwanted biases. long-standing problem in both language and vision is that biases (e.g. gender, racial) reflected in Internet language/images are inherited by large models, and reflected in their embeddings. One can imagine similar phenomena happen visual preferences (29). Humans may disagree on preference label, or even disagree with themselves if asked at different points in time. This may lead to noisy data if not filtered carefully, thus harming representations [54 43
Without sufficient demographic diversity in the annotator group, emerging biases may be reflected in the model. For example, some RLaietrainnd language models have been shown to develop a bias towards the opinions of high-income, liberal individuals over their non-RLHF counterparts [49]
Acknowledgements
We thank Tali Dekel and Richard Zhang for helpful discussions. This work was supported by the Sagol Weizmann-MM -MT Bridge Program and by a Packard Fellowship and a Sloan Research Fellowship to P.I. and an NSF GRFP Fellowship to S.S.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
[2]
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilbarco, Mitchell Wortsman, and Ludwig Schmidt. Openftaniigo: An open-source framework for training large autoregressive vision- lasion-anguage models, 2023.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
[5] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple frame- work for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 1 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
[7] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020.
volume 33, pp. 22243-22255
[8] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 1188-282,,42 2023.
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 2LL.1090CVVR.220995500000
[11] Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback. arXiv preprint arXiv:2310. /2/03, 2023.
[12] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference muchine learning, pp. 647-655, PMLR, 2014.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.

---

[14] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual represen- tations. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 9588 9597,9997.207 2021.
[15] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network, 2014.
[16] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models, 2023.
[17] Thomas FEL, Ivan F Rodriguez Rodriguez, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9432 9446. Curran Associates, Inc., 2022.
[18] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 80742-50768
Curran Associates,
Inc., 2023.
[19] Leon Gatys, Alexander Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances in neural information processing systems, 28, 2015.
[20] Leon A Gatys, Alexander S Bcker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprini arXiv: nee:::1080,0575 2015.
[21] Yuying Ge, Ruimao Zhang. Lingyun Wu, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re- identification of clothing images. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5332-5340, 2019.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[23] Martin N. Hebart, Charles Y. Zheng, Francisco Pereira, and Chris I. Baker. Revealing the multi- dimensional mental representations of natural objects underlying human similarity judgements. Nature Human Behaviour,
1185, October 2020. doi: 10. 00.100//41162-0050 022000000-00---.
[24] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng. Alexis and Chris I Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife, 12:e82580, feb 2023. ISSN 2050-084X. doi: 00.7554/eLife.885580
Kidder, Anna Corriveau, Maryam
[25] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning, arXiv preprint 2XXi:22166.02288 2021.
[26) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4904-4916. PMLR, 18-24 Jul 2021.
[27] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Pisio--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694-711. Springer, 2016.
[28] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
12

---

[29] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 36652-36663. Curran Associates, Inc., 2023. URL hRR.htttpp::/prrceeedngg wwwp:iinnnnnn..u.......... www.nnnurii.... cc/ (3acc688664460655506655555---623 Conferrc- pdf
[30] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023.
[31] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems, 35:2331 1113311-2330 133300. 2022.
[32] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
[33] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
[34] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retreeval-augmented generation for knowledge-intensive nlp tasks, 2021.
[35] Aoxue Li, Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning: Knowledge transfer with class hierarchy. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 7212-7220, 2019.
[36] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018.
[37] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation, 2022.
[38] Yiging Liang, Eliot Laidlaw. Alexander Meyerowitz, Srinath Sridhar, and James Tompkin. Semantic attention flow fields for monocular dynamic scene decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21797-21806, 2023.
[39] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski. and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[40] Kelly S. Mix. Similarity and numerical equivalence: Appearances count. Cognitive Develop- ment, 14(2):269-297, 1999. ISSN 0885-2014. doi: htt.:/ioorrr/011.0111/0/8333333 00005-2.
[41] Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Korn- blith. Human alignment of neural network representations. In 11th International Conference on Learning Representations, ICLR 2023, Kiguli, Rwanda, Mai 01-05, 2023. OpenReview.net, 2023.
[42] Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Her- mann, Andrew Lampinen, and Simon Kornblith. Improving neural network representations using human similarity judgments. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 50978-51000 10007 Curran Associates, Inc., 2023.
[43] Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael Mozer, Klaus-Rober Müller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstraction levels. arXiv preprint arXiv:2409.066:: 06509, 2024.
13

---

[44] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[45] Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3147-3157, 2023.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18-24 Jul 2021.
[47] Joshua Robinson, Ching-Yao Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04522 2020.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg. and Fei-Fei Li. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 1113:211-2252. 10.1////s 1263-015-0816-y.
2015. doi:
[49] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 29971-30004. PMLR, 23-29 Jul 2023. URL mlr httpa///aaaa...peee//////////aaa 2ttess//202//aaturrra22222 htmml
[50] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Can features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 806-813, 2014.
[51] Prafull Sharma, Julien Philip, Michaël Gharbi, Bill Freeman, Fredo Durand, and Valentin Deschaintre. Materialistic: Selecting similar materials in images. ACM Transactions Graphics (TOG), 14,, 2023.
[52] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. arXiv preprint 208791230..07931, 2023.
[53] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: text-to-image generation in any style. arXiv preprint arXiv:2336. 00983, 2023.
[54] Ilia Sucholutsky, I Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, al. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13011 2023.
[55] Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. ArXiv, a77/2312 2023.
[56] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9568-9578, June 2024.
[57] Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in single video. arXiv preprint 200.:::2033114445 2024.
14

---

[58] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 2096-2105. IEEE, 2023. doi: N0.10099CCV51000.202.....
[59] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Marina Mcila and Tong Zhang (eds.), Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320. PMLR, 18-24.2 Jul 2021.
[60] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv: Computer Vision and Pattern Recognition, 2019.
[61] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[62] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing, arXiv preprint arXivv2303.096/68 2023.
15

---

Appendix
In the Appendix, Section A contains additional experiments and ablations, Section B contains qualitative results and examples from different perceptual datasets, and Section CC method- ological/imllemnntaion details.
A
Experiments
A.1
Classification with the VTAB Benchmark
We evaluate on the Visual Task Adaptation Benchmark (VTAB) (60], a comprehensive set of nineteen classification datasets that is commonly used to study the transferability of representations. VTAB covers a broad set of domains beyond classic natural image datasets, including medical images, scene understanding, and fine- aine-grained classification. Several of the datasets are drawn from domains that not included in the NIGHTS dataset.
In Tables we show classification results on all VTAB datasets for each backbone, and its human- aligned version (denoted with -HA). The VTAB datasets are divided into three categories: Natural, Specialized, and Structured. Natural datasets include standard vision benchmarks; specialized datasets include benchmarks captured through specialized imaging equipment, such satellites and microscopes; structured datasets include benchmarks that focus on structure and layout, with both naturally capture and simulated images.
We find that allinne models largely lead worse performance on standard natural tasks. Re- sults on specialized tasks are mixed; for most datasets, human alignment helps or hurts by marginal amount. Results on structured datasets (Table are also mixed, however on particular datasets such and dSprites-Locatis
as sNORB, backbones. We remark, however, that these structured datasets may have limited evaluation utility. The Clevr-Count task, for instance, has numerical ground truth (i.e. the number of objects in an image) in which the distance from the correct answer is meaningful; thus may ill-suited to classification, and better suited to continuous evaluations such as RMSE or MAE, which we report in Section
alignment improves performance across
Natural
Specialized
Caltech101
CIFAR- 100
Flowers 102
Camelyon
Retinoputhy
EuroSAt AT
Resise45
Sun397
SVHN
DTD
Pets
Model
DINO
95.3
83.0
78.7
96.7
93.1
74.8
70.8
85.8
97.2
93.8
77.9
DINO-HA
93.6
80.8
72.6
93.2
91.0
69.7
73.2
83.7
97.3
93.2
76.9
DINOv2
95.5
88.97
82.8
99.77
81.9
61.2
86.77
96.4
92.7
78.6
DINOv2-HA
96.2
87.7
80.2
97.9
89.7
77.0
63.3
84.6
97.0
93.5
77.9
CLIP
95.4
76.5
76.6
95.5
87.6
80.6
65.4
83.7
95.3
92.9
75.8
CLIP-HA
95.7
76.4
76.0
93.4
85.5
78.1
69.2
84.0
95.9
93.0
76.1
OpenCLIP
97.3
82.4
81.6
97.0
90.4
82.3
73.1
82.3
96.4
83.3
75.9
OpenCLIP-HA
96.0
75.8
77.3
93.5
83.7
76.6
70.9
82.9
96.5
92.6
76.2
SynCLR
95.5
74.3
80.5
98.8
92.9
81.4
56.2
86.0
96.6
78.8
SYnCLR-HA
95.0
69.7
77.8
95.9
90.5
75.7
70.9
85.1
96.8
93.4
78.0
Ensemble
97.4
86.8
84.3
98.8
94.4
83.9f
80.1
85.7
98.0
96.11
78.7
Ensemble-HH HA
97.3
85.0
81.4
96.4
91.9
82,5
82.21
83.5
97.9
96.11
78.1
Table 5: Performance on VTAB natural subset (left) and specialized subset (right).
A.2 Additional dataset ablations
In Section 4.5 we show that tuning on NIGHTS leads to larger performance improvements on object counting and instance retrieval than training on other triplet datasets. Here, we show that this finding is consistent across dense prediction tasks well. We evaluate models trained on NIGHTS, BAPPS, THINGS, and ImageNet- as described section 4.5 on semantic segmentation and depth
estimation. As shown in Fig. g 10 training on NIGHTS outperforms training on all other datasets.
16

---

Structured
Clevr-Count
KITTI Dii
SNORB-ER Elev
dSpccoo
dSpr-Omi
DMLab
Model
DINO
82,2
53.6
57.1
72.6
61.8
56.5
DINO-HIA
79.2
57.2
51.1
81.5
66.6
66.2
65.8
55.3
DINOv2
87.1
57.7
59.7
68.2
62.3
52.8
DINOv2-HA
83.6
59.0
53.7
79.2
74,1
69.1
64.7
55.9
50.2
CLIP IIA
72,9
57.8
51.4
80.2
60.3
64.6
42.6
47.2
OpenCLIP
78.5
52.0
55.0
67.9
38.
50.4
OpenCIIP.
76.7
57.6
51.5
84,9
64.1
63.3
46.7
50.1
SynCLR
90.91
63.8
57.3
68.1
55.2
76.9
52.1
60.5
SynCLR-HA
81.7
61.5
52.9
86.1
64.3
63.6
66.6
58.5
Ensemble
89.2
63.5
58.4
72.5
67.3
76.5
76.9
65.5
Ensemble-1t.
76.4
Table 6: Performance on VTAB structured subset.
We additionally run this ablation subset of VTAB (see Fig. TTa.. On these classification datasets, base models perform best; the exception is sNORB (pose prediction) for which NIGHTS is best. Amongst perceptual datasets, NIGHTS is sometimes outperformed by BAPPS/ImageNet. This result consistent with our findings in Section A.1, that tuning on NIGHTS often fails to improve classification performance.
Finally, we ablate the strength of alignment - i.e. the training loss when fine-tuning on different perceptual datasets. In Fig. HPp show the downstream performance DeepFashion2 when fine-tuning for increasing numbers of steps on different datasets. Tuning on NIGHTS outperforms other datasets over the full training trajectory. Moreover, while performance rises significantly with small amount of alignment to NIGHTS, it trends down after > >000 steps. This indicates that a small amount of alignment is helpful, however overfitting may harm performance.
0.8
Fuscal voc
ADCI0k
Clyscapcs
COCO
Wains
03
57
B4P95
31
03
34
PG
PA
MAUCI
MA
HAOCTI
PA (T1
RA (1)
Figure 9: Dataset ablations on semantic segmentation. Following the same procedure as in section 4.5 and the segmentation training in section 4.1 of the main paper, we ablate low/mid/high-leve similarity. We find that tuning on mid-level similarity with NIGHTS provides the most improvement, with many other cases degrading the base DINO backbone.
NYUV2
NYUV2 >>> 4D LF
None
12
5.2
NIGITS
ImageNet
5.0
THINGS
BAPPS
4.8
3.6
4
0.6
3.4
Figure Dataset ablations on depth estimation. Following the ablation and depth training setups described in the main paper, we evaluate low/mid/high-lert similarity. We find that tuning on mid-level similarity with NIGHTS results in the lowest error (RMSE), and that the other cases often result in worse performance than base DINO
17

---

Dataset ation: VTAB (subset) Accuracy (1)
DeepFashion2
100
90
None
NIGHTS
80
ImageNet
THINGS
70
0.05
60
2000 4000 6000
SUN397
resisc45
EuroSAT
Pets
Flowers102
sNORB-Azim
Training stans
Figure 11: (left)) Dataset ablations subset VTAB. Following same procedure as in section 4.5 and the segmentation training section 4.1 the main paper, we ablate low/mid/high-ler llveel similarity. We find that all perceptual datasets degrade the DINO backbone, with the exception of sNORB, for which NIGHTS is best. We also note that training BAPP or ImageNet often better preserves base model performance than NIGHTS. (right) Training-step ablations DeepFashion2. We ablate low/mid/high-leeell similarity and models finetuned
different numbers steps on instance retrieval. find a small amount of finetuming on NIGHTS best, after which performances declines (though remains above the base model). Note that the legend applies to both the left and right figure.
A.3 Additional RAG results
We replicate our RAG experiment OpenFlamingo in Section 4.2 IDEFICS2, recently released 8B multimodal model achieving state-of-the-an results across several benchmarks [56]. As shown in Fig. 12 across the same four classification datasets, performance consistently improved when using NIGHTS- models in the RAG pipeline. These results validate our RAG results on OpenFlamingo, suggesting that performance gains from perceptual alignment are not specific to particular VLM.
Diatete Setrepatty
Strer View House fambers
modd Fandon
636
606
0P
0P
0.00
COOCLL
Bosane
ased
SyolLe
Figure 12: Retrieval-augmeented generation results image classification for four different VTAB datasets. We evaluate the IDEFICS2 model, SB-parameter multimodal model released more recently than Open- Flamingo and capable of in-context learning. We follow the same experimental setup in retrieving examples as detailed in 4.2
A.4 Perceptual alignment for CNNs
Our main experiments assess the effects of perceptual alignment for array of ViT models, these are state-of-the-art pretrained vision representations. Of further interest, however, is whether the same effects are observed for pre-trained CNNs as well.
We assess the effects of perceptual alignment with NIGHTS on two popular CNNs: ResNet50 and ConvNeXt-B, using the same loss described in Section 3.2 We evaluate counting and instance retrieval tasks; our results are shown in Tables L.b. Similarly to ViTs, human-aligned CNNs outperform their pretrained counterparts across both tasks. Note that we report results for both ConvNeXt and ResNet, however acknowledge that the accuracies for ResNet are likely too small to draw conclusions from.
Due to lack open-source LoRA implementations CNNs, we instead train MLPs on top the respective final layers. Previous work [18] indicated that training MIPs with NIGHTS can still affect representations, but may be less effective than full fine-tuning. Nonetheless, we still see downstream improvements with this method.
18

---

Clevr-Count
Preponshion2
Model
RMSE
MAE
Model
Top-1
Top-3
Top-5
ConvNeXt
2,045
1,522
ConvNeXt
2.12
3.56
4.59
ConyNeXt-HA
1.631
1.193
ConvNeXt-HA
2.81
4.8
5,98
ResNet
3.140
2.551
ResNet
0.018
0.12
0.14
ResNet-HA 1.729
1.282
ResNet-HA
0.018
0.074
0.17
Table 7: Human-aligned vs. pretrained CNNs, evaluated counting and instance-retrieval. In most cases, human-aligned CNNs perform better.
B Qualitative examples B.1
Additional visualizations
See Figures 13 and 14 for additional examples of instance retrieval DeepFashion2 and object counting on Clevr-count.
Query
Query
Quary
Human-Aligned
Human Aligned
Human-Aligned
Base Model
Base Model
Base
Query
Query
Quary
Human-Aligned
Human-Aligned
Human-Alignad
Figure 13: Additional visualizations for top-3 retrieved examples for given query image in DeepFashion2.
Base DINO
Base Ensemblee
Query:
++
Query:
Query:
Aligan- CLIP
Human-Aligned DINO
Human Aligned Eisemble
Figure 14: Additional Clevr-count examples comparing base and human monnal nearest-neighbor image retrievals.
B.2 Dataset examples
See Figure 15 for examples of the triplet datasets we train on in this paper. In each triplet, the reference image (middle) and outlined image are labeled as similar pair. Perceptually-alignee models in this paper tuned on NIGHTS triplets (top row), whose image variations encompass variety mid-level perceptual attributes including object count, identity, layout, subject pose, and color. In contrast, the datasets studied in our ablation encompass differing image attributes: BAPPS
19

---

NIGHTS triplets
BAPPS triplets
THINGS triplets
ImageNet triplets
Figure 15: Examples of triplets from the NIGHTS, BAPPS, THINGS, and ImageNet datasets, with the bordered images labeled more similar the reference (middle image each triplet).
focuses image patches with low-level distortions such CNN artifacts and color jitter, THINGS encodes similarity in concept space, and our constructed ImageNet triplets outline class boundaries.
C
Implementation Details
C.1 Training human-aligned models
The human-aligned models studied in this paper were fine-tuned with LoRA parameters: 16, = 0.5, = 0.0. All contrastive training (for image- and patch-level objectives) are trained with an Adam optimizer (Ir 0.0003), batch size of 16, and hinge loss margin of 0.05 detailed in 18] We train all models for epochs and evaluate on the checkpoint with the lowest validation loss. Train/val/test splits on NIGHTS, BAPPS, and THINGS were used as-provided in the dataset.
C.1.1 Dense prediction heads
Each semantic segmentation head a single linear layer mapping frozen DINO/DINOV2 patch tokens to segmentation map. We train with the Adam optimizer (a 0.0003) for 10 epochs with batch size of 16 and workers. The semantic segmentation head optimized with a Jaccard Index loss [AnB]
(cquivalently, mean Intersection-ovee Union) defined as ((A, B) = TAUB
Dense prediction heads are defined as a single linear layer mapping frozen DINO/DINOv2 patch tokens to a spatial map, whose values are then categorized into 256 bins. We train with the Adam optimizer (a = 0.0003) for 10 epochs with batch size of 128 and 12 workers. The scale-invariant log loss for prediction A and target B is defined as:
C(A. B)
Σ
d +0155
Σ d,
N
N
where d; = log(A + e) log( B + +), 0.001 to avoid gradient issues, and N is the number of valid pixels.
All training and evaluation for dense prediction tasks done on a single NVIDIA Titan RTX GPU.
C.2 Retrieval-augmentee generation
We follow the evaluation protocol the codebase provided by 22. constructing prompt containing three image/text examples, a query image, and a candidate class. We then average the model's s output logits, and repeat for each candidate class in a dataset form a softmaxed array of log probabilities. OpenF SrenFlaming''s 's classification prediction is determined as the class with the highest corresponding log probability, while classifications from IDEFICS2 are taken purely from a multiple- choice text response. Error bars were calculated as 95% confidence intervals over five trials with 1,000 andomly-sample queries each. All evaluation for RAG tasks done on single NVIDIA Titan RTX GPU.
20

---

C.2.1 Object counting
Our k- k-Nearest Neighbor evaluations reported in the Counting section are run over h E {1,3, 5, 10) for the training set, and the argmax with highest classification aceuracy is used to report the final performance on the test set. We use the train and test sets as provided for the FSC147 and CARPK datasets, construct a random 80:20 data split the Clevr-Count dataset. All evaluation for this section is done on single NVIDIA Titan RTX GPU.
C.3 Instance retrieval
We evaluate backbones on instance retrieval by extracting features (CLIP/OpenCLIP. DINO/DINOv2/SynCLI CLS token) from the gallery and query sets. For every query image, compute the cosine similarity with every image in the gallery. We then calculate top-k accuracy as the number of queries for which correct gallery pair image in the top retrievals.
embedding.
C.4 Dataset ablations
We train models for the dataset ablation section with identical hyperparameters as the human-aligned models (details in Section C.1 and the same amount of data the NIGHTS training set (13,900 triplets). We randomly select these triplets without replacement from the BAPPS and THINGS training sets, and construct the ImageNet triplets by randomly sampling images from one class and third image from different class.
C.5
VTAB classification
To evaluate models on VTAB, we follow standard procedure and train linear classifier on top the frozen representations using multinomial logistic regression, with the sci-kit learn implementation 42 For each classífier we perform hyperparameter search over the regularization strength using 10-fold cross validation over the training set, and report performance the validation set. We search over the values с = {1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6} where c the inverse regularization strength.
C.6 Additional compute details
See subsections above for experiment-spectin compute details. This full research project required additional compute for experiments and results that are not included in this paper; these computations also done on single NVIDIA Titan RTX, GeForce 2080, GeForce 3090, and V100 GPUs.
C.7
Licenses for existing assets
The datasets and models we use are released under the following licenses:
Dataset
License
NIGHTS
MIT
VTAB
Apache 2.0 BSD 2-Clause
BAPPS
THINGS
CCO 1.0 Universal
ImageNet
CC BY-NC Apache 2.0 CCO 1.0 Universal
Model
License
FSC147
DINO
Apache 2.0
CARPK
CLIP
MIT
Pascal VOC
CC BY 2.0
OpenCLIP
MIT
ADE20k
BSD 3-Clause
DreamSim
MIT
Cityscapes
CC BY-NC
SynCLR
Apache 2.0
COCO
CC BY 4.0
DAVIS2017
CC BY 4.0
NYUv2
MIT
4D Light Field
CC BY-NC-SA 4.0
SUN-RGBD
MIT
DeepFashion
CC BY-NC
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction are written with the paper results and scope fully in mind.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We describe limitations in the "Discussions" and Supplementary Material sections of our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification:
Our paper contains no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all experimental settings and hyperparameters in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide a zip of the code that produced our experimental results, as well as our project github at:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ wttaeeee//ooeeSuamaaaiiiiiiiioie
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccpublcccguuuddeeeoo Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide these details in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide 95% confidence interval bars across five seeds for our experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on compute resources in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics, and have determined that the research conducted conforms to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We detail the broader impacts of our work in the Discussions section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses such risks, as we do not have new pretrained models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers whose datasets and models we and provide license information in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "UhN1H37b9P", "reviewer": "Reviewer_kSUr", "summary": "This paper studies the question of when perceptual alignment supervision makes better vision representations. They fine-tune existing pre-trained vision backbones (eg, CLIP and DINO) with LoRA on NIGHTS, a synthetic triplet-based dataset of human similarity judgments. Models fine-tuned in this way generally perform better on vision tasks, ranging from dense prediction, RAG, counting, and object retrieval. However, the results can be the opposite when evaluating classification tasks, especially natural classes. They also considered fine-tuning on different types of human preference datasets, and found mid-level supervision to be more beneficial.", "questions": "I still find this paper could spark some interesting questions. More investigations on these aspects might increase the significance of this paper.\n1. When can human preference be harmful? One motivation of this paper is to provide some empirical guidance for future vision research, if human preference is introduced as the way in the language community. As I mentioned above, the benefit of it to vision is not very surprising, but the prevention of possible risks could be valuable. This paper has shown some and more in this direction might be more valuable than the benefits (as most improvements are just marginal).\n2. How should human preference be better defined and categorized? A precise definition of human preference is hard to obtain, but important. This paper has considered the level of supervision, and a discussion of other aspects could help readers.\n3. What distracting factors exist in current human preference datasets and how to isolate the effects of them? This is important to ensure reliable conclusions.", "limitations": "Limitations are discussed in the paper. Suggestions are flagged above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The tasks considered in this paper for evaluation are comprehensive and cover many different properties of the representation.\n2. The delivery of this paper is clear and easy to follow. Key motivation, setting, and conclusions of each part are easy to spot.\n3. Evaluating human preference for vision representations is an emerging topic and interesting for research.", "weaknesses": "1. My major concern is that although the topic is somehow interesting and the evaluation is comprehensive, many conclusions are expected and unsurprising. Provided in the form of triplets, the supervision of human preference is just another form of \"classes\", being very fine-grained and having flexible class definitions. This is similar to the learning signal of self-supervised contrastive learning, with a better form of data augmentation. In this regard, it is expected that human preference makes generally better representations.\n2. Regarding the performance drop in fine-grained classification (especially natural classes), this might be a result of the domain gap. Given that the model is fine-tuned on all synthetic images, it is unsurprising that discriminating natural images is harder afterward. The definition of human preference can be ambiguous, and the dataset for fine-tuning can have distracting factors (eg, synthetic distribution). These factors are not tackled well in the design of comparisons and could make some conclusions unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and appreciate that they found the evaluation comprehensive, the paper clear, and the topic interesting. We address questions/concerns below.\n\n**“It is expected that human preference make generally better representations.”**\n\nWe broadly agree with the reviewer that human preference is a form of fine-grained “classes”, with flexible class definitions, depending on the level/type of similarity judgments. Our key contribution is to elucidate how these classes should be defined to benefit representations: we find that the particular preferences in NIGHTS improve performance across many tasks.\n\nA key finding, in fact, is that *not all human preferences are always better*. In the global response section 1.1, and section 4.5 of the paper, we compare finetuning on multiple types of human preferences (both high- and low-level). Finetuning on NIGHTS outperforms other perceptual datasets (BAPPS, THINGS) across segmentation, depth estimation, retrieval, and counting. Finetuning on triplets formulated without any human preferences (from ImageNet) often outperforms models fine-tuned on BAPPS/THINGS, showing that some human preferences harm representations.\n\n**Synthetic-Real domain gap**\n\nWe briefly discuss the synthetic-real domain gap in section 4.5. We ablate the use of synthetic images in NIGHTS by including SynCLR in our experiments, which was pre-trained solely on generated images. SynCLR exhibits the same performance drops across fine-grained/natural datasets as real-trained backbones, indicating that they must result from the perceptual alignment. We will clarify this in our revision.\n\n**When can human preference be harmful?**\n\nAs detailed further in the next section, “human preference” lacks a rigorous definition in vision and language literatures; it can include aesthetic preferences and other harmful types of annotations. The scope of our work is to evaluate human perceptual alignment, which extracts human preference at a psychophysical level rather than a more cognitively-penetrable level. Nevertheless, we empirically observe several cases in which human preferences may harm representations:\n- In the global response, section 1.2, we show that overfitting to perceptual datasets harms performance.\n- In our dataset ablations (global response section 1.1, paper section 4.5) we find that finetuning on perceptual datasets with solely high-level (THINGS) or low-level (BAPPS) variations hurts performance for many tasks.\n- Finetuning on NIGHTS seems to degrade how well representations to distinguish between distinct fine-grained categories that are very visually similar (paper section 6.1). As mentioned above, due our evaluation of the synthetically-pretrained SynCLR, we attribute the performance drop to perceptual alignment.\n\nThere are other possibilities for harm outside the scope of the type of perceptual annotations we study:\n- Human preferences may reflect unwanted biases. A long-standing problem in both language and vision is that biases (e.g. gender, racial) reflected in Internet language/images are inherited by large models, and reflected in their embeddings. One can imagine similar phenomena in visual preferences [3].\n- Humans may disagree on a preference label, or even disagree with themselves if asked at different times. This may lead to noisy data if not filtered carefully, thus harming representations [1].\n- Without sufficient demographic diversity in the annotator group, emerging biases may be reflected in the model.  For example, some RLHF-trained language models have been shown to develop a bias towards the opinions of high-income, liberal individuals over their non-RLHF counterparts. [2]\n\n**How should human preference be better defined and categorized?**\n\nWe thank the reviewer for pointing this out; it is important to define human preferences in vision. In the language space, human preferences largely refers to ethical judgments, or aspects of the user interaction experience. In vision, we define preferences as concepts (or categories) that humans use for making image similarity judgments [1]. By training on these preferences, we make models learn a concept space that is aligned with the concepts that humans use to navigate the visual world. We will clarify this in our paper. In addition, we will refer the reader to a recent review paper, “Getting aligned on representational alignment” [1] that clarifies definitions related to aligning vision representations with human judgments. \n\nWe also note that aesthetic judgments (i.e. concepts that humans use to determine visual appeal) have been used to improve diffusion models [3]. We consider this a separate category of annotation, however can refer the reader to relevant works. \n\n**Distracting factors in human preference datasets**\n\nThe signal-to-noise ratio of human cognitive data (may it be behavioral or neural) is often low. Thus, it is important to perform a denoising step before using the data downstream. For behavioral data this can be achieved by collecting a large number of judgments and filtering out the low quality judgments or a strong regularization technique. Distracting factors may relate to long response times or other confounding factors. Isolating such effects could be achieved by showing human participants triplets of different kinds, with varying background or object complexity where complexity can be the visual richness of the scene(s) or the number of objects.\n\n[1] I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, J. Achterberg, J. B. Tenenbaum, et al. Getting aligned on representational alignment. arXiv, 2023.\n\n[2] Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P. and Hashimoto, T.. Whose opinions do language models reflect?. In ICML, 2023.\n\n[3] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "In3h77qI58", "reviewer": "Reviewer_9gB3", "summary": "As the title clearly indicates, this article investigates how aligning vision model representations to human perceptual judgments impacts the performance of models relying on these representations for downstream tasks such as dense prediction (semantic segmentation and depth estimation), retrieval-augmented generation, object counting and instance retrieval. Given a vision model backbone, it is finetuned with LoRA on the NIGHTS dataset, which contains human similarity judgments over synthetically-generated image triplets (NeurIPS 2023). For each considered downstream task, several backbones are compared to their \"human-aligned version\" with globally better results when the backbone is fine-tuned on human perception.", "questions": "* is the local patch level used for training for segmentation and depth estimation only or is it the same training protocol (with global and local levels) for all experiments? If so, what is the contribution of adding (or not) the heads that are trained at a local level?\n\n* will the code (zip file) be released with the paper?", "limitations": "An experiment on the VTAB benchmark, whose results are reported in the supplementary material, leads to lower performance with fine-tuned representations. This case is significantly discussed in Section 5, with several hypotheses proposed to explain the phenomenon.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* the scientific question raised by the paper is simple but interesting and the study shows it is worth investigation. The problem is clearly stated and introduced, the method and protocol are well explained and the article is well written overall. This bullet point is not just a false pretext to artificially add a \"strength\" to the review, it is a real pleasure to read.\n\n* the method to fine-tune the backbones relies on well-known and standard methods and tools (visual transformer, triplet loss...), which leads to a convincing methodology. The reader is not confused by an incomprehensible labyrinthine system and can therefore fully concentrate on the results of the study.\n\n* the experimental part is impressive, with an evaluation of a large variety of tasks, with various backbones as well. This is obviously a strong part of the paper: although the original idea was simple and interesting, the general quality of the study is finally supported by all these experiments. The supplementary material and the code (zip file) allow a precise investigation of the experiments and ensure reproducibility.\n\n* Section 4.5 investigates the use of alternative datasets to fine-tune the model. Some were proposed by previous works and others were built by the author to test a specific question. All of them have a size similar to NIGHTS. The resulting analysis is interesting and is a nice complement to the study.", "weaknesses": "* the alignment is performed both at the image level (section 3.2) and patch level (section 3.3) and the authors claim (lines 164-166) that \"Additionally, we find that local patch-level representations can be improved by tuning on image-level perceptual judgments, and show performance increases on semantic segmentation and depth estimation\". However, in the following all experiments only compare a backbone with its \"human aligned version\" (HA) and it is not clear what is the contribution of the alignment at a global and a local level. On line 148, the text says that the model consists to \"train heads for semantic segmentation and depth estimation [at the local level]\". Globally, the manuscript could be more explicit on this \"details\": is the local patch level used for training for segmentation and depth estimation only or is it the same training protocol (with global and local levels) for all experiments? If so, what is the contribution of adding (or not) the heads that are trained at a local level?\n\n* one may regret that the negative result on VTAB is not reported in the main paper and only discussed in the Limitation without further investigation. However, one must recognize that the paper already contains significant experimental works, that allow to support the main demonstration.\n\n* the fact that the same backbones are used in several experiments/tasks is interesting but sometimes it leads to models that are very weak baselines in comparison to the state of the art. In particular, in Section 4.3, specialized class-agnostic models on few-shot counting have much better performance than those reported (less than 0.15 MAE/RMSE on FSC147 for SafeCount or CounTR)\n\n**minor** \n* the experimental protocol for \"object counting\" reports that $k$ is determined on the training set among 4 values then the best one is used on the test set. Reporting the performance for each $k$ on the training set, or at least the one that was retained on the test set, would be useful.\n* line 95: a reference is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We are glad that the reviewer finds the method convincing, the paper well-written, and the experiments comprehensive. We address specific concerns and questions below.\n\n**Global v. local representations**\n\nWe provide details regarding training patch tokens in the global response, Section 4 of the paper, and further clarifications below.\n\nGlobal representations refer to the output embedding for CLIP/OpenCLIP, and the CLS token for all other backbones. These alone are used for counting, instance retrieval, RAG, and classification, as they contain information across the image. In these global tasks, we either compute k-Nearest Neighbors over these global representations, or train linear probes. Local features are used for dense tasks (segmentation, depth estimation);  good spatial features are necessary, as the output depends on per-pixel labels rather than image-level labels. Following the procedures from [1], we train segmentation and depth estimation heads on top of only the local features. \n\nWe thank the reviewer for raising these questions, and will clarify our methodology in our revision.\n\n**VTAB results**\n\nWe appreciate that the reviewer brought up this concern yet recognized the significant coverage of experiments in the main paper. Our main contribution is to empirically identify how tuning on perceptual datasets impacts transfer performance; for a thorough investigation, this includes negative impact. We acknowledge, however, that readers may be most directly interested in where this alignment succeeds, and structure the paper to highlight these applications while retaining all our findings.\n\n**Comparison to state-of-the-art task-specific models**\n\nAs the reviewer notes, we use the same backbones across all experiments and tasks. This was done to evaluate general-purpose representations, and we acknowledge that these may be outperformed in cases such as counting by task-specific models. We do not aim to achieve state-of-the-art performance on any single task; rather, we demonstrate how a representation becomes comparatively better (or worse) over an array of multiple tasks. By evaluating the competency of a single representation over multiple tasks – which would not be possible with task-specific models – we gain insight into how general-purpose feature spaces are affected by alignment.\n\n**Counting experiment parameters**\n\nBelow we report the training performance for each value of $k$ in the counting task.\n\n*DINO:*\n| k | Acc. | RMSE | MAE |\n|---------|-------|-------|-------|\n| 1 | 0.267 | 1.749 | 1.313 |\n| 3 | 0.284 | 1.753 | 1.292 |\n| **5** | 0.285 | 1.692 | 1.255 |\n| 10 | 0.277 | 1.742 | 1.293 |\n\n*DINO-HA:*\n| k | Acc. | RMSE | MAE |\n|---------|-------|-------|-------|\n| 1 | 0.271 | 1.706 | 1.331 |\n| 3 | 0.288 | 1.701 | 1.284 |\n| 5 | 0.289 | 1.694 | 1.251 |\n| **10** | 0.290 | 1.708 | 1.261 |\n\n**Code release**\n\nOur code will be fully open-sourced along with the release of our paper. \n\nWe also appreciate the reviewer’s note of a missing reference and will address this in revision. \n\n[1] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8U53HJWQrr", "reviewer": "Reviewer_LKEh", "summary": "This paper aligns representations with human perception on mid-level semantics to improve performance on various downstream tasks. Specifically, it does this by pre-training models on additional synthetic image triplets, where the visual similarity within each triplet is annotated by human subjects.", "questions": "- Regarding dense prediction tasks, there are cases where pre-training with human alignment impairs performance, e.g. DINOv2(-HA) on VOC, etc. The authors stated that DINOv2 has seen these data, and considering that DINOv2-HA has also seen these data, how does this explain the performance drop? Since the only difference in the dataset for models with or without HA should be NIGHTS, this needs clarification.\n- It seems odd that the performance change from using ImageNet and THINGS for pre-training is reversed in Figure 8 for the counting task, and pre-training on THINGS looks extremely bad for retrieval tasks. Can the authors provide an explanation for this phenomenon?\n- Compared with previous work [1], the authors use additional patch-level alignment during pre-training. Isn't there some redundancy between the average pooling of patches and the [CLS] token? How does patch-level alignment affect performance? Additionally, since some patches from negative samples and reference images are similar, it seems counterintuitive to push them away in the embedding space.\n\n    [1] *\"Improving neural network representations using human similarity judgment,\"* Oh et al., NeurIPS 2023", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The dataset this paper used for additional pre-training is rather small compared to the standard datasets, yet authors show that there is still a noticeable improvement on multiple tasks.", "weaknesses": "- Lacks general instructions on when aligning with human perception would benefit representation learning. From the title, I would expect a series of studies providing insights into which levels of alignment benefit or impair different types of tasks. The empirical study in its current version is not comprehensive enough to compensate for the lack of theoretical novelty compared to previous work [1].\n\n    [1] *\"Improving neural network representations using human similarity judgment,\"* Oh et al., NeurIPS 2023\n\n- It would be better if the authors could provide some empirical analysis to give insights into why mid-level alignment is better than other levels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We address questions and concerns below.\n\n**Which levels of alignment benefit or impair different tasks?**\n\nTo strengthen our empirical results, we extend our comparison of different “levels” of alignment (i.e. fine-tuning on THINGS, BAPPS, ImageNet) to the majority of tasks evaluated in the paper: counting, retrieval, segmentation, depth estimation, and a selection of natural/specialized/structured datasets from VTAB. We refer the reviewer to section 1.1 of the global response for our new results and observations. We also examine how the *strength* of alignment (i.e. number of training steps) impacts performance, in section 1.2 of the global response.\n\nTaken together with our paper, we glean the following insights:\n\n- Finetuning on NIGHTS generally benefits representations over base models on object counting, segmentation, depth estimation, instance retrieval, RAG, and some structured classification datasets (such as smallnORB).\n- Fine-tuning on perceptual datasets that are solely low-level (BAPPS) or high-level (THINGS) typically performs worse than fine-tuning on NIGHTS, and in many cases worse than the base models themselves. This demonstrates that representation quality depends on the type of perceptual data, and in fact some forms of perceptual alignment can harm representations.\n- Perceptual alignment impairs performance on most natural classification datasets, particularly fine-grained. This phenomenon is consistent across perceptual datasets; we discuss potential reasons in section 6 of the paper. Low-level alignment (BAPPS) seems to best preserve base model performance in this case. \n- A small amount of alignment may yield the most benefits; training for >1000 timesteps appears to cause a decline in retrieval performance. \n\n**Why is mid-level alignment better than other levels?**\n\nWe address this question directly in our global response, section 3. Please let us know if further detail or follow-up is needed.\n\n**Performance drop on dense tasks.**\n\nThe reviewer is correct that NIGHTS is the only difference for models with/without HA. We flag downstream datasets that the pretrained model has seen because if a dataset is already in-distribution for a backbone, fine-tuning on different data may be more likely to change the feature space such that that dataset is more out-of-distribution. We do not see this as the sole explanation for the results, simply a possible factor. We will clarify this in our revision. \n\n**Why does THINGS hurt retrieval?**\n\nThe triplet choices in THINGS reflect coarse-grained/high-level semantic similarity, i.e., the concepts that humans use for judging object similarity [1, 2]. Thus, THINGS is ill-suited to retrieval tasks (of which counting is one) because they rely on the local/fine-grained similarity structure of representations, rather than the global/coarse-grained similarity structure. Muttenthaler et al. [3] found that learning a linear transform to match the THINGS triplet odd-one-out choices on top of ImageNet-trained models deteriorates downstream performance for tasks where local similarity structure is important, such as few-shot learning on single domain datasets. The authors observed that changing a model’s representation space without preserving the local similarity structure of the pretrained representations decreased few-shot learning performance, and harmed their nearest neighbor structures. \n\nFu et al. [4] further found that fine-tuning vision models on NIGHTS, which better reflects human local similarity structure, leads to decreased performance on THINGS. This indicates that the concepts captured in NIGHTS – which are helpful for retrieval – are different from those useful for THINGS.\n\n**Questions on patch-level training**\n\nWe provide further details and intuition regarding patch-level training in section 2 of the global response. Below we address specific questions:\n\n*Isn’t there redundancy between the average pooling of patches and the CLS token?*\n\nWhile there is some redundancy among patch and CLS features, they do not encode identical information as the former is a uniform pooling over the image space, and the latter does not have such a constraint (previously, Zhai et al. [5] train the SigLIP model and use attention-pooled patch tokens as their global embedding instead of using a separate CLS token). In our work, we take the simplest approach to include all feature information available to tune on, and we find it effective for dense prediction tasks.\n\n*Since some patches from negative samples and reference images are similar, it seems counterintuitive to push them away in the embedding space.*\n\nWe agree with the reviewer; this was our intuition for supervising on the average-pooled patches, rather than using a dense contrastive loss. Average pooling allows us to supervise on a global representation, but also propagates that supervision to the patch tokens, enabling evaluation on dense tasks. In preliminary experiments, we found that patch-level alignment without pooling led to poor results, likely for this reason. \n\n[1] Hebart, M.N., Zheng, C.Y., Pereira, F. et al. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nat Hum Behav 4, 1173–1185 (2020).\n\n[2] Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., & Kornblith, S. (2022). Human alignment of neural network representations. In ICLR 2023.\n\n[3] Muttenthaler, L., Linhardt, L., Dippel, J., Vandermeulen, R. A., Hermann, K., Lampinen, A., & Kornblith, S. Improving neural network representations using human similarity judgments. In NeurIPS, 2023.\n\n[4] Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, Tali., and Isola, P. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023.\n\n[5] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" In ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CovHVc0vJE", "reviewer": "Reviewer_rQrx", "summary": "This paper investigates the effects of aligning pretrained vision models to human judgments. Image-level and patch-level learning objectives are proposed for fine-tuning the pretrained models. The experimental results demonstrate that fine-tuning pretrained models (such as CLIP, DINO, and DINOv2) on the NIGHTS dataset leads to performance improvements in various downstream tasks, including semantic segmentation, depth estimation, retrieval-augmented generation (RAG), counting, and instance retrieval. Ablation experiment regarding the fine-tuning dataset is conducted to demonstrate the perceptual qualities embedded in the NIGHTS dataset.", "questions": "Please refer to the weaknesses.", "limitations": "The authors discuss some of the limitations of their work in Section 5. But I would like them to consider some of my concerns above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed perceptual alignment process is simple yet effective, leading to performance improvements across various downstream tasks.\n2. This paper discussed the effects of aligning vision models using datasets with different levels of variations, providing emprical evidence for further research.", "weaknesses": "1. This paper demonstrates the effects of perceptual alignment by showing the performance improvement in downstream tasks. However, the presented evidence cannot fully explain \"how\" perceptual alignment affects model features as general-purpose representations. Specifically, it is unclear what the difference is between the features before and after perceptual alignment, and why such differences lead to performance improvements.\n\n2. This paper investigates the effects of aligning vision models using dataset with mid-level variations leads to a better “general-purpose representation”. However, the reasons why using datasets with such mid-level variations (instead of datasets with high or low levels of variations) for alignment results in better representations remain unclear.\n3. Experiments in this paper are conducted using ViT based models. However, the effects of perceptual alignment for CNN based models are not explored.\n4. This paper compares the effects of aligning vision models using datasets with low, mid, and high levels of variations on counting and instance retrieval tasks. However, to better demonstrate the high perceptual qualities embedded in the NIGHTS dataset, comparisons should be conducted in a wider range of downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We address questions and concerns below.\n\n**How does perceptual alignment affect model features?**\n\nWe emphasize that this paper aims to answer this question in terms of the competency of representations. There is a rich precedence of understanding and comparing representations via their competencies at downstream tasks [1-2]. Evaluating general-purpose features on transfer tasks – particularly with simple methods such as KNNs and linear probes – allows us to quantify what the features capture and make decodable.\n\nWe further flesh out our empirical evaluation in the global response, section 1.1, where we additionally compare different levels and types of perceptual alignment across segmentation, depth estimation, and classification. We hope that our additional evaluations provide insight into which datasets and tasks are represented in a useful way by finetuned models. We also provide further discussion of the learned feature space in section 3.\n\nWe finally note that while we probe representations in terms of competency, understanding them in terms of their mechanism is an important direction for future work. \n\n**Why is mid-level alignment better than other levels?**\n\nWe address this question directly in our global response, section 3. Please let us know if further detail is needed, or there are follow-ups.\n\n**Perceptual alignment for CNNs**\n\nWe thank the reviewer for their suggestion. We assess the effects of perceptual alignment with NIGHTS on two popular CNNs: ResNet50 and ConvNeXt-B, using the same loss described in the paper*. We evaluate on counting and instance retrieval tasks; our results are shown below:\n\n**Counting (Clevr-Count):**\n| Model | RMSE | MAE |\n|---------|-------|-------|\n| ConvNeXt| 2.045| 1.522|\n| ConvNeXt-HA|**1.631**| 1.193|\n| ResNet | 3.140| 2.551|\n| ResNet-HA | **1.729**|  **1.282**|\n\n**Instance Retrieval (DF2)** **:\n| Model | Top-1 | Top-3 | Top-5| \n|---------|-------|-------|-------|\n| ConvNeXt| 2.12| 3.56| 4.59 |\n| ConvNeXt-HA|**2.81**|**4.8**| **5.98**\n| ResNet | 0.018 | **0.12** | 0.14 |\n| ResNet-HA | 0.018 | 0.074 | **0.17**|\n\n\n*Due to time constraints we were unable to implement LoRA tuning for the CNNs, and instead trained MLPs on top of the final-layer improvements. Previous work [3] indicated training on NIGHTS using MLPs can steer alignment in the right direction but may be less effective; nonetheless we still see downstream improvements using this method. \n\n**We report results for both ConvNeXt and ResNet, however acknowledge that the accuracy numbers for ResNet are likely too small to draw conclusions from. \n\n**Comparisons against other perceptual datasets.**\n\nWe thank the reviewer for their suggestion. In addition to our comparison on counting and instance retrieval, we further evaluate models trained on BAPPS, THINGS, and ImageNet triplets on a wider range of downstream tasks: segmentation, depth estimation, and a and a selection of natural/specialized/structured datasets from VTAB. Our full results can be found in the global response, section 1.1, and we will include these in our revision.\n\n[1] Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. ArXiv, abs/2312.17742, 2023.\n\n[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICLR, 2021\n\n[3] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5w9inkhIFU", "reviewer": "Reviewer_VQay", "summary": "The paper investigates the benefits of aligning vision model representations with human perceptual judgments to improve their performance across various computer vision tasks. The study fine-tunes state-of-the-art models using a dataset of human similarity judgments and demonstrates that this alignment enhances performance in tasks like semantic segmentation, depth estimation, and instance retrieval. The results suggest that integrating human perceptual knowledge as an inductive bias can improve vision model representations without significantly sacrificing performance on other tasks, including specialized domains like medical imaging and 3D environments.", "questions": "How does the patch-level objective differ from the image-level objective? Does it show different performance on various downstream tasks?", "limitations": "Introduced in Section 5.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The experiment evaluation is comprehensive. The paper evaluates the impact of human perceptual alignment on a wide range of computer vision tasks, including semantic segmentation, depth estimation, instance retrieval, and counting. The paper also use multiple state-of-the-art models (e.g., CLIP, DINO, DINOv2, SynCLR) and a detailed experimental setup.\n\n2. The idea of introduces alignment for vision models with human perceptual judgments is reasonable and innovative.\n\n3. The analysis is insightful. The paper provides a nuanced discussion on the benefits and limitations of human perceptual alignment, offering insights into how different levels of perceptual judgments (low-level, mid-level, high-level) impact model performance on various tasks.", "weaknesses": "Writing Part:\n1. There is a question mark in a citation on line 95.\n2. The order of the references is shuffled; reference pages usually come before the supplementary files.\n3. The paper lacks a conclusion section.\n\nSuggestions for Experiments:\n1. More experiments on advanced vision-language models (VLMs) are expected, such as Llava, MiniGPT-4, and instructBLIP.\n2. While not necessarily a weakness, it would be interesting to see the performance improvement on CLIP-blind image pairs, as introduced in the paper \"Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments. We are glad that the reviewer found our evaluation to be comprehensive, the paper idea innovative, and the analysis insightful. We address questions and concerns below.\n\n**Writing comments**\n\nWe thank the reviewer for their helpful notes and will fix the citation typos in revision. We intend for the “Discussion” section at the end of the paper to serve as the conclusion, and can revise it to clarify this – and expand upon our conclusions – in revision. \n\n**Further experiments on VLMs**\n\nWe thank the reviewer for the suggestion. In preliminary experiments, we found that LLaVA, MiniGPT-4, and instructBLIP were not well suited to in-context prompting (potentially due to their instruction tuning and optimization for other tasks). However, we replicated our RAG experiment on IDEFICS2, a recently released 8B multimodal model achieving state-of-the-art results across several multimodal benchmarks [2]. Our results are shown in Fig. 1 of the PDF attached to our global response. We find that across the same four datasets evaluated in Section 4.2 of the paper, performance consistently improves when using NIGHTS-tuned models in the RAG pipeline. The exceptions are DINOv2 with the Diabetic Retinopathy dataset and DINO/DINOv2 for the SVHN dataset. Overall, these results support and validate our results on OpenFlamingo; we will add them to our revision. \n\n**CLIP-Blind image pairs**\n\nWe thank the reviewer for suggesting the MMVP-VLM benchmark [2]. Unfortunately, evaluating on this benchmark requires computing the similarity of text-image pairs. We only fine-tuned the CLIP vision encoder; thus it no longer shares the same feature space as the text encoder, making it difficult to fairly run a multimodal evaluation and interpret any positive or negative results. \n\n**Clarification on patch-level objective**\n\nWe provide further details and intuition regarding patch-level training in section 2 of the global response. Below we address specific questions:\n\n*How does the patch-level objective differ from the image-level objective? Does it show different performance on various downstream tasks?*\n\nFinetuned patch tokens were necessary to evaluate NIGHTS fine-tuning on dense tasks (depth estimation, segmentation). In preliminary experiments, we found that the image-level objective did not induce any significant changes in the patch tokens. The patch-level objective is functionally the same as the image-level objective, the only difference being that it propagates the similarity annotations more directly to the patch tokens. \n\nNote that only the fine-tuned patch tokens are needed for segmentation/depth-estimation, and only the CLS token for image-level tasks (all others).\n\n[1] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024b.\n\n[2] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” CoRR, vol. abs/2401.06209, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful questions and feedback. We are glad they found:
* The paper is clear, well-written, and easy to follow. [kSUr, 9gB3]
* The experiments are comprehensive. [kSUr, 9gB3, VQay, rQrx]
* The analysis is insightful and interesting. [9gB3, VQay]
* The paper studies a simple but interesting scientific question worth investigating [kSUr, 9gB3]; the key method is convincing and effective [9gB3, rQrx, VQay]

We present key results and details here, and respond to individual questions in reviewer-specific responses.

**1. What are the benefits/drawbacks of different levels of perceptual alignment?**

**1.1 Ablating perceptual datasets**
We extend our dataset ablation to our full range of tasks. We LoRA-tune DINO on triplets from BAPPS, THINGS, and ImageNet, using the procedure from Section 4.5. We emphasize that fine-tuning on these datasets ablates the *level* and *type* of perceptual alignment. BAPPS contains judgments of low-level distortions, THINGS contains semantic-level distortions, and ImageNet contains no perceptual judgments, instead grouping images by semantic category.

We evaluate models tuned on these datasets on segmentation, depth estimation, and a VTAB subset. Results are in Fig. 2-4 of the attached PDF. We observe:
- For all dense tasks, training on NIGHTS outperforms all other datasets. The dataset ranking varies across evaluations, but NIGHTS-tuned models consistently transfer best.
- For many tasks, training on ImageNet triplets or the base model, outperforms THINGS or BAPPS. This indicates that the types of perceptual judgments in NIGHTS specifically are helpful, whereas training solely on low-level or high-level variations may harm representations.
- On VTAB datasets, base models perform best. The exception is sNORB (pose prediction), for which NIGHTS is best. Amongst perceptual datasets, NIGHTS is sometimes outperformed by BAPPS/ImageNet.

**1.2 Ablating training time**

We further evaluate how the *strength* of alignment – i.e. training loss when finetuning on perceptual datasets – impacts features. We evaluate DINO checkpoints tuned for increasing # steps on NIGHTS/BAPPS/THINGS/ImageNet on instance retrieval. Results are in Fig. 5 of the PDF. We observe:
- Tuning on NIGHTS outperforms other datasets over the full training trajectory.
- Performance rises significantly with a small amount of alignment to NIGHTS, however it consistently trends down after >1000 steps. This indicates that a small amount of alignment is helpful for the downstream task, however overfitting to human preferences may be harmful.

**2. Clarification on patch token training**

For dense prediction, we modify the original training objective for stronger supervision on patch tokens, which are output alongside the CLS token and are the basis for our segmentation and depth maps. Although supervising only on the CLS token can still modify patch tokens (as tuning model weights affects all feature outputs), our initial experiments showed that the CLS-based objective was too global to change the patch tokens in any impactful way. Thus, we switch to a loss more directly tied to local features.

In more detail: our local objective only differs from the global objective in how the feature is extracted: instead of computing $L(\texttt{CLS}_A,\texttt{CLS}_B)$, we compute $L(cat[\texttt{CLS}_A, pool(\texttt{PATCH}_A)], cat[\texttt{CLS}_B, pool(\texttt{PATCH}_B)])$. $\texttt{CLS}$ is of dimension $(1, d)$, $\texttt{PATCH}$ is $(s, s, d)$ where $s$ is the number of patches along each spatial dimension, and we spatially average the patch tokens to get dimension $(1, d)$. We then concatenate the CLS and pooled patch tokens to get dimension $(1, 2d)$.

Only our experiments in 4.1 (semantic segmentation and depth estimation) use this patch objective, as they are the only ones that require local features; all other applications in the paper are evaluated with CLS tokens (original objective detailed in 3.2).
We appreciate the reviewer feedback to clarify this, and will do so in our revision.

**3. Why does tuning on NIGHTS outperform high/low-level datasets?**

We hypothesize that variations found in both BAPPS and THINGS are restricted to solely high- or low-level, whereas the mid-level judgments in NIGHTS cover some measure of both (see Fig. 11 in paper for examples of the difference). Previous work [1] has found that similarity judgments by perceptual metrics trained on BAPPS correlate better to low-level metrics such as color than semantic attributes. THINGS contains solely semantic distortions, reflecting the concepts humans use to judge object-level similarity rather than visual concepts.

In contrast, NIGHTS contains a broad set of visual variations including style, pose, color, and count. Previous work [1] has found that models fine-tuned on NIGHTS seem to attend to both low-level attributes and semantic attributes . Aligning a feature space to these concepts may be useful for visual tasks requiring some semantic knowledge, such as retrieval, counting, segmentation, etc. This hypothesis may also explain why tuning on NIGHTS hurts performance on fine-grained tasks, in which images that are perceptually similar may belong to different categories.

The main contribution of our paper is to empirically identify how tuning on mid-level variations, in comparison to other perceptual datasets, impacts transfer performance across a wide variety of downstream tasks. Further understanding the mechanism of the respective learned representations is a rich avenue for future work.

We greatly appreciate that the reviewers see perceptual alignment as an interesting scientific question worth investigating. We hope that our paper encourages research in this emerging topic and sparks fruitful discussion.

[1] Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, Tali., and Isola, P. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023.

Meta-review: More perceptual alignment with human similarity judgments benefits visual representations.

Four out of five reviewers recommend accepting the paper (6, 8, 5, and 6) while one reviewer LKEh with 4 did not give follow-up feedback on the authors' responses after the rebuttal period. After reading the paper, the reviewer's questions have been adequately addressed. 

Overall, the paper is well-written and easy to follow. As commonly acknowledged by the four reviewers, the idea of perceptual alignment with human similarity judgments is interesting. The experiments are comprehensive with many different types of tasks and model backbones. The results are strong and convincing. The experiment designs address the claims on the alignment very well. 

Please ensure the reviewers' comments are incorporated into the final version.
Decision: Accept (poster)