Evaluating alignment between humans and neural network representations in image-based learning tasks
Authors: Can Demircan
Tankred Saanum
Leonardo Pettini
Marcel Binz
Blazej M Baczkowski
Christian F. Doeller
Mona M. Garvert
Eric Schulz
Venue: NeurIPS 2024 poster (2024)
Keywords: human alignment
neural network representations
generalization
function learning
decision-making
Primary area: neuroscience_and_cognitive_science
TL;DR: We compare humans and various pretrained neural networks in image-based learning tasks and identify factors that make neural networks perform more human-like.

Abstract: Humans represent scenes and objects in rich feature spaces, carrying information that allows us to generalise about category memberships and abstract functions with few examples. What determines whether a neural network model generalises like a human? We tested how well the representations of $86$ pretrained neural network models mapped to human learning trajectories across two tasks where humans had to learn continuous relationships and categories of natural images. In these tasks, both human participants and neural networks successfully identified the relevant stimulus features within a few trials, demonstrating effective generalisation. We found that while training dataset size was a core determinant of alignment with human choices, contrastive training with multi-modal data (text and imagery) was a common feature of currently publicly available models that predicted human generalisation. Intrinsic dimensionality of representations had different effects on alignment for different model types. Lastly, we tested three sets of human-aligned representations and found no consistent improvements in predictive accuracy compared to the baselines. In conclusion, pretrained neural networks can serve to extract representations for cognitive models, as they appear to capture some fundamental aspects of cognition that are transferable across tasks. Both our paradigms and modelling approach offer a novel way to quantify alignment between neural networks and humans and extend cognitive science into more naturalistic domains.

Evaluating alignment between humans and neural network representations in image-based learning tasks
Can Demircan 1,2. Tankred Saanum? Leonardo Pettini
Marcel Binz
Blazej M Baczkowski4.5
Christian F Docller 4,67,88 Mona M Carvert 4,910
Eric Schulz
Institute for Comann-Centered AI, Helmholtz Computational Health Center, Munich, Germany
Max Planck Institute for Biological Cybernetics Tübingen, Germany
Max Planck School of Cognition Leipzig, Germany
1 Max Planck Institute for Human Cognitive Brain Sciences Leipzig, Germany
University of Tübingen Tübingen, Germany
Kavli Institute for Systems Neuroscience Trondheim, Norway
Leipzig University Leipzig, Germany
Technical University Dresden Dresden, Germany
Julius-Marn Wwwa--aaannllianniieeee Würzburg Würzburg, Germany
Max Planck Institute for Human Development Berlin, Germany @ @elmmoltz-munichh.ee
Abstract
Humans represent scenes and objects in rich feature spaces, carrying information that allows generalise about category memberships and abstract functions with few examples. What determines whether a neural network model generalises like human? We tested how well the representations of 86 pretrained neural network models mapped to human learning trajectories across two tasks where humans had learn continuous relationships and categories of natural images. these tasks, both human participants and neural networks successfully identified the relevant stimulus features within few trials, demonstrating effective gener- alisation. We found that while training dataset size was core determinant of alignment with human choices, contrastive training with multi-modal data (text and imagery) was a common feature of currently publicly available models that predicted human generalisation. Intrinsic dimensionality of representations had different effects on alignment for different model types. Lastly, we tested three sets of human-aligned, representations and found no consistent improvements predictive accuracy compared to the baselines. In conclusion, pretrained neural networks can serve to extract representations for cognitive models, as they appear to capture some fundamental aspects of cognition that are transferable across tasks. Both our paradigms and modelling approach offer a novel way to quantify align- ment between neural networks and humans and extend cognitive science into more naturalistic domains.
Introduction
Research on representational alignment between neural networks and humans has gained significant attention in recent years [1, 2]. Comparisons across the systems have provided important insights into neural network representations [3, 4], human cognition and the brain [5, 6, 8, 9], and the development of more robust machine learning systems [10, 11. 12]. In the sensory domain, the comparisons have been predominantly made through two families of behavioural tasks. One common approach is to compare object recognition performance across humans and neural networks [13]. This
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A
Experiment 1: Category Learning
C
Moteliktificalllett
Wrong!
This Image was meant for Folty!
Foodrelatedleainngee hww.://eeelingeeteeeketteeeeeeee heeerkkithhe--elttee
Experiment 2: Reward Learning
8
less
rewirdd
Task descriptions. (A) An example trial from the category learning task, where an incorrect decision is made. (B) An example trial from the reward learning task where the best option is chosen and highlighted in orange. (C) Example images from the THINGS database [30]. The database has low dimensional semantically interpretable embedding [27]. which is derived from human similarity judgements. The example images are placed in the most three prominent dimensions of this embedding. In both tasks, participants were randomly assigned to one of these three dimensions.
Figure 1:
The associated category membership and rewards for the two tasks are displayed.
a fruitful approach for understanding if the two systems use the same features for object recognition [14, 15, 16]. are susceptible similar distortions [17. 18, 19, 20], and struggle with similar images [21]. Another common approach is to use similarity judgement tasks, which may entail reporting pairwise similarity scores [22, 23, 24], arranging stimuli in a 2D space based on their similarity [25, 26), or choosing the odd-one-out in triplets of stimuli [27, 28]. Using these tasks, previous work has identified the factors that contribute to neural networks representing stimuli similarly to humans, both in low-level perception [29] and semantic judgements [3].
However, similarity judgements do not begin to capture the complexity of tasks humans use their representations for. Humans rely on rich representations for making judgements and acting in the world. For example, an apple has multitude of features, such colour, taste, shape, and brand. Depending on the context, people can use these features and make predictions about the apple's s taste, the environmental impacts of growing it, or the significance of it in different mythological and religious settings. What determines whether a neural network model represents an object like an apple with the same richness and flexibility?
In this work, we investigated people's ability to learn functional relationships on naturalistic images in a few-shot setting, and what neural network models best predict human choices. We adapted two commonly used learning paradigms from the cognitive psychology literature: category learning (Fig. IA) and reward learning (Fig. IB). However, instead of using repeating artificial stimuli, we presented human participants with unique naturalistic images sampled from the THINGS database [30] in each trial, requiring them to continuously generalise. To understand whether neural networks contain sufficiently rich representations that allow for such generalisation, we tested 86 different neural networks [27, 31, 32, 33, 34, 35, 11, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 10, 12, 52, 53, 54, 55]. These networks varied in their loss function, training diet, and the modality of training data. In summary, we found that:
While almost all pretrained models generalised above chance level and predicted human behaviour in both naturalistic learning tasks, contrastive language image pretraining (CLIP) [45] consistently yielded the best predictions of human behaviour. We furthermore showed that this could not fully attributed to the training diet alone.
Multiple factors were important for human alignment, including task performance, model size, training diet, separation of different classes in representations, and the similarity of the representations the generative embedding of the task.
Of the tested human-aligned neural networks, no method consistently improved human alignment in our tasks compared to non-aligned baselines. However, two of the methods (Harmonization [11] and gLocal [10]) yielded improvements in task accuracy average.

---

Human
Task
Supervised
Self-Superriied
Multimodal
Language Reward Learning
Category Leaming
Reward Learning
C
Category Learning
p(Comrect
0.5
0.0
80
120
20
40
60
80
120
20
60
Trial
Trial
Trial
Trial
Figure 2: Learning trajectories human participants and neural networks. Neural networks can perform as well as humans. (A & B) Accuracy of human participants across trials for the category and the reward learning tasks respectively. Shaded lines indicate 95% confidence intervals. (C D) Example learning curves for the neural network representations in the category and the reward learning tasks respectively. The best-performing models from each model type are shown.
2 Experiments
We design our experiments around naturalistic images from the THINGS database [30, 27J. Each image in the database depicts a collection of entities (animals or objects) and comes with an embedding with 49 human interpretable features, which was built by Hebart et al. [27] predict human similarity judgements of these objects. Each feature reflects a semantically meaningful property such as whether an image contains metallic objects, food, animals In our experiments, humans learned functions defined over these individual embedding dimensions. We chose category learning and reward learning experiments, as they are well-established paradigms test function learning and generalisation in human participants. However, unlike traditional paradigms, we used naturalistic images and no images were repeated, requiring generalisation.
Category learning:
Human participants (n = 91) completed 120 trials of an online category learning presented with novel image in each trial. They
task, where they images to one two dinosaurs, Julty or Folty, using key presses. Participants were told that the two dinosaurs had completely non-overlapping preferences for what gifts they enjoyed. After each trial, we gave participants feedback on whether their choice of delivery was correct. An example trial from the task shown in Fig. 1A. Participants were assigned to one of three conditions, where in each condition the category boundary was defined over a different THINGS embedding dimension. The three chosen dimensions map how metallic, food-related, and animal-related the shown image is (Fig. IC). For instance, in one condition non non-metallic images should be classified to Folty, and metallic images to Jolty. For each participant, 120 unique stimuli from the THINGS database were sampled. A median split over the assigned feature of the sampled stimuli determined the category boundary.
asked to deliver these
Reward learning: Human participants (n = 82) completed 60 trials of a reward learning paradigm [56], in which they were asked maximise their accumulated reward throughout the task. In each trial, participants were presented with two images and were asked to select one using key presses. After making a choice, the associated reward with each option was shown. task is shown in Fig. 1B. Participants were assigned to one of the three conditions, was done in the category learning task. Stimuli were sampled the same way as the category learning task. For each participant, the values of the task-relevant feature were re-scaled linearly between 0 and 100. Additional details about the experimental paradigms are described in Appendix A.
example trial from the
3 Behavioural analyses
Humans learn to generalise quickly. The learning curves of the participants are shown in Fig. 2A and 2B. To measure whether and how fast people learned in the two experiments, we analysed their choice data using mixed-effects logistic regression models. In the category learning task, we predicted whether a participant made the correct choice using an intercept and the trial number. We found that participants performed this task above chance level, indicated by a significant
3

---

0.15
A
Task Supervised
Self-Superiiee
Language
McFadden's R
Multimodal
Human Aligned
0.10
0.05
0.00
( thi bouu
Besh OONg Olig
phars
5.55 100 PortEna
di
0.15
McFadderiss
0.10
0.05
0.00
1. owe porne
of (ottoo
Sall venee
did 19Ne
Figure 3: Model fits to human choice data. In both category learning (A) and reward learning tasks (B), several CLIP models predict human choices the best, even better than the generative features of the tasks. How well the models fitted human choice was more heterogeneously distributed for supervised, seperssue and language models. Plotted are the cross- validaad McFadden's 's R² of each representation for the category learning and the reward learning tasks respectively. Higher values indicate better fits to human behaviour. 0 marks the alignment of random model.
intercept (3) = 1.14 0.09, 2 13.18, P < .001), and that their performance improved over trials (3 = 0.32 0.05, 2 6.89, < .001), indicating learning effect. This suggests that people can very efficiently extract the relevant feature dimension in high-dimensional naturalistic environments despite seeing each stimulus only once. For the reward learning task, we predicted whether participant chose the image on the right using the reward difference between the two images, the trial number, and the interaction of the two predictors. We found that the reward difference ( = 0.89 + 0.07, = 12.56, P < .001), and the interaction of this difference with the trial number (3 = 0.34 = 0.04, = 9.30, P < .001) predicted choice, again indicating a learning effect. We further characterise how quickly humans learn the task in Fig. 10 in Appendix C and provide the full specification of the mixed-effects models in Appendix A.
4 Model-based analyses
To understand what kind of representations are needed to predict human choices, we tested represen- tations extracted from several pretrained neural networks on our tasks.
Most representations predict human choice above chance level. CLIP makes the best predictions. The representations were extracted from the penultimate layer if the models had a classification layer, and from the final layer otherwise. For the transformer models, the [CLS] token representations were extracted. To extract representations from language models, we provided them with the prompt A photo of X where X was the category label of the task image. fastText was only provided with the category label instead.
We trained linear models to predict either reward or category membership from each neural network model's extracted representations. The models were provided with image-target pairs until trial

---

t- 1 as training data and made predictions for the image trial t. For the category learning task, we used an lg regularised logistic regression model, and for the reward learning task, we used Bayesian linear regression model with spherical Gaussian priors. We used the estimates from the linear models to predict participant choice using mixed-effects logistic regression in leave-one-trial- out cross-validation. For the category learning task, we regressed the probability estimates of the logistic regression models onto participant choice. For the reward learning task, we regressed the reward estimate differences between the left and the right options onto choice. Example learning curves for the two tasks are shown Fig. 2C and Fig. 2D, Finally, we measure alignment to human choices using McFadden's 's R [57]. which is computed follows:
Moddel
McFadden's 's R² =
LRmdom
where LModel is the negative log likelihood of given model and LRandom is the negative log likelihood of a random model.
We observed most of the representations we tested can do our task and predict human behaviour above chance level across the two tasks (as visualized in Fig. 3A and Fig. 3B). CLIP models were the top 7 (6) models for the category (reward) learning task in predicting human choices. In total, 16 (7) of the 86 candidate representations predicted participant behaviour better than the ground truth representations that were used to generate the task, Of these 16 (7) representations, 14 (6) were CLIP models. One was a large vision transformer, trained in supervised manner on ImageNet [58]. A human-aligned variant of DINO-v2 provided better fit than the generative task representations in the category learning task. The rest of the supervised and self-supervised vision models, as well as the language models, had heterogeneous distribution in how well they predicted human behaviour. To provide better intuition for how human participants and CLIP were similar, we display example trials where both CLIP and humans make the same incorrect decisions in Fig. 11 in Appendix C.
Which factors contribute to alignment?
Why are CLIP models substantially better aligned with humans in our task? We conducted series of analyses to better understand which model properties contribute to alignment. We pooled the data across the two tasks and excluded the language models from all analyses except those shown in Fig. 4A and Fig. 4E, as comparing other properties across vision and language models (e.g. model size) is not meaningful. We first tested if larger models predicted human choice better. While is common for more expressive models perform better at downstream computer vision tasks [59, 60, 61], previous work has shown that this is not a robust predictor of human alignment [3, 9]. In our tasks, found that larger models predicted human choices better (p 0.48, p < .001, Fig. 4B), which contradicts previous findings. Next, we considered the number of images seen during training, which is predictive of higher accuracy in image recognition [62] and human alignment. In tasks, found that models trained on more images were more predictive of human choices as well (p = 0.52, p 001, Fig. 4C).
Then, we analysed which, if any, properties of the models' representations were predictive of their alignment with human choices. First, we considered how well the THINGS classes were separated in the representations of each model. Following Kornblith et [63], the class-separation was computed as follows:
R² 1
dlvittiin / dooual
(2)
K
N₂
N2
cos(x Xx,
n deoual
K
K
N
N
cos(x
Xk.n)
within
(3)
KN
K2N,Ne
k= m
j= k=
where Xt.m is the representation of image in object class k. K is the total number of classes, and N& is the total number of images in class k. cos(-.)) denotes cosine similarity between representations. The R measure is between O and 1, where higher scores indicate a low within-class distance to across-class distance ratio, ie. high class separation. Previous work has shown a positive link between class separation and image classification [63], as well as recall [64]. Similar to these findings, we found that models that had higher class separation were more predictive of human choices (p = 0.29. p .01. Fig. 4D).
5

---

Task
Supervised
Self-Supervised
Multimodal
Language
p 0.73
0.48
p= 0.52
0.15
Mofacierris R²
0.05
0.00
0.5
0.6
0.7
0.8
10a
10'
10a
10'
Task Accuracy
Model Parameters
Treining Images
D
E
p
0.15
MoFadden's
0.10
0.05
0.0
0.5
1.0
0.0
0.5 CKA Similarity with the Task Embedding
1,0
10
20
30
40
50
Class SeparSenn
Intrinsic Dimensionality
Figure 4: Several factors contribute to alignment. Models trained on more data and with more trainable parameters predict human choices with higher accuracy. Turning representations, those that better separate image classes and are more similar to the generative task features exhibit stronger alignment with human choices.
We then considered whether the similarity of the representations with the generative task features was predictive of how well different representations predicted human choices. For this, we used linear Centered Kernel Alignment (CKA) [65], which computes the similarity between the generative task representations T and neural network representations X as follows:
| T
CKA(T,X)
| |
where ||e denotes the Frobenius norm. We found that representations that were more similar the generative task embedding predicted human choices better (p 0.67, p ,001, Fig. 4E).
Lastly, tested whether the intrinsic dimensionality of representations was related to alignment. Lower in- trinsic dimensionality of neural networks in late layers is positively linked to better classification performance [66]. The degree which network compresses its inputs is also directly linked to its ability to generalize [67, 68, 69, 70, 71]. In the human alignment literature, similar measure named expressed dimensionality has been studied in the context neural eepresentationsHooweee However, diverging from Ansuini et al. [66], one study found neg- ative correlation between alignment and this measure [72], and another study found no link [9]. We used the TwoNN method proposed by Facco et al. [73] to estimate intrinsic dimensionality, which makes use of the nearest neighbour distances. First, we linearly scaled all the features to between 0 and 1. We then computed pairwise distances
Supervised
Self-Supervised
Multimodal
-0.5
0.0
0.5
Intrinsic Dimensionality Alignment(p)
Figure 5: Lower intrinsic dimensional- ity linked with higher alignment most strongly for the multimodal models, and lesser extent with supervised ones.

---

CLIP
SimCLR
CLIP SimCLR
Class- Clase-Seaanin Separateaaannnnnn (R*)
Intrinsic Dimensionallyy
MoFadden's R²
0.10
Task Accuracy
0.7
0.5
OKA Similarity with
Task Embedding
0.5
20
0.05
0.6
0.4
0.00
0.3
0.3
Small
Base
Large
Small
Bane
Large
Small
Base
Large
Small
Base
Large
Small
Base
Large
Figure 6: The effect of CLIP loss while controlling for model size and data. We observed that CLIP loss increases alignment when data size and architecture are controlled. Here plotted are (A) McFadden's R²² (B) task accuracy, (C) class-separation, (D) similarity with the task embedding, and
(E) intrinsic dimensionality across model sizes and loss functions.
for each pair of data points. Then, we calculated = r4/r, where r', and 1'2 are the shortest distances from datapoint i. Later, the empirical cumulative distribution FemP(N) was computed by sorting H4 and normalising by the total data points N. The slope of linear model that maps log μa, to -log 1 --emepp (14) with no intercept gives the intrinsic dimensionality measure.
Pooling over all model types, lower intrinsic dimensionality was significantly associated with align- ment (p- = 00.32, p .03, Fig. 4F). However, found that this relationship was most strongly driven by the multimodal models and to a lesser extent by supervised models (Fig. 5). That input compression and dataset size are positively related to alignment most strongly for CLIP models suggests that the contrastive multimodal training regime unlocks desirable scaling properties in these models. See Fig. 12 in Appendix C for pairwise correlations between the investigated factors.
Are CLIP models well aligned only due to their high data diet?
While we found that models trained with contrastive language image loss predicted human behaviour the best, there remains an important confound. These models are also the ones that are trained on the largest datasets (400M to 2B images). Therefore the direct benefits of multimodal training remain unclear. To address this point, we turned to models provided by Mu et al. [52]. Here, the same models are trained on a large dataset (YFCC15M [74, 45)) using three different losses: i) CLIP loss that penalises for the distance between corresponding pairs of text and image representations ii) a SimCLR [39] loss that pushes the representations of the augmented and the original image close to each other and away from others, and iii) a CLIP + SimCLR loss.
First, we found that CLIP models always fit human data better than SimCLR models, and CLIP + SimCLR models made the best predictions when controlling for model size (Fig. 6A). This suggests that the advantage provided by the CLIP models cannot solely be attributed to the training data. We found the same ranking of models in terms of how well they did the tasks (Fig. 6B). Yet, contrary to our expectations, the SimCLR models had higher class separation than CLIP models (Fig. 6C), as well as better alignment with the generative task features (Fig. 6D), and lower intrinsic dimensionality (Fig. 6D). This was surprising because, in our previous analyses, we found these properties to be associated with models that predicted human choice better. However, there still may be other confounds that impacted the findings. For example, controlling for training data is not straightforward, as text-image pairs may carry more information than augmented versions of the same image, providing unfair advantage to the multimodal models.
Do alignment methods transfer to our learning tasks?
Lastly, we evaluated the performance of models that were explicitly aligned to be more human-like This comparison included three sets of models. Fel al. [11] have aligned models through method called Harmonization. In addition to the standard supervised training, the models are trained to use the same visual features of images that humans use. The second part is achieved by aligning the networks' saliency maps with feature importance maps obtained from human judgment. This results in networks that perform better in ImageNet and that are aligned with humans. Next, et al. [12] have curated carefully created synthetic dataset. They later fine-tuned pretrained
human similarity judgements models such as CLIP using Low-Rank Adaptation [75] to derive a metric named DreamSim that outperforms other models in predicting human similarity judgements. Lastly, Muttenthaler et al.
7

---

Harmonization
Dreamnsi
glocal
0.05
0.00
0.05
0.15
0.00
0.15
0.2 0.0
0.2
0.5
0.0
0.5
20
McFaciden's R²
Task Accurany
Clasa-Separation (R²)
A. C.A Similarit with the Task Embedding
& Intrinsic Dimensionality
Figure 7: We compared models aligned to humans through three different methods against baselines that had the same architecture and that were pretrained on the same data. We found no consistent improvement in human alignment. Here plotted are (A) McFadden's s R², (B) task accuracy, (C) class-separation, (D) similarity with the task embedding, and (D) intrinsic dimensionality across model sizes and loss functions.
[10] have fine-tuned representations of pretrained models through novel transformation named gLocal, which aligns the global representational space to be more human-like by trying to predict human similarity judgements, while preserving the local structure through contrastive loss that encourages the representations to stay close to their original positions. For these comparisons, we used the models openly provided by the authors.
First, we found that none of the alignment methods improved alignment our task consistently, with some instances of Harmonised and gLocal models improving alignment (Fig. 7A). Alignment improved task accuracy on average for Harmonised and gLocal models (Fig. 7B). Class separation was lower for all Harmonization and DreamSim models, whereas it increased for all gLocal models tested (Fig. 7C). We also observed that the similarity between the representations and the task embedding decreased after alignment for Harmonization and DreamSim, but increased in most of the models after gLocaa alignment (Fig. 7D). Lastly, we observed heterogeneous patterns in the change of intrinsic dimensionality across the three alignment methods, with gLocal reducing the intrinsic dimensionality for all but one of the tested models (Fig. 7E).
How do our tasks compare to other alignment measures?
Lastly, to better characterise how our cognitive tasks fit in the alignment literature, we compared them to previously established measures (Fig. 8). We found the strongest correlation with the THINGS odd-one-out judgements [3, 27] (p 0.54 for zero-shot, and = 0.61 for probing). Given the two tasks use the same images, and the ground truth of our tasks was constructed from the odd-one- out judgements, this strong relationship is expected. However, the correlations are still moderate, indicating important differences across tasks. Comparisons with an independent similarity judgement [23] task showed a weaker correlation (p 0.35), and we found no correlation with a fine- tine-grained two-alternative forced choice task [12]. Lastly, we compared alignment in our task alignment on the ClickMe dataset [76], which was used to build the Harmonization models [11]. We observed negative correlation (p- -0.48) here, suggesting that pixel-level alignment and semantically bound global image alignment might be at odds.
Discussion
In this work, we investigated the alignment of neural network representations humans. To study this, we measured how well different neural network representations predict human choices two newly developed learning tasks. Of the 86 tested representations, all but one predicted human choice above chance level. We furthermore identified several important factors for human alignment, such as large model size, training regime, and low intrinsic dimensionality. These results expand on previous work in both human alignment and cognitive modelling. From an alignment perspective, we considered more challenging tasks compared previous studies. Previous work has predominantly focused on simple image exposure and similarity judgments. We believe our findings complement this research by addressing unexplored aspects of alignment, which are generalisation and information integration across an extended horizon. From cognitive modelling perspective, we demonstrated that off-the-shelf pretrained neural networks can serve as representations for cognitive models [77], which allows to push cognitive models into more naturalistic domains.
8

---

Supervised
Seff Supervised Peterson (8) p 0.35
Multimodal a- GL0?
Mumondhaler (2123)
Mutoomhater al (2025) p-0
(2023)
(2022)
p- 0.54
pe 40 A6
Ned Andieeeeee
S
0.10
0.05
0.00
10
50
50
55
0.2
044
80 2AFC Accurscy
0:00
0.25
0.50 0.75
Odid-Orn-- Out Zaro-Shut Ausurwayy
Gell-Grae Probing Acsuracy
Parwisee Simiarty Corrlstion
ChobMettamotion Aigamment
Figure 8: How do our tasks compare to other alignment methods? Our tasks offer similar (but not identical) results with two of the four similarity judgement tasks. There is strong negative relationship with the ClickMe dataset, which focuses on localised pixel-level alignment.
5.1 Related work
How do our findings compare to previous work on alignment? First, previous research has shown that CLIP representations align well with human representations using brain imaging [78, 9, 79] and similarity judgments [3]. In line with this, we also found that the contrastive language-image loss improved alignment when controlling for data size and architecture. We furthermore found that training on large datasets generally improved alignment. Yet, it remains unclear whether supervised training on massive datasets alone can achieve high alignment similar that of CLIP models. For example, Muttenthaler et al. [3] showed that models trained in a supervised manner on the JFT-3B dataset [61] can outperform CLIP predicting human similarity judgments. However, since this is proprietary dataset, we could not make this comparison.
We observed some findings that diverge from previous studies using different experimental paradigms. Muttenthaler et al. [3] and Conwell et al. [9] found no consistent correlation between a model's number of parameters and its alignment with human similarity judgments and visual cortex activity. In contrast, we found that models with more parameters were more predictive of human choices. Another significant divergence is how intrinsic dimensionality relates alignment. Elmooziin and Bonner [72] found that vision models with higher latent dimensionality better predict visual cortex activity, Conwell al. [9] found no correlation. In contrast to this, we observed that lower intrinsic dimensionality led to increased alignment for CLIP models. We hypothesize that both of the observed discrepancies are due the higher cognitive demands required by our tasks, highlighting the importance of studying alignment in more complex settings. That being said, an alternative explanation for the latter discrepancy could be due to differences measuring latent dimensionality. Both Elmoznino and Bonner [72] and Conwell et al. [9] use the squared sum of the eigenvalues of principal components divided by the sum of squares of eigenvalues, assuming representations lie on a linear manifold. However, previous work shows that later layers in vision models lie on a curved manifold [66]. Thus, using principal components might not be the best method for this estimation.
Lastly, we found that a method designed for increasing human alignment, DreamSim [12], actually hurt alignment in our task. On the other hand, ooca [10] and Harmonization [11] improved both performance and human alignment for some models but not all of them. However, the gLocal transform heavily utilises the THINGS dataset, as made use of the triplet odd-one-out similarity judgement data [27], making it difficult to interpret how well it generalises to other settings, Taken together, these results highlight the importance of studying how well different alignment methods transfer across tasks, as we have done in this work.
5.2 Limitations
There are several limitations and extensions of our work that deserve to be highlighted. The main limitation concerns the interactions between factors in the tested neural networks, making it difficult to isolate specific factors. For example, we would like to test the influence of loss function keeping all other factors equal. Ideally, would train all combinations of architectures, model sizes, loss functions, and datasets, but this is computationally infeasible.
While we controlled for factors such as training data size and architecture in our comparison of CLIP to other models, there may still be confounding variables we haven't 't accounted for. For instance, it's not straightforward to compare the information content of image-text pairs used in CLIP training to

---

image-only data used in other models. Text- -ext-image pairs might inherently carry more information than single images, potentially giving multimodal models an advantage that's difficult to quantify. This and other subtle differences in training paradigms could influence our results in ways that are challenging to isolate and measure. Lastly, there can also other families of models that may outperform CLIP models we haven't considered, such video models, generative models, image segmentation models.
We furthermore tested only two experimental paradigms. Future research should explore whether the considered representations predict human behaviour with nonlinear task rules and extend to other paradigms. In particular, one should also consider tasks beyond those generated through the embedding from Hebart al. [27]. Previous work has shown that is possible automatically generate a large set of text- based category learning problems using large language models [80]. It might be interesting to test whether these methods can extended to generate tasks involving visual stimuli and use these tasks to test whether our findings generalise to wider setting.
Finally, we only measured human alignment by looking at behaviour. However, fully confirm our results, it would also be important to investigate the alignment to brain data. Hence, future work should replicate our experiments in an MRI scanner and compare the representations of neural networks to people's ' brain activity.
5.3
Conclusion
The findings presented in this work have implications both for machine learning and cognitive science. For machine learning, our task and modelling approach offers new way to measure the human alignment of neural network representations and use this metric while building human-aligned neural networks. Alignment at this level can pave the way for artificial systems that can generalise semantically rich tasks, making them more robust and powerful. For cognitive science, our findings create the opportunity to study several other problems in naturalistic settings by showing that people can do learning tasks with naturalistic stimuli and that pretrained neural networks can be used to extract representations for cognitive models. This could open up the door for whole new cognitive science that uses naturalistic tasks and environments and thereby increase the validity of the cognitive sciences more generally.
Acknowledgments and Disclosure of Funding
This work was funded by the Max Planck Society, the Volkswagen Foundation, as well as the Deutsche Forschungsgemeensccaatt (DFG, German Research Foundation) under Germany's Excel- lence Strdee--EXC2266/11399777665
10

---

References
[1] Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Korbllihh, Sunayana Rane, Talia Konkle, Thomas O'Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Robert Robert Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment, 2023.
[2] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj. Rishi Rajalingham, Elias Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy. Franziska Geiger, Kailyn Schmidt, Daniel L.. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, 2020. doi: 10.1101/407007. URL httpps//wwet http://wwrrgoottneeeeeeeeeeeeeee biorxiv. org/ wwwttnntttetecoccccmmoomoo wttpeeeet/raaae//////ee/// 00120/01/02/407007
[3] Lukas Muttenthalerr Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. In The Eleventh International Conference on Learning Representations, 2023. URL https: //op://////eeeeiee/t net/forum? id-ReDQ10UQRoX.
[4] Ilia Sucholutsky and Tom Griffiths. Alignment with human representations supports robust few-shot learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 73464-73479. Curran Associates, Inc., 2023. URL https:: nwurips. wewprips.cc/ppppr_ffllee pdf.
[5] Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep Supervised, but Not Un- supervised, Models May Explain IT Cortical Representation. PLOS Computational Bi- ology. (0(11):e100391 November 2014. ISSN 1553-7358. 1003915. URL https: petpss//jit j/unnnals plos.
doi: 10.1371/journal.chh wtt.:scommboo/aattclleeoooo burnal.pcbll 1003915. Publisher: Public Library of Science
[6] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C.
Frank, James J. DiCarlo, and Daniel L. K. Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3):e2014196 18, January 18. URL https: hats:///wwe //www//ww///////////// pnas 2014196118. Publisher: Proceedings of the National Academy of Sciences.
2021. doi:
wwww. org/ htt//doo/10... 1073/07//mmm //nas.
[7] Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H. McDermott. Many but not all deep neural network audio models capture brain responses and exhibit hierarchical region correspondence. November 2022. URL https: / /www. bi biorxiv. org/ content/n0.... 1101/ Pages: 2022.09 0w..099.66.6666 Section: New Results.
2022. 09. 506680066..
[8] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrativee modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 11 (8((())))20064444 https://www pnas. org/doi/10/ /tt//0.o111111////////. 10731007/paaa ///aaa...../. 2105646118. Publisher: Proceedings of the National Academy of Sciences.
18, November 2021. doi: 10. 1.073/pas.211555 18. URL
[9] Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, 2023. doi: 10.1101/2022.03. 28 48.488888. URL https: / //www. biorxiv..vv. org/ htt////eneeeeenne//eeeeeeee
.3. 28, 485868.
[10] Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel. Robert A. Vandermeulen, Katherine Her- mann, Andrew Kyle Lampinen, and Simon Kornblith. Improving neural network representations using human similarity judgments. In Shirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URI. https: httpp::///pppeeeeee http///ppeeeeeeeeeeeeeee wttpeewwwwwwieeeeeeeeeeeee .ett/forumm/d--//adpoooo Nddhuuuuuuxx
[11] Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[12] Stephanie Fu, Netanel Yakir Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using

---

synthetic data. In Seirty- Conference on Neural Information Processing Systems, 2023.
URL https::/openreviieet htt.://www.....ttoooooooooaaaaa.
[13]
Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-are deep artificial neural networks. Journal of Neuro- science, 38(33):7255-7266) 2018. ISSN 0270-6474, doi: 10. 10.153//EUUOSSC.088- 18.2018..
URL hwwph::tt///// //ww.. /nn.uppo///.....eoooocccciicc wtteeeii.rggooonooeeeeeeeeo http://oteet/////////222200
[14] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision, October 2021. URL http://aat /wwpvv/xvvvv////////////////.a wwrzxiv..og/aee htg/abb/21000 07411.
arXiv:2106.07411 [cs, q-bio].
[15] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traines CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https:t/////. http:::///eeeeeeeeeeeeee
http//foormmmdd-yyhbaaa j09XX.
[16] Nicholas Baker, Hongjing Lu, Gennady Frlikhman, and Philip J. Kellman. Deep convolutional networks do not classify based global object shape. PLOS Computational Biology, 14(12):
1 1-43, 12 2018. doi: 10.137 /371/journallp//// 1006613.
[17] Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, edi- tors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: https::/proceeddingg neurips wt/cc//ppppp/....c.p files/ https/ppppp///////ep/// 08776666e0001155sfff06666aa Paper .opprr.pf.
[18] Brandon RichardWebster, Samuel E. Anthony, and Walter J. Scheirer. PsyPhy: A psychophysics driven evaluation framework for visual recogmition. IEEE Transactions on Pattern Analysis and 2019. doi: 10. N..1109/TPMMI2000 2018 2849998..
Machine Intelligence, 4 (1)9222280-222
[19] Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th International Conference on Computer Communication and Networks (ICCCN), pages 1-7, 2017. doi: 10. 1,.1109/CCCCN2200 8038465.
[20] Hossein Hosscini, Baicen Xiao, Mayoore Jaiswal, and Radha Poovendran. On the limitation of convolutional neural networks in recognizing negative images. In Xuewen Chen, Bo Luo, Feng Luo, Vasile Palade, and M. Arif Wani, editors, 16th IEEE International Conference on Machine Learning and Applications, ICMLA 2017, pages 352-358. IEEE, 2017. doi: 10.1 10.1109//CMLA.2001 20..200770-110 136.
[21] Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or impossible dichotomous data difficulty masks model differences (on imagenet and beyond). In International Conference on Learning Representations, 2022. URL https: openrevien...e.. net/forum? Sate:rummiidC_aaaCEIeI3_3
[22] Kamila M. Jozwik, Nikolaus Kriegeskorte, Katherine R. Storrs, and Marieke Mur. Deep Convolutional Neural Networks outperform feature-based but not categorical models ex- plaining object similarity judgments. Frontiers in Psychology, 8, 2017. ISSN 1664-1078. doi: http:///fppgg21...111..
[23] Joshua C. Peterson, Joshua Т. Abbott, and Thomas L. Griffiths. Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations, Cognitive Science, 42(8):2648-266) November 2018. ISSN 1551-6709, doi: 10.1 /allll//cos..2661
[24] Raja Marjieh, Pol van Rijn, Ilia Sucholutsky, Theodore R. Sumers, Harin Lee, Thomas L. Griffiths, and Nori Jacoby. Words are all you need? Language as an approximation for human similarity judgments, February 2023. URL http:://rxixt brrg/abs//0n arXiv:2206.0 005 [cs, stat].
aaaaaabb//2//// 2206. 04100000000...........0000022000 04105.
[25] Radoslaw M. Cichy, Nikolaus Kriegeskorte, Kamila M. Jozwik, Jasper J.F. van den Bosch, and Ian Charest. The spatiotemporal neural dynamics underlying perceived similarity for real-world objects. Neurolmage, 124:12-24, 2019. ISSN 1053-81 19. doi: https://doi.or//10 10166 1016/j htff:/eeeuuiimmee221900033333330
12

---

[26] Marcie L. King, Iris LA.. Groen, Adam Steel, Dwight J. Kravitz, and Chris I. Baker. Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images. Neurolmage, 197:368-380 382, 2019. ISSN 1053-81 19. doi: https///oi..or/ 10. www.:/uugeeee20....44....
[27] Martin N. Hebart, Charles Y. Zheng, Francisco Pereira, and Chris I. Baker. Revealing the multi- dimensional mental representations of natural objects underlying human similarity judgements.
Nature Human Behaviour, 4(11):1173-1188 November 2020. ISSN 2397-3374. doi: 10. 1038/ URL https: //www www.rrrau..... com http://riilesseee
Number: 11 Publisher: Nature Publishing Group.
[28] Brett D. Roads and Bradley C. Love. Enriching ImageNet with human similarity judgments and psychological embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3547-3557, June 2021.
[29] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, April 2018. URL
http://arxiv.oott abs httpa/ass///...///// 18011 03924. 1eXiv:180100399 [cs].
[30] Martin N. Hebart, Adam H. Dickter, Alexis Kidder, Wan Y. Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I. Baker. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14(10)=0023392. October 2019, ISSN 1932- 6203. doi: 10.137 www.roallloooeeoo.ooo articleiid=10
j/wp/als..alllss plos arg/plosone/ pon. 0223792. Publisher: Public Library of Science.
URL https: :
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL arXiv:2010.1 1929 [cs].
wwwg/osg/////eeeoooee/..ooeeeeee ha////aa/200....0 11929.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL http: /tappiv//xiv... http://axxvv.../aabb/// httg//bb/1112...00
arXiv:1512.03385 [cs].
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, August 2021.
URL
14030. arXiv:2103. 14030 [cs].
[34] Zhuang Liu, Hanzi Mao, Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s, March 2022. URL
org/ /////bb//// hts//abb/22..... 22511.3355......
arXiv:2201.0355 [es].
[35] Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel L. Yamins, and James J. DiCarlo. CORnet: Modeling the Neural Mechanisms of Core Object Recognition, September 2018. URL https://wwat Section: New Results.
biorxiv. www..ww.wog/ooooeeeeeeeeeoooeeee
111004088881.11.1 Pages: 408385
[36] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Sule-Super Vision Transformers, May 2021. URL http:: http///axvvvvo// httg:/bss/21000 14294. arXiv:2104. 14294 [cs].
[37] Ishan Misra and Laurens van der Maaten. Sele-Super Learning of Pretext-Invarian Repre- sentations, December 2019. URL http: htt////axxvvvvve org/ 1atb:/991...111 01991. arXiv:1912.0199) [cs].
[38] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, January 2021. URL http:://axixi org/ hts//bb//2006 09882. arXiv:2006.09882 [cs].
[39] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, June 2020. URL http::/arxiv..oom org/ abs/2002. 0520000.077......... arXiv:2002.05709 [cs, stat].
[40] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variane-Iivarinnc-Covaaann Regularization for Self-Supervised I Learning, January 2022. 2105. 04906. 1055 049000000661.01 [cs].
13

---

[41] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self- Supervised Learning via Redundancy Reduction, June 2021. URL http://axxiv..com org/bb// 2103.03230. arXiv:2103.03230 [es, q-bio].
[42] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learnng by Predicting Image Rotations, March 2018. URL http::///arrri..oorr/////// 07728.
10Xiv:1803.07728 [es].
[43] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Mo- mentum Contrastive Learning, March 2020. URL http:///axtii http:://aawiww....aaa/////////// 04297.
arXiv:2003.04297 [cs].
[44] Mehdi Noroozi and Paolo Favaro. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles, August 2017. URL
org/ aaaaa/bs/1133 09246.
arXiv:1603.099: 092466 [cs].
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February
2021. URL http:://taaatt org/ ab///1//a/23///. 00020. arXiv:2103.00020 [cs].
[46] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedess Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal Sentence Encoder, April 2018. URL http: // //axiiv.vvv//////////////ar org/ //bbs/ 1.03.11175, Xrx::: 1803 11.111177 175 [es].
[47] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http: MEKKv:1810004980 [es].
/ //aaxiv..rg/abaeee..... wwwg/aa/////// http//1181..008.....
[48] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, February 2020. http://aaar
org/abs 1910 01 108. arXiv:1910.011 108 [cs].
[49] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Opti- mized BERT Pretraining Approach, July 2019. UtL.ttt: orgg/bs///07.1111
arXiv:1907.11 1692 [cs].
[50] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020. URL http: //tp:vaaaaivv////a org/ http//bbs2000.o.o 14165. arXiv:2005. 14165 [cs].
[51] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in Pre-Training Distributed Word Representations, December 2017. URL http: //arxiv. org/ waarg/////////// abs/ 09405. arXiv:1712.09400 [cs].
[52] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xic. Slip: Self-supervision meets
language-image pre-training. 2021.
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yo-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[54] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL https://doi. org/10.. 5281/ zenodo. 5143773.
[55] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws
14

---

for contrastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-288888888 2829, 2023.
[56] Can Demircan, Leonardo Pettini, Tankred Saanum, Marcel Binz, Blazej Metody Baczkowski, Christian Doeller, Mona Garvert, and Eric Schulz. Decision-making with naturalistic options. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
[57] Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972.
[58] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. R..10///VRR.2000.520000
[59] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 491-507, Cham, 2020. Springer International Publishing. ISBN 978-3-030- 58558-7.
[60] Mingxing Tan and Quoc Le. EfficientNer: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 09-15 Jun 2019.
[61] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2104-12113, June 2022.
[62] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era, 2017.
[63] Simon Korbblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features?, 2021.
[64] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8242- 8252. PMLR, 13 18 Jul 2020. URL https: https:://proceeding.s mllr. wttp:/s.mwr.....sss.eeee vttsss//119/rohh20.. html.
[65] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited, July 2019. httttt////a/t: arXiv:1905.00 004444 [cs, q-bio, stat].
BRLLttp::// http:///arxxa arxivv org/abs/ 0aabs/999..... 190410015500.....1
[66] Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks, 2019.
[67] Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory, pages 25-55. PMLR, 2018.
[68] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[69] Tankred Saanum, Noémi Éltetö, Peter Dayan, Marcel Binz, and Eric Schulz. Reinforcement learning with simple sequence priors. Advances in Neural Information Processing Systems, 36, 2024.
[70] Tankred Saanum, Peter Dayan, and Eric Schulz. Predicting the future with simple world models. arXiv preprint 178355 2024.
[71] Ravid Shwartz-Zii Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00 000100 2017.
[72] Eric Elmoznino and Michael F Bonner. cortex benefit from high latent dimensionality. PLOS Computational Biology, 20(1) 2024.
neural network models of visual
[73] Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 711):12140. 2017.
15

---

[74] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Comnunications of the ACM, 59(2):64-7)) 73, January 2016. ISSN 1557-7317. doi: 10.1 145/2812802. URL.
attp://áxx www.oooo/..o... 0144/2812800
[75] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang,
Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[76] Drew Linsley, Dan Shicbler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. International Conference on Learning Representations (ICLR), 2019.
[77] Ruairidh M. Baitleday, Joshua C. Peterson, and Thomas L. Griffiths. Capturing human cate-
gorization of natural images by combining deep networks and cognitive models. Nature Com- munications, 11(1):5418, October 2020. ISSN 2041-1723, doi: 10. 10.1008//41477-0000 1774-144-022---------
URL https: /www. www..natur....mmmmmmmmmm nature. wom. Nature Publishing Group.
18946- z. Number: Publisher:
[78] Lukas Muttenthaler and Martin N. Hebart. THINGSvision: A Python Toolbox for Streamlining the Extraction of Activations From Deep Neural Networks. Frontiers in Neuroinformaties, 15, 2021. ISSN 1662-5196. URL https: //wps: frontiersin, wrrg/articles///eerrrrrrrreeeeee 10. 33899 fninf. 2021 679838.
[79] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12):441511426, November 2023.
ISSN 2522-5839. doi: 10.1038/s42256-023 023-00753-y URL http://dx. //dp:dii.x//.....//////o orggoo////. 1038/ s42256- 023 023-00753...
[80] Akshay K Jagadish, Julian Coda-Forno, Mirko Thalmann, Eric Schulz, and Marcel Binz. Ecologically rational meta-learned inference explains human category learning. arXiv preprint arXiv:2402.0/82/, 2024.
[81] Joshua R. de Leeuw. jsPsych: A JavaScript library for creating behavioral experiments in Web browser. Behavior Research Methods, 47(1):1-12, March 2015. ISSN 1554-3528, doi: 10.3758/s13428-0144
URL https: //dii org/10 10 $758/s134288 014-014---00 1144-0058-3-
[82] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2021. URL htttts::////ttttt http://www....oooo.r projeecct. org/
[83] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting Linear Mixed-Efffetts Models Using lme4. Journal of Statistical Software, 17:1-48, October 2015. ISSN 1548-7660. doi: 10.18637/js..v067.000 URL https: //tp://oooooooo/o org.///00oo//0////////// 18637/ jss. v067 101.
[84] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vander- plas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Re- search, 12885522855-2830 2011. ISSN 1533-7928. URL http:://jmmii org wwwgg/apprs//vie pedregosalia, html.
[85] Laura Mai Stoinski, Jonas Perkuhn, and Martin N. Hebart. THINGSplus: New Norms and Metadata for the THINGS Database of 1,854 Object Concepts and 26,107 Natural Object com/exu9f/.
Images, July 2022. URI. httph:tt///// htttps//pssyaxxiv
[86] Maarten Speekenbrink, Shelley Channon, and David R Shanks. Learning strategies in amnesia. Neuroscience & Biobehavioral Reviews, 32(2):292-310, 2008.
[87] Samuel J Gershman. A unifying probabilistic view of associative learning. PLoS computational biology, 11(1))ee004467, 2015.
[88] Marcel Binz, Samuel J. Gershman, Eric Schulz, and Dominik Endres. Heuristics from bounded meta-learned inference. Psychological Review, 1042-1077 October 2022. ISSN 1939- 1471. doi: 00.1037/e00000330.
[89] Raja Marjieh. The universal law of generalization holds for naturalistic stimuli, Jan 2024. URL osf. Lo/rbkhh
16

---

A Methods
Participants For the category learning task, we recruited 98 participants (48 females, 50 males, mean age= 28.92y, SD= 7.32) on the Prolific platform. Participants with less than 50% accuracy were excluded from the analyses, leaving with 91 participants. A base payment of & 1.50 was made, and participants could earn an additional bonus of & 6.00. The median completion time was 12 minutes and 38 seconds. The inclusion criteria included having minimum approval rate of 97%, and a minimum number of 15 previous submissions on Prolific. Participation in the reward learning study was exclusion criterion. For the reward learning task, 99 participants were recruited (49 females, 49 males, other, mean age = 27.9 y. SD = 9.13). After applying the 50% accuracy criteria, were left with 82 participants. A base payment of & 2.00 was made, and an additional performance-depnndee
bonus of $4.00 was offered. The median completion time was 9 minutes and 26 seconds. The inclusion criteria included having a minimum approval rate of 95%, and minimum number of 10 previous submissions on Prolific. All participants agreed to their anonymized data being used for research. The study was approved by the ethics committee of of the medical faculty of the University of Tübingen (number 701/2020BO) Participants gave consent for their data to be anonymously analyzed by agreeing data protection sheet approved by the data protection officer of the MPG (Datenshhuzzbcautrraee der MPG, Max-Planck-Gesde Gesellschaftt zur Förderung der Wissenschaften).
Tasks and Stimuli Both tasks were run online in forced full-screen mode. Participants were shown written instructions and were asked to complete comprehension check questions before they could start the tasks. In both tasks, participants were given unlimited time to make decisions. In the category learning task, binary (correct versus wrong) feedback was given for 2s. In the reward learning task, the associated reward with the stimuli was shown for 1.5s, and there was an inter-trial interval of 1s where participants
shown a blank screen. Throughout both tasks, the estimated total payment of participants was shown on the upper part of the screen. At the end of the tasks, participants were asked whether they thought their data should be used for analysis. Across both tasks, all but one participant responded saying their data should be analyzed, whose data was anyway excluded due to poor performance. The category learning task was programmed using jsPsych [81], whereas the reward learning task was programmed in plain JavaScript.
For each participant, 120 stimuli were sampled independently from the THINGS database. Because the loadings of the features were not uniformly distributed, we made 5 equally sized bins of the loadings for the assigned feature and sampled object categories uniformly from these bins. From these object categories, the specific images were assigned randomly. For details on the used features and the embedding, see Hebart et al. [27].
Behavioural Analyses We used mixed-effects logistic models for both category and reward learning analyses. For category learning, we predicted correct responses per trial, using trial number fixed effect and including participant-ssecifie random effects for intercept, trial number, and assigned task rule. In the reward learning model, we predicted whether the image the right is selected, incorporating the trial number, reward difference between images, and their interaction as fixed effects. These factors, along with the assigned task rule, were also modelled participant-specific random effects. Both models effectively captured task structure, learning progression, and individual variability in performance. In R formula notation[82, 83), the model for category learning is denoted as follows:
correct correct_choice ~ + trial (1 trial dimension participant)
For the reward learning task, the following model was used:
Tight_choice- ~-1 + trial hight_lett_rewarre difference + (-1 trial + dimension +
rtwprrewwweriieeeeeeeeeee ddifrdrenffffeeeeee I participant)
where -1 denotes no intercept.
Software, Data, & Compute Resources The code to reproduce the reported results is available at com/ candemircan aaaandemiraa//attuaalloooo /attrallooooceeecaoc and we provide anonymised human choice data on https://osf io hht52/. For the learning models we used 1me4 [83] and scikit-lean [84]. To extract representations from neural networks we used thingsvision [78].
17

---

Computations were performed on an academic SLURM cluster. Feature extraction was done on single Nvidia A100 GPU (40GB) under 24 hours. The linear models were parallelised across several jobs that used single core and 8GB RAM and were completed in under 48 hours. The mixed-effects models were similarly parallelised and completed under 24 hours.
Our Task
Peterson et el. (2018)
Fu et all (2023)
4 Painwise Similerty
AFCFC% % Accurasy
a MoFadden'ss
0.02
Comelation (p)
0.20
20
0.00
0.10
-0.03
0.00
0.10
-10
Harmonization
DreamSim
gLocal
Harmonization
DreamSim
glocal
Harmaiization
DreamSim
glocal
Figure 9: Change of human alignment for different methods on different datasets
10
10
10
p-value
10
10
10
10
20
40
60
80
100
120
20
30
40
50
60
Trial
Figure 10: Participant Performance Against Chance Level Each Trial. Trial-by Trial-by-triaa p-values from 1 sample t-tests testing accuracy against chance level for (A) category learning task and the (B) reward learning task.
Modelling
For the category learning task, used & regularised logistic regression model optimize regression weights. We relied on scikit-learn's LogisticRegression class which internally optimizes the following objective:
W arg max
e, log(p(a|x;, w)) - (1 log(1 log(p(e w)) + w/12 (5)
For the reward learning task, used Bayesian linear regression model to infer posterior distribution over regression weights. We relied on scikit-learn's BayesianRidg. class which infers posterior distribution assuming spherical Gaussian priors (i.e., p(w) A/ A II) and
Gaussian likelihood (i.c., p(u x,, w) = =A(w" x,, 8-1)). Based these assumptions, the posterior
18

---

Metal/Artificia/HHard
64
44
65
76
62
49
71
76
44
48
87
64
Shimal-related/Organi
30
72
46
23
32
24
Figure 11: Example trials showing the similarity between CLIP and human decisions that show disagreement with the task embedding. Each row shows three trials from different condition. shows the option chosen by all CLIP models and the human participant, whereas grey text shows the decision made by the task embedding. As the tasks were generated using the task embedding, all the choices shown here made by CLIP and humans are suboptimal. Shown examples are from the second half of the task, to eliminate the learning process confound. The original images are replaced with copyright-free free alternatives from the THINGSplus database [85].
Orange highlighted
distribution can be computed in closed form:
p(w X, y) (MmN,, SN)
(6)
mN = x
(7)
S = AI 3X T
(8)
where X and y denote the stacked inputs and targets respectively.
We run both models from scratch on each trial using all previously observed input-target pairs. The choice of these models was motivated by previous investigations in similar but oow-dimeniiona settings [86, 87, 88].
The l2 penalty term for the logistic regression model described above was determined via grid search maximise task performance, were fitted to maximise the log marginal likelihood on the task performances.
per participant basis. For the linear regression model, À and
The estimates from these models were used in mixed-effects logistic regression models with leave- one-out predictions assess participant choices. For category learning, we used logistic regression probability estimates predictors. In the reward learning task, used the difference in estimated rewards from linear regression models as predictors. In both cases, these predictors were included as both fixed and random effects, allowing to account for individual differences while maintaining the group effects. These correspond to the following models in R formula notation for category and reward learning respectively:
human_ccooce choice ~-1 boobability_estimaan (-1 probabbiity_eetimiele estimate participant)
human
human_choice ~-1 + estimated_rwward differeecce + (-1 esstiated_rewardddiiieeee
participant)
For the mixed-effects models, the training data was centred and divided by standard deviation.
The same scaling parameters applied to the test data.
19

---

0.75
Task Accuracy
1.00
0.45
0.73
0.36
0.84
0.46
Model Parameters
0.25
Training Images
0.73
0.59
1.00
0.38
0.59
-0.55
0.00
Q
Class-Spparation (RT)
0.36
0.34
0.38
1.00
0.37
-0.65
-0.25
Similarity
Task Embedding
0:84
0.59
1.00
-0.38
-0.50
Intrinsic
Dimensionaiity
-0.46
-0.39
0.55
-0.65
-0.38
1,00
Task Accuracy
Model Parametero
Training imagen
Class Sepsrsteaatnonnno (Ro)
OKA Simitarity with the Task Bridediding
Intirssi Dimensionality
-0.75
-1.00
Figure 12: Pairwise Spearman correlations between the factors investigated that contribute align- ment.
Additional human-alignment tasks
Results reported Fig. 8A Fig. include comparisons for the overlap of models between those reported by Muttenthaler et al. [3] and the ones tested. For Fig. 8B & Fig. 8C, tested all the vision models reported in our paper. However, for the Peterson et al. [23] dataset, we only found only computed for supervised models, as the method requires computing gradients for ImageNet classes, which could only do for the supervised models that had ImageNet classification heads.
subset of the original data reported in the paper'. The BiccMe-Harmonizee alignment
B Testing aligned models on other datasets
Above, also report how different alignment methods perform different datasets (Fig. 9). Harmonization on average slightly more human-like on the two external tested datasets compared to baselines. DreamSim shows mixed results for the Peterson al. [23] dataset, but it shows improvement on the NIGHTS dataset [12]. This is not surprising, as this dataset was used to build DreamSim. Lastly, gLocal shows mixed results.
'Specifically, we tested similarity judgements obtained on Animal, Fruit, and Vegetable categories. The data obtained from [89]
20

---

C
Additional results
Above we provide some additional results supporting our claims in the main text. Fig. 10 shows participants can do both tasks above chance level very early on in the task. Fig. 11 shows some incorrect choices made by humans and also by CLIP models, and Fig. 12 shows pairwise correlations between the factors we investigated that contribute to alignment.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions are clearly marked in the introduction, and they directly match the provided results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalise to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, there is dedicated section for this.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
There are no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the relevant details for the experiments and the analyses that are needed for reproducing the results. We will also share the data and the code before the conference GitHub and OSF.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the data and the code available in online repositories. We will make them available before the conference.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(eglii/guides/Coeeefe wwwpiiiiiissoooooiooiioooooooooo
onpooicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /náps. wwtppbcccc///ccccccccccccccccltt wwtpc//ggg/eeeeeeee /alesee/oodSmbmii ssiinnoossssiiiooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are provided in the Results section of the main text, and in the Methods section in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The error bars and lines are shown in the figures. The captions explain what the error bars represent (95% confidence intervals).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the approximate compute resources used for our work in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //teurip// cc. cc/ Spppblic/Ethcsuuiiiiii
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are potential positive or negative societal impacts that may arise directly from our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data we release do not have a risk of misuse. No new models are provided. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the pretrained models we used. The used images either had public domain or CCO copyright license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww/coo//aaooooo datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: All the code and the data, which will be released before the conference, documented.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide a summary of the task and the instructions given to the partici- pants both in the main text and the Appendix. The experiment code, including the full instructions given to the participants, will be made available before the conference.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The behavioural studies carried no risks for participants. Both studies were approved by the local ethics committee.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "eeGSKNUe7r", "reviewer": "Reviewer_pzQG", "summary": "The authors evaluate alignment between humans and 77 pre-trained neural network vision models using learning tasks. Previous work has focused on comparing alignment between humans and models with similarity judgments alone; instead, here the authors asked participants to perform simple category learning and reinforcement learning tasks. These responses were compared  with linear models fit on top of the pre-trained networks. The analysis is very thoughtful and thorough, and the article goes into detail regarding the factors that influence which models are best aligned: task accuracy, model size, training, etc.\n\nI see this work as providing a strong contribution to understanding human vs. model alignment in vision tasks. I would like to see it published at NeurIPS, pending the concern I mentioned below.", "questions": "I don't fully understand the methodological choices in getting the pre-trained models to predict the human responses. In the categorization task, my understanding is that a regularized logistic regression was trained to predict the gold category labels. Then, the predicted category probability was used in a logistic mixed model. \n\nWith this approach, you would seemingly lose a lot of information (potentially relevant features) in the model embeddings beyond its category prediction, before mapping this prediction to human responses. Did you consider fitting a mapping between the model features and participant responses more directly? If not, why not? I would be willing to raise my score depending on the response to this question.\nEDIT: Thank you for trying this and for the explanation. I see the current approach as better justified now, and raised my score accordingly.\n\nAlso, the description of the mixed model in the appendix is not satisfying, as the variables aren't defined clearly. Please explain in the rebuttal.\n\nTypos\n\"Exemplary learning curves\" (pg 4)", "limitations": "This section is fine", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This article has many strengths\n- Kudos for comparing 77 different models\n- Thoughtful and rigorous model comparison\n- Detailed analysis of which factors lead models to fit the human data better\n- Excellent visuals for understanding the results", "weaknesses": "My main concern is about the method for fitting participant responses (see question below).\n\nThe categorization task of delivering images to two dinosaurs, Julty and Folty, is a bit silly and may have had participants overthinking the task.  You may have received cleaner data with a simpler framing: learning Category A vs. Category B. But I don't see this as a major issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Did you consider fitting a mapping between the model features and participant responses more directly? If not, why not? \n\nWe agree with the reviewer that fitting the linear/logistic regression models directly to human choices is an interesting thought (as it has for example recently been applied in a different domain by [1]). This modelling approach typically requires a lot of data to work well. For example, [1] has used data from around 1 million human choices, whereas we have only 60 or 120 choices per participant. Hence, we expected that this approach would not work well in our setting. Nevertheless, we have now also tried to fit the embeddings directly onto human choice. However, we received much poorer fits (mean $R^2$= -0.76, max $R^2$= -0.05, min $R^2$= -3.98), as this approach does not directly capture the structure of the task and likely leads to overfitting.\n\n> Also, the description of the mixed model in the appendix is not satisfying, as the variables aren't defined clearly. Please explain in the rebuttal.\n\nUnder the Behavioural Analyses section in Appendix A, we will add the following in the camera-ready version:\n\n**“We used mixed-effects logistic models for both category and reward learning analyses. For category learning, we predicted correct responses per trial, using trial number as a fixed effect and including participant-specific random effects for intercept, trial number, and assigned task rule. In the reward learning model, we predicted whether the image on the right is selected, incorporating the trial number, reward difference between images, and their interaction as fixed effects. These factors, along with the assigned task rule, were also modelled as participant-specific random effects. Both models effectively captured task structure, learning progression, and individual variability in performance.”**\n\nAnd under the Modelling in Appendix A, we will add the following:\n\n**“We used mixed effects logistic regression models with leave-one-out predictions to assess participant choices in both tasks. For category learning, we used logistic regression probability estimates as predictors. In the reward learning task, we used the difference in estimated rewards from linear regression models as predictors. In both cases, these predictors were included as both fixed and random effects, allowing us to account for individual differences while maintaining the group effects. These correspond to the following models in R formula notation:**\n\n**Category Learning:**\n\n```\nhuman_choice ~ -1 + probability_estimate + (-1 + probability_estimate | participant)\n```\n\n**Reward Learning** \n\n```\nhuman_choice ~ -1 + estimated_reward_difference + (-1 + estimated_reward_difference | participant)\n```\n\n**Where -1 denotes no intercept.\"**\n\n> The categorization task of delivering images to two dinosaurs, Julty and Folty, is a bit silly and may have had participants overthinking the task. You may have received cleaner data with a simpler framing: learning Category A vs. Category B.\n\nThanks for raising this point. It has been shown that using “fun” cover stories improves data quality in learning tasks[2]. In our experience, this also makes the tasks more engaging, especially in online settings. Furthermore, before starting the main experiment,  participants were required to answer a few comprehension questions, and they were not allowed to participate in the main task until answering these questions correctly. Therefore, we believe participants had a good understanding of the task. We will include these questions in the Appendix for the camera-ready version.\n\n> Typos \"Exemplary learning curves\" (pg 4)\n\nThanks. We will change this to **“Example learning curves”**\n\n[1] Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D., & Griffiths, T. L. (2021). Using large-scale experiments and machine learning to discover theories of human decision-making. Science, 372(6547), 1209-1214.\n\n[2] Feher da Silva, C., Lombardi, G., Edelson, M., & Hare, T. A. (2023). Rethinking model-based and model-free influences on mental effort and striatal prediction errors. Nat. Hum. Behav., 7(6), 956–969.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nmA0aICbo", "reviewer": "Reviewer_NCLd", "summary": "The authors propose a novel representational alignment metric tied to sequential human behavior and leverage it to analyze factors in NN design and training that contribute to increased alignment with humans.", "questions": "- why were participants with <50% accuracy excluded?\n- why might the alignment numbers be so low? McFadden's R^2 is typically considered to be excellent fit in 0.2-0.4 range, yet all the results cap out at about 0.15", "limitations": "- Limitations are adequately discussed (assuming weaknesses and questions raised above are addressed)", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Great analysis of factors contributing to alignment of NNs with humans\n- Well-written manuscript, easy to follow\n- Great descriptions of human and NN experiments; important for reproducibility\n- Important and timely contribution to a growing field (representational alignment)\n- I agree with the authors that its important to tie notions of alignment to behavior (a metric is only useful if it actually predicts something downstream)", "weaknesses": "- The authors claim to make 2 contributions: a novel alignment metric and an analysis of factors contributing to human alignment of various NN that leverages that alignment metric. The authors claim that their alignment metric is better than existing alignment metrics as it requires \"generalisation and information integration across an extended horizon\". However, there isn't a comprehensive comparison to previous alignment metrics to see (a) how correlated the new metric is to existing metrics, and (b) whether human behavior on the tasks can be predicted from previous metrics. For (b) for example, the authors could take human pairwise similarity judgments and treat them as features from which to learn the task and then correlate that with human responses (i.e., treating the sim judgments as a kernel). This kind of analysis would go a long way in validating the proposed metric and comparing/contrasting it against existing alignment metrics.\n- Perhaps I missed it, but I didn't see an analysis of inter-rater reliability/noise roof. This would be important to put the McFadden R^2 numbers into perspective and understand how good alignment *could* be.\n- Happy to raise my score if these points and questions below are resolved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  The authors claim that their alignment metric is better than existing alignment metrics as it requires \"generalisation and information integration across an extended horizon\".\n\nWe believe that our method is not necessarily better than other metrics but that alignment is multifaceted. We will clarify this stance further in the main text by updating line 248 as follows: \n**\"Previous work has predominantly focused on simple image exposure and similarity judgments. We believe our findings complement this research by addressing unexplored aspects of alignment, which is  generalisation and information integration across an extended horizon.\"**\n\n> how correlated the new metric is to existing metrics\n\nThis is an excellent question that’s crucial for adequately placing our work in the field of alignment. To address this, we compared our alignment metric to four different metrics computed on different datasets. The results are shown in Rebuttal Figure 1. We will add the following to the manuscript:\n\n**\"We compared alignment on four datasets against ours, which included:**\n\n**I) Comparison of 22 models on the odd-one-out similarity judgments on THINGS [1] ($\\rho=0.54$).**\n\n**II) Comparison of 79 models on pairwise similarity judgments from an independent dataset [2] ($\\rho=0.45$).**\n\n**III) Comparison of 79 models on two-alternative forced choice data, which is a part of the NIGHTS dataset from the DreamSim paper that was not used to train the model [3] ($\\rho=0.28$).**\n\n**IV) Comparison of 25 models on alignment to the validation split of the ClickMe dataset [4,5], which was used to fine-tune the Harmonizer models ($\\rho=-0.43$).**\n\n**While our metric positively correlated with the metrics from the first three datasets, there were important differences. We found a negative correlation between our metric and the ClickMe-Harmonizer feature alignment, suggesting that pixel-level alignment and semantically bound global image alignment might be at odds.**\"\n\nFor the odd-one-out data, we compared the models reported in the original paper [1]. For [2,3], we ran the comparisons for all the models. The ClickMe-Harmonizer alignment was only computed for supervised models, as the method requires computing gradients for ImageNet classes, which we could only do for the supervised models that had ImageNet classification heads.\n\n> the authors could take human pairwise similarity judgments and treat them as features from which to learn the task\n\nThis is a great suggestion. Unfortunately, pairwise similarity judgements do not exist for THINGS. However, the ground truth features of the task were generated from an embedding optimised to predict odd-one-out similarity judgements on THINGS[7]. We report how well these models predict human choice in Figure 2. While the ground truth model ranks in the top 13, it is surpassed by several multimodal models.\n\n> I didn't see an analysis of inter-rater reliability/noise roof.\n\nThe sequence of observations that were given to the participants in our tasks were randomly assigned and none were identical. Therefore it is not possible to calculate inter-rater reliability. However, the question of how good can alignment be is important.  We initially thought the models trained on the ground truth features would establish a ceiling. However, to our surprise, several models exceeded this.\n\n> why were participants with less than 50% accuracy excluded?\n\nIn online studies, it is common to have noisy data, where sometimes participants don’t engage with the task faithfully. Therefore, it is common to exclude participants who perform below chance. Nevertheless, when we fit our models to all participants’s data (Rebuttal Figure 4), the $R^2$ values we obtain are highly correlated with the $R^2$ values after exclusion ($\\rho=0.96$ for category learning, $\\rho=0.66$ for reward). Our qualitative results stay the same, where the top-ranking models are CLIP variants. Therefore, our findings are not dependent on this filtering step. \n\n\n> why might the alignment numbers be so low?\n\nWe believe there are several reasons for this. First, it is difficult to predict human choices early in the task for any model, given the noise associated with uncertainty. Indeed, when we look at $R^2$ values in the first half of the task, the maximum values are 0.05 and 0.1 for the category and the reward tasks. Whereas if we look at the second half, they go as high as 0.22 and 0.21, which is in the range that you suggested. Regardless of the split, the top-performing models are CLIP models. This is a unique difficulty of learning tasks, as unsupervised similarity judgements do not require learning and therefore a similar noise effect is not expected early in the task.\n\nSecond, even when we use the ground truth features to predict human choice, we see that $R^2$ values aren’t high, which again speaks to the difficulty of the task.\n\nLastly,  it’s difficult to compare McFadden’s $R^2$ values across tasks. Some simple learning paradigms might have best-fitting models with $R^2$ values <.2[7], whereas other similar paradigms can go up to the range you suggested [8].\n\n\n[1] Human alignment of neural network representations. Muttenthaler et al., ICLR 2023\n\n[2] Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations. Peterson et al., Cognitive Science 2018\n\n[3] DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data. Fu et al., NeurIPS 2023\n\n[4] Harmonizing the object recognition strategies of deep neural networks with humans. Fel et al., NeurIPS 2022\n\n[5] Learning what and where to attend, Linsley et al., ICLR 2019\n\n[6] Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Hebart et al., Nat. Hum. Beh. 2020\n\n[7] Finding structure in multi-armed bandits. Schulz et al., Cognitive Psychology 2020\n\n[8] Model-Based Influences on Humans’ Choice and Striatal Prediction Errors, Daw et al., Neuron 2011", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bPJVt3zkQO", "reviewer": "Reviewer_MCn8", "summary": "This paper explores the alignment between human cognitive processes and neural network representations in image-based learning tasks. The authors evaluated 77 pretrained neural network models to see how their representations aligned with human learning patterns in two tasks: category learning and reward learning. \n\nThey found that models with contrastive training, especially CLIP models, effectively predicted human generalization. Factors such as training dataset size, model size, and representation properties were identified as influential for alignment with human cognition.", "questions": "See Strengths & Weaknesses", "limitations": "Yes, the limitations identified by the authors are accurate and well-documented. Regarding the weakness I mentioned, I reserve the right to increase the score if the authors adequately address my major concerns.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes substantial contributions by:\n1. Introducing two novel tasks to evaluate human-model alignment.\n2. Demonstrating that larger contrastive training datasets and multi-modal data improve predictions of human behavior.\n3. Conducting extensive and engaging human experiments.\n\nAdditionally, I found that:\n- Figure 4 is particularly interesting and stimulates a lot of valuable discussion.\n- It would have been very insightful to include a comparison with generative models (e.g., EBM, VAE) and relate these findings to the brain's generative hypothesis.", "weaknesses": "Despite its strengths, the paper has several weaknesses. These are categorized into major problems (**M**) and minor problems (_m_).\n\n**M1**: There is a potential bias in using the THINGS database for both training and evaluation, especially with the gLocal models, which might lead to circular reasoning. The space trained with gLocal is implicitly designed to perform well on these specific tasks, which could inflate the perceived performance of the models. Addressing this concern by diversifying the evaluation datasets or providing a more detailed justification for using THINGS would strengthen the paper. You could use a measure from DreamSim and Harmonization as control.\n\n_m1_: The use of $\\ell_2$ regularization in the regression model may penalize some tokens disproportionately due to their high activation values. Standardizing all activations before performing regression could mitigate this issue and provide a fairer comparison.\n\n_m2_: Concerning the use of the TwoNN method for estimating intrinsic dimensionality, could you verify that standardization instead of linear scaling yield the same results, as we know that some high norm token (outlier token) exist and could totally distort your distances.\n\n_m3_: Figure 7 lacks clarity on which models are used in the harmonization method? You have 7 Harmonization model in Fig. 3 and only 3 in Fig. 7. Or at least specify how they were chosen.\n\n_m4_: Could you also add a metric to complement McFadden (e.g., avg accuracy) ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It would have been very insightful to include a comparison with generative models.\n\nThanks for the great suggestion! We now included 3 masked autoencoder models[1] trained on ImageNet in our comparison. We chose these specific models because they are SOTA classification models trained in a generative manner. They don’t do well in predicting human choice (mean $R^2$ = 0.05, min $R^2$=0.03, max $R^2$=0.06). Therefore, it appears that training generative models doesn't provide a particular advantage, at least in our comparisons. We will include these results in the camera-ready version.\n\n> **M1**: There is a potential bias in using the THINGS database for both training and evaluation, especially with the gLocal models,\n\nYes, this is an important limitation that we have also pointed out in line 279. Nevertheless, we wanted to include this comparison with that caveat. To our surprise, despite the shared data across the gLocal training and our task,  we did not observe consistent improvements in model fit with the gLocal compared to the baselines.\n\n> Addressing this concern by diversifying the evaluation datasets (...) You could use a measure from DreamSim and Harmonization as control.\n\nThank you for these suggestions. We have now compared the alignment methods tested across two additional tasks that do not use the THINGS data (Rebuttal Figure 2). Similar to our initial findings, gLocal leads to improved alignment in limited cases. We found that Dreamsim consistently improves alignment on its test set, but no consistent improvements are observed on the independent dataset.\n\nAdditionally, we have incorporated four more baseline models for Harmonization and one more for DreamSim. These new baselines led to some modest improvements in our task, although they were not consistent and were smaller than those achieved by the gLocal models. As observed in our task, Harmonization models tend to reduce alignment in most cases. We will include these results in the camera-ready version. \n\nWe would also have liked to make this comparison for the Harmonization metric on the Harmonization dataset. However, this was not possible because the Harmonization metric requires a trained Imagenet classification head, which Dreamsim and gLocal models do not have. \n\n> (...) or providing a more detailed  justification for using THINGS would strengthen the paper. \n\nWe chose to conduct our experiments with the THINGS data because the ground truth features that we used were only available for THINGS. This is because the ground truth features are a low dimensional embedding that was learned to predict human odd-one-out similarity judgements on the THINGS dataset. This embedding is central to the design of our tasks and unfortunately, it does not generalise to new image data. Therefore, we had to use the THINGS images. We will add this justification to the camera-ready version.\n\n> m1: (…) Standardizing all activations before performing regression could mitigate this issue and provide a fairer comparison.\n\nThanks for pointing out this important detail. This is indeed what we have done in our comparisons. We scaled the features using the mean and the standard deviation of the training data and applied the same transformation to the test data. We did this both in our mixed-effects models and also the linear and logistic regressions that were used to fit the features to the tasks.\n\n> m2: Concerning the use of the TwoNN method for estimating intrinsic dimensionality, could you verify that standardization instead of linear scaling yield the same results\n\nThanks for the suggestion. We have now made this comparison including the new models we added and observe that min-max scaling ($\\rho$=-0.23, $p$=0.04) and standard scaling ($\\rho$=-0.24, $p$=0.03) yield near identical results.\n\n> m3: Figure 7 lacks clarity on which models are used in the harmonization method?\n\nWe were only including the Harmonization models for which we had baselines. We have now added baselines for all the Harmonization models and therefore include all Harmonization models in our comparison. The results are highly similar to what was reported in the initial submission.\n\n> m4: Could you also add a metric to complement McFadden (e.g., avg accuracy) ?\n\nYes, please see Rebuttal Figure 3. The maximum accuracies that are achieved are 68% for the category task and 67% for the reward task. Sorting the models by accuracy and $R^2$ alters the ordering of the models slightly, although the rank correlations are very high both for category ($\\rho$=.97) and the reward ($\\rho$=.93) tasks and our qualitative results stay the same, where 9 of the top 10 models are CLIP variants in both tasks. We will include the average accuracy in the final version.\n\n[1]He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFHNfrYBo0", "reviewer": "Reviewer_PLVt", "summary": "This article assesses how AI embedding can predict human behavior on 2 different tasks. In those tasks, humans have to learn continuous relationships and categories of natural images. Overall the authors tested the predictability of 77 retrained neural networks, including supervised, semi-supervised, and multimodal models. The authors found that a larger training size, a higher number of parameters, and lower intrinsic dimensionality improve the human alignment of the tested model. In addition, the authors claim that multimodal networks are better aligned with humans. In the last part, the authors test 3 different alignment methods and show that only the gLocal method shows a better predictability of human performance on the considered task.", "questions": "* Q1: The section 3 is not clear enough. For example (line 95): « Using an intercept and the trial number ». What does it mean? To my (maybe not sufficient) knowledge, the intercept is the constant term in a linear fit. If this is the case, what kind of data are you using to regress to find the fit? And most importantly, how does this intercept relate to human performance? \n\n* Q2: Still in section 3, the results of the statistical tests are poorly explained. For example (line 98): what does (Beta, z) relate to? Is that the intercept and the number of trials? Is that something else? This should be explicitly stated and should be put under the carpet!\n\n* Q3: In section 4, there is not enough detail on how you train the linear probe to predict the performance of the category learning and the reward learning task. Especially in the case of the ‘linear regression mode with spherical prior » (line 121). What does it mean, what is the loss you exactly optimize to find the regression parameters? Why Gaussian prior? This should be motivated and better explained...\n\n* Q4: Even more dark for me is line 171. T is the generative task representation. What does it mean, and how this is obtained? What is the generative task? What’s the experimental protocol for this task (if there is one)… \n\n* Q5: I am lost in Figure 5. If I am not wrong clip is represented by the yellow color (multimodal, as confirmed in Fig 3). But in the caption of the same figure you say: « lower intrinsic dimensionality only increases alignment for CLIP models ». But the supervised model (red bar) showcases a similar trend. Is that a typo? Do I miss something here?\n\n* Q6: In Figures 5, 6, there is no error bar and statistical test to assess if the shown results are significant. For example in Fig6, the differences between CLIP, SimCLR, and CLIP+SimCLR seem to be rather small and I cannot evaluate if those differences are significant.\n\n* Q7: In Figure 3, would it be possible to include inter-human reliability (i.e. how much different participants are aligned with each other)\n\n* Q8: Is it not clear how to control the number of training data in CLIP and SimCLR? In CLIP, this is pairs of data (language/image) whereas SimCLR is only on images (and it’s an augmented version, but it might not count as it is not external information per say).\n\n* Q9: The authors suggest that the training size is a confound (and I agree), but the model’s number of parameters might also be a confound (that might be related to the training size). This is a problem because most of the multimodal models have generally a high number of parameters (see Fig B). How do I know if this is multimodality is the number of parameters that increase the human alignment? Would it be possible to run a similar experiment as in Fig 6, but testing the number of parameters? \n\n* Q10: I have the feeling there is no enough data (and a too small variety of tested model) to back the claim : « We found that only one alignement method improved predictive accuracy » (line 15). For example, for the harmonization technique only supervised models have been tested, and several other models have been tested with gLocal. How do you know that « harmonising » other type of model do not improve the predictability on your task. And I am even sure that the tested architectures are the same for the supervised harmonized and supervised gLocal models ? How do you want me to be convinced if on top of the alignement methods, other parameters can vary (as architecture…). This analysis clearly lack rigor...", "limitations": "The authors properly discuss the limitation of their work", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This article introduces a novel behavioral benchmark to compare humans and machines. This benchmark (provided that the data are public) should be a useful asset in the human-machine comparison toolbox. The number of tested models is both large and also diverse.", "weaknesses": "I found 3 weaknesses: \n* 1 - Given the data presented in the article, I have the feeling that claiming that ‘multimodality better predicts human performance’ is a bit overstated. It seems to have plenty of confound factors (besides the training side) that haven’t been explored and that might prove that this claim is not entirely true\n* 2 - Some of the parts in the article are very unclear with several details missing. I have the feeling I did not have clear enough to judge the methods used by the authors (which is problematic). Please refer to the question section for more details.\n* 3 - The last analysis (on the alignment method) is pretty weak compared to the rest of the article (not enough models and not enough model diversity, no statistical analysis…). \n\nNote that I am willing to increase my rating if the questions are properly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (...) claiming that 'multimodality better predicts human performance' is a bit overstated.\n \nWe agree that many factors contribute to alignment, as detailed in Figure 4. However, the comparison of SLIP, SimCLR, and CLIP models in Figure 6 suggests image language training is important, as architecture and dataset are matched and downstream performance is comparable [1]. We acknowledge there may be unexplored factors and welcome suggestions for additional comparisons.\n\n> Q1: The section 3 is not clear enough.\n\nFor category learning, we used a mixed-effects logistic model to predict correct responses per trial. Main effects included intercept and trial number, where the intercept models above chance performance and the trial number models learning. For reward learning, we predicted which image was chosen using a similar model with trial number, reward difference, and their interaction as main effects. Here, the interaction captures the learning effect. The same variables were also used as random effects for individual differences.\n\n> Q2: (...) For example (line 98): what does (Beta, z) relate to? Is that the intercept and the number of trials? \n\nThe first set of numbers are for the intercept and the second for the trial number. $\\hat{\\beta}$ is the estimated regression coefficient. z is the test statistic comparing predictor coefficients against their standard errors to test the null hypothesis that the predictor’s coefficient is 0.\n\n> Q3: In section 4, there is not enough detail on how you train the linear probe\n\nWe added the following details to the Appendix:\n\n**\"For the category learning task, we used an  $\\ell_2$ regularised logistic regression model. We relied on `scikit-learn`’s `LogisticRegression` class which optimizes the following objective:**\n\n\n$$\\mathbf{w}^* = \\underset{\\mathbf{w}}{argmax} \\sum_{i=1}^N -c_i \\log(p(c_i | \\mathbf{x}_i, \\mathbf{w})) - (1-c_i) \\log(1-\\log(p(c_i | \\mathbf{x}_i, \\mathbf{w})) + \\dfrac{1}{2} ||\\mathbf{w} ||^2_2$$\n\n**For the reward learning task, we used a Bayesian linear regression model to infer a posterior distribution over regression weights. We relied on `scikit-learn`’s `BayesianRidge` class which infers a posterior distribution assuming spherical Gaussian priors (i.e., $p(\\mathbf{w}) = \\mathcal{N}(0, \\lambda^{-1}\\mathbf{I})$) and Gaussian likelihood (i.e., $p(y_i | \\mathbf{x}_i, \\mathbf{w}) = \\mathcal{N}(\\mathbf{w}^{\\top}\\mathbf{x}_i, \\beta^{-1})$). Here, posterior distribution can be computed in closed form:**\n\n$$p(\\mathbf{w} | \\mathbf{X}, \\mathbf{y}) = \\mathcal{N}(\\mathbf{m}_N, \\mathbf{S}_N)$$ \n$$\\mathbf{m}_N = \\beta \\mathbf{S}_N \\mathbf{X}^{\\top} \\mathbf{y}$$ \n$$\\mathbf{S}^{-1}_N =  \\lambda \\mathbf{I} + \\beta \\mathbf{X}^{\\top}\\mathbf{X}$$\n\n**where $\\mathbf{X}$ and $\\mathbf{y}$ denote the stacked inputs and targets respectively.**\n\n**We run both models from scratch on each trial using all previously observed input-target pairs. The choice of these models was motivated by previous investigations in similar but low-dimensional settings [2,3,4].“**\n\n> Q4: What is the generative task? \n\nT is the ground truth of the task, obtained by [5]. They trained a low-dimensional sparse embedding model on the odd-one-out similarity judgements on THINGS. For both tasks, participants were assigned to conditions defined over one of the top three embedding dimensions (Figure 1C). Please see lines 72-91 for a detailed account.\n\n> Q6: In Figures 5, 6, there is no error bar and statistical test to assess if the shown results are significant.\n\nPlease see Rebuttal Figures 5 & 6. In Figure 5, we only report the intrinsic dimensionality analysis because the rest are not significant, which we earlier pointed in Appendix Table 3. We hope this also addresses Q5. For Figure 6, we found CLIP $R^2$ values were higher than SimCLR $R^2$ values ($\\hat{\\beta} = .014, t=1.98, p=.048$), and SimCLR + CLIP $R^2$ values were higher than CLIP $R^2$ values ($\\hat{\\beta} = .03, t=3.91, p<.0001$).\n\n> Q7: In Figure 3, would it be possible to include inter-human reliability\n\nAs the sequences of observations were randomly assigned and not identical, inter-rater reliability cannot be calculated.\n\n> Q8: Is it not clear how to control the number of training data in CLIP and SimCLR?\n\nWe'll add: **\"Comparing image-language pretraining with augmented image training remains challenging, as language may provide information beyond an augmented version of the same image.\"**\n\n> Q9: the model's number of parameters might also be a confound\n\nModel size is important (Figure 4B), but some smaller CLIP models outperform larger ones, and some large self-supervised models are outperformed by smaller CLIP models. Appendix Table 3 shows no relationship between size and score for CLIP models, suggesting performance isn't solely driven by model size.\n\n> Q10:I have the feeling there is no enough data (and a too small variety of tested model) to back the claim : « We found that only one alignment method improved predictive accuracy » (line 15).\n\nWe  added more baseline comparisons and tested alignment methods on two additional datasets (Rebuttal Figure 1 & 2). DreamSim consistently improves alignment in its test set, gLocal shows the largest improvements in our task, while Harmonization degrades alignment. We are limited to the models trained by the original authors, which allows us to only offer a qualitative report. Harmonization alignment is only possible for supervised models by the nature of the method.\n\n[1] SLIP: Self-supervision meets Language-Image Pre-training, Mu et al., arxiv 2021\n\n[2] Learning strategies in amnesia, Speekenbrink et al., Neuroscience & Biobehavioral Reviews 2008\n\n[3] A unifying probabilistic view of associative learning, Gershman, PLOS CB 2015\n\n[4] Heuristics from bounded meta-learned inference, Binz et al., Psychological Review 2022\n\n[5] Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Hebart et al., Nat. Hum. Beh. 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive and helpful feedback. Their input was immensely valuable to further improve our manuscript. The reviewers’ assessment was overall positive, with each reviewer recommending an initial score of at least borderline accept:

- Reviewer pzQG saw our work “as providing a strong contribution to understanding human vs. model alignment” and mentioned that they “would like to see it published at NeurIPS” given that the pending concerns are addressed.
- Reviewer NCLd highlighted that our work is an “important and timely contribution” to the field.
- Reviewer MCn8 stated that our paper “makes substantial contributions.”
- Reviewer PLVt said that our work is a “useful asset in the human-machine comparison toolbox.”

In response to the reviewers' feedback, we have made the following major modifications to our manuscript:

- We have compared the different alignment methods on two other datasets to address the data confound of the gLocal models (Reviewers MCn8, PLVt).
- We have added an analysis comparing our proposed metric to previous alignment metrics (Reviewer NCLd).
- We have added an analysis and a discussion on fitting a mapping between the model features and participant responses (Reviewer pzQG).
- We have added additional descriptions of our model and fitting procedure to the Appendix (Reviewers pzQG, PLVt).
- We have added several new analyses, such as including participants behaving at chance level, to show that our results are robust (Reviewers NCLd, MCn8).
- We now also report average accuracies in addition to $R^2$ values (Reviewer MCn8).

We describe these and other smaller changes in detail in our responses to the individual reviews below. We again want to thank the reviewers for their time and for actively taking part in the review process.

Meta-review: All reviewers agreed that the paper presents a strong contribution to understanding how strategies learned by today's deep neural network models of vision compare to those of humans during object classification. I recommend this paper for acceptance, with the caveat that I hope the authors can include a section in the appendix of their revision where they validate models on an established benchmark (e.g., ImageNet) to increase confidence in the implementation of each. This is particularly important with the figures created over the rebuttal period, which show mismatches in how models relate to different alignment benchmarks.
Decision: Accept (poster)