Alignment with human representations supports robust few-shot learning
Authors: Ilia Sucholutsky
Thomas L. Griffiths
Venue: NeurIPS 2023 spotlight (2023)
Keywords: representation learning
supervised learning
human alignment
few-shot learning
TL;DR: Models that are highly aligned with human representations are better at few-shot learning and are more robust to domain shift and adversarial examples

Abstract: Should we care whether AI systems have representations of the world that are similar to those of humans? We provide an information-theoretic analysis that suggests that there should be a U-shaped relationship between the degree of representational alignment with humans and performance on few-shot learning tasks. We confirm this prediction empirically, finding such a relationship in an analysis of the performance of 491 computer vision models. We also show that highly-aligned models are more robust to both natural adversarial attacks and domain shifts. Our results suggest that human-alignment is often a sufficient, but not necessary, condition for models to make effective use of limited data, be robust, and generalize well.

Alignment with human representations supports robust few-shot learning
Ilia Sucholutsky Department of Computer Science
Thomas Griffiths
Departments of Psychology and Computer Science
Princeton University 1r226t1princ edu
Princeton University
tomg@princeton.. edu
Abstract
Should we care whether Al systems have representations of the world that are simi- lar to those of humans? We provide an that there should be U-shaped relationship between the degree of representational aligmment with humans and performance on few-shot learning tasks. We confirm this prediction empirically, finding such relationship in analysis of the perfor- mance of 491 computer vision models. We also show that highly-aligne models more robust both natural adversarial attacks and domain shifts. Our results suggest that human alignment is often a sufficient, but not necessary, condition for models make effective use of limited data, be robust, and generalize well.
analysis that suggests
Introduction
As Al systems increasingly deployed in settings that involve interactions with humans, exploring the extent to which these systems are aligned with humans becomes more significant. While this exploration has largely focused the alignment of the values of Al systems with humans 7 [6), the alignment of their representutions is also important. Representing the world in the same way is precursor to being able to express common values and to comprehensible generalization. To the extent that humans have accurate representations of the world, representational alignment is also an effective source of inductive bias that might make it possible learn from limited data.
As motivating example, imagine a meeting between a 16th century alchemist and a 21st century chemist. They live in the same physical world and are intimately familiar with the materials that comprise it, but they would have significant difficulty expressing their values and generalizing the results of an experiment they observe together. The alchemist would likely learn poorly from examples of a reaction demonstrated by the chemist, not having the right inductive biases for the way the world actually works. The alchemist and the chemist lack representational alignment they represent the world in fundamentally different ways and this impedes generalization and learning.
In this paper, provide theoretical and empirical investigation of the consequences of repre- sentational alignment with humans for AI systems, using popular computer vision tasks as way to study this phenomenon. We define representational alignment the degree to which the latent representations of model match the latent representations of humans for the same set stimuli, and Several recent papers have proposed ways to measure [23] 24), explain 26 20 and even improve [2] the representational alignment of models. However, many models that score low these alignment
refer to models that representationally aligned with humans being "human-alignee
metrics still have high performance on downstream tasks like image classification 20 26 6
So, there any real advantages (or disadvantages) using human-aligned models? To answer this question, we develop an information information-theoretic theoretic framework for analyzing representational alignment. This framework enables us to make predictions about emergent behavior in numan-aligne models.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

REPRESENTATIONS
Dhir
AGG
Zage
Lenur
Clees
0.0
0.4
Exple
0.8
0.2
0.6
Alighment
Chery
Rage
Exgl+
Cherp
Lemar
Chrip
0.4
0.6
0.5
0.1
0.8
Figure I: Schematic of representational alignment between two agents, Alice and Bob. A: Shared data (x) is shown both agents (images from the animal subset of the similarity datasets from Peterson et al. (29 B: Both agents form representations (zx and fB(x)) of the objects they observe. C: Agents are asked to produce pairwise similarity matrices corresponding their representations. The similarity judgments can then be compared to measure alignment between the agents.
A-NEW
TEACHER
STUDENT
Chimp
Encl
Got
Well
Tiger
Eagle
Vell
Lerrur
abject to Wall than to Lemur!
Figure 2: Schematic of triplet-based supervised learning where one agent a teacher and the other student. A: new object is shown only to the Teacher. B: The Teacher forms a representation the object and sends the Student triplet relating the new object to two objects previously observed by both agents. C: The Student interprets the triplet in their own representation space and eliminates the half-plane where the new object cannot be located (shaded in red) according the triplet.
In particular, our framework predicts that few-shot transfer learning performance should have U-shaped relationship with alignment. To verify the predictions of our framework and to probe for additional properties that arise as a consequence of alignment, we conduct series experiments in which we assess the downstream properties of buman-aligned models compared to their non-aligned counterparts. From our experiments comparing 491 large computer-visie vision models to over 425,000 human judgments (across 1200 participants), we identify three properties:
Models with either high or low alignment with humans are better at few-shot learning (FSL) than models with medium alignment, even correcting for pre-training performance.
Human- -alnan- models are more robust to natural ad adversarial examples, even when correct- ing for classification performance on the pre-training dataset.
Human -amman-aligned models are more robust domain shift, even when correcting for classifica- tion performance on the pre-training dataset.
The U-shaped relationship between alignment and few-shot learning helps to explain why previous results have not consistently observed benefits of representational alignment, Overall, our results suggest that representational alignment can provide real and significant benefits for downstream tasks, but that it may be sufficient rather than necessary condition for these benefits to emerge.

---

2
[6] This concept, referred to as Al alignment, is an important Related Work
With AI systems entering mainstream usage, aligning these models with human values and under- standing is increasingly important
but still largely open problem 43 in part due to the difficulty of formalizing it [34, or even reach-
ing consensus on a definition [66] In this paper, we focus on formalizing a specific aspect of AI
alignment: representational alignment.
Representational alignment is measure of agreement between the representations of two learning agents (one of whom is typically a human). There are numerous names, definitions, measures, and uses of this form of alignment across various fields, including cognitive science, neuroscience, and machine learning. Some of the other names include latent space alignment [88, concept(ual)
alignment [35 26 system alignment [8] (30 I. representational similarity analysis (RSA) [17] and model alignment (24]. Shepard (32 proposed that human representations can be recovered by using behavioral data to measure the similarity of set of stimuli and then finding embeddings that satisfy those similarity associations using methods like multi nulti-ddmensional scaling (MDS). Similarly, in neuroscience, Representational Similarity Analysis (RSA) is a popular technique for relating neural, behavioral, and computational representations of the same set of stimuli via similarity analysis (17]. While similarity analysis has clearly proven itself to be a powerful method, exhaustively collecting pairwise similarity judgments is expensive (O(N2) judgments for N stimuli) and there have been numerous proposals aiming to develop more efficient methods of recovering human representations.
Jamieson and Nowak [15] proposed an active learning scheme for querying for human judgments using triplets of the form "is closer to than to c?" and derived bounds on the number of required queries for lossless completion of the full similarity matrix using such queries. When an approximate solution is acceptable, Peterson et al. [29] showed that pre-trained computer vision models can be used approximate human perceptual similarity judgments over images. Marjich et al. (23) showed that human perceptual similarity can be more accurately, but still efficiently, approximated from natural language descriptions of the stimuli of interest (for example by using large language models to estimate similarity over pairs of these descriptions). Marjieh et al. [24] extended this result to more domains (vision, audio, and video) and measured alignment for hundreds of pre- pre-trained models.
Several recent studies have also attempted to identify what design choices lead to improved rep- resentational alignment in models [20 26 6 although Moschella et al. (25) found that even with variation design choices, many models trained on the same dataset end up learning similar 'relative representations' (embeddings projected into a relational form like similarity matrix), or in other words, converge to the same representational space. Tucker et al. (38) showed that representational alignment emerges not only in static settings like image classification, but also dynamic reinforcement learning tasks involving human-robot interaction. Several studies have also focused on studying alignment specifically in humans, both between different people and for single person but across
multiple tasks and domains B 30
Although several recent papers have proposed ways to measure 23 24],
explain 26 20), and
even improve 29 6 the representational alignment of models, few have focused on studying the downstream impact of a model being representationally aligned with humans, and many studies simply rely on the intuition that better alignment leads to better performance to justify pursuing increased alignment. While there is recent evidence to suggest that alignment may help humans learn across domains and perform zero-shot generalization [1. there is also evidence to suggest that alignment may not always be beneficial for models, with models scoring low alignment metrics
achieving higher performance on downstream tasks like image classification 20 26 6) Our goal in this paper is to conduct in-depth study into the downstream effects of representational alignment. Our theoretical and empirical results both validate and explain the apparently conflicting results seen in previous literature special cases of the (more complex than previously suspected) effects of representational alignment.
3 Theory
If we want to measure representational alignment across different architectures with potentially mismatched embedding coordinate spaces and dimensionalities, then we need definition of represen- tation spaces that enables comparisons of such spaces. Inspired by the cognitive science literature on
3

---

using non-man non-metric similarity triplets to recover hidden psychological representations, and building on rigorous computational theory that analyzes such triplets [[5], we propose triplet-based definition of representation learning. We summarize representational alignment under our framework in the schematic shown in Figure
Definition 3.1. As proposed by Jamieson and Nowak [5]. for an ordered set of n objects in d dimensions represented as a vector RTd, similarity triplet corresponding the question 'is I closer to 2 than to zę ?' is membership query of the form I where Fijk = fz E RTd
l0, Fygl < [2; m/l).
Definition 3.2. For an ordered set of objects I E Rnd and model M with embeddings fM R² + Rdar the triplet-based representation space of M is the set of all triplets Sx) [(i,j,k) e fI....,m} 1 1e * j.j + k, * i} where M rijk = fo E Rnd (fa) - fM((z)) <
[fM(2.)) - ffM(())))
Remark 3.3. Since switching the order of j and k in triplet deterministically flips the query between 0 and 1, it is easy to see that, for a set of objects, a representation space determined by a set of a(n-1)(n-2) unique triplets. Thus, for a set of n objects, Sa (I) can be represented as a binary just 2
vector of length mnn- and we use this interpretation in the remainder of this section.
Definition 3.4. Consider training set E Rnd and a corresponding target representation space Sr(()). We define representation learning with a parametrized model Ma as optimizing the objective ming (z) Sa,, (z:) where E is a loss function penalizing divergence between the target and learned representation spaces.
Sucholutsky et al. (36] showed that many ML objectives and associated supervision signals, ranging from similarity judgments for contrastive learning to labels for classification, can be converted into similarity triplets making them compatible with this definition of representation learning. We also immediately get an intuitive definition of alignment between two agents (e.g. a model and person) as the rate of agreement
matched triplets (also known simple match coefficient).
Definition 3.5. Consider a dataset J E RRd and two agents, A and B, with corresponding represen- tation spaces A and (z)) We define representational misalignment between A and B as DR(A, B: x) [S.(z)-Sp(zi) Representational alignment is then simply 1 (R,
m(n-l)((n- 2)/2
Definition 3.6. Consider single set of three points t = (I,, (I, Tj mx) E X S 3d sampled uniformly at random, and two agents, and B. We define stochastic representational misalignment between A and B as D (A,, X) simply 1 (P,A,
P(1 xerg ige = 1 ser ). Stochastic representational alignment is then
Since each triplet corresponds to one bit of information [5] and we have a probabilistic definition of alignment, we can now use information theory to define what it means for agent to supervise another (potentially misaligned) agent. We visualize supervised learning under our framework in Figure 2 For perfectly aligned models, Jamieson and Nowak (15) derive the following result.
Lemma 3.7. Consider input space X C RTd shared data x ~ X, new object e E Rd, and models A and B with (p(A, B; X)) 0. (nllog(n) triplet queries are required for B to identify the location of relative to all objects in x.
Proposition 3.8. For input space X C Rnd, shared data X, new object E Rdd and models triplets required for B to learn c.
A, B with Dr (A B; X)
( ( (00
d dlog(n)
Proof. Consider communication game involving Alice and Bob, a dataset that they both have their own representations for (we call this 'shared data'), and new object that only Alice can see. Alice can only communicate with Bob by giving binary responses to triplet queries based on Alice's own representations of the objects. The goal is for Bob to learn the location of e with respect to the other objects in x. (A, B; X) = is the probability for any triplet drawn from x c to be flipped in Bob's 's representation space relative to Alice's. s. This is equivalent to Alice and Bob communicating over binary symmetric channel where the probability of bit flip is c. The capacity of this channel is C. = 1 Hee, 1 - <) 1 + log(e) + (1 +) log(1 - e). For error-free communication over this channel, the highest achievable rate is < C., meaning at least T. bits are required to communicate each useful bit over this channel. By Lemma 3.7
useful bits are required for Bob to learn
dllg(n)) ) bits are required. the location of over channel with no error, and thus a total of 2 C,
D

---

Theorem 3.9 (Few-shot learning). Consider imput space X S R R", shared data x ~ X. new objects c E IR rd and three models A, B, and B2 with (A B) EB, and D (A, Bzg X) EB2 If |0.5 <B,l < 00.5 BBb then B2 requires fewer queries to learn than B₁ does.
Proof. From Proposition 3.8 it immediately follows that B1 requires O ((mdlogg))) queries and C.p.
B2 requires D md login)
queries. Ce f0, + [0. 1] is symmetric convex function with
Cens
minimum of 0 mnd log(n!)
0.5 and maxima of 1 at 0, 1. Thus, if 0.5 EB,
< |0.5 EBul. then
O and log(n)
Q Cenz
Cen,
Sucholutsky et al. [36] showed that commonly used supervision signals, like hard and soft classi- fication labels, can be converted into sets of triplets. Reducing the number of required triplets to learn the location of new object is equivalent to reducing the number of required labels. It follows from Theorem 3.9 that high or low alignment leads to high few-shot learning performance on downstream tasks like classification (where learning a new class can be formulated as learn- ing the location of the centroid or boundaries of that class using a small number of labels). while medium alignment leads to low few-shot learning performance. Intuitively, this comes result of mutual information between the teacher and student representations being minimized when alignment is 0.5. In other words, if Bob knows that most bits coming from Alice are flipped (i.e. DA B) > 0.5), then going forward, Bob can flip all the bits coming from Alice to achieve a 0.5).
lower error rate (i.e. (P B) 1
We note that generalizing under domain-shift can be considered a form of zero-shot transfer learning and selecting adversarial examples can be seen selecting objects that maximize representational disagreement between a model and a human. Under these interpretations, our framework also predicts that highly-aligned models are robust to both domain shift and adversarial examples though the framing in these cases is less intuitive than for the few-shot learning case. We share some preliminary theoretical analyses of robustness properties in the Supplement.
4 Experiments
Our theoretical analysis shows how representational alignment serve source of inductive bias that reduces the number of bits that need to be acquired from data. In particular, Theorem 3.9 predicts an unexpected U-shaped relationship between alignment and few-shot learning performance. We now present a series of experiments designed to test this prediction and examine whether it extends to robustness to natural adversarial examples and domain shift.
Models. The pre-trained models evaluated in this paper are taken from the PyTorch Image Models package 41 The full list of 491 models used in our experiments can be found in the Supplement.
Data. All of the models used in this paper were pre-trained on ImageNet- Ik [31] and had their performance evaluated on the ImageNet- IK validation set. Adversarial robustness was measured on the ImageNet-A dataset of natural adversarial examples that have been found foil most ImageNet models [11] Zero-shot domain shift robustness was measured on the ImageNet-R R dataset of renditions (e.g., paintings, toys, statues, etc.) of ImageNet classes [00 and the ImageNet-Skee -keechh dataset of black-and-white sketches of the ImageNet classes [40] All ImageNet and ImageNet variant results come from the PyTorch Image Models package [11) Few-shot transfer learning performance was evaluated on the CIFAR 100 [8] test set with n (1, 5, 10, 20, 40, 80) examples per class used for few-shot learning and the remaining examples used for evaluation. We measure alignment by computing the three metrics described below on the six image datascts from Peterson et al. (29 and their respective sets of similarity judgments consisting of over 425,000 judgments across 1200 participants, as described by Marjich et al. 23 We average each alignment metric over the six datasets.
A Alignment metrics. The representational alignment metrics we consider are correlation over pairwise similarity judgments and agreement between similarity triplets (i.e., the proportion of triplets that have the same response for the matched sets of representations). Similarity triplets provide non-metric information since in each query we discard the actual pairwise distances and only retain which one larger. Analogously, in the pairwise case, if we Spearman rank correlation then are comparing
5

---

Spormmen
Tiplits
hunnn
Sparmen
Tiplits
Lineur prate
Linesr srobe
60
0.2
40
20
2.0
acuntyy PM
2.4
1 PN
1 NN
60-
8.2
40
Corstelion
0.9
0.4-
2 100 AV
2- 2-00pt KM
60
40
high
82-
3.0
Trining exomptes per
Taining oxamples dess
Figure 3: Left: Average n-shot transfer learning performance using linear probing, 1-layer classifi- cation head, and a 2-layer classification head on CIFAR 100 of the five models with highest, lowest, and closest-to-ther toosestto-thh-maan levels for each of Pearson (pp), Spearman (ps), and triplet alignment. Right: Pearson correlation (with 95% confidence intervals) between n-shot transfer learning performance using each classification head on CIFAR 100 and each 2² alignment metric.
the relative magnitudes of pairwise distances (of which the triplets form subset) and discarding the magnitudes. On the other hand, Pearson correlation over pairwise similarities takes into account the magnitudes of the pairwise similarity judgments which, if accurate, can provide more information about fine-gra mme-grai differences that affect alignment. We use all three metrics in our analyses and refer to them as Spearman pairwise alignment, Pearson pairwise alignment, and triplet alignment.
Few-shot learning. We measure few-shot transfer learning performance using three methods: linear probing and two classifier heads. For linear probing, we take the embeddings coming from the penultimate layer of a model and fit logistic regression using 'scikit-learn' heads, we use one- and two- hiddeddenn layer neural network implemented in PyTorch (27]. both with an input dropout rate of 0.8 and the latter with ReLU hidden layer. Recent work [19 has shown that linear probing usually performs on par with fine-tuning and is a more accurate measure of the quality of internal representations that finettnning distorts. The heads are provided in the supplement and fixed hyperparameter values (selected based on exploratory experiments to confirm stable convergence) were used for every dataset/model for consistency.
(28). For the classifier
Correcting correlation for ImageNet performance. For each of our experiments, ImageNet- performance can be confounding variable when trying to understand the relationship between alignment and a downstream property. To account for this, when measuring correlation between aligmment and other properties, we compute partial correlation with ImageNet- Top-1 validation accuracy as a covariate using the 'Pingouin' Python package (39
4.1 Results
Alignment predicts few-shot learning performance. For each alignment metric, we compare the n-shot transfer learning performance of the five models with the highest alignment, the five models alignment as shown in Figure 3 We find that according all three alignment metrics, the most aligned models have far better shot learning performance at all levels of n.
with the lowest alignment, and the five models with the nearest-to-the-met
Alignment predicts adversarial robustness. For each alignment metric, also compare the performance on natural adversarial examples of the five models with the highest alignment, the five models with the lowest alignment, and the five models with the nearest-to-the the- alignment as seen in Table I We again find that according to all three alignment metrics, the most aligned models perform far better both Top-1 and Top-5 predictions for ImageNet-A.
Alignment predicts domain-shift robustness. Finally, for each alignment metric, we also compare the domain-shin shift robustness of the five models with the highest alignment, the five models with the

---

Table Average top-1 and top-5 accuracy on ImageNet-A, ImageNet-S, and ImageNet-R of the five models with highest, lowest, and closest-to-the-mean levels of Pearson (pp), Spearman (ps), and triplet alignment.
Table Pearson correlation between Top- and Top-5 accuracy on ImageNet-A, ImageNet-R, and ImageNer-Sketch (-S) and Pearson (pp), Spearman (ps), and triplet 2² alignment metrics.
Aligument metrics
Alignment metries
Pp
Ps
Triplet
Pp
As
Triplet
High 5
59.93
46,83
44.56
IN-A
0.239
0.151
0.157
IN-A (Top 1)
Mid 5
10.26
12.30
18.35
(Top 1)
(p=0.000)
(p=0.001)
(p=0.000)
Low 5
30.89
21.80
29.99
IN-A
0.210
0.112
0.121
High
84.92
73,86
72.20
(Top 5)
(p=0.000)
(p=0.013)
(p=0.007)
IN-A
Mid
36.54
38.14
46.63
(Top 5)
IN-R
0.166
0.164
0.182
Low 5
59.10
49.36
57.70
(Top
(p=0.000)
(p=0.000)
(p=0.000)
High
61.57
53.78
54.27
IN-R (Top 1)
IN-R
0.173
0.170
0.194
Mid
39.16
40.01
41.80
(Top 5)
(p=0.000)
(p=0.000)
(p=0.000)
Low 5
51.33
46.76
50.16
IN-S
0.191
0.164
0.179
High 5
75.92
68.61
68.82
IN-R (Top 5)
(Top 1)
(p=0.000)
(p=0.000)
(p=0.000)
Mid 5
55,08
56.81
57.67
Low 5
66.55
61.84
65.22
IN-S
0.211
0.197
0.218
(Top 5)
(p=0.000)
(p=0.000)
(p=0.000)
High 5 Mid
48.06
40.37
40.31
IN-S
26.90
28.02
29.85
(Top 1)
Low 5
38.11
33,90
37.04
High
71.10
62.45
62.24
IN-S (Top 5)
Mid
44.90
46.67
48.58
Low 5
58.91
53.43
57.57
lowest alignment, and the five models with the nearest-to-the-ment alignment seen in Table 1 We find that for all three aligmment metrics, the most aligned models perform far better on both Top-1 and Top-5 predictions for both ImageNet-R and ImageNet-S.
Alignment has non-linear relationships with downstream properties. Based on the results in Table and Figure 3 while models with low alignment underperform models with high alignment, they seem to consistently outperform models with medium alignment. Furthermore, when looking at the entire set of models in Figure 4 (Left), there appears to be a non-linear, potentially quadratic relationship between alignment and few-shot learning performance. This matches the U-shaped behavior predicted by Theorem 3.9 To test this relationship, we measure the correlations between the downstream properties of interest and the 2² transformations of each alignment metric where >²(x;) = (=== μ is the mean, and a is the standard deviation. We find statistically significant, positive correlations between the 2² alignment metrics and few-shot transfer learning performance as seen in Figure 3 (Right). Using 22 alignment metrics also results in slightly stronger correlations for domain-shift robustness but weaker correlations for adversarial robustness as seen in Table
Alignment is domain-specific metric. Since CIFARI00 is made up of naturalistic object images and so were the pre-training dataset and alignment evaluation datasets, we now test few-shot transfer learning performance examples per class to evaluate how alignment generalizes to datasets that look very different than the ones on which was measured. FMNIST is dataset of grayscal clothing item images. While fairly different, it is still made of everyday objects as are the datasets on which we measured alignment. Meanwhile, MNIST is dataset of grayscale handwritten digit images and is completely different from the datasets on which measured alignment. Thus, CIFAR 100 is most similar to the datasets on which alignment was measured, followed by FMNIST and then MNIST. As shown in Figure { (Rihht) for all three datasets, we find statistically significant correlations between our Pearson alignment 22 metric and few-shot transfer learning performance. However, we find that the effect size decreases for FMNIST and even more so for MNIST. This suggests that like any other metric, there are limits to how well representational alignment can generalize, and that it has the strongest predictive power when measured on datasets similar to those which downstream properties are being evaluated.
the MNIST (1 and FMNIST 42 datasets with n E (1, 5, 10, 20, 40, 80)
7

---

70
3.04p% - 3, 94.75
CIFRRDOD FMNIST KINIST
CIFA1100 1-shol aicuunayy (96)
Comelatter with test accuracy
50
0.3
0.2
30
20
10
20
Pearson pairwise alignment (ep)
30
40 50 60
70
Training examples per class in]
20
40
80 80
Figure 4: Left: Comparing 1-shot learning performance CIFAR100 Pearson pairwise alignment for all 491 models. Right: Pearson correlation (with 95% confidence intervals) between n-shot transfer learning performance using linear probing and the Pearson (pp) 2² alignment metric for CIFAR 100, FMNIST, and MNIST.
Interactions of architecture and alignment. The results above suggest that when alignment is measured on near-distribution datasets, it can predict about 20% of the variance in one-shot learning performance. This is impressive given the large number of factors we know have impact on downstream performance (architecture, optimizer, random seed, initialization, early termination, hardware effects stochasticity, and many other hyperparameters) We also investigated whether some of these factors can predict alignment or interact with to affect its relationship with downstream performance. Going through the results of our 491 models (see the Supplemental Information for the full results), we find the following insights ranked from most supporting evidence to least: I) knowledge distillation [1] seems to increase both alignment and few-shot learning performance; 2) increasing depth/size tends to increase at least one alignment and FSL performance (increasing ResNet(9] depth improves alignment but not FSL performance increasing depth for Resnet- increases few-shot learning performance and not alignment; increasing Swin Transformer 22 increases both); 3) pre-training ImageNet22 k/22k before fine-tuning ImageNetlk increases
size
both alignment and few-shot learning performance; 4) batch-normalizztion 14 for VGG 33 leads
to lower alignment but slightly higher FSL performance: 5) pruning EfficientNet slightly decrease alignment but maintain FSL performance. We also find that for many architectures, the models cluster together (i.e. almost all MobileNets
3 seems
[13] have Now-to-medium alignment and low FSL performance, almost all NFNets 4 have high alignment and medium-to-hie FSL performance, all BEiT models [2] have very low alignment and high FSL performance). These findings suggest that architecture and training decisions can impact alignment as well as its effect on downstream properties. Thus researchers may be able directly optimize for alignment when selecting, designing. or optimizing model architectures.
Discussion
Our experimental results confirm our theoretical predictions not only do highly-aligned models exhibit better few-shot learning and robustness properties than models with lower alignment, but we also observe the U-shaped relationship with alignment predicted in Theorem 3.9 We now dissect the theoretical and empirical results via three key questions.
Which alignment metric should we be using? We find that Spearman pairwise alignment is almost perfectly correlated with triplet alignment (p 0.992). This suggests that triplets, though only small fraction of the full set of quadruplets (i.e., all queries of the form "X is closer to Y than A is to B")., already capture the majority of the non-metric information contained the entire set of quadruplets. Pearson pairwise alignment has strong, though not quite as strong, correlation with triplet alignment (p = 0.824). While all three metrics have statistically significant correlations with the downstream properties we care about, Pearson pairwise alignment seems to have the strongest correlations. This suggests that there recoverable metric information about representations encoded in the magnitudes of human similarity judgments, even when these judgments potentially noisy due to being elicited

---

without anchor stimuli that would ground the scale. The information -hhooretic representation learning framework would need to be extended in future work to quantify this additional information.
Which positive downstream properties do human-aligned models exhibit? As predicted by our information-theoretic representation learning framework, our experiments suggest that very numan-aligned models are better at few-shot transfer learning, more robust to natural adversarial examples, and more robust to test-time domain shift than models with lower degrees of alignment. The correlations between alignment and each downstream property were positive and statistically significant and, in every experiment we conducted, the models with the highest level of alignment outperformed the other models. These results seem to confirm the intuition that human alignment useful in tasks where we want to use human supervision to elicit human behavior. While the models with the highest level of alignment clearly exhibit the best downstream performance across all three sets of tasks, our results suggest an additional unexpected insight: there is a U-shaped relationship between alignment and two of the properties we test: robustness to domain shift and few-shot transfer learning performance. Thus, while a high degree of alignment may be sufficient for eliciting desirable properties in models, it does not appear to be necessary. In fact, in cases where achieving high alignment is impractical (e.g., due to limitations on human-labeled data), is possible that better results may be achieved by avoiding alignment altogether.
Are there downsides and limitations to representational alignment? We have shown that alignment is not fomain-agnostte metric and that its downstream effects are strongest when evaluated on in-distribution datasets. also clear that increasing alignment can damage performance across multiple criteria when that increase moves the alignment level into the medium range. But are there downsides to increasing alignment of models that are already at or past that range? Throughout this study, one of our key assumptions was that the task being solved is designed and specified by humans, or at least easily solvable by humans. However, there are numerous domains where humans have poor performance or where representations of the problem or stimuli are not helpful for solving the task and a different set of inductive biases are required. For example, many domains targeted by deep learning- such as protein folding, drug design, and social network analysis require geometric inductive biases [5]. In these cases, the goal should be to achieve alignment with the underlying laws governing the system of interest (e.g.. physical forces mathematical laws), rather than with humans. Finally, we note that as in all cases where models are trained with human data, social biases found in the human data may be reflected even amplified in representationally alllgged models.
Conclusion
Our findings confirm the intuition that representational alignment with humans elicits desirable human- like traits in models, including the ability to generalize from small data and lower susceptibility to adversarial attacks and domain shift. However, as both our theory and experiments suggest, increasing alignment is not always desirable, first due to a U-shaped relationship between alignment and desirable traits, and second, because there are domains where human representations simply are not useful. Notably, our discovery of the U-shaped relationship serves to resolve the tension between previously conflicting findings regarding whether alignment improves performance. We hope that our framework and results motivate further study into both the positive and negative consequences of aligning models with humans across diverse domains. We believe that representational alignment is a quantifiable and tangible route for making progress on the general AI alignment problem by allowing us to measure agreement between models and humans even over abstract domains.
Limitations and broader impact. We discussed the limitations and potential negative impacts of alignment in Section but we also want to note the potential limitations and impacts of our study of alignment. First, we note that our approach assumes access to internal representations of models, but many new models today are being released via closed-source APIs that provide access only to model outputs. We hope that this and future studies on representational alignment will encourage developers to provide more open access to model internals. Second, we note that our experiments only evaluated alignment with the aggregate representational space of a large group of people which may not actually be individually representative of any of those people. Our proposed theory allows for alignment with either individuals or groups, but is difficult to collect a large set of similarity judgments from individual participants. To ensure that we can understand whose voices, biases, and beliefs our models are actually aligned with, we believe that finding a way to account for individual differences is an important future direction.
9

---

Acknowledgements
We would like to thank Lukas Muttenthaler for excellent discussions that helped shape some of the ideas explored in this paper. This work was supported by ONR grant (800014-18-1-2873) to TLG and an NSERC fellowship (567554-2022) to IS.
References
[1] K. Aho, B. D. Roads, and B. C. Love. System alignment supports cross-domain learning and zero-shot generalisation. Cognition, 227:105200, 2022. ISSN 0010-0277. doi: https:
htbb.orrg1..0016...oogggig 20tpenniitooo.22....... 1022000
[2] H. Bao, L. Dong, S. Piao, and F. Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https://
[3] I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T.-Y. Lin, Shlens, and B. Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing
Systems, 14-22227- 2021.
[4] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059-1071. PMLR, 2021.
[5] M. M. Bronstein. J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2 13478, 2021.
[6] Fel, I. Felipe, D. Linsley, and T. Serre. Harmonizing the object recognition strategies of deep neural networks with humans. arXiv preprint arXiv:2211.04533 2022.
[7] Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020.
[8] R. L. Goldstone and B. J. Rogosky. Using relations within conceptual systems to translate across conceptual systems. Cognition, 84(3):295-320, 2002. ISSN 0010-0277. doi: https: //doi.org/10. 101 6/50010-027700000000000000 000000000000227770000555---
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[10] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of -utto--distribution generalization. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
pages
[11] D. Hendrycks, K. Zhao. S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271 2021.
[12] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint 2015.
1rXiv:1553.025::
[13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 7704.04841, 2017.
[14] S. loffe and C. Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 458-456. pmir, 2015.
[15] K. Jamieson and R. D. Nowak. Low-dimensional embedding using adaptively selected ordinal data. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1077-1084, 2011.
[16] H. Kirchner, Smith, Thibodeau, K. McDonell, and L.. Reynolds. Researching alignment research:
Unsupervised analysis. arXiv preprint arXiv:2206.0288 2022.
10

---

[17] N. Kriegeskorte, M. Mur, and P. Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. doi: 000099//eemo0000042400000
[18] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pre- trained features and underperform out-of-distributiinn In International Conference on Learning Representations, 2022. URL http:tt//ppppeeeeiiiee
[20] M. Kumar, N. Houlsby, N. Kalchbrenner, and E. D. Cubuk. Do better ImageNet classifiers assess perceptual similarity better? Transactions of Machine Learning Research, 2022.
[21] Y. Lecun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11) (((11):22782333)) 1998. doi: 10.1 109/5.726791.
[22] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[23] R. Marjich, Sucholutsky, T. R. Sumers, N. Jacoby, and T. L. Griffiths. Predicting human similarity judgments using large language models. arXiv preprint arXiv:2202.04728, 2022.
[24] R. Marjich, P. van Rijn, I. Sucholutsky, T. R. Sumers, H. Lee, T. L. Griffiths, and N. Jacoby. Words are all you need? Capturing human sensory similarity with textual descriptors. arXiv 2022.
preprint arXivv2206.044
[25] L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint MXXx:2209..1430 2022.
[26] L. Muttenthaler, J. Dippel, L. Linhardt, R. A. Vandermeulen, and S. Korblithh Human alignment of neural network representations. arXiv preprint arXiv:2211 (..i::2211.0221,, 2022.
[27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison. A. Kopf, E. Yang, Z. Vetioo, M. Raison, A. Tejani,
S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 5024-8035. Curran Associates, Inc., 2019.
[28] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-288 2830, 2011.
[29] J. C. Peterson, J. T. Abbott, and T. L. Grifliths. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive science, 42(8):2648 26699 2018.
[30] B. D. Roads and B. C. Love. Learning as the unsupervised alignment of conceptual systems. Nature Machine Intelligence, 2(1):76 82, 2020.
[31] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 1 115(3):21 11-252, 2015.
[32] R. N. Shepard. Multidimensional scaling, tree-fitting, and clustering. Science, 210(4468): 390-398, 1980.
[33] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[34] N. Soares and B. Fallenstein. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report. 8, 2014.
[35] A. Stolk, L. Verhagen, and I. Toni. Conceptual alignment: How brains achieve mutual un- derstanding. Trends in Cognitive Sciences, 20(3):180-19) 2016. ISSN 1364-6613. doi:
http:://dii.or/11..001. 10w.///.......0////1 1.07.
[36] I. SSuhholusky, R. Marjieh, N. Jacoby, and I. Griffiths. On the informativeness of supervision signals. arXiv preprint arXiv:2211.01 2022.

---

[37] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, 14. PMLR, 2019.
[38] M. Tucker, Y. Zhou, and A. Shah. Latent space alignment using adversarially guided self-play. International Journal of Flmman-Compttr Interaction, 38(18-20) 2022.
[39] R. Vallat. Pingouin: Statistics in Python. Journal of Open Source Software, 303) 2018. doi: 10.21105/joss.00022.
[40] H. Wang. S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing
local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[41] R. Wightman.
PyTorch Image Models.
httpp://ggtthab..cmm/mmgggg.iai
pytorch- image-models 2019.
[42] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:170..07444 2017.
[43] E. Yudkowsky. The Al alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 2016.
12

---

A Additional theoretical results
Definition A.1. Consider model with input space X C RTd previously observed data x X, and k class centroids E R&dd learned by A. We define domain shift as an update to the class centroids c + c* R&d, Domain shift sensitivity then the proportion of triplets flipped as result of this update.
a (c)c, = E |A c)
SA(2, c
[S((2; c)l
From this definition and Theorem 3.9. it immediately follows that sensitivity to domain shift should have the same U-shaped relationship with alignment that few-shot learning does in cases where the teacher model robust to domain shift.
Corollary A.2. (Alignment and domain-shiff robustness), Consider input space X C RRd shared data x ~ X. and three models, A. B1, and B2 with (p B1:X] = EB, and Dp (A, By; X) EB2''
Let c E IR kd be k class centroids learned by A, B1 and Bg. If af, = and |0.5 - <Bi << |0.5 l then оB, (c,c*) (c,
We can also use this framework to define robustness to adversarial examples. We assume that an adversarial example is an object that maximizes perceptual (i.e. representational) disagreement between the teacher and the student.
Definition A.3. Consider imput space X C Rnd shared data æ X,, and two models, and B, with DA B1; X) = An adversarial example is an object E IR that maximizes disagreement between A and B on S(z;e), the subset of nnn 1)/2 triplets relating the objects in e.
e = max ( - SB(x; e)ll
(I)
we we immediately get the following result. Lemma A.4. Consider an input space X C Rnd. and two agents, A and B. DP(A, =
Using Definition 3.6
E m(n 1)m22)/
We can now show that a model that is more aligned with the teacher will, on average, also be more robust to adversarial examples.
Theorem A.5. (Alignment and adversarial robustness). Consider input space X C Rnd shared data x ~ X. and three models, A. B1, and B2 with (p X) EB, and Dp(A, By; X) EB, (B < <BB, then E) [maxee [}5A(=;)) SB, (2; e]l]] Flmmaxexx [1; e) Sa, (z; ell]
Proof. Note that for a set of & binomial random variables X; ~ Bin(n, p). the expectation of the Fwx; n. p))) where F(x; n, p) = P(X, S s). In the case of adversarial examples, let X, be a random variable corresponding to the set of objects sampled uniformly from the input space X C R nd then U = [XA(();)) - SB. (X;e)ll1.3 Y ~ Bin(n(n- SB, (X;e)ll1,V V Bin(n(n 1)/2, EB, In that case. the expected disagreement of and B, on an adversarial example ((U())) En 2n(a-1/2(1 1)/2, <B, )")
k-th order statistic E[ ([)(k)) = 0
1)/2, and similarly V = =I|SA((;e
F(2; n 1 1)/2, )r) and for A and B2 it is ([)))) Ex= 1)/221 (1-F If ef < <B then F(x;; 1(n > F(; 1)/2,cR, and thus (.U) < F(V.))) 11
Remark A.6. While this theorem shows that increased alignment generally leads to increased adversarial robustness, this relies on representational metric of adversarial examples. However, in practice, adversarial robustness is often measured using hard classification error as simple proxy. This proxy does not capture the fine- aineggraieed degree of misalignment between humans and a model on each example. As a result, when measuring adversarial robustness using this proxy, the effect of alignment may dampened by the U-shaped effect seen other classification settings as mentioned above.
B List of 491 models used in experiments
adv we-_inception_yee v3, _3,
beit beit_base___ paasepptchh66.. 1achh16224444 beit base cactee_pahhl___11 33,44, beit harre_ppachlle 224, beit, saarge_patchllle botnet26t_255 256, cait $24 224, cait, $2i__24__250 _384,
bat_ ress
13

---

006463444000 cait xs24_ as24_384, cait xxs_4_224____ 224, cait xxs24_384 cait xxs36 224, cait xxs__xx366__00000 38_,,, coat lite mini, coat lite small, coat lite tiny, coat mini, coat tiny, convit base, convit small, convit_tiny, convmixer_102 1024_20__0 20_ks9 __1,, convmixer_ 1aee__1366_220 20, convmixer_ 2aaes__68_3__ con- vnext_base, convnext convnext_hase 3833_8n422fF Laa_llree____nn222iiii_
convnext_ _aaseet__bb_____e _ants__n_e___ntkk convnext laapeexx_llgeein22fttt
convnext_large con-
convnext_ small,, convnext_ting csp- darknet53, 0spresnet50, cspresnext50, deit cest_base__atle www.__astccchb....m 1__4__ deit_ ceit_bas__patchl6. deit fin__smalllpatchh6 _24, ceii__tiy__ue siny_patchhe 2224, densenet121, densenet161, 16 densenet201, densenetblur1 dla102, dla dla102x2, d9a dla34, dla46_ _la dla46x_c. dla60, dla60 res2net, dla60 res2next, dla60x, dla60x dm nfnet f0, dm nfnet fl, dm nfnet f2, dpn107, dpn131, dpn68, dpn68b, dpn92, dpn98, eca botnext26ts 256, eca halonext26ts, eca nfnet 10, nfnet 11, eca nfnett 12, eca resnet33ts, ecaresnet269d, ecaresnet26t, ecaresnet50d, ecaresnet50d_ pruuned ecaresnet50r, ecaresmetlight, efficientnet_bil bO, efficientnet_bi b1, efficientmet_bll _pppnedd,,,___,---ee efficientnet_br b2, efficientnet_h22 prreruuuee______ruded_ee_e pruned, efficientnet_b3. @fficientmet_b3_prrneee efficientnet_enn em, efficientnet_es es, efficientnet_es perunnede_____edddd__eeeee dfficienttet_litte) efficientnetv2 r___, efffiiientnevv__r..cc _r___, efficientnntv2__v. ___ ens inneption_resnel v2, ese_ ese_vovnet19b_ee ese_ vovnet39b, fbnetc_100, fbnntv3_b, fbnetv3_d, fbnetv3_g, gg__fficiietnetv2_nw_L geres- net33ts, gcresnet50t, gcresnext26ts, gcresnext50ts, gernet_, gernet_m, gernet_s ghostnet_100, vlb, seuon_resmetio vlb,
resnext26ts, ecaresnet ecaresnet1Old prumed,
efficientnet_hh __e,tttttetttttt_eeeee efficientnet_el, Pnacientmet_el_prrunnee
coto__nccptton__ v3.
_ gluon_resmetion _vvidd gllon_resnette vlc, gluon_ @luon_resnet152 vld, gluon resnet34 __bb,
iisoonresnet111111 101 vls,
gluon_ resmet50 vlb, (ea_reeneet011_24444e gluon_resnext101 (xxxxxx00006444d,,,__ deluon_resnxx500 __xxxxx gluon_senet1 gluon_seeesneee seresnext101 __32xd,, Setnneeeeeeeeeeeeeeeee_ _3x4d, Xicoo_xception65.. gmixer 24 224, gmip_s16_222 22,4,, halo2botnet50ts. 2_6,,,_566___ halonet26t, halonet50ts, haloregnetz.b. _b, hardcorenas, _a hardcorenas_b. b. hardcorenas_c, hardcorenas_d. hardcorenas_c. hardcorenas_,, hrnet_wl hrnet_w32, hrnet_40, hrnet_w44, hr- Rrsnext1011322x60
@luoo_resmet152 vls,
gluon 1esooneeeeett11_11 vlb, gluon_reenette
gluon_resnett0__ye
gluon_seresnext101 (x4
Amer_w18_small hrnet_w18 118__mmall_i hmmetw18_small__e hrnet_w30,
inception_resnee v2, inception__v _3, inception v4, jx nest base. jx nest small, jx nest tiny, lambda renorddrreeet2ppt 256, lambda resnet26t, lambda resnet50ts, lamhalobotnet50ts 256, lenet 050, lenet 005, Ic- net_100, Sreacy_senet154 legacy_ seresnet101, legacy seascys_rresne15222 legacy_seresnet34,, Sggacy_seresnetti legacy_ Sgracy_seresnext101 __32xd,, Beaecyssereenex22__552
net_w48, hrnet_w644 ig rg__esnext101_ 1aaxxt0011_xxxxdd
legacy Jegacy_scresnetlS scresnet18,
mixnet_ mixnet_ mixnet_s, mixnet_xl, mnasnet_ 100, mnasnet small, mobilenetv2 050, mobilenetv2_ 100, mobilenetv2_1104 mobilenetv2_11200
__x3_d,__xd_____ mixer_ 616__1166__1 2__24, mixer_bll ______16622_______1111111
mobilenetv2_1400,
Baatlenetv3_lrge_Juue
mo-
bilenetv3 Laannetv3large__0000 100rge_100mmiii mobilenetv3_rg __w,, nasnetalarge, nf nf_regnet_bl, of re_eee05000, nfnet_1O.
pit_b_224, pit_s_224, pit_ti_224, pit_xs_224 pnasnet5large, regnetx_002 regnetx_0Ok reg- netx_006, regnetx_ 0088 regnetx_(11 regnetx_032 032,, regnetx_040, regnetx_0644 064, regnetx_080,
regnetx 120, regnetx_ 160, regnetx 320, regnety 002, regnety 004, regnety_ 006, regnety_ _008
regnety_0ue 0066, regnety _02,,003320000 regnety_04g regnety_ 064,
080, regnety_122) 120, regnety_ 160. regnety 3200 regnetz_ bl6, regnetz c16 regnetz d32, regnetz d8, regnetz_cs c8, repvgg a2, repvgg_bi bO, repvgg bl, repvgg_bleg repvgg 62, repvgg b2g4, repvgg_b3, b3, repvgg_ b3g4, res2net50 26w 4s, res2net50_266
6s. res2net50 26w _8, 1__stiilleddeddd_eeee_________ resmip_22 2____24__ii 214_distilled__22 224,, resmip_36 1amip_36_2200 224, resmlp_36 @o_distilled__22
1es2net101_26w 4s, res2net500
res2next50, resmlp_12 12 12_224, resmlp_12 12 distilled 224, resmlp 24 224, res2net50_480 _0___8w_22_
resmip_big 24_224, resmip_bin big 24 20_1224_dii distilled 224, resnestl0le, resnest 4d, resnest50d_4s224006
resmip_big Saaulpbbig_24 2iiii_24222_iii 224_in22ftlk resnest200e, resnest269e, resnest26d, resnest50d, resnest50d_Is4xie resnet101, resnet101d, resnet152, resnet152d, resnet18, resnetl8d, resnet200d, resnet26, resnet26d, resnet26t, resnet32ts. resnet33ts, resnet34, resnet34d, resnet50, resnet50 gn. resnet50d, resnet51q, resnet6lq, resnetblur50, resnetrs101, resnetrs152, resnetrs200, resnetrs270, resnetrs350, resnetrs420, resnetrs50, resnetv2_101. resnetv2_IOlks resnetv2_500 assnetv2_50x1 1_tt bit distilled, resnetv2_50x1_bine resnext101 _3xxxd resnext26ts, resnext50_32xer resnext50d _3x4d, rexnet_100 rexnet_130 rexnet 150, rexnet_200 se- botnet33ts 256, sehalonet33ts, selecsls42b, selecsls60, selecsls60b, semnasnet 075, semnas- 100_ seresnet152d, seresnet33ts. seresnet50, seresnext26d 32x4d, seresnex126t_32xx4d seresnext26is, seresnext50_32x44 skresnet18, skresnet34, skresnext50_32xxe spnasnet_100, ssl_resnet18, ssl resnet50, ssl "e_rrennxt10111 1o1___32xll0 16d, ssl red_resnext11111 _0___x4,,, ssl_ ssl_resnext101_333 __xxdd, ssl 2a__rennext50_32x44 swin_base base_pptccheeeeeeeee pltppppaaatphphaaaee wwt._wiidwww2...mmm 12_384, swin_base_paree Lar__asse_ptcch__viie patch4_windoww
swin Brrgge__aach______ window 12 _84.
swin laige__ag__pachhh window4_windww7...m
14

---

swin small patch4 wtl___iinwww__00
swin tiny pery__ptcch_._. wwwdwww_nddw2..
[oss_resnetll 18,
swsl resnet50, swsl resmextle1111 32x16d.
swsl reseeetel1111.. 32x4d.
swsl resnext101 32x8d. tf_ f__efficientnet_bll __o_nss__________ tf ff_efficientnet__i
swsl @ass_resnext50_3220 __2x4d,,
tf [f_effcceettet__b0)
tf "f_efficientnet_bbee ______b0_pppppp___
efficcentmet bl,
tf_ @f_efficientntt_ll
tf L_e_efeieetttt____ _bl ns,
If Lefficienttee_bble ____ttttt______p________
tf ff_efficientnet___ ___ns, _s, tf df_efficientnet b4, tf_efficientnet_be ______________________________ee
tf Lfffcieenteet__... b3,
tf L_effccieennet__b3. sareeet_bb3___0 _b3__p. tf_efficientnet_b4____
If effficientttt Bacpieettb___mm__ b3_ns, If wwf.icientnettiiecee b5,
tf_ [[_efficientnet__ b5_ ___p.
If ef_efficieette_t__ b5 ns,
tf wf__fficceettet__eee
if ff_efficieentett_nee __t_neet_bb________ee_ ns. If efficientnet b7 tf ewficientnetttcccc b0 4e, tf efficientmet cc b0 8e, tf wwf.ecieetttettecce cc b1 8e, efficientnet el, tf effeeciennneet_eee em, tf efficientnet es, tf efffccientnett liteO, tf efficientnet litel, tf #ffffficieetett22 b,, of #f_effccientmety211.
If Leefficienttet_ieeee Bactitttt_________e_ b6_ap,
tf effiiieenteet_bi b7. tf efffciieentee__bi ____pp
If efficientnet lite2, tf efficientnet lite3, tf efficientnet lite4,
Lefficcienteet2211
ff_efficientmetv22 62, @ in221ft,,,,, tf ff__efffcientnet22_nne tf [f_efficientneev2__m. ialmmm_nnn___i in21ft1k,, tf ff_efficientteet2_5.. siiteentnet2__siilfiiii
tf efffficiennteet2
tf_ tf_inception_ye tf mf_mobileeetv3 Larrele_aa______ 100,
tf tf_mixnet_l_ tf mf_mmxee___. of df_mobilenttv3 Barttii3llaee_eeeee large_minimal _100,
tf if_mixnet_s
If mobilenetv3 lar_arrrre_00 075, If m_mobilenet33.1 2aato333_mall______ tf t__mobilenets3ssme 10mal_100,, if {__mobilenetv3_see 3at333ssmmmmmaa __smal_mmiimal
tinynet_a, tinynet_,, tinynet_c. tinynet_, tinynete, tnt tnt_s_aatch6.. 224, tv tv_densenet121, Iv_resnet101, tv Iv_resnet152, tv_resnet34, tv tv_resnet50, (v_resnext50_322 (exexxt50332xdd, twins_pepvt_base wwins_pepvt__tmm large, twins twins_pepvt pepvt pcppt_ssmalll twins jyt base, twins twins_svt svr_large, twins wwins_syt_small. vggl1, vggl1 bn, vgg13, vg bn, vggl6, vggl6 bn, vgg19, vgg19_ bn. vis- Jormer_small, vit base p 224. vit base patch16 224 miil, vit base patch16 vit Bi__baee_patch322 224, vit_basee_patie httpppaachhhapaeee vit_small_patch 224, vit_ L_ssll_pptcc|l 2338, vit mi__small_patchhs 224, vit vit_ vit_small_patch32_ vit si__iny__eee 1in__patch16 2222,, vit Liinn__ptcch66____ __844,
base_patchh16__1_.1 384,
sit_base_paae www.bbaasecch________e 224, vit ut_base_r50_s16__1
vit_small_rae @aalllrr6__s___ 222__32___________2222_2_2222_ 224, vit vit_
vit tuttttnny___s_ 1aayy__s66_p0_ _2,22,2422 vit vit_tiny_r_stee
384, wide resnet101 2, @ideeresnet50_1 _, xception,
xception41, xception65, xception71, xcit_large___ sape__24_p_ 21__21______________1__626662222 224,
lcit_large_24_100 _144_____384_____1 __s84_ditt,
xcit_ xcit_large_244 ___p____p_________ _4__,_224,
xcit_large La__l_rge____pp_ _4_p___pp___ 24__8__24____ 224 _2s__istt, xcit_ mit_mediimm_ 24_p16_224 _dist, xcit medium 24 _4_p8 _2_4222420 dist, xcit nano 12 p16 384 384 dist, xcit nano 12_p8 _84 dist, xcit xcit_small 12 12_p16_384 dist,
xcit_large___ 2aciie____________ $1$$__4__isss
xcit xcit_medium_]3 @reciumm_4_ppi_ ______1_________ 224.
xcit medium 24 [4__16______ _ist, xcit medium 24 2__p8_________________ 224, xcit nano 12 p16 224 dist, xcit nano 1amoo_12_ppp 224 dist, xcit_small 12_p16 222442224___0 dist, xcit_smai samall_12__$$ 12_p8 224 dist, _2_____24________
xcit nano 12 22__p66__ 224,
22_24. 1it_small_122_ph 224,
12_ _t______________ 224,
xcit_small 12_p8 224,
xcit_small @apaal_112_________ _________________ dist,
xcit_ssmall 24_pll_________11_____ 224,
xcit_small 24 24_p16 224 dist,
xcit xcit_ small 24 24_p166384 dist,
xcit_small 1aamaall244_$$8__ _4__2__222_ _2_4,
xcit_small ___________________ 24__8_224_____ 224__ist.
xcit_ _2_____32________11 __________dss,, ___________________ _istt, dist, xcit_tiny_2444
1at_ny__2___6_2___
xcit_tiny__ Siii_tiny_____p___ ___p1__p___66_ 22_p16_224_000000__222_22_2222_2 2___224_ds,,,
xcit_tiny_122 028_224,
224_dist, scit_tiny_11____ _______________
xcit_tiny_244
24________6______ 2,__222__s,,
xcit _____tty_____________ pppp_111______111 dist,
xcit_tiny_24 2appiy_______222__1__1 224,
xcit tint_tiny________ ______2_______________1_____ 224 dist, xcit tiny_nny______ 24 18_p___ 380_3844_000
C Correlations with ImageNet validation accuracy
We note that alignment metrics also have a (weak) U-shaped relationship with ImageNet validation accuracy (see Figure 5). We also note the following correlations between ImageNet validation accuracy and other properties which show that validation accuracy does not explain all the variance in downstream properties:
ImageNet-A: 0.81
ImageNet-S: 0.86
ImageNet-R: 0.82
CIFARI0 1-shot learning: 0.68
FMNIST 1-shot learning: 0.10
MNIST I-shot learning: 0.07
15

---

85
Insgelee vulisation acC.
80
70-
Pearson pair algrement
Figure 5: Comparing Pearson pairwise alignment and ImageNet validation accuracy.
D Outlier examples
We compared performance the original 'noisy" ImageNet and relabeled ImageNet-Real. 100 All models had better performance on Real., but the improvement has a weak inverted-U relationship with alignment. This may seem surprising, but our understanding is that the ReaL dataset was collected by relabelling images on which models (pre-2020) disagreed with the original labels. Models with low performance by today's standards (and alignment near 0.5) were used select which images to relabel, so it is unsurprising that models with alignment near 0.5 benefit most from this relabeling. This suggests that ImageNet labels may need be revisited once again as ReaL relabelling may have been biased.
E Reproducibility: Code and results data
All code and full results data are provided as part the supplemental information. will share them publicly after the anonymity period is over. All experiments were conducted AWS 1x1.16xlarge" instance (no GPUs).
16


Reviews:
[{"review_id": "C6iNqPmwJT", "reviewer": "Reviewer_CS1b", "summary": "- The authors predict that there should be an U-shaped relationship between the degree of representational alignment and performance on few-shot tasks based on information theoretic analysis, and empirically show that is the case using hundreds of vision models. They further show that the degree of alignment predicts robustness to ImageNet-A, domain-shift (e.g. ImageNet-R, -S.) ", "questions": "- Figure 4 Left: Curious to see if there is any characteristics of models that are in the top left of the graph (the ones with test accuracy around 60% and alignment being less than 20), as these points seem to be rather outliers.", "limitations": "- The analysis seems to be limited to ImageNet-based models. Recently there are more capable vision-language models such as BLIP-2, which combines a vision model with Large Language Model. While the current paper conducts detailed analysis, it would be a more interesting work if they also study these recent vision-language models.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The U-shaped results are novel and interesting phenomenon. With both information theoretical analysis and detailed empirical analysis, the results seem credible, and provide new perspective on the previously conflicting findings as to whether improving alignment increases performance.\n- They study three different alignment metrics.\n- The degree of alignment not only explains the few-shot performance, but also predicts robustness to distributional shift (e.g. ImageNet-A, -R, -S) which intuitively make sense.", "weaknesses": "- Minor:\n\t- I believe ImageNet-A is just a subset of ImageNet-test set that happens to be difficult, but the term \"adversarial robustness\" often refers to images with actual adversarial noise, so I think it should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We’re really glad that you appreciated the U-shaped results and both the theory and experiments that support them!\n\n**Comment 1: I believe ImageNet-A is just a subset of ImageNet-test set that happens to be difficult, but the term \"adversarial robustness\" often refers to images with actual adversarial noise, so I think it should be clarified.**\n\n**Response**: Thanks for pointing this out! ImageNet-A (Hendrycks et al., 2021) is actually **not** a subset of ImageNet-test but was rather new images collected from scratch from websites like iNaturalist and Flickr that match 200 of the classes in ImageNet-1k. However, unlike typical adversarial examples (which are modifications of existing images), these are so-called “natural adversarial examples” that are real images that fool the vast majority of image classifiers. We have updated the introduction, experiments, and discussion sections to clarify these two points and now instead use the term “natural adversarial examples” as suggested by the ImageNet-A authors. \n\n**Comment 2: Figure 4 Left: Curious to see if there is any characteristics of models that are in the top left of the graph (the ones with test accuracy around 60% and alignment being less than 20), as these points seem to be rather outliers.**\n\n**Response**: Great question! Those models have the following architectures: \tbeit_base_patch16_384 & beit_base_patch16_224. Both of these models also maintain their top-left positions when evaluating performance on ImageNet-A, -R, & -S and one-shot performance on MNIST and FMNIST. To check whether this is a general property of BEiT models [1], we tried out 3 additional architectures from the Pytorch Image Models zoo: beit_large_patch16_512, beit_large_patch16_384, beit_large_patch16_224 and find that they also land in the top-left corner. It appears that bi-directional pre-training of image models leads to very non-human-like but effective representations. We have updated the architecture discussion section to include this insight. Thank you for the suggestion!\n\n[1] Bao, Hangbo, Li Dong, Songhao Piao, and Furu Wei. \"Beit: Bert pre-training of image transformers.\" arXiv preprint arXiv:2106.08254 (2021).\n\n**Comment 3: The analysis seems to be limited to ImageNet-based models. Recently there are more capable vision-language models such as BLIP-2, which combines a vision model with Large Language Model. While the current paper conducts detailed analysis, it would be a more interesting work if they also study these recent vision-language models.**\n\n**Response**:\nWe are currently running additional experiments with the new models that have been added to the Pytorch Image Models (timm) package in the months that have passed since we conducted the experiments with the 491 models currently listed in the paper, including vision-language models.  Unfortunately, this is pretty time-consuming since our GPU access is limited and the additional experiments involve running 600+ additional large models. We will update the camera-ready version of the paper once the new experiments finish running. Additionally, we note that we included all our code in the supplemental information so that readers can reproduce our experiments with new or custom models. \n"}, {"review_id": "4dp2vfxvJV", "reviewer": "Reviewer_kVdE", "summary": "In this paper, the authors present a theory for a u-shaped effect in terms of model representation alignment, where models that are more and less aligned with human representations should have better performance over a number of tasks. They then perform a large number of experiments, comparing 491 models with a representation approximated based on 425,000 human judgements. They find that the quantitative results largely hold with their theory. \n\nEdit: Based on the author's rebuttal and further discussion I have raised my initial review to an accept.", "questions": "1. How do the authors account for the effect being more J-shaped than U-shaped as predicted?\n2. Why did the authors choose to only look at high-5/meanest-5/lowest-5 models?", "limitations": "The authors identify the issues around the choice of alignment, otherwise there are no major potential negative societal issues.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has a large number of strengths. It is well-written and well-presented. The theory is novel and valuable, providing insights for future researchers and developers. The experiments are exhaustive and well-motivated. The depth of discussion around justifying the choices in the experiments is excellent. The experiments also largely follow the theory, which suggests useful takeaways to a broad audience.", "weaknesses": "This is a strong paper, but there's a couple of weaknesses which dampen my enthusiasm somewhat. \n\nThe largest of these weaknesses is that while the authors repeatedly refer to a U-shaped effect, the results actually suggest a J-shaped effect. The low alignment models consistently demonstrate a significantly worse performance than the high-alignment models Despite this, there's no discussion of this point throughout the paper. The reasoning seems intuitively straightforward, despite note following from the theory. While it is true that a model could \"realize\" to interpret signals in an \"opposite\" manner (to use the paper's term) with low alignment, this realization is still extra effort. In a model trained on backprop, it is not always possible to \"flip\" the use of features in a large, multidimensional space. This or similar discussion accounting for this issue should be brought up in the paper. \n\nSecond, while a large number of experiments are run, the choice of only a top 5, bottom 5, and mean 5 is somewhat odd. This represents less than 3% of the models used in the experiments. This is understandable in some cases, like presenting individual values in tables. But in Figures like Figure 3 a more continual effect could have been demonstrated with a typical standard deviation off the lines. If the authors chose to only use 15 models, some experiment helping to demonstrate that the results weren't a fluke would be beneficial. For example, ranking the 491 models in term of alignment and visualizing some performance metric across all 491. As it is, readers need to look through a dense CSV file for this information. There's notably no readMe or similar to explain the labels for the CSV file making it difficult to understand what's going on here without digging through code. Some clarity on this would be appreciated and would benefit the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We’re grateful that you enjoyed the paper and expressed such enthusiasm for the theory and experiments!\n\n**Comment 1: The largest of these weaknesses is that while the authors repeatedly refer to a U-shaped effect, the results actually suggest a J-shaped effect. The low alignment models consistently demonstrate a significantly worse performance than the high-alignment models Despite this, there's no discussion of this point throughout the paper. The reasoning seems intuitively straightforward, despite note following from the theory. While it is true that a model could \"realize\" to interpret signals in an \"opposite\" manner (to use the paper's term) with low alignment, this realization is still extra effort. In a model trained on backprop, it is not always possible to \"flip\" the use of features in a large, multidimensional space. This or similar discussion accounting for this issue should be brought up in the paper. (Question 1: How do the authors account for the effect being more J-shaped than U-shaped as predicted?)**\n\n**Response**: This is a great point and the asymmetry of the empirical U is something we’ve been thinking about a lot since writing the paper! As you can see in Figure 4 (left), the left arm of the U does indeed extend as high as the right arm with a few models having alignment at around 20 but few-shot learning performance at around 60. Those models have the following architectures: \tbeit_base_patch16_384 & beit_base_patch16_224. Both of these models also maintain their top-left positions when evaluating alignment against performance on ImageNet-A, -R, & -S and one-shot performance on MNIST and FMNIST. To check whether this is a general property of BEiT models [1], we tried out 3 additional architectures from the Pytorch Image Models zoo: beit_large_patch16_512, beit_large_patch16_384, beit_large_patch16_224 and find that they also land in the top-left corner. It appears that bi-directional pre-training of image models leads to very non-human-like but effective representations. \n\nHowever, the left arm of the U is indeed still much sparser than the right arm. This suggests that, while it is clearly possible for models to have very low alignment, and some indeed do, generic computer vision training tends to produce more aligned models. This supports the findings from Moschella et al. (2022), who find that a very large swath of computer-vision models trained on the same data have similarly structured relative representations. We have added this discussion and additional results to the Results and Discussion sections. \n\n[1] Bao, Hangbo, Li Dong, Songhao Piao, and Furu Wei. \"Beit: Bert pre-training of image transformers.\" arXiv preprint arXiv:2106.08254 (2021).\n\n[2] Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication.\" The Eleventh International Conference on Learning Representations. 2022.\n\n**Comment 2: Second, while a large number of experiments are run, the choice of only a top 5, bottom 5, and mean 5 is somewhat odd. This represents less than 3% of the models used in the experiments. This is understandable in some cases, like presenting individual values in tables. But in Figures like Figure 3 a more continual effect could have been demonstrated with a typical standard deviation off the lines. If the authors chose to only use 15 models, some experiment helping to demonstrate that the results weren't a fluke would be beneficial. For example, ranking the 491 models in term of alignment and visualizing some performance metric across all 491. As it is, readers need to look through a dense CSV file for this information. There's notably no readMe or similar to explain the labels for the CSV file making it difficult to understand what's going on here without digging through code. Some clarity on this would be appreciated and would benefit the paper. (Question 2: Why did the authors choose to only look at high-5/meanest-5/lowest-5 models?)**\n\n**Response**: Thank you for bringing this up! We initially used the 5-5-5 format for Figure 3 (left) and Table 1 because we thought it would be a simpler, more intuitive format for readers to visualize and interpret the data. However, the key results and claims are based on the analysis of the **full set of models** as can be seen in **Figure 3 (right), Figure 4, and Table 2** of the paper. For example, Figure 4 (left) directly compares one-shot learning performance with alignment as you suggest. We have now updated the captions and in-text descriptions to clarify that the results are based on all models rather than the subset of 15 and have added additional figures like Figure 4 that visualize other downstream properties for all 491 models to the appendix. \n\nAdditionally, we apologize for the dense supplemental information. We have prepped a README file that describes the columns of full_results.csv and have added additional comments in the associated iPython notebook that explains how to use the notebook to analyze the results file and reproduce all the figures in the paper as well as additional figures that compare various performance metrics against our three alignment metrics. \n"}, {"review_id": "hjonC9cHoa", "reviewer": "Reviewer_wjPt", "summary": "The work provides theoretical and empirical analysis for the relationship between human alignment, few-shot performance, and robustness to distribution shift. The theoretical analysis predicts models that have high and low alignment with human annotations will have better performance on few-shot and distribution shift datasets. The experiments verify the predicted relationship between representational alignment and downstream performance. ", "questions": "- It would be helpful to see the distribution of alignment metrics and some analysis on what factors other than validation accuracy affect the alignment metric. Given that validation accuracy is almost entirely predictive of robustness and downstream performance (greater than .9 R-squared), it seems that validation accuracy and alignment would be highly correlated. \n\n- In the abstract it’s stated that highly-aligned models are more robust to adversarial attacks. This claim seems overly strong as the experiments are conducted with natural adversarial images, and not with white or black box attacks. \n\n- Given theorem 3.9, would the relationship between alignment and few-shot performance be v-shaped and not u-shaped? \n", "limitations": "The authors adequately address the limitations of the method. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Alignment between humans and ML models has been studied considerably from the perspective of interpretability and safety. Understanding how and whether alignment can improve sample-efficiency and robustness is a less explored, but important research question which this work makes progress towards. \n\n- The paper theoretically and empirically shows that alignment is not monotonic with respect to robustness/sample efficiency which may explain why previous works have found conflicting results regarding the benefits of alignments. I think this finding would be useful to the community. Also the information-theoretic framework proposed by the authors for analyzing representational alignment would likely be useful to future work. \n\n- The experiments are thorough and adequately support the claims. \n\n- The paper is well organized and the tables and plots clearly illustrate the main empirical findings. ", "weaknesses": "Other works [1][2] have shown that the validation accuracy on an in-distribution (ID) dataset is highly correlated with accuracy on out-of-distribution (OOD) and downstream datasets. For reference, the R-squared value for pretrained models evaluated on ImageNet and ImageNetV2 is .996 and other distribution shifts have similar correlation. Given that in-distribution validation accuracy is almost entirely predictive of out-of-distribution accuracy, I think the current motivation, which is to show that alignment is predictive of downstream performance, is weak. Representational alignment as a predictive metric for robustness and downstream transfer performance is less accurate than standard ID validation accuracy. Also obtaining validation accuracy is easier since it only requires one-hot labeling, as opposed to similarity matrices. \n\n\nIf the main motivation of the paper is to show that improving alignment will confer benefits such as robustness and few-shot performance, then it should show that alignment measures something more than in-distribution validation accuracy. Otherwise, robustness and few-shot performance could simply be increased by improving the model on the validation set. \n\nOverall I think the paper is interesting and original. My main concern is for the motivation, but would be happy to change my rating if this concern is adequately addressed. \n\n\n[1] Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization\n\n[2] Do ImageNet Classifiers Generalize to ImageNet?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We’re very glad to hear that the key result and information-theoretic framework will be useful for the community!\n\n**Comment 1: Other works [1][2] have shown that the validation accuracy on an in-distribution (ID) dataset is highly correlated with accuracy on out-of-distribution (OOD) and downstream datasets. For reference, the R-squared value for pretrained models evaluated on ImageNet and ImageNetV2 is .996 and other distribution shifts have similar correlation. Given that in-distribution validation accuracy is almost entirely predictive of out-of-distribution accuracy, I think the current motivation, which is to show that alignment is predictive of downstream performance, is weak. Representational alignment as a predictive metric for robustness and downstream transfer performance is less accurate than standard ID validation accuracy. Also obtaining validation accuracy is easier since it only requires one-hot labeling, as opposed to similarity matrices.\nIf the main motivation of the paper is to show that improving alignment will confer benefits such as robustness and few-shot performance, then it should show that alignment measures something more than in-distribution validation accuracy. Otherwise, robustness and few-shot performance could simply be increased by improving the model on the validation set.\nOverall I think the paper is interesting and original. My main concern is for the motivation, but would be happy to change my rating if this concern is adequately addressed.**\n\n**Response**: We are very glad you brought up this concern as it’s very important to us that we correctly communicate the motivation for this study. In representation learning research, two of the major goals are 1) to figure out what makes a representation “good” and 2) to figure out how to learn these “good” representations from data. However, as we mention in the paper, there is much disagreement in the literature, and even diverging experimental results, about whether human-like representations are actually “good”.  Meanwhile, for alignment researchers, it is important to know what the downstream effects are of aligning models with humans and whether this affects model behavior in positive (or negative) ways. Finally, cognitive scientists and neuroscientists often study representational similarity between humans and other humans as well as between humans and models to try to understand human cognition and communication. We believe our results and framework are important for all three of these communities. We definitely do not see our main motivation as simply being a way to improve or predict few-shot learning performance, for which we agree there are other, better metrics. Rather, our goals are  1) establishing a clear link between alignment and downstream properties, 2) explaining/resolving previous disagreement about the value of human-like representations and 3) providing these three communities with a framework they can all use in future studies. This is also why we submitted the paper with cognitive science as the primary area rather than supervised learning or computer vision. We have now updated the introduction and the discussion to clarify these key motivations.\n\n**Comment 2: It would be helpful to see the distribution of alignment metrics and some analysis on what factors other than validation accuracy affect the alignment metric. Given that validation accuracy is almost entirely predictive of robustness and downstream performance (greater than .9 R-squared), it seems that validation accuracy and alignment would be highly correlated.**\n\n**Response**: We include a scatterplot in the rebuttal pdf of Pearson alignment plotted against ImageNet validation accuracy and find a slight U-shaped curve and a weak positive correlation ($\\rho=0.235$). To clarify, we believe that the causal link goes the other way: alignment affects validation accuracy, robustness, and few-shot learning performance. \n\nWe also note the following correlations between ImageNet validation accuracy and other properties which show that validation accuracy does not explain all the variance in downstream properties:\n- ImageNet-A: 0.81\n- ImageNet-S: 0.86\n- ImageNet-R: 0.82\n- CIFAR10 1-shot learning: 0.68\n- FMNIST 1-shot learning: 0.10\n- MNIST 1-shot learning: 0.07\n \nWe have now included this additional figure and insights in the Results section. \n\n**Comment 3: In the abstract it’s stated that highly-aligned models are more robust to adversarial attacks. This claim seems overly strong as the experiments are conducted with natural adversarial images, and not with white or black box attacks.**\n\n**Response**: We have updated the abstract and main text to instead refer to robustness to natural adversarial examples. \n\n**Comment 4: Given theorem 3.9, would the relationship between alignment and few-shot performance be v-shaped and not u-shaped?**\n\n**Response**: Thank you for bringing this up! For binary symmetric channels, the channel capacity is equal to $1-H(p)$ where $H(p):=-p\\log_2(p)-(1-p)\\log_2(1-p)$ is the binary entropy function and $p$ is the probability of a bit flip. As a result, capacity (and hence the relationship between alignment and few-shot performance) is U-shaped. We have added a short clarification about this to the theory section.\n"}, {"review_id": "499BcSOfb2", "reviewer": "Reviewer_225s", "summary": "This paper explores the relationship between the alignment of deep learning based representations with humans and the downstream performance of the NN models. Using triplets as a non-metric measure of similarity, the authors use information theory to show that alignment and downstream performance would have a U-shaped relationship, where low and high alignment would give a good downstream performance. ", "questions": "* Why does correlation with test accuracy reduce with more training examples (Fig. 3 and 4 right)?\n* How are the classifier heads parameters chosen?\n* What is meant by supervised learning in Section 3? Does it mean standard classification task? It is not clear.\n* What are quadruplets mentioned in Discussion?\n* Large datasets like ImageNet and CIFAR have a lot of \"noisy\" and misclassified examples on which the deep models are trained on. Do the different alignment metrics reveal any interesting facts on the model's alignment or non-alignment on these \"outlier\" examples?\n", "limitations": "* Most transfer learning results are shown for transfer from ImageNet to CIFAR, which is limiting.\n* Exact contribution of triplets is not clear from the paper, as existing correlation metrics can already show the U-shaped relation and performance relations.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The authors use existing ideas of triplets from cognitive science and combine information theory to show why human alignment may not always lead to better performance, especially for few-shot downstream learning. More importantly, the authors give an intuition and theoretical backing for why some models can have low alignment with humans and still perform well in downstream learning.\n* The authors highlight the significance of ensuring human alignment of neural network representations by studying the role of alignment on two critical characteristics: adversarial robustness and domain shift robustness. They further show how human alignment plays a role in downstream learning, mainly when limited data is available.\n* Using a significantly large number of models and large-scale existing human-annotated data, the authors empirically validated the theoretical claims made in the paper, showing the benefit of very high alignment with robustness while also revealing the U-shaped relationship between few-shot learning and alignment. The authors performed critical statistical testing to highlight the results.\n* The interaction of architecture model choices and alignment is insightful and could be helpful for future work.", "weaknesses": "* The contribution seems limited, the main contribution seems to be the showing of the U-shaped relationship. The primary notion of triplets is derived from prior work.\n* The importance of triplets seems limited. Pearson and Spearman also clearly provide similar takeaways. Why are triplets needed? Not clear from the evaluation section.\n* Evaluation is also limited in some aspects, although the paper explores more than 400 models. Most transfer learning results are shown for CIFAR after ImageNet pre-training. The result on MNIST and FMNIST show quite weak results, meaning that strong domain shift during transfer learning makes the metric less impactful. But transfer learning is usually most important in these settings (e.g., medical domain, face classification, etc.).\n* There is a lack of clarity in the writing of the paper. Section 3 is difficult to follow, and it is not always clear how the different definitions flow, and how the pipeline works. E.g., not clear what is the significance of Def. 3.4, Discussion section mentions \"quadruplets\", but they are never mentioned in the paper. Not clear at all.\n* Adversarial robustness is measured for real-world adversarial images. More common adversarial examples used in literature, e.g., $\\ell_p$ ball perturbations are not explored.\n* Evaluation section compares the triplets with pairwise measures like Pearson and Spearman correlations. But it is not clear from Section 3 as to why it is interesting to compare triplets to pairwise. Do triplets give benefits over Pearson/Spearman? Are there any tradeoffs?\n* Results shown in the tables are not well discussed\n* Transfer learning results only analyzed when we have \"head\" networks. Not clear if the method works for full network fine-tuning or not, something that is often used in practice.\n* No justification given on the choice of the heads, e.g., dropout rate, etc. Were they hyper parameter-tuned?\n* Not clear what and how the ImageNet accuracy correction is performed. It seems important, so more intuition should be provided.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are really glad you appreciated the scale of our experiments and the architectural insights. We are especially grateful for your question about “outlier” examples as this led to an exciting new insight detailed below. \n\n**1: Contribution seems limited to U-shaped relationship.**\n\nThank you for bringing up this concern! Please see the 2nd paragraph in the general response to all reviewers for clarification on the motivation/contribution.\n\n\n**2: Why are triplets needed?**\n\nAs mentioned, the theoretical aspects of this work are a key contribution. The information-theoretic argument we make in the main theorem leverages the fact that we can convert triplets into bits which is necessary to prove the link between representational alignment and downstream properties. Knowing this link exists, we can then study several ways to measure the representational alignment between two agents in practice. For example, Pearson & Spearman similarity analysis is close to what neuroscientists use to compare brain data with DNN activations (one of the reasons why we submitted with ‘neuroscience and cognitive science’ as the primary area). As some reviewers mentioned, our findings being supported experimentally by both triplet analysis and more common methods provides strong confirmation for the theory. We have updated the Introduction to reflect this. \n\n**3: Most transfer results shown for CIFAR. MNIST and FMNIST show weak results, meaning strong domain shift makes the metric less impactful.**\n\nThat strong domain-shift weakens the relationship between alignment and transfer learning is an excellent point and we see it as a key insight of the experimental results. A key question in representational learning is whether human-like representations are ‘good’, with conflicting findings in the literature on whether they lead to better performance. Our discovery of the U-shaped curve relieves this tension, but our experiments furthermore show that representational alignment is also susceptible to domain-shift (e.g., just because we are very aligned on animals we should not assume we are equally aligned on fruits) and thus if we want our models to learn human-like representations, we should ensure they are human-like globally. This result is similarly of interest to the AI alignment community as it suggests we should study whether other kinds of alignment are also susceptible to domain-shift. We have updated the Discussion to emphasize this insight from the (F)MNIST experiments. \n\n**4: Lack of clarity: Def. 3.4,  \"quadruplets\", “supervised learning”**\n\nThank you for pointing this out! By supervised learning, we do mean classification. This is also the source of confusion in Def 3.4 since that defines the task setup, and we have updated it accordingly. We also added a definition of quadruplets to this section, by which we meant queries of the form “X is closer to Y than A is to B” ($|X-Y|<|A-B|$). With this definition in place, it is also much clearer why triplets form a subset of quadruplets. \n\n**5: Adversarial robustness is measured for real-world adversarial images.**\n\nWe have updated the text to refer to robustness to “natural adversarial examples” as also suggested by some of the other reviewers. \n\n**6: Results in tables are not well discussed.**\n\nWe have added additional discussion of both tables, including a clarification suggested by reviewer kVdE. \n\n**7: Transfer learning results only with head networks. How are the classifier head parameters chosen?**\n\nFine-tuning all models would be prohibitively expensive. Recent work [1] has shown that linear probing usually performs on par with fine-tuning and is a more accurate measure of the quality of internal representations which fine-tuning distorts. The heads are provided in the supplement (FSL_exps.py) and fixed hyperparameter values (selected based on exploratory experiments to confirm stable convergence) were used for every dataset/model for consistency. For example, as mentioned in section 4 paragraph “Few-shot learning”, all NN heads had dropout set to 0.8. We have updated the paragraph accordingly. \n\n[1] Kumar, Ananya, et al. \"Fine-tuning can distort pretrained features and underperform out-of-distribution\" arXiv:2202.10054 (2022)\n\n**8: Not clear what and how the ImageNet accuracy correction is performed.**\n\nAs described in section 4 paragraph “Correcting correlation for ImageNet performance”, we computed partial correlation with ImageNet accuracy as a covariate. Partial correlation is a popular statistical technique for measuring how associated two random variables are after removing the effect of confounding covariates. We have updated the paragraph.\n\n**9: Why does correlation reduce with more training examples (Fig. 3 and 4 right)?**\n\nInformation theory predicts exactly this kind of effect since passing additional redundant bits over a noisy channel (i.e. more examples per class) increases the probability that all information is successfully recovered. We have added this insight to the Discussion. \n\n**10: What do alignment metrics reveal on \"outlier” examples?**\n\nBased on your suggestion, we compared performance on the original ‘noisy’ ImageNet and relabeled ImageNet-ReaL [2]. All models had better performance on ReaL, but the improvement has a weak inverted-U relationship with alignment. This may seem surprising, but the ReaL dataset was collected by relabelling images on which models (pre-2020) disagreed with the original labels. Models with low performance by today’s standards (and alignment near 0.5) were used to select which images to relabel, so it is unsurprising that models with alignment near 0.5 benefit most from this relabeling. This is exciting as it suggests that ImageNet labels need to be revisited once again as ReaL may be biased! We have added this to the Results and Discussion. Thank you for the suggestion!\n\n[2] Beyer, Lucas, et al. \"Are we done with ImageNet?\" arXiv:2006.07159 (2020)\n"}]

Author rebuttal:
**We thank all the reviewers for their valuable feedback! We are encouraged that the reviewers found the paper to be “well-written”, “well-presented”, and “well-organized”; the theory to be “novel”, “interesting”, “valuable”, and “useful” for the community; and the experiments to be “detailed”, “exhaustive”, “well-motivated”, “thorough”, and “insightful”. The constructive feedback from the reviewers generally consisted of 1) clarity suggestions; and 2) insight/analysis suggestions. We have implemented all suggestions provided by the reviewers and responded with details of those changes to each reviewer individually. We are grateful for the suggestions as not only did they help us improve clarity but they also 1) helped us distill the motivation into a much stronger, more concise statement; 2) made our theory section smoother and more accessible to a broader audience; and 3) resulted in exciting new insights from the experimental results.**

Since two reviewers asked about the motivation, we provide a shared response here. In representation learning research, two of the major goals are 1) to figure out what makes a representation “good” and 2) to figure out how to learn these “good” representations from data. However, as we mention in the paper, there is much disagreement in the literature, and even diverging experimental results, about whether human-like representations are actually “good”.  Meanwhile, for alignment researchers, it is important to know what the downstream effects are of aligning models with humans and whether this affects behavior in positive (or negative) ways. Finally, cognitive scientists and neuroscientists often study representational similarity between humans & other humans as well as between humans & models to try to understand human cognition and communication. We believe our results and framework are important for all three of these communities. We definitely do not see our main contribution as simply being able to predict few-shot learning performance based on analysis of triplets, but rather as 1) establishing a clear link between alignment and downstream properties, 2) explaining/resolving previous disagreement about the value of human-like representations in the literature, and 3) providing these three communities with a novel information-theoretic framework they can all use in future studies. This is also why we submitted the paper with cognitive science as the primary area rather than supervised learning or computer vision. We have now updated the introduction and the discussion to clarify these key motivations and contributions.

Finally, we note that the figure in the attached PDF is intended for **Reviewer wjPt**.

**Please let us know if you have any more questions or suggestions!**


Meta-review: This work provides significant empirical and theoretical contributions which help support the intuition that representations more aligned with humans’ lead to better (e.g. more robust) behavior.  This is an important finding for science and AI (preference) alignment, as this intuition has often been assumed without support.  The experiments demonstrate that this is in fact the case in computer vision in the context of robustness to (natural) adversarial robustness and domain shift (for few shot learning, the result is more subtle; moderately aligned models are the worst, in line with the authors’ theoretical results).  Reviewers found their concerns were adequately addressed in rebuttals and all favor acceptance and I am happy to concur.
Decision: Accept (spotlight)