Title: Alignment with human representations supports robust few-shot learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation learning

supervised learning

human alignment

few-shot learning
Abstract: Should we care whether AI systems have representations of the world that are similar to those of humans? We provide an information-theoretic analysis that suggests that there should be a U-shaped relationship between the degree of representational alignment with humans and performance on few-shot learning tasks. We confirm this prediction empirically, finding such a relationship in an analysis of the performance of 491 computer vision models. We also show that highly-aligned models are more robust to both natural adversarial attacks and domain shifts. Our results suggest that human-alignment is often a sufficient, but not necessary, condition for models to make effective use of limited data, be robust, and generalize well.

Alignment with human representations supports robust few-shot learning

Abstract

Should we care whether AI systems have representations of the world that are similar to those of humans? We provide an information-theoretic analysis that suggests that there should be a U-shaped relationship between the degree of representational alignment with humans and performance on few-shot learning tasks. We confirm this prediction empirically, finding such a relationship in an analysis of the performance of 491 computer vision models. We also show that highly-aligned models are more robust to both natural adversarial attacks and domain shifts. Our results suggest that human alignment is often a sufficient, but not necessary, condition for models to make effective use of limited data, be robust, and generalize well.

1 Introduction

As AI systems are increasingly deployed in settings that involve interactions with humans, exploring the extent to which these systems are aligned with humans becomes more significant. While this exploration has largely focused on the alignment of the values of AI systems with humans [7, 16], the alignment of their representations is also important. Representing the world in the same way is a precursor to being able to express common values and to comprehensible generalization. To the extent that humans have accurate representations of the world, representational alignment is also an effective source of inductive bias that might make it possible to learn from limited data.

As a motivating example, imagine a meeting between a 16th century alchemist and a 21st century chemist. They live in the same physical world and are intimately familiar with the materials that comprise it, but they would have significant difficulty expressing their values and generalizing the results of an experiment they observe together. The alchemist would likely learn poorly from examples of a reaction demonstrated by the chemist, not having the right inductive biases for the way the world actually works. The alchemist and the chemist lack representational alignment - they represent the world in fundamentally different ways - and this impedes generalization and learning.

In this paper, we provide a theoretical and empirical investigation of the consequences of representational alignment with humans for AI systems, using popular computer vision tasks as a way to study this phenomenon. We define representational alignment as the degree to which the latent representations of a model match the latent representations of humans for the same set of stimuli, and refer to models that are representationally aligned with humans as being 'human-aligned.' Several recent papers have proposed ways to measure [23, 24], explain [26, 20], and even improve [29, 6] the representational alignment of models. However, many models that score low on these alignment metrics still have high performance on downstream tasks like image classification [20, 26, 6].

So, are there any real advantages (or disadvantages) to using human-aligned models? To answer this question, we develop an information-theoretic framework for analyzing representational alignment. This framework enables us to make predictions about emergent behavior in human-aligned models.

A-SHAREDDATA

B-REPRESENTATIONS

C-PAIRWISESIMILARITY

Chimp

Wolf--.Eagle

Lemur

Chimp

1

0.8

0.4

0

Chimp

Eagle

+

0.8

0.2

0.1

Lemur

Eagle

0.4

0.2

1

0.6

Lemur

0.1

0.6

1

Alignment

Chimp

Eagle

Lemur

Eagle

Chimp

1

0.8

0.4

0.1

Lemur

0.8

1

0.5

0

A

Chimp

Eagle

0.4

0.5

1

0.6

Lemur

0.1

0

0.6

1

A-NEWDATA

B-TEACHER

C-STUDENT

Chimp

Eagle

Gotit!

Lemur

Eagle

Lemur

Chimp

Thenewobject iscloser

toWolf thantoLemur!

In particular, our framework predicts that few-shot transfer learning performance should have a U-shaped relationship with alignment. To verify the predictions of our framework and to probe for additional properties that arise as a consequence of alignment, we conduct a series of experiments in which we assess the downstream properties of human-aligned models compared to their non-aligned counterparts. From our experiments comparing 491 large computer-vision models to over 425,000 human judgments (across 1200 participants), we identify three properties:

Models with either high or low alignment with humans are better at few-shot learning (FSL) than models with medium alignment, even correcting for pre-training performance.

Human-aligned models are more robust to natural adversarial examples, even when correcting for classification performance on the pre-training dataset.

Human-aligned models are more robust to domain shift, even when correcting for classification performance on the pre-training dataset.

The U-shaped relationship between alignment and few-shot learning helps to explain why previous results have not consistently observed benefits of representational alignment. Overall, our results suggest that representational alignment can provide real and significant benefits for downstream tasks, but that it may be a sufficient rather than necessary condition for these benefits to emerge.

2 Related Work

With AI systems entering mainstream usage, aligning these models with human values and understanding is increasingly important [7, 16]. This concept, referred to as AI alignment, is an important but still largely open problem [43], in part due to the difficulty of formalizing it [34], or even reaching consensus on a definition [16]. In this paper, we focus on formalizing a specific aspect of AI alignment: representational alignment.

Representational alignment is a measure of agreement between the representations of two learning agents (one of whom is typically a human). There are numerous names, definitions, measures, and uses of this form of alignment across various fields, including cognitive science, neuroscience, and machine learning. Some of the other names include latent space alignment [38], concept(ual) alignment [35, 26], system alignment [8, 30, 1], representational similarity analysis (RSA) [17], and model alignment [24]. Shepard [32] proposed that human representations can be recovered by using behavioral data to measure the similarity of a set of stimuli and then finding embeddings that satisfy those similarity associations using methods like multi-dimensional scaling (MDS). Similarly, in neuroscience, Representational Similarity Analysis (RSA) is a popular technique for relating neural, behavioral, and computational representations of the same set of stimuli via similarity analysis [17]. While similarity analysis has clearly proven itself to be a powerful method, exhaustively collecting pairwise similarity judgments is expensive ( O ( N 2 ) judgments for N stimuli) and there have been numerous proposals aiming to develop more efficient methods of recovering human representations.

Jamieson and Nowak [15] proposed an active learning scheme for querying for human judgments using triplets of the form 'is a closer to b than to c ?' and derived bounds on the number of required queries for lossless completion of the full similarity matrix using such queries. When an approximate solution is acceptable, Peterson et al. [29] showed that pre-trained computer vision models can be used to approximate human perceptual similarity judgments over images. Marjieh et al. [23] showed that human perceptual similarity can be more accurately, but still efficiently, approximated from natural language descriptions of the stimuli of interest (for example by using large language models to estimate similarity over pairs of these descriptions). Marjieh et al. [24] extended this result to more domains (vision, audio, and video) and measured alignment for hundreds of pre-trained models.

Several recent studies have also attempted to identify what design choices lead to improved representational alignment in models [20, 26, 6], although Moschella et al. [25] found that even with variation in design choices, many models trained on the same dataset end up learning similar 'relative representations' (embeddings projected into a relational form like a similarity matrix), or in other words, converge to the same representational space. Tucker et al. [38] showed that representational alignment emerges not only in static settings like image classification, but also dynamic reinforcement learning tasks involving human-robot interaction. Several studies have also focused on studying alignment specifically in humans, both between different people and for a single person but across multiple tasks and domains [8, 30].

Although several recent papers have proposed ways to measure [23, 24], explain [26, 20], and even improve [29, 6] the representational alignment of models, few have focused on studying the downstream impact of a model being representationally aligned with humans, and many studies simply rely on the intuition that better alignment leads to better performance to justify pursuing increased alignment. While there is recent evidence to suggest that alignment may help humans learn across domains and perform zero-shot generalization [1], there is also evidence to suggest that alignment may not always be beneficial for models, with models scoring low on alignment metrics achieving higher performance on downstream tasks like image classification [20, 26, 6]. Our goal in this paper is to conduct an in-depth study into the downstream effects of representational alignment. Our theoretical and empirical results both validate and explain the apparently conflicting results seen in previous literature as special cases of the (more complex than previously suspected) effects of representational alignment.

3 Theory

If we want to measure representational alignment across different architectures with potentially mismatched embedding coordinate spaces and dimensionalities, then we need a definition of representation spaces that enables comparisons of such spaces. Inspired by the cognitive science literature on

using non-metric similarity triplets to recover hidden psychological representations, and building on rigorous computational theory that analyzes such triplets [15], we propose a triplet-based definition of representation learning. We summarize representational alignment under our framework in the schematic shown in Figure 1.

Definition 3.1. As proposed by Jamieson and Nowak [15], for an ordered set of n objects in d dimensions represented as a vector x ∈ ❘ nd , a similarity triplet corresponding to the question 'is x i closer to x j than to x k ?' is a membership query of the form ✶ x ∈ r ijk where r ijk = { x ∈ ❘ nd : | x i -x j | < | x i -x k |} .

̸

Definition 3.2. For an ordered set of objects x ∈ ❘ nd and model M with embeddings f M : ❘ d → ❘ d M , the triplet-based representation space of M is the set of all triplets S M ( x ) = { ( i, j, k ) ∈ { 1 , ..., n } 3 : ✶ x ∈ r M ijk , i = j, j = k, k = i } where r M ijk = { x ∈ ❘ nd : | f M ( x i ) -f M ( x j ) | < | f M ( x i ) -f M ( x k ) |} .

̸

̸

Remark 3.3 . Since switching the order of j and k in a triplet deterministically flips the query between 0 and 1, it is easy to see that, for a set of n objects, a representation space is determined by a set of just n ( n -1)( n -2) 2 unique triplets. Thus, for a set of n objects, S M ( x ) can be represented as a binary vector of length n ( n -1)( n -2) 2 and we use this interpretation in the remainder of this section.

Definition 3.4. Consider a training set x ∈ ❘ nd and a corresponding target representation space S T ( x ) . We define representation learning with a parametrized model M θ as optimizing the objective min θ ℓ ( S T ( x ) , S M θ ( x )) where ℓ is a loss function penalizing divergence between the target and learned representation spaces.

Sucholutsky et al. [36] showed that many ML objectives and associated supervision signals, ranging from similarity judgments for contrastive learning to labels for classification, can be converted into similarity triplets making them compatible with this definition of representation learning. We also immediately get an intuitive definition of alignment between two agents (e.g. a model and a person) as the rate of agreement on matched triplets (also known as simple match coefficient).

Definition 3.5. Consider a dataset x ∈ ❘ nd and two agents, A and B , with corresponding representation spaces S A ( x ) and S B ( x ) . We define representational misalignment between A and B as D R ( A,B ; x ) = || S A ( x ) -S B ( x ) || 1 n ( n -1)( n -2) / 2 . Representational alignment is then simply 1 -D R ( A,B ; x ) .

Definition 3.6. Consider a single set of three points t = ( x i , x j , x k ) ∈ X ⊆ ❘ 3 d sampled uniformly at random, and two agents, A and B . We define stochastic representational misalignment between A and B as D P ( A,B ; X ) = P ( ✶ x ∈ r A ijk = ✶ x ∈ r B ijk ) . Stochastic representational alignment is then simply 1 -D P ( A,B ; X ) .

Since each triplet corresponds to one bit of information [15], and we have a probabilistic definition of alignment, we can now use information theory to define what it means for one agent to supervise another (potentially misaligned) agent. We visualize supervised learning under our framework in Figure 2. For perfectly aligned models, Jamieson and Nowak [15] derive the following result.

Lemma 3.7. Consider input space X ⊆ ❘ nd , shared data x ∼ X , new object c ∈ R d , and models A and B with D P ( A,B ; X ) = 0 . Ω( d log( n )) triplet queries are required for B to identify the location of c relative to all objects in x .

Proposition 3.8. For input space X ⊆ ❘ nd , shared data x ∼ X , new object c ∈ R d , and models A,B with D P ( A,B ; X ) = ϵ , Ω( d log( n ) 1+ ϵ log( ϵ )+(1 -ϵ ) log(1 -ϵ ) ) triplets are required for B to learn c .

Proof. Consider a communication game involving Alice and Bob, a dataset x that they both have their own representations for (we call this 'shared data'), and a new object c that only Alice can see. Alice can only communicate with Bob by giving binary responses to triplet queries based on Alice's own representations of the objects. The goal is for Bob to learn the location of c with respect to the other objects in x . D P ( A,B ; X ) = ϵ is the probability for any triplet drawn from x ∪ c to be flipped in Bob's representation space relative to Alice's. This is equivalent to Alice and Bob communicating over a binary symmetric channel where the probability of a bit flip is ϵ . The capacity of this channel is C ϵ = 1 -H ( ϵ, 1 -ϵ ) = 1 + ϵ log( ϵ ) + (1 -ϵ ) log(1 -ϵ ) . For error-free communication over this channel, the highest achievable rate is R < C ϵ , meaning at least 1 C ϵ bits are required to communicate each useful bit over this channel. By Lemma 3.7, Ω( d log( n )) useful bits are required for Bob to learn the location of c over a channel with no error, and thus a total of Ω( d log( n ) C ϵ ) bits are required.

Theorem 3.9 ( Few-shot learning ) . Consider input space X ⊆ ❘ nd , shared data x ∼ X , new objects c ∈ R md and three models A , B 1 , and B 2 with D P ( A,B 1 ; X ) = ϵ B 1 and D P ( A,B 2 ; X ) = ϵ B 2 . If | 0 . 5 -ϵ B 1 | < | 0 . 5 -ϵ B 2 | , then B 2 requires fewer queries to learn c than B 1 does.

Proof. From Proposition 3.8, it immediately follows that B 1 requires Ω( md log( n ) C ϵ B 1 ) queries and B 2 requires Ω( md log( n ) C ϵ B 2 ) queries. C ϵ : [0 , 1] → [0 , 1] is a symmetric convex function with a minimum of 0 at ϵ = 0 . 5 and maxima of 1 at ϵ = 0 , 1 . Thus, if f | 0 . 5 -ϵ B 1 | < | 0 . 5 -ϵ B 2 | , then Ω( md log( n ) C ϵ B 1 ) > Ω( md log( n ) C ϵ B 2 ) .

Sucholutsky et al. [36] showed that commonly used supervision signals, like hard and soft classification labels, can be converted into sets of triplets. Reducing the number of required triplets to learn the location of a new object is equivalent to reducing the number of required labels. It follows from Theorem 3.9 that high or low alignment leads to high few-shot learning performance on downstream tasks like classification (where learning a new class can be formulated as learning the location of the centroid or boundaries of that class using a small number of labels), while medium alignment leads to low few-shot learning performance . Intuitively, this comes as a result of mutual information between the teacher and student representations being minimized when alignment is at 0 . 5 . In other words, if Bob knows that most bits coming from Alice are flipped (i.e. D P ( A,B ) = ϵ > 0 . 5 ), then going forward, Bob can flip all the bits coming from Alice to achieve a lower error rate (i.e. D P ( ¯ A,B ) = 1 -ϵ < 0 . 5 ).

We note that generalizing under domain-shift can be considered a form of zero-shot transfer learning and selecting adversarial examples can be seen as selecting objects that maximize representational disagreement between a model and a human. Under these interpretations, our framework also predicts that highly-aligned models are robust to both domain shift and adversarial examples though the framing in these cases is less intuitive than for the few-shot learning case. We share some preliminary theoretical analyses of robustness properties in the Supplement.

4 Experiments

Our theoretical analysis shows how representational alignment can serve as a source of inductive bias that reduces the number of bits that need to be acquired from data. In particular, Theorem 3.9 predicts an unexpected U-shaped relationship between alignment and few-shot learning performance. We now present a series of experiments designed to test this prediction and examine whether it extends to robustness to natural adversarial examples and domain shift.

Models. The pre-trained models evaluated in this paper are taken from the PyTorch Image Models package [41]. The full list of 491 models used in our experiments can be found in the Supplement.

Data. All of the models used in this paper were pre-trained on ImageNet-1k [31] and had their performance evaluated on the ImageNet-1k validation set. Adversarial robustness was measured on the ImageNet-A dataset of natural adversarial examples that have been found to foil most ImageNet models [11]. Zero-shot domain shift robustness was measured on the ImageNet-R dataset of renditions (e.g., paintings, toys, statues, etc.) of ImageNet classes [10] and the ImageNet-Sketch dataset of black-and-white sketches of the ImageNet classes [40]. All ImageNet and ImageNet variant results come from the PyTorch Image Models package [41]. Few-shot transfer learning performance was evaluated on the CIFAR100 [18] test set with n ∈ { 1 , 5 , 10 , 20 , 40 , 80 } examples per class used for few-shot learning and the remaining examples used for evaluation. We measure alignment by computing the three metrics described below on the six image datasets from Peterson et al. [29] and their respective sets of similarity judgments consisting of over 425,000 judgments across 1200 participants, as described by Marjieh et al. [23]. We average each alignment metric over the six datasets.

Alignment metrics. The representational alignment metrics we consider are correlation over pairwise similarity judgments and agreement between similarity triplets (i.e., the proportion of triplets that have the same response for the matched sets of representations). Similarity triplets provide non-metric information since in each query we discard the actual pairwise distances and only retain which one is larger. Analogously, in the pairwise case, if we use Spearman rank correlation then we are comparing

Pearson

Spearman

Triplets

Pearson

Spearman

Triplets

80

Linear probe

0.4

Linearprobe

09

40

0.2

Correlation withtestaccuracy

20

0.0

Test accuracy (%)

0.4

NN a-

1-layer NN

40

0.2

20

0.0

80

0.4

2-layer NN

2-layer NN

09

High 5

0.2

F

40

Mid 5

Low 5

20

F

0.0

0

50

0

50

0

50

0

50

0

50

0

50

Training examples per class (n)

Training examples per class (n)

the relative magnitudes of pairwise distances (of which the triplets form a subset) and discarding the magnitudes. On the other hand, Pearson correlation over pairwise similarities takes into account the magnitudes of the pairwise similarity judgments which, if accurate, can provide more information about fine-grained differences that affect alignment. We use all three metrics in our analyses and refer to them as Spearman pairwise alignment, Pearson pairwise alignment, and triplet alignment.

Few-shot learning. We measure few-shot transfer learning performance using three methods: linear probing and two classifier heads. For linear probing, we take the embeddings coming from the penultimate layer of a model and fit a logistic regression using 'scikit-learn' [28]. For the classifier heads, we use a one- and two-hidden layer neural network implemented in PyTorch [27], both with an input dropout rate of 0.8 and the latter with a ReLU hidden layer. Recent work [19] has shown that linear probing usually performs on par with fine-tuning and is a more accurate measure of the quality of internal representations that fine-tuning distorts. The heads are provided in the supplement and fixed hyperparameter values (selected based on exploratory experiments to confirm stable convergence) were used for every dataset/model for consistency.

Correcting correlation for ImageNet performance. For each of our experiments, ImageNet-1k performance can be a confounding variable when trying to understand the relationship between alignment and a downstream property. To account for this, when measuring correlation between alignment and other properties, we compute partial correlation with ImageNet-1k Top-1 validation accuracy as a covariate using the 'Pingouin' Python package [39].

4.1 Results

Alignment predicts few-shot learning performance. For each alignment metric, we compare the n -shot transfer learning performance of the five models with the highest alignment, the five models with the lowest alignment, and the five models with the nearest-to-the-mean alignment as shown in Figure 3. We find that according to all three alignment metrics, the most aligned models have far better n -shot learning performance at all levels of n .

Alignment predicts adversarial robustness. For each alignment metric, we also compare the performance on natural adversarial examples of the five models with the highest alignment, the five models with the lowest alignment, and the five models with the nearest-to-the-mean alignment as seen in Table 1. We again find that according to all three alignment metrics, the most aligned models perform far better on both Top-1 and Top-5 predictions for ImageNet-A.

Alignment predicts domain-shift robustness. Finally, for each alignment metric, we also compare the domain-shift robustness of the five models with the highest alignment, the five models with the

lowest alignment, and the five models with the nearest-to-the-mean alignment as seen in Table 1. We find that for all three alignment metrics, the most aligned models perform far better on both Top-1 and Top-5 predictions for both ImageNet-R and ImageNet-S.

Alignment has non-linear relationships with downstream properties. Based on the results in Table 1 and Figure 3, while models with low alignment underperform models with high alignment, they seem to consistently outperform models with medium alignment. Furthermore, when looking at the entire set of models in Figure 4 (Left), there appears to be a non-linear, potentially quadratic relationship between alignment and few-shot learning performance. This matches the U-shaped behavior predicted by Theorem 3.9. To test this relationship, we measure the correlations between the downstream properties of interest and the z 2 transformations of each alignment metric where z 2 ( x i ) = ( x i -µ σ ) 2 , µ is the mean, and σ is the standard deviation. We find statistically significant, positive correlations between the z 2 alignment metrics and few-shot transfer learning performance as seen in Figure 3 (Right). Using z 2 alignment metrics also results in slightly stronger correlations for domain-shift robustness but weaker correlations for adversarial robustness as seen in Table 2.

Alignment is a domain-specific metric. Since CIFAR100 is made up of naturalistic object images and so were the pre-training dataset and alignment evaluation datasets, we now test few-shot transfer learning performance on the MNIST [21] and FMNIST [42] datasets with n ∈ { 1 , 5 , 10 , 20 , 40 , 80 } examples per class to evaluate how alignment generalizes to datasets that look very different than the ones on which it was measured. FMNIST is a dataset of grayscale clothing item images. While fairly different, it is still made up of everyday objects as are the datasets on which we measured alignment. Meanwhile, MNIST is a dataset of grayscale handwritten digit images and is completely different from the datasets on which measured alignment. Thus, CIFAR100 is most similar to the datasets on which alignment was measured, followed by FMNIST and then MNIST. As shown in Figure 4 (Right), for all three datasets, we find statistically significant correlations between our Pearson alignment z 2 metric and few-shot transfer learning performance. However, we find that the effect size decreases for FMNIST and even more so for MNIST. This suggests that like any other metric, there are limits to how well representational alignment can generalize, and that it has the strongest predictive power when measured on datasets similar to those on which downstream properties are being evaluated.

70

0.04p2 -3.12pp + 94.75

0.5

CIFAR100

FMNIST

()

60

MNIST

Correlation with test accuracy

0.4

50

0.3

40

0.2

30

0.1

20

10

0.0

10

20

30

40

50

60

70

0

Training examples per class (n)

20

40

60

80

Pearsonpairwise alignment(pp)

Interactions of architecture and alignment. The results above suggest that when alignment is measured on near-distribution datasets, it can predict about 20% of the variance in one-shot learning performance. This is impressive given the large number of factors we know have an impact on downstream performance (architecture, optimizer, random seed, initialization, early termination, hardware effects on stochasticity, and many other hyperparameters). We also investigated whether some of these factors can predict alignment or interact with it to affect its relationship with downstream performance. Going through the results of our 491 models (see the Supplemental Information for the full results), we find the following insights ranked from most supporting evidence to least: 1) knowledge distillation[12] seems to increase both alignment and few-shot learning performance; 2) increasing depth/size tends to increase at least one of alignment and FSL performance (increasing ResNet[9] depth improves alignment but not FSL performance increasing depth for Resnet-RS[3] increases few-shot learning performance and not alignment; increasing Swin Transformer [22] size increases both); 3) pre-training on ImageNet21k/22k before fine-tuning on ImageNet1k increases both alignment and few-shot learning performance; 4) batch-normalization [14] for VGG [33] leads to lower alignment but slightly higher FSL performance; 5) pruning EfficientNet [37] seems to slightly decrease alignment but maintain FSL performance. We also find that for many architectures, the models cluster together (i.e. almost all MobileNets [13] have low-to-medium alignment and low FSL performance, almost all NFNets [4] have high alignment and medium-to-high FSL performance, all BEiT models [2] have very low alignment and high FSL performance). These findings suggest that architecture and training decisions can impact alignment as well as its effect on downstream properties. Thus researchers may be able to directly optimize for alignment when selecting, designing, or optimizing model architectures.

5 Discussion

Our experimental results confirm our theoretical predictions - not only do highly-aligned models exhibit better few-shot learning and robustness properties than models with lower alignment, but we also observe the U-shaped relationship with alignment predicted in Theorem 3.9. We now dissect the theoretical and empirical results via three key questions.

Which alignment metric should we be using? We find that Spearman pairwise alignment is almost perfectly correlated with triplet alignment ( ρ = 0 . 992 ). This suggests that triplets, though only a small fraction of the full set of quadruplets (i.e., all queries of the form 'X is closer to Y than A is to B'). , already capture the majority of the non-metric information contained in the entire set of quadruplets. Pearson pairwise alignment has a strong, though not quite as strong, correlation with triplet alignment ( ρ = 0 . 824 ). While all three metrics have statistically significant correlations with the downstream properties we care about, Pearson pairwise alignment seems to have the strongest correlations. This suggests that there is recoverable metric information about representations encoded in the magnitudes of human similarity judgments, even when these judgments are potentially noisy due to being elicited

without anchor stimuli that would ground the scale. The information-theoretic representation learning framework would need to be extended in future work to quantify this additional information.

Which positive downstream properties do human-aligned models exhibit? As predicted by our information-theoretic representation learning framework, our experiments suggest that very human-aligned models are better at few-shot transfer learning, more robust to natural adversarial examples, and more robust to test-time domain shift than models with lower degrees of alignment. The correlations between alignment and each downstream property were positive and statistically significant and, in every experiment we conducted, the models with the highest level of alignment outperformed the other models. These results seem to confirm the intuition that human alignment is useful in tasks where we want to use human supervision to elicit human-like behavior. While the models with the highest level of alignment clearly exhibit the best downstream performance across all three sets of tasks, our results suggest an additional unexpected insight: there is a U-shaped relationship between alignment and two of the properties we test: robustness to domain shift and few-shot transfer learning performance. Thus, while a high degree of alignment may be sufficient for eliciting desirable properties in models, it does not appear to be necessary. In fact, in cases where achieving high alignment is impractical (e.g., due to limitations on human-labeled data), it is possible that better results may be achieved by avoiding alignment altogether.

Are there downsides and limitations to representational alignment? Wehave shown that alignment is not a domain-agnostic metric and that its downstream effects are strongest when evaluated on in-distribution datasets. It is also clear that increasing alignment can damage performance across multiple criteria when that increase moves the alignment level into the medium range. But are there downsides to increasing alignment of models that are already at or past that range? Throughout this study, one of our key assumptions was that the task being solved is designed and specified by humans, or at least easily solvable by humans. However, there are numerous domains where humans have poor performance or where our representations of the problem or stimuli are not helpful for solving the task and a different set of inductive biases are required. For example, many domains targeted by deep learning - such as protein folding, drug design, and social network analysis - require geometric inductive biases [5]. In these cases, the goal should be to achieve alignment with the underlying laws governing the system of interest (e.g., physical forces or mathematical laws), rather than with humans. Finally, we note that as in all cases where models are trained with human data, social biases found in the human data may be reflected or even amplified in representationally-aligned models.

6 Conclusion

Our findings confirm the intuition that representational alignment with humans elicits desirable humanlike traits in models, including the ability to generalize from small data and lower susceptibility to adversarial attacks and domain shift. However, as both our theory and experiments suggest, increasing alignment is not always desirable, first due to a U-shaped relationship between alignment and desirable traits, and second, because there are domains where human representations simply are not useful. Notably, our discovery of the U-shaped relationship serves to resolve the tension between previously conflicting findings regarding whether alignment improves performance. We hope that our framework and results motivate further study into both the positive and negative consequences of aligning models with humans across diverse domains. We believe that representational alignment is a quantifiable and tangible route for making progress on the general AI alignment problem by allowing us to measure agreement between models and humans even over abstract domains.

Limitations and broader impact. We discussed the limitations and potential negative impacts of alignment in Section 5, but we also want to note the potential limitations and impacts of our study of alignment . First, we note that our approach assumes access to internal representations of models, but many new models today are being released via closed-source APIs that provide access only to model outputs. We hope that this and future studies on representational alignment will encourage developers to provide more open access to model internals. Second, we note that our experiments only evaluated alignment with the aggregate representational space of a large group of people which may not actually be individually representative of any of those people. Our proposed theory allows for alignment with either individuals or groups, but it is difficult to collect a large set of similarity judgments from individual participants. To ensure that we can understand whose voices, biases, and beliefs our models are actually aligned with, we believe that finding a way to account for individual differences is an important future direction.

References

K. Aho, B. D. Roads, and B. C. Love. System alignment supports cross-domain learning and zero-shot generalisation. Cognition , 227:105200, 2022. ISSN 0010-0277. doi: https: //doi.org/10.1016/j.cognition.2022.105200.

H. Bao, L. Dong, S. Piao, and F. Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=p-BhZSz59o4 .

I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T.-Y . Lin, J. Shlens, and B. Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems , 34:22614-22627, 2021.

A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning , pages 1059-1071. PMLR, 2021.

M. M. Bronstein, J. Bruna, T. Cohen, and P. Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

T. Fel, I. Felipe, D. Linsley, and T. Serre. Harmonizing the object recognition strategies of deep neural networks with humans. arXiv preprint arXiv:2211.04533 , 2022.

I. Gabriel. Artificial intelligence, values, and alignment. Minds and machines , 30(3):411-437, 2020.

R. L. Goldstone and B. J. Rogosky. Using relations within conceptual systems to translate across conceptual systems. Cognition , 84(3):295-320, 2002. ISSN 0010-0277. doi: https: //doi.org/10.1016/S0010-0277(02)00053-7.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8349, 2021.

D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15262-15271, 2021.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

K. G. Jamieson and R. D. Nowak. Low-dimensional embedding using adaptively selected ordinal data. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1077-1084, 2011.

J. H. Kirchner, L. Smith, J. Thibodeau, K. McDonell, and L. Reynolds. Researching alignment research: Unsupervised analysis. arXiv preprint arXiv:2206.02841 , 2022.

N. Kriegeskorte, M. Mur, and P. Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience , 2, 2008. ISSN 1662-5137. doi: 10.3389/neuro.06.004.2008.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=UYneFzXSJWh .

M. Kumar, N. Houlsby, N. Kalchbrenner, and E. D. Cubuk. Do better ImageNet classifiers assess perceptual similarity better? Transactions of Machine Learning Research , 2022.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

R. Marjieh, I. Sucholutsky, T. R. Sumers, N. Jacoby, and T. L. Griffiths. Predicting human similarity judgments using large language models. arXiv preprint arXiv:2202.04728 , 2022.

R. Marjieh, P. van Rijn, I. Sucholutsky, T. R. Sumers, H. Lee, T. L. Griffiths, and N. Jacoby. Words are all you need? Capturing human sensory similarity with textual descriptors. arXiv preprint arXiv:2206.04105 , 2022.

L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430 , 2022.

L. Muttenthaler, J. Dippel, L. Linhardt, R. A. Vandermeulen, and S. Kornblith. Human alignment of neural network representations. arXiv preprint arXiv:2211.01201 , 2022.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

J. C. Peterson, J. T. Abbott, and T. L. Griffiths. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive science , 42(8):2648-2669, 2018.

B. D. Roads and B. C. Love. Learning as the unsupervised alignment of conceptual systems. Nature Machine Intelligence , 2(1):76-82, 2020.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

R. N. Shepard. Multidimensional scaling, tree-fitting, and clustering. Science , 210(4468): 390-398, 1980.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

N. Soares and B. Fallenstein. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report , 8, 2014.

A. Stolk, L. Verhagen, and I. Toni. Conceptual alignment: How brains achieve mutual understanding. Trends in Cognitive Sciences , 20(3):180-191, 2016. ISSN 1364-6613. doi: https://doi.org/10.1016/j.tics.2015.11.007.

I. Sucholutsky, R. Marjieh, N. Jacoby, and T. L. Griffiths. On the informativeness of supervision signals. arXiv preprint arXiv:2211.01407 , 2022.

M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

M. Tucker, Y. Zhou, and J. A. Shah. Latent space alignment using adversarially guided self-play. International Journal of Human-Computer Interaction , 38(18-20):1753-1771, 2022.

R. Vallat. Pingouin: Statistics in Python. Journal of Open Source Software , 3(31):1026, 2018. doi: 10.21105/joss.01026.

H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

R. Wightman. PyTorch Image Models. https://github.com/rwightman/ pytorch-image-models , 2019.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

E. Yudkowsky. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker , 2016.

A Additional theoretical results

Definition A.1. Consider model A with input space X ⊆ ❘ nd , previously observed data x ∼ X , and k class centroids c ∈ ❘ kd learned by A . We define domain shift as an update to the class centroids c → c ∗ ∈ ❘ kd . Domain shift sensitivity is then the proportion of triplets flipped as a result of this update.

From this definition and Theorem 3.9, it immediately follows that sensitivity to domain shift should have the same U-shaped relationship with alignment that few-shot learning does in cases where the teacher model is robust to domain shift.

Corollary A.2. (Alignment and domain-shift robustness). Consider input space X ⊆ ❘ nd , shared data x ∼ X , and three models, A , B 1 , and B 2 with D P ( A,B 1 ; X ) = ϵ B 1 and D P ( A,B 2 ; X ) = ϵ B 2 . Let c ∈ ❘ kd be k class centroids learned by A , B 1 and B 2 . If σ A ( c, c ∗ ) = 0 and | 0 . 5 -ϵ B 1 | < | 0 . 5 -ϵ B 2 | , then σ B 1 ( c, c ∗ ) < σ B 2 ( c, c ∗ ) .

We can also use this framework to define robustness to adversarial examples. We assume that an adversarial example is an object that maximizes perceptual (i.e. representational) disagreement between the teacher and the student.

Definition A.3. Consider input space X ⊆ ❘ nd , shared data x ∼ X , and two models, A and B , with D P ( A,B 1 ; X ) = ϵ B . An adversarial example is an object e ∈ R d that maximizes disagreement between A and B on S ( x ; e ) , the subset of n ( n -1) / 2 triplets relating the objects in x to e .

Using Definition 3.6 we immediately get the following result.

Lemma A.4. Consider an input space X ⊆ ❘ nd , and two agents, A and B . D P ( A,B ; X ) = E [ || S A ( X ) -S B ( X ) || 1 n ( n -1)( n -2) / 2 ] .

We can now show that a model that is more aligned with the teacher will, on average, also be more robust to adversarial examples.

Theorem A.5. (Alignment and adversarial robustness). Consider input space X ⊆ ❘ nd , shared data x ∼ X , and three models, A , B 1 , and B 2 with D P ( A,B 1 ; X ) = ϵ B 1 and D P ( A,B 2 ; X ) = ϵ B 2 . If ϵ B 1 < ϵ B 2 , then E [max e ∈ x || S A ( x ; e ) -S B 1 ( x ; e ) || 1 ] < E [max e ∈ X || S A ( x ; e ) -S B 2 ( x ; e ) || 1 ] .

Proof. Note that for a set of k binomial random variables X i ∼ Bin ( n, p ) , the expectation of the k -th order statistic is E [ X ( k ) ] = ∑ n x =0 (1 -F ( x ; n, p ) k ) where F ( x ; n, p ) = P ( X i ≤ x ) . In the case of adversarial examples, let X i be a random variable corresponding to the set of objects sampled uniformly from the input space X ⊆ ❘ nd then U = || S A ( X ; e ) -S B 1 ( X ; e ) || 1 , Y ∼ Bin ( n ( n -1) / 2 , ϵ B 1 ) and similarly V = || S A ( X ; e ) -S B 2 ( X ; e ) || 1 , V ∼ Bin ( n ( n -1) / 2 , ϵ B 1 ) . In that case, the expected disagreement of A and B 1 on an adversarial example is E [ U ( n ) ] = ∑ n ( n -1) / 2 x =0 (1 -F ( x ; n ( n -1) / 2 , ϵ B 1 ) n ) and for A and B 2 it is E [ V ( n ) ] = ∑ n ( n -1) / 2 x =0 (1 -F ( x ; n ( n -1) / 2 , ϵ B 2 ) n ) . If ϵ B 1 < ϵ B 2 , then F ( x ; n ( n -1) / 2 , ϵ B 1 ) > F ( x ; n ( n -1) / 2 , ϵ B 2 ) and thus E [ U ( n ) ] < E [ V ( n ) ] .

Remark A.6 . While this theorem shows that increased alignment generally leads to increased adversarial robustness, this relies on a representational metric of adversarial examples. However, in practice, adversarial robustness is often measured using hard classification error as a simple proxy. This proxy does not capture the fine-grained degree of misalignment between humans and a model on each example. As a result, when measuring adversarial robustness using this proxy, the effect of alignment may be dampened by the U-shaped effect seen in other classification settings as mentioned above.

B List of 491 models used in experiments

adv_inception_v3, bat_resnext26ts, beit_base_patch16_224, beit_base_patch16_384, beit_large_patch16_224, beit_large_patch16_384, botnet26t_256, cait_s24_224, cait_s24_384,

cait_s36_384, cait_xs24_384, cait_xxs24_224, cait_xxs24_384, cait_xxs36_224, cait_xxs36_384, coat_lite_mini, coat_lite_small, coat_lite_tiny, coat_mini, coat_tiny, convit_base, convit_small, convit_tiny, convmixer_1024_20_ks9_p14, convmixer_1536_20, convmixer_768_32, con-vnext_base, convnext_base_384_in22ft1k, convnext_base_in22ft1k, convnext_large, con-vnext_large_384_in22ft1k, convnext_large_in22ft1k, convnext_small, convnext_tiny, cspdarknet53, cspresnet50, cspresnext50, deit_base_patch16_224, deit_base_patch16_384, deit_small_patch16_224, deit_tiny_patch16_224, densenet121, densenet161, densenet169, densenet201, densenetblur121d, dla102, dla102x, dla102x2, dla169, dla34, dla46_c, dla46x_c, dla60, dla60_res2net, dla60_res2next, dla60x, dla60x_c, dm_nfnet_f0, dm_nfnet_f1, dm_nfnet_f2, dpn107, dpn131, dpn68, dpn68b, dpn92, dpn98, eca_botnext26ts_256, eca_halonext26ts, eca_nfnet_l0, eca_nfnet_l1, eca_nfnet_l2, eca_resnet33ts, eca_resnext26ts, ecaresnet101d, ecaresnet101d_pruned, ecaresnet269d, ecaresnet26t, ecaresnet50d, ecaresnet50d_pruned, ecaresnet50t, ecaresnetlight, efficientnet_b0, efficientnet_b1, efficientnet_b1_pruned, efficientnet_b2, efficientnet_b2_pruned, efficientnet_b3, efficientnet_b3_pruned, efficientnet_b4, efficientnet_el, efficientnet_el_pruned, efficientnet_em, efficientnet_es, efficientnet_es_pruned, efficientnet_lite0, efficientnetv2_rw_m, efficientnetv2_rw_s, efficientnetv2_rw_t, ens_adv_inception_resnet_v2, ese_vovnet19b_dw, ese_vovnet39b, fbnetc_100, fbnetv3_b, fbnetv3_d, fbnetv3_g, gc_efficientnetv2_rw_t, gcresnet33ts, gcresnet50t, gcresnext26ts, gcresnext50ts, gernet_l, gernet_m, gernet_s, ghostnet_100, gluon_inception_v3, gluon_resnet101_v1b, gluon_resnet101_v1c, gluon_resnet101_v1d, gluon_resnet101_v1s, gluon_resnet152_v1b, gluon_resnet152_v1c, gluon_resnet152_v1d, gluon_resnet152_v1s, gluon_resnet18_v1b, gluon_resnet34_v1b, gluon_resnet50_v1b, gluon_resnet50_v1c, gluon_resnet50_v1d, gluon_resnet50_v1s, gluon_resnext101_32x4d, gluon_resnext101_64x4d, gluon_resnext50_32x4d, gluon_senet154, gluon_seresnext101_32x4d, gluon_seresnext101_64x4d, gluon_seresnext50_32x4d, gluon_xception65, gmixer_24_224, gmlp_s16_224, halo2botnet50ts_256, halonet26t, halonet50ts, haloregnetz_b, hardcorenas_a, hardcorenas_b, hardcorenas_c, hardcorenas_d, hardcorenas_e, hardcorenas_f, hrnet_w18, hrnet_w18_small, hrnet_w18_small_v2, hrnet_w30, hrnet_w32, hrnet_w40, hrnet_w44, hr-net_w48, hrnet_w64, ig_resnext101_32x16d, ig_resnext101_32x8d, inception_resnet_v2, inception_v3, inception_v4, jx_nest_base, jx_nest_small, jx_nest_tiny, lambda_resnet26rpt_256, lambda_resnet26t, lambda_resnet50ts, lamhalobotnet50ts_256, lcnet_050, lcnet_075, lc-net_100, legacy_senet154, legacy_seresnet101, legacy_seresnet152, legacy_seresnet18, legacy_seresnet34, legacy_seresnet50, legacy_seresnext101_32x4d, legacy_seresnext26_32x4d, legacy_seresnext50_32x4d, mixer_b16_224, mixer_b16_224_miil, mixnet_l, mixnet_m, mixnet_s, mixnet_xl, mnasnet_100, mnasnet_small, mobilenetv2_050, mobilenetv2_100, mobilenetv2_110d, mobilenetv2_120d, mobilenetv2_140, mobilenetv3_large_100, mo-bilenetv3_large_100_miil, mobilenetv3_rw, nasnetalarge, nf_regnet_b1, nf_resnet50, nfnet_l0, pit_b_224, pit_s_224, pit_ti_224, pit_xs_224, pnasnet5large, regnetx_002, regnetx_004, reg-netx_006, regnetx_008, regnetx_016, regnetx_032, regnetx_040, regnetx_064, regnetx_080, regnetx_120, regnetx_160, regnetx_320, regnety_002, regnety_004, regnety_006, regnety_008, regnety_016, regnety_032, regnety_040, regnety_064, regnety_080, regnety_120, regnety_160, regnety_320, regnetz_b16, regnetz_c16, regnetz_d32, regnetz_d8, regnetz_e8, repvgg_a2, repvgg_b0, repvgg_b1, repvgg_b1g4, repvgg_b2, repvgg_b2g4, repvgg_b3, repvgg_b3g4, res2net101_26w_4s, res2net50_14w_8s, res2net50_26w_4s, res2net50_26w_6s, res2net50_26w_8s, res2net50_48w_2s, res2next50, resmlp_12_224, resmlp_12_distilled_224, resmlp_24_224, resmlp_24_distilled_224, resmlp_36_224, resmlp_36_distilled_224, resmlp_big_24_224, resmlp_big_24_224_in22ft1k, resmlp_big_24_distilled_224, resnest101e, resnest14d, resnest200e, resnest269e, resnest26d, resnest50d, resnest50d_1s4x24d, resnest50d_4s2x40d, resnet101, resnet101d, resnet152, resnet152d, resnet18, resnet18d, resnet200d, resnet26, resnet26d, resnet26t, resnet32ts, resnet33ts, resnet34, resnet34d, resnet50, resnet50_gn, resnet50d, resnet51q, resnet61q, resnetblur50, resnetrs101, resnetrs152, resnetrs200, resnetrs270, resnetrs350, resnetrs420, resnetrs50, resnetv2_101, resnetv2_101x1_bitm, resnetv2_50, resnetv2_50x1_bit_distilled, resnetv2_50x1_bitm, resnext101_32x8d, resnext26ts, resnext50_32x4d, resnext50d_32x4d, rexnet_100, rexnet_130, rexnet_150, rexnet_200, se-botnet33ts_256, sehalonet33ts, selecsls42b, selecsls60, selecsls60b, semnasnet_075, semnas-net_100, seresnet152d, seresnet33ts, seresnet50, seresnext26d_32x4d, seresnext26t_32x4d, seresnext26ts, seresnext50_32x4d, skresnet18, skresnet34, skresnext50_32x4d, spnasnet_100, ssl_resnet18, ssl_resnet50, ssl_resnext101_32x16d, ssl_resnext101_32x4d, ssl_resnext101_32x8d, ssl_resnext50_32x4d, swin_base_patch4_window12_384, swin_base_patch4_window7_224, swin_large_patch4_window12_384, swin_large_patch4_window7_224,

swin_small_patch4_window7_224, swin_tiny_patch4_window7_224, swsl_resnet18, swsl_resnet50, swsl_resnext101_32x16d, swsl_resnext101_32x4d, swsl_resnext101_32x8d, swsl_resnext50_32x4d, tf_efficientnet_b0, tf_efficientnet_b0_ap, tf_efficientnet_b0_ns, tf_efficientnet_b1, tf_efficientnet_b1_ap, tf_efficientnet_b1_ns, tf_efficientnet_b2, tf_efficientnet_b2_ap, tf_efficientnet_b2_ns, tf_efficientnet_b3, tf_efficientnet_b3_ap, tf_efficientnet_b3_ns, tf_efficientnet_b4, tf_efficientnet_b4_ap, tf_efficientnet_b4_ns, tf_efficientnet_b5, tf_efficientnet_b5_ap, tf_efficientnet_b5_ns, tf_efficientnet_b6, tf_efficientnet_b6_ap, tf_efficientnet_b6_ns, tf_efficientnet_b7, tf_efficientnet_b7_ap, tf_efficientnet_b7_ns, tf_efficientnet_cc_b0_4e, tf_efficientnet_cc_b0_8e, tf_efficientnet_cc_b1_8e, tf_efficientnet_el, tf_efficientnet_em, tf_efficientnet_es, tf_efficientnet_lite0, tf_efficientnet_lite1, tf_efficientnet_lite2, tf_efficientnet_lite3, tf_efficientnet_lite4, tf_efficientnetv2_b0, tf_efficientnetv2_b1, tf_efficientnetv2_b2, tf_efficientnetv2_b3, tf_efficientnetv2_l, tf_efficientnetv2_l_in21ft1k, tf_efficientnetv2_m, tf_efficientnetv2_m_in21ft1k, tf_efficientnetv2_s, tf_efficientnetv2_s_in21ft1k, tf_inception_v3, tf_mixnet_l, tf_mixnet_m, tf_mixnet_s, tf_mobilenetv3_large_075, tf_mobilenetv3_large_100, tf_mobilenetv3_large_minimal_100, tf_mobilenetv3_small_075, tf_mobilenetv3_small_100, tf_mobilenetv3_small_minimal_100, tinynet_a, tinynet_b, tinynet_c, tinynet_d, tinynet_e, tnt_s_patch16_224, tv_densenet121, tv_resnet101, tv_resnet152, tv_resnet34, tv_resnet50, tv_resnext50_32x4d, twins_pcpvt_base, twins_pcpvt_large, twins_pcpvt_small, twins_svt_base, twins_svt_large, twins_svt_small, vgg11, vgg11_bn, vgg13, vgg13_bn, vgg16, vgg16_bn, vgg19, vgg19_bn, vis-former_small, vit_base_patch16_224, vit_base_patch16_224_miil, vit_base_patch16_384, vit_base_patch32_224, vit_base_patch32_384, vit_base_patch8_224, vit_base_r50_s16_384, vit_small_patch16_224, vit_small_patch16_384, vit_small_patch32_224, vit_small_patch32_384, vit_small_r26_s32_224, vit_small_r26_s32_384, vit_tiny_patch16_224, vit_tiny_patch16_384, vit_tiny_r_s16_p8_224, vit_tiny_r_s16_p8_384, wide_resnet101_2, wide_resnet50_2, xception, xception41, xception65, xception71, xcit_large_24_p16_224, xcit_large_24_p16_224_dist, xcit_large_24_p16_384_dist, xcit_large_24_p8_224, xcit_large_24_p8_224_dist, xcit_large_24_p8_384_dist, xcit_medium_24_p16_224, xcit_medium_24_p16_224_dist, xcit_medium_24_p16_384_dist, xcit_medium_24_p8_224, xcit_medium_24_p8_224_dist, xcit_nano_12_p16_224, xcit_nano_12_p16_224_dist, xcit_nano_12_p16_384_dist, xcit_nano_12_p8_224, xcit_nano_12_p8_224_dist, xcit_nano_12_p8_384_dist, xcit_small_12_p16_224, xcit_small_12_p16_224_dist, xcit_small_12_p16_384_dist, xcit_small_12_p8_224, xcit_small_12_p8_224_dist, xcit_small_12_p8_384_dist, xcit_small_24_p16_224, xcit_small_24_p16_224_dist, xcit_small_24_p16_384_dist, xcit_small_24_p8_224, xcit_small_24_p8_224_dist, xcit_small_24_p8_384_dist, xcit_tiny_12_p16_224, xcit_tiny_12_p16_224_dist, xcit_tiny_12_p16_384_dist, xcit_tiny_12_p8_224, xcit_tiny_12_p8_224_dist, xcit_tiny_12_p8_384_dist, xcit_tiny_24_p16_224, xcit_tiny_24_p16_224_dist, xcit_tiny_24_p16_384_dist, xcit_tiny_24_p8_224, xcit_tiny_24_p8_224_dist, xcit_tiny_24_p8_384_dist

C Correlations with ImageNet validation accuracy

We note that alignment metrics also have a (weak) U-shaped relationship with ImageNet validation accuracy (see Figure 5). We also note the following correlations between ImageNet validation accuracy and other properties which show that validation accuracy does not explain all the variance in downstream properties:

ImageNet-A: 0.81

ImageNet-S: 0.86

ImageNet-R: 0.82

CIFAR10 1-shot learning: 0.68

FMNIST 1-shot learning: 0.10

MNIST 1-shot learning: 0.07

20

30

40

50

60

70

Pearson pairwise alignment

60

65

70

75

80

85

ImageNet validation acc.

D Outlier examples

We compared performance on the original 'noisy' ImageNet and relabeled ImageNet-ReaL [10]. All models had better performance on ReaL, but the improvement has a weak inverted-U relationship with alignment. This may seem surprising, but our understanding is that the ReaL dataset was collected by relabelling images on which models (pre-2020) disagreed with the original labels. Models with low performance by today's standards (and alignment near 0.5) were used to select which images to relabel, so it is unsurprising that models with alignment near 0.5 benefit most from this relabeling. This suggests that ImageNet labels may need to be revisited once again as ReaL relabelling may have been biased.

E Reproducibility: Code and results data

All code and full results data are provided as part of the supplemental information. We will share them publicly after the anonymity period is over. All experiments were conducted on an AWS 'x1.16xlarge' instance (no GPUs).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the relationship between the alignment of deep learning based representations with humans and the downstream performance of the NN models. Using triplets as a non-metric measure of similarity, the authors use information theory to show that alignment and downstream performance would have a U-shaped relationship, where low and high alignment would give a good downstream performance.\n\nSTRENGTHS:\n* The authors use existing ideas of triplets from cognitive science and combine information theory to show why human alignment may not always lead to better performance, especially for few-shot downstream learning. More importantly, the authors give an intuition and theoretical backing for why some models can have low alignment with humans and still perform well in downstream learning.\n* The authors highlight the significance of ensuring human alignment of neural network representations by studying the role of alignment on two critical characteristics: adversarial robustness and domain shift robustness. They further show how human alignment plays a role in downstream learning, mainly when limited data is available.\n* Using a significantly large number of models and large-scale existing human-annotated data, the authors empirically validated the theoretical claims made in the paper, showing the benefit of very high alignment with robustness while also revealing the U-shaped relationship between few-shot learning and alignment. The authors performed critical statistical testing to highlight the results.\n* The interaction of architecture model choices and alignment is insightful and could be helpful for future work.\n\nWEAKNESSES:\n* The contribution seems limited, the main contribution seems to be the showing of the U-shaped relationship. The primary notion of triplets is derived from prior work.\n* The importance of triplets seems limited. Pearson and Spearman also clearly provide similar takeaways. Why are triplets needed? Not clear from the evaluation section.\n* Evaluation is also limited in some aspects, although the paper explores more than 400 models. Most transfer learning results are shown for CIFAR after ImageNet pre-training. The result on MNIST and FMNIST show quite weak results, meaning that strong domain shift during transfer learning makes the metric less impactful. But transfer learning is usually most important in these settings (e.g., medical domain, face classification, etc.).\n* There is a lack of clarity in the writing of the paper. Section 3 is difficult to follow, and it is not always clear how the different definitions flow, and how the pipeline works. E.g., not clear what is the significance of Def. 3.4, Discussion section mentions \"quadruplets\", but they are never mentioned in the paper. Not clear at all.\n* Adversarial robustness is measured for real-world adversarial images. More common adversarial examples used in literature, e.g., $\\ell_p$ ball perturbations are not explored.\n* Evaluation section compares the triplets with pairwise measures like Pearson and Spearman correlations. But it is not clear from Section 3 as to why it is interesting to compare triplets to pairwise. Do triplets give benefits over Pearson/Spearman? Are there any tradeoffs?\n* Results shown in the tables are not well discussed\n* Transfer learning results only analyzed when we have \"head\" networks. Not clear if the method works for full network fine-tuning or not, something that is often used in practice.\n* No justification given on the choice of the heads, e.g., dropout rate, etc. Were they hyper parameter-tuned?\n* Not clear what and how the ImageNet accuracy correction is performed. It seems important, so more intuition should be provided.\n\nQUESTIONS:\n* Why does correlation with test accuracy reduce with more training examples (Fig. 3 and 4 right)?\n* How are the classifier heads parameters chosen?\n* What is meant by supervised learning in Section 3? Does it mean standard classification task? It is not clear.\n* What are quadruplets mentioned in Discussion?\n* Large datasets like ImageNet and CIFAR have a lot of \"noisy\" and misclassified examples on which the deep models are trained on. Do the different alignment metrics reveal any interesting facts on the model's alignment or non-alignment on these \"outlier\" examples?\n\nLIMITATIONS:\n* Most transfer learning results are shown for transfer from ImageNet to CIFAR, which is limiting.\n* Exact contribution of triplets is not clear from the paper, as existing correlation metrics can already show the U-shaped relation and performance relations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work provides theoretical and empirical analysis for the relationship between human alignment, few-shot performance, and robustness to distribution shift. The theoretical analysis predicts models that have high and low alignment with human annotations will have better performance on few-shot and distribution shift datasets. The experiments verify the predicted relationship between representational alignment and downstream performance.\n\nSTRENGTHS:\n- Alignment between humans and ML models has been studied considerably from the perspective of interpretability and safety. Understanding how and whether alignment can improve sample-efficiency and robustness is a less explored, but important research question which this work makes progress towards. \n\n- The paper theoretically and empirically shows that alignment is not monotonic with respect to robustness/sample efficiency which may explain why previous works have found conflicting results regarding the benefits of alignments. I think this finding would be useful to the community. Also the information-theoretic framework proposed by the authors for analyzing representational alignment would likely be useful to future work. \n\n- The experiments are thorough and adequately support the claims. \n\n- The paper is well organized and the tables and plots clearly illustrate the main empirical findings.\n\nWEAKNESSES:\nOther works [1][2] have shown that the validation accuracy on an in-distribution (ID) dataset is highly correlated with accuracy on out-of-distribution (OOD) and downstream datasets. For reference, the R-squared value for pretrained models evaluated on ImageNet and ImageNetV2 is .996 and other distribution shifts have similar correlation. Given that in-distribution validation accuracy is almost entirely predictive of out-of-distribution accuracy, I think the current motivation, which is to show that alignment is predictive of downstream performance, is weak. Representational alignment as a predictive metric for robustness and downstream transfer performance is less accurate than standard ID validation accuracy. Also obtaining validation accuracy is easier since it only requires one-hot labeling, as opposed to similarity matrices. \n\n\nIf the main motivation of the paper is to show that improving alignment will confer benefits such as robustness and few-shot performance, then it should show that alignment measures something more than in-distribution validation accuracy. Otherwise, robustness and few-shot performance could simply be increased by improving the model on the validation set. \n\nOverall I think the paper is interesting and original. My main concern is for the motivation, but would be happy to change my rating if this concern is adequately addressed. \n\n\n[1] Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization\n\n[2] Do ImageNet Classifiers Generalize to ImageNet?\n\nQUESTIONS:\n- It would be helpful to see the distribution of alignment metrics and some analysis on what factors other than validation accuracy affect the alignment metric. Given that validation accuracy is almost entirely predictive of robustness and downstream performance (greater than .9 R-squared), it seems that validation accuracy and alignment would be highly correlated. \n\n- In the abstract it’s stated that highly-aligned models are more robust to adversarial attacks. This claim seems overly strong as the experiments are conducted with natural adversarial images, and not with white or black box attacks. \n\n- Given theorem 3.9, would the relationship between alignment and few-shot performance be v-shaped and not u-shaped?\n\nLIMITATIONS:\nThe authors adequately address the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a theory for a u-shaped effect in terms of model representation alignment, where models that are more and less aligned with human representations should have better performance over a number of tasks. They then perform a large number of experiments, comparing 491 models with a representation approximated based on 425,000 human judgements. They find that the quantitative results largely hold with their theory. \n\nEdit: Based on the author's rebuttal and further discussion I have raised my initial review to an accept.\n\nSTRENGTHS:\nThis paper has a large number of strengths. It is well-written and well-presented. The theory is novel and valuable, providing insights for future researchers and developers. The experiments are exhaustive and well-motivated. The depth of discussion around justifying the choices in the experiments is excellent. The experiments also largely follow the theory, which suggests useful takeaways to a broad audience.\n\nWEAKNESSES:\nThis is a strong paper, but there's a couple of weaknesses which dampen my enthusiasm somewhat. \n\nThe largest of these weaknesses is that while the authors repeatedly refer to a U-shaped effect, the results actually suggest a J-shaped effect. The low alignment models consistently demonstrate a significantly worse performance than the high-alignment models Despite this, there's no discussion of this point throughout the paper. The reasoning seems intuitively straightforward, despite note following from the theory. While it is true that a model could \"realize\" to interpret signals in an \"opposite\" manner (to use the paper's term) with low alignment, this realization is still extra effort. In a model trained on backprop, it is not always possible to \"flip\" the use of features in a large, multidimensional space. This or similar discussion accounting for this issue should be brought up in the paper. \n\nSecond, while a large number of experiments are run, the choice of only a top 5, bottom 5, and mean 5 is somewhat odd. This represents less than 3% of the models used in the experiments. This is understandable in some cases, like presenting individual values in tables. But in Figures like Figure 3 a more continual effect could have been demonstrated with a typical standard deviation off the lines. If the authors chose to only use 15 models, some experiment helping to demonstrate that the results weren't a fluke would be beneficial. For example, ranking the 491 models in term of alignment and visualizing some performance metric across all 491. As it is, readers need to look through a dense CSV file for this information. There's notably no readMe or similar to explain the labels for the CSV file making it difficult to understand what's going on here without digging through code. Some clarity on this would be appreciated and would benefit the paper.\n\nQUESTIONS:\n1. How do the authors account for the effect being more J-shaped than U-shaped as predicted?\n2. Why did the authors choose to only look at high-5/meanest-5/lowest-5 models?\n\nLIMITATIONS:\nThe authors identify the issues around the choice of alignment, otherwise there are no major potential negative societal issues.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
- The authors predict that there should be an U-shaped relationship between the degree of representational alignment and performance on few-shot tasks based on information theoretic analysis, and empirically show that is the case using hundreds of vision models. They further show that the degree of alignment predicts robustness to ImageNet-A, domain-shift (e.g. ImageNet-R, -S.)

STRENGTHS:
- The U-shaped results are novel and interesting phenomenon. With both information theoretical analysis and detailed empirical analysis, the results seem credible, and provide new perspective on the previously conflicting findings as to whether improving alignment increases performance.
- They study three different alignment metrics.
- The degree of alignment not only explains the few-shot performance, but also predicts robustness to distributional shift (e.g. ImageNet-A, -R, -S) which intuitively make sense.

WEAKNESSES:
- Minor:
	- I believe ImageNet-A is just a subset of ImageNet-test set that happens to be difficult, but the term "adversarial robustness" often refers to images with actual adversarial noise, so I think it should be clarified.

QUESTIONS:
- Figure 4 Left: Curious to see if there is any characteristics of models that are in the top left of the graph (the ones with test accuracy around 60% and alignment being less than 20), as these points seem to be rather outliers.

LIMITATIONS:
- The analysis seems to be limited to ImageNet-based models. Recently there are more capable vision-language models such as BLIP-2, which combines a vision model with Large Language Model. While the current paper conducts detailed analysis, it would be a more interesting work if they also study these recent vision-language models.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27