SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ilia Sucholutsky
is2961@princeton.edu
Thomas L. Griffiths
Departments of Psychology and Computer Science
tomg@princeton.edu
Should we care whether AI systems have representations of the world that are similar to those of humans? We provide an information-theoretic analysis that suggests that there should be a U-shaped relationship between the degree of representational alignment with humans and performance on few-shot learning tasks. We confirm this prediction empirically, finding such a relationship in an analysis of the performance of 491 computer vision models. We also show that highly-aligned models are more robust to both natural adversarial attacks and domain shifts. Our results suggest that human alignment is often a sufficient, but not necessary, condition for models to make effective use of limited data, be robust, and generalize well.
As AI systems are increasingly deployed in settings that involve interactions with humans, exploring the extent to which these systems are aligned with humans becomes more significant. While this exploration has largely focused on the alignment of the values of AI systems with humans [7, 16], the alignment of their representations is also important. Representing the world in the same way is a precursor to being able to express common values and to comprehensible generalization. To the extent that humans have accurate representations of the world, representational alignment is also an effective source of inductive bias that might make it possible to learn from limited data.
As a motivating example, imagine a meeting between a 16th century alchemist and a 21st century chemist. They live in the same physical world and are intimately familiar with the materials that comprise it, but they would have significant difficulty expressing their values and generalizing the results of an experiment they observe together. The alchemist would likely learn poorly from examples of a reaction demonstrated by the chemist, not having the right inductive biases for the way the world actually works. The alchemist and the chemist lack representational alignment – they represent the world in fundamentally different ways – and this impedes generalization and learning.
In this paper, we provide a theoretical and empirical investigation of the consequences of representational alignment with humans for AI systems, using popular computer vision tasks as a way to study this phenomenon. We define representational alignment as the degree to which the latent representations of a model match the latent representations of humans for the same set of stimuli, and refer to models that are representationally aligned with humans as being "human-aligned." Several recent papers have proposed ways to measure [23, 24], explain [26, 20], and even improve [29, 6] the representational alignment of models. However, many models that score low on these alignment metrics still have high performance on downstream tasks like image classification [20, 26, 6].
So, are there any real advantages (or disadvantages) to using human-aligned models? To answer this question, we develop an information-theoretic framework for analyzing representational alignment. This framework enables us to make predictions about emergent behavior in human-aligned models.
A - SHARED DATA
Figure 1: Schematic of representational alignment between two agents, Alice and Bob. A: Shared data  $(x)$  is shown to both agents (images are from the animal subset of the similarity datasets from Peterson et al. [29]). B: Both agents form representations  $(f_{A}(x)$  and  $f_{B}(x))$  of the objects they observe. C: Agents are asked to produce pairwise similarity matrices corresponding to their representations. The similarity judgments can then be compared to measure alignment between the agents.
B - REPRESENTATIONS
Chimp	Wolf	Eagle	Lemur
1	0.8	0.4	0
0.8	1	0.2	0.1
0.4	0.2	1	0.6
0	0.1	0.6	1
Chimp	Wolf	Eagle	Lemur
1	0.8	0.4	0.1
0.8	1	0.5	0
0.4	0.5	1	0.6
0.1	0	0.6	1
C - PAIRWISE SIMILARITY
A - NEW DATA
B-TEACHER
C- STUDENT
Figure 2: Schematic of triplet-based supervised learning where one agent is a teacher and the other a student. A: A new object is shown only to the Teacher. B: The Teacher forms a representation of the object and sends the Student a triplet relating the new object to two objects previously observed by both agents. C: The Student interprets the triplet in their own representation space and eliminates the half-plane where the new object cannot be located (shaded in red) according to the triplet.
In particular, our framework predicts that few-shot transfer learning performance should have a U-shaped relationship with alignment. To verify the predictions of our framework and to probe for additional properties that arise as a consequence of alignment, we conduct a series of experiments in which we assess the downstream properties of human-aligned models compared to their non-aligned counterparts. From our experiments comparing 491 large computer-vision models to over 425,000 human judgments (across 1200 participants), we identify three properties:
- Models with either high or low alignment with humans are better at few-shot learning (FSL) than models with medium alignment, even correcting for pre-training performance.
- Human-aligned models are more robust to natural adversarial examples, even when correcting for classification performance on the pre-training dataset.
- Human-aligned models are more robust to domain shift, even when correcting for classification performance on the pre-training dataset.
The U-shaped relationship between alignment and few-shot learning helps to explain why previous results have not consistently observed benefits of representational alignment. Overall, our results suggest that representational alignment can provide real and significant benefits for downstream tasks, but that it may be a sufficient rather than necessary condition for these benefits to emerge.
With AI systems entering mainstream usage, aligning these models with human values and understanding is increasingly important [7, 16]. This concept, referred to as AI alignment, is an important but still largely open problem [43], in part due to the difficulty of formalizing it [34], or even reaching consensus on a definition [16]. In this paper, we focus on formalizing a specific aspect of AI alignment: representational alignment.
Representational alignment is a measure of agreement between the representations of two learning agents (one of whom is typically a human). There are numerous names, definitions, measures, and uses of this form of alignment across various fields, including cognitive science, neuroscience, and machine learning. Some of the other names include latent space alignment [38], concept(ual) alignment [35, 26], system alignment [8, 30, 1], representational similarity analysis (RSA) [17], and model alignment [24]. Shepard [32] proposed that human representations can be recovered by using behavioral data to measure the similarity of a set of stimuli and then finding embeddings that satisfy those similarity associations using methods like multi-dimensional scaling (MDS). Similarly, in neuroscience, Representational Similarity Analysis (RSA) is a popular technique for relating neural, behavioral, and computational representations of the same set of stimuli via similarity analysis [17]. While similarity analysis has clearly proven itself to be a powerful method, exhaustively collecting pairwise similarity judgments is expensive ( $O(N^2)$  judgments for  $N$  stimuli) and there have been numerous proposals aiming to develop more efficient methods of recovering human representations.
Jamieson and Nowak [15] proposed an active learning scheme for querying for human judgments using triplets of the form "is  $a$  closer to  $b$  than to  $c$ ?" and derived bounds on the number of required queries for lossless completion of the full similarity matrix using such queries. When an approximate solution is acceptable, Peterson et al. [29] showed that pre-trained computer vision models can be used to approximate human perceptual similarity judgments over images. Marjieh et al. [23] showed that human perceptual similarity can be more accurately, but still efficiently, approximated from natural language descriptions of the stimuli of interest (for example by using large language models to estimate similarity over pairs of these descriptions). Marjieh et al. [24] extended this result to more domains (vision, audio, and video) and measured alignment for hundreds of pre-trained models.
Several recent studies have also attempted to identify what design choices lead to improved representational alignment in models [20, 26, 6], although Moschella et al. [25] found that even with variation in design choices, many models trained on the same dataset end up learning similar 'relative representations' (embeddings projected into a relational form like a similarity matrix), or in other words, converge to the same representational space. Tucker et al. [38] showed that representational alignment emerges not only in static settings like image classification, but also dynamic reinforcement learning tasks involving human-robot interaction. Several studies have also focused on studying alignment specifically in humans, both between different people and for a single person but across multiple tasks and domains [8, 30].
Although several recent papers have proposed ways to measure [23, 24], explain [26, 20], and even improve [29, 6] the representational alignment of models, few have focused on studying the downstream impact of a model being representationally aligned with humans, and many studies simply rely on the intuition that better alignment leads to better performance to justify pursuing increased alignment. While there is recent evidence to suggest that alignment may help humans learn across domains and perform zero-shot generalization [1], there is also evidence to suggest that alignment may not always be beneficial for models, with models scoring low on alignment metrics achieving higher performance on downstream tasks like image classification [20, 26, 6]. Our goal in this paper is to conduct an in-depth study into the downstream effects of representational alignment. Our theoretical and empirical results both validate and explain the apparently conflicting results seen in previous literature as special cases of the (more complex than previously suspected) effects of representational alignment.
3 Theory
If we want to measure representational alignment across different architectures with potentially mismatched embedding coordinate spaces and dimensionalities, then we need a definition of representation spaces that enables comparisons of such spaces. Inspired by the cognitive science literature on
using non-metric similarity triplets to recover hidden psychological representations, and building on rigorous computational theory that analyzes such triplets [15], we propose a triplet-based definition of representation learning. We summarize representational alignment under our framework in the schematic shown in Figure 1.
Definition 3.1. As proposed by Jamieson and Nowak [15], for an ordered set of  $n$  objects in  $d$  dimensions represented as a vector  $x \in \mathbb{R}^{nd}$ , a similarity triplet corresponding to the question 'is  $x_i$  closer to  $x_j$  than to  $x_k$ ?' is a membership query of the form  $\mathbb{1}_{x \in r_{ijk}}$  where  $r_{ijk} = \{x \in \mathbb{R}^{nd} : |x_i - x_j| < |x_i - x_k|\}$ .
Definition 3.2. For an ordered set of objects  $x \in \mathbb{R}^{nd}$  and model  $M$  with embeddings  $f_{M}: \mathbb{R}^{d} \to \mathbb{R}^{d_{M}}$ , the triplet-based representation space of  $M$  is the set of all triplets  $S_{M}(x) = \{(i,j,k) \in \{1,\dots,n\}^{3}: \mathbb{1}_{x \in r_{ijk}^{M}}, i \neq j, j \neq k, k \neq i\}$  where  $r_{ijk}^{M} = \{x \in \mathbb{R}^{nd}: |f_{M}(x_{i}) - f_{M}(x_{j})| < |f_{M}(x_{i}) - f_{M}(x_{k})|\}$ .
Remark 3.3. Since switching the order of  $j$  and  $k$  in a triplet deterministically flips the query between 0 and 1, it is easy to see that, for a set of  $n$  objects, a representation space is determined by a set of just  $\frac{n(n - 1)(n - 2)}{2}$  unique triplets. Thus, for a set of  $n$  objects,  $S_{M}(x)$  can be represented as a binary vector of length  $\frac{n(n - 1)(n - 2)}{2}$  and we use this interpretation in the remainder of this section.
Definition 3.4. Consider a training set  $x \in \mathbb{R}^{nd}$  and a corresponding target representation space  $S_T(x)$ . We define representation learning with a parametrized model  $M_{\theta}$  as optimizing the objective  $\min_{\theta} \ell(S_T(x), S_{M_{\theta}}(x))$  where  $\ell$  is a loss function penalizing divergence between the target and learned representation spaces.
Sucholutsky et al. [36] showed that many ML objectives and associated supervision signals, ranging from similarity judgments for contrastive learning to labels for classification, can be converted into similarity triplets making them compatible with this definition of representation learning. We also immediately get an intuitive definition of alignment between two agents (e.g. a model and a person) as the rate of agreement on matched triplets (also known as simple match coefficient).
Definition 3.5. Consider a dataset  $x \in \mathbb{R}^{nd}$  and two agents,  $A$  and  $B$ , with corresponding representation spaces  $S_A(x)$  and  $S_B(x)$ . We define representational misalignment between  $A$  and  $B$  as  $D_R(A,B;x) = \frac{||S_A(x) - S_B(x)||_1}{n(n - 1)(n - 2) / 2}$ . Representational alignment is then simply  $1 - D_R(A,B;x)$ .
Definition 3.6. Consider a single set of three points  $t = (x_i, x_j, x_k) \in X \subseteq \mathbb{R}^{3d}$  sampled uniformly at random, and two agents,  $A$  and  $B$ . We define stochastic representational misalignment between  $A$  and  $B$  as  $D_P(A, B; X) = P(\mathbb{1}_{x \in r_{ijk}^A} = \mathbb{1}_{x \in r_{ijk}^B})$ . Stochastic representational alignment is then simply  $1 - D_P(A, B; X)$ .
Since each triplet corresponds to one bit of information [15], and we have a probabilistic definition of alignment, we can now use information theory to define what it means for one agent to supervise another (potentially misaligned) agent. We visualize supervised learning under our framework in Figure 2. For perfectly aligned models, Jamieson and Nowak [15] derive the following result.
Lemma 3.7. Consider input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , new object  $c \in \mathbb{R}^d$ , and models  $A$  and  $B$  with  $D_P(A, B; X) = 0$ .  $\Omega(d \log(n))$  triplet queries are required for  $B$  to identify the location of  $c$  relative to all objects in  $x$ .
Proposition 3.8. For input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , new object  $c \in \mathbb{R}^d$ , and models  $A, B$  with  $D_P(A, B; X) = \epsilon$ ,  $\Omega\left(\frac{d \log(n)}{1 + \epsilon \log(\epsilon) + (1 - \epsilon) \log(1 - \epsilon)}\right)$  triplets are required for  $B$  to learn  $c$ .
Proof. Consider a communication game involving Alice and Bob, a dataset  $x$  that they both have their own representations for (we call this 'shared data'), and a new object  $c$  that only Alice can see. Alice can only communicate with Bob by giving binary responses to triplet queries based on Alice's own representations of the objects. The goal is for Bob to learn the location of  $c$  with respect to the other objects in  $x$ .  $D_P(A, B; X) = \epsilon$  is the probability for any triplet drawn from  $x \cup c$  to be flipped in Bob's representation space relative to Alice's. This is equivalent to Alice and Bob communicating over a binary symmetric channel where the probability of a bit flip is  $\epsilon$ . The capacity of this channel is  $C_{\epsilon} = 1 - H(\epsilon, 1 - \epsilon) = 1 + \epsilon \log(\epsilon) + (1 - \epsilon) \log(1 - \epsilon)$ . For error-free communication over this channel, the highest achievable rate is  $R < C_{\epsilon}$ , meaning at least  $\frac{1}{C_{\epsilon}}$  bits are required to communicate each useful bit over this channel. By Lemma 3.7,  $\Omega(d \log(n))$  useful bits are required for Bob to learn the location of  $c$  over a channel with no error, and thus a total of  $\Omega(\frac{d \log(n)}{C_{\epsilon}})$  bits are required.
Theorem 3.9 (Few-shot learning). Consider input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , new objects  $c \in \mathbb{R}^{md}$  and three models  $A, B_1,$  and  $B_2$  with  $D_P(A, B_1; X) = \epsilon_{B_1}$  and  $D_P(A, B_2; X) = \epsilon_{B_2}$ . If  $|0.5 - \epsilon_{B_1}| < |0.5 - \epsilon_{B_2}|$ , then  $B_2$  requires fewer queries to learn  $c$  than  $B_1$  does.
Proof. From Proposition 3.8, it immediately follows that  $B_{1}$  requires  $\Omega\left(\frac{md\log(n)}{C_{\epsilon_{B_1}}}\right)$  queries and  $B_{2}$  requires  $\Omega\left(\frac{md\log(n)}{C_{\epsilon_{B_2}}}\right)$  queries.  $C_{\epsilon} : [0,1] \to [0,1]$  is a symmetric convex function with a minimum of 0 at  $\epsilon = 0.5$  and maxima of 1 at  $\epsilon = 0,1$ . Thus, if  $|0.5 - \epsilon_{B_1}| < |0.5 - \epsilon_{B_2}|$ , then  $\Omega\left(\frac{md\log(n)}{C_{\epsilon_{B_1}}}\right) > \Omega\left(\frac{md\log(n)}{C_{\epsilon_{B_2}}}\right)$ .
Sucholutsky et al. [36] showed that commonly used supervision signals, like hard and soft classification labels, can be converted into sets of triplets. Reducing the number of required triplets to learn the location of a new object is equivalent to reducing the number of required labels. It follows from Theorem 3.9 that high or low alignment leads to high few-shot learning performance on downstream tasks like classification (where learning a new class can be formulated as learning the location of the centroid or boundaries of that class using a small number of labels), while medium alignment leads to low few-shot learning performance. Intuitively, this comes as a result of mutual information between the teacher and student representations being minimized when alignment is at 0.5. In other words, if Bob knows that most bits coming from Alice are flipped (i.e.  $D_P(A,B) = \epsilon > 0.5$ ), then going forward, Bob can flip all the bits coming from Alice to achieve a lower error rate (i.e.  $D_P(\bar{A},B) = 1 - \epsilon < 0.5$ ).
We note that generalizing under domain-shift can be considered a form of zero-shot transfer learning and selecting adversarial examples can be seen as selecting objects that maximize representational disagreement between a model and a human. Under these interpretations, our framework also predicts that highly-aligned models are robust to both domain shift and adversarial examples though the framing in these cases is less intuitive than for the few-shot learning case. We share some preliminary theoretical analyses of robustness properties in the Supplement.
Our theoretical analysis shows how representational alignment can serve as a source of inductive bias that reduces the number of bits that need to be acquired from data. In particular, Theorem 3.9 predicts an unexpected U-shaped relationship between alignment and few-shot learning performance. We now present a series of experiments designed to test this prediction and examine whether it extends to robustness to natural adversarial examples and domain shift.
Models. The pre-trained models evaluated in this paper are taken from the PyTorch Image Models package [41]. The full list of 491 models used in our experiments can be found in the Supplement.
Data. All of the models used in this paper were pre-trained on ImageNet-1k [31] and had their performance evaluated on the ImageNet-1k validation set. Adversarial robustness was measured on the ImageNet-A dataset of natural adversarial examples that have been found to foil most ImageNet models [11]. Zero-shot domain shift robustness was measured on the ImageNet-R dataset of renditions (e.g., paintings, toys, statues, etc.) of ImageNet classes [10] and the ImageNet-Sketch dataset of black-and-white sketches of the ImageNet classes [40]. All ImageNet and ImageNet variant results come from the PyTorch Image Models package [41]. Few-shot transfer learning performance was evaluated on the CIFAR100 [18] test set with  $n \in \{1, 5, 10, 20, 40, 80\}$  examples per class used for few-shot learning and the remaining examples used for evaluation. We measure alignment by computing the three metrics described below on the six image datasets from Peterson et al. [29] and their respective sets of similarity judgments consisting of over 425,000 judgments across 1200 participants, as described by Marjieh et al. [23]. We average each alignment metric over the six datasets.
Alignment metrics. The representational alignment metrics we consider are correlation over pairwise similarity judgments and agreement between similarity triplets (i.e., the proportion of triplets that have the same response for the matched sets of representations). Similarity triplets provide non-metric information since in each query we discard the actual pairwise distances and only retain which one is larger. Analogously, in the pairwise case, if we use Spearman rank correlation then we are comparing
Figure 3: Left: Average  $n$ -shot transfer learning performance using linear probing, a 1-layer classification head, and a 2-layer classification head on CIFAR100 of the five models with highest, lowest, and closest-to-the-mean levels for each of Pearson  $(\rho_{P})$ , Spearman  $(\rho_{S})$ , and triplet alignment. Right: Pearson correlation (with 95% confidence intervals) between  $n$ -shot transfer learning performance using each classification head on CIFAR100 and each  $z^{2}$  alignment metric.
the relative magnitudes of pairwise distances (of which the triplets form a subset) and discarding the magnitudes. On the other hand, Pearson correlation over pairwise similarities takes into account the magnitudes of the pairwise similarity judgments which, if accurate, can provide more information about fine-grained differences that affect alignment. We use all three metrics in our analyses and refer to them as Spearman pairwise alignment, Pearson pairwise alignment, and triplet alignment.
Few-shot learning. We measure few-shot transfer learning performance using three methods: linear probing and two classifier heads. For linear probing, we take the embeddings coming from the penultimate layer of a model and fit a logistic regression using 'scikit-learn' [28]. For the classifier heads, we use a one- and two-hidden layer neural network implemented in PyTorch [27], both with an input dropout rate of 0.8 and the latter with a ReLU hidden layer. Recent work [19] has shown that linear probing usually performs on par with fine-tuning and is a more accurate measure of the quality of internal representations that fine-tuning distorts. The heads are provided in the supplement and fixed hyperparameter values (selected based on exploratory experiments to confirm stable convergence) were used for every dataset/model for consistency.
Correcting correlation for ImageNet performance. For each of our experiments, ImageNet-1k performance can be a confounding variable when trying to understand the relationship between alignment and a downstream property. To account for this, when measuring correlation between alignment and other properties, we compute partial correlation with ImageNet-1k Top-1 validation accuracy as a covariate using the 'Pingouin' Python package [39].
Alignment predicts few-shot learning performance. For each alignment metric, we compare the  $n$ -shot transfer learning performance of the five models with the highest alignment, the five models with the lowest alignment, and the five models with the nearest-to-the-mean alignment as shown in Figure 3. We find that according to all three alignment metrics, the most aligned models have far better  $n$ -shot learning performance at all levels of  $n$ .