SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alignment predicts adversarial robustness. For each alignment metric, we also compare the performance on natural adversarial examples of the five models with the highest alignment, the five models with the lowest alignment, and the five models with the nearest-to-the-mean alignment as seen in Table 1. We again find that according to all three alignment metrics, the most aligned models perform far better on both Top-1 and Top-5 predictions for ImageNet-A.
Alignment predicts domain-shift robustness. Finally, for each alignment metric, we also compare the domain-shift robustness of the five models with the highest alignment, the five models with the
Table 1: Average top-1 and top-5 accuracy on ImageNet-A, ImageNet-S, and ImageNet-R of the five models with highest, lowest, and closest-to-the-mean levels of Pearson  $(\rho_{P})$  Spearman  $(\rho_{S})$ , and triplet alignment.
Alignment metrics
ρp	ρs	Triplet
IN-A (Top 1)	High 5	59.93	46.83	44.56
Mid 5	10.26	12.30	18.35
Low 5	30.89	21.80	29.99
IN-A (Top 5)	High 5	84.92	73.86	72.20
Mid 5	36.54	38.14	46.63
Low 5	59.10	49.36	57.70
IN-R (Top 1)	High 5	61.57	53.78	54.27
Mid 5	39.16	40.01	41.80
Low 5	51.33	46.76	50.16
IN-R (Top 5)	High 5	75.92	68.61	68.82
Mid 5	55.08	56.81	57.67
Low 5	66.55	61.84	65.22
IN-S (Top 1)	High 5	48.06	40.37	40.31
Mid 5	26.90	28.02	29.85
Low 5	38.11	33.90	37.04
IN-S (Top 5)	High 5	71.10	62.45	62.24
Mid 5	44.90	46.67	48.58
Low 5	58.91	53.43	57.57
Table 2: Pearson correlation between Top1 and Top-5 accuracy on ImageNet-A, ImageNet-R, and ImageNet-Sketch (-S) and Pearson  $(\rho_{P})$ , Spearman  $(\rho_{S})$ , and triplet  $z^{2}$  alignment metrics.
Alignment metrics
ρp	ρs	Triplet
IN-A (Top 1)	0.239	0.151	0.157
(p=0.000)	(p=0.001)	(p=0.000)
IN-A (Top 5)	0.210	0.112	0.121
(p=0.000)	(p=0.013)	(p=0.007)
IN-R (Top 1)	0.166	0.164	0.182
(p=0.000)	(p=0.000)	(p=0.000)
IN-R (Top 5)	0.173	0.170	0.194
(p=0.000)	(p=0.000)	(p=0.000)
IN-S (Top 1)	0.191	0.164	0.179
(p=0.000)	(p=0.000)	(p=0.000)
IN-S (Top 5)	0.211	0.197	0.218
(p=0.000)	(p=0.000)	(p=0.000)
lowest alignment, and the five models with the nearest-to-the-mean alignment as seen in Table 1. We find that for all three alignment metrics, the most aligned models perform far better on both Top-1 and Top-5 predictions for both ImageNet-R and ImageNet-S.
Alignment has non-linear relationships with downstream properties. Based on the results in Table 1 and Figure 3, while models with low alignment underperform models with high alignment, they seem to consistently outperform models with medium alignment. Furthermore, when looking at the entire set of models in Figure 4 (Left), there appears to be a non-linear, potentially quadratic relationship between alignment and few-shot learning performance. This matches the U-shaped behavior predicted by Theorem 3.9. To test this relationship, we measure the correlations between the downstream properties of interest and the  $z^2$  transformations of each alignment metric where  $z^2(x_i) = \left(\frac{x_i - \mu}{\sigma}\right)^2$ ,  $\mu$  is the mean, and  $\sigma$  is the standard deviation. We find statistically significant, positive correlations between the  $z^2$  alignment metrics and few-shot transfer learning performance as seen in Figure 3 (Right). Using  $z^2$  alignment metrics also results in slightly stronger correlations for domain-shift robustness but weaker correlations for adversarial robustness as seen in Table 2.
Alignment is a domain-specific metric. Since CIFAR100 is made up of naturalistic object images and so were the pre-training dataset and alignment evaluation datasets, we now test few-shot transfer learning performance on the MNIST [21] and FMNIST [42] datasets with  $n \in \{1, 5, 10, 20, 40, 80\}$  examples per class to evaluate how alignment generalizes to datasets that look very different than the ones on which it was measured. FMNIST is a dataset of grayscale clothing item images. While fairly different, it is still made up of everyday objects as are the datasets on which we measured alignment. Meanwhile, MNIST is a dataset of grayscale handwritten digit images and is completely different from the datasets on which measured alignment. Thus, CIFAR100 is most similar to the datasets on which alignment was measured, followed by FMNIST and then MNIST. As shown in Figure 4 (Right), for all three datasets, we find statistically significant correlations between our Pearson alignment  $z^2$  metric and few-shot transfer learning performance. However, we find that the effect size decreases for FMNIST and even more so for MNIST. This suggests that like any other metric, there are limits to how well representational alignment can generalize, and that it has the strongest predictive power when measured on datasets similar to those on which downstream properties are being evaluated.
Figure 4: Left: Comparing 1-shot learning performance on CIFAR100 to Pearson pairwise alignment for all 491 models. Right: Pearson correlation (with  $95\%$  confidence intervals) between  $n$ -shot transfer learning performance using linear probing and the Pearson  $(\rho_{P})z^{2}$  alignment metric for CIFAR100, FMNIST, and MNIST.
Interactions of architecture and alignment. The results above suggest that when alignment is measured on near-distribution datasets, it can predict about  $20\%$  of the variance in one-shot learning performance. This is impressive given the large number of factors we know have an impact on downstream performance (architecture, optimizer, random seed, initialization, early termination, hardware effects on stochasticity, and many other hyperparameters). We also investigated whether some of these factors can predict alignment or interact with it to affect its relationship with downstream performance. Going through the results of our 491 models (see the Supplemental Information for the full results), we find the following insights ranked from most supporting evidence to least: 1) knowledge distillation[12] seems to increase both alignment and few-shot learning performance; 2) increasing depth/size tends to increase at least one of alignment and FSL performance (increasing ResNet[9] depth improves alignment but not FSL performance increasing depth for Resnet-RS[3] increases few-shot learning performance and not alignment; increasing Swin Transformer [22] size increases both); 3) pre-training on ImageNet21k/22k before fine-tuning on ImageNet1k increases both alignment and few-shot learning performance; 4) batch-normalization [14] for VGG [33] leads to lower alignment but slightly higher FSL performance; 5) pruning EfficientNet [37] seems to slightly decrease alignment but maintain FSL performance. We also find that for many architectures, the models cluster together (i.e. almost all MobileNets [13] have low-to-medium alignment and low FSL performance, almost all NFNets [4] have high alignment and medium-to-high FSL performance, all BEiT models [2] have very low alignment and high FSL performance). These findings suggest that architecture and training decisions can impact alignment as well as its effect on downstream properties. Thus researchers may be able to directly optimize for alignment when selecting, designing, or optimizing model architectures.
Our experimental results confirm our theoretical predictions – not only do highly-aligned models exhibit better few-shot learning and robustness properties than models with lower alignment, but we also observe the U-shaped relationship with alignment predicted in Theorem 3.9. We now dissect the theoretical and empirical results via three key questions.
Which alignment metric should we be using? We find that Spearman pairwise alignment is almost perfectly correlated with triplet alignment  $(\rho = 0.992)$ . This suggests that triplets, though only a small fraction of the full set of quadruplets (i.e., all queries of the form "X is closer to Y than A is to B"), already capture the majority of the non-metric information contained in the entire set of quadruplets. Pearson pairwise alignment has a strong, though not quite as strong, correlation with triplet alignment  $(\rho = 0.824)$ . While all three metrics have statistically significant correlations with the downstream properties we care about, Pearson pairwise alignment seems to have the strongest correlations. This suggests that there is recoverable metric information about representations encoded in the magnitudes of human similarity judgments, even when these judgments are potentially noisy due to being elicited
without anchor stimuli that would ground the scale. The information-theoretic representation learning framework would need to be extended in future work to quantify this additional information.
Which positive downstream properties do human-aligned models exhibit? As predicted by our information-theoretic representation learning framework, our experiments suggest that very human-aligned models are better at few-shot transfer learning, more robust to natural adversarial examples, and more robust to test-time domain shift than models with lower degrees of alignment. The correlations between alignment and each downstream property were positive and statistically significant and, in every experiment we conducted, the models with the highest level of alignment outperformed the other models. These results seem to confirm the intuition that human alignment is useful in tasks where we want to use human supervision to elicit human-like behavior. While the models with the highest level of alignment clearly exhibit the best downstream performance across all three sets of tasks, our results suggest an additional unexpected insight: there is a U-shaped relationship between alignment and two of the properties we test: robustness to domain shift and few-shot transfer learning performance. Thus, while a high degree of alignment may be sufficient for eliciting desirable properties in models, it does not appear to be necessary. In fact, in cases where achieving high alignment is impractical (e.g., due to limitations on human-labeled data), it is possible that better results may be achieved by avoiding alignment altogether.
Are there downsides and limitations to representational alignment? We have shown that alignment is not a domain-agnostic metric and that its downstream effects are strongest when evaluated on in-distribution datasets. It is also clear that increasing alignment can damage performance across multiple criteria when that increase moves the alignment level into the medium range. But are there downsides to increasing alignment of models that are already at or past that range? Throughout this study, one of our key assumptions was that the task being solved is designed and specified by humans, or at least easily solvable by humans. However, there are numerous domains where humans have poor performance or where our representations of the problem or stimuli are not helpful for solving the task and a different set of inductive biases are required. For example, many domains targeted by deep learning – such as protein folding, drug design, and social network analysis – require geometric inductive biases [5]. In these cases, the goal should be to achieve alignment with the underlying laws governing the system of interest (e.g., physical forces or mathematical laws), rather than with humans. Finally, we note that as in all cases where models are trained with human data, social biases found in the human data may be reflected or even amplified in representationally-aligned models.
Our findings confirm the intuition that representational alignment with humans elicits desirable human-like traits in models, including the ability to generalize from small data and lower susceptibility to adversarial attacks and domain shift. However, as both our theory and experiments suggest, increasing alignment is not always desirable, first due to a U-shaped relationship between alignment and desirable traits, and second, because there are domains where human representations simply are not useful. Notably, our discovery of the U-shaped relationship serves to resolve the tension between previously conflicting findings regarding whether alignment improves performance. We hope that our framework and results motivate further study into both the positive and negative consequences of aligning models with humans across diverse domains. We believe that representational alignment is a quantifiable and tangible route for making progress on the general AI alignment problem by allowing us to measure agreement between models and humans even over abstract domains.
Limitations and broader impact. We discussed the limitations and potential negative impacts of alignment in Section 5, but we also want to note the potential limitations and impacts of our study of alignment. First, we note that our approach assumes access to internal representations of models, but many new models today are being released via closed-source APIs that provide access only to model outputs. We hope that this and future studies on representational alignment will encourage developers to provide more open access to model internals. Second, we note that our experiments only evaluated alignment with the aggregate representational space of a large group of people which may not actually be individually representative of any of those people. Our proposed theory allows for alignment with either individuals or groups, but it is difficult to collect a large set of similarity judgments from individual participants. To ensure that we can understand whose voices, biases, and beliefs our models are actually aligned with, we believe that finding a way to account for individual differences is an important future direction.
We would like to thank Lukas Mutenthaler for excellent discussions that helped shape some of the ideas explored in this paper. This work was supported by an ONR grant (N00014-18-1-2873) to TLG and an NSERC fellowship (567554-2022) to IS.
[1] K. Aho, B. D. Roads, and B. C. Love. System alignment supports cross-domain learning and zero-shot generalisation. Cognition, 227:105200, 2022. ISSN 0010-0277. doi: https://doi.org/10.1016/j.cognition.2022.105200.
[2] H. Bao, L. Dong, S. Piao, and F. Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=p-BhZSz59o4.
[3] I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T.-Y. Lin, J. Shlens, and B. Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems, 34:22614-22627, 2021.
[4] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059-1071. PMLR, 2021.
[5] M. M. Bronstein, J. Bruna, T. Cohen, and P. Velicković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[6] T. Fel, I. Felipe, D. Linsley, and T. Serre. Harmonizing the object recognition strategies of deep neural networks with humans. arXiv preprint arXiv:2211.04533, 2022.
[7] I. Gabriel. Artificial intelligence, values, and alignment. *Minds and machines*, 30(3):411-437, 2020.
[8] R. L. Goldstone and B. J. Rogosky. Using relations within conceptual systems to translate across conceptual systems. Cognition, 84(3):295-320, 2002. ISSN 0010-0277. doi: https://doi.org/10.1016/S0010-0277(02)00053-7.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[11] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[12] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
[13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[14] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[15] K. G. Jamieson and R. D. Nowak. Low-dimensional embedding using adaptively selected ordinal data. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1077-1084, 2011.
[16] J. H. Kirchner, L. Smith, J. Thibodeau, K. McDonell, and L. Reynolds. Researching alignment research: Unsupervised analysis. arXiv preprint arXiv:2206.02841, 2022.
[37] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.
[38] M. Tucker, Y. Zhou, and J. A. Shah. Latent space alignment using adversarially guided self-play. International Journal of Human-Computer Interaction, 38(18-20):1753-1771, 2022.
[39] R. Vallat. Pingouin: Statistics in Python. Journal of Open Source Software, 3(31):1026, 2018. doi: 10.21105/joss.01026.
[40] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[41] R. Wightman. PyTorch Image Models. https://github.com/rwrightman/pytorch-image-models, 2019.
[42] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[43] E. Yudkowsky. The AI alignment problem: why it is hard, and where to start. *Symbolic Systems Distinguished Speaker*, 2016.
A Additional theoretical results
Definition A.1. Consider model  $A$  with input space  $X \subseteq \mathbb{R}^{nd}$ , previously observed data  $x \sim X$ , and  $k$  class centroids  $c \in \mathbb{R}^{kd}$  learned by  $A$ . We define domain shift as an update to the class centroids  $c \rightarrow c^* \in \mathbb{R}^{kd}$ . Domain shift sensitivity is then the proportion of triplets flipped as a result of this update.
$$
\sigma_ {A} (c, c ^ {*}) := E [ \frac {| | S _ {A} (x ; c) - S _ {A} (x ; c ^ {*}) | | _ {1}}{| S _ {A} (x ; c) |} ]
$$
From this definition and Theorem 3.9, it immediately follows that sensitivity to domain shift should have the same U-shaped relationship with alignment that few-shot learning does in cases where the teacher model is robust to domain shift.
Corollary A.2. (Alignment and domain-shift robustness). Consider input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , and three models,  $A$ ,  $B_1$ , and  $B_2$  with  $D_P(A, B_1; X) = \epsilon_{B_1}$  and  $D_P(A, B_2; X) = \epsilon_{B_2}$ . Let  $c \in \mathbb{R}^{kd}$  be  $k$  class centroids learned by  $A$ ,  $B_1$  and  $B_2$ . If  $\sigma_A(c, c^*) = 0$  and  $|0.5 - \epsilon_{B_1}| < |0.5 - \epsilon_{B_2}|$ , then  $\sigma_{B_1}(c, c^*) < \sigma_{B_2}(c, c^*)$ .
We can also use this framework to define robustness to adversarial examples. We assume that an adversarial example is an object that maximizes perceptual (i.e. representational) disagreement between the teacher and the student.
Definition A.3. Consider input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , and two models,  $A$  and  $B$ , with  $D_P(A, B_1; X) = \epsilon_B$ . An adversarial example is an object  $e \in \mathbb{R}^d$  that maximizes disagreement between  $A$  and  $B$  on  $S(x; e)$ , the subset of  $n(n - 1)/2$  triplets relating the objects in  $x$  to  $e$ .
$$
e = \max  _ {X} | | S _ {A} (x; e) - S _ {B} (x; e) | | _ {1} \tag {1}
$$
Using Definition 3.6 we immediately get the following result.
Lemma A.4. Consider an input space  $X \subseteq \mathbb{R}_{nd}$ , and two agents,  $A$  and  $B$ .  $D_P(A,B;X) = E[\frac{||S_A(X) - S_B(X)||_1}{n(n - 1)(n - 2) / 2}]$ .
We can now show that a model that is more aligned with the teacher will, on average, also be more robust to adversarial examples.
Theorem A.5. (Alignment and adversarial robustness). Consider input space  $X \subseteq \mathbb{R}^{nd}$ , shared data  $x \sim X$ , and three models,  $A$ ,  $B_{1}$ , and  $B_{2}$  with  $D_{P}(A,B_{1};X) = \epsilon_{B_{1}}$  and  $D_{P}(A,B_{2};X) = \epsilon_{B_{2}}$ . If  $\epsilon_{B_1} < \epsilon_{B_2}$ , then  $E[\max_{e \in x}||S_A(x;e) - S_{B_1}(x;e)||_1] < E[\max_{e \in X}||S_A(x;e) - S_{B_2}(x;e)||_1]$ .
Proof. Note that for a set of  $k$  binomial random variables  $X_{i} \sim \text{Bin}(n, p)$ , the expectation of the  $k$ -th order statistic is  $E[X_{(k)}] = \sum_{x=0}^{n} (1 - F(x; n, p)^{k})$  where  $F(x; n, p) = P(X_{i} \leq x)$ . In the case of adversarial examples, let  $X_{i}$  be a random variable corresponding to the set of objects sampled uniformly from the input space  $X \subseteq \mathbb{R}^{nd}$  then  $U = ||S_{A}(X; e) - S_{B_{1}}(X; e)||_{1}, Y \sim \text{Bin}(n(n-1)/2, \epsilon_{B_{1}})$  and similarly  $V = ||S_{A}(X; e) - S_{B_{2}}(X; e)||_{1}, V \sim \text{Bin}(n(n-1)/2, \epsilon_{B_{1}})$ . In that case, the expected disagreement of  $A$  and  $B_{1}$  on an adversarial example is  $E[U_{(n)}] = \sum_{x=0}^{n(n-1)/2} (1 - F(x; n(n-1)/2, \epsilon_{B_{1}})^{n})$  and for  $A$  and  $B_{2}$  it is  $E[V_{(n)}] = \sum_{x=0}^{n(n-1)/2} (1 - F(x; n(n-1)/2, \epsilon_{B_{2}})^{n})$ . If  $\epsilon_{B_{1}} < \epsilon_{B_{2}}$ , then  $F(x; n(n-1)/2, \epsilon_{B_{1}}) > F(x; n(n-1)/2, \epsilon_{B_{2}})$  and thus  $E[U_{(n)}] < E[V_{(n)}]$ .
Remark A.6. While this theorem shows that increased alignment generally leads to increased adversarial robustness, this relies on a representational metric of adversarial examples. However, in practice, adversarial robustness is often measured using hard classification error as a simple proxy. This proxy does not capture the fine-grained degree of misalignment between humans and a model on each example. As a result, when measuring adversarial robustness using this proxy, the effect of alignment may be dampened by the U-shaped effect seen in other classification settings as mentioned above.
B List of 491 models used in experiments
adv_inception_v3, bat_resnext26ts, beit_base PATCH16_224, beit_base PATCH16_384, beit_largePatch16_224, beit_largePatch16_384, botnet26t_256, cait_s24_224, cait_s24_384,