SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Whereas the local structure of gLocal and global representations closely resembles that of the original representation, the global structure instead more closely resembles the naive transformed representation. We quantify global representational similarity using linear centered kernel alignment (LCKA) [43, 14]. LCKA can be thought of as measuring the similarity between principal components (PCs) of two representations weighted by the amount of variance they explain; see further discussion in Appx. G. It thus primarily reflects similarity between the large PCs that define global representational structure. As shown in Fig. 3 (left), LCKA indicates that the gLocal/global representations are more similar to the naive transformed representation than to the untransformed representation, suggesting that the gLocal, global, and naive transforms induce similar changes in global structure. We further measure LCKA between representations obtained by setting all singular values to zero except for those corresponding to either largest 10 PCs or the remaining 758 PCs. LCKA between representations retaining the largest 10 PCs resembles LCKA between the full representations (Fig. 3 middle). However, when retaining only the remaining PCs, the gLocal/global representations are more similar to the untransformed representation than to the naive-transformed representation (Fig. 3 right).
Figure 4: The gLocal transformed representation captures global structure of the naive transformed representation, as shown by PCA, but local structure of the untransformed representation, as shown by t-SNE with perplexity 10. Visualizations reflect embeddings of 10 images from each of the 260 Entity-13 ImageNet subclasses obtained from CLIP ViT-L and are colored by the superclasses. 2D embeddings are rotated to align with the gLocal representation using orthogonal Procrustes. The t-SNE fitting process is initialized using PCA; we pick the embedding with the lowest loss from 10 runs.
In Fig. 4, we visualize the effects of different transformations using PCA, which preserves global structure, and t-SNE, which preserves local structure. As described by Van der Maaten & Hinton [89], PCA "focus[es] on keeping the low-dimensional representations of dissimilar datapoints far apart," whereas t-SNE tries to "keep the low-dimensional representations of very similar datapoints close together." In line with the results above, the global structure of the gLocal representation revealed by PCA closely resembles that of the naive transformed representation, whereas the local structure of the gLocal representation revealed by t-SNE closely resembles that of the untransformed representation.
To complement these analyses, in Appx. B we explore what the alignment transforms alter about the global category structure of the model representations. Generally, categories within a single superordinate category move more closely together, while different superordinate categories move apart, but with sensible exceptions — e.g. food and drink move closer together.
4.3 Few-shot learning
Table 1: 5-shot FS results using the original or transformed representations.  $\dagger$  indicates the highest accuracy for each dataset. Results are averaged over 5 runs.
Model \ Transform	Entity-13	Entity-30	CIFAR100-Coarse	CIFAR100	SUN397	DTD
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
CLIP-RN50 (WIT)	63.99	67.96	57.86	59.80	44.27	47.43	38.77	39.59	57.21	58.79	54.32	54.95
CLIP-ViT-L/14 (WIT)	65.34	71.94†	66.92	69.97†	66.97	68.48	72.22	73.03	69.87	71.13	62.84	63.27
CLIP-ViT-L/14 (LAION-400M)	65.33	69.02	62.93	65.99	68.88	69.58	73.57	72.98	70.25	71.08	67.81	66.71
CLIP-ViT-L/14 (LAION-2B)	65.98	71.24	65.64	67.93	72.43	73.48†	79.01†	78.48	71.62	72.62†	69.49†	68.44
In this section, we examine the impact of the gLocal transform on few-shot classification accuracy on downstream datasets. We consider a standard fine-grained few-shot learning setup on CIFAR-100 [48], SUN397 [90], and the Describable Textures Dataset [DTD, 12], as well as a coarse-grained setup on Entity-{13,30} of BREEDS [77].
In coarse-grained FS, classes are grouped into semantically meaningful subclasses. This is a more challenging setting than the standard fine-grained scenario, for which there does not exist a superordinate grouping. In fine-grained FS, training examples are uniformly drawn from all (sub-)classes. For coarse-grained FS, we classify according to subclasses rather than fine-grained classes, and choose  $k$  training examples uniformly at random for each superclass. This implies that not every subclass is contained in the train set if the number of training samples is smaller than the number of subclasses. On Entity-{13,30}, subclasses between train and test sets are guaranteed to be disjoint due to a subpopulation shift. To achieve high accuracy, models must consider examples from unseen members of a superclass similar to the few examples it has seen. Task performance is dependent on how well the partial information contained in the few examples of a superclass can be exploited by a model to characterize the entire superclass with its subclasses. Hence, global similarity structure is more crucial than local similarity structure to perform well on this task. We calculate classification accuracy across all available test images of all subclasses, using coarse labels.
We find that transforming the representations via gLocal transforms substantially improves performance over the untransformed representations across both coarse-grained and fine-grained FS tasks for all image/text models considered (Tab. 1). For CLIP models trained on LAION, however, we do not observe improvements for CIFAR-100 and DTD, which are the most fine-grained datasets. For ImageNet models, the gLocal transform improves performance on Entity-{13,30}, but has almost no impact on the performance for the remaining datasets (see Appx. D.1).
4.4 Anomaly detection
Here, we evaluate the performance of representations on  $k$ -nearest neighbor AD with and without using the gLocal transform. AD methods typically return an anomaly score for which a detection threshold has to be chosen. Our method is evaluated using each training class as the nominal class and we report the average AUROC. Following §3.1, we compute the representations for each normal sample of the training set and then evaluate the model with representations from the test set. We set  $k = 5$  for our experiments but found that performance is fairly insensitive to the choice of  $k$  (see Appx. D.2). For measuring the distance between representations, we use cosine similarity.
In Tab. 2, we show that the gLocal transform substantially improves AD performance across all datasets considered in our analyses, for almost every image/text model. However, as in the few-shot setting, we observe no improvements over the untransformed representation space for ImageNet models (see Appx. D.2). In Tab. 3, we further investigate performance on distribution shift datasets,
where improvements are particularly striking. Here, global similarity structure appears to be crucial for generalizing between the superclasses. Tab. 2 additionally reports current state-of-the-art (SOTA) results on the standard benchmarks; SOTA results are not available for the distribution shift datasets. SOTA approaches generally use additional data relevant to the AD task, such as outlier exposure data or textual supervision for the normal class [53, 13, 36], whereas we use only human similarity judgments. Our transformation also works well in non-standard AD settings (see Appx. D.2).
Table 2: One-vs-rest nearest neighbor based AD results; with and without transformation.  $\dagger$  indicates the highest accuracy for each dataset.
Model \ Transform	CIFAR10	CIFAR100	CIFAR100-Coarse	ImageNet30	DTD
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
CLIP-RN50 (WIT)	89.44	91.19	90.83	92.92	86.47	89.29	98.89	99.05	90.67	92.29
CLIP-ViT-L/14 (WIT)	95.14	98.16	91.41	97.19	88.5	95.83	98.91	99.75†	92.02	94.9
CLIP-ViT-L/14 (LAION-400M)	98.8	98.39	98.66	98.53	97.86	97.77	99.51	99.69	95.54	96.44
CLIP-ViT-L/14 (LAION-2B)	98.97	99.11†	98.76	98.97†	98.05	98.5†	99.29	99.74	94.87	96.78†
SOTA	99.6 [53]	-	97.34 [13]	99.9 [53]	94.6 [53]
Table 3: One-vs-rest AD with a class distribution shift between train and test sets; with and without transformation.  $\dagger$  indicates the highest accuracy for each dataset.
Model \ Transform	Entity-13	Entity-30	Living-17	Nonliving-26	Cifar100-shift
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
CLIP-RN50 (WIT)	90.22	92.03	91.64	93.71	94.62	93.28	87.49	91.09	76.27	80.33
CLIP-ViT-L/14 (WIT)	88.54	93.23†	91.31	95.53	97.31	97.7†	84.73	92.53	73.69	87.11
CLIP-ViT-L/14 (LAION-400M)	90.79	92.71	92.49	95.04	96.56	96.32	90.09	93.18	91.09	92.7
CLIP-ViT-L/14 (LAION-2B)	90.33	93.08	92.1	95.48†	96.96	97.37	88.82	93.54†	89.73	93.94†
4.5 Representational alignment
We have shown above that the gLocal transform provides performance gains on FS and AD tasks. Here, we examine whether these performance gains come at the cost of alignment with human similarity judgments as compared to the naive transform, which does not preserve local structure. Thus, we examine the impact of the gLocal transform on human alignment, using the same human similarity judgment datasets evaluated in Muttenthaler et al. [61] plus an additional dataset. Specifically, we perform representational similarity analysis [RSA; 46] between representations of two image/text models — CLIP RN50 and CLIP ViT-L/14 — and human behavior for four different human similarity judgment datasets [31, 65, 66, 10, 41]. RSA is a method for comparing neural network representations to representations obtained from human behavior [46]. In RSA, one first obtains representational similarity matrices (RSMs) for the human behavioral judgments and for the neural network representations (more details in Appx. E). These RSMs measure the similarity between pairs of examples according to each source. As in previous work [10, 41, 61], we use the Spearman rank correlation coefficient to quantify the similarity of these RSMs. We find that there is almost no trade-off in representational alignment for the gLocal transform compared to the naively transformed representations (see Tab. 4). Hence, the gLocal transform can improve representational alignment while preserving local similarity structure.
In Fig. 5, we further demonstrate how the representational changes introduced by the linear and gLocal transforms lead to greater alignment with human similarity judgments by visualizing the RSMs on each dataset. The global similarity structure captured by the RSMs is qualitatively identical between naive and gLocal transforms, and both of these transforms lead to RSMs that closely resemble human RSMs (see Fig. 5). Human similarity judgments for data from Hebart et al. [31] were collected in the form of triplet odd-one-out choices. Therefore, we used VICE [60] — an approximate Bayesian method for inferring mental representations of object concepts from human behavior — to obtain an RSM for those judgments. Human RSMs are sorted into five different concept clusters using  $k$ -means for datasets from King et al. [41] and Cichy et al. [10] and using the THINGS concept hierarchy [30] for RSMs from Hebart et al. [31]. A visualization for RSMs obtained from CLIP RN50 representations and a more detailed discussion on RSA can be found in Appx. E.
<sup>4</sup>Human similarity judgments were collected by either asking participants to arrange natural object images from different categories on a computer screen [66, 10, 41] or in the form of triplet odd-one-out choices [31].
Table 4: The gLocal transform yields both a high degree of alignment with datasets of human similarity judgments and good performance on FS/AD tasks. Performance on human similarity datasets is measured as odd-one-out accuracy on a held-out test set for THINGS or Spearman's  $\rho$  for the other three datasets, using either the original representations, the naively transformed representations, or representations transformed via the gLocal transform. For 5-shot FS and AD, we report the average performance across all tasks in Tab. {1, 2, 3}.
Dataset \ Transform	CLIP-RN50 (WIT)	CLIP-ViT-L/14 (WIT)	CLIP-ViT-L/14 (LAION-2B)
original	naive	gLocal	original	naive	gLocal	original	naive	gLocal
Human similarity datasets:
Hebart et al. [31]	52.78%	59.92%	59.89%	46.71%	60.13%	60.05%	47.50%	60.47%	60.38%
King et al. [41]	0.386	0.650	0.645	0.355	0.638	0.637	0.292	0.620	0.613
Cichy et al. [10]	0.557	0.721	0.716	0.363	0.732	0.732	0.395	0.735	0.718
Peterson et al. [65, 66]	0.364	0.701	0.705	0.260	0.688	0.688	0.314	0.689	0.660
Downstream tasks:
5-shot FS (avg.)	53.05%	41.20%	55.01%	67.71%	47.62%	69.91%	71.10%	50.71%	72.27%
Anomaly detection (avg.)	89.65%	90.75%	91.52%	90.16%	92.79%	95.19%	94.79%	93.09%	96.65%
Although neural networks achieve near-human-level performance on a variety of computer vision tasks, they may not optimally capture global object similarities. By contrast, humans represent
concepts using rich semantic features including superordinate categories and other global constraints for performing object similarity judgments [66, 41, 10, 31, 60]. These representations may contribute to humans' strong generalization capabilities [21, 24]. Here, we investigated the impact of aligning neural network representations with human similarity judgments on different FS and AD tasks.
We find that naively aligning neural network representations, without regularizing the learned transformations to preserve structure in the original representation space, can impair downstream task performance. However, our gLocal transform, which combines an alignment loss that optimizes for representational alignment with a local loss that preserves the nearest neighbor structure from the original representation, can substantially improve downstream task performance while increasing representational alignment. The transformed representation space transfers surprisingly well across different human similarity judgment datasets and achieves almost equally strong alignment as the
Figure 5: RSMs for human behavior and CLIP ViT-L/14 [WIT; 70] for four different human similarity judgment datasets [31, 66, 10, 41]. We contrast RSMs obtained from the network's original representations (second column), the naively aligned representations [61] (third column), and the representations  $+$  gLocal transform (rightmost column) against RDMs directly constructed from human similarity judgments (leftmost column). Yellower colors indicate greater similarity; bluer colors indicate greater dissimilarity.
naive approach [61], indicating that it captures human notions of object similarity. In addition, it substantially improves downstream task performance compared to both original and naively aligned representations across a variety of FS and AD tasks. The gLocal transform yields state-of-the-art (SOTA) performance on the CIFAR-100 coarse AD task, and approaches SOTA on other AD benchmarks.
Ablations. In addition, we show that the gLocal transform can readily be used in combination with other approaches that are specifically designed for few-shot learning with pretrained representations of image/text models — such as Tip-Adapter [93]. We observe considerable improvements compared to using Tip-Adapter without applying the gLocal transform (see Appx. D.1). This indicates that
methods designed to improve the transferability of pretrained representations without incorporating knowledge about human global similarity structure can additionally benefit from the gLocal transform. Using a human-adversarial triplet dataset, where each odd-one-out choice is determined to be an object that is different from the human choice, for optimizing the gLocal transform either yields no change in a model's representation space and thus no difference in downstream task performance or changes a model's representation space in a way that substantially deteriorates its downstream task performance (see Appx. D.4). These observations corroborate our findings and suggest that the benefits of the gLocal transform most likely stem from including an inductive bias about global object similarity.
Limitations. Our work has some limitations. First, as we show in Appx. D, the gLocal transform fails to consistently improve downstream task performance on ImageNet. We conjecture that the gLocal transform can succeed only if representations capture the concepts by which human representations are organized, and ImageNet representations may not. Second, human similarity judgments are more expensive to acquire than other forms of supervision, and there may be important human concepts that are captured neither by the 1854 images we use for alignment nor by the pretrained representations.
Conclusion. Our results imply that even with hundreds of millions of image/text pairs, image/text contrastive learning does not learn a representation space with human-like global organization. However, since the gLocal transform successfully aligns contrastive representations' global structure using only a small number of images, these representations do seem to have a pre-existing representation of the concepts by which human representations are globally organized. Why does this happen? One possibility is that image/text pairs do not provide adequate global supervision, and thus contrastive representation learning is (near-)optimal given the data. Alternatively, contrastive learning may not incorporate signals that exist in the data into the learned representation because it imposes only local constraints. Previous work has shown that t-SNE and UMAP visualizations reflect global structure only if they are carefully initialized [42]. Given the similarity between contrastive representation learning and t-SNE/UMAP [15] and the known sensitivity of contrastive representations to initialization [52], it is plausible contrastive representations also inherit their global structure from their initialization. Although our gLocal transform provides a way to perform post-hoc alignment of representations from image/text models using human similarity judgments, there may be alternative initialization strategies or objectives that can provide the same benefits during training, using only image/text pairs.
LM, LL, JD, and RV acknowledge funding from the German Federal Ministry of Education and Research (BMBF) for the grants BIFOLD22B and BIFOLD23B. LM acknowledges support through the Google Research Collabs Programme. We thank Pieter-Jan Kindermans for helpful comments on an earlier version of the manuscript.
[1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[2] Maria Attarian, Brett D Roads, and Michael C Mozer. Transforming neural network visual representations to predict human judgments of similarity. In NeurIPS 2020 Workshop SVRHM, 2020.
[3] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
[4] Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. CoRR, abs/2002.10445, 2020.
[5] Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? IEEE transactions on visualization and computer graphics, 24 (1):152-162, 2017.
[6] Jeffrey S Bowers, Gaurav Malhotra, Marin Dujmović, Milton Llera Montero, Christian Tsvetkov, Valerio Biscione, Guillermo Puebla, Federico Adolfi, John E Hummel, Rachel F Heaton, et al.
[91] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.
[92] Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619–8624, 2014. doi: 10.1073/pnas.1403112111.
[93] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In Shai Avidan, Gabriel Brostow, Moustapha Cisse, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision – ECCV 2022, pp. 493–510, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19833-5.
A.1 Model features
We extract penultimate layer features of four different ImageNet-models — AlexNet [47], VGG-16 [85], ResNet-18, and ResNet-50 [28] — and image encoder features of four different image/text models — CLIP RN50 and CLIP ViT-L/14 trained on WIT [70]; CLIP ViT-L/14 trained on Laion-400M [81] and Laion-2B [82] respectively. For extracting the model features, we use the Python library thingsvision [59].
A.2 gLocal probing
To optimize the gLocal transforms, we use standard SGD with momentum and perform cross-validation according to the procedure proposed in Muttenthaler et al. [61]. For finding the optimal gLocal transform, we perform an extensive grid search over four different hyperparameter values — the learning rate,  $\eta$ , the strength of the regularization term  $\lambda$ , the global-local trade-off parameter  $\alpha$ , and the temperature parameter,  $\tau$ , used in the softmax expression for the local contrastive loss term (see Eq. 5). Specifically, we perform an extensive grid search over the Cartesian product of the following sets of hyperparameters:
-  $\eta \in \{0.0001, 0.001, 0.01, 0.1\}$ ,
-  $\lambda \in \{0.01, 0.1, 1.0, 10.0\}$ ,
-  $\alpha \in \{0.05, 0.1, 0.25, 0.5, 1.0\}$ ,
-  $\tau \in \{0.1, 0.25, 0.5, 1.0\}$ .
We use the same  $\eta$  and  $\lambda$  grids for global probing. We use PyTorch [63] for implementing the probes and PyTorch lightning to accelerate training. We choose the gLocal transform that achieves the lowest alignment loss (see alignment term in Eq. 6). Among the values in the above grid, we find that a combination of  $(\alpha = 0.1, \lambda = 0.1, \eta = 0.001)$  yields the lowest alignment loss/highest probing odd-one-out accuracy for both CLIP RN50 (WIT) and CLIP ViT-L/14 (WIT) (see Fig. 6). A combination of  $(\alpha = 0.25, \lambda = 0.1, \eta = 0.001)$  gives the second lowest alignment loss/highest probing odd-one-out accuracy for CLIP RN50 (WIT) and CLIP ViT-L/14 (WIT).
(a) CLIP RN50 (WIT)
(b) CLIP ViT-L/14 (WIT)
Figure 6: Among all hyperparameter combinations considered in our grid search, a combination of  $(\alpha = 0.1, \lambda = 0.1, \eta = 0.001)$  for Eq. 6 in §3 yields the best odd-one-out accuracy on a held-out test set for both CLIP RN50 and CLIP ViT-L/14 (WIT).
For each  $(\alpha, \lambda)$  combination we select that combination with the best probing odd-one-out accuracy on a held-out test among the set of possible learning rate,  $\eta$ , and temperature value,  $\tau$ , combinations determined by the above grid. We observe that  $\eta = 0.001$  generally gives the best results across the different  $(\alpha, \lambda)$  combinations, whereas performance is fairly insensitive to the value of  $\tau$ . Since neither  $(\alpha = 0.1, \lambda = 0.1)$  nor  $(\alpha = 0.25, \lambda = 0.1)$  are values at the edges of the hyperparameter
grid, it is plausible to assume that both the contrastive local loss and the regularization term in Eq. 6 in §3 are necessary to obtain a transformation that leads to a best-of-both-worlds representation.