SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although our goal has been to find a transform that induces both increased representational alignment and improved downstream task performance, we considered  $\alpha = 1.0$  to examine whether downstream task performance can potentially be improved by excluding the alignment loss. Note that  $\alpha = 1.0$  causes the optimization process to ignore the alignment loss. Unsurprisingly we did not find that to be the case. We remark that minimizing both the local contrastive loss and the regularization preserves the local similarity structure of the original representation space but does not inject any additional information into the representations. Moreover, it is non-trivial to choose a transform that works well across all downstream tasks without including the alignment loss. Therefore, we exclude  $\alpha = 1.0$  in Fig. 6.
Compute. We used a compute time of approximately 400 hours on a single Nvidia A100 GPU with 40GB VRAM for all linear probing experiments — including the hyperparameter sweep. The computations were performed on a standard, large-scale academic SLURM cluster.
A.3 Few-shot learning
Here, we use  $n_s$ -fold cross-validation for finding the optimal  $\ell_2$ -regularization parameter, where  $n_s$  refers to the number of shots per class. We select the parameter from the following set of values,  $\{1\mathrm{e} + 6, 1, 1\mathrm{e} + 5, 1\mathrm{e} + 4, 1\mathrm{e} + 3, 1, 1\mathrm{e} + 2, 1\mathrm{e} + 1, 1, 1\mathrm{e} - 1, 1\mathrm{e} - 2, 1\mathrm{e} - 3, 1\mathrm{e} - 4\}$ . We use the scikit-learn [64] implementation of (multinomial) logistic regression and refit the regression after selecting the optimal regularization parameter.
Compute. We used a compute time of approximately 5600 CPU-hours of  $2.90\mathrm{GHz}$  Intel Xeon Gold 6326 CPUs for all few-shot experiments. Computations were performed on a standard, large-scale academic SLURM cluster.
A.4 Anomaly Detection
In this section, we outline our anomaly detection experimental setting in more detail. In the anomaly detection settings that we consider in our analyses normal/anomaly classes are determined via the original classes in the data. Here, each of the original classes is once selected as a normal class with the remaining classes being anomalous and, vice versa, each class in the data is once selected as an anomalous class with the other classes being normal. After embedding the training images from either the normal or the anomalous class in a model's representation space, at inference time a model must classify whether a new image belongs to the normal data or whether it deviates from it and is thus considered an anomaly. For each example in the test set, a model yields an anomaly score where higher scores indicate more probability of an example being anomalous. Using the binary anomaly labels and the anomaly scores for each of the examples, we can then compute the area-under-the-receiver-operating-characteristic-curve (AUROC) to quantify the performance of the model.
One-vs-rest. Given a dataset (e.g., CIFAR-10) with  $C$  classes, one class (e.g., "airplane") is chosen to be the normal class and the remaining  $C - 1$  classes of the dataset are considered anomalies. Each of the  $C$  classes is once selected as a normal class and the AUROC is averaged across the classes.
Leave-one-out (LOO). In contrast to the "one-vs-rest" setting, in LOO we define one class of the dataset as an anomaly and the remaining classes as normal. Similarly to the "one-vs-rest" setting, this results in  $C$  evaluations for a dataset with  $C$  classes.
In both "one-vs-rest" and LOO AD settings, we evaluate model representations in the following way: First, we compute the representations  $X_{\text{train}}$  of the normal samples in the train set. Then, we compute the representations of all test set examples  $X_{\text{test}}$ . For each test set representation, we compute the cosine similarity to all normal train set representations,  $X_{\text{train}}$ , and select the  $k$  nearest neighbor samples that have the highest cosine similarity.
The anomaly score of a test set representation is then defined as the average cosine distance to the  $k$  nearest train representations.  $k$  is a hyperparameter that determines the number of nearest neighbors over which the anomaly score is computed. We choose  $k = 5$  for our experiments but show that performance is fairly insensitive to the value of  $k$  (see Tab. 12).
Compute. For all AD experiments, we used a compute time of approximately 20 hours on a single Nvidia A100 GPU with 40GB VRAM. Computations were performed on a standard, large-scale academic SLURM cluster.
B What changes in the global structure of the representations after alignment?
In this section, we attempt to build some intuitions for how the global structure of the representations changes after alignment. To do so, we analyze the movements of the representations of items and superordinate categories in the THINGS dataset. Specifically, we compute cosine differences between the CLIP-ViT-L/14 representations of each pair of items in THINGS and then compute how these distances change under the transforms.
We show the pairs of items that change the most in distance in Table 5. Items that are semantically related, like "curry" and "scrambled egg", tend to move closer together, and therefore have transformed distances that are smaller than their original distance. By contrast, items like "handcuff" and "stethoscope", which are semantically unrelated but perhaps have some slight visual similarity, tend to move farther apart. The distance changes under the gLocal transforms are correlated with, though generally less varied than, those under the naively-aligned transform.
To more broadly analyze the change in global structure, we then look at how distances between pairs of items change within and across superordinate categories (the top-down categories from THINGS). We show the results in Fig. 7. Under the naively aligned transform, the items within each superordinate category tend to move slightly closer together — the diagonal is slightly blue — while the items from different categories tend to move substantially farther apart — the off-diagonal is mostly red. That is, the representations are broadly moving in a way that reflects the overall human semantic organization of the categories.
There are a few notable standouts: the categories of drink, food, plant, and animal change particularly much, and in particularly interesting ways. These categories each move much farther relative to all other categories (such as tool or musical instrument) than those other categories move relative to each other. This perhaps reflects the particular semantic salience of food, drink, plants, and animals from a human perspective. Furthermore, food and drink are one of the few pairs of superordinate categories between which distances actually decrease after the transform, presumably reflecting the strong semantic ties between these categories. Similarly, animals move less far from plants than from any other category, perhaps reflecting the fact that the animate/inanimate distinction is one of the strongest features in human semantic representations [73].
Under the gLocal transform, the pattern of changes is strongly correlated with the naively aligned transform  $(r = 0.96, p \leq 10^{-16})$ . However, in keeping with the regularization, the magnitude of the changes varies less.
Table 5: Distances between pairs of individual items from THINGS, ranked by the relative change in cosine distance from before to after naive alignment (normalized by original distance). The top items move much closer together under naive alignment, while the bottom ones move much farther apart. (All results are from CLIP-ViT-L/14.)
Item 1	Item 2	original dist.	naively aligned dist.	gLocal dist.
curry	scrambled egg	0.303120	0.005276	0.401019
otter	warthog	0.305242	0.005530	0.382150
parfait	spaghetti	0.457553	0.009115	0.540346
otter	rhinoceros	0.327497	0.006530	0.456641
:
stethoscope	wheat	0.263908	1.284535	0.935891
grass	wallet	0.277866	1.347424	1.056572
cat	traffic light	0.285151	1.378671	0.981944
handcuff	sugar cube	0.272936	1.308337	0.904380
Figure 7: How does the global structure of the representations change after alignment? Here, we analyze the movements of the representations of pairs of items from different superordinate categories from the THINGS dataset. The squares on the diagonal indicate the change in distance between items within a superordinate category, while the squares off the diagonal indicate changes between pairs of items from the corresponding pair of superordinate categories. A red color indicates the items from the categories move farther apart from each other after alignment, blue indicates moving closer together. Generally, items within a superordinate category move slightly closer together under naive alignment, while those in different categories move farther apart. A similar overall pattern is reflected in both the naively-aligned transform (left) and gLocal (right) ones, though under gLocal alignment there is a greater overall spreading of the representations. (All results are from CLIP-ViT-L/14.)
C Visualization of neighboring images
To provide further insight into the difference between the effects of the naive and gLocal transforms, in Fig. 8 we visualize the neighbors of nine anchor images. In order to show a diverse set of images, we pick the nearest neighbors in the CLIP ViT-L/14 (WIT) embedding space subject to the constraint that each neighbor comes from a different class from the original images and the nearer neighbors. In accordance with the results in §4.2, we find that the neighbors in the untransformed and gLocal spaces are generally similar, whereas neighbors in the naive representation space are frequently different. The naive transform appears to discard all non-semantic properties of images, whereas the untransformed and gLocal representation spaces are sensitive to pose, color, and numerosity. In cases where the closest neighbor differs between the naive and gLocal representations (third and fourth row), the neighbors in the gLocal representation are arguably better matches to the anchor.
Figure 8: Comparison of neighbors in the ImageNet validation set for representations with different transforms. We visualize the 10 closest images subject to the constraint that each comes from a unique class. The anchor images are shown in the leftmost column. The three rows corresponding to each anchor image show their nearest neighbors in the untransformed, gLocal transformed, and naively transformed representations.
D Additional results on downstream tasks
In this section, we provide additional few-shot learning and anomaly detection results for all ImageNet and image/text models that we considered in our analyses (see §A.1). We start this section by demonstrating a strong relationship between the performances of the different downstream tasks.
Figure 9: Here, we show anomaly detection AUROC averaged across all tasks reported in Tab. {2, 3} as a function of the average 5-shot classification performance for all ImageNet and CLIP models (see §A.1), using either the original representations or the representations transformed via the naive or gLocal transformations.
Downstream task relationship. We observe a strong positive relationship  $(r = 0.8524, p \leq 10^{-6})$  between the average few-shot learning and the average anomaly detection performance for all ImageNet and image/text models that we considered in our analyses (see Fig. 9). This observation holds for both the original representation space and the representations transformed via the naive or gLocal transformations. This indicates that both downstream tasks require similar representations for similarly strong performance.
D.1 Few-shot learning
In the following section, we show additional few-shot learning results. Specifically, we report 5-shot performance of ImageNet models and show few-shot results as a function of the number of samples used during fitting.
Results for ImageNet models. In Tab. 6 we report additional 5-shot results for ImageNet models. The gLocal transforms improve few-shot accuracy on Entity-{13,30} from BREEDS but the impact on few-shot performance is either inconsistent or negative for CIFAR-100 coarse, CIFAR-100, SUN397, and DTD of which the latter three are more fine-grained datasets than the other three.
Table 6: 5-shot FS results using the original or transformed representations.
Model \ Transform	Entity-13	Entity-30	CIFAR100-Coarse	CIFAR100	SUN397	DTD
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
AlexNet	37.49	42.13	27.73	26.93	32.33	31.44	28.70	23.29	26.43	19.12	37.05	30.54
ResNet-18	56.71	58.19	52.94	51.27	42.97	41.84	38.12	35.92	37.26	34.52	48.53	45.90
ResNet-50	49.27	59.18†	50.16	54.11†	50.99†	49.48	47.99†	44.75	47.17†	44.26	54.02†	51.49
VGG-16	51.78	56.49	44.69	45.97	39.08	37.03	34.54	28.08	37.12	29.89	45.39	38.01
Results for Tip-Adapter [93]. To investigate whether the gLocal transforms improve downstream task performance of image/text models in few-shot learning settings in combination with techniques that were specifically designed for this task, we evaluate the CLIP models combined with Tip-Adapter [93]. Tip-Adapter is designed to produce effective few-show classifiers from pretrained image/text models by linearly combining the outputs of two modules:
(i) a zero-shot classifier, where the similarity of the input in embedding space with the embedded textual description of each class determines the output
(ii) a key-value cache model that computes its output by summing the one-hot labels of all stored few-shot data, weighted by the similarities of the associated image embeddings to the embedded input sample
For the evaluation of gLocal transforms, we transform both the text embedding and any image embedding involved in the inference part. The hyper-parameters that we use are  $\alpha = 1$  and  $\beta = 5.5$  which have been identified as optimal for ImageNet classification by Zhang et al. [93].
We observe that even in conjunction with a method specifically designed for few-shot learning, gLocal transforms improve few-shot accuracy in most scenarios (see Tab. 7), in particular on Entity-{13,30} from BREEDS. Furthermore, the performance increase appears to be stronger for the CLIP models trained on the OpenAI WIT datasets compared to LAION (see Fig. 10). For the smaller of the two LAION datasets [LAION-400M; 81], the effect of the gLocal transform on CLIP with Tip-Adapter is negligible compared to its effect on models trained on the larger LAION-2B [82] or OpenAI's WIT dataset [70].
Table 7: 5-shot FS results using the original or transformed representations in combination with Tip-Adapter.
Model \ Transform	Entity-13	Entity-30	CIFAR100-Coarse	CIFAR100	SUN397	DTD
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
CLIP-RN50 (WIT)	66.29	69.95	59.24	61.76	31.41	33.69	38.53	44.82	53.42	55.75	47.98	47.91
CLIP-ViT-L/14 (WIT)	68.17	77.15†	67.50	73.52†	55.59	66.73	45.79	64.51	64.77	69.80	57.96	59.72
CLIP-ViT-L/14 (LAION-400M)	72.87	72.04	69.60	67.21	75.07	71.38	69.65	68.27	71.24†	70.30	68.96†	65.69
CLIP-ViT-L/14 (LAION-2B)	73.19	75.54	70.50	71.79	77.45†	76.81	71.87	73.33†	70.24	71.17	66.10	64.43
5-Shot Accuracy: original + Tip-Adapter
Figure 10: 5-shot FS results using the original or transformed representations in combination with Tip-Adapter.
Effect of transforms for different numbers of training samples. When varying the number of training samples for the few-shot experiments described in §4.3 we observe consistent improvements of the gLocal transforms across shots. Excluding the high-variance setting of 2-shot learning, we either find stable improvements in accuracy for image/text models, or a downward trend for ImageNet models on some tasks. This corroborates our findings from §4.3. Results appear to be robust to changes in the training set size, in particular for the CLIP models. Yet, we observe the most substantial benefits in low data regimes. See Fig. 11 for more details.
Figure 11: Change in average accuracy for different numbers of training samples per super-class (top row) or (sub-)class (bottom row) used for few-shot learning. Error bands depict  $95\%$  Confidence Intervals (CIs), computed over 5 different runs.
Few-shot learning on ImageNet-1K. In addition to the few-shot learning results that we presented in §4.3, here we report few-shot classification results on ImageNet-1K [17]. For both the original and gLocal-transformed CLIP representations, we trained a linear classifier with different numbers of shots,  $k$ , i.e., the number of training examples per class. We find that the gLocal-transformed representation achieves better ImageNet validation set accuracy compared to the original representation for both CLIP ViT-L/14 (WIT) and CLIP ViT-L/14 (LAION-2B) in small training data settings where  $k > 5$ . We don't observe a significant difference between the two representation spaces for CLIP RN50 (WIT) and CLIP ViT-L/14 (LAION-400 M) which generally perform worse than the other two models across all training settings (see Fig. 12).
Number of shots
Number of shots
Figure 12: ImageNet validation set accuracy as a function of the number of training examples per class used for training a linear classifier on the original and the gLocal-transformed representations. For all CLIP models, we report the mean validating set accuracy across three different seeds.
D.2 Anomaly detection
In addition to the results of image/text models for the "one-vs-rest" anomaly detection (AD) setting that we presented in §4.4, here we show "one-vs-rest" AD performance of ImageNet models. While the gLocal transform considerably improves AD performance over the untransformed representations
across the different datasets for image/text models (see Tab. {2, 3}, for ImageNet models we do not observe any improvements over the original representation space (see Tab. {8, 9}).
Furthermore, we present results for the non-standard Leave-one-out (LOO) setting and for "CIFAR10-vs-CIFAR100" for all image/text and Imagenet models that we considered. In the "CIFAR10-vs-CIFAR100" AD task, all data of CIFAR10 is considered to be the normal class, and each sample from the CIFAR100 dataset is considered an anomaly. Similarly to the previously reported AD results, the gLocal transform substantially improves AD performance compared to the original representations for image/text models across all datasets but does not appear to have a considerable impact on the performance of ImageNet models (see Tab. {10, 11}).
Table 8: One-vs-rest nearest neighbor based AD results; with and without transformation. ImageNet30 results for ImageNet models are omitted due to overlap with train data.
Model \ Transform	CIFAR10	CIFAR100	CIFAR100-Coarse	ImageNet30	DTD
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
AlexNet	89.43	85.63	92.34	88.53	87.53	82.75	-	-	86.33	79.51
ResNet-18	92.19	84.96	95.06	89.71	92.16	84.89	-	-	94.38	89.55
ResNet-50	94.74	89.74	96.46	93.76	94.3	91.2	-	-	94.47	91.52
VGG-16	90.33	88.0	93.56	91.97	89.78	88.16	-	-	91.15	85.5
Table 9: One-vs-rest AD with a class distribution shift between train and test sets; with and without transformation.
Model \ Transform	Entity-13	Entity-30	Living-17	Nonliving-26	Cifar100-shift
original	gLocal	original	gLocal	original	gLocal	original	gLocal	original	gLocal
AlexNet	83.84	81.45	85.38	83.71	87.04	79.09	81.45	78.84	80.21	76.37
ResNet-18	91.84	88.01	93.18	90.32	96.82	89.11	90.97	88.82	81.83	76.84
ResNet-50	89.59	87.24	93.51	89.63	98.27	89.87	90.61	88.65	84.73	84.45
VGG-16	89.78	88.87	90.7	91.56	94.72	89.98	89.78	89.32	83.42	81.91
Table 10: LOO nearest neighbor based AD results and "CIFAR-10 vs. CIFAR-100" AD results; with and without using the gLocal transform.
Model \ Transform	CIFAR10	CIFAR100	Cifar100-Coarse	Cifar10 vs Cifar100
original	gLocal	original	gLocal	original	gLocal	original	gLocal
AlexNet	67.64	62.7	58.94	55.83	63.33	59.37	69.87	68.01
ResNet-18	72.35	63.47	64.86	58.53	71.38	62.89	81.42	73.71
ResNet-50	76.62	68.27	66.91	61.26	74.78	68.63	84.27	80.17
VGG-16	68.45	64.31	59.92	57.89	65.81	64.28	73.55	74.7
CLIP-RN50	70.32	72.46	59.91	61.43	65.63	68.07	72.55	76.78
CLIP-ViT-L/14 (WIT)	84.91	91.3	67.08	72.24	73.48	80.7	85.24	92.78
CLIP-ViT-L/14 (LAION-400M)	93.0	92.37	74.05	74.15	82.13	82.88	94.44	94.68
CLIP-ViT-L/14 (LAION-2B)	93.55	95.23	76.88	77.46	84.67	85.78	93.18	95.26
Table 11: LOO nearest neighbor based AD results; with and without using the gLocal transform.
Model \ Transform	Entity-13	Entity-30	Living-17	Non-Living-26
original	gLocal	original	gLocal	original	gLocal	original	gLocal
AlexNet	62.05	59.73	58.2	56.23	61.02	56.07	56.27	54.93
ResNet-18	74.26	68.88	70.6	64.7	76.48	70.79	66.61	63.82
ResNet-50	72.46	73.67	73.37	72.49	83.5	82.45	68.16	68.29
VGG-16	70.26	68.03	66.38	63.24	73.31	65.99	64.87	62.95
CLIP-RN50 (WIT)	69.99	71.12	63.49	63.72	72.52	70.04	62.55	63.4
CLIP-ViT-L/14 (WIT)	71.68	74.88	68.96	70.58	82.9	82.72	63.94	67.33
CLIP-ViT-L/14 (LAION-400M)	69.98	71.44	65.68	66.26	77.35	77.4	65.19	66.27
CLIP-ViT-L/14 (LAION-2B)	70.77	72.49	66.55	67.68	80.07	80.09	65.43	67.99
The nearest neighbor hyperparameter  $k$ . From the results reported in Tab. 12 it can be inferred that the nearest neighbor hyperparameter  $k$  does not have a considerable impact on AD task performance across the different datasets. Here, we report the impact of  $k$  on the performance of CLIP ViT-L/14 (WIT) but the observation holds across all image/text models.
Table 12: Nearest Neighbor AD performance of CLIP ViT-L/14 for different  $k$  .
kDataset \ Transform	2	5	10	20
original	gLocal	original	gLocal	original	gLocal	original	gLocal
CIFAR-10	95.37	98.16	95.14	98.16	94.86	98.11	94.50	98.03
CIFAR-100	91.90	97.22	91.41	97.19	90.93	97.08	90.39	96.92
CIFAR-100-coarse	89.28	95.9	88.50	95.83	87.73	95.66	86.81	95.41
CIFAR-100-shift	74.48	87.06	73.69	87.11	73.00	87.01	72.29	86.85
ImageNet30	98.95	99.74	98.91	99.75	98.85	99.77	98.78	99.77
Entity-13	88.37	92.5	88.54	93.23	88.45	93.63	88.28	93.95
Entity-30	91.26	95.11	91.31	95.53	91.22	95.74	91.03	95.91
D.3 Global versus gLocal transform
Aside from the AD performance of CLIP RN50 and CLIP ViT-L/14 (WIT), the gLocal transform leads to more substantial improvements on downstream tasks than the global transform. In Tab 13, we report the average few-shot and anomaly detection performances using the global or gLocal transforms. For FS, we average performance over all results reported in Tab. 1, and for AD we average performance across all results reported in Tab. {2, 3, 6}.
Table 13: Comparison of the average downstream task performance global and gLocal transforms.
Model \ Transform	AD	FS
global	gLocal	global	gLocal
AlexNet	81.16	81.76	28.43	28.91
ResNet-18	84.62	86.91	43.42	44.61
ResNet-50	93.19	89.56	51.32	50.55
VGG-16	87.32	88.36	37.23	39.25
CLIP-RN50 (WIT)	92.12	91.52	54.63	55.00
CLIP-ViT-L/14 (WIT)	95.49	95.19	69.92	69.91
CLIP-ViT-L/14 (LAION-400M)	95.72	96.08	68.80	69.40
CLIP-ViT-L/14 (LAION-2B)	96.33	96.65	72.00	72.27
D.4 Human-adversarial experiments