SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthew Wallingford*† Vivek Ramanujan*† Alex Fang† Aditya Kusupati† Roozbeh Mottaghi† Aniruddha Kembhavi Ludwig Schmidt†† Ali Farhadi† †University of Washington PRIOR, Allen Institute for AI ‡LAION {mcw244,ramany}@cs.washington.edu
We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks given few or no labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model to recall and conditions its parameters on relevant data seen throughout pretraining, thereby priming it for the test distribution. Neural Priming can be performed at inference, even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a  $2.45\%$  improvement in accuracy on ImageNet and  $3.81\%$  accuracy improvement on average across standard transfer learning benchmarks. Further, using Neural Priming at inference to adapt to distribution shift, we see a  $1.41\%$  accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the challenge of limited labeled data and changing distributions. Code is available at https://github.com/RAIVNLab/neural-priming.
Humans have a vast store of prior experience which we draw on to flexibly perform a diverse range of tasks [20, 5, 4, 12]. While engaging in an activity, we naturally retrieve relevant information or schema in a cognitive phenomena known as Priming [34]. This process ensures that necessary knowledge is readily accessible in memory, leading to enhanced performance for the task at hand [43]. Pre-trained, general-purpose models such as CLIP [40] and ALIGNN [23] have extensive prior knowledge learned from large-scale, diverse datasets. These datasets seek to capture all natural variation in real data within their distribution. Can these models also benefit from something like priming? We observe that models trained even on the largest of such datasets often substantially improve in performance when fine-tuned on task-specific data. This begs the question of what the model learns from fine-tuning on the target dataset, if it already trained on many similar examples during pre-training.
We speculate that the effect of fine-tuning a pre-trained model on task-specific data is similar to that of priming. Given the sheer size and diversity of the pre-training dataset it becomes challenging for the model to find a consistent solution that is optimal for all subsets of the data. This becomes particularly evident for open-vocabulary models such as CLIP, where multiple natural language descriptions can correspond to a single image, highlighting the challenge of accommodating diverse interpretations. We hypothesize that training on the downstream dataset re-aligns the model to the specific objective.
With this in consideration, we propose Neural Priming. Specifically, Neural Priming recalls a subset of the pre-training data similar to the target distribution, re-aligns the natural language
Figure 1: A diagram of Neural Priming, our proposed method. Neural Priming is a framework for leveraging an open-vocabulary model's own pre-training data to improve performance on downstream tasks. Neural Priming encompasses two processes: 1. Collecting a priming pool of relevant examples from the pre-training set to prime with and 2. using these examples to attune our model to a given task. We show performance improvements across a wide range of transfer learning and robustness benchmarks.
descriptions to the downstream task, and quickly adapts the model to the subset. We perform extensive experiments on 7 transfer learning and 4 distribution shift datasets to validate our method. We use the OpenCLIP [40, 52] ViT [8] set of models pre-trained on the LAION-2B and 400M [44]. We find Neural Priming leads to significant accuracy improvements, particularly when labeled data is scarce and in specialized domains. Concretely, Neural Priming improves accuracy by  $2.45\%$  on ImageNet and  $4.25\%$  on average across the other 6 datasets over the base CLIP model. In the few-shot setting, Neural Priming improves accuracy by  $3.81\%$  on average over recent methods on standard transfer learning benchmarks. We show Neural Priming is efficient and can be performed on-the-fly. For datasets containing more than 2 billion images, we can prime our model to ImageNet in less than 2 minutes with a single commercial GPU.
Neural Priming is flexible and can be used with variable degrees of information about the downstream distribution. When the model has language-only task descriptions, our approach can efficiently retrieve a priming pool of relevant examples from the pre-training set and attune the model to this data. At inference time, given a set of test images to classify, Neural Priming is able to use test examples to adapt the priming pool to distribution shifts. When we have access to training examples in the few-shot setting, Neural Priming can filter the priming pool to align with the training distribution.
- We introduce Neural Priming, a novel method that leverages retrieval from the pre-training dataset for efficient and accurate adaptation of large, pre-trained models to downstream tasks.
- Neural Priming achieves state-of-the-art zero-shot and few-shot accuracy on the standard transfer learning datasets – up to  $4.25\%$  and  $3.81\%$  improvements respectively over baselines (Section 4.2).
- Neural Priming also enables transductive learning and improves performance on standard distribution shift datasets by  $2.51\%$  on average, all without using any additional data (Section 4.3).
- Our approach generalizes to various architectures and pre-training datasets while being complementary to techniques [33, 39] that improve zero-shot performance of open-vocabulary models.
2.1 Open-Vocabulary Models and Zero-shot Inference
Open-vocabulary models have proven to be an effective approach for transfer learning. Such models enable training on vast amounts of web-scale images without the need for labor-intensive human
labeling by leveraging pre-existing natural language descriptions [40]. Open-vocabulary models have set state-of-the-art on ImageNet [7] as well other transfer learning benchmarks [53, 1, 41].
Open-vocabulary models offer additional capabilities beyond standard pre-trained models. They can perform zero-shot inference, where predictions are made without training on target data. Additionally, they are robust to distribution shifts [40], enable prompt-tuning methods [39, 33], and can be used for text-based retrieval [10]. Zero-shot can have different meanings in the literature. In the context of this paper, we consider zero-shot as the experimental setting in which the model receives no training examples drawn from the training distribution.
Prompt-tuning has emerged as a popular research direction in the domains of large language and open-vocabulary models. In the context of open-vocabulary models, prompt-tuning can involve modifying the textual prompts or queries used during the training or inference of the model to improve its understanding of visual content or achieve specific goals. In the original CLIP paper, Radford et al. [40] design hand-crafted prompt templates for ImageNet and other transfer learning datasets and show that this leads to substantial accuracy improvements. More recently, other work [33, 55] has used machine learning approaches to learn the prompts rather than hand-crafting them.
2.2 Distribution Shifts
Robustness to distribution shift is a key property of good machine learning models as it represents a notion of reliability. In particular, studies on natural distribution shifts, including ImageNet-V2 [41], ImageNet-Sketch [51], ImageNet-R [18], and ImageNet-A [19], find that models have a consistent performance drop when exposed to a distribution at inference time not seen during train time [49]. In order to focus on robustness and eliminate the confounder of better models being generally better, this performance gap is measured through effective robustness, which is the robustness improvement over ImageNet trained models. Prior work has shown that the performance of models on in distribution and out of distribution is highly correlated across many algorithmic training interventions, except for cases where training on larger and more diverse datasets increases robustness [35].
The most significant recent improvement in effective robustness [41] is the introduction of open-vocabulary models. At its time of release, CLIP [40] achieved unprecedented effective robustness on a variety of distribution shifts. Studies have suggested that these models achieve high effective robustness through their data distribution [9], a result of training on large amounts of web-scraped data. However, these models are still worse at downstream tasks than models fine-tuned on in-distribution data. Moreover, fine-tuning on downstream data causes robustness on other data distributions to deteriorate [40, 52]. Many mitigation methods have been proposed to such as Wise-FT, FLYP, LP-FT, and model surgery [52, 15, 28, 29]. Our paper differs from these methods in goal: whereas they seek to keep model robustness while gaining the benefits of fine-tuning on task-specific data, we seek the benefits of fine-tuning while not collecting any in-distribution data. Hence these methods are complementary to Neural Priming, and we employ Wise-FT in our model attunement procedure.
2.3 Transductive Learning
Transductive learning [13, 6] focuses on leveraging unlabeled data during inference. It differs from traditional supervised learning, which solely relies on labeled data at train time. Related to transductive learning is test-time training [48, 14, 45]. Test-time training involves adapting and refining the model's predictions based on the specific testing examples encountered. Transductive learning differs from test-time training in that test-time training only considers one test sample at a time, whereas transductive aims to learn from the entire test set.
2.4 Few-Shot Learning
Few-shot learning research aims to address the challenge of learning from a limited number of labeled examples. In many real-world scenarios, acquiring large labeled datasets is impractical or costly. Older lines of work have focused on meta-training small models [47, 11, 37, 22] on small-scale datasets. More recently, the approach for few-shot learning has shifted towards training large, general-purpose models such as CLIP [40] and ALIGN [23] on web-scale datasets.
2.5 Retrieval-Augmented Models
In language, works have demonstrated the effectiveness of retrieval from text corpora or structured data for tasks such as question answering [3, 16, 25]. In general, these methods seek to recover facts either from a large corpus or knowledge graph, then use those to complete tasks. This differs from our scenario, where exact examples at inference time do not necessarily exist in the pre-training corpus. REACT [30] and SuS-X [50] are retrieval-augmented methods for open-vocabulary models which use search to fine-tune with relevant examples [30]. We differ from Liu et al. [30] in that they add a substantial number of new parameters whereas we do not. Additionally, our approach is significantly more efficient, both computationally and in terms of number of samples, enabling use at inference for additional improvement (Section 3.1.2). We differ from [50] in that their work uses semantic retrieval whereas Neural Priming leverages language for fast initial filtering and image search for accurate retrieval. Further, Neural Priming shows that models can improve by revisiting examples seen throughout pretraining whereas other works retrieve new examples from external datasets.
Neural Priming is the process of retrieving relevant information from the pre-training dataset and leveraging it for a specific task. We study it in the context of vision-language contrastive pre-training, so the form our task description takes is a set of class names,  $\mathcal{C}$ , already in natural language. A CLIP model [40] consists of a vision embedding model,  $V$ , and a language embedding model,  $L$ , each producing a vector representation in  $\mathbb{R}^d$ . The pre-training dataset,  $\mathcal{D}$ , consists of a large number of image-text pairs collected from the web. The text component can be noisy, potentially containing irrelevant or inaccurate information about the image content.
We break our method down into two main steps: 1. Collecting the priming pool, where we gather data from our pre-training dataset relevant to a particular task and 2. model attunement, where we leverage this data to improve our model.
3.1 Collecting the Priming Pool
3.1.1 Leveraging Natural Language Task Information
The goal of this step is to collect an initial pool of images relevant to the task at hand given the previously defined natural language description  $\mathcal{C}$ . For example, if our task is a set of dog breeds, ideally we would collect sets of images belonging to those breeds and label them accordingly. A simple way to prime is by using retrieval to gather relevant data points from our pre-training dataset. An existing method for language-based retrieval involves using the CLIP text embedding of a class description  $c \in \mathcal{C}$  for retrieval using semantic similarity scores on the pre-training set [2, 42]. However, with neural priming, prioritizing precision over recall is crucial, considering the size, diversity, and noise of the pre-training dataset. This form of semantic retrieval has a major downside: it is not clear where to threshold similarity scores to retrieve the most relevant images. Threshold too late and we allow unrelated images to be included in our pool. Further, this threshold is often specific to a category, making it infeasible to search at scale.
Our approach to language-based priming is to search for the existence of the class name,  $c \in \mathcal{C}$ , in the captions of our pre-training dataset to retrieve images relevant to a particular category. We organize these image-text pairs into separate categorical clusters  $\{B_c\}$  according to the class name  $c$  mentioned in their captions. This approach has a few advantages over semantic retrieval: 1. After setting up an inverted index search structure for text retrieval [26, 46], exact string matching is far faster than semantic retrieval, even when approximate nearest neighbor strategies are employed [24], 2. with exact string search, the category boundary is clear and therefore does not require per category tuning, 3. the retrieval results are overall qualitatively more relevant. Finally, to leverage the semantic understanding of our CLIP model, we filter the priming pool using CLIP similarity score. We do this by constructing a "zero-shot" CLIP classifier, as defined by Radford et al. [40], and removing examples from categorical clusters that do not align with their label according to the CLIP model.
Figure 2: A qualitative example of our approach for transductive image filtering. Given an initial priming pool, acquired through natural-language text search on the captions of our pre-training dataset (Section 3.1), we filter out irrelevant examples using images from our test set. (left) we show examples from the great owl categorical cluster of our priming pool before filtering, (center) we show an example image from the same category of ImageNet-V2, (right) example retrievals using image embedding similarity from the entire priming pool. The visual similarity of the retrievals are apparent, and they are generally from the appropriate categorical cluster. Doing this filtering results in a significantly more relevant priming pool.
3.1.2 Leveraging Image Information at Test Time
At inference time, the model can narrow the relevant priming pool even further by utilizing information about the test distribution. To do this, given an image  $x$  in our test set, we compute the cosine similarity using our CLIP image encoder  $V$ ,  $\cos(V(x), V(y))$  for every  $y \in P$ , our priming pool. We retrieve examples with the top- $k$  cosine similarity scores ( $k = 10$  in most of our experiments). We do this collectively for every image in the test set and collect the retrievals to form a filtered priming pool. If an example is retrieved twice, we de-duplicate them in the final priming pool. Since we do this for all images in the test set, we consider this the transductive setting to align with prior work [13, 6].
3.2 Attuning CLIP to the Priming Pool
The goal of this step is to modify our CLIP model to take advantage of the data in the priming pool  $P$ . We first construct the task-specific zero-shot linear head  $W_{z} \in \mathbb{R}^{d \times n}$  using the text encoder and the natural language names of each class, where  $d$  is the feature dimension and  $n$  is the number of classes. To get logits for a particular example,  $x$ , we compute  $W_{z} \cdot L(x)$ , so our prediction is  $\arg \max_{c} W_{z} \cdot L(x)$ .
To attune our CLIP model to the priming pool, we perform nearest-class mean (NCM) [32] on all retrieved examples to obtain a classification head from the priming pool. Namely for a given class  $c$ , we compute a centroid  $\tilde{y}_c = \frac{1}{|B_c|}\sum_{x\in B_c}L(x)$  and then normalize this centroid to produce a class embedding  $y_{c} = \tilde{y}_{c} / \| y_{c}\|$ . We define the collection of centroids as matrix  $W_{ft} = [y_c]_c\in \mathbb{R}^{d\times n}$ . To expand this to few-shot scenarios, we mix the labeled data into the corresponding categorical clusters before performing NCM. Finally, we ensemble  $W_{z}$  and  $W_{ft}$  using a mixing coefficient  $\alpha \in [0,1]$  as  $W_{\alpha} = (1 - \alpha)\cdot W_{ft} + \alpha \cdot W_{z}$ , which is our final classification head. We choose alpha according to a heuristic  $\alpha = e^{-|P| / \sigma^2}$  which can be derived from a Bayesian prior over the text features. We experiment with varying values of  $\sigma$  in Table 10. We also find that  $\alpha$  can be effectively chosen through cross-validation on a held-out portion of the retrieval set. Intuitively, if we do not have much data in our priming pool, we want it to influence our model less. We use NCM as the classifier as it has shown to be sample-efficient [47]. For comparison to a WISE-FT classifier [52] see Table 3.
Our key results include: 1. Priming improves performance over baselines in the few-shot setting by  $3.81\%$  on average across all datasets and  $2.4\%$  on ImageNet in the zero-shot setting 2. Priming in the
Table 1: Performance of Neural Priming and comparable methods in the zero-shot setting. Priming consistently improves top-1 accuracy across standard transfer learning data sets. Performance reported for the OpenCLIP ViT-B-16 model pretrained on LAION-2B.
ImageNet	Stanford Cars	FGVC Aircraft	Flowers102	Food101	Oxford Pets	SUN397
CLIP [40, 21]	68.30	87.40	25.86	71.65	86.58	90.21	67.35
Retrieval + Finetuning	70.28	87.95	26.22	72.15	86.63	90.35	68.01
VLM [33]	69.35	87.88	28.54	72.11	86.31	90.24	67.73
CuPL [39]	70.25	88.63	29.64	72.32	86.20	91.16	70.80
Priming (Ours)	70.75	89.30	33.03	79.81	86.66	91.87	71.21
Priming + CuPL (Ours)	71.38	90.23	36.00	80.04	86.86	91.85	72.35
transductive, or on-the-fly, setting further improves performance over baselines by  $2.51\%$  accuracy and  $1.09\%$  over standard Neural Priming 3. Priming is complementary to existing prompt-tuning methods. Our finding indicates that images in the priming set impart distinct information to the model compared to textual class descriptions. We include full details of hyperparameter choices and error bars included in the appendix.
4.1 Datasets and Architectures
We evaluate on standard transfer learning and distribution shift benchmarks. ImageNet [7] is a large-scale, general classification dataset that has been well-studied in both transfer learning and distribution shift. ImageNetV2 [41] is one of its natural distribution shift test sets, made by reproducing the original data collection procedure of ImageNet, but even modern large-scale pre-trained models have performance drops on it. ImageNet Sketch [51] and ImageNet-R [18] are natural distribution shifts created by assembling sketches and various renditions of the ImageNet classes. ImageNet-A [19] is a natural adversarial distribution shift of ImageNet, created by collecting images that are misclassified by ResNets. StanfordCars [27], FGVCAircraft [31], Flowers102 [36], and OxfordPets [38] are fine-grained classification datasets which require understanding subtle visual differences between classes and are commonly used for transfer learning benchmarks [17, 21, 40, 55]. SUN397 [53] is a large-scale scene recognition dataset with 397 scene categories.
We perform our experiments with OpenCLIP models [52] trained on LAION-2B and 400M [44]. We choose OpenCLIP because their pretrain datasets are publicly available, therefore we can control what data is introduced to the model. The model architecture reported in the main paper is the B-16 variant trained on LAION-2B unless otherwise stated and we report L-14 and B-32 in the Appendix C.
4.2 Zero-shot Results
In this setting, our model only has access to data it has seen during pre-training, in this case LAION-2B. Neural Priming improves top-1 accuracy by  $2.45\%$  on ImageNet and  $4.25\%$  on average across 6 other diverse downstream datasets compared to the CLIP baseline. In the zero-shot setting, Neural Priming outperforms the 3-shot CLIP model on StanfordCars and FGVCAircraft. This result is particularly noteworthy since traditionally training on in-distribution data generally outperforms zero-shot techniques [54]. Note that we do not present error-bars for the zero-shot experiments as the process is deterministic.
We also compare with VLM [33] and CuPL [39], two zero-shot prompt-tuning methods which obtain natural language descriptions of each class using language models, and a retrieval with fine-tuning baseline. For implementation details on how the retrieval and fine-tuning are performed see Appendix H. Interestingly, we find that Neural Priming is complementary to existing prompt-tuning methods. The accuracy improvements from CuPL and VLM are additive with Neural Priming. For example, CuPL and Neural Priming each improve performance by  $3.78\%$  and  $7.17\%$  respectively on FGVCAircraft. Ensembling the methods results in  $10.74\%$  improvement over the baseline (Table 2). This surprising result suggests that the textual class descriptions in CuPL and VLM provide unique information to the model that differ from the information obtained from the images in the priming set.
Figure 3: Performance of Neural Priming and comparable methods in the few-shot setting. We find consistent improvement across shot numbers and datasets. In particular, Neural Priming especially excels for fine-grained datasets such as FGVCAircraft and Flowers102. We hypothesize that such fine-grained captioned images are not well represented in LAION-2B, therefore revisiting this subset of data improves the model more.
Another observation is that Neural Priming is especially effective for specialized domains such as StanfordCars, Flowers102, and FGVCAircraft. We speculate this is due to the fact that the label space and image content differs from the majority of the pre-training set. For example, although airplanes occur frequently in LAION-2B, they are rarely described according to their specific model such as Boeing 737-200. Therefore, recalling and priming the model on pre-train images with such fine-grained classes significantly improves the model. For analysis of LAION-2B with regards to label statistics see Appendix B.