Neural Priming for Sample-Efficient Adaptation
Authors: Matthew Wallingford
Vivek Ramanujan
Alex Fang
Aditya Kusupati
Roozbeh Mottaghi
Aniruddha Kembhavi
Ludwig Schmidt
Ali Farhadi
Venue: NeurIPS 2023 poster (2023)
Keywords: Transfer Learning
Distribution Shift
Test-Time Training
TL;DR: Technique which leverages retrieval methods for recalling relevant pretrain data used to condition the network for improved transfer learning/distribution shift accuracy.

Abstract: We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks given few or no labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model to recall and conditions its parameters on relevant data seen throughout pretraining, thereby priming it for the test distribution. Neural Priming can be performed at test time in even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a 2.45% improvement in accuracy on ImageNet and 3.81% accuracy improvement on average across standard transfer learning benchmarks. Further, using our test time inference scheme, we see a 1.41% accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the common challenge of limited labeled data and changing distributions. Code and models are open-sourced at [https://www.github.com/RAIVNLab/neural-priming](https://www.github.com/RAIVNLab/neural-priming).

Neural Priming for Sample-Efficient Adaptation
Matthew Wallingford
Vivek Ramanujan"i
Alex Fang"
Aditya Kusupati Roozbeh Mottaghi Aniruddha Kembhavi
Ludwig Schmidt Ali Farhadi
University of Washington PRIOR, Allen Institute for AI *LAION
1mcw244,ramanv ramanymmavvve @cs. washington, edu
Abstract
We propose Neural Priming, a technique for adapting large pretrained models distribution shifts and downstream tasks given few labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model recall and conditions its parameters relevant data through- out pretraining, thereby priming it for the test distribution. Neural Priming can be performed at inference, even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves ac- curacy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a 2.45% improvement in accuracy on ImageNet and 3.81 % accuracy improvement learning benchmarks. Further, using Neural Priming at inference to adapt to dis- tribution shift, we see a 1.41% accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the chal- lenge of limited labeled data and changing distributions. Code is available https: /ttpt/ggttttt/tt////
average across standard transfer
Introduction
Humans have vast store of prior experience which we draw on to flexibly perform a diverse range of tasks [20, 5, 12]. While engaging in an activity, we naturally retrieve relevant information or schema in a cognitive phenomena known as Priming [34]. This process ensures that necessary knowledge is readily accessible in memory, leading to enhanced performance for the task at hand [43]. Pre-trained, general-purpose models such as CLIP [40] and ALIGN [23] have extensive prior knowledge learned from large-scale, diverse datasets. These datasets seek to capture variation in real data within their distribution. Can these models also benefit from something like priming? We observe that models trained even on the largest of such datasets often substantially improve in performance when fine-tuned on task-specific data. This begs the question of what the model learns from fine-tuning on the target dataset, if it already trained on many similar examples during pre-training
natural
We speculate that the effect of fine-tuning a pre-trained model on task seee-speifiic data is similar to that of priming. Given the sheer size and diversity of the pre-training dataset it becomes challenging for the model find consistent solution that is optimal for all subsets of the data. This becomes particularly evident for open- ven-vvoabullary models such as CLIP, where multiple natural language descriptions can correspond to a single image, highlighting the challenge of accommodating diverse interpretations. We hypothesize that training on the downstream dataset re-aligns the model to the specific objective.
With this in consideration, we propose Neural Priming. Specifically, Neural Priming recalls subset of the pre-training data similar to the target distribution, re-aligns the natural language
Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Task Description
Pre-training Data
CLIP Model
Stimulus
Maltese Poodle
German
Shepherd
Priming Set
P
Recall
Attunement
Figure I: A diagram of Neural Priming, our proposed method. Neural Priming is a framework for leveraging open-vocabulary model's own pre-training data to improve performance stream tasks. Neural Priming encompasses two processes: 1. Collecting a priming pool of relevant examples from the pre-training set to prime with and 2. using these examples to attune our model to given task. We show performance improvements across a wide range of transfer learning and robustness benchmarks.
down-
descriptions to the downstream task, and quickly adapts the model to the subset. perform extensive experiments 7 transfer learning and 4 distribution shift datasets to validate method. We use the OpenCLIP [40, 52] VIT [8] set of models pre-trained on the LAION-2B and 400M [44]. We find Neural Priming leads to significant accuracy improvements, particularly when labeled data scarce and in specialized domains. Concretely, Neural Priming improves accuracy by 2.45% ImageNet and 4.25% on average across the other 6 datasets over the base CLIP model. In the few-shot setting, Neural Priming improves accuracy by 3.81% % on average over recent methods on standard transfer learning benchmarks. We show Neural Priming is efficient and can be performed on-the-fly. For datasets containing more than billion images, we can prime our model to ImageNet in less than minutes with single commercial GPU.
Neural Priming is flexible and can be used with variable degrees of information about the down- stream distribution. When the model has language-only task descriptions, approach can efficiently retrieve priming pool of relevant examples from the pre-training set and attune the model to this data. At inference time, given
of test images classify, Neural Priming is able to test examples to adapt the priming pool to distribution shifts. When we have access to training examples in the few-shot setting, Neural Priming can filter the priming pool to align with the training distribution.
We make the following contributions:
We introduce Neural Priming, a novel method that leverages retrieval from the pre-training dataset for efficient and accurate adaptation of large, pre-trained models to downstream tasks.
Neural Priming achieves state-of-the-art zero-shot and few-shot accuracy the standard transfer learning datasets - up to 4.25% and 3.81 % improvements respectively over baselines (Section 4.2).
Neural Priming also enables transductive learning and improves performance on standard distri- bution shift datasets by 2.51% on average, all without using any additional data (Section 4.3).
Our approach generalizes to various architectures and pre- pre-training datascts while being comple- mentary to techniques [33, 39] that improve zero-shot performance of open-vocabulary models.
Related Work
2.1 Open-Vocabulary Models and Zero-shot Inference
Open-vocabulary models have proven to an effective approach transfer learning. Such models enable training vast amounts of web-scale images without the need for labor-intensive human

---

labeling by leveraging pre-existing natural language descriptions [40]. Open-vocabulagd models have set state-of-the of-thh--rt on ImageNet [7] as well other transfer learning benchmarks [53, 1, 41].
open-vocabulary models offer additional capabilities beyond standard pre-trained models. They can perform zero-shot inference, where predictions made without training on target data. Additionally, they are robust to distribution shifts [40], enable prompt-tuning methods [39,33], and can be used for text-based retrieval [10]. Zero-shot can have different meanings in the literature. In the context of this paper, we consider zero-shot as the experimental setting in which the model receives no training examples drawn from the training distribution.
Prompt-tuning has emerged as a popular research direction in the domains of large language and open-vocabulary models. In the context of open- vocabullay models, prompt-tuning modifying the textual prompts or queries used during the training or inference of the model to improve its understanding of visual content or achieve specific goals. In the original CLIP paper, Radford et al. [40] design hand-crafted prompt templates for ImageNet and other transfer learning datasets and show that this leads to substantial accuracy improvements. More recently, other work [33, 55] has used machine learning approaches to learn the prompts rather than hand-crafting them.
involve
2.2 Distribution Shifts
Robustness to distribution shift is a key property of good machine learning models as represents a notion of reliability. In particular, studies on natural distribution shifts, including ImageNet- V2 [41], ImageNett-Setch [51], ImageNet-R [18], and ImageNet-A [19], find that models have a consistent performance drop when exposed to distribution at inference time not seen during train time [49]. In order to focus on robustness and eliminate the confounder of better models being generally better, this performance gap is measured through effective robustness, which is the robustness improvement over ImageNet trained models. Prior work has shown that the performance of models on in distribution and out of distribution is highly correlated across many algorithmic training interventions, except for cases where training on larger and more diverse datasets increases robustness [35].
The most significant recent improvement in effective robustness [41] is the introduction of open- vocabulary models. At its time of release, CLIP [40] achieved unprecedented effective robustness on variety of distribution shifts. Studies have suggested that these models achieve high effective robustness through their data distribution [9], a result of training on large amounts of web-scraped data. However, these models are still worse at downstream tasks than models fine-tuned on in-distribution data. Moreover, fine-tuning on downstream data causes robustness on other data distributions to deteriorate [40, 52]. Many mitigation methods have been proposed to such as Wise-FT, FLYP, LP-FT, and model surgery [52, 15, 28, 29]. Our paper differs from these methods in goal: whereas they seek to keep model robustness while gaining the benefits of fine-tuning task-specific data, seek the benefits of fine-tuning while not collecting any in in-distribution data. Hence these methods are complementary to Neural Priming, and employ Wise-FT in our model attunement procedure.
2.3 Transductive Learning
Transductive learning [13, 6] focuses on leveraging unlabeled data during inference. It differs from traditional supervised learning, which solely relies on labeled data at train time. Related to transductive learning is test-time training [48, 14, 45]. Test-time training involves adapting and refining the model's predictions based on the specific testing examples encountered. Transductive learning differs from test-time training in that test-time training only considers one test sample at time, whereas transductive aims learn from the entire test set.
2.4 Few-Shot Learning
Few-shot learning research aims to addresses the challenge of learning from a limited number of labeled examples. many real-world scenarios, acquiring large labeled datasets is impractical or
costly. Older lines of work have focused on meta-training small models [47, 11, 37, 22] on small- scale datasets. More recently, the approach for few-shot learning has shifted towards training large, pureroe- models such as CLIP [40] and ALIGN [23] on web-scale datasets.
3

---

2.5 Retrieval-Augmented Models
In language, works have demonstrated the effectiveness of retrieval from text corpora or structured data for tasks such as question answering [3, 16, 25]. In general, these methods seek to recover facts either from a large corpus or knowledge graph, then use those to complete tasks. This differs from our scenario, where exact examples at inference time do not necessarily exist in the pre-training corpus. REACT [30] and SuS-X [50] are -augeeee- methods for open- veen-vocabulary models which use search to fine-tune with relevant examples [30]. We differ from Liu et al. [30] in that they add a substantial number of new parameters whereas we do not. Additionally,
approach is significantly more efficient, both computationally and in terms of number of samples, enabling use at inference for additional improvement (Section 3.1.2). We differ from [50] in that their work uses semantic retrieval whereas Neural Priming leverages language for fast initial filtering and image search for accurate retrieval. Further, Neural Priming shows that models can improve by revisiting examples seen throughout pretraining whereas other works retrieve new examples from external datasets.
Method
Neural Priming is the process of retrieving relevant information from the pre-training dataset and leveraging it for specific task. We study it in the context of vision-language contrastive pre-training, so the form our task description takes is a set of class names, C, already natural language. CLIP model [40] consists of vision embedding model, V. and language embedding model, L, each Rd. The pre-training dataset, D, consists of a large number of image-text pairs collected from the web. The text component can be noisy, potentially containing irrelevant or inaccurate information about the image content.
producing a vector representation in
We break our method down into two main steps: 1. Collecting the priming pool, where we gather data from our pre-training dataset relevant to particular task and model attunement, where we leverage this data to improve our model.
3.1 Collecting the Priming Pool
3.1.1 Leveraging Natural Language Task Information
The goal of this step is collect an initial pool of images relevant to the task hand given the previously defined natural language description C. For example, if our task is a set of dog breeds, ideally we would collect sets of images belonging to those breeds and label them accordingly. A simple way to prime is by using retrieval to gather relevant data points from our pre-training dataset. An existing method for language- based retrieval involves using the CLIP text embedding of a class description E C' for retrieval using semantic similarity scores on the pre-training set [2, 42]. However, with neural priming, prioritizing precision over recall is crucial, considering the size, diversity, and noise of the pre-training dataset. This form of semantic retrieval has a major downside: it is not clear where threshold similarity scores to retrieve the most relevant images. Threshold too late and we allow unrelated images to be included in our pool. Further, this threshold is often specific to category, making it infeasible to search at scale.
Our approach to language-based priming is to search for the existence of the class name, c E C, in the captions of our pre-training dataset to retrieve images relevant to a particular category. We organize these image-text pairs into separate categorical clusters { [B,} according to the class name c mentioned in their captions. This approach has few advantages over semantic retrieval: 1. After setting up an inverted index search structure for text retrieval [26, 46], exact string matching is far faster than semantic retrieval, even when approximate nearest neighbor strategies are employed [24], 2. with exact string search, the category boundary is clear and therefore does not require per category tuning, 3. the retrieval results are overall qualitatively more relevant. Finally, to leverage the semantic understanding of our CLIP model, we filter the priming pool using CLIP similarity score. We do this by constructing a "zero-shot" CLIP classifier, as defined by Radford et al. [40], and removing examples from categorical clusters that do not align with their label according to the CLIP model.

---

C:: preAt BTY TVL
Transductive Image Filter (ImageNet-V2)
Inibal Diverse Priming Pool
Filtered Priming
(Great Grey Owl)
A qualitative example of our approach for transductive image filtering. Given an initial priming pool, acquired through naturalee text search on the captions our pre-training dataset (Section 3.1), we filter out irrelevant examples using images from our test set. (left) we show examples from the great owl categorical cluster of our priming pool before filtering. (center) we show example image from the same category of ImageNet-V2, (right) example retrievals using image embedding similarity from the entire priming pool. The visual similarity of the retrievals are apparent, and they are generally from the appropriate categorical cluster. Doing this filtering results significantly more relevant priming pool.
Figure 2:
3.1.2 Leveraging Image Information at Test Time
At inference time, the model can narrow the relevant priming pool even further by utilizing information about the test distribution. To do this, given image r our set, we compute the cosine similarity using our CLIP image encoder V, cos(V(a:;, Vuv) for every y E P. our priming pool. We retrieve examples with the top-k cosine similarity scores (k- 10 in most of our experiments). We do this collectively for every image in the test set and collect the retrievals to form filtered priming pool. If an example retrieved twice, we de- depuucie them in the final priming pool. Since we do this for all images in the test set, we consider this the transductive setting to align with prior work [13, 6].
3.2 Attuning CLIP to the Priming Pool
The goal of this step is to modify our CLIP model to take advantage of the data in the priming pool P. We first construct the task-specific zero-shot linear head W, RRdx using the text encoder and the natural language names of each class, where d is the feature dimension and n is the number of classes. To get logits for partieular example, æ, we compute W. L(x), so our prediction arg max, W, (2x).
To attune our CLIP model to the priming pool, we perform nearest-class mean (NCM) [32] on all retrieved examples to obtain a classification head from the priming pool. Namely for given class we compute a centroid de | EzEB. L(x)) and then normalize this centroid to produce class embedding Ve Vo/ llelll We define the collection of centroids matrix Wtr [Me]e E ggdn, To expand this to few-shot scenarios, we mix the labeled data into the corresponding categorical clusters before performing NCM. Finally, ensemble W, and Wy using a mixing coefficient W. = (1-11 a) Wfz a W., which our final classification head. We choose alpha according to heuristic e = which can be derived from Bayesian prior over the text features. We experiment with varying values of a in Table 10. We also find that a can be effectively chosen through cross-validation data in our priming pool, we want it to influence our model less. We use NCM as the classifier as has shown to be sample-efficient [47]. For comparison to WISE-FT classifier [52] see Table 3.
[0, as
held-out portion of the retrieval set. Intuitively, if we do not have much
Experiments
Our key results include: Priming improves performance over baselines in the few-shot setting by 3.81 % on average across all datasets and 2.4% on ImageNet in the zero-shot setting 2. Priming in the
5

---

Table 1: Performance of Neural Priming and comparable methods in the zero-shot setting. Priming consistently improves top-1 accuracy across standard transfer learning data sets. Performance reported for the OpenCLIP ViT-B-16 model pretrained on LAION-2B.
Stanford Cars
FGVC Aircraft
Oxford Pets
ImageNet
Flowers
Food101
SUN397
CLIP [40, 21]
68.30
87.40
25.86
71.65
86.58
90.21
67.35
Retrieval Finetuning
70.28
87.95
26.22
72.15
86.63
90.35
68.01
VLM [33]
69.35
87.88
28.54
72.11
86.31
90.24
67.73
CuPL [39]
70.25
88.63
29.64
72.32
86.20
91.16
70.80
Priming (Ours)
70.75
89.30
33.03
79.81
86.66
91.87
71.21
Priming CuPL (Ours)
71.38
90.23
36.00
80.04
86.86
91.85
72.35
transductive, on-the-fly, setting further improves performance baselines by 2.51% accuracy and 1.09% over standard Neural Priming 3. Priming is complementary to existing prompt-tuning methods. Our finding indicates that images in the priming impart distinet information to the model compared to textual class descriptions, We include full details of hyperparameter choices and error bars included in the appendix.
4.1 Datasets and Architectures
We evaluate on standard transfer learning and distribution shift benchmarks. ImageNet [7] a large- scale, general classification dataset that has been well-studied in both transfer learning and distribution shift. ImageNetV2 [41] is one of its natural distribution shift test sets, made by reproducing the original data collection procedure of ImageNet, but even modern large-scale pre-trained models have performance drops on it. ImageNet Sketch [51] and ImageNet-R [18] are natural distribution shifts created by assembling sketches and various renditions of the ImageNet classes. ImageNet-A [19] is a natural adversarial distribution shift of ImageNet, created by collecting images that are misclassified by ResNets. StanfordCars [27]. FGVCAireraft [31], Flowers 102 [36], and OxfordPets [38] are tme-grained classification datasets which require understanding subtle visual differences between classes and are commonly used for transfer learning benchmarks [17, 21, 40, 55J. SUN397 [53] is a large-scale scene recognition dataset with 397 scene categories.
We perform our experiments with OpenCLIP models [52] trained on LAION-2B and 400M [44]. We choose OpenCLIP because their pretrain datasets are publicly available, therefore we can control what data is introduced to the model. The model architecture reported in the main paper is the B-16 variant trained on LAION-2B unless otherwise stated and we report L-14 and B-32 in the Appendix C.
4.2 Zero-shot Results
In this setting, our model only has access to data it has seen during pre-training in this case LAION- 2B. Neural Priming improves top-1 accuracy by 2.45% on ImageNet and 4.25% on average across other diverse downstream datasets compared to the CLIP baseline. In the zero-shot setting, Neural Priming outperforms the 3-shot CLIP model StanfordCars and FGVCAircraft. This result is particularly noteworthy since traditionally training on in-distribution data generally outperforms zero-shot techniques [54]. Note that we do not present error-bars for the zero-shot experiments as the process is deterministic.
We also compare with VLM [33] and CuPL [39], two zero-shot prompt-tuning methods which obtain natural language descriptions of each class using language models, and retrieval with fine- tuning baseline. For implementation details on how the retrieval and fine-tuning are performed see Appendix H. Interestingly, we find that Neural Priming is complementary to existing prompt-tuning methods. The accuracy improvements from CuPL and VLM are additive with Neural Priming. For example, CuPL and Neural Priming each improve performance by 3.78% % and 7.17% respectively on FGVCAircraft. Ensembling the methods results in 10.74% improvement over the baseline (Table 2). This surprising result suggests that the textual class descriptions in CuPL. and VLM provide unique information to the model that differ from the information obtained from the images in the priming set.

---

ImageNet
Stanford
72
70
90
Accuracy
Accuracy
69
Neural Priming
89
Neural Priming
CoOp
CoOp
67
NCM (Baseline)
NCM (Baseline)
Zero-shot OpenCLIP
Zero-shot OpenCLIP
1
2
10
1 2
10
# Training Samples per Class
Training Samples per Class
FGCV-Aircraft
Flowers102
42.5
40.0
90
Accuracy cy (%6)
37.5
85
35.0
32.5
Priming
Neural Priming
30.0
CoOp
CoOp
NCM (Baseline)
NCM (Baseline)
27.5
Zero-shot OpenCLIP
Zero-shot OpenCLIP
25.0
2
10
# Training Samples per Class
# Training Samples per Class
Figure 3: Performance of Neural Priming and comparable methods in the few-shot setting. We find consistent improvement across shot numbers and datasets. In particular, Neural Priming especially excels for fine-grained datasets such as FGVCAircraft and Flowers 102. We hypothesize that such fine- grained captioned images are not well represented in LAION-2B, therefore revisting this subset of data improves the model more.
Another observation is that Neural Priming is especially effective for specialized domains such as StanfordCars, Flowers 102, and FGVCAircraft. We speculate this is due to the fact that the label space and image content differs from the majority of the pre-training set. For example, although airplanes occur frequently in LAION-2B, they are rarely described according to their specific model such Boeing 737-200. Therefore, recalling and priming the model on pre-train images with such fine-grained classes significantly improves the model. For analysis of LAION-28 with regards to label statistics see Appendix B.
In contrast, for datasets which are more aligned with LAION-2B and the distribution of internet images, such as ImageNet and SUN397, the accuracy gain provided from Neural Priming is smaller in comparison, albeit still significant. In the limit of this trend, Food101 sees almost no improvement across all methods, and even training on in-distribution data for the few-shot case barely improves the accuracy. We speculate that this is because images similar to those in Food101 are already well- represented in LAION-2B, rendering additional food images of marginal informational value. We provide analysis of how well the attributes of each dataset are captured by LAION-2B in Appendix B.
To be precise, when refer to term "shot number" throughout the experiments section, number labeled examples from the target training set. We do not consider images retrieved from LAION-2B shots in this setting because they obtained from the pre-training set.
the
4.3 Few-Shot Results
Neural Priming improves performance for all datasets and shots in the few-shot setting. We compare with CoOp, recent method for few-shot prompt-tuning tuning, and Nearest-class-Mett (NCM) baseline. On average across all shots and datasets Neural Priming improves by 3.81% in accuracy over the closest baseline. Results can be found in Figure and Table 9 of the Appendix.
Notably, we find that Neural Priming can match the accuracy of models trained with substantial number of training examples without using any of the labeled training data for all of the evaluated
7

---

Table 2: Performance of Neural Priming and relevant methods for the transductive setting. Neural Priming finds examples similar to the test image at inference to optimize the model. Models are evaluated zero-shot on 4 distribution shift datasets. Neural Priming excels on distribution shifts which differ significantly from the natural language description of the class names. Performance reported for the OpenCIIP ViT-B-16 model pretrained on LAION-2B.
ImageNet Sketch
ImageNet-V2 V2
ImageNet-R
ImageNet-A
CLIP [21, 20]
59.35
64.57
57.05
35.95
TPT [45]
59.84
78.74
52.75
36.92
Priming (Ours)
60.12
77.98
58.29
37.56
Transduct. Priming (Ours)
60.76
79.37
59.97
38.20
datasets (Figure 3). Similar to the zero-shot setting, we observe that Neural Priming is complementary with prompt-tuning methods (Appendix F), Additionally, we observe that as the shot number increases, improvement over the baseline decreases. At 1-shot the improvement in accuracy over the baselines 5.63% on average, while at 10-shot the improvement is 2.04%. Intuitively, as the model receives more target training data, obtaining additional examples from the pretrain set becomes less necessary.
4.4 Transductive Results
We compare Neural Priming in the transductive setting on 4 standard distribution shift datasets, ImageNet- V2, ImageNet Sketch, ImageNet-R and ImageNet-A. Distribution shift datasets are natural application of adaptation at test-time. Often real-world datasets differ from the training data, therefore models should be able to adapt on-the-fly. this setting, the model can learn from the test images without labels before making predictions. We compare with Test-Time Prompt-Tuning (TPT), a state-of-the-art method which uses a self-supervised objective to learn from test data.
We find that Neural Priming with images in the test set improves performance over standard Neural Priming by 1.09% as well as 2.51% over TPT across the 4 distribution shifts (Table 2). Looking at Figure 2, we qualitatively see that the priming pool more closely matches the test images after filtering for the closest images in the initial priming pool. Though the distribution shift can often be imperceptible such as between ImageNet and ImageNetv2, quantitatively we see that the transductive filtering step finds images the pretraining close to the test distribution.
The transductive retrieval for 50,000 images in the test set on average takes 96 seconds for a priming pool of 1 million images, while retraining the classifier takes average 1 11.5 seconds for priming pool of size 10,000 on standard hardware. We provide further analysis run-time efficiency of the on-the-fly variant of Neural Priming in Appendix G.
Flowerr--100
Stanford
FGVC Aircraft
Top-1 Acturacy
Top-1 Accuracy
500
36
Tap-1 Accuracy
91.75
28
89
82
21.50
22.25
76
28
91.00-
40
Samples
Samples per
Samples
Samples per
Figure 4: Ablation over the number of samples per class in the priming pool. We observe a
consistent zero-shot accuracy improvement the number of samples drawn from our pool increases.
4.5 Ablations
We investigate the impact of the priming pool size on the zero-shot accuracy of downstream tasks (Figure 4). Our analysis reveals that as the size of the priming pool increases, there is a general improvement in accuracy. However, there are certain limitations associated with enlarging the pool. The majority of classes in the downstream task have limited number of available images.
8

---

Consequently, when we retrieve larger number of images for the priming pool, they tend to contain more noise and mislabeled samples. Furthermore, for rare classes, the number of images obtained through exact string search is often less than 100. To address this, a potential extension could involve utilizing a language model to generate alias names for classes, which could then be used to perform additional string searches, thereby expanding the initial priming pool size.
ImageNet
Stanford
Flowers-102
Top- Accuraty
15
CLI Zero-shot
Noural Priming
T0u 1 Secuuio
30-
CUP Zeroshet
Priming
Top- Accurccy
60
CLIP Zerosshet
Priming
Top- Acccurcyy
40
25
CUP Zero-shok
BB
86
8/22
1/16
1/16
1/32
1/54
B/1.6 Model (VIT)
U14
B/12
8:16
1/4
Model (MTT)
Model (VI))
Model (VIT)
Figure 5: Analyzing the effect of model capacity on Neural Priming, We find the relative error reduction stays consistent even as the scale of the model increases.
We also analyze the impact of the architecture on the accuracy improvement achieved by Neural Priming in the zero-shot setting (Figure 5). To examine this, we conduct experiments using models of varying capacities, namely ViT B-32, B-16, and L-14. We observe that the gains remains consistent across the models. This finding suggests that even as we scale the architecture's capacity, our method will continue to yield significant and consistent relative error reduction.
Limitations
Neural Priming has a few potential limitations. Firstly, requires that the pre-train dataset contains images similar to those in the downstream task. Though all of the datasets we benchmark have abundant relevant data, it is possible for more out-of-distribution datasets that LAION-2B simply does not contain related or queryable images. Secondly, accurate class names are required for retrieval. Meaningful class names for some datasets can be difficult to obtain. For example, in the Plowers102 dataset, some flower species given by their latin names, which leads to poor retrieval. This issue generally affects open- open-vocabulary models which require accurate class names to initialize the zero-shot classifier. This limitation may be resolved by using language models to replace class names with their more commonly known synonyms. Lastly, Neural Priming requires access to the pre-training data set which is not always possible such as in the case of OpenAl variant of CLIP. In this case a surrogate dataset would likely suffice, such as using LAION-2B.
Discussion & Conclusion
We present Neural Priming, method to improve the performance of open-vocabulary models by leveraging their own large-scale, diverse pre-training data with no additional data required. With Neural Priming, we demonstrate how construct a high quality priming pool of examples from the pre-training dataset relevant to a particular task and how to utilize this pool to improve our model. We further show that our method effective across a variety of downstream tasks and settings. In particular, our method can be used in situations where only natural language descriptions of relevant classes are given, when we have the ability to adapt at inference time, and when we are provided with few labeled in-distribution examples. In all settings, our framework demonstrates a substantial improvement in performance over existing interventions, and is in fact complementary with current prompt-tuning and robustness methods. Our method is also computationally cheap, not requiring any modification of model backbone weights and only a fast text search on the pre-training corpus.
The efficacy of Neural Priming leads to some interesting questions for future work. For example, if the model has seen this data before, why does it help to recall them? We hypothesize that this is due to the fact that the diversity of these datasets introduces competing objectives, which are difficult for the model to optimize directly. For example, the same kind of image could appear with multiple captions and vice-versa, making it difficult to prompt CL.IP model trained on such data at inference time for a particular task. A systematic study of this could elucidate important limitations of current large-scale training paradigms.

---

Acknowledgments
We are grateful Sarah Pratt, Mitchell Wortsman, and Romain Beaumont for helpful discussions and feedback. Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303 166, DARPA N66001-19-2-4031. DARPA W911NF- 15-1 1--1-,533, and gifts from Allen Institute for Artificial Intelligence, Google, and Apple. Ludwig Schmidt and Alex Fang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML, CCF-2019844) Open Philanthropy, Google, and the Allen Institute for AI.
References
[1] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[2] R. Beaumont.
Clip
Retrieval, 2021.
URL https://gt thub.
clip-retrieval.
[3] Borgeaud, A. Mensch, Hoffmmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driess- che, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 1206-2240 PMLR, 2022.
[4] G. Brod, M. Werkle-Bergnee and Y. L. Shing. The influence of prior knowledge on memory: developmental cognitive neuroscience perspective. Frontiers behavioral neuroscience, 7:139, 2013.
[5] A. L. Brown and M. J. Kane. Preschool children can learn to transfer: Learning to learn and learning from example. Cognitive psychology, 20(4):493-523, 1988.
[6] O. Chapelle, B. Schölkopf, and A. Zien. A discussion semi-supervised learning and transduc- tion. In Semi-supervised learning, pages 473-478. MIT Press, 2006.
[7] J. Deng, W. Dong, R. Socher, Li. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 1 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[9] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L.. Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In K. Chaudhuri, S. Jegelka, I.. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 6216-6234. PMLR, 2022. http////22fffa2222aa// html.
URL https: /ttp:////ccccceeeeee/eee/ mlr.
[10] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 11097, 2021.
[11] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1136-1135. PMLR, 2017.
[12] N. T. Franklin and M. J. Frank. Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning. PLoS computational biology, 16(4)ee 2020.
[13] A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. 130 2013.
arXiv preprint
[14] Y. Gandelsman, Y. Sun, Chen, and Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-29385, 2022.
10

---

[15] S. Goyal, A. Kumar, S. Garg, Z. Kolter, and A. Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models, arXiv preprint arXiv:2212.00638, 2022.
[16] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929-3938. PMLR, 2020.
[17] K. He, R. Girshick, and P. Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918-4927, 4927, 2019.
[18] D. Hendrycks, Basart, N. Mu, S. Kadavath, Wang. E. Dorundo, Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021.
[19] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[20] L. Hermer-Vane Vazquez, A. Moffet, and P. Munkholm. Language, space, and the development of cognitive flexibility in humans: The case of two spatial memory tasks. Cognition, 79(3): 263-299, 2001.
[21] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https://doi. /ttp:://doo.org110. org/10. 6281/zenodo 5143773.
[22] M. A. Jamal and G.-J. Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11719-11727, 2019.
[23] C. Jia, Y. Yang, Y. Xia, Y--T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint 200.00502. 2021.
[24] J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535 547,507.0000 2019.
[25] U. Khandelwal, O. Levy, D. Jurafsky, Zettlemoyer, and Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172. 2019.
[26] D. E. Knuth. Retrieval on secondary keys. The art of computer programming: Sorting and Searching, 3:550-567, 1997.
[27] Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[28] A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform oft-of-distributiin. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httLttts////pperrriiii net wttp///rrrumunnnideaee/ee
[29] Y. Lee, A. S. Chen, Tajwar, A. Kumar, H. Yao, Liang, and C. Finn. Surgical fine-tuning improves adaptation to distribution shifts. CoRR, abs/2210.11466. 2022. doi: 10.48550/arXiv. 1466. URL https: org/ 10. 4854448550/00 48550/arXiv. aar.vv..vvvv. 2216.
[30] H. Liu, K. Son, J. Yang, C. Liu, J. Gao, Y. J. Lee, and C. Li. Learning customized visual models with retrieval-augmented knowledge. arXiv preprint arXiv:2301.07099, 2023.
[31] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.5555, 2013.
[32] C. Manning, P. Raghavan, and H. Schütze Vector space classification. Introduction to Informa- tion Retrieval, pages 289-317,2008

---

[33] S. Menon and C. Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022.
[34] D. E. Meyer and R. W. Schvaneveldt. Facilitation in recognizing pairs of words: evidence of dependence between retrieval operations. Journal of experimental psychology, 90(2):227, 1971.
[35] J. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy the line: on the strong correlation between out-o--distribution and in-distribution generalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 7721-7735. PMLR, 2021. URL http:/ http::::ppocceeding.. mml. pwwss/ hassss/v13//mm/ htt///mmlmmmb....mmm//ee html..
[36] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[37] B. Oreshkin, P. Rodríguez López, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
[38] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[39] Pratt, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification, arXiv preprint arXiv:2209.033200 2022.
[40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[41] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[42] A. Rege, A. Kusupati, A. Fan, Q. Cao, S. Kakade, Jain, A. Farhadi, et al. Adanns: A framework for adaptive semantic search. arXiv preprint arXiv:2305. 2023.
[43] H. L.. Roediger and K. B. McDermott. Creating false memories: Remembering words not presented in lists. Journal of experimental psychology: Learning, Memory, and Cognition, 21 (4):803, 1995.
[44] C. Schuhmann, R. Beaumont, Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[45] M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test- time prompt tuning for zero-shot generalization in vision-language models. arXiv preprint arXiv:2209.07511, 2022.
[46] Sivic and A. Zisserman. Video google: text retrieval approach object matching in videos. In Computer Vision, IEEE International Conference on, volume 3, pages 1470-1470. IEEE Computer Society, 2003.
[47] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[48] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International conference machine learning, pages 2249-2 PMLR, 2020.
[49] R. Taori, A. Dave V. Shankar, N. Carlini, Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-
12, 2020, virtual, 2020. URL https: https:://prooeeedinggg wwwpediiinnnnnnniinneeeeee httpppssccppppp/////22/ Abstract.tttttttae
12

---

[50] V. Udandarao. A. Gupta, and S. Albanie. Sus-x: Training- free name-only transfer of vision- language models. arXiv preprint arXiv:2211.16198, 2022.
[51] H. Wang, S. Ge, Z. C. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In H. M. Wallach, H. Larochelle, A. Beygelz- imer, F. d A'Alché-Bel Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Infor- Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10506-10518, 2019. URL https: https://proceediigg wwwpeeeddinnnnuuiissss.omm 3aofcab000r5044902256222292 Abstract html.
mation Processing Systems 32: Annual Conference
[52] M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[53] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to 7oo. 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492 IEEE, 2010.
[54] R. Zhang,
W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 493-510 Springer, 2022.
[55] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
13

---

A Implementation Details
We perform last layer retraining for Neural Priming. We experimented with fine-tuning all layers, but found minimal performance benefits (Table 4). It has been noted in few-shot literature [47,37] that retraining only the last layer sufficient when few training examples are present and confirm this empirically. When retraining the last layer, we ensemble the text-classifier. We use mixing co-efficient to ensemble the text-classifier with the image centroids. We found that as the number of samples in the image centroid increases, the ensemble should more heavily favor the image centroid. Formally use e -1 /o as the mixing-coefficicn where [P] is the number of image examples with a equal to 100.
The OpenCLIP models we use can be found http://ggihub.com//looomndddiiii We use the following specific models: (ViT-B-32, baionnb_s3eed s0t__333bbbb______ _b9k)), (ViT-B-16, saion2b__34b_baee _88k),, (ViT-L-14, laion2b drion2b_s32b____ _s3b__82k))
clip.
For the zero-shot CLIP baseline presented Table 2, we use the baseline prompts from [33]. When ensembling with the image centroids from NCM, use the text-classifier initialized with these prompts unless otherwise stated.
To perform fast substring search, we set up Full Text Search (FTS) database over the metadata shards of LAION-2B using SQLite. Each shard SQLite database composed of 18.1 million consisting of information about the URL of the image location, constructed from the original distributed shards from LAION-5B [44]. The nature FTS and storage reading attributes means that it is much faster to search for rare substrings. More details on this are provided in Appendix G.
Table 3: Comparing Neural Priming to other classification approaches. NCM uses only the image features. WISE-FT fine-tunes a linear classifier and ensembles with the text classifier.
Shots
Neural Priming
NCM
Wise-FT
0
71.38
68.3
67.75
I
71.56
68.89
68, 14
2
71.81
68.85
68.25
5
72.03
69.12
68.78
10
72.36
70.12
69.02
B LAION-2B Analysis
We analyze the prevalence of each domain in the LAION pretrain set (Figure 6). We find that for datasets where the domain images and captions occur infrequently such as Flowers and StanfordCars, the performance gain from Neural Priming is significantly larger. We measure domain prevalence LAION-2B by substring searching for captions which contain the class names and counting the total number of images. Image content is another factor for measuring how close the pretraining set is to the downstream dataset in feature space, however this is computationally expensive for dataset of 2 billion. Also this would depend the model used extract features. We choose to look at the caption statistics as a proxy because is computationally feasible with a dataset the size LAION-2B and model agnostic.
C Other Architectures and Datasets
We present the full results for VIT B/16, B/32 and ViT L/14 on LAION-2B for ImageNet, Flowers 102, StanfordCars, FGVC-Aircraft, and OxfordPets. We find similar results across architectures. Results can be found in Table 6. We also reproduce experiments for the LAION-400m dataset with the B/16 architecture.
D Dataset Statistics
For ImageNet we performed URL deduplication using the LAION-2B URL list provided on Hug- gingFace. We found no ImageNet images in the pretrain data. We also randomly sampled images
14

---

Table Comparison of full fine-tuning and NCM with Neural Priming, We find NCM performs
similarly in the low-sample regime.
Stanford Cars
FGVC Aircraft
Oxford Pets
ImageNet
Flowers
Food101
SUN397
CLIP [40,
68.30
87.40
25.86
71.65
86.58
90.21
67.35
Retrieval + FT
70.28
87.95
26.22
72.15
86.63
90.35
68.01
VLM [33]
69.35
87.88
28.54
72.11
86.31
90.24
67.73
CuPL [39]
70.25
88.63
29.64
72.32
86.20
91.16
70.80
Priming (Ours)
70.75
89.30
33.03
79.81
86.66
91.87
71.21
Priming CuPL. (Ours)
71.38
90.23
36.00
80.04
86.86
91.85
72.35
Prevalence of Datasets In LAION-2B
1.0
Relative Frequency
0.8
0.6
0.4
0.2
0.0
Stanford Cars
FGVC-AirCraft
Flowers102
Food101
OxfordPets
ImageNet
SUN397
Figure 6: Frequency of dataset class names in the LAION-2B dataset normalized to 1. We find Neural Priming improves accuracy more for datasets that rare.
from the other datasets (Flowers102, FGVCAirCraft, etc.) and looked for duplicates, though found no exact matches. The exact dataset statistics are reported for context in Table 7.
E Comparison to REACT
We do not compare directly with REACT [30] the main paper for two reasons. The first that REACT adds a significant number of parameters (50 % for the B/16 model) to the network, making an unequal comparison from a parameter and FLOPS perspective. Second, REACT uses 6-10 million images as the support set for each dataset which makes it computationally infeasible to rerun ourselves for all the data sets we benchmark on and without the parameter adding version. compare on ImageNet using the numbers reported in their work for comparable models trained on LAION-2B (Table 8).
We differ from REACT in few key aspects. First, we realign the images labels to the downstream task. For example, for the OxfordPets data set, given an image which has the caption "maltese on leash walking through the parking with its owner" we map the label to "maltese". REACT trains contrastively using the original captions. We find this step significantly affects accuracy and sample-efficiency. For comparison, REACT retrieves 6- 10 million images from LAION whereas we use 10,000-10 100,000, Second, our method is significantly more efficient step so can be performed at test-time. This is due to our multi-stage filtering with the first stage of caption filtering being extremely fast. Third, REACT adds 50% additional parameters the ViT B/16 model.
retrieval and fine-tuning
15

---

Table Performance Neural Priming and the CLIP baseline with the VIT B/16 architecture. We observe similar trends as for the LAION-2B dataset. Neural Priming improves over the zero-shot performance with similar relative error reduction.
Stanford Cars
FGVC Aircraft
Oxford Pets
ImageNet
Flowers102
Food101
SUN397
CLIP [40, 21]
66.72
82.47
16.05
71.47
86.33
88.68
61.13
Neural Priming (Ours)
69.21
86.83
26.73
82.17
86.62
90.08
62.27
Table 6: Zero-shot results for ViT B/32 and L/14 architectures. We find similar results across architectures. Surprisingly, the relative error often decreases more with architecture scale.
Capacity ImageNet Stanford Cars
Flowers102
FGVCAircraft
Food101
Oxford Pets SUN397
Baseline (NCM)
VIT-B/32
67.12
84,21
67.40
23.79
82.53
90.62
61.97
ViT-B/16
68.30
87.40
71.65
25,86
86.58
90.21
67.35
VIT-1/14
76.62
91.82
75.14
33.36
90.96
93.40
71.29
Neural Priming
VIT-B/32
68.38
87.45
72.26
31.68
82.91
91.25
63.84
ViT-B/16
70.75
90.23
79.81
33.03
86.66
91.87
71.21
VIT-L/14
78.72
93.12
84.48
43.62
91.06
94.05
72.49
F
Few-Shot
We report the few-shot performance of a model pretrained on LAION-2B with CuPL prompts (Table 9). We find that better prompts synergize with Neural Priming similar the zero-shot case. The mixing coefficients used for these experiments are set by the schedule given in section A.
G Transductive Run-Time Analysis
Transductive run-time can be broken into two components: 1. Time to get the initial priming pool and 2. Time to perform exact retrieval on image features of the priming pool.
G.1 Initial Priming Pool construction timings
Here we study the wall-clock time for construction of the initial priming pool on consumer hardware (Intel 10900k CPU and Samsung 980 Pro NVME drive). Wall-clock times for priming pool construc- tion vary greatly depending on variety of factors. The two main factors affecting timings are: 1. speed of the storage on which the fast text search (FTS) databases are stored (see Appendix A) and 2. whether not LAION-2b is stored locally (if it is not stored locally, then you need to access the
10a
Seconds to search
10
10a
10 1
200
400
600
800
1000
Sorted query index
Figure 7: Wall-clock times by query for construction of the initial priming pool. The queries used here are the ones used for the construction of the ImageNet pool. Due the nature of full text search, rare queries are very fast, with most classes taking less than second to search all of LAION-2b.
16

---

Dataset
Number of Classes
Train Size
Test Size
ImageNet
1000
1281167
50000
Stanford Cars
196
8144
8041
FGVC Aireraft
100
6667
3333
Flowers 102
102
2040
6149
Food101
102
75750
25250
Oxford Pets
37
3680
3669
SUN397
397
19850
19850
Table 7: Dataset statistics for every dataset evaluated on in Section 4. We evaluated using the accuracy metric for every dataset.
Table 8: Comparison with REACT on ImageNet for varying capacities. Neural Priming outper- forms REACT on ImageNet without adding parameters while retraining on 1 1% of the data REACT does (10 million vs 100,000.)
B/16
L/14
REACT
67.5
76.4
Neural Priming
68.38
78.72
original URL. to get the image, which adds another factor of variation). Given the massive changes in performance between the two options for (2), we only give wall-clock timing estimates for (1).
FTS databases in SQLite use an inverted index over the tokens of their entries. In this case, the tokenization over full words and some more common suffix strings. Given this tokenization, is easier to search for less common phrases and words. In Figure 7, we analyze this effect with respect ImageNet priming pool construction. We see that most queries take less than second to search all of LAION-2b. A few very common queries take much longer. For example, the most common query, T-shirt, takes >100s search. The overall time complete these searches for ImageNet was 21 minutes. For smaller datasets with less common class names, this process is significantly shorter. This is the majority of the runtime of our overall method.
G.2 Exact Retrieval Time
The next part of our process was performing exact retrieval on the priming pool at test time. The method for this is described in Section 3.1.2 This is generally efficient, as the priming pool is significantly filtered version of the pre-training dataset. To filter priming pool of 1. mm images to 5 1 using the test set of ImageNetV2 took 3.2 minutes consumer hardware (two 3090 GPUs), not including the initial feature extraction time since these are easily pre-computed.
H Other Implementation Details
Retrieval + Finetuning We retrieve 100 images per class using HNSW, an approximate nearest neighbors method, with the CLIP text-embedding from the class name. We contrastive fine-tuning with the CLIP objective as the pretrain data only has natural language descriptions as labels. We fine-tune for with learning rate of 1e-6 and batch size 1024.
CoOp [55] CoOp tries to learn the embeddings for the prompts using few examples for each the classes. We the official implementation' provided the authors with default hyperparameters. For ImageNet experiments, we the models for 50 epochs regardless of the shots. For the smaller datasets like Stanford Cars, Flower102 and FGCV-Aireraft, 2-shot and 5-shot models for 100 epochs, and 10-shot models for 200 epochs. We use the standard train-test split for all the datasets the CLIP models provided by OpenAI, we use OpenCI.I [21] models for fair comparison against Neural Priming.
train the 1-shot models for 50 epochs,
consistency across all the methods. Lastly, instead of
Shttps://githain http:///wwwbbaao/k/kaaaaa///////
17

---

Table 9: Performance on Neural Priming (NP) with prompt-tuning method in the few-shot setting, We find that prompt-tuning methods are compatible with Neural Priming even when training examples are available.
Shots
ImageNet
Stanford Cars
Flowers102
FGVCAíreraft
CuPL
NP
CuPL
CuPL
NP + CuPL
71.38
90.11
80.04
1
71.56
71.32
90.31
90.17
80.77
80.72
36,39
36.55
2
71.81
71.50
90.55
90.21
88.90
88.60
38.16
37.94
5
72.03
71.92
91.03
90.93
94.66
93.43
40.01
39.84
10
72.36
72.27
91.54
91.49
94.83
94.75
43.83
43.67
TPT [45] Test-time Prompt Tuning (TPT) combines the ideas of test-time training [48] and prompt learning [55]. For a given test image, TPT tries learn a soft prompt that maximizes agreement between multiple augmented views. We the official implementation' for all the experiments and train TPT with 64 augmented views to ensure the best performance on all the datasets.
ImageNet
Stanford Cars
FGVC-Aircraft
Flowers102
68.9
87.54
25.9
71.94
10
69.1
88.99
29.75
74.97
100
70.75
89.3
33.03
79.81
1000
70.64
89.11
31.32
78.81
Table 10: An ablation over the sigma value used to calculate the mixing coefficient between the image and text features. A larger sigma value is associated with a stronger prior for the text features.
I Confidence Intervals
We report the confidence intervals and average accuracy for the few-shot experiments (Table 11). For the zero-shot experiments, performance does not vary. For the few-shot experiments variation derives from sampling the training examples. We average accuracy across 10 runs and find the variance to be minimal (less than 1% for most settings).
Broader Impact
One potential risk of Neural Priming is amplifying bias in the dataset. By retraining on a subset of the data, is possible to condition the model in a way that biased given a biased prompt or captions. For downstream tasks which are sensitive to bias, the risk could be mitigated by filtering the captions for inappropriate text or using an auxiliary model to detect inappropriate images. This approach of filtering the priming dataset is potential direction for reducing bias in model the fly. Another potential drawback of Neural Priming is that it increases the carbon footprint of the trained model by retraining it more data. However, the size of our priming set is small relative to LAION-21 -B (less than 1%) and we only retrain on the last layer which minimizes compute cost.
'nttps://github httpe:///githuub..ommasahaah
18

---

Table 11: Few-shot results with confidence intervals. We report the average performance of Neural Priming across 10 runs with the standard deviation. We find that Neural Priming has lower variance than the baseline as the constant priming pool leads less variation in the centroid of images.
Shots
ImageNNt
Stanford Cars
Flowers102
RGVCAireran
Neural Priming
Baseline (NCM)
Neural Priming
Baseline (NCM)
Neural Priming
Baseline (NCM)
Neural Priming
Baseline (NCM)
71,53
80.77
27.56
88.
83.
30.1
72.02 0.04
7043 0.02
91 03 0.12
89.19 0.18
94.66 0:10
80 46 0.20
40.01 0.11
35.50 0.77
10
72,34 0.03
70.90 0.04
91.51 0.30
90.86 1 0.16
94,83 0.01
92.74 1 0.02
13.83 0.09
40.84 1 0.42
19


Reviews:
[{"review_id": "rtDivr8OO8", "reviewer": "Reviewer_xxj2", "summary": "This work proposed a new method, named Neural Priming, which uses retrieval to adapt vision-language foundation models to downstream tasks given few or no labeled examples in the test time. Specifically, Neural Priming first uses the set of class names in a new task to retrieve relevant samples from the pre-training dataset to form the priming pool and also uses the test examples to narrow the priming pool. Then, the CLIP model can be attuned using the priming pool by constructing the linear prediction head. The experiments are performed in three settings: zero-shot, few-shot and transductive.", "questions": "My major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details. I’m willing to increase my rating if my concerns can be well addressed.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using retrieval to adapt large vision-language models to downstream tasks is interesting and intuitive. Compared with similar approaches, such as REACT, this work does not need to add new trainable modules. \n2. Experiments on several settings (zero-shot, few-shot and transductive) show the effectiveness and broad applicability of the proposed method. Ablation studies on the number of samples per class in the priming pool and the model capacity are interesting.", "weaknesses": "1. Regarding the retrieval idea for test-time adaptation, the major difference between Neural Priming and REACT is that REACT adds new training modules while Neural Priming only constructs a linear prediction head. It is not clear to me how significant this difference is and how original the proposed method is. Also, there is no direct comparison between these two approaches in experiments. Since Neural Priming and REACT are used in similar settings, it is not clear to me if the proposed method can largely improve over REACT. \n2. It is not clear how sensitive the proposed approach is to the retrieval database. From experiments, we see the improvement will be less significant if the target dataset is well presented in the pre-training dataset (see lines 237-241). On the other hand, the performance will also drop if there is a large gap between the target dataset and the retrieval dataset. Another question is whether the retrieval database has to be the pre-training dataset? Or we can arbitrarily construct a large-scale dataset as retrieval database.\n3. In the few-shot setting, what is the comparison with CoOp+TPT (which is better than CoOp according to [44])?\n4. In the transductive setting, it is unfair to compare TPT as TPT only relies on a single test sample while the proposed method takes all samples in the test set into account. I think another baseline called TENT (https://openreview.net/forum?id=uXl3bZLkr3c) should be considered. \n5. Writing can be improved. There are many repeated sentences across the whole paper. For example, the quantitative results have been mentioned in both introduction and experiment sections repeatedly.\n6. Not sure what “3-shot CLIP model” refers to in line 214.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review. We are glad you found the paper interesting and intuitive and appreciate the feedback. We hope our response addresses your concerns and interested in any further comments you may have. \n\n**1. Regarding the retrieval idea for test-time adaptation, the major difference between Neural Priming and REACT is that REACT adds new training modules while Neural Priming only constructs a linear prediction head.**\n\nAt a high level, Neural Priming has four main contributions when compared with REACT [1] and other methods [2]. The first is that Neural Priming is far more sample-efficient in the number of examples drawn from the pretraining set. REACT [1] fine-tunes with 1-10 million examples whereas Neural Priming uses as few as 10,000. Second, Neural Priming is far more efficient in the retrieval process due to the initial text filtering step. Neural Priming takes around ~10 minutes on one rtx-6k and a Samsung Evo 970 SSD, whereas REACT requires ~72 hours. Third, due to the multi-stage filtration and the label realignment phase (captions are changed to fit the target categories), Neural Priming achieves 2.32% better performance on ImageNet. Fourth, we do not introduce new parameters or have architecture-specific modules, making Neural Priming compatible with any open-vocabulary model. Finally, we show that for even a fixed dataset, our method can find subsets which improve downstream performance. Previous methods [1,2,3] are focused on improving performance by adding new data to the model. This is a useful scientific contribution.\n\n**There is no direct comparison between these two approaches in experiments.**\n\nIn table 6 on page 15 of the appendix we included a comparison on ImageNet between REACT and Neural Priming. The REACT results are taken from [1] as reproducing their experiments would be prohibitively expensive. We’ve also included the numbers in the below table for your reference. We did not include direct comparison in the main paper for two reasons. REACT adds a significant number of new parameters to the model, and requires 1-10 million retrieved samples whereas Neural Sampling uses 10-100 thousand. Still, Neural Priming performs favorably. \n\nTop-1 Accuracy on Zero-Shot ImageNet\n\n|| B/16  |L/14|\n|:-:|:-:|:-:|\n| REACT|67.5|76.4|\n| Neural Priming |**68.38**|**78.72**|\n\nWe’d also like to note that the baseline, retrieval+fine tuning, in Table 1 of the main paper uses an implementation of REACT without adding parameters and using the same number of retrieved samples as Neural Priming for a direct comparison. \n\n**2. How sensitive is the proposed approach to the retrieval database? Does the retrieval database have to be the pre-training dataset?**\n\nWe’ve run experiments using LAION-2B trained model to retrieve from the LAION-400M database which can be found below. We find similar results, albeit retrieving from smaller scale databases improves accuracy less. We primarily focused on using the pretraining dataset as the retrieval database to understand the effect of resampling data without adding new data. It’s clear from past work that adding more data will improve models and one of our contributions is showing that performance can be improved even with the data held fixed. \n\nAblation over Retrieval Database.\n|| ImageNet | StanfordCars | FGVC-Aircraft | Flowers102 |\n|:-:|:-:|:-:|:-:|:-:|\n| CLIP|68.3|87.4|25.86|71.65|\n| Priming (LAION-400M)| 70.21|88.14|30.52|76.23|\n| Priming (LAION-2B)  |70.75|89.3|33.03|79.81|\n\n\n**3. In the few-shot setting, what is the comparison with CoOp +TPT (which is better than CoOp according to [44])?**\n\nCoOp+TPT is not a straight-forward comparison in the few-shot setting as TPT requires training on the test sample which the other few-shot methods are not designed for. We’ve run the CoOp+TPT baseline and report the results below. Also CoOp+TPT takes 12 hours on one RTX-6k and is not feasible to run on-the-fly. Even with these caveats, we still outperform CoOp+TPT.\n\nTop-1 Accuracy on Few-Shot ImageNet \n\n|| 1-Shot  | 2-Shot  |5-Shot  |10-Shot \n|:-:|:-:|:-:|:-:|:-:|\n| CoOp| 65.80 | 67.70 | 69.30 | 70.00 |\n| TPT+CoOp|66.97 |  70.04 |  71.02 |  71.89 |\n| Neural Priming|**71.56**| **71.81** |**72.03** |**72.36** |\n\n**4. In the transductive setting, it is unfair to compare TPT as TPT only relies on a single test sample while the proposed method takes all samples in the test set into account. I think another baseline called TENT should be considered.**\n\nThis is a fair point, and we appreciate you pointing us to an additional baseline. We will add this baseline to the paper and have included abbreviated results below.\n\nComparing with TENT [4] in Transductive Setting\n|| ImageNet-V2| ImageNet-Sketch |\n|:-:|:-:|:-:|\n| CLIP| 59.35|57.05 |\n| TENT|59.91|57.40|\n| Transduct. Priming |60.76| 59.97|\n\n\n**5. There are many repeated sentences across the whole paper.**\n\nThanks for the feedback. We’ve removed redundant sentences throughout the paper, particularly between the introduction and experiments. \n\n**6. Not sure what “3-shot CLIP model” refers to in line 214.**\n\nThe 3-shot CLIP model refers to a CLIP model trained with 3 examples per class which are randomly sampled from the training set. In other words, a Neural Primed model with no training examples performs as well as a standard model trained on 3 examples per class. We have revised the paper to make this more clear to readers. \n\n[1] Learning Customized Visual Models with Retrieval-Augmented Knowledge\n\n[2] K-LITE: Learning Transferable Visual Models with External Knowledge\n\n[3] Internet Explorer: Targeted Representation Learning on the Open Web\n\n[4] Tent: Fully Test-time Adaptation by Entropy Minimization"}, {"review_id": "mrWzgOcIc5", "reviewer": "Reviewer_89m2", "summary": "This paper presents a novel approach to zero-shot learning in vision-language models. The authors propose a new tunning schema using prior knowledge in the CLIP model. First, the authors construct a priming pool, which consists of clusters of retrieval images by the CLIP model. Each cluster collects a specific class by searching all text-image pair owning that class name in the text. The authors then tuned this priming pool using the test set. For each image in the test set, the 10 best matching images are selected and de-duplicated to make up the final priming pool. The authors claim that such an operation can adapt the priming pool to the test domain. At last, the authors tune the CLIP model by adding a linear head at the end of the model. The linear head is the weighted sum of a zero-shot trained head and NCM of all retrieved examples per category. Experiments show a 2.45% improvement on ImageNet and a 3.81% improvement on average across some traditional transfer learning benchmarks. Overall, this paper provides valuable insights and promising directions for future research in this field.", "questions": "1. Will the method improve finetuning efficiency of other prompt tuning methods? I mean after applying it, how many training times or steps can be saved for prompt-tuning?\n2. Is the method stable among different tasks? \n3. How is the influence of the filtering step? If remove it, how much accuracy will lose? Also, is there any ablation on the choice of $\\alpha$?\n\n\n\nThe following are suggestions rather than comments, the authors may take them advisedly.\n1. **Notations.** I am in favor of notations used in SimCLR, which are much clear to read and easy to follow as they use different font styles for different concepts. I would suggest a slightly modified version as follows: sets, using \\mathcal command like $\\mathcal{S}$; neural networks, using bold font with parameter subscript, like $f_{\\theta}$, number field, like the real number set, using $\\mathbb{R}^d$, integers using $\\mathbb{N}$; loss function, using \\mathcal command with subscript the \\mathrm environment, like $\\mathcal{L}_{\\mathrm{CLIP}}$; scalars using plain font and vectors using bold fond to distinguish. Currently, the notations carry only limited meaning, make readers need to look back to their definitions several times to remember them. For example, Line 130 uses $\\mathcal{R}^d$ for real numbers while Line 178 uses $\\mathbb{R}^{d\\times n}$, V,P,B,L stand for model, data pool (while training data using font $\\mathcal{D}$), data clusters, and model but using the same font style.  I also recommend not to use L as a notation for the model, it can be confused with the loss function.\n\n2. **Fontsize of figures.** Currently fontsizes of figures are set carelessly. Fig. 1 using a giant fontsize, much larger than the context, seems not necessary, while some words in Fig.2 are too small to tell. I also recommend you to use some images to tell readers the meaning of each component in Fig. 1, like putting some training images above the pretraining data, and putting some images from the priming pool beneze the priming set, using a neural network model to replace the CLIP block and color those parameters that need to be tuned.\n\n3. **The hyphen.** In the title you use **Data-Efficient**, words connected by the hyphen are in capital. While in **Zero-shot**, **Nearest-class-Mean**,  they are not.", "limitations": "1. The method needs to know all test set to filter the priming set, online learning may exceed the scope of it.\n2. Filtering step may consume considerable time and computation resources, not affordable for ordinary researchers.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "# Novelty:\nThis paper provides a novel perspective of aligning images by the class name directly. Previous methods are often trapped in playing with feature representations and embeddings. None has looked into whether using strings directly, is a good practice. This work fills this unseen point with good improvements. The method can leverage massive training data, to help the model adapt better to specific tasks.\n\n# Clarity:\nThis paper is extremely well-written and easy to follow.", "weaknesses": "# Novelty:\nThe key innovation of this work can be split into three parts, 1) **class name retrieval to construct priming pooling**, 2) **filtering the priming pooling using cosine similarity with the test set**, and 3) **tuning the CLIP model by adding a linear head using NCM of the priming pooling**. So I will give my opinion to them separately as follows. \n\n1. **Retrieval**:\n\na) **Retrieval is not a new idea in transfer and zero-shot learning.** I think using retrieval is not new in both domains of transfer learning or few-shot\\zero-shot learning, like in NeurIPS2021,2017 paper \"Re-rankingforimageretrievalandtransductive few-shot classification\", \"Few-ShotLearningThroughanInformation RetrievalLens\", there may be too many to list. \n\nb) **String retrieval  I think is not powerful enough.** So the new stuff is using the class name as the retrieval feature. This is somewhat too direct for me. Is there a possibility that CLIP representations combining the string retrieval perform better? According to my experience, this is usually the case. The authors do not discuss the possibilities of using other tools, instead, they come out with a belief that string retrieval is the fastest one so it is superior to other methods, which I may respectfully disagree. First, time efficiency should not be a major problem in constructing the priming pool, after all, it is not required to be online learned or real-time computed. In your pipeline, you first get the naive priming pool from the training data. Then you prune it to preserve the top 10 similar images with each image of the test data. This step won't consume more time if you use semantics as you need to go through all the data in the priming set either way. Then at inference time you only need to compute the mean of each class, there is no need for exact string matching so it also won't consume more time if you use semantics or other stuff. So using semantics or not won't influence the inference time at all, I can't understand why only string is used. Perhaps some experiments comparing those retrieval methods can be helpful to demonstrate the advantages of this work.\n\nc) **Potential conflicts.** Another problem of only using string retrieval is that it only conflicts with each other. Like the class \"guinea pig\" is not a pig at all, \"wolf spider\" is not a wolf, and \"sea snake\" may not contain sea in the image. I am interested in how this method can tell the \"guinea pig\" category (339 in ImageNet) and the \"pig\" category (342 in ImageNet)? On the hand, combining string with semantics can handle this case, as the embedding of guinea pigs will be more similar to the mouse's. Also, I don't understand why the results of string retrieval will be more relevant, can the authors explain why?\n\n2. **Filtering the priming set.** I think the authors can try combining CLIP similarity with Pixel similarity in ranking, inducing more metrics to cooperate. In transfer learning and few-shot learning, such methodology is usually adopted.\n\n3. **NCM head.**  This step uses the final outputs of the language model to predict the mean. Why not also use the output of the Image encoder? Or how about the intermediate representations of the CLIP model? I think the authors can add an explanation of why the output of the language model is the best. Also, tuning a new head is not a new thing. It is popular in incremental learning, transfer, and zero-shot learning. But it is not necessary to limit your imagination in here. Many works have discovered some other parts of the CLIP model, like attention layers, and convolutional kernels, MLPs have rich semantics and an important role in making predictions. If the authors can discuss why they choose to add a linear head instead of tuning other parts, I think this section can be more convincing.  For example, the authors can compute the gradients of each parameter when input new examples of test sets, and find out that the terminal linear layer has the most significant gradients thus influencing the results on new examples the most, so tuning them is vital and more efficient than tuning other parts.\n\nOverall, the three parts are a bit split and there seem no strong connections among them, at least not provided by the authors. They seem like A+B+C. If the authors can reorganize them, and engage them together with one purpose of enhancing transfer learning capability, I think this paper can be more fluent.\n\n\n# Experiments:\n1. **No errors and error bars.** While it is recommended by NeurIPS author guidelines, papers in this domain seem seldom report error bars. I think it could enhance the paper as some figures are close, and errors can help us understand the true advantage of the proposed method. As in Tab. 1, in ImageNet, FGVC, Food101, and SUN397, the differences are smaller than 0.5. We may suspect it within the variance of measures. Also in Fig.3, performance is close, error bars can further tell the differences between methods. \n2. **Why no LAION-5B?** I see many relevant papers conducted on the LAION-5B dataset. LAION-2B is a new one and includes it very well. However, omitting LAION-5B may be hard to align the results of this paper to previous work.\n3. **Ablation is not enough.** Like in the novelty part, the superiority of the chosen components to their alternatives need to be clarified by the experiments. This will make this submission stronger.\n4. \"\"Fair comparason.** As you add a head to the model, you may control the equivalence of all parameters and training steps of all methods. This is important when performance is close.\n\n\n\n# Minor issue:\n1. Line 143 \"However,143 withneuralpriming,prioritizingprecisionoverrecalliscrucial,consideringthesize,diver\", better to add references.\n2. Line 164, the last \"for\" is redundant.\n3. It is not very clear whether Neural Priming is another name of this method or the task this method tries to handle. Better clarify and unify it throughout the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review. We hope our response addresses your concerns. Due to the character limit during the rebuttal phase, we’ve focused on the most important concerns and will address the remaining comments at the start of the discussion period. \n\n**Using retrieval is not new in domains of transfer, few-shot, and zero-shot learning like in [1], [2].**\n\nThanks for pointing us to these works. We have added them to our paper with discussion. The use of retrieval in general is not new, but these works differ significantly from Neural Priming in both overall objective and method details. The goal of our work is to efficiently improve vision-language models by leveraging their large-scale pretrain dataset. [1] formulates a new optimization based on information retrieval for training small convolutional models from scratch. Also, the method requires human annotated labels, whereas Neural Priming can leverage web captioned data. [2] formulates a training objective for learning a better similarity graph. Similar to [1], [2] uses human annotated images and is focused on better pretraining from small-scale datasets.\n\n**String retrieval is not powerful enough. Authors should try combining CLIP similarity with Pixel similarity in ranking.**\n\nYes, that is exactly what Neural Priming does (line 156-159). We do a multi-stage filtering and string retrieval is just the first phase. After the initial filtering phase, the more expensive techniques of CLIP similarity or image embedding similarity (transductive setting) are used. \n\n**Time efficiency should not be a problem in constructing the priming pool.**\n\nAgreed, time-efficiency is not a major concern for the priming pool. However, performing exact nearest neighbor search over large-scale datasets such as LAION-2B is infeasible on current hardware. Other works [3] use approximate nearest neighbors, but this leads to empirically worse results as can be seen from the retrieval ablation study in Table 1 of this response and in Table 1 of the main paper. \n\n**Experiments comparing retrieval methods can demonstrate the advantages of this work.**\n\nThe baseline of retrieval+fine-tuning uses approximate nearest neighbors search. We found that the approximate semantic search retrieved many images that did not fall into the target class. Below we’ve included a table showing the comparison of different retrieval methods and have added it to the paper.\n\nTable 1. Ablation Over Filtering Methods\n|| ImageNet | Stanford Cars | FGVC-Aircraft | Flowers102 |\n|:-:|:-:|:-:|:-:|:-:|\n| CLIP| 68.3| 87.4|25.86| 71.65|\n| Retrieval + Fine-Tune|70.28|87.95|26.22|72.15|\n| Neural Priming (No CLIP Filter)| 70.35| 88.19| 31.56|76.91|\n| Neural Priming (No Label Alignment)|70.64| 88.38| 30.38|77.18|\n| Neural Priming| **70.75**| **89.3**| **33.03**| **79.81**|\n\n**Using string retrieval can cause conflicts.The class \"guinea pig\" is not a pig at all, \"wolf spider\" is not a wolf\".**\n\nThis is a good observation and a major reason we introduced CLIP similarity to further filter these images (section 3.1.1). In Figure 1 of our rebuttal PDF, we show random examples of images removed by the CLIP similarity filter for the \"pig\" class and the remaining \"pig\" images in the priming pool. From Figure 1, the CLIP filtering step removes essentially all of the guinea pigs. We will add this motivation and figure to the paper. \n\n**NCM uses the outputs of the language model, why not also use the output of the Image encoder?**\n \nWe do use the output of the image encoder to construct the centroid for NCM.  On line 175-176: *we perform nearest-class mean (NCM) on all retrieved examples per category to get a pool specific classification head*. \n\n**No errors and error bars. The differences are smaller than 0.5. Fig. 3 has no error bars.**\n\nFor Table 1, zero-shot and transductive results do not have error bars because zero-shot inference is deterministic. Also, the improvement on ImageNet over CLIP is 3.08% which is considerable for this benchmark.\n\nWe provide confidence intervals (CI) for the few-shot experiments in Table 8 (section I of the appendix). The CI for ImageNet is .05% and the average improvement across all shot numbers is 1.63%. For Figure 3, the error in table 8 is so minimal that it is not visible in the plot. Average improvement is 3.57% across the 4 datasets listed, and average CI is .103%. \n\n**The superiority of the chosen components to their alternatives need to be clarified.**\n\nIn Table 1 of this response we’ve presented an ablation study over the various retrieval components in Neural Priming which we’ve added to the paper. \n\n**As you add a head to the model, you may control the equivalence of all parameters and training steps of all methods.**\n \nAll methods we benchmark against require a linear head, therefore Neural Priming has the same or fewer new parameters. Our method does not use gradient steps and is ~155x more efficient on average in the number of training FLOPS than REACT [3], TPT [4], and CoOp [5]. \n\n**Filtering step is not computationally affordable for ordinary researchers.**\n\nWe agree that accessibility and computational efficiency are important. One motivation for Neural Priming is to enable others to leverage large-scale data sources without needing massive compute budgets. Neural Priming can be run on a single machine with a 3090 TI and 1 TB SSD. Comparable methods such as REACT [3] require retrieving and fine-tuning on 10 million auxiliary images (100,000 times more images than Neural Priming) which we agree can be prohibitively expensive.\n\n[1] Few-Shot Learning Through an Information Retrieval Lens\n\n[2] Re-ranking for image retrieval and transductive few-shot classification\n\n[3] Learning Customized Visual Models with Retrieval-Augmented Knowledge\n\n[4] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models\n\n[5] Learning to Prompt for Vision-Language Models\n"}, {"review_id": "846oc4xvTt", "reviewer": "Reviewer_Ekvm", "summary": "The paper proposes a technique for adapting large pre-trained models to distribution shifts and downstream tasks given few or no labeled examples named Neural Priming. The method can be used at test time on very large scale datasets. Experiments on various datasets show the effectiveness of Neural Priming.", "questions": "The questions here are for open discussion.\n1. I wonder if all the filtered priming pools for a specific downstream task are integrated into one dataset, will it be a better pretraining/finetuning dataset (excluding noises)?\n2. Is there a possibility that some samples are harmful (e.g. low quality or wrongly labeled), but selected into the priming pools or even the filtered priming pool?", "limitations": "Limitations are discussed in terms of the method itself. But issues on broader aspects such as model safety, data leakage are not included. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Neural Scaling method is novel, with a good motivation (diversity of pretraining datasets introduces competing objectives) and clear presentation.\n2. The technical details of Neural Scaling is simple yet effective: Classical ideas of information retrieval, clustering and prototype calculation were used. This may possibly open a line of research on advancing the proposed techniques.\n3. Neural Scaling has high efficiency on saving the time of doing linear probing. ", "weaknesses": "1. In line 182, what is P and \\sigma? The main text should include their meanings rather than putting them in the appendix. Also, ablation studies/discussions can be included for the choice of \\sigma.\n2. I am not familiar with the performance of the compared baselines e.g. VLM (whether these methods are strong and competitive), thus I cannot determine to which extent Neural Priming outperforms other relative methods, especially SOTA ones. While this do not effect the paper's advantage in novelty, it does draw questions on the \"absolute\" performance of Neural Priming. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough and thoughtful review. We hope to address your concerns below and are happy to engage in further discussion. \n\n**Weaknesses:**\n\n**1. In line 182, what is P and $\\sigma?$ The main text should include their meanings rather than putting them in the appendix. Also, ablation studies/discussions can be included for the choice of $\\sigma.$**\n\nP is the size of the priming pool, and we’ve made that more clear in the text. The formula for the mixing coefficient, $\\alpha$ can be derived by considering the language embedding as a Gaussian prior and updating the prior with examples from the priming pool. $\\sigma$ is the certainty over the language embedding prior. We’ve included an ablation over the choice of $\\sigma$ below. \n\n| $\\sigma$ | ImageNet | Stanford Cars | FGVC-Aircraft | Flowers102 |\n|:-:|:-:|:-:|:-:|:-:|\n| 1      | 68.9     | 87.54         | 25.9          | 71.94      |\n| 10     | 69.1     | 88.98         | 29.75         | 74.97      |\n| 100    | 70.75    | 89.3          | 33.03         | 79.81      |\n| 1000   | 70.64    | 89.11         | 31.32         | 78.81      |\n\n\n\n\n\n\n\n\n**2. I am not familiar with the performance of the compared baselines e.g. VLM (whether these methods are strong and competitive), thus I cannot determine to which extent Neural Priming outperforms other relative methods, especially SOTA ones. While this do not effect the paper's advantage in novelty, it does draw questions on the \"absolute\" performance of Neural Priming.**\n\nVLM [1]  is a recent paper published at ICLR 2023 which to our knowledge is state of the art for zero-shot along with CuPL [2]. Based on the recommendation from Reviewer XXJ2 we’ve added the TPT+CoOp and TENT [1] baseline for the transductive and few-shot setting which can be found below. If there are other methods that should be considered we’d be happy to compare with them.\n\nTop-1 Accuracy on Few-Shot ImageNet \n|             | 1-Shot  | 2-Shot  | 5-Shot  | 10-Shot |\n|:-:|:-:|:-:|:-:|:-:|\n| TPT+CoOp       | 66.97   | 70.04   | 71.02   | 71.89   |\n| Neural Priming | **71.56** | **71.81** | **72.03** | **72.36** |\n\nComparing with TENT in Transductive Setting\n\n|| ImageNet-V2| ImageNet-Sketch |\n|:-:|:-:|:-:|\n| CLIP| 59.35|57.05 |\n| TENT|59.91|57.40|\n| Transduct. Priming |60.76| 59.97|\n\n**Questions:**\n\n**1. I wonder if all the filtered priming pools for a specific downstream task are integrated into one dataset, will it be a better pretraining/finetuning dataset (excluding noises)?**\n\nIf we understand correctly, the question is if we take all of the priming pools across downstream tasks and combine them, will it boost accuracy in general. That's an interesting idea. With the 11 tasks we evaluate in the paper, the diversity of classes is likely too small for generalizing to new tasks (i.e. most classes for a new task would not be in our combined priming pool). If hundreds or thousands of tasks were used to create a combined priming pool, it's quite possible that it could produce a fine-tuning dataset that would be more accurate for new tasks. \n\n\n**2. Is there a possibility that some samples are harmful (e.g. low quality or wrongly labeled), but selected into the priming pools or even the filtered priming pool?**\n\nYes, during our experiments we found that using the priming pool before filtering with CLIP similarity score (zero-shot setting) or image similarity (transductive setting) can decrease accuracy. \n\n\n**Limitations:**\n\n**1. Limitations are discussed in terms of the method itself. But issues on broader aspects such as model safety, data leakage are not included.**\n\nWe agree and have added content on model safety to the limitations section. We recognize that biased class names could lead to harmful images being retrieved to the priming pool and lead to unintended behavior of the model. We also agree that data leakage is an issue with large-scale models pretrained on data scraped from the internet. We suspected there might have been data leakage so we ran deduplication between the images in the priming pool and images in the test sets, but found no overlap. We contend that data leakage is more of an issue with the pretraining set since Neural Priming does not introduce any new data. \n\n[1] Tent: Fully Test-Time Adaptation by Entropy Minimization"}, {"review_id": "xuQFsb1lsX", "reviewer": "Reviewer_AXQG", "summary": "This paper introduces neural priming, a retrieval based approach for adapting large pretrained models to downstream tasks. Neural priming first collects a priming pool of pretraining data relevant to the downstream task using natural language descriptions, then at test-time the priming pool is further filtered by keeping the top-k most similar pretraining samples to the test samples w.r.t. cosine similarity of the CLIP embeddings. From this filtered pool, a classification head is set to the nearest-class mean and averaged with the CLIP ZS text embeddings (similar to WiSE). Across several classification datasets, neural priming is shown to improve over existing prompt-tuning and test time adaptation methods in the zero-shot and few-shot settings, and is complementary to prompt-tuning methods.  ", "questions": "Does the CLIP ZS baseline in Table 1 just include the class name or does it also include context (i.e. 'a photo of a {} flower' instead of 'a photo of a {}'\")?", "limitations": "One big limitation that I did not see mentioned was the time it takes to retrieve the priming pool from the original pretraining data, especially for large datasets like LAION 2B. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem of fitting pretrained classifiers to downstream tasks effectively, specifically models like CLIP, is of great interest to the ML community and the retrieval based approach of neural priming is both simple and a pleasant departure from current robust finetuning methods. Overall the paper was well-written, easy to follow, and the evaluation shows consistent improvements over existing prompt tuning and retrieval-based transfer learning methods. I also enjoyed the open questions posed in the discussion section on why retrieval methods work well for transfer learning which I could see sparking future work in this area. \n\nWhile the method is simple and there are some eval limitations that I would like addressed, I think this paper is of value to the community and is above the acceptance threshold. ", "weaknesses": "Given the simplicity of the method and the vast number of related works, my critiques mainly lie in differentiating from related works more and adding in some simple yet relevant baselines. \n\n**Related works**\n\nMy main critique with the related works section is the lack of differentiation from test-time training/adaptation and transductive learning in section 2.3, as well as few-shot learning in section 2.4. \n\nNot a critique and more of a suggestion, but a recent paper[4] explores how removing the least relevant items in a pretraining dataset can improve transfer learning accuracy and I believe would be good motivation for neural priming. There are also some robust CLIP finetuning methods[1] and prompt tuning methods[2,3] that should be mentioned in related works. \n\n[1] [Using Language to Extend to Unseen Domains](https://arxiv.org/abs/2210.09520) \\\n[2] [Visual Prompt Tuning for Test-time Domain Adaptation](https://arxiv.org/abs/2210.04831) \\\n[3] [Conditional Prompt Learning for Vision-Language Models](https://arxiv.org/abs/2203.05557) \\\n[4] [A Data-Based Perspective on Transfer Learning](https://arxiv.org/abs/2207.05739)\n\n**Evaluation**\nHere the main thing that is lacking is a description of the baselines. There is some brief descriptions of CoOp (without a citation) but I found it hard to interoperate exactly what these baselines were. \\\nWhy not compare to CLIP LP or WiSE for the few shot setting? It seems like that would be a natural baseline (even just setting the ZS weights to the few shot examples, from what I understand the NCM baseline uses the priming examples). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough and thoughtful review. We appreciate your insightful feedback, and hope to address your concerns below. \n\n**Given the simplicity of the method and the vast number of related works, my critiques mainly lie in differentiating from related works more and adding in some simple yet relevant baselines.**\n\nThanks for the feedback. We recognize that the discussion in related work could have provided better context for Neural Priming. We have added further related works and added discussion about the differences with Neural Priming. \n\nTo our knowledge, the most closely related work is REACT [1]. This paper retrieves images from an external data source, and fine-tunes with a gating mechanism. First, Neural Priming differs in that we keep the base architecture fixed whereas REACT adds additional parameters. Second, Neural Priming shows that performance can be improved by finding subsets of the original pretraining data, whereas REACT [1] and previous methods [2,3] add data from external sources. Third, Neural Priming is magnitudes more efficient and can be on-the-fly for transductive learning and domain-shift.\n\nFor few-shot works we’ve added the following works with discussion based on feedback from reviewer 89m2 and further literature review. \n- Atlas: Few-shot Learning with Retrieval Augmented Language Models, \n- Training-free name-only transfer of vision-language models\n- Few-Shot Learning Through an Information Retrieval Lens\n\nFor transductive learning we’ve added the following with discussion:\n- Re-ranking for image retrieval and transductive few-shot classification.\n- ZegCLIP: Towards Adapting CLIP for Zero-Shot Semantic Segmentation\n\n**Related works section lacks of differentiation from test-time training/adaptation and transductive learning, as well as few-shot learning.**\n\nWe realize the difference between test-time training and transductive learning was not well delineated in the related works. We’ve better explained that test-time training is online and typically handles one test example at a time whereas transductive has access to the entire test set. We’ve also added discussion to make it clear that few-shot differs in that the model has access to a few labeled training examples and no access to the test set. \n\n**The main thing that is lacking is a description of the baselines. There is some brief descriptions of CoOp (without a citation) but I found it hard to interpret.**\n\nThanks for the feedback, we’ve added the citation for CoOp [4] in the experiments section and added details for the NCM and CoOP baseline. CoOp is a popular few-shot method for language-vision models which learns context vectors for each class given a set of labeled examples. In other words, rather than hand-engineering templates such as \"a photo of {}, a type of flower\", CoOp learns the templates from few-shot data. Based on your later comment, we’ve also added WiSE-FT as a baseline and clarified that NCM uses the few-shot examples with weight ensembling. We elaborate further on the details of NCM below and have added them to the experiments section. \n\n**Questions:**\n\n**1. Does the CLIP ZS baseline in Table 1 just include the class name or does it also include context (i.e. 'a photo of a {} flower')?**\n\nYes, we’ve used the openAI templates such as “a photo of {}, a type of flower”, “a photo of {}, a type of aircraft”, etc. We’ve added this detail to the experiments section.\n\n**Limitations:**\n\n**1. One big limitation was the time it takes to retrieve the priming pool from the original pretraining data.**\n\nOne of the major advantages of Neural Priming is the speed compared to similar methods. In figure 7 of the appendix and figure 2 of the rebuttal pdf we plot the wall clock time for retrieving the 1000 classes of ImageNet from LAION-2B. In total to retrieve 1 million images from LAION-2B it takes ~10 minutes on one rtx-6k and evo 970 SSD. For comparison, to retrieve the 10 million images required for REACT [1] using approximate nearest neighbors (FAISS) with text-to-image similarity on the same hardware takes ~72 hours. \n\n\n**2. Why not compare to CLIP LP or WiSE? From what I understand the NCM baseline uses the priming examples.**\n\nTo clarify, the NCM baseline uses the few-shot training examples, not the priming examples. Also NCM uses the same weight interpolation between the centroid of the few-shot examples and the language classifier as Neural Priming. We also performed standard fine-tuning with weight ensembling and found that NCM with weight ensembling performed better, which has been observed for few-shot settings [5,6]. We realize this wasn’t clear, and have added details to the experiments section. We’ve also added the WiSE-FT baseline to the paper and the results on ImageNet can be found below. \n\nTop-1 Few-Shot Accuracy on ImageNet\n\n| Shots | Neural Priming | NCM  | Wise-FT |\n|-|:-:|:-:|:-:|\n| 0     |     71.38      | 68.3 |  67.75  |\n| 1     |     71.56      | 68.89|  68.14  |\n| 2     |     71.81      | 68.85|  68.25  |\n| 5     |     72.03      | 69.12|  68.78  |\n| 10    |     72.36      | 70.12|  69.02  |\n\n[1] Learning Customized Visual Models with Retrieval-Augmented Knowledge\n\n[2] K-LITE: Learning Transferable Visual Models with External Knowledge\n\n[3] Internet Explorer: Targeted Representation Learning on the Open Web\n\n[4] Learning to Prompt for Vision-Language Models\n\n[5] Prototypical Networks for few-shot learning\n\n[6] Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?\n\n\n"}]

Author rebuttal:
We thank the reviewers for their thorough, insightful comments and have made revisions based on their feedback. We are glad they found the work well motivated, novel, and the contributions to be of value to the community. We have included a PDF with figures which we reference in our responses. Below are the individual responses to each reviewer which we hope address their concerns. We are happy to engage in further discussion and interested in any additional feedback the reviewers may have. 

Meta-review: This paper received diverging reviews. Reviewer 89m2 expressed concerns regarding novelty and experiment, whereas the rest 3 reviewers acknowledge the contribution of this work with unanimous acceptance. Upon reading through the paper and all the discussions, the AC feels that this work does contain enough contributions to be accepted, and the proposed method seems interesting. The authors did a good job addressing the concerns while reviewer 89m2 did not follow up with more questions. In light of this consideration, the AC recommends acceptance.
Decision: Accept (poster)