Title: Neural Priming for Sample-Efficient Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Transfer Learning

Distribution Shift

Test-Time Training
Abstract: We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks given few or no labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model to recall and conditions its parameters on relevant data seen throughout pretraining, thereby priming it for the test distribution. Neural Priming can be performed at test time in even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a 2.45% improvement in accuracy on ImageNet and 3.81% accuracy improvement on average across standard transfer learning benchmarks. Further, using our test time inference scheme, we see a 1.41% accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the common challenge of limited labeled data and changing distributions. Code and models are open-sourced at [https://www.github.com/RAIVNLab/neural-priming](https://www.github.com/RAIVNLab/neural-priming).

Neural Priming for Sample-Efficient Adaptation

Abstract

We propose Neural Priming , a technique for adapting large pretrained models to distribution shifts and downstream tasks given few or no labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model to recall and conditions its parameters on relevant data seen throughout pretraining, thereby priming it for the test distribution. Neural Priming can be performed at inference, even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a 2 . 45% improvement in accuracy on ImageNet and 3 . 81% accuracy improvement on average across standard transfer learning benchmarks. Further, using Neural Priming at inference to adapt to distribution shift, we see a 1 . 41% accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the challenge of limited labeled data and changing distributions. Code is available at https://github.com/RAIVNLab/neural-priming .

1 Introduction

Humans have a vast store of prior experience which we draw on to flexibly perform a diverse range of tasks [20, 5, 4, 12]. While engaging in an activity, we naturally retrieve relevant information or schema in a cognitive phenomena known as Priming [34]. This process ensures that necessary knowledge is readily accessible in memory, leading to enhanced performance for the task at hand [43]. Pre-trained, general-purpose models such as CLIP [40] and ALIGN [23] have extensive prior knowledge learned from large-scale, diverse datasets. These datasets seek to capture all natural variation in real data within their distribution. Can these models also benefit from something like priming? We observe that models trained even on the largest of such datasets often substantially improve in performance when fine-tuned on task-specific data. This begs the question of what the model learns from fine-tuning on the target dataset, if it already trained on many similar examples during pre-training.

We speculate that the effect of fine-tuning a pre-trained model on task-specific data is similar to that of priming. Given the sheer size and diversity of the pre-training dataset it becomes challenging for the model to find a consistent solution that is optimal for all subsets of the data. This becomes particularly evident for open-vocabulary models such as CLIP, where multiple natural language descriptions can correspond to a single image, highlighting the challenge of accommodating diverse interpretations. We hypothesize that training on the downstream dataset re-aligns the model to the specific objective.

With this in consideration, we propose Neural Priming . Specifically, Neural Priming recalls a subset of the pre-training data similar to the target distribution, re-aligns the natural language

Maltese

Poodle

Pre-training Data

p

Recall

Attunement

German

Shepherd

Stimulus

CLIP Model

descriptions to the downstream task, and quickly adapts the model to the subset. We perform extensive experiments on 7 transfer learning and 4 distribution shift datasets to validate our method. We use the OpenCLIP [40, 52] ViT [8] set of models pre-trained on the LAION-2B and 400M [44]. We find Neural Priming leads to significant accuracy improvements, particularly when labeled data is scarce and in specialized domains. Concretely, Neural Priming improves accuracy by 2.45% on ImageNet and 4.25% on average across the other 6 datasets over the base CLIP model. In the few-shot setting, Neural Priming improves accuracy by 3.81% on average over recent methods on standard transfer learning benchmarks. We show Neural Priming is efficient and can be performed on-the-fly. For datasets containing more than 2 billion images, we can prime our model to ImageNet in less than 2 minutes with a single commercial GPU.

Neural Priming is flexible and can be used with variable degrees of information about the downstream distribution. When the model has language-only task descriptions, our approach can efficiently retrieve a priming pool of relevant examples from the pre-training set and attune the model to this data. At inference time, given a set of test images to classify, Neural Priming is able to use test examples to adapt the priming pool to distribution shifts. When we have access to training examples in the few-shot setting, Neural Priming can filter the priming pool to align with the training distribution.

We make the following contributions:

We introduce Neural Priming , a novel method that leverages retrieval from the pre-training dataset for efficient and accurate adaptation of large, pre-trained models to downstream tasks.

Neural Priming achieves state-of-the-art zero-shot and few-shot accuracy on the standard transfer learning datasets - up to 4 . 25% and 3 . 81% improvements respectively over baselines (Section 4.2).

Neural Priming also enables transductive learning and improves performance on standard distribution shift datasets by 2 . 51% on average, all without using any additional data (Section 4.3).

Our approach generalizes to various architectures and pre-training datasets while being complementary to techniques [33, 39] that improve zero-shot performance of open-vocabulary models.

2 Related Work

2.1 Open-Vocabulary Models and Zero-shot Inference

Open-vocabulary models have proven to be an effective approach for transfer learning. Such models enable training on vast amounts of web-scale images without the need for labor-intensive human

labeling by leveraging pre-existing natural language descriptions [40]. Open-vocabulary models have set state-of-the-art on ImageNet [7] as well other transfer learning benchmarks [53, 1, 41].

Open-vocabulary models offer additional capabilities beyond standard pre-trained models. They can perform zero-shot inference, where predictions are made without training on target data. Additionally, they are robust to distribution shifts [40], enable prompt-tuning methods [39, 33], and can be used for text-based retrieval [10]. Zero-shot can have different meanings in the literature. In the context of this paper, we consider zero-shot as the experimental setting in which the model receives no training examples drawn from the training distribution.

Prompt-tuning has emerged as a popular research direction in the domains of large language and open-vocabulary models. In the context of open-vocabulary models, prompt-tuning can involve modifying the textual prompts or queries used during the training or inference of the model to improve its understanding of visual content or achieve specific goals. In the original CLIP paper, Radford et al. [40] design hand-crafted prompt templates for ImageNet and other transfer learning datasets and show that this leads to substantial accuracy improvements. More recently, other work [33, 55] has used machine learning approaches to learn the prompts rather than hand-crafting them.

2.2 Distribution Shifts

Robustness to distribution shift is a key property of good machine learning models as it represents a notion of reliability. In particular, studies on natural distribution shifts, including ImageNet-V2 [41], ImageNet-Sketch [51], ImageNet-R [18], and ImageNet-A [19], find that models have a consistent performance drop when exposed to a distribution at inference time not seen during train time [49]. In order to focus on robustness and eliminate the confounder of better models being generally better, this performance gap is measured through effective robustness, which is the robustness improvement over ImageNet trained models. Prior work has shown that the performance of models on in distribution and out of distribution is highly correlated across many algorithmic training interventions, except for cases where training on larger and more diverse datasets increases robustness [35].

The most significant recent improvement in effective robustness [41] is the introduction of openvocabulary models. At its time of release, CLIP [40] achieved unprecedented effective robustness on a variety of distribution shifts. Studies have suggested that these models achieve high effective robustness through their data distribution [9], a result of training on large amounts of web-scraped data. However, these models are still worse at downstream tasks than models fine-tuned on in-distribution data. Moreover, fine-tuning on downstream data causes robustness on other data distributions to deteriorate [40, 52]. Many mitigation methods have been proposed to such as Wise-FT, FLYP, LP-FT, and model surgery [52, 15, 28, 29]. Our paper differs from these methods in goal: whereas they seek to keep model robustness while gaining the benefits of fine-tuning on task-specific data, we seek the benefits of fine-tuning while not collecting any in-distribution data . Hence these methods are complementary to Neural Priming, and we employ Wise-FT in our model attunement procedure.

2.3 Transductive Learning

Transductive learning [13, 6] focuses on leveraging unlabeled data during inference. It differs from traditional supervised learning, which solely relies on labeled data at train time. Related to transductive learning is test-time training [48, 14, 45]. Test-time training involves adapting and refining the model's predictions based on the specific testing examples encountered. Transductive learning differs from test-time training in that test-time training only considers one test sample at a time, whereas transductive aims to learn from the entire test set.

2.4 Few-Shot Learning

Few-shot learning research aims to addresses the challenge of learning from a limited number of labeled examples. In many real-world scenarios, acquiring large labeled datasets is impractical or costly. Older lines of work have focused on meta-training small models [47, 11, 37, 22] on smallscale datasets. More recently, the approach for few-shot learning has shifted towards training large, general-purpose models such as CLIP [40] and ALIGN [23] on web-scale datasets.

2.5 Retrieval-Augmented Models

In language, works have demonstrated the effectiveness of retrieval from text corpora or structured data for tasks such as question answering [3, 16, 25]. In general, these methods seek to recover facts either from a large corpus or knowledge graph, then use those to complete tasks. This differs from our scenario, where exact examples at inference time do not necessarily exist in the pre-training corpus. REACT [30] and SuS-X [50] are retrieval-augmented methods for open-vocabulary models which use search to fine-tune with relevant examples [30]. We differ from Liu et al. [30] in that they add a substantial number of new parameters whereas we do not. Additionally, our approach is significantly more efficient, both computationally and in terms of number of samples, enabling use at inference for additional improvement (Section 3.1.2). We differ from [50] in that their work uses semantic retrieval whereas Neural Priming leverages language for fast initial filtering and image search for accurate retrieval. Further, Neural Priming shows that models can improve by revisiting examples seen throughout pretraining whereas other works retrieve new examples from external datasets.

3 Method

Neural Priming is the process of retrieving relevant information from the pre-training dataset and leveraging it for a specific task. We study it in the context of vision-language contrastive pre-training, so the form our task description takes is a set of class names, C , already in natural language. A CLIP model [40] consists of a vision embedding model, V , and a language embedding model, L , each producing a vector representation in R d . The pre-training dataset, D , consists of a large number of image-text pairs collected from the web. The text component can be noisy, potentially containing irrelevant or inaccurate information about the image content.

We break our method down into two main steps: 1. Collecting the priming pool, where we gather data from our pre-training dataset relevant to a particular task and 2. model attunement, where we leverage this data to improve our model.

3.1 Collecting the Priming Pool

3.1.1 Leveraging Natural Language Task Information

The goal of this step is to collect an initial pool of images relevant to the task at hand given the previously defined natural language description C . For example, if our task is a set of dog breeds, ideally we would collect sets of images belonging to those breeds and label them accordingly. A simple way to prime is by using retrieval to gather relevant data points from our pre-training dataset. An existing method for language-based retrieval involves using the CLIP text embedding of a class description c ∈ C for retrieval using semantic similarity scores on the pre-training set [2, 42]. However, with neural priming, prioritizing precision over recall is crucial, considering the size, diversity, and noise of the pre-training dataset. This form of semantic retrieval has a major downside: it is not clear where to threshold similarity scores to retrieve the most relevant images. Threshold too late and we allow unrelated images to be included in our pool. Further, this threshold is often specific to a category, making it infeasible to search at scale.

Our approach to language-based priming is to search for the existence of the class name, c ∈ C , in the captions of our pre-training dataset to retrieve images relevant to a particular category. We organize these image-text pairs into separate categorical clusters { B c } according to the class name c mentioned in their captions. This approach has a few advantages over semantic retrieval: 1. After setting up an inverted index search structure for text retrieval [26, 46], exact string matching is far faster than semantic retrieval, even when approximate nearest neighbor strategies are employed [24], 2. with exact string search, the category boundary is clear and therefore does not require per category tuning, 3. the retrieval results are overall qualitatively more relevant. Finally, to leverage the semantic understanding of our CLIP model, we filter the priming pool using CLIP similarity score. We do this by constructing a 'zero-shot' CLIP classifier, as defined by Radford et al. [40], and removing examples from categorical clusters that do not align with their label according to the CLIP model.

Initial Diverse Priming Pool

(Great Grey Owl)

Filtered Priming Pool

Transductive Image Filter

(ImageNet-V2)

GT:

greatgreyowl

3.1.2 Leveraging Image Information at Test Time

At inference time, the model can narrow the relevant priming pool even further by utilizing information about the test distribution. To do this, given an image x in our test set, we compute the cosine similarity using our CLIP image encoder V , cos( V ( x ) , V ( y )) for every y ∈ P , our priming pool. We retrieve examples with the topk cosine similarity scores ( k = 10 in most of our experiments). We do this collectively for every image in the test set and collect the retrievals to form a filtered priming pool. If an example is retrieved twice, we de-duplicate them in the final priming pool. Since we do this for all images in the test set, we consider this the transductive setting to align with prior work [13, 6].

3.2 Attuning CLIP to the Priming Pool

The goal of this step is to modify our CLIP model to take advantage of the data in the priming pool P . We first construct the task-specific zero-shot linear head W z ∈ R d × n using the text encoder and the natural language names of each class, where d is the feature dimension and n is the number of classes. To get logits for a particular example, x , we compute W z · L ( x ) , so our prediction is arg max c W z · L ( x ) .

To attune our CLIP model to the priming pool, we perform nearest-class mean (NCM) [32] on all retrieved examples to obtain a classification head from the priming pool. Namely for a given class c , we compute a centroid ˜ y c = 1 | B c | ∑ x ∈ B c L ( x ) and then normalize this centroid to produce a class embedding y c = ˜ y c / ∥ y c ∥ . We define the collection of centroids as matrix W ft = [ y c ] c ∈ R d × n . To expand this to few-shot scenarios, we mix the labeled data into the corresponding categorical clusters before performing NCM. Finally, we ensemble W z and W ft using a mixing coefficient α ∈ [0 , 1] as W α = (1 -α ) · W ft + α · W z , which is our final classification head. We choose alpha according to a heuristic α = e -| P | /σ 2 which can be derived from a Bayesian prior over the text features. We experiment with varying values of σ in Table 10. We also find that α can be effectively chosen through cross-validation on a held-out portion of the retrieval set. Intuitively, if we do not have much data in our priming pool, we want it to influence our model less. We use NCM as the classifier as it has shown to be sample-efficient [47]. For comparison to a WISE-FT classifier [52] see Table 3.

4 Experiments

Our key results include: 1. Priming improves performance over baselines in the few-shot setting by 3.81% on average across all datasets and 2.4% on ImageNet in the zero-shot setting 2. Priming in the

transductive, or on-the-fly, setting further improves performance over baselines by 2.51% accuracy and 1.09% over standard Neural Priming 3. Priming is complementary to existing prompt-tuning methods. Our finding indicates that images in the priming set impart distinct information to the model compared to textual class descriptions. We include full details of hyperparameter choices and error bars included in the appendix.

4.1 Datasets and Architectures

We evaluate on standard transfer learning and distribution shift benchmarks. ImageNet [7] is a largescale, general classification dataset that has been well-studied in both transfer learning and distribution shift. ImageNetV2 [41] is one of its natural distribution shift test sets, made by reproducing the original data collection procedure of ImageNet, but even modern large-scale pre-trained models have performance drops on it. ImageNet Sketch [51] and ImageNet-R [18] are natural distribution shifts created by assembling sketches and various renditions of the ImageNet classes. ImageNet-A [19] is a natural adversarial distribution shift of ImageNet, created by collecting images that are misclassified by ResNets. StanfordCars [27], FGVCAircraft [31], Flowers102 [36], and OxfordPets [38] are fine-grained classification datasets which require understanding subtle visual differences between classes and are commonly used for transfer learning benchmarks [17, 21, 40, 55]. SUN397 [53] is a large-scale scene recognition dataset with 397 scene categories.

We perform our experiments with OpenCLIP models [52] trained on LAION-2B and 400M [44]. We choose OpenCLIP because their pretrain datasets are publicly available, therefore we can control what data is introduced to the model. The model architecture reported in the main paper is the B-16 variant trained on LAION-2B unless otherwise stated and we report L-14 and B-32 in the Appendix C.

4.2 Zero-shot Results

In this setting, our model only has access to data it has seen during pre-training, in this case LAION2B. Neural Priming improves top-1 accuracy by 2.45% on ImageNet and 4.25% on average across 6 other diverse downstream datasets compared to the CLIP baseline. In the zero-shot setting, Neural Priming outperforms the 3-shot CLIP model on StanfordCars and FGVCAircraft. This result is particularly noteworthy since traditionally training on in-distribution data generally outperforms zero-shot techniques [54]. Note that we do not present error-bars for the zero-shot experiments as the process is deterministic.

We also compare with VLM [33] and CuPL [39], two zero-shot prompt-tuning methods which obtain natural language descriptions of each class using language models, and a retrieval with finetuning baseline. For implementation details on how the retrieval and fine-tuning are performed see Appendix H. Interestingly, we find that Neural Priming is complementary to existing prompt-tuning methods. The accuracy improvements from CuPL and VLM are additive with Neural Priming. For example, CuPL and Neural Priming each improve performance by 3.78% and 7.17% respectively on FGVCAircraft. Ensembling the methods results in 10.74% improvement over the baseline (Table 2). This surprising result suggests that the textual class descriptions in CuPL and VLM provide unique information to the model that differ from the information obtained from the images in the priming set.

0

1

2

5

10

# Training Samples per Class

66

67

68

69

70

71

72

Accuracy (%)

ImageNet

Neural Priming

CoOp

NCM (Baseline)

Zero-shot OpenCLIP

0

1

2

5

10

# Training Samples per Class

88

89

90

91

Accuracy (%)

Stanford Cars

Neural Priming

CoOp

NCM (Baseline)

Zero-shot OpenCLIP

0

1

2

5

10

# Training Samples per Class

25.0

27.5

30.0

32.5

35.0

37.5

40.0

42.5

Accuracy (%)

FGCV-Aircraft

Neural Priming

CoOp

NCM (Baseline)

Zero-shot OpenCLIP

0

1

2

5

10

# Training Samples per Class

75

80

85

90

95

Accuracy (%)

Flowers102

Neural Priming

CoOp

NCM (Baseline)

Zero-shot OpenCLIP

Another observation is that Neural Priming is especially effective for specialized domains such as StanfordCars, Flowers102, and FGVCAircraft. We speculate this is due to the fact that the label space and image content differs from the majority of the pre-training set. For example, although airplanes occur frequently in LAION-2B, they are rarely described according to their specific model such as Boeing 737-200 . Therefore, recalling and priming the model on pre-train images with such fine-grained classes significantly improves the model. For analysis of LAION-2B with regards to label statistics see Appendix B.

In contrast, for datasets which are more aligned with LAION-2B and the distribution of internet images, such as ImageNet and SUN397, the accuracy gain provided from Neural Priming is smaller in comparison, albeit still significant. In the limit of this trend, Food101 sees almost no improvement across all methods, and even training on in-distribution data for the few-shot case barely improves the accuracy. We speculate that this is because images similar to those in Food101 are already wellrepresented in LAION-2B, rendering additional food images of marginal informational value. We provide analysis of how well the attributes of each dataset are captured by LAION-2B in Appendix B.

To be precise, when we refer to term 'shot number' throughout the experiments section, we mean the number of labeled examples from the target training set. We do not consider images retrieved from LAION-2B as shots in this setting because they are obtained from the pre-training set.

4.3 Few-Shot Results

Neural Priming improves performance for all datasets and shots in the few-shot setting. We compare with CoOp, a recent method for few-shot prompt-tuning, and a Nearest-class-Mean (NCM) baseline. On average across all shots and datasets Neural Priming improves by 3.81% in accuracy over the closest baseline. Results can be found in Figure 3 and Table 9 of the Appendix.

Notably, we find that Neural Priming can match the accuracy of models trained with a substantial number of training examples without using any of the labeled training data for all of the evaluated

datasets (Figure 3). Similar to the zero-shot setting, we observe that Neural Priming is complementary with prompt-tuning methods (Appendix F). Additionally, we observe that as the shot number increases, improvement over the baseline decreases. At 1-shot the improvement in accuracy over the baselines is 5.63% on average, while at 10-shot the improvement is 2.04%. Intuitively, as the model receives more target training data, obtaining additional examples from the pretrain set becomes less necessary.

4.4 Transductive Results

We compare Neural Priming in the transductive setting on 4 standard distribution shift datasets, ImageNet-V2, ImageNet Sketch, ImageNet-R and ImageNet-A. Distribution shift datasets are a natural application of adaptation at test-time. Often real-world datasets differ from the training data, therefore models should be able to adapt on-the-fly. In this setting, the model can learn from the test images without labels before making predictions. We compare with Test-Time Prompt-Tuning (TPT), a state-of-the-art method which uses a self-supervised objective to learn from test data.

We find that Neural Priming with images in the test set improves performance over standard Neural Priming by 1.09% as well as 2.51% over TPT across the 4 distribution shifts (Table 2). Looking at Figure 2, we qualitatively see that the priming pool more closely matches the test images after filtering for the closest images in the initial priming pool. Though the distribution shift can often be imperceptible such as between ImageNet and ImageNetv2, quantitatively we see that the transductive filtering step finds images in the pretraining close to the test distribution.

The transductive retrieval for 50,000 images in the test set on average takes 96 seconds for a priming pool of 1 million images, while retraining the classifier takes on average 11.5 seconds for a priming pool of size 10,000 on standard hardware. We provide further analysis of run-time efficiency of the on-the-fly variant of Neural Priming in Appendix G.

20

40

60

80

100

Samples per class

76

78

80

Top-1 Accuracy

Flowers-102

20

40

60

80

100

Samples per class

88

89

90

Top-1 Accuracy

Stanford Cars

20

40

60

80

100

Samples per class

28

30

32

34

36

Top-1 Accuracy

FGVC Aircraft

20

40

60

80

100

Samples per class

91.00

91.25

91.50

91.75

Top-1 Accuracy

Oxford Pets

4.5 Ablations

We investigate the impact of the priming pool size on the zero-shot accuracy of downstream tasks (Figure 4). Our analysis reveals that as the size of the priming pool increases, there is a general improvement in accuracy. However, there are certain limitations associated with enlarging the pool. The majority of classes in the downstream task have a limited number of available images.

Consequently, when we retrieve a larger number of images for the priming pool, they tend to contain more noise and mislabeled samples. Furthermore, for rare classes, the number of images obtained through exact string search is often less than 100. To address this, a potential extension could involve utilizing a language model to generate alias names for classes, which could then be used to perform additional string searches, thereby expanding the initial priming pool size.

B/32

B/16

L/14

Model (ViT)

70

75

Top-1 Accuracy

ImageNet

Neural Priming

CLIP Zero-shot

B/32

B/16

L/14

Model (ViT)

84

86

88

90

92

Top-1 Accuracy

Stanford Cars

Neural Priming

CLIP Zero-shot

B/32

B/16

L/14

Model (ViT)

70

75

80

85

Top-1 Accuracy

Flowers-102

Neural Priming

CLIP Zero-shot

B/32

B/16

L/14

Model (ViT)

25

30

35

40

Top-1 Accuracy

FGVC Aircraft

Neural Priming

CLIP Zero-shot

We also analyze the impact of the architecture on the accuracy improvement achieved by Neural Priming in the zero-shot setting (Figure 5). To examine this, we conduct experiments using models of varying capacities, namely ViT B-32, B-16, and L-14. We observe that the gains remains consistent across the models. This finding suggests that even as we scale the architecture's capacity, our method will continue to yield significant and consistent relative error reduction.

5 Limitations

Neural Priming has a few potential limitations. Firstly, it requires that the pre-train dataset contains images similar to those in the downstream task. Though all of the datasets we benchmark have abundant relevant data, it is possible for more out-of-distribution datasets that LAION-2B simply does not contain related or queryable images. Secondly, accurate class names are required for retrieval. Meaningful class names for some datasets can be difficult to obtain. For example, in the Flowers102 dataset, some flower species are given by their latin names, which leads to poor retrieval. This issue generally affects open-vocabulary models which require accurate class names to initialize the zero-shot classifier. This limitation may be resolved by using language models to replace class names with their more commonly known synonyms. Lastly, Neural Priming requires access to the pre-training data set which is not always possible such as in the case of OpenAI variant of CLIP. In this case a surrogate dataset would likely suffice, such as using LAION-2B.

6 Discussion & Conclusion

We present Neural Priming , a method to improve the performance of open-vocabulary models by leveraging their own large-scale, diverse pre-training data with no additional data required. With Neural Priming , we demonstrate how to construct a high quality priming pool of examples from the pre-training dataset relevant to a particular task and how to utilize this pool to improve our model. We further show that our method is effective across a variety of downstream tasks and settings. In particular, our method can be used in situations where only natural language descriptions of relevant classes are given, when we have the ability to adapt at inference time, and when we are provided with few labeled in-distribution examples. In all settings, our framework demonstrates a substantial improvement in performance over existing interventions, and is in fact complementary with current prompt-tuning and robustness methods. Our method is also computationally cheap, not requiring any modification of model backbone weights and only a fast text search on the pre-training corpus.

The efficacy of Neural Priming leads to some interesting questions for future work. For example, if the model has seen this data before, why does it help to recall them? We hypothesize that this is due to the fact that the diversity of these datasets introduces competing objectives, which are difficult for the model to optimize directly. For example, the same kind of image could appear with multiple captions and vice-versa, making it difficult to prompt a CLIP model trained on such data at inference time for a particular task. A systematic study of this could elucidate important limitations of current large-scale training paradigms.

Acknowledgments

We are grateful to Sarah Pratt, Mitchell Wortsman, and Romain Beaumont for helpful discussions and feedback. Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, and gifts from Allen Institute for Artificial Intelligence, Google, and Apple. Ludwig Schmidt and Alex Fang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML, CCF-2019844), Open Philanthropy, Google, and the Allen Institute for AI.

References

A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems , 32, 2019.

R. Beaumont. Clip Retrieval , 2021. URL https://github.com/rom1504/ clip-retrieval .

S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning , pages 2206-2240. PMLR, 2022.

G. Brod, M. Werkle-Bergner, and Y. L. Shing. The influence of prior knowledge on memory: a developmental cognitive neuroscience perspective. Frontiers in behavioral neuroscience , 7:139, 2013.

A. L. Brown and M. J. Kane. Preschool children can learn to transfer: Learning to learn and learning from example. Cognitive psychology , 20(4):493-523, 1988.

O. Chapelle, B. Schölkopf, and A. Zien. A discussion of semi-supervised learning and transduction. In Semi-supervised learning , pages 473-478. MIT Press, 2006.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 6216-6234. PMLR, 2022. URL https://proceedings.mlr.press/v162/fang22a.html .

H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 , 2021.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

N. T. Franklin and M. J. Frank. Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning. PLoS computational biology , 16(4):e1007720, 2020.

A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. arXiv preprint arXiv:1301.7375 , 2013.

Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems , 35:29374-29385, 2022.

S. Goyal, A. Kumar, S. Garg, Z. Kolter, and A. Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint arXiv:2212.00638 , 2022.

K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In International conference on machine learning , pages 3929-3938. PMLR, 2020.

K. He, R. Girshick, and P. Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4918-4927, 2019.

D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV , 2021.

D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15262-15271, 2021.

L. Hermer-Vazquez, A. Moffet, and P. Munkholm. Language, space, and the development of cognitive flexibility in humans: The case of two spatial memory tasks. Cognition , 79(3): 263-299, 2001.

G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773 .

M. A. Jamal and G.-J. Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11719-11727, 2019.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. V . Le, Y . Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 , 2021.

J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019.

U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 , 2019.

D. E. Knuth. Retrieval on secondary keys. The art of computer programming: Sorting and Searching , 3:550-567, 1997.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=UYneFzXSJWh .

Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn. Surgical fine-tuning improves adaptation to distribution shifts. CoRR , abs/2210.11466, 2022. doi: 10.48550/arXiv. 2210.11466. URL https://doi.org/10.48550/arXiv.2210.11466 .

H. Liu, K. Son, J. Yang, C. Liu, J. Gao, Y . J. Lee, and C. Li. Learning customized visual models with retrieval-augmented knowledge. arXiv preprint arXiv:2301.07094 , 2023.

S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

C. Manning, P. Raghavan, and H. Schütze. Vector space classification. Introduction to Information Retrieval , pages 289-317, 2008.

S. Menon and C. Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

D. E. Meyer and R. W. Schvaneveldt. Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. Journal of experimental psychology , 90(2):227, 1971.

J. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 7721-7735. PMLR, 2021. URL http://proceedings.mlr.press/v139/miller21b.html .

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

B. Oreshkin, P. Rodríguez López, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems , 31, 2018.

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

S. Pratt, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.03320 , 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019.

A. Rege, A. Kusupati, A. Fan, Q. Cao, S. Kakade, P. Jain, A. Farhadi, et al. Adanns: A framework for adaptive semantic search. arXiv preprint arXiv:2305.19435 , 2023.

H. L. Roediger and K. B. McDermott. Creating false memories: Remembering words not presented in lists. Journal of experimental psychology: Learning, Memory, and Cognition , 21 (4):803, 1995.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Testtime prompt tuning for zero-shot generalization in vision-language models. arXiv preprint arXiv:2209.07511 , 2022.

J. Sivic and A. Zisserman. Video google: A text retrieval approach to object matching in videos. In Computer Vision, IEEE International Conference on , volume 3, pages 1470-1470. IEEE Computer Society, 2003.

J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with selfsupervision for generalization under distribution shifts. In International conference on machine learning , pages 9229-9248. PMLR, 2020.

R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ d8330f857a17c53d217014ee776bfd50-Abstract.html .

V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of visionlanguage models. arXiv preprint arXiv:2211.16198 , 2022.

H. Wang, S. Ge, Z. C. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 10506-10518, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 3eefceb8087e964f89c2d59e8a249915-Abstract.html .

M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7959-7971, 2022.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Trainingfree adaption of clip for few-shot classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 493-510. Springer, 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

A Implementation Details

We perform last layer retraining for Neural Priming. We experimented with fine-tuning all layers, but found minimal performance benefits (Table 4). It has been noted in few-shot literature [47, 37] that retraining only the last layer is sufficient when few training examples are present and we confirm this empirically. When retraining the last layer, we ensemble the text-classifier. We use a mixing co-efficient to ensemble the text-classifier with the image centroids. We found that as the number of samples in the image centroid increases, the ensemble should more heavily favor the image centroid. Formally we use α = e -| P | 2 /σ as the mixing-coefficient, where | P | is the number of image examples with σ equal to 100.

The OpenCLIP models we use can be found at https://github.com/mlfoundations/open_clip. We use the following specific models: (ViT-B-32, laion2b_s34b_b79k) , (ViT-B-16, laion2b_s34b_b88k) , (ViT-L-14, laion2b_s32b_b82k) .

For the zero-shot CLIP baseline presented in Table 2, we use the baseline prompts from [33]. When ensembling with the image centroids from NCM, we use the text-classifier initialized with these prompts unless otherwise stated.

To perform fast substring search, we set up a Full Text Search (FTS) database over the metadata shards of LAION-2B using SQLite. Each shard is a SQLite database composed of 18.1 million rows consisting of information about the URL of the image location, constructed from the original distributed shards from LAION-5B [44]. The nature of FTS and storage reading attributes means that it is much faster to search for rare substrings. More details on this are provided in Appendix G.

B LAION-2B Analysis

We analyze the prevalence of each domain in the LAION pretrain set (Figure 6). We find that for datasets where the domain images and captions occur infrequently such as Flowers102 and StanfordCars, the performance gain from Neural Priming is significantly larger. We measure domain prevalence in LAION-2B by substring searching for captions which contain the class names and counting the total number of images. Image content is another factor for measuring how close the pretraining set is to the downstream dataset in feature space, however this is computationally expensive for a dataset of 2 billion. Also this would depend on the model used to extract features. We choose to look at the caption statistics as a proxy because it is computationally feasible with a dataset at the size of LAION-2B and model agnostic.

C Other Architectures and Datasets

We present the full results for ViT B/16, B/32 and ViT L/14 on LAION-2B for ImageNet, Flowers102, StanfordCars, FGVC-Aircraft, and OxfordPets. We find similar results across architectures. Results can be found in Table 6. We also reproduce experiments for the LAION-400m dataset with the B/16 architecture.

D Dataset Statistics

For ImageNet we performed URL deduplication using the LAION-2B URL list provided on HuggingFace. We found no ImageNet images in the pretrain data. We also randomly sampled images

Stanford Cars

FGVC-AirCraft

Flowers102

Food101

OxfordPets

ImageNet

SUN397

0.0

0.2

0.4

0.6

0.8

1.0

Relative Frequency

Prevalence of Datasets In LAION-2B

from the other datasets (Flowers102, FGVCAirCraft, etc.) and looked for duplicates, though we found no exact matches. The exact dataset statistics are reported for context in Table 7.

E Comparison to REACT

We do not compare directly with REACT [30] in the main paper for two reasons. The first is that REACT adds a significant number of parameters (50 % for the B/16 model) to the network, making an unequal comparison from a parameter and FLOPS perspective. Second, REACT uses 6-10 million images as the support set for each dataset which makes it computationally infeasible to rerun ourselves for all of the data sets we benchmark on and without the parameter adding version. We compare on ImageNet using the numbers reported in their work for comparable models trained on LAION-2B (Table 8).

We differ from REACT in a few key aspects. First, we realign the images labels to the downstream task. For example, for the OxfordPets data set, given an image which has the caption "maltese on a leash walking through the parking with its owner" we map the label to "maltese". REACT trains contrastively using the original captions. We find this step significantly affects accuracy and sample-efficiency. For comparison, REACT retrieves 6-10 million images from LAION whereas we use 10,000-100,000. Second, our method is significantly more efficient in the retrieval and fine-tuning step so can be performed at test-time. This is due to our multi-stage filtering with the first stage of caption filtering being extremely fast. Third, REACT adds 50% additional parameters to the ViT B/16 model.

F Few-Shot

We report the few-shot performance of a model pretrained on LAION-2B with CuPL prompts (Table 9). We find that better prompts synergize with Neural Priming similar to the zero-shot case. The mixing coefficients used for these experiments are set by the schedule given in section A.

G Transductive Run-Time Analysis

Transductive run-time can be broken up into two components: 1. Time to get the initial priming pool and 2. Time to perform exact retrieval on image features of the priming pool.

G.1 Initial Priming Pool construction timings

Here we study the wall-clock time for construction of the initial priming pool on consumer hardware (Intel 10900k CPU and Samsung 980 Pro NVME drive). Wall-clock times for priming pool construction vary greatly depending on a variety of factors. The two main factors affecting timings are: 1. speed of the storage on which the fast text search (FTS) databases are stored (see Appendix A) and 2. whether or not LAION-2b is stored locally (if it is not stored locally, then you need to access the

0

200

400

600

800

1000

Sorted query index

10

1

10 0

10 1

10 2

Seconds to search

original URL to get the image, which adds another factor of variation). Given the massive changes in performance between the two options for (2) , we only give wall-clock timing estimates for (1) .

FTS databases in SQLite use an inverted index over the tokens of their entries. In this case, the tokenization over full words and some more common suffix strings. Given this tokenization, it is easier to search for less common phrases and words. In Figure 7, we analyze this effect with respect to ImageNet priming pool construction. We see that most queries take less than 1 second to search all of LAION-2b. A few very common queries take much longer. For example, the most common query, T-shirt, takes >100s to search. The overall time to complete these searches for ImageNet was 21 minutes. For smaller datasets with less common class names, this process is significantly shorter. This is the majority of the runtime of our overall method.

G.2 Exact Retrieval Time

The next part of our process was performing exact retrieval on the priming pool at test time. The method for this is described in Section 3.1.2 This is generally efficient, as the priming pool is a significantly filtered version of the pre-training dataset. To filter a priming pool of 1.1m images to 51k using the test set of ImageNetV2 took 3.2 minutes on consumer hardware (two 3090 GPUs), not including the initial feature extraction time since these are easily pre-computed.

H Other Implementation Details

Retrieval + Finetuning We retrieve 100 images per class using HNSW, an approximate nearest neighbors method, with the CLIP text-embedding from the class name. We use contrastive fine-tuning with the CLIP objective as the pretrain data only has natural language descriptions as labels. We fine-tune for with a learning rate of 1 e6 and batch size 1024.

CoOp [55] CoOp tries to learn the embeddings for the prompts using a few examples for each of the classes. We use the official implementation 2 provided by the authors with default hyperparameters. For ImageNet experiments, we the models for 50 epochs regardless of the shots. For the smaller datasets like Stanford Cars, Flower102 and FGCV-Aircraft, we train the 1-shot models for 50 epochs, 2-shot and 5-shot models for 100 epochs, and 10-shot models for 200 epochs. We use the standard train-test split for all the datasets to ensure consistency across all the methods. Lastly, instead of the CLIP models provided by OpenAI, we use OpenCLIP [21] models for fair comparison against Neural Priming .

Table 10: An ablation over the sigma value used to calculate the mixing coefficient between the image and text features. A larger sigma value is associated with a stronger prior for the text features.

I Confidence Intervals

We report the confidence intervals and average accuracy for the few-shot experiments (Table 11). For the zero-shot experiments, performance does not vary. For the few-shot experiments variation derives from sampling the training examples. We average accuracy across 10 runs and find the variance to be minimal (less than 1% for most settings).

J Broader Impact

One potential risk of Neural Priming is amplifying bias in the dataset. By retraining on a subset of the data, it is possible to condition the model in a way that is biased given a biased prompt or captions. For downstream tasks which are sensitive to bias, the risk could be mitigated by filtering the captions for inappropriate text or using an auxiliary model to detect inappropriate images. This approach of filtering the priming dataset is a potential direction for reducing bias in a model on the fly. Another potential drawback of Neural Priming is that it increases the carbon footprint of the trained model by retraining it on more data. However, the size of our priming set is small relative to LAION-2B (less than 1%) and we only retrain on the last layer which minimizes compute cost.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces neural priming, a retrieval based approach for adapting large pretrained models to downstream tasks. Neural priming first collects a priming pool of pretraining data relevant to the downstream task using natural language descriptions, then at test-time the priming pool is further filtered by keeping the top-k most similar pretraining samples to the test samples w.r.t. cosine similarity of the CLIP embeddings. From this filtered pool, a classification head is set to the nearest-class mean and averaged with the CLIP ZS text embeddings (similar to WiSE). Across several classification datasets, neural priming is shown to improve over existing prompt-tuning and test time adaptation methods in the zero-shot and few-shot settings, and is complementary to prompt-tuning methods.\n\nSTRENGTHS:\nThe problem of fitting pretrained classifiers to downstream tasks effectively, specifically models like CLIP, is of great interest to the ML community and the retrieval based approach of neural priming is both simple and a pleasant departure from current robust finetuning methods. Overall the paper was well-written, easy to follow, and the evaluation shows consistent improvements over existing prompt tuning and retrieval-based transfer learning methods. I also enjoyed the open questions posed in the discussion section on why retrieval methods work well for transfer learning which I could see sparking future work in this area. \n\nWhile the method is simple and there are some eval limitations that I would like addressed, I think this paper is of value to the community and is above the acceptance threshold.\n\nWEAKNESSES:\nGiven the simplicity of the method and the vast number of related works, my critiques mainly lie in differentiating from related works more and adding in some simple yet relevant baselines. \n\n**Related works**\n\nMy main critique with the related works section is the lack of differentiation from test-time training/adaptation and transductive learning in section 2.3, as well as few-shot learning in section 2.4. \n\nNot a critique and more of a suggestion, but a recent paper[4] explores how removing the least relevant items in a pretraining dataset can improve transfer learning accuracy and I believe would be good motivation for neural priming. There are also some robust CLIP finetuning methods[1] and prompt tuning methods[2,3] that should be mentioned in related works. \n\n[1] [Using Language to Extend to Unseen Domains](https://arxiv.org/abs/2210.09520) \\\n[2] [Visual Prompt Tuning for Test-time Domain Adaptation](https://arxiv.org/abs/2210.04831) \\\n[3] [Conditional Prompt Learning for Vision-Language Models](https://arxiv.org/abs/2203.05557) \\\n[4] [A Data-Based Perspective on Transfer Learning](https://arxiv.org/abs/2207.05739)\n\n**Evaluation**\nHere the main thing that is lacking is a description of the baselines. There is some brief descriptions of CoOp (without a citation) but I found it hard to interoperate exactly what these baselines were. \\\nWhy not compare to CLIP LP or WiSE for the few shot setting? It seems like that would be a natural baseline (even just setting the ZS weights to the few shot examples, from what I understand the NCM baseline uses the priming examples).\n\nQUESTIONS:\nDoes the CLIP ZS baseline in Table 1 just include the class name or does it also include context (i.e. 'a photo of a {} flower' instead of 'a photo of a {}'\")?\n\nLIMITATIONS:\nOne big limitation that I did not see mentioned was the time it takes to retrieve the priming pool from the original pretraining data, especially for large datasets like LAION 2B.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a technique for adapting large pre-trained models to distribution shifts and downstream tasks given few or no labeled examples named Neural Priming. The method can be used at test time on very large scale datasets. Experiments on various datasets show the effectiveness of Neural Priming.\n\nSTRENGTHS:\n1. The proposed Neural Scaling method is novel, with a good motivation (diversity of pretraining datasets introduces competing objectives) and clear presentation.\n2. The technical details of Neural Scaling is simple yet effective: Classical ideas of information retrieval, clustering and prototype calculation were used. This may possibly open a line of research on advancing the proposed techniques.\n3. Neural Scaling has high efficiency on saving the time of doing linear probing.\n\nWEAKNESSES:\n1. In line 182, what is P and \\sigma? The main text should include their meanings rather than putting them in the appendix. Also, ablation studies/discussions can be included for the choice of \\sigma.\n2. I am not familiar with the performance of the compared baselines e.g. VLM (whether these methods are strong and competitive), thus I cannot determine to which extent Neural Priming outperforms other relative methods, especially SOTA ones. While this do not effect the paper's advantage in novelty, it does draw questions on the \"absolute\" performance of Neural Priming.\n\nQUESTIONS:\nThe questions here are for open discussion.\n1. I wonder if all the filtered priming pools for a specific downstream task are integrated into one dataset, will it be a better pretraining/finetuning dataset (excluding noises)?\n2. Is there a possibility that some samples are harmful (e.g. low quality or wrongly labeled), but selected into the priming pools or even the filtered priming pool?\n\nLIMITATIONS:\nLimitations are discussed in terms of the method itself. But issues on broader aspects such as model safety, data leakage are not included.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to zero-shot learning in vision-language models. The authors propose a new tunning schema using prior knowledge in the CLIP model. First, the authors construct a priming pool, which consists of clusters of retrieval images by the CLIP model. Each cluster collects a specific class by searching all text-image pair owning that class name in the text. The authors then tuned this priming pool using the test set. For each image in the test set, the 10 best matching images are selected and de-duplicated to make up the final priming pool. The authors claim that such an operation can adapt the priming pool to the test domain. At last, the authors tune the CLIP model by adding a linear head at the end of the model. The linear head is the weighted sum of a zero-shot trained head and NCM of all retrieved examples per category. Experiments show a 2.45% improvement on ImageNet and a 3.81% improvement on average across some traditional transfer learning benchmarks. Overall, this paper provides valuable insights and promising directions for future research in this field.\n\nSTRENGTHS:\n# Novelty:\nThis paper provides a novel perspective of aligning images by the class name directly. Previous methods are often trapped in playing with feature representations and embeddings. None has looked into whether using strings directly, is a good practice. This work fills this unseen point with good improvements. The method can leverage massive training data, to help the model adapt better to specific tasks.\n\n# Clarity:\nThis paper is extremely well-written and easy to follow.\n\nWEAKNESSES:\n# Novelty:\nThe key innovation of this work can be split into three parts, 1) **class name retrieval to construct priming pooling**, 2) **filtering the priming pooling using cosine similarity with the test set**, and 3) **tuning the CLIP model by adding a linear head using NCM of the priming pooling**. So I will give my opinion to them separately as follows. \n\n1. **Retrieval**:\n\na) **Retrieval is not a new idea in transfer and zero-shot learning.** I think using retrieval is not new in both domains of transfer learning or few-shot\\zero-shot learning, like in NeurIPS2021,2017 paper \"Re-rankingforimageretrievalandtransductive few-shot classification\", \"Few-ShotLearningThroughanInformation RetrievalLens\", there may be too many to list. \n\nb) **String retrieval  I think is not powerful enough.** So the new stuff is using the class name as the retrieval feature. This is somewhat too direct for me. Is there a possibility that CLIP representations combining the string retrieval perform better? According to my experience, this is usually the case. The authors do not discuss the possibilities of using other tools, instead, they come out with a belief that string retrieval is the fastest one so it is superior to other methods, which I may respectfully disagree. First, time efficiency should not be a major problem in constructing the priming pool, after all, it is not required to be online learned or real-time computed. In your pipeline, you first get the naive priming pool from the training data. Then you prune it to preserve the top 10 similar images with each image of the test data. This step won't consume more time if you use semantics as you need to go through all the data in the priming set either way. Then at inference time you only need to compute the mean of each class, there is no need for exact string matching so it also won't consume more time if you use semantics or other stuff. So using semantics or not won't influence the inference time at all, I can't understand why only string is used. Perhaps some experiments comparing those retrieval methods can be helpful to demonstrate the advantages of this work.\n\nc) **Potential conflicts.** Another problem of only using string retrieval is that it only conflicts with each other. Like the class \"guinea pig\" is not a pig at all, \"wolf spider\" is not a wolf, and \"sea snake\" may not contain sea in the image. I am interested in how this method can tell the \"guinea pig\" category (339 in ImageNet) and the \"pig\" category (342 in ImageNet)? On the hand, combining string with semantics can handle this case, as the embedding of guinea pigs will be more similar to the mouse's. Also, I don't understand why the results of string retrieval will be more relevant, can the authors explain why?\n\n2. **Filtering the priming set.** I think the authors can try combining CLIP similarity with Pixel similarity in ranking, inducing more metrics to cooperate. In transfer learning and few-shot learning, such methodology is usually adopted.\n\n3. **NCM head.**  This step uses the final outputs of the language model to predict the mean. Why not also use the output of the Image encoder? Or how about the intermediate representations of the CLIP model? I think the authors can add an explanation of why the output of the language model is the best. Also, tuning a new head is not a new thing. It is popular in incremental learning, transfer, and zero-shot learning. But it is not necessary to limit your imagination in here. Many works have discovered some other parts of the CLIP model, like attention layers, and convolutional kernels, MLPs have rich semantics and an important role in making predictions. If the authors can discuss why they choose to add a linear head instead of tuning other parts, I think this section can be more convincing.  For example, the authors can compute the gradients of each parameter when input new examples of test sets, and find out that the terminal linear layer has the most significant gradients thus influencing the results on new examples the most, so tuning them is vital and more efficient than tuning other parts.\n\nOverall, the three parts are a bit split and there seem no strong connections among them, at least not provided by the authors. They seem like A+B+C. If the authors can reorganize them, and engage them together with one purpose of enhancing transfer learning capability, I think this paper can be more fluent.\n\n\n# Experiments:\n1. **No errors and error bars.** While it is recommended by NeurIPS author guidelines, papers in this domain seem seldom report error bars. I think it could enhance the paper as some figures are close, and errors can help us understand the true advantage of the proposed method. As in Tab. 1, in ImageNet, FGVC, Food101, and SUN397, the differences are smaller than 0.5. We may suspect it within the variance of measures. Also in Fig.3, performance is close, error bars can further tell the differences between methods. \n2. **Why no LAION-5B?** I see many relevant papers conducted on the LAION-5B dataset. LAION-2B is a new one and includes it very well. However, omitting LAION-5B may be hard to align the results of this paper to previous work.\n3. **Ablation is not enough.** Like in the novelty part, the superiority of the chosen components to their alternatives need to be clarified by the experiments. This will make this submission stronger.\n4. \"\"Fair comparason.** As you add a head to the model, you may control the equivalence of all parameters and training steps of all methods. This is important when performance is close.\n\n\n\n# Minor issue:\n1. Line 143 \"However,143 withneuralpriming,prioritizingprecisionoverrecalliscrucial,consideringthesize,diver\", better to add references.\n2. Line 164, the last \"for\" is redundant.\n3. It is not very clear whether Neural Priming is another name of this method or the task this method tries to handle. Better clarify and unify it throughout the paper.\n\nQUESTIONS:\n1. Will the method improve finetuning efficiency of other prompt tuning methods? I mean after applying it, how many training times or steps can be saved for prompt-tuning?\n2. Is the method stable among different tasks? \n3. How is the influence of the filtering step? If remove it, how much accuracy will lose? Also, is there any ablation on the choice of $\\alpha$?\n\n\n\nThe following are suggestions rather than comments, the authors may take them advisedly.\n1. **Notations.** I am in favor of notations used in SimCLR, which are much clear to read and easy to follow as they use different font styles for different concepts. I would suggest a slightly modified version as follows: sets, using \\mathcal command like $\\mathcal{S}$; neural networks, using bold font with parameter subscript, like $f_{\\theta}$, number field, like the real number set, using $\\mathbb{R}^d$, integers using $\\mathbb{N}$; loss function, using \\mathcal command with subscript the \\mathrm environment, like $\\mathcal{L}_{\\mathrm{CLIP}}$; scalars using plain font and vectors using bold fond to distinguish. Currently, the notations carry only limited meaning, make readers need to look back to their definitions several times to remember them. For example, Line 130 uses $\\mathcal{R}^d$ for real numbers while Line 178 uses $\\mathbb{R}^{d\\times n}$, V,P,B,L stand for model, data pool (while training data using font $\\mathcal{D}$), data clusters, and model but using the same font style.  I also recommend not to use L as a notation for the model, it can be confused with the loss function.\n\n2. **Fontsize of figures.** Currently fontsizes of figures are set carelessly. Fig. 1 using a giant fontsize, much larger than the context, seems not necessary, while some words in Fig.2 are too small to tell. I also recommend you to use some images to tell readers the meaning of each component in Fig. 1, like putting some training images above the pretraining data, and putting some images from the priming pool beneze the priming set, using a neural network model to replace the CLIP block and color those parameters that need to be tuned.\n\n3. **The hyphen.** In the title you use **Data-Efficient**, words connected by the hyphen are in capital. While in **Zero-shot**, **Nearest-class-Mean**,  they are not.\n\nLIMITATIONS:\n1. The method needs to know all test set to filter the priming set, online learning may exceed the scope of it.\n2. Filtering step may consume considerable time and computation resources, not affordable for ordinary researchers.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a new method, named Neural Priming, which uses retrieval to adapt vision-language foundation models to downstream tasks given few or no labeled examples in the test time. Specifically, Neural Priming first uses the set of class names in a new task to retrieve relevant samples from the pre-training dataset to form the priming pool and also uses the test examples to narrow the priming pool. Then, the CLIP model can be attuned using the priming pool by constructing the linear prediction head. The experiments are performed in three settings: zero-shot, few-shot and transductive.\n\nSTRENGTHS:\n1. The idea of using retrieval to adapt large vision-language models to downstream tasks is interesting and intuitive. Compared with similar approaches, such as REACT, this work does not need to add new trainable modules. \n2. Experiments on several settings (zero-shot, few-shot and transductive) show the effectiveness and broad applicability of the proposed method. Ablation studies on the number of samples per class in the priming pool and the model capacity are interesting.\n\nWEAKNESSES:\n1. Regarding the retrieval idea for test-time adaptation, the major difference between Neural Priming and REACT is that REACT adds new training modules while Neural Priming only constructs a linear prediction head. It is not clear to me how significant this difference is and how original the proposed method is. Also, there is no direct comparison between these two approaches in experiments. Since Neural Priming and REACT are used in similar settings, it is not clear to me if the proposed method can largely improve over REACT. \n2. It is not clear how sensitive the proposed approach is to the retrieval database. From experiments, we see the improvement will be less significant if the target dataset is well presented in the pre-training dataset (see lines 237-241). On the other hand, the performance will also drop if there is a large gap between the target dataset and the retrieval dataset. Another question is whether the retrieval database has to be the pre-training dataset? Or we can arbitrarily construct a large-scale dataset as retrieval database.\n3. In the few-shot setting, what is the comparison with CoOp+TPT (which is better than CoOp according to [44])?\n4. In the transductive setting, it is unfair to compare TPT as TPT only relies on a single test sample while the proposed method takes all samples in the test set into account. I think another baseline called TENT (https://openreview.net/forum?id=uXl3bZLkr3c) should be considered. \n5. Writing can be improved. There are many repeated sentences across the whole paper. For example, the quantitative results have been mentioned in both introduction and experiment sections repeatedly.\n6. Not sure what “3-shot CLIP model” refers to in line 214.\n\nQUESTIONS:\nMy major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details. I’m willing to increase my rating if my concerns can be well addressed.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}