SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In contrast, for datasets which are more aligned with LAION-2B and the distribution of internet images, such as ImageNet and SUN397, the accuracy gain provided from Neural Priming is smaller in comparison, albeit still significant. In the limit of this trend, Food101 sees almost no improvement across all methods, and even training on in-distribution data for the few-shot case barely improves the accuracy. We speculate that this is because images similar to those in Food101 are already well-represented in LAION-2B, rendering additional food images of marginal informational value. We provide analysis of how well the attributes of each dataset are captured by LAION-2B in Appendix B.
To be precise, when we refer to term "shot number" throughout the experiments section, we mean the number of labeled examples from the target training set. We do not consider images retrieved from LAION-2B as shots in this setting because they are obtained from the pre-training set.
4.3 Few-Shot Results
Neural Priming improves performance for all datasets and shots in the few-shot setting. We compare with CoOp, a recent method for few-shot prompt-tuning, and a Nearest-class-Mean (NCM) baseline. On average across all shots and datasets Neural Priming improves by  $3.81\%$  in accuracy over the closest baseline. Results can be found in Figure 3 and Table 9 of the Appendix.
Notably, we find that Neural Priming can match the accuracy of models trained with a substantial number of training examples without using any of the labeled training data for all of the evaluated
Table 2: Performance of Neural Priming and relevant methods for the transductive setting. Neural Priming finds examples similar to the test image at inference to optimize the model. Models are evaluated zero-shot on 4 distribution shift datasets. Neural Priming excels on distribution shifts which differ significantly from the natural language description of the class names. Performance reported for the OpenCLIP ViT-B-16 model pretrained on LAION-2B.
ImageNet-V2	ImageNet-R	ImageNet Sketch	ImageNet-A
CLIP [21, 40]	59.35	64.57	57.05	35.95
TPT [45]	59.84	78.74	52.75	36.92
Priming (Ours)	60.12	77.98	58.29	37.56
Transduct. Priming (Ours)	60.76	79.37	59.97	38.20
datasets (Figure 3). Similar to the zero-shot setting, we observe that Neural Priming is complementary with prompt-tuning methods (Appendix F). Additionally, we observe that as the shot number increases, improvement over the baseline decreases. At 1-shot the improvement in accuracy over the baselines is  $5.63\%$  on average, while at 10-shot the improvement is  $2.04\%$ . Intuitively, as the model receives more target training data, obtaining additional examples from the pretrain set becomes less necessary.
4.4 Transductive Results
We compare Neural Priming in the transductive setting on 4 standard distribution shift datasets, ImageNet-V2, ImageNet Sketch, ImageNet-R and ImageNet-A. Distribution shift datasets are a natural application of adaptation at test-time. Often real-world datasets differ from the training data, therefore models should be able to adapt on-the-fly. In this setting, the model can learn from the test images without labels before making predictions. We compare with Test-Time Prompt-Tuning (TPT), a state-of-the-art method which uses a self-supervised objective to learn from test data.
We find that Neural Priming with images in the test set improves performance over standard Neural Priming by  $1.09\%$  as well as  $2.51\%$  over TPT across the 4 distribution shifts (Table 2). Looking at Figure 2, we qualitatively see that the priming pool more closely matches the test images after filtering for the closest images in the initial priming pool. Though the distribution shift can often be imperceptible such as between ImageNet and ImageNetv2, quantitatively we see that the transductive filtering step finds images in the pretraining close to the test distribution.
The transductive retrieval for 50,000 images in the test set on average takes 96 seconds for a priming pool of 1 million images, while retraining the classifier takes on average 11.5 seconds for a priming pool of size 10,000 on standard hardware. We provide further analysis of run-time efficiency of the on-the-fly variant of Neural Priming in Appendix G.
Figure 4: Ablation over the number of samples per class in the priming pool. We observe a consistent zero-shot accuracy improvement as the number of samples drawn from our pool increases.
4.5 Ablations
We investigate the impact of the priming pool size on the zero-shot accuracy of downstream tasks (Figure 4). Our analysis reveals that as the size of the priming pool increases, there is a general improvement in accuracy. However, there are certain limitations associated with enlarging the pool. The majority of classes in the downstream task have a limited number of available images.
Consequently, when we retrieve a larger number of images for the priming pool, they tend to contain more noise and mistrabeled samples. Furthermore, for rare classes, the number of images obtained through exact string search is often less than 100. To address this, a potential extension could involve utilizing a language model to generate alias names for classes, which could then be used to perform additional string searches, thereby expanding the initial priming pool size.
Figure 5: Analyzing the effect of model capacity on Neural Priming. We find the relative error reduction stays consistent even as the scale of the model increases.
We also analyze the impact of the architecture on the accuracy improvement achieved by Neural Priming in the zero-shot setting (Figure 5). To examine this, we conduct experiments using models of varying capacities, namely ViT B-32, B-16, and L-14. We observe that the gains remain consistent across the models. This finding suggests that even as we scale the architecture's capacity, our method will continue to yield significant and consistent relative error reduction.
Neural Priming has a few potential limitations. Firstly, it requires that the pre-train dataset contains images similar to those in the downstream task. Though all of the datasets we benchmark have abundant relevant data, it is possible for more out-of-distribution datasets that LAION-2B simply does not contain related or queryable images. Secondly, accurate class names are required for retrieval. Meaningful class names for some datasets can be difficult to obtain. For example, in the Flowers102 dataset, some flower species are given by their latin names, which leads to poor retrieval. This issue generally affects open-vocabulary models which require accurate class names to initialize the zero-shot classifier. This limitation may be resolved by using language models to replace class names with their more commonly known synonyms. Lastly, Neural Priming requires access to the pre-training data set which is not always possible such as in the case of OpenAI variant of CLIP. In this case a surrogate dataset would likely suffice, such as using LAION-2B.
We present Neural Priming, a method to improve the performance of open-vocabulary models by leveraging their own large-scale, diverse pre-training data with no additional data required. With Neural Priming, we demonstrate how to construct a high quality priming pool of examples from the pre-training dataset relevant to a particular task and how to utilize this pool to improve our model. We further show that our method is effective across a variety of downstream tasks and settings. In particular, our method can be used in situations where only natural language descriptions of relevant classes are given, when we have the ability to adapt at inference time, and when we are provided with few labeled in-distribution examples. In all settings, our framework demonstrates a substantial improvement in performance over existing interventions, and is in fact complementary with current prompt-tuning and robustness methods. Our method is also computationally cheap, not requiring any modification of model backbone weights and only a fast text search on the pre-training corpus.
The efficacy of Neural Priming leads to some interesting questions for future work. For example, if the model has seen this data before, why does it help to recall them? We hypothesize that this is due to the fact that the diversity of these datasets introduces competing objectives, which are difficult for the model to optimize directly. For example, the same kind of image could appear with multiple captions and vice-versa, making it difficult to prompt a CLIP model trained on such data at inference time for a particular task. A systematic study of this could elucidate important limitations of current large-scale training paradigms.
We are grateful to Sarah Pratt, Mitchell Wortsman, and Romain Beaumont for helpful discussions and feedback. Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, and gifts from Allen Institute for Artificial Intelligence, Google, and Apple. Ludwig Schmidt and Alex Fang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML, CCF-2019844), Open Philanthropy, Google, and the Allen Institute for AI.
[1] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[2] R. Beaumont. Clip Retrieval, 2021. URL https://github.com/rom1504/clip-retrieval.
[3] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206-2240. PMLR, 2022.
[4] G. Brod, M. Werkle-Bergner, and Y. L. Shing. The influence of prior knowledge on memory: a developmental cognitive neuroscience perspective. Frontiers in behavioral neuroscience, 7:139, 2013.
[5] A. L. Brown and M. J. Kane. Preschool children can learn to transfer: Learning to learn and learning from example. Cognitive psychology, 20(4):493-523, 1988.
[6] O. Chapelle, B. Scholkopf, and A. Zien. A discussion of semi-supervised learning and transduction. In Semi-supervised learning, pages 473-478. MIT Press, 2006.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[9] A. Fang, G. Ilharco, M. Wortman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 6216-6234. PMLR, 2022. URL https://proceedings.mlr.press/v162/fang22a.html.
[10] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097, 2021.
[11] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017.
[12] N. T. Franklin and M. J. Frank. Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning. PLoS computational biology, 16(4):e1007720, 2020.
[13] A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. arXiv preprint arXiv:1301.7375, 2013.
[14] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-29385, 2022.
[50] V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of vision-language models. arXiv preprint arXiv:2211.16198, 2022.
[51] H. Wang, S. Ge, Z. C. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10506-10518, 2019. URL https://proceedings.neurips.cc/paper/2019/bit/3eefceb8087e964f89c2d59e8a249915-Abstraction.html.
[52] M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[53] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[54] R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 493-510. Springer, 2022.
[55] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
We perform last layer retraining for Neural Priming. We experimented with fine-tuning all layers, but found minimal performance benefits (Table 4). It has been noted in few-shot literature [47, 37] that retraining only the last layer is sufficient when few training examples are present and we confirm this empirically. When retraining the last layer, we ensemble the text-classifier. We use a mixing co-efficient to ensemble the text-classifier with the image centroids. We found that as the number of samples in the image centroid increases, the ensemble should more heavily favor the image centroid. Formally we use  $\alpha = e^{-|P|^2 / \sigma}$  as the mixing-coefficient, where  $|P|$  is the number of image examples with  $\sigma$  equal to 100.
The OpenCLIP models we use can be found at https://github.com/mlfoundations/open Clip. We use the following specific models: (ViT-B-32, laion2b_s34b_b79k), (ViT-B-16, laion2b_s34b_b88k), (ViT-L-14, laion2b_s32b_b82k).
For the zero-shot CLIP baseline presented in Table 2, we use the baseline prompts from [33]. When ensembling with the image centroids from NCM, we use the text-classifier initialized with these prompts unless otherwise stated.
To perform fast substring search, we set up a Full Text Search (FTS) database over the metadata shards of LAION-2B using SQLite. Each shard is a SQLite database composed of 18.1 million rows consisting of information about the URL of the image location, constructed from the original distributed shards from LAION-5B [44]. The nature of FTS and storage reading attributes means that it is much faster to search for rare substrings. More details on this are provided in Appendix G.
Table 3: Comparing Neural Priming to other classification approaches. NCM uses only the image features. WISE-FT fine-tunes a linear classifier and ensembles it with the text classifier.
Shots	Neural Priming	NCM	Wise-FT
0	71.38	68.3	67.75
1	71.56	68.89	68.14
2	71.81	68.85	68.25
5	72.03	69.12	68.78
10	72.36	70.12	69.02
B LAION-2B Analysis
We analyze the prevalence of each domain in the LAION pretrain set (Figure 6). We find that for datasets where the domain images and captions occur infrequently such as Flowers102 and StanfordCars, the performance gain from Neural Priming is significantly larger. We measure domain prevalence in LAION-2B by substring searching for captions which contain the class names and counting the total number of images. Image content is another factor for measuring how close the pretraining set is to the downstream dataset in feature space, however this is computationally expensive for a dataset of 2 billion. Also this would depend on the model used to extract features. We choose to look at the caption statistics as a proxy because it is computationally feasible with a dataset at the size of LAION-2B and model agnostic.
C Other Architectures and Datasets
We present the full results for ViT B/16, B/32 and ViT L/14 on LAION-2B for ImageNet, Flowers102, StanfordCars, FGVC-Aircraft, and OxfordPets. We find similar results across architectures. Results can be found in Table 6. We also reproduce experiments for the LAION-400m dataset with the B/16 architecture.
For ImageNet we performed URL dedduplication using the LAION-2B URL list provided on HuggingFace. We found no ImageNet images in the pretrain data. We also randomly sampled images
Table 4: Comparison of full fine-tuning and NCM with Neural Priming. We find NCM performs similarly in the low-sample regime.
ImageNet	Stanford Cars	FGVC Aircraft	Flowers102	Food101	Oxford Pets	SUN397
CLIP [40, 21]	68.30	87.40	25.86	71.65	86.58	90.21	67.35
Retrieval + FT	70.28	87.95	26.22	72.15	86.63	90.35	68.01
VLM [33]	69.35	87.88	28.54	72.11	86.31	90.24	67.73
CuPL [39]	70.25	88.63	29.64	72.32	86.20	91.16	70.80
Priming (Ours)	70.75	89.30	33.03	79.81	86.66	91.87	71.21
Priming + CuPL (Ours)	71.38	90.23	36.00	80.04	86.86	91.85	72.35
Figure 6: Frequency of dataset class names in the LAION-2B dataset normalized to 1. We find Neural Priming improves accuracy more for datasets that are rare.
from the other datasets (Flowers102, FGVCAirCraft, etc.) and looked for duplicates, though we found no exact matches. The exact dataset statistics are reported for context in Table 7.
E Comparison to REACT
We do not compare directly with REACT [30] in the main paper for two reasons. The first is that REACT adds a significant number of parameters (50% for the B/16 model) to the network, making an unequal comparison from a parameter and FLOPS perspective. Second, REACT uses 6-10 million images as the support set for each dataset which makes it computationally infeasible to rerun ourselves for all of the data sets we benchmark on and without the parameter adding version. We compare on ImageNet using the numbers reported in their work for comparable models trained on LAION-2B (Table 8).
We differ from REACT in a few key aspects. First, we realign the images labels to the downstream task. For example, for the OxfordPets data set, given an image which has the caption "maltese on a leash walking through the parking with its owner" we map the label to "maltese". REACT trains contrastively using the original captions. We find this step significantly affects accuracy and sample-efficiency. For comparison, REACT retrieves 6-10 million images from LAION whereas we use 10,000-100,000. Second, our method is significantly more efficient in the retrieval and fine-tuning step so can be performed at test-time. This is due to our multi-stage filtering with the first stage of caption filtering being extremely fast. Third, REACT adds  $50\%$  additional parameters to the ViT B/16 model.
Table 5: Performance of Neural Priming and the CLIP baseline with the ViT B/16 architecture. We observe similar trends as for the LAION-2B dataset. Neural Priming improves over the zero-shot performance with similar relative error reduction.
ImageNet	Stanford Cars	FGVC Aircraft	Flowers102	Food101	Oxford Pets	SUN397
CLIP [40, 21]	66.72	82.47	16.05	71.47	86.33	88.68	61.13
Neural Priming (Ours)	69.21	86.83	26.73	82.17	86.62	90.08	62.27
Table 6: Zero-shot results for ViT B/32 and L/14 architectures. We find similar results across architectures. Surprisingly, the relative error often decreases more with architecture scale.
Capacity	ImageNet	Stanford Cars	Flowers102	FGVCAircraft	Food101	Oxford Pets	SUN397
Baseline (NCM)
ViT-B/32	67.12	84.21	67.40	23.79	82.53	90.62	61.97
ViT-B/16	68.30	87.40	71.65	25.86	86.58	90.21	67.35
ViT-L/14	76.62	91.82	75.14	33.36	90.96	93.40	71.29
Neural Priming
ViT-B/32	68.38	87.45	72.26	31.68	82.91	91.25	63.84
ViT-B/16	70.75	90.23	79.81	33.03	86.66	91.87	71.21
ViT-L/14	78.72	93.12	84.48	43.62	91.06	94.05	72.49
F Few-Shot
We report the few-shot performance of a model pretrained on LAION-2B with CuPL prompts (Table 9). We find that better prompts synergize with Neural Priming similar to the zero-shot case. The mixing coefficients used for these experiments are set by the schedule given in section A.
G Transductive Run-Time Analysis
Transductive run-time can be broken up into two components: 1. Time to get the initial priming pool and 2. Time to perform exact retrieval on image features of the priming pool.
G.1 Initial Priming Pool construction timings
Here we study the wall-clock time for construction of the initial priming pool on consumer hardware (Intel 10900k CPU and Samsung 980 Pro NVME drive). Wall-clock times for priming pool construction vary greatly depending on a variety of factors. The two main factors affecting timings are: 1. speed of the storage on which the fast text search (FTS) databases are stored (see Appendix A) and 2. whether or not LAION-2b is stored locally (if it is not stored locally, then you need to access the
Figure 7: Wall-clock times by query for construction of the initial priming pool. The queries used here are the ones used for the construction of the ImageNet pool. Due to the nature of full text search, rare queries are very fast, with most classes taking less than 1 second to search all of LAION-2b.
Dataset	Number of Classes	Train Size	Test Size
ImageNet	1000	1281167	50000
Stanford Cars	196	8144	8041
FGVC Aircraft	100	6667	3333
Flowers102	102	2040	6149
Food101	102	75750	25250
Oxford Pets	37	3680	3669
SUN397	397	19850	19850
Table 7: Dataset statistics for every dataset evaluated on in Section 4. We evaluated using the accuracy metric for every dataset.
Table 8: Comparison with REACT on ImageNet for varying capacities. Neural Priming outperforms REACT on ImageNet without adding parameters while retraining on  $1\%$  of the data REACT does (10 million vs 100,000.)
B/16	L/14
REACT	67.5	76.4
Neural Priming	68.38	78.72
original URL to get the image, which adds another factor of variation). Given the massive changes in performance between the two options for (2), we only give wall-clock timing estimates for (1).
FTS databases in SQLite use an inverted index over the tokens of their entries. In this case, the tokenization over full words and some more common suffix strings. Given this tokenization, it is easier to search for less common phrases and words. In Figure 7, we analyze this effect with respect to ImageNet priming pool construction. We see that most queries take less than 1 second to search all of LAION-2b. A few very common queries take much longer. For example, the most common query, T-shirt, takes  $>100$ s to search. The overall time to complete these searches for ImageNet was 21 minutes. For smaller datasets with less common class names, this process is significantly shorter. This is the majority of the runtime of our overall method.
G.2 Exact Retrieval Time
The next part of our process was performing exact retrieval on the priming pool at test time. The method for this is described in Section 3.1.2 This is generally efficient, as the priming pool is a significantly filtered version of the pre-training dataset. To filter a priming pool of  $1.1\mathrm{m}$  images to  $51\mathrm{k}$  using the test set of ImageNetV2 took 3.2 minutes on consumer hardware (two 3090 GPUs), not including the initial feature extraction time since these are easily pre-computed.
H Other Implementation Details
Retrieval + Finetuning We retrieve 100 images per class using HNSW, an approximate nearest neighbors method, with the CLIP text-embedding from the class name. We use contrastive fine-tuning with the CLIP objective as the pretrain data only has natural language descriptions as labels. We fine-tune for with a learning rate of 1e-6 and batch size 1024.