SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CoOp [55] CoOp tries to learn the embeddings for the prompts using a few examples for each of the classes. We use the official implementation<sup>2</sup> provided by the authors with default hyperparameters. For ImageNet experiments, we the models for 50 epochs regardless of the shots. For the smaller datasets like Stanford Cars, Flower102 and FGCV-Aircraft, we train the 1-shot models for 50 epochs, 2-shot and 5-shot models for 100 epochs, and 10-shot models for 200 epochs. We use the standard train-test split for all the datasets to ensure consistency across all the methods. Lastly, instead of the CLIP models provided by OpenAI, we use OpenCLIP [21] models for fair comparison against Neural Priming.
$^{2}$ https://github.com/KaiyangZhou/CoOp
Table 9: Performance on Neural Priming (NP) with prompt-tuning method in the few-shot setting. We find that prompt-tuning methods are compatible with Neural Priming even when training examples are available.
Shots	ImageNet	Stanford Cars	Flowers102	FGVCAircraft
NP + CuPL	NP	NP + CuPL	NP	NP + CuPL	NP	NP + CuPL	NP
0	71.38	71.15	90.23	90.11	80.04	79.78	35.70	35.38
1	71.56	71.32	90.31	90.17	80.77	80.72	36.39	36.55
2	71.81	71.50	90.55	90.21	88.90	88.60	38.16	37.94
5	72.03	71.92	91.03	90.93	94.66	93.43	40.01	39.84
10	72.36	72.27	91.54	91.49	94.83	94.75	43.83	43.67
TPT [45] Test-time Prompt Tuning (TPT) combines the ideas of test-time training [48] and prompt learning [55]. For a given test image, TPT tries to learn a soft prompt that maximizes agreement between multiple augmented views. We use the official implementation<sup>3</sup> for all the experiments and train TPT with 64 augmented views to ensure the best performance on all the datasets.
σ	ImageNet	Stanford Cars	FGVC-Aircraft	Flowers102
1	68.9	87.54	25.9	71.94
10	69.1	88.98	29.75	74.97
100	70.75	89.3	33.03	79.81
1000	70.64	89.11	31.32	78.81
Table 10: An ablation over the sigma value used to calculate the mixing coefficient between the image and text features. A larger sigma value is associated with a stronger prior for the text features.
I Confidence Intervals
We report the confidence intervals and average accuracy for the few-shot experiments (Table 11). For the zero-shot experiments, performance does not vary. For the few-shot experiments variation derives from sampling the training examples. We average accuracy across 10 runs and find the variance to be minimal (less than  $1\%$  for most settings).
One potential risk of Neural Priming is amplifying bias in the dataset. By retraining on a subset of the data, it is possible to condition the model in a way that is biased given a biased prompt or captions. For downstream tasks which are sensitive to bias, the risk could be mitigated by filtering the captions for inappropriate text or using an auxiliary model to detect inappropriate images. This approach of filtering the priming dataset is a potential direction for reducing bias in a model on the fly. Another potential drawback of Neural Priming is that it increases the carbon footprint of the trained model by retraining it on more data. However, the size of our priming set is small relative to LAION-2B (less than  $1\%$ ) and we only retrain on the last layer which minimizes compute cost.
<sup>3</sup>https://github.com/azshue/TPT
Table 11: Few-shot results with confidence intervals. We report the average performance of Neural Priming across 10 runs with the standard deviation. We find that Neural Priming has lower variance than the baseline as the constant priming pool leads to less variation in the centroid of images.
Shots	ImageNet	Stanford Cars	Flowers102	FGVCAircraft
Neural Priming	Baseline (NCM)	Neural Priming	Baseline (NCM)	Neural Priming	Baseline (NCM)	Neural Priming	Baseline (NCM)
1	71.53 ± 0.04	69.81 ± 0.05	90.31 ± 0.16	88.57 ± 0.48	80.77 ± 0.14	76.09 ± 1.14	36.39 ± 0.22	27.56 ± 1.12
2	71.83 ± 0.05	70.05 ± 0.05	90.55 ± 0.15	88.73 ± 0.43	88.90 ± 0.13	82.50 ± 0.74	38.16 ± 0.17	30.01 ± 0.67
5	72.02 ± 0.04	70.43 ± 0.02	91.03 ± 0.12	89.19 ± 0.18	94.66 ± 0.10	89.46 ± 0.29	40.01 ± 0.11	35.50 ± 0.77
10	72.34 ± 0.03	70.90 ± 0.04	91.54 ± 0.10	90.86 ± 0.16	94.83 ± 0.01	92.74 ± 0.02	43.83 ± 0.09	40.84 ± 0.42