SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Embeddings. For these evaluations, we use what we call "streaming" embeddings (see Figure 24). In this setup, we use the context window of the LLM to encode a single example at a time. The final embeddings are then averaged and used in-context with TART's reasoning module. This is in contrast to the vanilla and LOO embeddings which use multiple examples in-context with the base LLM to obtain the embeddings.
Evaluation. Figure 23 shows the performance of base in-context learning with TART across the three different model families. Observe that while in-context learning is bottlenecked by the context window of the base LLM, TART is able to learn from  $10\mathrm{x}$  more examples and exhibits an increasing trend in accuracy with number of examples across models.
E.3 Generalization across architectures
In this section, we demonstrate that it is possible to train TART reasoning modules on architectures beyond transformers. More concretely, we train a reasoning module that has the Hyena architecture [30] using the same synthetic logistic regression tasks.
E.3.1 Setup and Hyper-parameters
We instantiate a reasoning module with 12 Hyena blocks, a hidden dimension size of 256, and a sequence length of 2050. We train with a batch size of 16, using a learning rate of 5e-05. We sample data with a noise parameter  $(\alpha)$  equal to 1. We train the model for 5000 steps. For our evaluations, we use the final checkpoint (i.e., 5000) of the reasoning module.
E.3.2 Evaluation
We evaluate the Hyena-based reasoning module applied to GPT-NEO (125M) on 6 datasets: SMS Spam, SST, AG-News-0, DBpedia-14-0, Youtube, and Rotten Tomatoes. As seen in Figure 25, the Hyena-based reasoning module transfers to natural language tasks and performs competitively with the transformer-based reasoning module
E.4 Performance in low sample regime
In Figure 26, we evaluate our models in the low-sample regime where the number of in-context examples range from  $k = 4 - 10$ . TART outperforms both in-context learning and task-specific fine-tuning baselines.
(a)  $\mathrm{k} = {18}$
(b)  $\mathrm{k} = {32}$
(c)  $\mathrm{k} = {48}$
(d)  $\mathrm{k} = {64}$
Figure 25: Comparing TART with a Hyena version of TART. Instead of using a Transformer module, we use a Hyena model to learn the probabilistic reasoning task. Hyena-TART is comparable to TART across datasets and achieves better performance for larger value of  $k$  ( $= 64$ ).
E.5 Extension to other modalities: TART is domain-agnostic!
We begin by providing a description of the datasets we used to evaluate TART on audio and vision tasks, and then provide additional results comparing our algorithm with baselines.
E.5.1 Dataset details
For audio classification, we use the Speech Commands (Version 0.01) dataset [40]. Speech Commands is a multi-class classification task where the task is to detect preregistered keywords by classifying utterances into a predefined set of words. We construct a 3 binary classification task over the keywords "stop" and "go", "up" and "down", and "yes" and "no" (see Table 6 for more details).
For image classification, we use CIFAR-10 [17] and MNIST [19]. Both tasks are multi-class classification tasks. We create 3 binary classification tasks for each of the datasets. For CIFAR-10 the tasks are: airplane vs. bird, bird vs. horse, and ship vs. automobile. For MNIST the tasks are: 0 vs. 8, 1 vs. 6 and 2 vs. 4. See Table 6 for more details.
For both the audio and image datasets, we sample a class-balanced set of 256 samples from the training set. For the test sets, we filter the original test sets to only include samples of the two classes we are learning to predict for (i.e., airplane and bird for CIFAR10 and 0 and 8 for MNIST).
(a)  $\mathrm{k} = 4$
(b)  $\mathrm{k} = 6$
(c)  $\mathrm{k} = 8$
Figure 26: TART in low sample regime
(d)  $\mathrm{k} = {10}$
Dataset	Modality	Train size	Test size
MNIST (0 vs. 8)	image	256	1954
MNIST (1 vs. 6)	image	256	1940
MNIST (2 vs. 4)	image	256	2014
Speech Commands (stop vs. go)	audio	256	500
Speech Commands (up vs. down)	audio	256	508
Speech Commands (yes vs. no)	audio	256	525
CIFAR-10 (airplane vs. bird)	image	256	2000
CIFAR-10 (bird vs. horse)	image	256	2000
CIFAR-10 (ship vs. automobile)	image	256	2000
Table 6: Dataset statistics for all audio and image evaluation datasets.
E.5.2 Algorithms for comparison
For these evaluations, we use the "streaming embeddings" described in Figure 24 to obtain the embedding for TART. We evaluate over  $k = [18, 32, 48, 64, 128, 256]$ .
We compare against two baseline task-adaptation methods: 1) full fine-tuning and 2) adapters. We use the same architectures as described in Appendix E.1.2. For vision tasks, we use Google's 307M
(a) MNIST (1 vs. 6)
(b) MNIST (2 vs. 4)
Figure 27: Additional MNIST binary classification tasks. TART is competitive with task-specific full fine-tuning and adapters.
(a) CIFAR-10 (ship vs. automobile)
Figure 28: Additional CIFAR-10 binary classification tasks. TART is competitive with task-specific full fine-tuning and adapters.
(b) CIFAR-10 (bird vs. horse)
parameter pretrained Vision Transformer (ViT) model [44]: VIT-LARGE-PATCH16-224. For audio tasks, we use OpenAI's 1.5B parameter pretrained Whisper model [31]: WHISPER-LARGE.
Hyperparameter search For each baseline, we perform an extensive hyperparameter search over number of epochs and learning rate for each dataset in order to optimize performance. We search over a range of learning rates (1e-3, 5e-04, 1e-4, 5e-5, 1e-5, and 8e-6) and a range of epochs (5, 10, 15 and 20). For all models we use a batch size of 1. We perform our hyperparameter searches for a fixed number of train samples (128) and run our hyperparameter searches over 3 random seeds.
E.5.3 Evaluation
We plot the accuracy as a function of the number of examples for TART, fine-tuning and adapter in Figure 27 (MNIST), Figure 28 (CIFAR-10), and Figure 29 (Speech Commands). TART is competitive with both these baselines, showing how task-agnostic methods can compete with task-specific adaptation methods across different modalities.
Our submission focuses on understanding in-context learning and how it compares to task-specific fine-tuning approaches. We view our work as a general understanding paper and, to the best of our knowledge, see no negative consequences of our work. We hope that our work furthers the state of understanding of large language models (LLMs). Furthermore, because TART is a plug-and-play solution that doesn't require any fine-tuning, we hope that our method can help domain experts (e.g., lawyers, scientists) incorporate LLMs into their workflows. In this sense, we view TART as a
(a) Speech Commands (yes vs. no)
(b) Speech Commands (up vs. down)
Figure 29: Additional Speech Commands binary classification tasks. TART is competitive with task-specific full fine-tuning and adapters.
contribution that is democratizing ML by making them more accessible and usable to experts in a variety of domains.
G Compute Resource Estimates
Resources for training TART We use a single NVIDIA RTX A6000 GPU ( $2/hr) for 6 hours to train our TART inference head, costing a total of$ 18 to train.
Resources for training task-specific baselines We use 4 NVIDIA A100 GPU's ( $3.5 per GPU/hr) for 100 hours for hyperparameter tuning, costing a total of$ 1,400. We use 8 NVIDIA A100 GPU's ( $3.5 per GPU/hr) for 50 hours to fine-tune all task-adaptation baseline models. Total cost for fine-tuning amounted to$ 1,440.
Resources for inference We use 8 NVIDIA A100 GPU's ( $3.5 per GPU/hr) for 480 hours for all inference runs costing a total of$ 13,440.
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.790 ± 0.036	0.552 ± 0.049	0.724 ± 0.070	0.852 ± 0.019	0.824 ± 0.031
AG-News-1	0.828 ± 0.021	0.541 ± 0.037	0.779 ± 0.137	0.903 ± 0.021	0.868 ± 0.020
AG-News-2	0.751 ± 0.023	0.513 ± 0.014	0.626 ± 0.057	0.765 ± 0.025	0.755 ± 0.012
AG-News-3	0.743 ± 0.031	0.502 ± 0.017	0.736 ± 0.035	0.786 ± 0.025	0.727 ± 0.066
Civil Comments	0.559 ± 0.027	0.499 ± 0.002	0.520 ± 0.033	0.515 ± 0.019	0.555 ± 0.036
DBPedia-0	0.866 ± 0.030	0.611 ± 0.091	0.802 ± 0.020	0.825 ± 0.012	0.837 ± 0.022
DBPedia-1	0.778 ± 0.036	0.579 ± 0.100	0.766 ± 0.056	0.740 ± 0.041	0.778 ± 0.044
DBPedia-2	0.798 ± 0.042	0.609 ± 0.136	0.862 ± 0.041	0.908 ± 0.011	0.832 ± 0.048
DBPedia-3	0.812 ± 0.032	0.611 ± 0.135	0.817 ± 0.034	0.859 ± 0.025	0.848 ± 0.028
IMDB	0.537 ± 0.022	0.507 ± 0.007	0.625 ± 0.013	0.560 ± 0.021	0.556 ± 0.014
Rotten Tomatoes	0.535 ± 0.030	0.550 ± 0.043	0.689 ± 0.019	0.541 ± 0.037	0.524 ± 0.018
SMS Spam	0.869 ± 0.063	0.736 ± 0.099	0.925 ± 0.011	0.833 ± 0.015	0.886 ± 0.023
SST	0.544 ± 0.021	0.542 ± 0.024	0.715 ± 0.009	0.555 ± 0.026	0.547 ± 0.009
Youtube	0.784 ± 0.047	0.658 ± 0.089	0.833 ± 0.089	0.768 ± 0.100	0.715 ± 0.136
Table 7: Standard deviation of accuracy, number of examples  $= {18}$  ,GPT-NEO (125M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.805 ± 0.029	0.498 ± 0.002	0.758 ± 0.127	0.601 ± 0.072	0.669 ± 0.044
AG-News-1	0.825 ± 0.023	0.517 ± 0.019	0.916 ± 0.015	0.702 ± 0.097	0.690 ± 0.034
AG-News-2	0.758 ± 0.027	0.500 ± 0.000	0.596 ± 0.031	0.500 ± 0.001	0.588 ± 0.026
AG-News-3	0.754 ± 0.033	0.501 ± 0.003	0.661 ± 0.093	0.552 ± 0.041	0.613 ± 0.022
Civil Comments	0.575 ± 0.019	0.500 ± 0.003	0.525 ± 0.049	0.500 ± 0.000	0.515 ± 0.008
DBPedia-0	0.861 ± 0.018	0.508 ± 0.016	0.786 ± 0.054	0.638 ± 0.109	0.704 ± 0.038
DBPedia-1	0.813 ± 0.020	0.499 ± 0.010	0.787 ± 0.067	0.599 ± 0.078	0.710 ± 0.059
DBPedia-2	0.870 ± 0.035	0.502 ± 0.025	0.878 ± 0.071	0.701 ± 0.113	0.767 ± 0.052
DBPedia-3	0.850 ± 0.050	0.502 ± 0.003	0.864 ± 0.034	0.603 ± 0.114	0.734 ± 0.030
IMDB	0.550 ± 0.027	0.507 ± 0.005	0.590 ± 0.046	0.500 ± 0.000	0.526 ± 0.012
Rotten Tomatoes	0.544 ± 0.027	0.491 ± 0.013	0.589 ± 0.074	0.500 ± 0.000	0.522 ± 0.023
SMS Spam	0.901 ± 0.038	0.867 ± 0.030	0.892 ± 0.052	0.867 ± 0.004	0.851 ± 0.042
SST	0.572 ± 0.016	0.517 ± 0.002	0.617 ± 0.074	0.517 ± 0.000	0.539 ± 0.007
Youtube	0.847 ± 0.047	0.611 ± 0.084	0.810 ± 0.060	0.528 ± 0.000	0.598 ± 0.103
Table 8: Standard deviation of accuracy, number of examples  $= {18}$  ,PYTHIA (160M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.813 ± 0.021	0.511 ± 0.013	0.702 ± 0.102	0.790 ± 0.033	0.689 ± 0.060
AG-News-1	0.851 ± 0.020	0.511 ± 0.014	0.801 ± 0.086	0.891 ± 0.037	0.752 ± 0.032
AG-News-2	0.744 ± 0.034	0.509 ± 0.009	0.622 ± 0.083	0.711 ± 0.070	0.652 ± 0.047
AG-News-3	0.763 ± 0.026	0.508 ± 0.014	0.626 ± 0.016	0.775 ± 0.035	0.706 ± 0.027
Civil Comments	0.561 ± 0.029	0.489 ± 0.009	0.491 ± 0.032	0.540 ± 0.029	0.533 ± 0.030
DBPedia-0	0.851 ± 0.009	0.531 ± 0.044	0.811 ± 0.116	0.813 ± 0.057	0.812 ± 0.043
DBPedia-1	0.760 ± 0.037	0.546 ± 0.088	0.750 ± 0.129	0.718 ± 0.067	0.754 ± 0.041
DBPedia-2	0.800 ± 0.032	0.567 ± 0.110	0.850 ± 0.086	0.904 ± 0.018	0.851 ± 0.055
DBPedia-3	0.848 ± 0.025	0.528 ± 0.046	0.739 ± 0.133	0.785 ± 0.132	0.849 ± 0.011
IMDB	0.552 ± 0.031	0.550 ± 0.044	0.630 ± 0.016	0.608 ± 0.014	0.526 ± 0.025
Rotten Tomatoes	0.574 ± 0.029	0.539 ± 0.031	0.638 ± 0.037	0.618 ± 0.049	0.507 ± 0.010
SMS Spam	0.830 ± 0.112	0.613 ± 0.261	0.883 ± 0.130	0.911 ± 0.035	0.885 ± 0.045
SST	0.574 ± 0.019	0.561 ± 0.062	0.707 ± 0.024	0.636 ± 0.025	0.531 ± 0.015
Youtube	0.762 ± 0.116	0.584 ± 0.144	0.753 ± 0.140	0.726 ± 0.052	0.769 ± 0.084
Table 9: Standard deviation of accuracy, number of examples  $= {18}$  ,BLOOM (560M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.808 ± 0.030	0.551 ± 0.041	0.795 ± 0.049	0.874 ± 0.022	0.830 ± 0.034
AG-News-1	0.883 ± 0.014	0.577 ± 0.055	0.852 ± 0.070	0.911 ± 0.021	0.902 ± 0.011
AG-News-2	0.764 ± 0.019	0.540 ± 0.027	0.705 ± 0.063	0.812 ± 0.019	0.782 ± 0.019
AG-News-3	0.798 ± 0.025	0.521 ± 0.039	0.762 ± 0.046	0.813 ± 0.012	0.806 ± 0.028
Civil Comments	0.575 ± 0.054	0.498 ± 0.002	0.554 ± 0.029	0.543 ± 0.022	0.579 ± 0.044
DBPedia-0	0.886 ± 0.021	0.632 ± 0.096	0.884 ± 0.019	0.866 ± 0.013	0.859 ± 0.020
DBPedia-1	0.809 ± 0.031	0.593 ± 0.062	0.802 ± 0.030	0.783 ± 0.022	0.813 ± 0.023
DBPedia-2	0.886 ± 0.011	0.586 ± 0.078	0.916 ± 0.029	0.932 ± 0.013	0.899 ± 0.017
DBPedia-3	0.868 ± 0.022	0.565 ± 0.041	0.902 ± 0.016	0.908 ± 0.012	0.868 ± 0.023
IMDB	0.537 ± 0.040	0.543 ± 0.029	0.609 ± 0.029	0.556 ± 0.024	0.551 ± 0.036
Rotten Tomatoes	0.549 ± 0.035	0.554 ± 0.036	0.667 ± 0.036	0.550 ± 0.045	0.528 ± 0.020
SMS Spam	0.903 ± 0.027	0.768 ± 0.105	0.931 ± 0.014	0.861 ± 0.027	0.920 ± 0.011
SST	0.573 ± 0.009	0.564 ± 0.045	0.711 ± 0.022	0.594 ± 0.022	0.551 ± 0.022
Youtube	0.810 ± 0.072	0.759 ± 0.074	0.854 ± 0.044	0.773 ± 0.052	0.722 ± 0.113
Table 10: Standard deviation of accuracy, number of examples  $= {32}$  ,GPT-NEO (125M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.835 ± 0.020	0.499 ± 0.002	0.791 ± 0.100	0.855 ± 0.017	0.712 ± 0.028
AG-News-1	0.900 ± 0.012	0.504 ± 0.004	0.911 ± 0.022	0.926 ± 0.009	0.738 ± 0.028
AG-News-2	0.773 ± 0.012	0.510 ± 0.006	0.687 ± 0.034	0.742 ± 0.044	0.683 ± 0.025
AG-News-3	0.823 ± 0.024	0.514 ± 0.017	0.792 ± 0.022	0.833 ± 0.011	0.697 ± 0.023
Civil Comments	0.596 ± 0.024	0.499 ± 0.007	0.588 ± 0.052	0.536 ± 0.020	0.542 ± 0.016
DBPedia-0	0.890 ± 0.016	0.514 ± 0.026	0.880 ± 0.049	0.777 ± 0.055	0.782 ± 0.044
DBPedia-1	0.833 ± 0.018	0.522 ± 0.023	0.834 ± 0.061	0.768 ± 0.024	0.760 ± 0.022
DBPedia-2	0.912 ± 0.010	0.522 ± 0.031	0.916 ± 0.033	0.903 ± 0.012	0.859 ± 0.019
DBPedia-3	0.879 ± 0.021	0.520 ± 0.026	0.900 ± 0.019	0.863 ± 0.018	0.812 ± 0.022
IMDB	0.541 ± 0.041	0.510 ± 0.008	0.630 ± 0.017	0.575 ± 0.024	0.540 ± 0.018
Rotten Tomatoes	0.576 ± 0.042	0.495 ± 0.007	0.659 ± 0.064	0.568 ± 0.027	0.542 ± 0.019
SMS Spam	0.937 ± 0.017	0.856 ± 0.080	0.913 ± 0.044	0.953 ± 0.014	0.877 ± 0.036
SST	0.602 ± 0.015	0.509 ± 0.013	0.705 ± 0.021	0.605 ± 0.043	0.542 ± 0.003
Youtube	0.862 ± 0.045	0.626 ± 0.081	0.874 ± 0.046	0.762 ± 0.052	0.654 ± 0.050
Table 11: Standard deviation of accuracy, number of examples  $= {32}$  ,PYTHIA (160M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.825 ± 0.031	0.503 ± 0.003	0.802 ± 0.041	0.818 ± 0.014	0.739 ± 0.049
AG-News-1	0.880 ± 0.012	0.516 ± 0.026	0.898 ± 0.043	0.912 ± 0.034	0.829 ± 0.010
AG-News-2	0.771 ± 0.009	0.519 ± 0.025	0.736 ± 0.071	0.773 ± 0.039	0.684 ± 0.040
AG-News-3	0.810 ± 0.016	0.509 ± 0.013	0.753 ± 0.058	0.805 ± 0.031	0.742 ± 0.030
Civil Comments	0.587 ± 0.025	0.500 ± 0.011	0.549 ± 0.016	0.564 ± 0.062	0.555 ± 0.009
DBPedia-0	0.857 ± 0.035	0.592 ± 0.083	0.801 ± 0.125	0.822 ± 0.043	0.834 ± 0.038
DBPedia-1	0.802 ± 0.031	0.558 ± 0.047	0.870 ± 0.037	0.800 ± 0.041	0.813 ± 0.034
DBPedia-2	0.879 ± 0.018	0.609 ± 0.025	0.938 ± 0.018	0.920 ± 0.031	0.903 ± 0.021
DBPedia-3	0.866 ± 0.035	0.634 ± 0.110	0.812 ± 0.163	0.911 ± 0.004	0.876 ± 0.037
IMDB	0.553 ± 0.046	0.541 ± 0.024	0.636 ± 0.016	0.600 ± 0.018	0.536 ± 0.024
Rotten Tomatoes	0.589 ± 0.037	0.575 ± 0.039	0.713 ± 0.029	0.630 ± 0.026	0.527 ± 0.015
SMS Spam	0.933 ± 0.017	0.762 ± 0.033	0.956 ± 0.026	0.905 ± 0.056	0.917 ± 0.017
SST	0.579 ± 0.032	0.502 ± 0.020	0.739 ± 0.013	0.638 ± 0.035	0.562 ± 0.017
Youtube	0.799 ± 0.117	0.710 ± 0.182	0.887 ± 0.033	0.756 ± 0.115	0.850 ± 0.024
Table 12: Standard deviation of accuracy, number of examples  $= {32}$  ,BLOOM (560M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.812 ± 0.021	0.560 ± 0.046	0.832 ± 0.023	0.876 ± 0.012	0.833 ± 0.016
AG-News-1	0.904 ± 0.009	0.623 ± 0.051	0.924 ± 0.026	0.932 ± 0.007	0.923 ± 0.009
AG-News-2	0.786 ± 0.008	0.566 ± 0.023	0.791 ± 0.016	0.824 ± 0.010	0.797 ± 0.010
AG-News-3	0.822 ± 0.030	0.533 ± 0.029	0.800 ± 0.042	0.840 ± 0.015	0.825 ± 0.034
Civil Comments	0.591 ± 0.029	0.497 ± 0.003	0.568 ± 0.038	0.554 ± 0.029	0.614 ± 0.028
DBPedia-0	0.911 ± 0.011	0.676 ± 0.097	0.892 ± 0.017	0.898 ± 0.012	0.894 ± 0.023
DBPedia-1	0.823 ± 0.025	0.644 ± 0.115	0.838 ± 0.058	0.819 ± 0.018	0.833 ± 0.026
DBPedia-2	0.902 ± 0.015	0.622 ± 0.114	0.931 ± 0.032	0.940 ± 0.005	0.913 ± 0.007
DBPedia-3	0.883 ± 0.023	0.620 ± 0.129	0.891 ± 0.015	0.911 ± 0.006	0.889 ± 0.021
IMDB	0.557 ± 0.033	0.522 ± 0.025	0.641 ± 0.008	0.564 ± 0.017	0.577 ± 0.020
Rotten Tomatoes	0.572 ± 0.014	0.548 ± 0.045	0.710 ± 0.010	0.575 ± 0.048	0.572 ± 0.029
SMS Spam	0.882 ± 0.044	0.860 ± 0.036	0.946 ± 0.019	0.881 ± 0.013	0.925 ± 0.010
SST	0.570 ± 0.019	0.563 ± 0.032	0.705 ± 0.019	0.575 ± 0.028	0.542 ± 0.035
Youtube	0.810 ± 0.039	0.792 ± 0.081	0.923 ± 0.014	0.874 ± 0.029	0.832 ± 0.053
Table 13: Standard deviation of accuracy, number of examples  $= {48}$  ,GPT-NEO (125M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.846 ± 0.018	0.496 ± 0.007	0.871 ± 0.015	0.871 ± 0.005	0.732 ± 0.036
AG-News-1	0.923 ± 0.005	0.528 ± 0.044	0.942 ± 0.004	0.935 ± 0.009	0.780 ± 0.040
AG-News-2	0.798 ± 0.009	0.509 ± 0.004	0.731 ± 0.040	0.782 ± 0.018	0.722 ± 0.019
AG-News-3	0.845 ± 0.012	0.515 ± 0.014	0.787 ± 0.046	0.852 ± 0.011	0.718 ± 0.026
Civil Comments	0.605 ± 0.037	0.512 ± 0.009	0.607 ± 0.039	0.556 ± 0.014	0.556 ± 0.018
DBPedia-0	0.905 ± 0.020	0.549 ± 0.057	0.924 ± 0.020	0.830 ± 0.034	0.798 ± 0.021
DBPedia-1	0.840 ± 0.021	0.592 ± 0.062	0.867 ± 0.024	0.808 ± 0.031	0.780 ± 0.011
DBPedia-2	0.916 ± 0.009	0.636 ± 0.081	0.937 ± 0.017	0.923 ± 0.013	0.870 ± 0.022
DBPedia-3	0.888 ± 0.022	0.618 ± 0.096	0.927 ± 0.011	0.885 ± 0.007	0.827 ± 0.026
IMDB	0.557 ± 0.034	0.503 ± 0.008	0.632 ± 0.013	0.566 ± 0.030	0.545 ± 0.008
Rotten Tomatoes	0.601 ± 0.026	0.480 ± 0.014	0.683 ± 0.033	0.556 ± 0.033	0.562 ± 0.018
SMS Spam	0.956 ± 0.008	0.869 ± 0.044	0.966 ± 0.011	0.960 ± 0.008	0.891 ± 0.035
SST	0.612 ± 0.024	0.500 ± 0.025	0.702 ± 0.046	0.594 ± 0.042	0.560 ± 0.023
Youtube	0.872 ± 0.019	0.654 ± 0.063	0.904 ± 0.033	0.826 ± 0.046	0.666 ± 0.027
Table 14: Standard deviation of accuracy, number of examples  $= {48}$  ,PYTHIA (160M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.829 ± 0.029	0.507 ± 0.005	0.829 ± 0.071	0.847 ± 0.022	0.788 ± 0.028
AG-News-1	0.909 ± 0.010	0.543 ± 0.052	0.940 ± 0.011	0.935 ± 0.004	0.856 ± 0.014
AG-News-2	0.789 ± 0.016	0.511 ± 0.012	0.715 ± 0.057	0.793 ± 0.027	0.708 ± 0.011
AG-News-3	0.832 ± 0.018	0.530 ± 0.038	0.834 ± 0.029	0.843 ± 0.014	0.763 ± 0.015
Civil Comments	0.602 ± 0.030	0.506 ± 0.011	0.605 ± 0.044	0.592 ± 0.039	0.566 ± 0.008
DBPedia-0	0.889 ± 0.022	0.703 ± 0.085	0.933 ± 0.021	0.888 ± 0.024	0.873 ± 0.027
DBPedia-1	0.817 ± 0.023	0.734 ± 0.076	0.891 ± 0.028	0.836 ± 0.036	0.834 ± 0.024
DBPedia-2	0.900 ± 0.011	0.847 ± 0.037	0.949 ± 0.013	0.940 ± 0.009	0.914 ± 0.013
DBPedia-3	0.884 ± 0.020	0.815 ± 0.105	0.928 ± 0.024	0.917 ± 0.015	0.891 ± 0.037
IMDB	0.565 ± 0.033	0.545 ± 0.033	0.641 ± 0.020	0.604 ± 0.022	0.542 ± 0.015
Rotten Tomatoes	0.605 ± 0.015	0.505 ± 0.005	0.704 ± 0.025	0.672 ± 0.042	0.531 ± 0.018
SMS Spam	0.933 ± 0.009	0.636 ± 0.130	0.929 ± 0.058	0.919 ± 0.032	0.914 ± 0.027
SST	0.610 ± 0.030	0.489 ± 0.004	0.695 ± 0.020	0.673 ± 0.029	0.538 ± 0.011
Youtube	0.834 ± 0.049	0.797 ± 0.090	0.805 ± 0.095	0.851 ± 0.020	0.865 ± 0.012
Table 15: Standard deviation of accuracy, number of examples  $= {48}$  ,BLOOM (560M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.817 ± 0.024	0.539 ± 0.044	0.855 ± 0.030	0.877 ± 0.013	0.830 ± 0.023
AG-News-1	0.904 ± 0.007	0.561 ± 0.048	0.923 ± 0.027	0.939 ± 0.003	0.922 ± 0.010
AG-News-2	0.790 ± 0.026	0.576 ± 0.022	0.814 ± 0.009	0.839 ± 0.009	0.816 ± 0.014
AG-News-3	0.833 ± 0.015	0.550 ± 0.035	0.803 ± 0.017	0.852 ± 0.013	0.842 ± 0.018
Civil Comments	0.581 ± 0.039	0.499 ± 0.002	0.587 ± 0.018	0.576 ± 0.039	0.605 ± 0.036
DBPedia-0	0.915 ± 0.005	0.652 ± 0.095	0.917 ± 0.011	0.908 ± 0.019	0.909 ± 0.025
DBPedia-1	0.825 ± 0.037	0.633 ± 0.104	0.838 ± 0.032	0.829 ± 0.012	0.852 ± 0.011
DBPedia-2	0.887 ± 0.020	0.606 ± 0.088	0.950 ± 0.007	0.952 ± 0.011	0.916 ± 0.014
DBPedia-3	0.873 ± 0.033	0.611 ± 0.136	0.898 ± 0.034	0.927 ± 0.007	0.909 ± 0.008
IMDB	0.558 ± 0.029	0.515 ± 0.024	0.646 ± 0.010	0.563 ± 0.021	0.566 ± 0.035
Rotten Tomatoes	0.601 ± 0.020	0.533 ± 0.053	0.708 ± 0.015	0.579 ± 0.049	0.556 ± 0.033
SMS Spam	0.869 ± 0.023	0.825 ± 0.074	0.934 ± 0.016	0.898 ± 0.008	0.927 ± 0.004
SST	0.570 ± 0.038	0.554 ± 0.041	0.712 ± 0.033	0.609 ± 0.021	0.567 ± 0.016
Youtube	0.790 ± 0.050	0.819 ± 0.050	0.926 ± 0.014	0.891 ± 0.031	0.837 ± 0.055
Table 16: Standard deviation of accuracy, number of examples  $= {64}$  ,GPT-NEO (125M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.851 ± 0.024	0.502 ± 0.003	0.858 ± 0.009	0.869 ± 0.010	0.764 ± 0.030
AG-News-1	0.925 ± 0.002	0.529 ± 0.030	0.937 ± 0.011	0.938 ± 0.004	0.820 ± 0.021
AG-News-2	0.812 ± 0.007	0.513 ± 0.013	0.752 ± 0.051	0.795 ± 0.010	0.738 ± 0.008
AG-News-3	0.851 ± 0.007	0.503 ± 0.004	0.820 ± 0.020	0.855 ± 0.009	0.741 ± 0.019
Civil Comments	0.606 ± 0.029	0.500 ± 0.001	0.659 ± 0.032	0.566 ± 0.023	0.566 ± 0.018
DBPedia-0	0.910 ± 0.013	0.518 ± 0.020	0.912 ± 0.027	0.858 ± 0.019	0.825 ± 0.015
DBPedia-1	0.839 ± 0.027	0.542 ± 0.028	0.897 ± 0.016	0.824 ± 0.031	0.788 ± 0.018
DBPedia-2	0.916 ± 0.011	0.609 ± 0.106	0.953 ± 0.013	0.922 ± 0.012	0.882 ± 0.019
DBPedia-3	0.887 ± 0.028	0.527 ± 0.022	0.940 ± 0.013	0.904 ± 0.007	0.857 ± 0.020
IMDB	0.556 ± 0.024	0.506 ± 0.005	0.619 ± 0.030	0.574 ± 0.017	0.552 ± 0.009
Rotten Tomatoes	0.624 ± 0.024	0.485 ± 0.020	0.686 ± 0.040	0.577 ± 0.033	0.568 ± 0.019
SMS Spam	0.937 ± 0.018	0.905 ± 0.017	0.960 ± 0.021	0.961 ± 0.006	0.899 ± 0.014
SST	0.606 ± 0.022	0.508 ± 0.022	0.688 ± 0.047	0.602 ± 0.036	0.567 ± 0.017
Youtube	0.888 ± 0.028	0.715 ± 0.097	0.897 ± 0.046	0.878 ± 0.050	0.675 ± 0.019
Table 17: Standard deviation of accuracy, number of examples  $= {64}$  ,PYTHIA (160M)
Dataset	TART	In-context learning	Fine-tuning full	Fine-tuning layer	Adapters
AG-News-0	0.836 ± 0.018	0.509 ± 0.008	0.850 ± 0.027	0.856 ± 0.008	0.799 ± 0.029
AG-News-1	0.918 ± 0.007	0.543 ± 0.033	0.900 ± 0.024	0.933 ± 0.008	0.875 ± 0.015
AG-News-2	0.799 ± 0.012	0.515 ± 0.019	0.784 ± 0.018	0.831 ± 0.022	0.732 ± 0.017
AG-News-3	0.836 ± 0.011	0.504 ± 0.003	0.811 ± 0.034	0.853 ± 0.011	0.784 ± 0.022
Civil Comments	0.602 ± 0.030	0.510 ± 0.012	0.573 ± 0.035	0.611 ± 0.025	0.572 ± 0.013
DBPedia-0	0.905 ± 0.015	0.667 ± 0.052	0.936 ± 0.018	0.902 ± 0.022	0.882 ± 0.020
DBPedia-1	0.809 ± 0.022	0.687 ± 0.117	0.887 ± 0.039	0.852 ± 0.032	0.853 ± 0.008
DBPedia-2	0.881 ± 0.026	0.799 ± 0.075	0.955 ± 0.022	0.947 ± 0.009	0.922 ± 0.014
DBPedia-3	0.877 ± 0.014	0.793 ± 0.106	0.906 ± 0.027	0.921 ± 0.017	0.899 ± 0.024
IMDB	0.571 ± 0.033	0.539 ± 0.020	0.621 ± 0.039	0.618 ± 0.013	0.542 ± 0.012
Rotten Tomatoes	0.597 ± 0.025	0.536 ± 0.047	0.684 ± 0.053	0.672 ± 0.041	0.543 ± 0.024
SMS Spam	0.907 ± 0.045	0.659 ± 0.133	0.942 ± 0.024	0.931 ± 0.030	0.909 ± 0.033
SST	0.620 ± 0.039	0.495 ± 0.015	0.672 ± 0.039	0.716 ± 0.032	0.554 ± 0.020
Youtube	0.844 ± 0.059	0.765 ± 0.137	0.879 ± 0.058	0.865 ± 0.033	0.883 ± 0.016
Table 18: Standard deviation of accuracy, number of examples  $= {64}$  ,BLOOM (560M)