SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the second hypothesis, we first evaluate whether fine-tuning hurts task-agnosticity. For this we evaluate two sets of accuracies: accuracy of a model fine-tuned for the specific task and the accuracy of a model on the task but fine-tuned on the AG News dataset. From Figures 11 and 12, we see that there is a drop in accuracy—over  $25.77\%$  across models and datasets. For the second part, we again decompose the drop in accuracy into a representation drop and a reasoning drop. The representation drop is computed by training a linear probe over the two models (task-specific fine-tuned and AG News fine-tuned) and looking at the difference between them. The reasoning drop, as before, is computed by subtracting this representation drop from the total drop. Figure 13 shows that most of
Figure 13: Effects of fine-tuning on task agnosticity. Scatter plot of reasoning loss against representation loss when the base model is trained on AG-News-0 and evaluated on other tasks. Across datasets (each point in plot represents a dataset), fine-tuning majorly impairs reasoning when transferring to tasks outside the specific fine-tuned task.
(a) GPT-NEO (125M)
Figure 14: Effects of fine-tuning on task-agnosticity of adapters. Accuracy of task-specific fine-tuned adapter vs. accuracy of adapter fine-tuned on AG-News-0 and evaluated on task. Fine-tuning consistently hurts the generalization ability of adapters across datasets.
this drop in task-agnosticity can be attributed to over-fitting of the reasoning abilities over the task for which the models are fine-tuned.
Adapters. Since adapters do not modify the underlying representations of the model, we look at how a single adapter generalizes across tasks. For this we train an adapter on the AG News dataset and evaluate it on the other datasets. We compare this set of accuracies with those obtained by task-specific adapters in Figure 14. The main conclusion is that task-specific adapters are not agnostic learners and over-fit to the task for which they are fine-tuned.
C Details for TART implementation
This section contains the details on training TART's reasoning module and extended results on the choice of embeddings from Section 3.
C.1 TART's reasoning module
Architecture details. We use the standard GPT-2 architecture [32] for training our reasoning module. We set the embedding size to 256, number of decoder layers to 12, and number of heads to 8 for a total of 22 million parameters. Since the GPT-2 backbone outputs a sequence of embeddings,
Figure 15: TART reasoning module architecture. The reasoning module takes as input sequences of  $(x, y)$  pairs of dimension  $d$ . A linear layer is used to project  $d$  to the hidden dimension size of the GPT-2 backbone. Finally, a linear layer is applied to the outputs of the backbone to generate predictions for each  $x_{k}$  in the input sequence.
Figure 16: Training loss vs. number of steps. The plot shows the variation in training loss as a function of the number of steps of gradient descent for TART's reasoning module.
we additionally add a linear layer in the end to convert the output to scalar values (see Figure 15). Additionally, the binary labels  $y$  are encoded as a one-hot vector to match the input dimension  $d$  of the corresponding covariates  $x$ .
Training procedure. We trained TART's reasoning module with a context length of 258 (allowing for up to 129 in-context examples). The batch size was set to 64, learning rate to 0.0001 and the model was trained for a total of 24000 epochs. Each batch of training data consists of sampling a sequence of 258 examples using eq. (3). In addition to these hyperparameters, we used a curriculum on the input dimensions and on the number of examples in the sequence to train our module—the input dimensions started from a value of 4 and were incremented by 4 every 1000 epochs while the number of examples started from 18 and were incremented by 30 every 1000 epochs.
Combining reasoning module with base LLM. We trained the reasoning module with input dimension set to 16. However, most base models produce representations which are much higher dimensional (ranging from 784 to 2048). In order to reduce the dimensionality of these representations, we perform PCA on the output embeddings of the base model, learning the components using only the training points available for that specific task. The test examples are then projected onto these principal components to produce 16 dimensional input representations.
C.2 Choice of representations
As discussed in Section 3.3 there are two possible options for forming the representations, the vanilla embeddings and the leave-one-out (LOO) embeddings. Figure 18 shows the schematic differences between the two style of embedding. In Figure 17, we plot the average accuracies across different datasets for both vanilla and LOO embeddings, observing that the LOO embeddings consistently perform better across the different model families.
(b) PYTHIA (160M)
Figure 17: LOO embeddings vs. Vanilla Embeddings. Comparison of TART performance when using LOO embeddings and vanilla embeddings. Vanilla embeddings see a performance collapse, but LOO embeddings do not.
(a) Vanilla embeddings
(b) LOO embeddings
Figure 18: TART Embedding Protocols. (a) For the vanilla embeddings, the test example is appended to the training set and the sequence is passed to the base model. The representation for each train example in this sequence is taken as the average embedding across all its tokens. (b) For the LOO embeddings, we generate embeddings for each train example separately by placing all the other train examples before it in the prompt and averaging the embeddings over the final example's tokens.
In this section, we provide a formal statement of Theorem 1 from Section 3.4. Our theorem quantifies the expected error of the Transformer, trained on synthetic data, on natural language tasks in terms of the change in the two input distributions.
We begin by introducing some notation. We denote the class of Transformer family by
$$
\mathcal {T} _ {\Theta} := \left\{T _ {\theta}: \mathbb {R} ^ {(2 k + 1) \times d} \mapsto \mathbb {R} \quad | \quad \theta \in \Theta \right\}, \tag {7}
$$
where  $k$  represents the maximum number of in-context examples the Transformer can support,  $d$  represents the input dimensions, and  $\Theta$  represents the corresponding parameter class over which the Transformer family is defined.
Observe that the Transformer family  $\mathcal{T}_{\Theta}$  takes as input a sequence of  $k$  train examples, each corresponding to two tokens of hidden dimension  $d$ : a covariate  $x \in \mathbb{R}^d$  and a binary label, encoded as a one-hot vector in  $d$  dimension. This sequence of train examples is followed by a test example, for which we only have the features  $x_{k+1}$ .
Given this background, let  $P_{\mathrm{syn}}$  denote the synthetic distribution over sequences  $\{(x_1, y_1), \ldots, (x_k, y_x), (x_{k+1})\}$ . Similarly, let  $P_{\mathrm{NL}}$  denote the corresponding distribution over sequences derived from natural language tasks where  $x_i$  denotes the LLM embeddings of the example. Recall from Section 3.2.1, the synthetic training distribution  $P_{\mathrm{syn}}$  is given by
Sequence  $s_t:w_t\sim \mathcal{N}(0,I_d)$ $x_{i,t}\sim \mathcal{N}(0,I_d),$ $y_{i,t}\sim \sigma (\alpha \langle x_{i,t},w_t\rangle)$  for  $i\in [k]$  ，(8)
for each training point  $(x_{i,t},y_{i,t})$ . The test point is also sampled similarly from an independent standard normal distribution. Let  $\ell :\mathbb{R}\times \mathbb{R}\mapsto \mathbb{R}$  be the loss function used for evaluating the performance of the reasoning module. Further, let us denote the expected loss under a distribution  $P$
$$
\operatorname {e r r} _ {P} (T) := \mathbb {E} _ {(s, y) \sim P} [ \ell (T (s), y) ], \tag {9}
$$
and the corresponding empirical distribution over samples  $S$  by  $\hat{\mathrm{er}} r_P$ , where the dependence on the samples is implicit. Given these samples, we denote the empirical risk minimizer
$$
T _ {S} = \arg \min  _ {T \in \mathcal {T} _ {\Theta}} \frac {1}{| S |} \sum_ {s \in S} \ell (T (s), y). \tag {10}
$$
In addition to these notation, we make the following Lipschitz assumption on the loss function  $\ell$  and the Transformer model  $T$ .
Assumption 1. [Lipschitz loss.] For any two output labels  $y, y'$ , the loss function  $\ell$  is Lipschitz with constant  $c_1$ , that is,
$$
\left| \ell (T r (s), y) - \ell (T r (s), y ^ {\prime}) \right| \leq c _ {1} | y - y ^ {\prime} |. \tag {11}
$$
Assumption 2. [Lipschitz models.] For any two input sequences  $s, s'$ , each any model  $T \in \mathcal{T}_{\Theta}$  is Lipschitz with constant  $L$ , that is,
$$
\left| T (s) - T \left(s ^ {\prime}\right) \right| \leq L \| s - s ^ {\prime} \|. \tag {12}
$$
Given this setup, we are now ready to state a formal version of Theorem 1.
Theorem 2 (Formal version of Theorem 1). Let  $T_{S} \in \mathcal{T}_{\Theta}$  denote the trained reasoning module on set  $S$  of synthetic logistic regression tasks with  $n_{\mathrm{syn}}$  sequences sampled from distribution  $P_{\mathrm{syn}}$  in eq. (3). Let the loss function  $\ell$  satisfy Assumption 1 and the model class  $\mathcal{T}_{\Theta}$  satisfy Assumption 2. Then, with probability at least  $1 - \delta$ , we have
$$
\operatorname {e r r} _ {P _ {\mathrm {N L}}} \left(T _ {S}\right) \leq c _ {1} \max  (1, L) \cdot W _ {1} \left(P _ {\mathrm {N L}}, P _ {\mathrm {s y n}}\right) + c _ {1} \cdot \sqrt {\frac {2 V C \left(\mathcal {T} _ {\Theta}\right) \ln m}{n _ {\mathrm {s y n}}}} + 4 \sqrt {\frac {2 \ln (4 / \delta)}{n _ {\mathrm {s y n}}}} + \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right), \tag {13}
$$
where  $W_{1}$  denotes the Wasserstein-1 metric and  $VC(\mathcal{T}_{\Theta})$  represents the VC dimension of class  $\mathcal{T}_{\Theta}$
Proof. We begin by decomposing the error  $\mathrm{err}_{P_{\mathbb{N}L}}(T_S)$  into three components as
$$
\operatorname {e r r} _ {P _ {\mathrm {N L}}} \left(T _ {S}\right) = \underbrace {\operatorname {e r r} _ {P _ {\mathrm {N L}}} \left(T _ {S}\right) - \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right)} _ {\text {(I)}} + \underbrace {\operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right) - \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right)} _ {\text {(I I)}} + \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right). \tag {14}
$$
We now upper bound each of the terms (I) and (II) separately.
Bound on Term (I). Let  $\gamma$  denote an arbitrary joint distribution over the distributions  $P_{\mathrm{syn}}$  and  $P_{\mathrm{NL}}$ . Then, we can bound the first term as
$$
\begin{array}{l} \operatorname {e r r} _ {P _ {\mathrm {N L}}} \left(T _ {S}\right) - \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right) = \mathbb {E} _ {P _ {\mathrm {N L}}} \left[ \ell (T (s), y) \right] - \mathbb {E} _ {P _ {\mathrm {s y n}}} \left[ \ell \left(T \left(s ^ {\prime}\right), y ^ {\prime}\right) \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {\gamma} [ \ell (T (s), y) - \ell (T (s ^ {\prime}), y ^ {\prime}) ] \\ \stackrel {(\mathrm {i i})} {\leq} \inf  _ {\gamma} \mathbb {E} _ {\gamma} | \ell (T (s), y) - \ell (T \left(s ^ {\prime}\right), y ^ {\prime}) |, \tag {15} \\ \end{array}
$$
where (i) follows from the independence of the two expectations and (ii) follows from that (i) holds for any arbitrary joint distribution  $\gamma$ . The final bound on this term now follows:
$$
\begin{array}{l} \inf  _ {\gamma} \mathbb {E} _ {\gamma} | \ell (T (s), y) - \ell (T (s ^ {\prime}), y ^ {\prime}) | = \inf  _ {\gamma} \int | \ell (T (s), y) - \ell (T (s), y ^ {\prime}) + \ell (T (s), y ^ {\prime}) - \ell (T (s ^ {\prime}), y ^ {\prime}) | d \gamma \\ \stackrel {(i)} {\leq} c _ {1} \inf  _ {\gamma} \int | y - y ^ {\prime} | - \| T (s ^ {\prime}) - T (s) \| d \gamma \\ \stackrel {\text {(i i)}} {\leq} c _ {1} \max  (1, L) \cdot \inf  _ {\gamma} \int | y - y ^ {\prime} | - \| s ^ {\prime} - s \| d \gamma \\ = c _ {1} \max  (1, L) \cdot W _ {1} \left(P _ {\mathrm {N L}}, P _ {\mathrm {s y n}}\right), \tag {16} \\ \end{array}
$$
where the inequalities (i) follows from Assumption 1 and (ii) follows from Assumption 2. This completes the bound on Term (I).
Bound on Term (II). Using a standard generalization bound [34, see Theorem 26.5], we have with probability at least  $1 - \delta$
$$
\begin{array}{l} \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right) - \operatorname {e r r} _ {P _ {\mathrm {s y n}}} \left(T _ {S}\right) \leq \mathcal {R} (\ell \circ \mathcal {T} _ {\Theta}) + 4 \sqrt {\frac {2 \ln (4 / \delta)}{n _ {\mathrm {s y n}}}} \\ \leq c _ {1} \cdot \mathcal {R} (\mathcal {T} _ {\Theta}) + 4 \sqrt {\frac {2 \ln (4 / \delta)}{n _ {\mathrm {s y n}}}} \\ \stackrel {(i)} {\leq} c _ {1} \cdot \sqrt {\frac {2 \operatorname {V C} \left(\mathcal {T} _ {\Theta}\right) \ln m}{n _ {\text {s y n}}}} + 4 \sqrt {\frac {2 \ln (4 / \delta)}{n _ {\text {s y n}}}} \tag {17} \\ \end{array}
$$
where  $\mathcal{R}(\ell \circ \mathcal{T}_{\Theta})$  denotes the Rademacher complexity of the class  $\mathcal{T}_{\Theta}$  composed with the loss function  $\ell$  and inequality (i) follows from Sauer's Lemma.
Combining the bounds in equations (16) and (17) completes the proof of the theorem.
D Extension to multi-class classification
In this section, we present an extension of TART to the multi-class classification setup. Recall from Section 3.2 that the module was trained on synthetic binary tasks sampled from the logistic model as
$$
\text {S e q u e n c e} s _ {t}: w _ {t} \sim \mathcal {N} (0, I _ {d}), \quad x _ {i, t} \sim \mathcal {N} (0, I _ {d}), \quad y _ {i, t} \sim \sigma (\alpha \langle x _ {i, t}, w _ {t} \rangle) \quad \text {f o r} i \in [ k ]. \tag {18}
$$
Observe that each sequence corresponded to a single hyperplane  $w_{t}$  which defined the binary classification problem. For the multi-class extension, suppose that there are  $L$  labels. For each sequence  $s_t$ , we sample one hyperplane per class in  $d$ -dimensional space:
$$
w _ {t, l} \sim \mathcal {N} (0, I _ {d}) \quad \text {f o r} l \in [ L ]. \tag {19}
$$
Each covariate  $x_{i,t}$  is sampled again from the standard Normal distribution as before. The label  $y_{i,t}$  is sampled from the multinomial distribution over  $[L]$  given by:
$$
\mathbb {P} \left(y _ {i, t} = l\right) \propto \exp \left(\left\langle x _ {i, t}, w _ {t, l} \right\rangle\right). \tag {20}
$$
Given this distribution over sequences  $s_t$ , we train the TART module as before, but with the linear layer with output dimension  $L$  instead of 1 as with the binary classification setup. We evaluate our
Dataset	Num Labels	Test Size	Max char length	Max token length	Avg. char length	Avg. token length
AGNews	4	2000	814	238	236.97	52.31
Ecommerce	4	2000	50328	10544	678.768	158.02
DBPedia-14	14	1988	691	240	284.08	67.52
Table 1: Dataset (test) statistics for multi-class datasets.
Dataset	Method	k=18	k=32	k=48	k=64
AGNews	TART	63.55 (+38.5)	64.45 (+39.6)	71.15 (+45.9)	72.35 (+45.4)
ICL	25.1	24.9	25.3	27.0
Ecommerce	TART	78.2 (+52.3)	79.4 (+52.6)	75.4 (+47.1)	79.3 (+50.8)
ICL	25.1	24.9	25.3	27.0
DBPedia-14	TART	52.4 (+45.1)	63.8 (+52.8)	63.1 (+49.5)	68.4 (+59.3)
ICL	7.3	11.0	13.6	9.1
Table 2: TART vs. Base ICL Performance on multi-class classification tasks. Evaluations of GPT-Neo-125M model in the multi-class setup.  $k =$  Number of in-context examples.
multi-class variant of TART on AGNews-4, Ecommerce-4 and the DBPedia-14 datasets (see Table 1 for dataset statistics).
Table 2 shows the evaluation our proposed multi-class extension of TART on the above three datasets. While the in-context learning performance of the base model (GPT-Neo-125M) is close to random, we see that the TART substantially improves upon this performance. It provides improvement of up to  $45.9\%$  on AGNews,  $52.6\%$  on Ecommerce, and  $59.3\%$  on DBPedia-14 datasets.
E Details for experimental evaluation
We describe supplementary experimental details from Section 4 as well as additional results for the natural language benchmark evaluations (Section E.2) and results for other modalities (vision and audio) (Section E.5).
We begin by providing dataset statistics and details of the baselines.
E.1.1 Dataset construction and statistics
Table 4 and 3 provides a detailed breakdown of dataset statistics. For each dataset, we use the original test sets with the exception of Civil Comments [8], AG News [48] and DBPedia [48]. For the multi-class datasets—AG News [48] and DBPedia [48] — we construct 4 binary classification tasks for each datasets. More concretely, AG News labels news articles into four categories: World, Sports, Business, and Science/Technology. We create a separate binary classification task for each category, sampling negatives from the remaining classes. DBPedia is a 14-way ontology classification dataset. We create 4 separate binary classification tasks for the educational institution, company, artist, and athlete ontologies, sampling negatives from the remaining classes. For the train set, we sample a class-balanced set of 64 examples from the original dataset. For each dataset, we sample 5 separate training sets, using 5 different random seeds. In evaluations, we evaluate TART and the baseline methods across each of these 5 different training sets.
Dataset	Test size	Max char length	Max token length	Avg. char length	Avg. token length
AG-News-0	3800	732	259	237.07	51.36
AG-News-1	3800	814	213	232.01	51.46
AG-News-2	3800	814	225	236.10	52.25
AG-News-3	3800	892	259	234.86	51.38
Civil Comments	11576	1000	634	272.72	61.73
DBPedia-0	10000	2081	629	300.94	65.83
DBPedia-1	10000	2081	629	298.81	66.91
DBPedia-2	10000	2081	883	286.85	66.53
DBPedia-3	10000	2081	629	275.81	63.88
IMDB	25000	12988	2972	1293.79	292.82
Rotten Tomatoes	1066	261	63	115.52	25.36
SMS Spam	4181	612	258	81.46	23.76
SST	2210	256	60	102.40	22.34
Youtube	250	1125	292	112.50	31.84
Table 3: Dataset (test) statistics for all NLP datasets.
Dataset	Max char length	Max token length	Avg. char length	Avg. token length
AG-News-0	701	256	236.15	51.53
AG-News-1	749	180	232.48	51.61
AG-News-2	735	256	241.70	53.91
AG-News-3	1002	258	241.21	53.21
Civil Comments	1000	347	280.97	63.44
DBPedia-0	707	207	300.48	65.79
DBPedia-1	1023	280	299.89	66.57
DBPedia-2	628	203	288.21	66.22
DBPedia-3	758	203	279.45	64.24
IMDB	7068	1630	1284.20	290.00
Rotten Tomatoes	260	62	112.46	24.82
SMS Spam	911	217	106.90	31.87
SST	248	56	101.97	22.34
Youtube	1089	767	90.12	29.98
Table 4: Dataset (train) statistics for all NLP datasets.
(b) PYTHIA (160M)
Figure 19: Comparison of all methods. TART significantly improves base in-context learning performance and is competitive with full-finetuning across model families.
E.1.2 Baseline methods
For each dataset, we compare TART to 4 baseline task-adaptation methods: 1) in-context learning, 2) full fine-tuning, 3) last layer fine-tuning, and 4) adapters. The last layer fine-tuning and the adapters are trained as follows:
(a) Number of examples  $= 18$
(b) Number of examples  $= 32$
(c) Number of examples  $= 48$
(d) Number of examples  $= 64$
Figure 20: Comparison of TART and task-adaptation approaches (GPT-NEO (125M)). We see that for GPT-NEO (125M), TART outperforms in-context learning and is competitive with full fine-tuning and adapters across all  $k$ .
- Last layer fine-tuning: Freeze all layers of transformer but the final transformer block and the language modeling head.
(d) Number of examples  $= 64$
Figure 21: Comparison of TART and task-adaptation approaches (PYTHIA (160M)). We see that for PYTHIA (160M), TART outperforms in-context learning and adapters and is competitive with full fine-tuning across all  $k$ .
- Adapter: Combine a frozen LLM base transformer model with a trainable adapter head—an MLP composed of a single linear layer followed by non-linearity.
(d) Number of examples  $= 64$
Figure 22: Comparison of TART and task-adaptation approaches (BLOOM (560M)). We see that for BLOOM (560M), TART outperforms in-context learning and adapters and is competitive with full fine-tuning across all  $k$ .
Hyperparameter search. For each baseline, we perform an extensive hyperparameter search over number of epochs and learning rate for each dataset in order to optimize performance. We search over a range of learning rates (1e-3, 1e-4, 3e-5, 1e-5, 8e-6), and range of epochs (5, 10, 15, 20, 50). For all models  $< 1$ B parameters, we use a batch size of 1. For all models  $>1$ B parameters, we use a batch
Model	TART	GPT-J (6B)	OPT (175B)	BLOOM (176B)	GPT-3 (175B)
Accuracy	0.634	0.608	0.637	0.595	0.673
Table 5: RAFT (HELM) Binary Classification Performance (Average Accuracy). TART is used with GPT-NEO (125M) model which is  ${1000}\mathrm{x}$  smaller than the corresponding 175B parameter models. TART outperforms BLOOM (176B) and is competitive with OPT (175B) and GPT-3 (175B).
(b) PYTHIA (160M)
Figure 23: Beyond context window constraints. Performance comparison with respect to number of in-context examples. Base in-context learning is bound with respect to total numbers of examples and performance saturates. TART is not bound by context length, and performance continues to scale as number of examples increases.
size of 8. We use these same batch sizes at evaluation time. We perform our hyperparameter searches with a fixed number of train samples (64). We run our hyperparameter searches over 3 random seeds.
E.2 NL benchmarks
In this section, we provide additional results deferred from Section 4 on the NLP benchmark evaluations, RAFT evaluations and demonstration of TART's data-scalability.
E.2.1 Performance on benchmark datasets
Figure 19 shows the performance of the baseline methods with TART averaged across the suite of 14 datasets. TART, while being task-agnostic, shows similar performance quality to task-specific approaches across the different model families, and consistently outperforms in-context learning.
Figures 20, 21, and 22 show the scatter plots of the accuracies of TART with the baseline methods across datasets and different values of in-context examples  $k$ . An interesting observation is that as the number of examples  $k$  increases from 18 to 64, the performance of fine-tuning improves at a better rate than that of TART.
E.2.2 Real-world Annotated Few-shot Tasks (RAFT) evaluation
For our evaluations on the RAFT benchmark [2], we follow the protocol (same train and test sets) used in HELM benchmark. The HELM benchmark [22] contains the evaluation results for many open and closed models enabling us to accurately compare the performance of TART with other models. We evaluate TART on all RAFT binary classification datasets (twitter-complaints, neurips-impact-statement-risks, overulling, ade-corpusv2, tweet-eval-hate, terms-of-service, tai-safety-research) with the exception of systematic-review-inclusion which contains zero positive samples in the train set. TART requires at least one example of each class in the training set. Table 5 contains a detailed performance comparison of TART with respect to other models. TART when combined with GPT-NEO (125M) is able to outperform BLOOM (176B) and is competitive with OPT (175B) and GPT-3 (175B), all of which have  $1000\mathrm{x}$  more parameters.
E.2.3 Beyond context length: TART is data-scalable
Setup. For these evaluations, we use the a subset of 6 datasets: AG-News-0, DBPedia-0, SST, SMS Spam, Youtube and Rotten Tomatoes. We evaluate the performance of TART over  $k = \left\lbrack  {{18},{32},{48},{64}}\right\rbrack$  , 128, 192, 256] where  $k$  is the number of in-context examples. When evaluating our base models, we
Figure 24: Stream embeddings. Another protocol for generating representations for in-context examples where each example is embedded by the base model separately.
evaluate over  $k = [8, 24]$ —values of  $k$  that maximize the context window. We use a lower-bound of 8 given that the maximum input sequence length in the training set for AG News is 256. With such a sequence length, the maximum number of in-context examples that fit in the context-window is 8, hence the lower bound.