Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements (2024)

Abstract: Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format’s influence on the in-context learning performance. We evaluate the impact of the prompt template across 21 models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.

heading: 1 Introduction
text: Pretrained language models have emerged as a dominant paradigm for solving many NLP problems in a unified framework (Brown et al., 2020; Chowdhery et al., 2022; Scao et al., 2023; Touvron et al., 2023a). In particular, these models can achieve impressive downstream results with just a few demonstrations given as a part of their input (Liu et al., 2021; Min et al., 2022c), which is often called a prompt in this case.
These few-shot or in-context learning (ICL) abilities (Brown et al., 2020) of large models are a subject of frequent study, as the primary factors behind them are not yet fully understood. For example, one
line of work investigates in-context learning within different theoretical frameworks (Xie et al., 2022; Garg et al., 2022; Akyürek et al., 2023). In addition, multiple publications study the importance of different prompt attributes, such as the order of input demonstrations (Lu et al., 2022a) and their labels (Min et al., 2022d).
As shown in Zhao et al. (2021); Min et al. (2022a), the prompt format (i.e., a transformation from a set of examples to a natural language input) is also highly important. However, this aspect is often overlooked in most existing studies. Namely, works proposing modifications of ICL frequently present their results for a specific template without specifying the criteria guiding its selection. Furthermore, even when the results are averaged over a set of templates, they are compared to methods that were evaluated on a different set of templates. We illustrate this common discrepancy in Appendix A. Such inconsistency can lead to a misinterpretation of the reported results: the difference between the performance of two methods may be explained by the variation across prompt formats rather than the methods themselves.
In this work, we evaluate the template sensitivity of 21 models from 8 families, including state-ofthe-art open-access models, such as Llama 2 (Touvron et al., 2023b) and Falcon (Almazrouei et al., 2023), as well as latest instruction-tuned models, such as Mistral (Jiang et al., 2023) and Llama 3 Instruct (AI@Meta, 2024). We show that this issue persists regardless of the model size and the number of demonstrations. Moreover, comparing various in-context learning enhancements while taking the template influence into account renders the superiority of one method over others less apparent. Therefore, it is likely that the gains reported for advanced prompting methods can often be attributed to a luckily chosen template.
6287
Crucially, there are no universally best templates for a given task. The best performing demonstration format for a fixed evaluation setting (i.e., the dataset, the model, the demonstration set, and the prediction method) does not transfer consistently across models (even within the same family), demonstration sets, or different prediction methods. We find this concerning, as even the best template for a given setting can produce poor results after slight changes, which makes “tuning” the template a very difficult task.
As a first step towards addressing template sensitivity in a practical way, we propose Template Ensembles — a test-time augmentation approach that averages model predictions over several prompt formats. This method is easy to implement and increases the average performance across templates for multiple prompting methods while reducing the sensitivity of these methods to the template choice.
In summary, our contributions are as follows:
1. We conduct a broad evaluation1 of prompt template sensitivity across 21 models and 4 datasets, showing that the performance gains similar to using in-context learning improvements can be achieved solely by selecting a proper template.
2. We show that the choice of the best template depends on a combination of factors and that it is not possible to transfer the best template between models or prompting methods without a negative impact on quality.
3. We propose Template Ensembles as a baseline solution for improving the template robustness for in-context learning.
1Our code and results of all evaluations can be found at github.com/yandex-research/mind-your-format

heading: 2 Background and Related Work

heading: 2.1 In-Context Learning
text: An important property of LLMs is their ability to learn new tasks from only a few demonstrations (Radford et al., 2019; Brown et al., 2020). This capability, known as in-context learning, forms the focus of our work. We focus on sequence classification, as it is the most widely studied task for understanding and improving ICL performance.
Formally, classifying an input xtest with incontext learning can be described as finding the class c in the space of label tokens C that yields a sequence with the highest probability according to a language model. The input sequence consists of demonstration inputs and labels (xi, yi) and a test input (xtest, c); to obtain a natural language input, demonstrations are formatted with a template.
Each template consists of four components: input and output verbalizers vI(x) and vO(y, C) that transform (xi, yi) into a natural language text, an intra-separator to divide input from output, and an inter-separator to join several demonstrations. Figure 1 shows an example of transforming a set of demonstrations into a context for ICL.

heading: 2.2 In-Context Learning Analysis
text: Recent work has shown that ICL can perform at levels comparable to finetuning (Chowdhery et al., 2022; Hoffmann et al., 2022). Still, in-context learning is known to be highly dependent on the way the model input is formed: a prompt is defined by several components, and altering any of them can lead to unpredictable changes in performance.
Template Selection There are multiple ways to construct a template for a task. The most straightforward approach is to use minimal templates (vI = {x}, vO = {C[y]}) or universal verbalizers like “input/output”, as done in Wang et al. (2023) and Wei et al. (2023).
Another strategy is to create task-specific templates. Jiang et al. (2020) generate paraphrases of templates for the relation extraction task. Authors show the sensitivity of masked language models to the prompt format and propose to ensemble predictions over the best templates. Compared to this method, our approach is task-agnostic and does not require evaluating all templates in advance.
Several studies aim to find templates that directly optimize in-context learning performance (Shin et al., 2020; Gao et al., 2021). Our work unifies the results of previous research, using the verbalizers proposed by Gao et al. (2021), as well as minimal and universal templates.
Choice and Order of Demonstrations The choice of examples for ICL is highly important, as they enable the model to condition on correct input and label distributions for the task (Wu et al., 2023; Nguyen and Wong, 2023; Min et al., 2022d; Chang and Jia, 2023). Furthermore, the order of examples also significantly affects the results and does not transfer between models even within the same family (Lu et al., 2022b; Zhao et al., 2021).
In this work, we analyze two recent methods for selecting demonstrations. Wang et al. (2023) propose learning latent concept variables for a task and using them to find examples that can best predict the task concept. We refer to this method as Implicit Topic Models or ITM. In turn, Z-ICL (Lyu et al., 2023) generates pseudo-demonstrations by retrieving most similar examples to the test sentence from an unlabeled dataset and assigning random labels to retrieved examples.
Crucially, both methods are evaluated on single templates that differ across two works. Therefore, it is unclear whether the reported performance gains arise from the methods themselves or from a particular combination of the example selection strategy, the model, and the chosen template.
Prediction Methods The standard approach for classification with LLMs is to compute the sequence probability with each of the possible labels and select the label with the highest probability. We refer to this method as DIRECT further on.
Alternatively, one can use more advanced prediction methods that aim to reduce the variance across prompt formats. The CALIBRATION method (Zhao et al., 2021) computes a correction factor based on the deviation of the model’s predictions for a placeholder input from a uniform distribution over labels and applies this factor to test set predictions.
Recent work has proposed multiple improvements of this method (Fei et al., 2023; Han et al., 2023; Zhou et al., 2024); to limit the scope of our study, we focus only on the base CALIBRATION approach in this work. Lastly, the CHANNEL prompting technique, proposed in Min et al. (2022b), maximizes P (x|y) instead of P (y|x).
Both of these methods aim to mitigate the issue of ICL sensitivity to the prompt template choice. However, as we show in Appendix A, these methods are evaluated on their own sets of templates. In this paper, we strive for a more unified view on the robustness of advanced prompting methods and compare their performance across a broader range of templates and models.
Prompt and Template Robustness Although the problem of prompt robustness is relatively wellknown, until recently, the discussion of template robustness has been limited. Notably, Sclar et al. (2023) present a highly relevant study of prompt format sensitivity, reporting a significant performance variation across formats even for large models or minor template changes. While their experiments are conducted in the standard setup (randomly selected examples and default prompting), our work instead focuses on alternative prompting and example selection methods, several of which (Zhao et al., 2021; Min et al., 2022b) were proposed to improve the prompt robustness of ICL. Similarly to papers in other subfields of machine learning arguing for a more consistent methodology (Dacrema et al., 2019; Musgrave et al., 2020; Platonov et al., 2023), the goal of our work is to demonstrate that disparate experiment setups lead to an invalid comparison of competing methods.
Moreover, several works study prompt robustness in a broader sense by considering models that use natural language instructions instead of labeled demonstrations (Webson and Pavlick, 2022; Leidinger et al., 2023; Weber et al., 2023). Recently, Mizrahi et al. (2023) have shown that very similar instructions can lead to drastic differences in task performance for a variety of instruction-tuned models. Although we study a similar issue, the focus of our work is on in-context learning and the transfer of best prompts between evaluation setups. Still, we find that instruction-tuned models lack in-context robustness as well, which confirms previous observations and emphasizes the need for language model evaluation that takes prompt design into account.

heading: 3 Setup & Methodology

heading: 3.1 Models and Data
text: We evaluate the robustness of in-context learning to template selection across a wide range of models on classification tasks. All models used in our work are listed in Table 1: we run experiments on model families frequently used in literature (such as OPT and BLOOM), as well as the latest models with the highest quality (such as Llama 2 and Falcon).
In preliminary experiments, we observed that the performance of some models in the few-shot regime lags behind their zero-shot results. Hence, we excluded these models from further investigation. Further details regarding this selection procedure can be found in Appendix B.
We experiment with 4 sequence classification datasets: SST-2 (Socher et al., 2013), DBPedia ontology classification task (Lehmann et al., 2015), AGNews (Zhang et al., 2015), and TREC Question Classification (Li and Roth, 2002). Although these datasets are frequently used in ICL studies, there is no consensus regarding the templates that should be used for each task.
One can construct an input for in-context learning from a set of demonstrations by using a template consisting of four parts, as illustrated in Figure 1. We present all options for verbalizers and separators for each dataset we study in Table 2.
Any combination of these components results in a valid template. This set of options results in 216 possible prompt formats for SST-2 and 168 for DBPedia, AGNews and TREC. A single evaluation run of all models on 10 random templates in one setup takes 17–48 hours on a single NVIDIA A100-80GB GPU, depending on the dataset.

heading: 3.2 Methods
text: Along with studying the robustness of standard in-context learning, we consider its improvements proposed in prior work. We focus on two main directions of ICL enhancements mentioned in Section 2: example selection and prediction methods. For each setting, we aggregate the results over 3 random seeds for example selection, with 10 random templates used for each seed and report the mean and standard deviation of classification accuracy, unless specified otherwise.
As a baseline for demonstration selection, we choose the most straightforward approach of selecting N random examples from the training dataset. Intuitively, selecting more relevant examples for ICL should yield better performance. Therefore, we investigate the template sensitivity of two demonstration selection methods described in Section 2.1: ITM and Z-ICL. Specifically, we select N = 4 examples using official implementations of each method.
Importantly, ITM requires training a concept model before choosing the examples. For GPT2 Large, this procedure takes approximately 30 hours on a single NVIDIA A100-80GB. Repeating it for each model would be infeasible, especially given that the largest model has 86 times more parameters. Therefore, we use the checkpoints of the GPT2 Large concept model provided by the authors to select demonstrations. Also, we reuse the same examples for all models, leveraging authors’ observations that demonstrations chosen with ITM can be transferred between models.
As discussed in Section 2, more advanced prediction techniques can improve in-context learning accuracy. Therefore, we compare DIRECT prompting with CHANNEL (Min et al., 2022b) and CALIBRATION (Zhao et al., 2021) prediction methods.

heading: 4 Evaluation

heading: 4.1 Baseline Results
text: We begin with analyzing the robustness of language models to the template choice in the baseline setup. Specifically, we evaluate models in zero-shot and few-shot settings, selecting 2/4 random demonstrations and using the DIRECT prediction method.
Our results in Table 3 show that even the most capable models such as Falcon and Llama 2 are highly sensitive to the prompt format; Appendix C contains the results for the full set of 19 base models, and Appendix L reports the results for instructiontuned models. Although the variance caused by this sensitivity makes it harder to observe the increase in ICL performance with the model size or the number of demonstrations, both trends still persist. However, even the largest models have standard deviations of scores up to 35% of their mean values.
To mitigate this lack of robustness, we could remove consistently underperforming prompt formats from the template pool. We analyze the impact of separate template components in Appendix D and find that there are no specific parts (for example, verbalizers or separators) which could be excluded from evaluation. Furthermore, we observe that a combination of “suboptimal” parts may result in an optimal template.

heading: 4.2 Prediction Methods
text: Next, we aim to evaluate the performance of different prediction methods in a unified setting. Ideally, we would like these modifications to reduce the
variance across templates, making the model behavior less dependent on the input format.
We evaluate CHANNEL and CALIBRATION methods in the 2-shot setting along with the DIRECT baseline. As depicted in Figure 2, both CHANNEL and CALIBRATION generally exhibit improved performance in comparison with DIRECT. Still, for a number of models and datasets, the range of scores for DIRECT substantially overlaps with those of advanced methods. This suggests that there are templates reaching the best performance with the DIRECT prediction method.
Additionally, Table 10 of Appendix E reveals that despite CALIBRATION yielding the highest mean accuracy more often than other methods, it is more sensitive to the template choice than CHANNEL. Similar findings for instruction-tuned models are contained in Appendix L. Therefore, the choice of the prediction method should likely rely on the downstream usage scenario and the target evaluation setting.

heading: 4.3 Example Selection Methods
text: Another area of ICL improvements that we evaluate on the matter of template sensitivity is the example selection strategy. We compare ITM and Z-ICL methods to the RANDOM baseline in 4-shot setting, since using 4 demonstration was the main evaluation setting in the works proposing these methods. We use DIRECT prediction method to evaluate the gains of advanced example selection strategies independently from other ICL modifications.
Results in Figure 3 and Table 14 illustrate that when taking template sensitivity into account, advanced example selection methods often perform worse than random choice baseline. ITM increases the average performance in most cases but still has a remarkably high standard deviation across templates. Examples selected using the Z-ICL method lead to more consistent but worse performance.
Note that our evaluation setup differs from those described in the original works, which might explain the discrepancy between our findings and the results reported by authors. Namely, we use the DIRECT prompting method and sample 10 random templates that may not include the templates used by authors of ITM and Z-ICL. To confirm template instability for prediction methods in their original implementations, we reproduce both methods and report our findings in Appendix F. We observe high sensitivity to the prompt format, which raises a question of how much the reported gains of these methods can be attributed to the methods themselves and not to the template choice.
We conclude that the prompt format should be viewed as important as the example selection or the prediction method for ICL evaluation. However, the search space of possible templates is infinite, which makes exhaustive search for each combination of the dataset, the model and the number of examples impractical. Ideally, the best template for one setting would be optimal for all others or at least for similar settings. However, as we demonstrate in the following section, this is not the case.

heading: 5 Template Transfer Evaluation

heading: 5.1 Setup
text: We begin by defining a successful transfer between ICL settings. In order to do so, we evaluate how the quality of model predictions varies across 30 random templates from Table 2. The results described in Appendix H demonstrate that the top-10 template on average yields 90% of the best template score. Therefore, if a prompt format is present in top-10 for both of the two compared setups, we can consider this an instance of successful transfer.
To compare sets of the best templates for a pair of settings, we compute Intersection-over-Union (IoU), also known as the Jaccard similarity coefficient (Jaccard, 1912), for top-10 best templates in each setting. We also consider using the ρ rank correlation coefficient (Spearman, 1904) as another measure of template transfer. However, its value can increase when low-performing templates have similar rankings in different ICL setups, while the transfer of efficient templates remains low. Still, we provide the results for this metric in Appendix I.
G P T -J
G P T -N
eo X O P T 6. 7B O P T 30 B O P T 66 B
B LO
O M
1. 7B
B LO
O M
3B
B LO
O M
7. 1B
P yt
hi a
6. 9B
P yt
hi a
12 B LL aM A 7B
LL aM
A 13
B
LL aM
A 30
B
LL aM
A 65
B Fa lc on 1B Fa lc on 7B Fa lc on 40 B
Ll am
a 2
13 B
Ll am
a 2
70 B
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B
Falcon 40B Llama 2 13B Llama 2 70B 0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Figure 4: IoU of top-10 templates for all base models with 2 random demonstrations and the DIRECT prediction method on the DBPedia dataset.

heading: 5.2 Transfer Between Models
text: Next, we analyze the transfer of the bestperforming templates between models in the baseline setup. Specifically, we collect the results of each model in the 2-shot learning setting with DIRECT prediction method and RANDOM demonstrations (fixed throughout the experiments) for 30 templates. A heatmap of IoU for the transfer of top-10 best templates between 19 base models on the DBPedia dataset is presented in Figure 4; for other datasets, please see Appendix J.
We observe that the IoU values exceed 0.5 only for a few model pairs on all datasets, meaning that the capacity for template transfer between models in the same setup is generally low. This is especially concerning for models within a single family: as these models are trained on the same data and have the same architecture, one would expect them to perform similarly on the same prompt formats.
These observations signify that comparing ICL methods across models with a single template can lead to incorrect conclusions: a template that is effective for one model can easily be one of the worst choices for another model.

heading: 5.3 Transfer Between Prediction Methods
text: As discussed in Section 4.2, no prediction method that we evaluate can consistently outperform others across all models and datasets. Therefore, to find an optimal setup for a new ICL improvement, one needs to evaluate every prediction technique in multiple templates. We investigate the possibility of finding a universally optimal prompt for different methods to reduce the total computational cost.
To answer this question, we calculate the IoU between top 10 performing templates for each method for a fixed set of demonstrations. Results in Table 4 display that similarly to the models, the transfer between prediction methods is also low. Consequently, the prompt format sensitivity issue creates a burden on authors of new ICL modifications; they must tune templates for every prediction method they want to combine with their own approach.

heading: 5.4 Transfer Between Demonstration Selection Methods
text: Having found that the best-performing templates are specific both to the model and the prediction method, we now aim to find whether the best formats would be the same for different demonstration sets in the same setup. Similarly to previous experiments, we calculate IoU for 10 templates that yield the highest scores for each method.
Results in Figure 5 illustrate that simply adding demonstrations, even if they were obtained with the same method, can significantly alter the ranking of the best templates. This justifies the necessity to evaluate example selection methods on a range of templates to avoid misinterpretation of the results.

heading: 5.5 Discussion
text: Based on the above findings, we conclude that the results of evaluation of various ICL improvements without consideration of template sensitivity issue are hardly reliable for several reasons. First, as the best templates do not transfer between models even within the same family, scoring a method across several models using the same format will inevitably lead to underestimation of the method for all models except the one for which the format was tuned. Next, as there is little evidence of transfer between setups, the format selection procedure needs to be precisely described and applied in all evaluated settings for a fair comparison.
Ra nd
om -2
Ra nd
om -4 IT M2 IT M4 z-I CL -2 z-I CL -4
Random-2
Random-4
ITM-2
ITM-4
z-ICL-2
z-ICL-4 0.2
0.4
0.6
0.8
1.0
Figure 5: IoU of 10 best templates for example selection methods on the AG News dataset. METHOD-N indicates that METHOD was used to select N examples.
In summary, we find that there are no universally well-performing prompt formats. Therefore, the results of in-context learning evaluation can be reliable only if they are aggregated over several templates or if each setting is evaluated in its best-performing template. The former approach requires accounting for the variance of the scores and makes comparison less apparent, while the latter can be computationally expensive.

heading: 6 Template Ensembles
text: To reduce the variance in performance caused by the template choice, we propose to ensemble model predictions across multiple templates. This approach is widely used in machine learning (Ho, 1995; Lakshminarayanan et al., 2017) for improving the predictive performance of the model, as well as its robustness, and can be viewed as a form of test-time augmentation (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015). Prior work on prompt ensembling has shown significant gains
by training a boosting algorithm on model outputs (Hou et al., 2023); by contrast, our method needs only the pretrained model predictions without additional training.
Formally, our method computes label probabilities across predictions for each of N templates, where N is the ensemble size, and outputs the label with the highest average probability. In early experiments, we tried selecting the most common label among the predictions; however, we found this voting strategy to perform poorly on tasks with a large number of classes. It is also important to note that ensembling N predictions involves running the model N times more compared to singleformat evaluation, which makes this approach more computationally expensive. We view template ensembles as a baseline solution for the problem of prompt format sensitivity and leave the exploration of more efficient methods to future work.
We begin with determining the minimal ensemble size that consistently reduces variance while increasing the average performance. We observe that for the majority of models and prediction methods, an ensemble achieves the best accuracy when its size reaches 4 or 5 (see an example in Figure 6), with further expansion being less effective. We also found that smaller ensembles may demonstrate unstable behavior, with the possibility of a drop in
performance if a suboptimal template is sampled. Therefore, we report results for ensembles of size 5 and average the results over 5 random seeds.
Next, we evaluate the performance gains of Template Ensembles for different prediction methods. Our findings in Tables 5 and 20 and Appendices K and L indicate that ensembles increase the accuracy for all evaluated models and prediction methods. Most importantly, they also significantly reduce the variance caused by the template choice for most setups. Therefore, we conclude that template ensembling allows to preserve the increase in accuracy provided by ICL modifications while mitigating the template sensitivity issue.

heading: 7 Conclusion
text: In this work, we study the inconsistencies in the evaluation of in-context learning advancements introduced by the template sensitivity of large language models. Specifically, we find that ICL improvements exhibit high variation across template formats and that it is not possible to reuse the same template across different modifications. This aspect is often overlooked in prior work, despite the fact that the impact of template selection on prediction accuracy may be comparable with the choice of demonstrations or prompting methods.
While we propose Template Ensembles as an initial solution to this problem, the general sensitivity of language models to minor prompt variations is yet to be addressed. Consequently, we believe that the research community should take this problem into account when developing new models, evaluation benchmarks, or in-context learning methods.
Limitations
Due to limited computational resources and the high cost for evaluation on a large range of models, we only focus on four classification datasets. Moreover, we only compare two example selection methods to a random baseline, potentially overlooking other effective approaches.
Additionally, the space of templates could be expanded for more comprehensive experimentation. For example, we did not explore label mapping, including random labels, which is an important aspect of the template.
We would like to notice that our study focuses on a template selection impact on a performance and a degree of template transfer between different setups but not on templates themselves. Future
work should further analyze not only which templates lead to a change in performance but also on why they affect it.

heading: A Templates from Prior Work
text: Tables 6 to 9 provide a comparison of all the templates used in the works presenting all methods we evaluate. Noticeably, prompt formats (and the choice of label words for some formats and datasets) used in works proposing investigated methods have no intersection. This is also concerning, since the original papers proposing these methods refer to each other. For instance, CHANNEL prompting outperforming CALIBRATION in Min et al. (2022b) might be explained by selecting a more favorable set of templates for the method proposed in the paper rather than by the advantages of the method itself.

heading: B Model Selection
text: Our initial evaluation pool consisted of 23 models. We evaluated each of them in 0-shot and 2- shot settings with three prediction methods on four datasets, resulting in 12 runs. For each run in both 0-shot and 2-shot setups, we compare the model performance averaged over 10 random templates.
Based on the results presented in Table 10, we restricted the final pool of models for evaluation to those that have a consistent increase in performance in the 2-shot setting, in other words, to those demonstrating a performance boost from ICL. More specifically, we kept the models that had 8 or more wins in 2-shot evaluation against 0-shot.

heading: C Full Baseline Results
text: Table 11 shows the results of evaluation of all 19 models in the default setting with a varying number of few-shot examples. These results illustrate that the template sensitivity issue is present in all models regardless of their size, and is not efficiently mitigated with the increase in the number of demonstrations.

heading: D Template Parts Analysis
text: In addition to studying prompt format sensitivity in general, we analyze how each part of a template impacts model performance. For instance, it could be possible that the inclusion of a certain verbalizer in a template consistently leads to a decline in accuracy, irrespective of the other components.
To find that out, we decompose all templates into their parts and measure the distribution of scores for different variations of each component separately. The results presented in Figure 7 illustrate
that even for state-of-the-art models, such as Llama 2 70B and Falcon 40B, many components exhibit high variance; also, the variance differs between two models. In other words, even if a certain template yields good performance and low variance for a given setup, it is not guaranteed to work consistently well in other setups, and changing a single component could have detrimental effects.
Along with the non-transferability of whole templates, we notice that individual components also do not transfer both between models and prediction methods. For instance, while “It was {}” ranks highest among output verbalizers for Llama 2 70B with the DIRECT prediction method, it is one of the worst for Falcon 40B.
Moreover, while a combination of best verbalizers is often a well-performing template, it is not necessarily the best one; the same is applicable for “bad” verbalizers too. For example, “input: {}\n sentiment: {}\n\n” is the best template for Falcon 40B with the DIRECT method, even though “sentiment: {}” is one of the “worst” output verbalizers for that model.
In summary, there is a complex interaction between the components of a template and their influence on model performance. We hypothesize that the transfer of both whole prompt templates and their parts is limited and requires further analysis.

heading: E Prediction Methods
text: We provide the results of advanced prediction methods evaluation for all models in 0-shot and 2-shot setting with random demonstrations in Table 10. We conclude from this comparison that neither of the advanced prediction strategies do not decrease prompt format sensitivity consistently across models and datasets. Moreover, when accounting for the spread in accuracy scores caused by this issue, the advantages of these methods over DIRECT become less apparent.

heading: F Reproduction of Results for Advanced Selection Methods
text: To evaluate the sensitivity of example selection methods to the template choice, we compare how the results reported in original works on these methods change when evaluated on a set of random templates instead of a predefined single one. For an accurate reproduction of original setups, we evaluate both Z-ICL and ITM using CHANNEL prompting with corresponding templates from Tables 6 and 7.
.
In Tables 12 and 13, we present the scores for a single template used in original implementations in the column “Reproduced”, aggregating results over 5 example selection seeds. The “Random” column shows average scores for fixed demonstrations on a set of 10 random templates.
We observe that the results obtained in our code differ from the ones reported in the papers presenting both methods (the “Paper” column). The cause of this discrepancy is presumably the difference in tokenization during preprocessing of the datasets. Both methods use the same codebase with tokenization specific to GPT tokenizers, which results in a significant drop in quality for OPT and LLaMA models. By contrast, our tokenization approach is more general and preserves the ICL performance of these models.
From these results, we conclude that both methods are not robust to the template choice, as the mean performance decreases for multiple models while the standard deviation across seeds increases. Therefore, the gains from advanced example selection methods are caused to a certain degree by the choice of a proper prompt format rather than the retrieved demonstrations.

heading: G Example Selection Methods
text: Full results of evaluation of demonstration selection techniques in 4-shot learning using the DIRECT prediction method are presented in Table 14. The results highlight that advanced example selection techniques often perform comparably to the random choice baseline when evaluated on multiple templates. One might argue that the prompt format choice is inseparable from the method itself and thus such a comparison is invalid. However, since the best-performing formats do not transfer between models or demonstration sets of different sizes selected with the same method, a proper evaluation would require finding the best template for each setup. This procedure both is computationally expensive and difficult to accomplish, as authors of example selection methods frequently omit the description of their format selection algorithm.

heading: H Accuracy as a Function of Template Rank
text: We plot the dependence of accuracy on the rank of the template in Figure 8. The results are aggregated across 19 models. Each model was evaluated on 30 random templates with the DIRECT prediction method and the same set of 2 randomly selected demonstrations. We observe that for SST-2 and AGNews datasets, the mean quality of the tenth-best template is within 0.9 of the best template score, which we consider a successful transfer. Despite the more rapid decay for DBPedia and TREC, taking variation across models into account, we still count first 10 formats as performing on par with the best one.

heading: I Transfer Evaluation with Spearman Rank Correlation
text: One of the possible means to evaluate template transfer is to calculate the Spearman rank correlation between scores of all templates. As can be
seen from Figure 9, this method yields higher correlations than IoU over 10 best formats, but the capacity for transfer is still far from perfect (for example, for SST-2 and TREC datasets).
J IoU Transfer For All Datasets
Similarly to Figure 4, we provide Intersection-overUnion of 10 best prompt formats for all 19 models and all datasets explored in our work in Figure 10. These heatmaps illustrate that the transfer of bestperforming templates between models is remarkably low for all datasets.

heading: K Additional Results For Template Ensembles
text: Tables 15 and 16 show the results of Template Ensembles evaluation on a broader set of models and datasets. For most setups, ensemble of size 5 exhibit better performance than a single template.

heading: L Evaluation of Instruct Models
text: We validate that our findings hold true even for the latest instruction-tuned models, such as Llama 3 8B Instruct and Mistral v0.3 7B Instruct. First, as we show in Table 17, in the baseline setting, an increase in number of demonstrations generally leads to a better performance of instruct models but does not significantly decrease variance of their final scores, similarly to what we observe in base models.
Second, Table 18 demonstrates that, after adjusting to template robustness, the default DIRECT prediction method performs on par with more advanced methods, e.g. CHANNEL and CALIBRATION, while sometimes having a less variance of the model’s scores.
Finally, we observe that the examples retrieved by Z-ICL method turn out to be consistently worse than two other methods. This is also in line with our observation for base models. However, in
contrast to our main results, where there was no evident winner between two other methods, the demonstrations selected with ITM method turn out to be slightly more robust to template choice for instruction-tuned models, as we show in Table 19.
L.1 Ensemble Results
Table 20 shows that applying Template Ensembles method to instruct models results in improved mean
classification accuracy with significantly reduced variance across different templates in all configurations that we test.
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
SST-2 GPT-J
GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
DBPedia 14
G P T -J
G P T -N
eo X
O P T
6. 7B O P T 30 B O P T 66 B
B LO
O M
1. 7B
B LO
O M
3B
B LO
O M
7. 1B
P yt
hi a
6. 9B
P yt
hi a
12 B LL aM A 7B
LL aM
A 13
B
LL aM
A 30
B
LL aM
A 65
B
Fa lc on
1B
Fa lc on
7B Fa lc on 40 B
Ll am
a 2
13 B
Ll am
a 2
70 B
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
TREC
G P T -J
G P T -N
eo X
O P T
6. 7B O P T 30 B O P T 66 B
B LO
O M
1. 7B
B LO
O M
3B
B LO
O M
7. 1B
P yt
hi a
6. 9B
P yt
hi a
12 B LL aM A 7B
LL aM
A 13
B
LL aM
A 30
B
LL aM
A 65
B
Fa lc on
1B
Fa lc on
7B Fa lc on 40 B
Ll am
a 2
13 B
Ll am
a 2
70 B
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
AG News
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
−0.4
−0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 9: Spearman rank correlation over 30 templates for models evaluated in the DIRECT-RANDOM-2-shot setting.
Model Direct Channel Calibration
Single Ensemble Single Ensemble Single Ensemble
LLaMA 7B 0.290.10 0.390.04 0.380.07 0.480.10 0.380.11 0.530.05 LLaMA 13B 0.350.09 0.430.09 0.360.08 0.460.03 0.420.09 0.550.01 LLaMA 30B 0.340.11 0.380.04 0.410.08 0.550.02 0.430.13 0.530.03 LLaMA 65B 0.380.08 0.480.07 0.380.09 0.560.01 0.450.11 0.55 0.04 Llama 2 13B 0.320.09 0.450.11 0.350.11 0.450.02 0.460.11 0.580.05 Llama 2 70B 0.410.07 0.420.09 0.410.10 0.460.03 0.510.06 0.570.02 Falcon 1B 0.260.09 0.330.06 0.330.06 0.430.06 0.330.06 0.440.01 Falcon 7B 0.320.09 0.360.05 0.330.08 0.490.12 0.370.11 0.480.06 Falcon 40B 0.360.07 0.390.06 0.410.06 0.520.03 0.450.06 0.530.05
Table 16: Comparison of 2-shot learning performance on the TREC dataset using ensembles of 5 templates and a single template. Results are averaged over 5 random seeds.
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
SST-2 GPT-J
GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
DBPedia 14
G P T -J
G P T -N
eo X
O P T
6. 7B O P T 30 B O P T 66 B
B LO
O M
1. 7B
B LO
O M
3B
B LO
O M
7. 1B
P yt
hi a
6. 9B
P yt
hi a
12 B LL aM A 7B
LL aM
A 13
B
LL aM
A 30
B
LL aM
A 65
B
Fa lc on
1B
Fa lc on
7B Fa lc on 40 B
Ll am
a 2
13 B
Ll am
a 2
70 B
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
TREC
G P T -J
G P T -N
eo X
O P T
6. 7B O P T 30 B O P T 66 B
B LO
O M
1. 7B
B LO
O M
3B
B LO
O M
7. 1B
P yt
hi a
6. 9B
P yt
hi a
12 B LL aM A 7B
LL aM
A 13
B
LL aM
A 30
B
LL aM
A 65
B
Fa lc on
1B
Fa lc on
7B Fa lc on 40 B
Ll am
a 2
13 B
Ll am
a 2
70 B
GPT-J GPT-NeoX
OPT 6.7B OPT 30B OPT 66B BLOOM 1.7B
BLOOM 3B BLOOM 7.1B
Pythia 6.9B Pythia 12B LLaMA 7B LLaMA 13B LLaMA 30B LLaMA 65B
Falcon 1B Falcon 7B Falcon 40B Llama 2 13B Llama 2 70B
AG News
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
.
.

heading: Future Work
text: Future
work should further analyze not only which templates lead to a change in performance but also on why they affect it