Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Authors: Melanie Sclar
Yejin Choi
Yulia Tsvetkov
Alane Suhr
Venue: ICLR 2024 poster (2024)
Keywords: large language models
sensitivity analysis
prompt engineering
evaluation
prompting
robustness
in-context learning
spurious features
Primary area: generative models

Abstract: As large language models (LLMs) are adopted as a fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format. We also show that format performance only weakly correlates between models, which puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FormatSpread, an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights. Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.

Published as a conference paper at ICLR 2024
QUANTIFYING LANGUAGE MODELS' SENSITIVITY TO SPURIOUS FEATURES IN PROMPT DESIGN or: How I learned to start worrying about prompt formatting
Melanie Sclar'
Yejin Choi Paul G. Allen School of Computer Science & Engineering, University of Washington
Yulia Tsvetkov!
Alane Suhr
2 Allen Institute for Artificial Intelligence msc asclarêcs wash hngtonned
UUniversity of California, Berkeley
ABSTRACT
As large language models (LLMs) are adopted as fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of sreaning-preservivg several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B Sensitivity remains even when increasing model size, the number of few-shot examples, or perform- ing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-basee based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standant porting performance on single format. We also show that format performance only weakly correlates between models, which puts into question the methodolog- ical validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FORMATSPREAD, an algorithm that rapidly evaluates sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.
design choices: prompt formatting. We find that
practice of re-
INTRODUCTION
As the capabilities of LLMs have rapidly improved, their sensitivity input prompt features has been used to optimize performance via prompt engineering (White et al. 2023). However, there has been little work characterizing this sensitivity, especially to seemingly innocuous feature choices that preserve prompt meaning and intent. In this work, we analyze the sensitivity of widely used, open-source LLMs to a class features that should not influence a prompt's interpretation: formatting choices. We find that pre- treitri LLMs sensitive to these choices in unpredictable ways, with accuracy varying in up to 76 points for LLaMA-2-1 1-B between equivalent formats, and ~ 10 accuracy points on average across 50+ tasks and several models. We also show that this variance is not eliminated by adding few-shot examples, increasing model size, or instruction tuning.
Designing prompt templates is critical part of effectively using pre-trained language model. This design process includes making choices about wording, choosing few-shot examples for in-context learning, and making decisions about seemingly trivial features like formatting. This process, and often even the resulting templates, is rarely reported or discussed in research papers, under the as- sumption that performance variance across these choices is insigmificant compared to variance across data points models. However, some anecdotal evidence points to formatting choices actually hav- ing a significant influence on model behavior (Aghajanyan, 2023). some cases, researchers report a limited number of manually generated formats to show that scaling trends hold despite perfor-
"FORMATSPREAD'S code be found at http:////////tttttt...o///aninaa

---

Published as a conference paper at ICLR 2024
Modified separator
Original formatting
Modified spacing between fields
Passage: sergge:stte <text>> snswer:<text>
Passage: <text>
passage: <text> Answer: <text>
Answer: <text>
Modified separator and spacing
Modified casing
PASSAGE <text> ANSWER <text>
Passage <text> Answer <text>
PASSAGE: <text>
ANSWER: <text>
Task Accuracy
0.036
Performance Spread Among Plausible Formats
0.804
Figure 1: Slight modifications in prompt format templating may lead to significantly different model performance for a given task. Each <text> represents different variable-length placeholder to be replaced with actual data samples. Example shown corresponds 1-shot LLaMA-2-7B perfor- mances for task280 from SuperNaturallnnsruutinn (Nadeem et al.] 2021) requires the model to, given a short passage, classify it into one of four types of stereotype anti-stereotype (gender, profession, race, and religion).
(Wang et al. 2022). This StereaSet-inspieed task
mance being significantly diffferent (Schick et al. 2021). The assumption that formatting does not influence overall model performance may become problematic when improvements over existing approaches attributed to the amount and source of training data, number of parameters, model architecture, without also accounting for changes in prompt format. Ignoring variance across for- mats may also negatively affect user experience, e.g. if users inadvertently choose formats the LLM does not perform well on.
Our proposed tool, FORMATSPREAD, enables a systematic analysis of these variances across wide set of semantically equivalent prompt formats within a user-specified computational budget. We find that choices in formatting few-shot examples during in-context learning introduce spurious biases that may lead to significantly different conclusions in model performance. The sensitivity to format- ting choices that we discover across widely-used, open-soure, models suggests that future research would benefit from reporting performance spread over sufficient sample of plausible formats, instead of simply reporting the formatting used and its performance, as is currently standard. More- over, we argue that this reporting is crucial when comparing the performance of different models, as we show the influence of formatting choices only weakly correlates between models, thus making and fixing a formatting choice could introduce significant confounding factor.
Fully exploring the space of prompt formats is intractable, as computation costs scale linearly with the number of formats considered. FORMATSPREAD efficiently explores the space of prompt for- mats under a user-specified computational budget using Bayesian optimization. FORMATSPREAA does not require access to the model weights, allowing its use API-gated models: we find spread up to 56 accuracy points with a median spread of 6.4 accuracy points with GPT3.5 across 320 for- mats and 53 tasks at a cost of under 1OUSD on average per task. Beyond facilitating evaluation, we also propose a suite of analyses to further characterize model sensitivity to formatting. Among other results, we show that the separability of continuous prompt embeddings correlates with the spread observed in task performance.
2 OVERVIEW
We evaluate LLM performance over the space of prompt formats that may plausibly be chosen by a non-adversarial user when designing a prompt for target task, where the space of formats is defined by a grammar 1I.. Our grammar's definition naturally induces a definition of semantic equivalence among formats. We quantify model sensitivity in terms of performance range in target task across the space of equivalent prompt formats to the original choice 4...). We cast the prob- lem of searching across this space as a bandit problem, and propose FORMATSPREAT ($3), which consists of a grammar ($3.1) and a procedure to estimate the minimum and maximum performance across a set of semantically equivalent formats given a pre-defined metric ( (3.2).. FORMATSPREAD uses Bayesian optimization to identify the expected performance range with low additional compu- tational cost (54.5) all without requiring access to model weights, which enables use API-gated
2

---

Published as a conference paper at ICLR 2024
LLMs. Furthermore, perform in-depth analysis of this observed sensitivity, including by quan- tifying the contribution of individual feature choices to the final performance ($4.3) and measuring the identifiability of a format based solely model's internal, continuous representation of any prompt via correlation with model performance $4.44
3
MEASURING SENSITIVITY WITH FORMATSPREAD
3.1 GRAMMAR OF PLAUSIBLE PROMPT FORMATS
We construct grammar that defines both the space of plausible prompt formats and semantic equiv- alence between formats. The grammar is manually constructed, as opposed to automatically induced from data, to guarantee a higher level of precision when defining the of equivalent formats. Our grammar is directly tested by verifying that it can generate the formatting associated with 100+ huper--aturalInstructooo tasks (Wang al. 2022
Our grammar consists of fields that are composed to create a prompt format. For example, the format "Passage: <text> II Answer: <text>', has basic fields 'Passage: <text>', and 'Answer: <text>', denoted
and 02 Each basic field consists of a descriptor (e.g. 1) and a text placeholder to replace with each data point. We define basic fields Bi (d, f) ;= f(d)s<text> using Backus-Naur notation, where d is descriptor string, 8 E] a separator, and f Fensing a function that alters d while preserving meaning. Thus, in example, d1 B1 (Passagee
'Passage'), a separator (e.g.
idd and 02 B1 BAnsweer, ' ',id), with id the identity function. We define joining several fields as B2 (X X,,c) == XicX2e...chl eXn, with cEC being space. Our example's prompt format may be written B²²) (a1, 11 1).
The grammar also supports enumeration, which defined as joining several basic fields, each rep- resenting a different list item. For example, the enumeration "Option (A) <text>, Option (B) : <text>, Option (C) <text>' may be written as ( 02, 03. 1I "), where a, = Bi (ej, ': ' idd.) In our example, ei represents "Option (A)', and may in turn be written as the concatenation & := d82 fiem with d = 'Option', 82 = (single space), and fuem(1) = (A) ' Each fuem transforms item using number format (e.g. letters or Roman numerals, denoted as Fuemz) and an item wrapper (e.g. (A) or [A], denoted as Fnemt).
In summary, we define valid prompt formats as those accepted by the following grammar:
Ba() = <text>
(d,,, s) = f(d)s with S f Feasing
B (d, 8, f) = f(d)sctext> with S1, f E F
X., c) = cX, with X; E (Bo, Bb, Bi, Bz, B3} Vi (d. In 81, S fe) = BLxh) (Bi(e1, where e, == f2(d) 82 fi(ie). je E No Vi. St. S2, fi Fuem. f2 E Fensin
B
(X T
(e f2).
Our grammar defines valid formats as finite compositions of Bo. By, B1, B2, B3. The sets S1, S2. C, Feasing. Fitem (two sets of separators, spaces, casing functions, and itemizing functions respectively) are pre-defined by the user. Throughout this work, we instantiate all sets with values typically ob- served in human -written prompt formats. We intentionally only modify the casing of descriptors (via Feusing) to guarantee semantic equivalence; one may also define a set of functions that paraphrases the descriptor, e.g., via synonym replacement. Appendix A.2 contains the full list values we use for the constant sets, as well as visualization of a prompt template generated from the grammar.
Prompt Format Equivalence. Two prompt formats P1, P2 are equivalent they represent the same rule application B,, the descriptors (if any) are the same, and the sub-elements (if any) are equivalent. Appendix(A.] A contains the formal definition of equivalence. The grammar's 's strict definition allows assume that sets of equivalent formats share equivalent meanings. When measuring sensitivity we explore only the space of formats equivalent to a task's 's original format.
3.2
Contextual Restrictions. We define restrictions to the combinations of spaces and separators to further ensure naturalness. For example, if B, (X) XX,.c) where e does not contain a newline, then each X, separators and any subcomponents" separators should not contain newline. This
3

---

Published as a conference paper at ICLR 2024
avoids unnatural formats like Input: <text> Cutput: :\, <text>, We also allow for adding conditions that force constants (separators, spaces, etc.) in different applications of B, to be equal. When measuring sensitivity to format perturbations, if two separators or spaces are equal in an original format, they are forced to jointly change to be considered equivalent. Appendix A.3 contaiis all contextual restrictions.
Final Prompt Construction. Given a valid format p accepted by the grammar, the fi- nal prompt is constructed by concatenating with space с an instruction string inst, n few- shot data points Di,..., Dn exemplifying the task, and a data point Dz+1 to be solved. All few-shot examples D, are formatted using p. Thus, the final prompt template is: inst e ((D) ((2))
C p(Dx) с p((n+1) Since Dn+1's ''s output will be generated by the model, an empty string is added in place of the answer in the last field in the template. Prompt con- struction will modify inst to match specific choices encoded in p: concretely, if p enumerates valid multiple-choice options as characters 2 Tn. we ensure inst refers to these choices as Tp....p.
3.2 MEASURING SENSITIVITY
We measure how plausible choices in prompt formatting influence quantifiable metrics of generated outputs. Given a set of plausible formats (pi...., 5. a dataset D, and a scalar metric m, let the performance interval be [min, m(p., D), max, m(p,, D)J. We define the performance spread or simply spread as max; m(p;, D) min; mpp;, D). Higher spread indicates more sensitivity mmmanticall--eqivalle
to variance within the space of plausible,
formats. While our method is agnostic to the scalar metric m used, and one could consider a number of metrics including text length, formality, or toxicity, throughout this work we focus our analysis on estimated task accuracy acc. Due to ease in automatic evaluation, here we evaluate on classification tasks.
Our goal is to compute spread for given model and task. A comprehensive approach would be to fully evaluate each plausible format on the entire evaluation dataset D. This increases the cost of reporting model's performance linearly with n, which becomes computationally infeasible for large values of n. Following prior gradient- gradient-free prompt engineering work (Zhou et al. 20233 Pryzant al. 2023). we model our problem as multi-arm bandit. Given a random sample of formats (arms) Pr.....PM Pr. for task, an arm p; 's hidden value is the actual performance m(p;, D) when evaluated on the full dataset D, and the reward for pulling the arm is an estimate m(p,, D) where D D D, [D] = B (mini-bated size) and no element of D has yet been evaluated with p;.
We assume a budget of E total data point evaluations. We first search for the highest performing for- mat with budget E/2, and then for the lowest performing format with budget E/2. Evaluations done for the first exploration are readily available for the second exploration, which yields a more infor- mative prior for many formats. We consider two well-known regret minimization bandit algorithms: Thompson sampling (used in FORMATSPREAD. and Upper Confidence Bound (UCB).
Thompson Sampling. This simple, high-performing Bayesian inference heuristic randomly draws each arm according to its probability of being optimal (Chapelle & Li 2011). Each m(p;, D) is modeled as random variable, and since with our target metric each data point evaluation Bemoulli trial, it is natural to model mp; D) as a Beta distribution. In each round, Thompson sampling draws from each m(p:, D) and chooses the best arm (Algorithm 1). It then updates i according to the number of observed successes r, and the corresponding B -r failures, within D.
Algorithm Thompson Sampling for Bernoulli Bandits
5(D) <- (()) 0 (success counters and total times armed was drawn counter)
for t 1.... 1....EBBdl do
for i I...., do
Take 0550) from Beta(a; S(") .B: (N/50 (00()-5('9 s("))
Draw s ((41)
arg max, g0() (or arg min in minimization problems) and observe reward
(!()) +r, N N((41)) N:(0
Thompson sampling allows for setting informative priors (a,, 3.) based domain knowledge to accelerate runtime. (ppendix/A.4)dettaal the exact priors we use. To our knowledge, we are the first to consider Bayesian sampling method for prompt optimization.
4

---

Published as a conference paper at ICLR 2024
Upper Confidence Bound (UCB) Sampling. UCB (Lai et al. 1985) computes an upper confidence bound to each arm's 's performance, derived from Chernoff' 's bound. The key difference with Thomp- son sampling is in how 0 is defined. In UCB's s frequentist approach, 0!") is assigned the estimated accuracy plus the upper confidence bound: 008h) + S./N, + ev (llg(())///) We use c = 2 following Pryzant et al. (2023). who find UCB with 2 to be most effective for prompt optimization.
Naive Sampling. Each prompt format is evaluated on E/n points (with appropriate rounding).
4
CHARACTERIZING PROMPT FORMAT VARIANCE WITH FORMATSPREAD
4.1 EXPERIMENTAL SETUP
Data. We use a subset of 53 tasks from Super-Naturallnstructio (Wang et al., 2022) with diverse human-written formats and instructions, comprising 19 multiple-choice tasks and 34 classification tasks with {2, 3, 4} basic fields. Appendix B.1 details the exact task selection procedure. To con- struct the final prompt template, we concatenate each task's 'ss instruction and formatted few-shot examples using \m)n as spacing. While selection and ordering of few-shot examples is a compo- nent of prompt design influencing features of model output (Lu et al. 2022), our work focuses on prompt formatting. To remove this confounder, we fix the exact choice and ordering of examples for each task and for a given number of shots n. Few-shot examples for each task are chosen randomly within each dataset and are not used for evaluation. We evaluate task data samples on an arbitrary order fixed across settings. Datasets are assumed to be of size 1,000 for fair evaluation across tasks.
Models. We evaluate LLaMA-2- 10AMAA--(7BBBI)) (3B. } (Touvron et al. 2023), Falcon-7B and Falcon- -BB Instruct (Almazrouei et al. 2023). GPT-3.5-Turbe (Schulman al. 2022). all autoregressive LMs.
Task Evaluation Metries. We use two popular measures for computing accuracy: exact prefix matching and probability ranking. exact prefix matching, we check if the output's prefix matches the expected answer after normalization (casing, spacing, newlines). Ranking accuracy computes the rate that the expected answer is the highest-ranked valid option (in multiple choice and classification tasks) according to the model's output distribution. Results are reported using ranking accuracy unless specified otherwise. Appendix B.2 shows additional analysis of exact prefix matching, with spreads even higher than those shown in Section 4.2 and including how formatting choice affeets task degeneration (i.c., not answering any valid option).
4.2 PROMPT FORMATS HAVE A LARGE PERFORMANCE SPREAD, NOT ELIMINATED BY INCREASING FEW- SHOT EXAMPLES OR MODEL SIZE, NOR WITH INSTRUCTION TUNING
For each evaluation task randomly sample 10 plausible prompt formats and use PORMATSPREAD to compute performance spread for each modeling and n--hoot choice (Figure 3 We find signifi-
cant performance spread across tasks, with median spread of 7.5 accuracy points across choices in the model and the number of few-shot examples. 20% of tasks consistently result in a spread of least 15 accuracy points for all LLaMA-2 settings, and at least 9 points for all Falcon settings. We observe several tasks with performance spread over 70 accuracy points. Because this analysis uses only 10 randomly sampled formats, it represents a lower bound of the true spreads for each task. Furthermore, there exists sigmificant performance spread regardless of increased model size (Figure 2a and Figure [1 for 1-ama-2-70B), instruction tuning (Figure 2b), or number of few-shot examples (Figure 2c Figure 2a and 2b plot 1- and 5-shot jointly). Appendix [ demonstrates sim- ilar results on a selection of nonscaasiiiication tasks, and expands the spread discussion to plotting the entire accuracy distribution, along with a dispersion metric.
Comparison trends between models are often reversed just by choosing different formats. As- suming model M is better than M' by at least d accuracy using prompt p. we compute how often M' achieves at least d higher accuracy than M under a different format p'. Figure 4 shows these trends are often reversed: LLaMA-2-133 13B and -70B reverse trend by least d 0.02 with prob- ability O. 0.141; LLaMA-2-7B and Falcon-7B reverse trend by at least d = 0.02 with probability 0. 1.. Strikingly, often both experiments (first using p. and then p') were statistically significant (p-value < 0.05) on 1000 s 76% and 47% respectively for the two model comparisons
We use one-sided McNemar tests, also known paired x2 tests, since we evaluate models on the same set of samples. We test the significance of M being better than M' under p. and M heing worse than M' under p'.
5

---

Published as a conference paper at ICLR 2024
Uame-2-133 spread
0.8
r-0.83 p-3.37 10-36
felcon- 7b-inst spread
0.3
mo.43
mo.02
0.6
pm311 x 10
mahote spread
0.5
p=1.59
0.2-
0.-
0.4-
0.1
0.0
8.000
0.10
0.0
0.2
0.8
Liama-2-7b spread
falcon-7b spread
nshotwil spread
(a) Llama-2-7B vs. 13B
(b) Falcon-7B vs. 7B-Intrucc
(c) vs. 5-shot (same task, model)
Figure 2: Spread comparison between evaluating the same task under different models or n-shots.
u mi
Uamo 7.
0.4
(M, AF)
(M,
(Limm-2-77,, L/ama-2-70b)
(Llama-2-75, Llama-2-133)
nshot-1
Protabiity that
accig'al accioll--occoo scciolel
(Uams 2 130, Limm 700)
Uama
nchot=]
0.3
LLama-2-70, ralcon-7b)
Ua to
(M, A')
(Talcon 7b. falcon 7b inst)
0.2
Utme--7te
nchot=]
44
rshot-)
falcon 76
0.0
falcon 7b
felcon-Tb-ee
nchag=5
Threshold sccuracy aifference (c) llog)
feteon-rt-ine
nshot=]
Figure 4: Probability that model M performs worse than M' by at least d when using format p', given that M performed better than M' by at least d using format p. 53 tasks, 1- and 5-shot.
nchot-5
0.7
Parformance
prompt formats
Figure 3: Spread across models and n-shots.
mentioned above. We find that formats yielding high performance for model M may not yield high performance for M', implying that formats may not be inherently good or bad (Appendix B.2)
4.3 How DO INDIVIDUAL FEATURES CONTRIBUTE TO PERFORMANCE?
We analyze how choices in particular constants (i.e. S₁ S2, C, Feasing Firm) independently influence task performance across different formats. Figure 5 shows the distribution of accuracy for 500 sam- pled prompts conditioned on the choice of S₁ (the separator between a descriptor and the text place- holder) for one task in Super-Nature aapper-aauurlllnstuutttonn When comparing the individual influence of two feature choices, we measure both weak and strong notions of dissimilarity between distributions of accuracy across prompts conditioned on a chosen feature. We say two constant choices yield weakly different accuracy distributions if the values between the first quartile (Q1) and third quartile (Qs) do not intersect. This is equivalent to the boxes in a boxplot not overlapping. We say two constant choices yield strongly different accuracy distributions if the ranges [2.5Q] 1.5Q3, 2.5Q3 1.5Q1] do not overlap (adjusted to end in a data point). This is equivalent to two boxplots with their whiskers not overlapping. In F An)t and , (fourth and sixth) are only weakly different.
We compute accuracy for 500 random formats with 250 samples each on 31 tasks for 1-shot Llama- 2-7B. Table [ shows that choices in S2, Fitmmt Feasing do not independently predict performance differences (weakly or strongly): although these features can have a large performance variance and thus should be explored with FORMATSPREAD, they cannot be used to independently predict accuracy changes. Other constant sets have varying degrees of differences, with S (separators) and Faemz (number format changes in enumerations) having the most individual impact. All tasks with strong dissimilarities are shown in Appendix B.4
Small prompt variations often yield large performance differences. Table a selection of tasks where changing single constant on format (e.g., casing in task322) results in large accuracy differences. Figure shows that regardless of the scoring criterion used, significant ratio of these atomic changes associated with large accuracy changes. For example, 24% of atomic changes have an associated accuracy change of at least 5 points when using exact prefix matching as scoring criteria (11% when using probability ranking).
The space of prompt format accuracy is highly non- non-monotonic, which makes local search algorithms over the space less effective. Let (P1, (P,,P2,P5 Ps) be prompt format triple such that P++1 is obtained by making an atomic change to P.. We argue that if the prompt format space is smooth, we should often see triples' accuracy to be strictly monotonic over i. We choose 24 tasks (13 multiple choice,

---

Published as a conference paper at ICLR 2024
Table 1: Tasks where at least two constants yield different performance (weakly different their boxes in a boxplot do not overlap, strongly if boxes including whiskers do not overlap).
0.55
Accuracy
0.45
Median Spread (range [0, I])
Weak Diff.
Strong Diff.
0.40
100 Ink
%
0.144
29%
1%
S₁
0.132
43%
22%
Separator Used
S2
0,238
0%
0%
Fitem
0.176
0%
0%
Figure 5:
Example of accuracy variance for dif- ferent choices of constants in S for task 1283.
Fuemt 0.173
45%
10%
Feasing
0.188
3%
Table 2: Examples of atomic changes' impact on accuracy using probability ranking (prefix match- ing shown in Table 4 ( represents text field; p2 yields higher accuracy than p1 for all tasks.
Task ld
Prompt Format (p1)
Prompt Format (p2)
Acc p1
Acc p2
Diff.
t80
passages (Ain answer:
passage (r ansver O
0.043
0.826
0.783
task317
Passage:: ( Answer::()
Paszage:s () st 0
0.076
0.638
0.562
task190
Sentence [T]- (1entterceT11)- O
Sentence [A]- ()sentence
0,360
0.614
0.254
task904
isput:: 0 \e output:: O
:ngut:: O \r outputz:()
0.418
0.616
0.198
task320
target O \e( \panswez O
targot () (n(H \nansver O
0.361
0.476
0.115
task322
COMME:: O ANSWER: ()
comment: 0 answer:
0.614
0.714
0.100
task279
Fassaee O.
PASSAGE () AMMNER
0.372
0.441
0.069
1l non-multiple choice), sample 300 (p1, P2, Pa) triples for each, and the compute accuracy (using
exact prefix matching) of each p3 250 samples. 32.4 and 33.6% of triples multiple-choice and non-multiple-choic tasks respectively. Given that random shuffling within a triple will result in monotonicity 33.3% of the time, this suggests that local search mechanisms like simulated annealing may not be effective as they require locally smooth search space.
monotonic for
4.4 PROMPT FORMATS ARE IDENTIFIABLE TRANSFORMATION OF PROMPT EMBEDDINGS
Prompt format choices represent deterministic transformation of the input, even if its impact on the resulting performance is hard to predict. We represent prompt embeddings as the last hidden layer obtained when processing the whole input prompt (immediately before generating the first token). We demonstrate that format choice yields highly identifiable transformation over this embedding, which suggests that formats can be seen as transformations of the output probability distribution.
For each task, and for both {1, 5}-shot settings, we collect prompt embeddings from LLaMA-2-7B corresponding to 10 randomly sampled valid formats for 1000 evaluation examples. We train an XGBoost (Chen & Guestrin 2016) classifier that maps from the top n principal components prompt embedding to the prompt format 3 We find that although the original prompt embeddings are of size 4,09d"] using just the top 100 principal components can result in a classifier with 20.98 accuracy in format identification for all 31 tasks analyzed. Figure 7 shows the accuracy of format classification given a fixed number of principal components We find that classifier accuracy given just the top components correlates moderately with the spread of performance in the prompts they represent (0.424, 8.04 . 10 10-6, 0.555 for the 5-shot setting; using exact prefix matching).
4.5 FAST EXPLORATION OF THE PROMPT FORMATTING SPACE: FORMATSPREAD
In Section we demonstrate that even when sampling just 10 formats from the space of plausible formats, we still observe significant performance spread on many tasks. However, this is only a lower
'e train with 800 vectors from each of the 10 formats (8000 vectors) and evaluate the remaining 200. "Equivalent to the dimension of hidden representations for LLaMA-2-7B. FFgure in the Appendix visualizes examples of the top two principal components for prompt formats.

---

Published as a conference paper at ICLR 2024
1.00
atomic change
Al
0.75
Scoring Criterion
Probabilty Ranking
Cumulative Ratio
0.75
# principal components
Exact Prefix Matching
Analyyed
Probabillty
yields
0.50
10
0.25
0.25
20
50
0.00
0.1 Absolute Accuracy Diff. (d) (log scale)
0.01
1.0
0.4
0.6
0.8
1.0
Format Classification Accuracy (a)
Figure 6: Probability that an atomic change (e.g. changing space, separator) has given impact in accuracy for two scoring criteria. 53 tasks, 30 sampled atomic changes each.
Figure 7: Cumulative ratio of tasks that can be classified with at most accuracy using the top principal components of the last decoding layer of the prompt.
0.7
Spread Increase (a) when inceastngg sampling from formats
0.16-
Thompson Samp ling
UCB Sampling
Probabiity of stread
increat 23
k 110 to ky
g
0.14
Naive Sampling
kum20 Kgm00
0.12
0.4
k,-40 ky-80
0.10
ky=00 Ry=1000
kg-240
0.08
k,-240 ky-320
kg 320 to kym000
0.06
0.04
0.1
0.02
0.01
10000
20000
30000
40000
50000
Spread (log)
Maximum number of evaluations allowed
Figure 8: Probability of observing spread in- crease of at least when increasing sample size from kn to l2 formats. 31 tasks, 100 trials each.
Difference between the true sample spread and each algorithm-found spread with re- spect E (evaluation budget). 320 formats, B = 20, average of 5 trials over 31 tasks shown.
Figure 9:
bound of the spread a task may exhibit when increasing the number of formats: for example, about 17% of tasks are expected increase their spread by least accuracy points when increasing from 10 to 20 sampled formats. Figure 8 quantifies the expected increase in spread when increasing the number of formats by evaluating 500 formats on 250 samples each and computing expected gains.
Figure 9 compares the efficiency of Thompson sampling, UCB, and naive sampling for estimating budget E (Section 3.2 To ensure accurate reports, we compute and show the true spread of the highest- and powest-performing formats chosen by each method using all data. With a budget of 51, 1,200 evaluations, Thompson sampling results in a spread within accuracy point of the true spread, while naive sampling finds spread within 4 points, and UCB within 11.
spread with respect
Finally, we use FORMATŠPREAD to measure sensitivity several models where inference is expensive. With a budget of 40,000 evaluations and 320 prompt formats, we find that 1 1-shot LLaMA-2-70B-ras using 4-bit quantization (Dettmers et al. (mcan=0.221, std=0.200, using probability ranking across 53 tasks; 25% of tasks had spread of 0.292 or higher, with a maximum spread of 0.876), and GPT-3.5 yields a median spread of 0.064 (mean=0.110) 110, std=0.115. across 53 tasks using exact prefix matching given that do not have access to the full logits; 25% of tasks had a spread of 0.148 or higher, with a maximum spread of 0.562), showing sensitivity formatting is still present even on larger models. 5-shot LLaMA-2- 70B still shows high spreads, with 25% of tasks having a spread of 0.310 and maximum of 0.841. See spread visualization Figure 25 and a list of best and worst formats found in Table
a median spread of 0.11
RELATED WORK
The task automatically finding the best-performing prompt for a given task without changing model parameters has recently gained attention, given the constantly improving yet somewhat unpre- dictable performance of LLMs. Prior work has often focused on discovering optimal prompts with gradient-based methods, which effective, but often lead to disfluent unnatural prompts (Shin
8

---

Published as a conference paper at ICLR 2024
et al. 2020), which can be mitigated with Langevin dynamics based method (Sh et al. 2022
Another approach is to learn, optimize, and insert continuous representations of prompts and tasks
as input to models (Qin & Eisner 2021 Lester et al. 2021 Ding et al. 2022 Ilharco et al. 2023
These methods also require access to the LLM's parameters, thus cannot be applied to models behind an API. In contrast, FORMAT free work has focused on edit- enumeration over human-written prompts (Prasad et al. 2023),
does not assume access to any model internals. Prior gradient-
reinforcement learning (Deng et al., 2022), and by using LLMs themselves (Zhou et al.) 2023 Gao
et al. 2021). These works aim to achieve competitive task performance, even if the meaning of the prompt or instruction modified. To our knowledge, we are the first to focus specifically on prompt formatting variance, a quintessential example of semantic equivalence.
Jailbreaking refers to the behavior of intentionally manipulating prompts to elicit inappropriate or sensitive responses, or otherwise reveal parts of the prompt that were intentionally not revealed.
While the objective differs from our work, jailbreaking works (Wei et al.) 2023] Zou et al., 2023)
share the underlying technical question of finding the lowest-performing prompt. Our methods differ, since Wei et al. (2023) evaluate human- -ennaraeee attacks to guide adversarial prompt design, and Zou et al. (2023) uses gradient-based search methods simultaneously across multiple models.
Some existing work has explored the influence of certain prompt design choices on model perfor- mance, for example the prompt's language (Gonen et al. 2022), the ordering of few-shot exam- ples (Lu et al. 2022), and their patterns (Madaan et al. (022)). Other work has focused on providing textual interpretations of continuous prompt representations (Khashabi al.) 2022]. Beyond au- toregressive LLMs, existing work has focused on performance variance in masked language mod- els (Elazar et al. [022 Jiang et al., 2020), Our work follows efforts in other domains that explore the influence of spurious features on research evaluations, e.g., in deep reinforcement learning (Islam)
ct al. 2017 Henderson al. 2018 and statistical machine translation (Clark et 201T
DISCUSSION
We introduce FORMATSPREAD, an algorithm that estimates the performance spread across prompt formatting choices h We use FORMATSPREAD to evaluate the spread of several widely-used open- source LLMs for classification tasks in few-shot learning settings. We find that spread is large regardless of model choice, even when increasing model size, number of few-shots, instruction tuning. FORMATSPREAD is designed to efficiently search the space of plausible prompt formats under a user-specified computational budget. For example, with computational budget of exploring only 5% of the entire search space for task with 2,500 test examples and 320 plausible formats, we are able to estimate spread within 2 accuracy points of the true spread.
when using
We also characterize the space of prompt formats, finding that it is largely non- non-monotonic and that few atomic features can be predictors of performance alone, although the separability of format embeddings is highly correlated with observed performance spread. These findings informed the design of our search procedure, where local search methods are not advantageous.
Our findings suggest that performance spread caused by arbitrary prompt formatting choices may influence conclusions made about model performance, especially when comparing models on bench- mark tasks. Thus, we recommend that work evaluating LLMs with prompting- based methods would benefit from reporting a range of performance across plausible formats. However, we want to em- phasize that dingle-format evaluation may still be sufficient for many use cases. For example, for researchers or practitioners who build systems on top of LLMs, choosing a single prompt format that works sufficiently well for use in this larger system is valid methodological choice. However, we encourage future research to compute FORMATSPREAD when comparing their systems to out- of-the-box models, ensure fair baseline representation. Furthermore, FORMATSPREAD can be used to identify lower-bound performance of a model or system. For example, when using a model for socially impactful tasks, such stereotype classification in Figure D it is important report the range of accuracy a non-adversarial user might encounter. Likewise, it is crucial to consider robustness to spurious features when claiming that models possess general abilities, such as theory of mind; and beneficial to report when e.g. exploring model biases. We leave it to future research to develop regularization procedures either during training or with an already-traines model to make models robust to diverse formatting choices.
We thoroughly descrihe the limitations of method in Appendix
9

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
We thank Jillian Fisher, Sachin Kumar, Angela Zhou, and the Berkeley NLP group for valuable discussions. This work was conducted while A.S. was Young Investigator at A12. This material based upon work partly funded by the DARPA CMO under Contract No. HR001 120C0124, by DARPA MCS program through NIWC Pacific (N66001-19-2-4031, by NSF DMS-2134012, IIS- 2125201, IIS-2203097, by NSF CAREER Grant No. IIS2142739, and an Alfred P. Sloan Foundation Fellowship. Any opinions. findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily state or reflect those of the United States Government or any agency thereof.
REFERENCES
Armen Aghajanyan. Tweet: Susan found MMLU performance jump 6-10 points in the 40s by formatting multiple choice as (A) not A in MMLU (for internal model). All evaluation of LLM'S s are broken. Evaluating a task requires marginalizing across all prompts
that describe the task, not point istim of one. June 2023. URL
https://twittes www.coo////////// Aww.AAreeenaaaaaeeeeeeaaaaaaaaaa http://www..stttttssssseese httpa///.......aaeee..eooaoa///.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-an performance. 2023.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems, 24, 2011.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD "16, pp. 785-794, New York, NY, USA, 2016. ACM. ISBN 978- 778-1-4500--232--2 doi: 145// httt:://o/ooi acm org 10. 00w...11/////11.222222 2993999788722
2939672.2939785.
Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. Better hypothesis testing for statisti- cal machine translation: Controlling for optimizer instability. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 176-181, 2011.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with rein- forcement learning. Proceedings of the 2022 Conference on Empirical Methods Natu- ral Language Processing, pp. 3369-3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 1226:////2222.mnmmmmmmaaa.
URL
attts://aclanthoole wwgp 2022. emnlp-main 222
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int80: 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: wwwpreeieeeeeeeeeeeemmee net wttp://wwww.eeeeoouueeeeeee.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. Openprompt: An open-source framework for prompt-learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 105-113, 2022.
Yanai Flazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- cls. Transactions of the Association for Computational Linguistics, 1:102-11031, 2021.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. pp. 3816-3830, August 2021. doi: (0.18653/v1/202) Iol- URL hetps: //aclantho ogg.hll/2ooorrr/002000/0 acl 1 2.95
10

---

Published as a conference paper at ICLR 2024
Hila Gonen, Srini Iyer. Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037, 2022.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. pp. 1935- 1952, July 2023. doi: 10.18653/v1/ 2023.acl-long.11e long.10...1..... 108. Utt.htttss///ccanttoool http:/oooy..rr//...aaaaae.a.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint
17X.. 2017.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Han- nanch Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt way- wardness: The curious case of discretized interpretation of continuous prompts. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 3631-3643 Seattle, United States, July URL
2022. Association for Computational Linguistics. doi: 10.18653/412022..mmss
wrtp naac ma n.. 266
Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4 1985.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-effciient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 0045-3059, 2021.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74- 2004.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically or- dered prompts and where to find them: Overcoming few-shot prompt order sensitivity. pp. 8086- 8098, May 2022. doi: 100.66661/0022222llloo.....555 URL htttps://cclatttiit ogy.
org acl22aacll-1 loog..n.555..
Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. What makes chain-of-thought prompting effective? a counterfactual study. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
2023.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Mcasuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356-53711 2021.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. GrIPS; Gradient-free, edit-based in- struction search for prompting large language models. pp. 3845 3864, May 2023. doi: 10.18653/
httthttts: potttps:/accantiint /ataaaaaannnttoooooeoaooaoo wrt/ www.::2w......m...............
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2335.033495, 2023.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HI.T), 2021.
11

---

Published as a conference paper at ICLR 2024
Timo Schick, Sahana Udupa, and Hinrich Schüize. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based based bias in nlp. Transactions of the Association for Computational Linguistics, 1:1408-1424, 2021.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's 's the shining is good movie, and good prompt too? arXiv preprint arXiv:2212. 2022.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. 10155///112220..mmmmmmooo //aclanthology. wtwpolloyyyrgg/22000 eww.222..emmmp...mmmmmmmmmmm samnlp-main. main. 346
pp. 4222-4235, November 2020. doi:
URL https:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mi- rali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Nauraaltaattllnsstuuttoonss Generalization via declarative instruc- tions on 1600+ NLP tasks. pp. 5085 5109, December 2022. doi: 10.18653/v I 1/202222222 /20022emmllpppp... main. 340. URL
http://cc..antolooooro///222222 emn wwnemmmllpemmmm 1p-ma i in. 340
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? arXiv preprint arXiv:2307.02483, 2023.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf El- nashar, Jesse Spencer-Smith, and Douglas C Schmidt, A prompt pattern catalog enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302. 1382, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations, 2019.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL Eorum? 2erumiid=222yxx22E.
Andy Zou, Zifan Wang. J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
12

---

Published as a conference paper at ICLR 2024
A GRAMMAR DEFINITION AND INSTANTIATION DETAILS
A.1 EQUIVALENCE RELATION DEFINITION
Precisely, p ~ P2 if and only if at least one of the following hold: p = P2 Bo: or pi
Bi (d; s;) with di d2: pr Bi (di, Si, fi) with di d2: ps = B , Xn, ci)
with 5 X32 1A j n; or p. = Bg" (d. -...,,,....n...... 81, 82, c. f) where d1 = d2 and jk,1 Jk.2 1I & k S It is possible that generated formats equivalent in their string representation are not equivalent according to this equivalence relation.
A.1.1 VISUALIZATION OF PROMPT FORMAT'S S PARSING AND FULL FORMAT GENERATION
<text>
OPTIONS:
CHOICE (A) <text> CHOICE (B) <text>
ANSWER: <text>
OPTIONS:
CHOICE (A) <text>
CHOICE (B) <text>
B ANSWER: <text>
casing 9 casing
separator 8
separator between text and option 8
item formatting item (includes enum. wrapper)
space &
Figure 10: Visualization of complex prompt format showing its parsing and which constants functions affect each part of the format.
Figure 10 shows visualization of how complex format is parsed using our defined grammar. A full prompt consists an instruction, n few-shots and a data point solve. For example, if the struction was Given sentence and two words that appear in it, answer which one of the two (A or B) appeared first in the sentence., full prompt may look follow. Note that always use !n)i space character between instruc- tion and few-shots. The example below shows a I-shot prompt. simply illustrative and does not correspond to any of the tasks considered.
Given a sentence and two words that appear in it, answer which one of the two (A or B) appeared first in the sentence.
The quick brown fox jumps OPTIONS:
CHOICE (A): fox CHOICE (3): brown
ANSWER:
Over the Lazy dog
OPTIONS:
CHOICE (A) lazy ; CHOICE (B) dog
ANSWER:
FORMATSPREAD forces all instantiations of multiple choice variable to change jointly to maintain coherence, and this includes text in the instruction. Therefore, when changing the option items from A and B to and II, the prompt will be generated follows.
13

---

Published as a conference paper at ICLR 2024
Given sentence and two words that appear in it, answer which one of the two II) appeared first in the sentence.
The quick brown fox jumps
OPTIONS:
CHOICE :I): fox CHOICE (IT): brown
ANSWER:
Over the 1azy dog
OPTIONS:
CHOICE (I): lazy 7 CHOICE (II): dog
ANSWER:
A.2 ALLOWED VALUES FOR EACH SET S2, C, FEASING, FITEM
S {"r n/
3 'n' 11' < sep " - In , M'n'
S {" ("!' (t')) (no space, single space, double space, tab)
C ("/m= 1/2 1/: \n)to//n '
'n |n/t'/ -
Feming = (f(x) = x. f(x) =x.title(), f(x) x. upper f(x x. x.lower))
Fram {0 + f(g(z)) such that f E Finemt Ag E Finmz)
Fueml [x + (x), X., x), x + x_), [x), +< x
Fiema [x 1.
x, + +x, x 2x215F + x + 1, ROMAN lower 0, ROMAN[x] upper))
Enumerations are indexed from (i.e., ", 3" rather than "0, 1, 2"). ROMAN x represents the Ro- man numerals written in regular ASCII characters. '0x215F' +x represent the series of Unicode characters for Roman numerals. denotes a spacing character for clarity.
A.3 RESTRICTIONS TO PROMPT FORMATS SPACES AND SEPARATORS' COMBINATIONS
We define several restrictions to ensure format naturalness. Users can additionally customize FOR- MATSPREAD by defining their own rules and restrictions between values. Our rules are as follows:
If (3 where does not contain a newline, then each X, separators and any subcomponents' separators should not contain a newline.
Similar to the rule above, BS" (d, Ji...... $1, 82, c, f1, fe) such that some separator contains newline (i.e. 81 contains newline and/or 82 contains newline) then the space c must also contain a newline.
For (d, 8, /) == [d)ss< text >, must not be the empty string (i.e., there has to be some separation between descriptor and text).
Having be an empty string space in Bl") is only allowed if the first n 1 components are Bi fields with an empty <text>, Similarly, the newline restrictions mentioned above only apply if the <text> is not empty. This rarely happens in prompt formats, but there are formats such as Question. where the Options: do not have corresponding field.
<text> Opt Opnio A. <text> B. <text>
A.4 THOMPSON SAMPLING PRIORS
For the first exploration (i.e., finding the best-performing prompt format), prior Beta(a, 3) Beta max accuracy. Our goal is to set an informative prior where the expected value of the prior distribution is the original format accuracy æ, since priori it is the only information have about performance.
an informative
fx 1 1.11).5, for all arms pt, where æ is the original format's
14

---

Published as a conference paper at ICLR 2024
This restricts the parameters as follows:
a
E (eteaaa,,
=
+ 3
B B
Since will modulate how confident is the prior, and want avoid the model being overconfi- dent, we fix B = 5. Because we want to have an informative prior Beta(o, () with a Gaussian-lik PDF, we foree a and / > 1. In extreme cases, forcing a might alter the expected value.
1.1 and 5 for The first exploration's priors thus exactly Beta(o, with a max (1-z all arms pi.
B-z
For the second exploration (i.e., finding the worst-performing prompt format), the model has access to the first explorations' counters S(E/B) and (B Therefore, we set the second exploration's priors to Beta (a ( { , (N!E/B) - (N!E/B)-S( S(E/B))
B ADDITIONAL EXPERIMENTS INFORMATION AND PLOTS
B. TASK SELECTION
We use a number of heuristics to filter Supper-Naturallnsttuttii tasks to our set of 53 evaluation tasks. Datasets should have least 1000 samples to be considered. We also remove tasks whose in- structions are too long (over 3,000 characters) and datasets with inputs longer than 2,000 characters, given that this makes performing inference scale intractable. We also filter datasets whose valid outputs include more than 20 different strings, given that we focus on classification tasks.
We also removed tasks where found priori performance on the task was 0% accuracy using LLaMA-2-7B 1-shot. Some aaper-Naturlllnstuuctiosssss tasks are derived from the same original dataset, but ask different questions. We did not include more than tasks from the same original dataset.
Finally, we also searched for having socially impactful tasks. Those tasks were the only Super- Naturallnsttrctionss tasks where we included format if one was not provided by the dataset.
The selected tasks were the following 53: task050, task065, task069, task070,
task114, task133, task155, task158, task161, task162, task163, task190, task213, task214, task220, task279, task280, task286,
task296, task297, task316, task317, task319, task320, task322,
task323, task325, task326, task327, task328, task335, task337,
task385, task580, task607, task608, task609, task904, task905, task1186, task1283, task1284, task1297, task1347, task1387, task1419, task1420, task1421, task1423, task1502, task1612,
task1678, task1724.
Ulam--2-78 spread
0.8
r=0.80
p-1.07 x 10
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
Llama-2-7b spread
Figure 11: Comparison between Llama-2-7B and Llama-2-70B spreads. Llama-2-70B was com-
puted using 4bit quantization (Dettmers al. 2022
15

---

Published as a conference paper at ICLR 2024
B.2 ADDITIONAL RESULTS FOR SECTION 4.2
Table 3: Ratio of prompt format pairs P2) such that if is worse than pa using model M1, then the same trend holds for M2.
Model (M1)
Model 2 (M2)
Performance Relative Ordering Preservation
Llama-2-7b
Llama-2-13b
57.46%
Llama-2-7b
Falcon-2-7b
55.91
Falcon-7b
Falcon-7b-Inst
61.1 11%
0.4
(M,A4) - (Llame2-7b, 2-33)
(Lleme-2-7b, Llame-2-133 33b)
(0) (Uama-2-76, Usma 7000
MM) (U-mm--277, (alcon-Ta)
Probaltic that
acctgrol pccipel
gien 40-0 1000
0.3
Probuliity that
pcctggial - arcipul
(fakkon-7b,
(M.
(Ulame2-/bb falcon-7b)
0.2
(M, (flconn78, falcon-7biinst)
0.2
0.1
6.1
0.0
0.0
Threshols dfference 0og)
Threshold diference (d)
(a) Option ranking metric.
(b) Exact prefix matching metric.
Figure 12: Probability of a prompt being worse than p' by at least d points using model M', given that prompt p was better than prompt p' when using model M.
Formats are not inherently good or bad. Table 3 shows that if format p1 has lower performance than format P2 under model Ad, there is 0.62 probability that this trend would hold under another model M' (random chance is 0.5). This weak relative order preservation suggests that prompt format performance in a model may not be extrapolated to a different model, or in other words, that there are no inherently good or bad formats. This finding is further supported by Figure which shows that findings of format being better or worse than another are often inconsistent across models.
Experiments with exact prefix matching accuracy. Here we show results with using exact prefix matching to compute accuracy. Often, failures in prefix matching are associated with degeneration, i..., cases where the model does not answer any of the valid options, motivating the use of ranking accuracy. Degeneration makes models (specially smaller models) more unlikely to have high accu- racy out of the box. As seen in Figure 6 prefix matching is linked to having higher changes when performing atomic changes. Moreover, exact prefix matching can lead to lower performance as gen- eration is less constrained (see Figure [6).. Table shows examples of atomic changes yielding large accuracy changes with exact prefix matching metric.
Figure 13c shows spread remains regardless of model size increase, architecture change, number of few-shot examples also when using exact prefix matching as accuracy metric. In line with the results shown for probability ranking in Section 4.22 Iishoe/ that the probability of reversing performance trends between two models just by changing prompt remains high when using exact prefix matching as metric. Strikingly, spread is significantly higher than in the probability ranking setting (see Figure [4), with median spread ranging from 12 to 28 accuracy points depending on the model used. This further motivates the need for running FORMATSPREAD when benchmarking models with this accuracy metric. This increased spread may be partly due to degeneration, as we will detail next.
Degeneration. Sometimes when model does not generate the correct answer with exact prefix matching, it also does not generate a valid response, i.c. degenerates. We will now quantify this phenomenon using 53 SupperVaturallnstructioos classification and multiple choice tasks.
Given a model, a task, and a format, let the centered mass be the ratio of examples where the model's output matched with any valid option (regardless of correctness). Table 5 shows that the correlation between accuracy and centered mass is moderate or high depending on the model. This suggests that very often when model does not return valid answer, it does not return any valid answer
16

---

Published as a conference paper at ICLR 2024
Unme spread
rm0.79
eo p 1:12 10
roo ped RSX 10
pm3.68 x 10-12
0.4
ralcon- -o-inn---tttt--
0.4
0.
0.2
0.0
Llama-2-7B spread
falcon- spread
spread
(a) Llama-2-7B vs.
(b) Falcon-7B vs. 7B-Instruct
(c) 1- vs. 5-shot (same task, model)
Figure 13: Spread comparison between evaluating the same task under different models or n-shots
using exact prefix matching as accuracy metric.
Uamo 72
(Uama-277,, Liama-2-330 (M. AP (ticonn77, falcon inet)
nshot-1
Usme 2-7
probastiyy tha acete ao00
nahot-5
Uama 130
0.2
nzhoc=1
Uama 2-13b
oven a
nshotws
nshot=1
nshotes
0.01
0.1
1.0
Threshold of accurscy atference (d) llog]
nshotel
bleon Te inst
Figure 15: Probability that model M performs worse than M' by at least d when using format p', given that M performed better than M' by at least d using format p. 53 tasks, 1- and 5-shot. Exact prefix matching metric.
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Performance spread across prompt formats
Figure 14: Spread across models and n-shots. Exact prefix matching metric.
Uamo 7b
Ubma 2 7b
Ulma-2-77
nshot-3
nshot-5
Uama 2:136
Uama-2-13b
nshoz=1
nshotu]
Uama-2-130
Uama-2-13b
falcon
falcon-7b
mhotts
nehore]
fsicon-Tb
felcon-Tb
nshor-d
nahar-s
falcon-bb-inat
nchot-1
blcon To
falcon
nshotw5
nshotes
0.0
0.2
0.4
9.6
0.8
0.0
0.2
0,4
0.6
0.8
Accuracy (Probability Ranking)
Accuracy (Exact Profix Matching)
(a) Accuracy boxplot for the selected 53 Super Natural-Instructions tasks, option ranking metric.
(b) Accuracy boxplot selected 53 Super Natural- Instructions lasks, prefix matching metric.
Figure 16: Accuracy metric used can strongly impact final performance. 53 Super Natural-
Instructions tasks shown. Ranking accuracy yields higher accuracies overall.
at all. This is especially true for Falcon models, where we observe an almost perfect correlation between accuracy and centered mass. In conclusion, prompt format chosen often do not solely affect accuracy, but they also affect the frequency in which a model is actually able to perform task. This will especially affect tasks for which there are no alternative metrics. Further research may focus specifically targeting features that cause degeneration.
Experiments with Instruction Induction tasks. All experiments thus far focused solely on classi- fication tasks. We will now focus tasks that require generating (short) text, and cannot be framed as classification tasks. We selected 10 tasks from Instruction Induction (Honovich et al. 2023) that require generating unique, valid string be considered correct response. Examples include identifying the second letter of a word, adding numbers, or answering a synonym to given word. Instruction Induction tasks also show wide range of difficulty, resulting in varied settings to be
17

---

Published as a conference paper at ICLR 2024
Table Examples of atomic changes' impact on accuracy using prefix matching (probability rank- ing shown in Table 2). represents text field; p2 yields higher accuracy than for all tasks.
Task ld
Prompt Format (p1)
Prompt Format 2 (p2)
Acc
Ace p
DifT.
task213
Ticles O SentencesAAS
ritles: Sentence<A21::))
0.113
0.475
0.362
II Sensence< {}
II Sentence
I1 Sentence<CS; O 11
Sentence<CC> 1:f)
bentenceehg: () Chnicess An
Sentence <hasss() Choicesit)n
<:>11 () \n. <1124111 O
<i>11: Vn <1131: Answer::()
task296
Sentence 1. 1. 1.S. O AnSentence IT
Sentence I : ::44 InSentence :: O) IrSertence 1ET J \nSentonce PE: Insentence :
0.201
0.522
0.321
= () InSencence ETT > 0
AnSentence
AnSentence AnSencence enencence VTT ) (nSentence
AnSentence
InSertence
:
Senteence WET J AnSencence
VIII : 0 VnSentence I 1.0
VIII 1 : O Insentence IX
InSentence :
IrSentence x
2.
0
task905
Tueet::: () An'abelsss sss 0
: :nlabel:::::
0.252
0.559
0.307
\nAnswers:: 0'
\einewer::
task317
Pazsages: () AnAnzaME::
0
Passage:: \aknzow:::()
0,245
0.546
0.301
task280
passage
passage: O' ansver:
0.332
0.612
0.28
task050
OENTRACE
InQUESTTON
0
SENTRACT)eNt \nQIRSPTON))N1-
0.244
0.504
0.26
\AANSMER
(LANSKER(aiL-0)
task070
Beginning
Aoildie 10
Beginning
(aMidal 11()
0.143
0.3
0.157
Hiddle 121
AnEnding
Middle AaEnding
Antnnwee
Table 5: Correlation between accuracy using exact prefix matching and the centered mass (the op- posite of degeneration). 53 tasks, 10 formats each, evaluated on 1000 samples.
correlation between accuracy & and centered mass
Model
n-shot
p- p-value
Alama-2-77
1
0.702
5. 5.1E-77
Llama-2-70
5
0.762
4.9E-98
Llama-2- --bb-
1
0.639
5,8E-61
Llama-2-134
5
0,662
9.2E-67
falcon-7b
1
0.936
7.1E-233
falcon-7b
5
0,933
8,4E-228
falcon-7b-instruce
1
0.962
3,6E-289
falcon-7b-instruct
5
0,958
5.5E-277
analyzed (see Figure 18b
Given that the collection does not contain human- human-generated formats, applied simple IIpput (An Output: ()' format. Results for 1-shot and 5-shot settings show spread is still high across models and n-shot choices (see Figure I7).
Tasks
antonyms, diff, first word letter, larger animal, letters 1 list, num to verbal, second word letter, singular to plural, sum, synonyms.
Deme-2-13b spread
p-3.34 10-
r-0.93
falcor- 7b-rrst spreed
0.8
p-7.40
Ishotes spread
rm0.42
D-6 62 10-3
0.6
0.6-
0.6
0.4
0.4
0.
0.2
02
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Liama-2-7b spread
falcon-7b spread
nisho=11 spread
(a) 1lama-2-7B vs. 13B
(b) Falcon-7B vs. 7B-Instruct
(c) 1- vs. 5-shot (same task, model)
Figure 17: Spread comparison between evaluating the same task under different models n-shots for Instruction Induction tasks. Exact prefix matching used as accuracy metric.
18

---

Published as a conference paper at ICLR 2024
Lama-277
Lama-2 2-7b
natale.
rshote:
1amo-2-7
Lama-2-77
nssoteS
nabate
nshotel
Ulmme22-11b
nahot=s
rshote5
falcon
falcon 7b
nahutes
rshotes
falcon binat.
falcon falcon-bb-irg -i-int
eshotsl
rshotel
falcon-BB-mnt
felcon-Ib-inet
Performance spread across prompt formats
Accuracc (Exact Matching)
(a) Spreads across models and m-shots.
(b) Accuracy variance by model
Figure 18:
Instruction Induction tasks' spreads and accuracy across models. Exact prefix matching used as accuracy metric.
Experiments with continuous metries in open-ended text generation tasks. Throughout the pa- per we focus on tasks with a single valid output, whether in classification tasks or in short-text generation tasks. This decision intentional, since guarantees that variation in the metric truly represents a variation in model performance. We have shown that spread remains high when con- sidering option ranking or exact prefix matching as accuracy metric.
Since LLMs are often used in more open-ended generation contexts, we will now explore the per- formance variance across prompt formats when considering sentence-length generation tasks (e.g. generate the next sentence of story, given the four initial sentences of story, generate a question whose answer the sentence given). To analyze the automatic generations, we use two widely used metrics: ROUGE-L (Lin 2004), and BERTScore (Zhang et al. 2019). The first is n-gram-basee metric, and the latter is model-based metric, and both are [0, 1 metrics where higher is better. Fig- ure shows that variance remains high for LLaMA-2-7B regardless of the metric and the number of n-shots considered, with LLaMA-2-7B 5-shot having 25% of tasks with a ROUGE-L spread of 0.098 or higher, and BERTScore spread of 0.09 or higher.
We observe that the median spread sometimes smaller than the accuracy tasks. This may be because although ROUGE, BERTScore, and accuracy all [0, 1] metrics, typical metric values may be different, which may in turn affect the final spread (an absolute difference). We leave it to future work quantify the differences in style content that each format may be inducing.
Finally, it is worth noting that text generation metrics are known to be noisier, and thus not all metric decreases necessarily correspond to a performance loss, is the case for accuracy in single-valid -alid-output tasks. We used 17 SuperNatural Instructions tasks: task037, task038,
Llama-2-7b
nshot=1 ROUGE-L
1lama-2-78
nshot=5 ROUGE-L
Llama-2-7b
nshot=1
BERTScore
Llama-2-7b
nshot=5 BERTScare
0.1
0.2
0.3
Performance spread 10 prompt formats
Figure 19: Spread across n-shots for LLaMA-2-7B considering ROUGE-L and BERTScore met- rics. 17 sentence-level open- generation tasks are considered, extracted from SuperNatural In- structions. 10 prompt formats are considered for each task.
task040, task067, task071, task072, task105, task216, task223, task240, task348, task389, task443, task845, task1326, task1401, task1613. We selected the 17 open-ended text generation tasks among those with at least 1000 samples, with some formatting present in the original task (e.g. "Passage:" <text>). We only considered tasks whose instructions were under 1,000 characters and that contained inputs no longer than 5,000 characters.
19

---

Published as a conference paper at ICLR 2024
We limit generations to 50 tokens. To parse model outputs more faithfully, and given that none of our expected generations include a newline, we only consider a model's generation up to the first newline (excluding leading spaces and newlines in given generation). This consideration is important given that often models start to generate a new data sample from scratch, immediately after generating the requested answer.
Characterizing a model's 's accuracy distribution beyond spread. Spread gives a quantitative jump in information with respect informing a single point the performance distribution since it mea- sures the distribution range (maximum minus minimum). However, distributions that may share the same range, may yield widely different probability of obtaining each value in the distribution. Fig- ure plots the accuracy distribution of 30 tasks, sorted in decreasing order by standard deviation. Tasks with high standard deviation reflect a higher likelihood of obtaining dissimilar values when making formatting selection; Figure 20 shows that the median standard distribution is a R 0.04, which can be considered high in our context.
task280
task1297
task1 90
task327
tack 31
Fraction of formats
#=00311
d'
0.10
d' 0.89
d' 0.98
0.00
task328
task304
tack2612
task236
task3284
0.50
0.25
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Performance Spread
task325
task323
task213
task158
task
1.0
Fraction of formats
#=0.0
y 0.05
0
a
a 0.04
0.5
0.0
task13B0
tack1283
task5B0
task2431
tack330
1.0
0-0.
5=0.04
0=0.0
d=0.04
0.5
0.0
0.0
0.4
0.8
0.0
0.4
0.8
0.0
0.4
0.8
0.0
0.4
0.8
0.0
0.4
0.8
Performance Spread
task903
task1186
task069
task905
1.0
Fraction of formats
2-0.4
0-03
0.03
0.5
0.0
task720
task065
task2678
task2419
task1387
1.0
0.02
0-022
0-022
a=0.2
0=002
0.5
0.0
0.1
0.4
0.7
0.1
0.4
0.7
0.1
0.4
0.7
0.1
0.4
0.7
0.1
0.4
0.7
Performance Spread
Figure 20: Accuracy distribution across 500 formats for 30 tasks evaluated on 250 samples cach, sorted by standard deviation in decreasing order. LLaMA-2-7B 1-shot, option ranking metrie.
On factors influencing spread besides prompt formatting. We believe many factors beyond for- matting may be influencing performance variance, but were unable to find a feature that reliably predicts spread. We found that the average prompt length in a task has negligible correlation with its performance spread: r 0.228 (p- 1.4 10 r -0.022 (p- 0.615) for option ranking metric, when jointly considering all models and n- shots. Similarly, the standard deviation of the prompt length had negligible correlation with spread: r 0.125 (p- = 0.004) for exact prefix matching, and r = -0.099 (p = 0.024) for option ranking metric. When considering each model individually, only LLaMA-2-7DB with exact prefix matching showed a correlation |r| > 0.5, with the average prompt length having a correlation r = 0.559 p 6.86 x 10-10 10, All other settings had |r < 0.36.
10"7) ) for exact prefix matching metric, and
20

---

Published as a conference paper at ICLR 2024
B.3 PCA EXAMPLES
Section 4.4 systematically analyzes whether we can predict the prompt format that generated given pre-softmax activation layer (i.e., prompt embeddings) by using solely its top-n principal components. Figure 2 shows the top two principal components for two different tasks where all 10 formats considered are easily identifiable solely with a prompt embedding's 's top two principal compoenents.
100
80
Principal Component 2
20
-2
40
-50
50
50
-25
0
25
50
75
Principal Component
Principal Component
Figure 21: Plot of the top two principal components of the last decoder layer of the prompt, as representation of the output probability distribution. Two different tasks shown, with each prompt format shown in a different color.
B.4 NOTABLE FEATURES
As discussed Section 4.3 sometimes the choice of a constant may lead to significantly different
accuracy ranges. Figures 23 and show all strongly dissimilar choices of constants found on any given task, across 53 Super Natural-Instructions tasks, and on both accuracy metrics consid- ered throughout the work. As be appreciated, choices of constants do not consistently predict performance in isolation.
21

---

Published as a conference paper at ICLR 2024
0.6
Accurscy
Accuracy
0.4
44
M
Separator
Soparator Used
0.6
Acouracy
Acouracy
84
0.2
100
10
Separator
Recctanc337111 J0re ew
metscl toutfle Human Wiittar Formani
Accuras
Accursor
Separator S,
Separator
TyseTHeSRES Phoo vou di)
Taox (errpp Cheioi
Accurss
Accurscy
0.2
0.1
Separator S,
Separator
Trak bokk Paaidde Cho
Twok Lokess (Matare Clesedl
09 G+
Acuurazy
0.4
0.0
0.0
1n
Separator St Used
Separator St Used
Tass Di0220 dhstgie Choios
Acourcyy
Pausa Used
Figure 22: Variance by feature value for Llama-7B 1-shot. Evaluated tasks with 500 formats each, and only plots with significant difference between values are shown. Exact prefix matching accuracy metric.
22

---

Published as a conference paper at ICLR 2024
0.350
Accuracy
Accuracy
0.325
0.300
HE1
0.275
100
10
M
Separator
61
Acouracy
0.25
0.00
tnec315 Iepur to
0,7
Accuraat
Accursay
0.5
0.6
tnt
Separator S
Separator st Used
Taok tanck37 (One repur Rest
nesiibe jone rout Human Wither Formar
Accurnyy
0.6
Accurscy
0.5
0.4
Separator &
Separator
Tulk Jav0ss N4lk pi Checol
Tusk uoktesz Crese
Accurszy
AccumnE
0.4
0.5
0.3
101
1n
Separator & Used
Separator St Used
Tork THd 10 iitto Chwict)
Accuasy
0.6
1n
10
Soparater & Used
Figure 23: Variance by feature value for Llama-7B 1-shot. Evaluated 31 tasks with 500 formats each, and only plots with strongly significant differences are shown. Option ranking accuracy evaluation metric. Only S₁ boxplots are shown here, see Figure 24 for all others.
23

---

Published as a conference paper at ICLR 2024
Trae Luseded 10me tor
Tea LeASAsS SMI CYesad
Accurady
Accurady
0.6
Space Usedd
Peust Used
Task: :182213 INAito Cho cat
Phard Used
Figure 24: Variance by feature value for Llama-7B 1-shot. Evaluated 31 tasks with 500 formats each, and only plots with strongly significant differences are shown. Option ranking accuracy evaluation metric. See Figure 23 for more plots.
B.5 THOMPSON SAMPLING RESULTS
Llama-2-700
nshot=1
Llama-2-700
nshot=5
GPT3.5
nshot=1
GPT3.5
nshot=5
0.0
0.2
0.4
0.6
0.8
Performance spread across prompt formats
Figure 25: Spreads found using Thompson Sampling for 320 formats, budget of 40000 evaluations. LLAMA-2-70b was evaluated with option ranking, and GPT3.5 with prefix matching given that we cannot access all logits.
Task
Model
Best Format
Worst Format
Best
Worst
Acc
Acc
tas0050
70B
senter.e II question O
Sentencektt(1e Question\==)An Answez\=()
answez ()
tns0065
Liama- 708
Santence
Sentence V) \a Sentence
Sentence
IEr:N An enne ()
Benlence Vi (1 Option): ( (II::
Sonten.co 53
AnOptian A.
(23)): Ansver ()
() Spt
AnAnsver An ()
Llamo-708
B2GINMING::: ()
HIDDLR Al (
SEGINNISC)A ()a KIDOLE <I>>
835
0.58
MIDDL3
()
ENDING:::
HIDOLE <II> ()
INDTHEAn (P ANSWERAA
Inv0070
Llamn-70R
Regi ar Engeet Os
Reginning ( 11 Kiddle 1.s 0
0.0
0.27
andiedi tel (eti ddle Aef AnRndingsss
Miodle
0 Ending O
(
() VrAnswersss
Insk
Llama- 70B
centence: ( <oep>annver:: O
Sentense: (AnAnSNer:
0.54
0.51
1186 1886
Lhma: 70B
SYSTEH SEFERENCE O). ORIGINAL REPERENCE, () ANBHER ()
System Reference: (Pnoriiginal Reference: (eAnaawez: ()
0.56
0.51
24

---

Published as a conference paper at ICLR 2024
Liamo-708
aysten reforence: () Vaorisinal safarencs: OF Veanswez: O
атвтEм REPERENCE ORIGINAL REFERENCE\=(PMG ANSHERAE()
1284
Liama 70B
SYSIEH RCFERENCE\E() ORIGINAL REPERENCENLL) ANEWER)-()
System Raference O Orseinal Refermnce () Ansaes ()
0.74
0.43
12511297
Liamo- 70B
(E Fact\cIZZALL()
fact ('aaae b 5- Vnguestion:
E88
0.40
KongoGuest
0
Oa 122
Or
}
O2
O.m (1vt : 0.1
Vi 3 (): las
[14 : (Joi2) ()o22a)
0: An
03
( (nanswers 0
Llamo-708
Se: () A
Senience:: (A Beasons: 0%
0.71
0.59
InCuestion:() (nAnsveri()
Question:: Oe Answers: 1
took 3347
LJamo- 70B
SENTENER Al: O
SENTENER Mi
Sentence i0 Sentence 1:1
{} AMEMER::: 1: O
(atnzwar(n {
task138
Llama: 70B
Premicas: (
(nRypothacts::))
premise (Ha hypochosis (Na
AnAnsweri
answer {}
lssk1419
Liama- 701
problem-
Apaptions- Va
Prebiem: ( Options:
<1> Or <2211 O) <3>11
<)>(} <>>p><<>>(
Ob, <4>-2 O. <5>0: 0
<o<p><<>> <aup> <)>
(nanswer- 0
<><p><<><) II Ansver: ()
task1420
70B
(Nnoptionaae 1)=()
28CBLEN- O'EEPTIONS (F
: :1:)(
:0)
:v):()
[1 lel ()aa (0
[el 1010)a (CAMSEER ()
tock
LJamo-70r
PROBLEM: O. OPTIONS: Ane +0 10 e $0 4 00
Problemss: O cptionssss Int :=0 \eiI 1:0 Aettr 1: (aEV )=:() AoV 1::() Ansamesss
O. ANSRER:
()
tonk 1423
Llmmn-7RR
problen: O opt ions: 15
PRONINN:: O. OPTIONELE T.- Ha 1I.- (Hin IIE.- (Na IV.- v.- O. ANSWER:: O
0.25
0.20
0: 5111 11 answers
21:1 O.
31 O. 4221
tonk 1522
Llama-708
Inputif) Outpur:() O
input O outpat O
0.59
0.45
70B
SENTENCE (b, ANEWER 0
Sontence:: () \nAnSwRE:: 0
tsk155
Llama 70B
Sontence: O Answar: (
sentonce::(] - ansvers:
0.55
0.49
t6k1
1Jmoo-70
Senterce (Pio Anawer (
Sentenceln 0 Indnsserln O
msk16122
Liamo-70B
sentenow!
() (Jaentwnott]
Sentwnce =+)= Swntence
0.64
0.35
O Va anewersi
(b) (FAAnneeE O
tock 162
Lama-708 70B
Senterce'nn Os \nAnewerin ()
SENTERCE \nkefl \HANSWER \r)e()
0.45
0.40
tas1163
Liama: 70B
SENTEMC3: (O, ANSWER:
SENTENCE\N (AAMMMERRAa O
0.47
0.35
1678
Llama 70B
FROBLEM: OFTIONS: Antal O-
PROBLEN \nYe() Va OPTIONS lake 111 () (311 (14: (:5)
0.21
0.21
[a] O- [e] O. [d] O- tel
O ANSKER: O
la ANSWER (a(t()
1754
Llama 70B
input: O octpet: O
IMTT::: COTPOT:::)
0.48
0.48
took 190
Llamo-708
Sentencw a O
Swntwoow
Sentence 1. 1.(0Aneentemn 2-(An
0.66
0.35
O)a (osee
Anavws ()
Llamo-70B
TI7L3:11 O \n Sentence 1i:
TITLElt sentence HAFf sentence (Hi -- sentance
0.99
0.50
() Sentence 20) 0
Senlenow 2II) ( SenLence
GinA) SHiLMmOH (oAALO)
la
CHOICESKE
O 1T1102 O.. AN
Ae HI in ANSURR::: O
14
Llama: 70B
ritle: (A scntence
title II sentense sentence (3)
0.92
0.04
Sentence Hi Sentence
sentence
111.: () Senterce 19. (A
() sentence [4] {) choices An 1.(h \eII. 1I anseer 0
choices: (ni.::: O Anst.:11
(A Answer:
task220
Llama- 7015
Bantencel): : (). Bonsancelt);
Bentence <i>: () Santence
0.99
0.83
(). SontencollI) ()-
<2>: O -- Sentencs <3>:
SentenceIVi: : O). sentencew) O
-- Sentence <1>: --
Choices: \a<a>- (Pa <b>-
Sentence <5>: () Choices:
( Answer: (
la 1 II () Answer:
teck229
Llamo-70B
Passacus: () Aa Answers: ()
Passage () Answer- O)
task2280
Liama 708
Passages: () Answerss ()
PA8SAGa\:= ANSMER\=()
0.84
0.04
186
1.amo-70R
INPUT- () OUTFUT- O
Ieputlt() (t()
0.51
206"
1.3m0-70R
Sentercet li: (). Sentencett 1:: O), ScateaceiII y:: (b. Sentenceiv 1:: (h
Sentince 1.: {): VnSencence u.: Oi (asentance III.: O: (nSentence IV.: 117 Ansencence V.: () AnSentence vr.: (): (ascntence VII.: (): Vasentence VIIL:: (br Ansensence IX.: ()) ence x.: ( \a <1>s:() <H1>::)) Ao Anewerla O
0.98
0.53
SenterceV lis O. sertencett
(), SenteneeVII y:: Or SentercevIit lss
Sentenceix 1: O. Sentencex ):s (). AA0 21/0(), Anewer 0
25

---

Published as a conference paper at ICLR 2024
tas2297
Llamo-708
Bentence ta]:: () \nSentence
Bentencel:
Bentence2:
[a]:: O1 \aSeaconce [e]:: ()
Sentenee3:
Sentenced:
inSentence [6]2: O7z Anāenzches
Sentences:
Sentence6;
[o]:: (): \asaatence
()
Bontence7:
Bentence8:
\nSentence [e]s: O37 knacnzenes
Sentences:
Sontence10:
[al:: (h iascatence 11] Or
Aa (00 Q (3)
O \n Answert
InSentence :l1:: ONa (a) (
(a1:: Oi Raswer O
166
Llamo-708
passage
0 Anansver
PASSAGR; AAAEEEER : (
0.51
0.49
m
Llamo-708
Passagela
O/ Ansumelja (
DASSAGRNE (1 ANSWERS-O (()
0.83
19k319
Limmo-708
(n3e O): V=00
Tasgel: O () Answee: (
0.77
0.58
transverinitdf
back
Lamo-708
Target:
Or Anfr \nAnswers
Terget in O An O In Answerin
0.77
O
1ask
Liama 70B
(AA (
Comment
0 AnSNer 0
makk323
Liamo-70B 70B
DomEENL (0 (eanswes V040
Comment 0) Anavee (
0.84
0.58
tnck225
Liamo- 7UB
()) (BANSHER)=()
Comment:
Oh Anawee: 0
task126
Liamo 70B
Conment:: () Answee:: 0
Comment: Or Anawar: (
0.58
0.51
127
Lamo--08
Comment O <eep>Ansver O
Comment
O Answer 0
0.88
128
1.mma-70R
comment: : -- s
Comment:
( Anguers 0
0.81
0.52
Llama 70B
post:: O answer:: i
Post:
O (Antnswrr: Q
0.52
0.50
tmsk33?
Liama 70B
past O (panswor O
O COSNSE O
0.85
0.63
Inok388
Llmmo-708
CONTEXT ( QUESTION i
contexci (n²=() Anguestion
0.38
0.11
O9TIONS Vel.- (). 2.- O. 2.-
(a)=() Anaptiona Lote 1 (:
( AMMMER O
)):(() )):() Ypanswer \ay-O
Llamo- 70B
Context- (Naa Question OA Optiors- \nn) O - {
Context (). Lan O. Opliona
0.78
0.40
(1 \ntt Q \nI1T
(An Answer ()
(). Answer 0
Llamo-70B
INPUT (o)=() \o corpor (0(((
Topat O- Ousput 0
0.68
0.51
bxk608
Liam- 70B
input: O. output: O
INFUT (ole() \n(e(
0.71
0.48
tmx.099
Liama- 70B
Input (nte() Va cutput \nktO
Input O, Octput O
0.71
0.51
tnc1504
Llamo-7018
Imput::: O ouspuc::: O
(N :
0.71
0.55
tas5505
Liama 7088
Twoot:: 1I labal::() II Anewer::
THE3T: la LABEL: Q
0.68
ANSWER: ()
GPT3.5
Sentercetn11( An QussticololL() Aa (a)=
swntsnoe: 0
questiion 0
0.67
0.1
()
GPT3.5
SENTSRCSi 0 InSENTENCEE 1..0
Sentence 1 A Sentence 3
0.82
0.33
0
:: () Sent 4 (
INSENTENCE1V
Sentence 5 : O <sep>-0pcion (3: 1 <sep>aation 11:11 O
AnSENTENCEV 110 C AOOPTION
()OFTTON (2 (EANSMER)=())
<aep>Annuer- (
GPT3.5
BEGENMING: (Ain MIDELENE (} Va eT (Pin
Beginning ( 11 Middle
Middie lal- 0 \n Middle [b]- () 11 Ending 0
0.83
0.65
ENDING: OA ANSMER:
Answer (
task070"
GPT3.5
Begiandar : Ormiedde III: () -- Middle (II): (NaaEniin
Beginnirak=() <cop>Midcle O II Hiddle ()
0,70
0.19
()
Antnswee
<oop>andina\E() <eep>>tnswrr\s
taskl 14
GPT33
sentencw
SENTENC:: ()- ANSKER: ()
B67
0.3
1nsk1186
GPT35
aysten reference origiral refarence 0 answes 0
0
systen bafurence: (Paoziginal Rnference: O'rrnnwer:: 0
033
1243
GPT3.5
SYSTEM REFERRECE () ORIGINAL REPERENCE : f ANSMER (
System Befecenceia (} VaDriginal Reference () \nAnsverin f
0.50
bosk 1784
GPT3.5
Systen
(n) Original Refer:ncs (n
system reference:s: 0 original roterence::: 0
066
Answer (n(t()
answers:: (}
back 207"
GPT3.5
Fact<c1>: 0 Fact <2>> Or
ractita. O ractlee.
Anguastion:: (1 (aai:
O 12)::
[a question : () lns.
1111:: O 191:
O viss
(1.. - (). (sa. 1).
vi VIII::
(2. (Ja. ().. (3%a
(h Velnewez::()
answer O
133
GPT335
Senterce:: 0
Sentence -h \ascasca 00) Laguestion (bs Andnswes
0.69
0.61
Questica: Answes::
tnsk11447
GPT3.5
BantenceI.::
(M SuntancetI.:::
Bentence (I) Sentence (II) ()
0.46
0.42
() ()(
(nAnsver::()
19771388
GPT3.5
Pr)nise:() Hyposheeix:: Ansoes: ()
PRENTeR:: OF \NWPPOTHRSTS: (): :HARSWER::
O)
1419
GPT3.5
Problen 0 Options
PROBLEN- (POPTIINS- > 0 2 O 31 0
ArIAIte [RIxe() ()) (IA(- (EA)F (I nnswer
0 51 () (AAAREEER-
0
26

---

Published as a conference paper at ICLR 2024
tas31420"
GP133
problen- O No options- (1) ()(21) (111)) () (IV) (F ( (e anower- 0
Probiam:: 2 0 -- Optionazs
AaE43: (b) Vorii]: O1
(ag4s41: Oi Ancie]: O
Vatuo: O
AnsweE::
lnsk1421
GPT335
Problon (e options telt
PROSLEN; O, OPTIONB:: Anta]:
0.30
0.01
\edu :23) -- 21::(} -- :1:0
O Anibl O
Velej= O
:)::() [1 Answes
Volels O. : (
1n231433
GPT3.5
PROBLEM: O2 /nGPT-ous: \r a
Probl ems: O) Ogt fonsss
0.50
0.10
Ya b O 0 la () VOMNSKER: (}
d
\oti): Witiil O):
Vatiiil: () Aativ]: ():
Anlv): 0 Ansuerss 0
1502
GPT33
Enput)y D'nout Va O SENTERC3:: O - ANSKER::()
input ++ :
0.54
155
GPT3.5
SENTERCE O ANSWER O SENTENCE)t() ANGVERAE()
0.48
0.40
15s
GPT3.5
Sentonse: ( Answer:()
msk161
GPT35
BenLsnce (AAAAsvvE- ()
0.40
0.31
fnsk1612
GPT35
sentence 1: 01 1I sentence
Bentence I : () \nBencence : O v Answez O)
0.66
): O. answer::
t6x1162
GPT35
Senterce
0
SENTENCRAE Or (oAwwwR)Ee
0.37
16k
GPT3.
Senterow
Or InAnzser (
sentence: (maanseer: 0
0:43
0.30
1668
GPT3.5
Problem 0 Options An (2)
PROBLEN: :
OPTIONESE
ARTET 1391
O <cep> III1 {) <sep> (III)
{ (EX) O [III] O
( <><p>>11 1TV1 () <sep> (V1 O
( M O In A
Answer O
I724
GPT3.5
IXPUT:: 11 COTPUT::()
INFUT: O OUIPUT: O
0.63
0.57
tssk190
GP133
Bentence (1) Or VnSentence
Bentenos<1>: (Nasentenca
0.16
(II) OF (adnover O)
O, Answez
O
3
GPT3.5
Tille: Kaepb
<1>:
TITL:: O. ( sentence)t
0:98
O II Sentence <2>: 0 1I
Senterce on <3>: O 11 Sentence
sent (e)) swniance (t O. CHOICES: (at42:0) (..:: ANSKER: (
<4>; () <ang2Choices: Jacis- <oup><li>> O ExegaAneent: 0
tnsk214"
GPT335
Litle: () (ncentencel)) O.
T[ Sentence )=: Sentenceltb 100 Sencenceleo 1= . Sentencely (od 1::
0:97
0.4
Sentenceiill Or sentencetis)
(r. Sentenceivi: O iachoicnes owes LecT> 0 <II> () Inanswer: {
Chaices:: [il. Ofi... 0 Answeris()
GPT3.5
Senterce [1]: O in Sentence
Sentenceyti.. 0 II
0.98
0.79
[21: (} An Sentence 131:
Sent Sentenceltiin i.: Q 1
O la Sentence [4]: 0 la
Sentence
Senterce 151: O choices: In <A>::(Aa <B>:: Ancwer: ()
Sentence
SentenceVtt... 0 Choices:
(n(1):() 11 (1:1: Anewer: 0
GPT3.5
pacsage O r snewer O
Pansage: (b. Anewer: 0
msk2280
GPT35
Passace - () <oap>Answez
Passage () -- Anawer 0
0.85
0.30
task286
GPT35
Input- ON Outpet- 0
INFOT:: 0 coTPOT:: 0
206
GPT3.5
Senterce
0 InSentence
Sentance (T))- i
0.95
0.68
() VoSentence
Sentence
(
(nBentencs } ( lasantence
Bentencs)t1111
* - () \nSentence f (
Sentence ITVI
InSentence 9
( (DEent
Sentence)1 0
O InSentence
Sentencek C
SnSentence 1 - 0 (TeAs- (h.
Sentencelt (977))
<B> () Jedmsaett: ()
Sentence 1912T)
Sentence (t ITXI
Sent (X)
297
GPT3.5
Sentence (811 (br InSentence
SENTENCE) 18) (}
0.36
0.06
(R1: Insentence (0): ()z
(nSPNTENCE (1)
[):
\nSentence 1D): () (ascatcace
(aSENTENCE)
(31: (b \nSentenze (F): (1
(nGENTENCE (3)
\nSentence 181: (I \stentence
ISSETTENE (3)
(H1: Ob \pSentence (2) O7
(SSENTENCE (2)
InSentence id); (b: (><1>110
ASENTENCE (G)
<sep> :))::;)); (nAnsver (
RSPNTENCEA (10)
SSSENTENCE
O III : O) -- AMNERR: 0
16
GPT3.5
(a (eanswer\alt(()
Passsge- O Anewer- O
0.49
0.48
GPT3.5
pacsaco: O answer: O
peoszge O enswer O
0.75
0.70
1
GPT3.5
TARGET:: ()
0 AnD (HANSMEDE:
Tazgel: 0v AnAnawees
0
0.66
GPT3.5
Targetrs:
( v0O \nAnsweress
target ( answer 0
0.73
0.68
()
122
GPT335
Comment:
0 - Anewer: O
comment:: :() anewer::(]
0.84
0.83
27

---

Published as a conference paper at ICLR 2024
CONMENT (
<eep>>ANGMER (
comment (
('nanssuu {
mok325
GPT33
conment )
Yeanswez i
Commentlys (/ Answerko
0881
0.74
comnect 0
answez (
Comment- () Answes- O
127
GPT3.
conment: : Ananswer: (
Comment ( Ansuer
0.86
0.82
Comment 0- Answer 0
CONMENT () < <omp?>ANEE 0
135
GPT3.5
Fostin)=() \ednewer/atte>
0.37
0.20
137
GPT3.5
:o \
B0st (r, Anewer O
0.57
0.51
GPT3.
ContwxL:: Apquessions Linss
Context: (). Queat ions
incptiona:: al:es (A
Optiona: \n;1] O [-1] (11-14:
(Fn c):1: Oh Vahnswer:: 0
(). AnsweE:()
GPT3.5
Queat
CONTEXT::
QURETION::
I1 lonase
B) 0
OFTIONB: Naft) U-- [11]
02- ( 11 Answerss
C). (171] - (1
ANSWERE
Input () ouspac 0
INPOT:: 0 0
Ind.008
GPT3.5
Input:: Output:: 0
InputAn O (noutputin
0.60
0.52
bnk6009
GPT3.5
IXPUTIn O \nOUTeUT)nn O
Input (nnooppt O
Ins5504
GPT35
INPUT (AACUUPT
Impat::
Octpets:
0.66
Ins5905
GPT3.5
Twest:
Label: ( Answar:
THE27 O II LABEL () |I AMBWER
0.72
0.63
Table 6: Best and worst formats found by using Thompson Sampling on 320 formats on Llama-70B and GPT3.5 E 40000, B 10. Tasks marked with an asterisk" indicate a format where the
Roman numerals used correspond to Unicode characters (starting at 0x215F) visualized as I, III. IV.
28

---

Published as a conference paper at ICLR 2024
C LIMITATIONS
As defined by our grammar, all equivalent formats are semantically equivalent to human read- ers. However, some of them are more likely to be used by humans than others. Spaces and separators are inspired from naturally-occurring formats, but some values are more un- usual, such as the spacing <sep> or the separator ::. Contextual restrictions enable disal- lowing undesired combinations of e.g. spaces and separators. However, formats may have multiple valid parses, and some may be more prone than others to unnatural character combi- nations. For example, let data sample be "Passage: lorem ipsum dolor sit amet. Answer: Yes'. Depending on if we consider the full stop to part of the passage or the format, we may parse it as B (Bi(Pasaage,' ': ',id), (Answerr,'' '.id)," ') or 'iid), (A 'did), 7). this work, we choose the for- mer parsing throughout tasks to ensure full sentences. This sometime leads equivalent for- mats to have less usual, yet trivially semantically equivalent resulting character combinations, e.g. B122 (Bi (Passage,'s : id), B1 (Answerr,''
(B1 (
(Passage:,;'
'did)'; 1). This last format would have the following string form on the example above: 'Passage: Lorem ipsum dolor
amet.; Answer: Yes'. We observe high performance spread both in these cases and beyond them. Contextual relations may also restrict these cases desired by the end user.
Additionally, we focus our evaluation on tasks that have reasonably short input instructions and input field length (see task selection details B.I. Future work may investigate on how input length affects final performance.
LLess than 20% of cases, based on manual inspection of 10 formats across 20 tasks.
29


Reviews:
[{"review_id": "qnuHYOzvjF", "reviewer": "Reviewer_RFEF", "summary": "This paper explores a very interesting question, which is the impact of different prompt formats of LLMs on the accuracy of downstream tasks. The authors found that this impact is significant to some extent and present a new algorithm called FORMATSPREAD, which can estimate the performance spread of different prompt formatting choices. FORMATSPREAD efficiently searches the space of plausible prompt formats within a specified computational budget.", "questions": "1. Add more analysis about cofounders, such as content length in the examplars, different tokenize methods, etc.\n2. Add further explanation of which specific tasks are more susceptible of changes in prompt formatting.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The issue discussed in this paper, i.e., LLM evaluation should not be limited to a single specific prompt, has good informativeness for the community.\n- This paper is well written and easy to understand. \n- The proposed construction of grammar rules for different prompt formats and the design of sensitivity evaluation are quite ingenious.", "weaknesses": "As the authors claim that LLMs are extremely sensitive to subtle changes in prompt formatting, with performance differences of up to 76 accuracy points, which is quite surprising. It is necessary to conduct a more in-depth analysis of these somewhat counterintuitive conclusions. For example, \n1. Is the difference in  prompt formats the only influencing factor, or do other confounders exist, such as the content length of in-context, different tokenize methods?\n2. It is difficult to predict the impact on specific task sensitivities. How can we analyze which types of tasks are more susceptible to prompt format influences, rather than just conducting sensitivity evaluations? This requires further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BpQRKA1iUZ", "reviewer": "Reviewer_fgsZ", "summary": "Authors claim that in recent trends, LLMs are the inevitable choice for language technologies; their sensitivity to prompt formatting results in critical issues. The sensitivity remains critical even after increasing model size, usage of in-context learning, and instruction-tuning. Given this situation, authors suggest evaluating the performance of LLMs with various prompt formatting, not with a single prompt format. The authors also report that format performance across LLMs has a low correlation, which supports the multi-prompt format evaluation of LLMs. To facilitate the multi-prompt format, the authors propose an analysis method, FORMATSPREAD, that evaluates a set of plausible prompt formats for a given task. The proposed method induces the concept of semantically equivalent formats and measures the performance gap among LLMs queried with the different formats but in a semantically equivalent set, procured by the help of Bayesian Optimization.", "questions": "- As the search space of the prompt is intractable, a good initial point (in this case, a prompt) would be crucial for successful entire optimization (as authors assume non-adversarial user). How can we be convinced that we start from a good initial point?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Focus on the subject that has not taken much attention from the community but should be addressed for robust application of LLMs\n- Evaluating LLMs over the prompt distribution provides a more informative understanding of the model's performance and robustness than evaluating only with a single prompt.\n- The proposed method can be utilized on API-gated models, with no need for access to model weights.", "weaknesses": "- The authors cast the problem of searching prompt space as a bandit problem. However, many current important applications of LLM assume multi-turn conversation between the user and LLM, which is highly dependent on the conversation history.\n- In Algorithm 1, the formulation assumes the reward of each arm is success or failure, but in the NLP field, there are many important tasks where the output of LM cannot be determined between success or failure (for example, a task needs human alignment). Does the formulation still hold for the tasks that cannot be evaluated in discrete value?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YZySQzD9Xu", "reviewer": "Reviewer_c9z9", "summary": "This paper explores the sensitivity of large language models (LLMs) to prompt formatting choices and investigates the impact of prompt design on LLM performance in few-shot (especially few-shot classification) settings. The authors find that even subtle changes in prompt formatting can have a significant effect on model behavior, with performance differences of up to 76 accuracy points. This sensitivity remains even when the model size or the number of few-shot examples is increased. \nThe authors argue that evaluating LLMs with a single prompt format is inadequate and propose reporting a range of performance across plausible formats. \nThey also demonstrate that format performance weakly correlates between models, questioning the validity of comparing models using a fixed prompt format. To facilitate systematic analysis, the authors introduce an algorithm called FORMATSPREAD, which quickly evaluates a sampled set of prompt formats for a given task. The paper emphasizes the importance of reporting and considering prompt format variations when comparing models and highlights the impact of formatting choices on model behavior by extensive experiments.", "questions": "1. In Section 3.2, the performance spread is quantified as the difference between the maximum and minimum values. I suggest that a more comprehensive approach would be to report both the range (max - min) and the standard deviation, providing a fuller understanding of the data distribution.\n\n2. In the experimental section, the authors explain that classification tasks were selected for their ease of automatic evaluation. I am curious about the challenges associated with measuring performance for text generation tasks, especially considering the benchmarks that have been proposed recently.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper investigates a critical issue for Large Language Models (LLMs), specifically the impact of formatting on the few-shot examples used in prompts.\n\n2. The assertion that \"The performance of Large Language Models (LLMs) is highly sensitive to prompt formatting choices, especially in few-shot settings,\" is substantiated by numerous experiments on few-shot classification tasks (e.g., Super-Natural Instructions) and short text generation tasks, such as identifying the second letter of a word, performing arithmetic, or responding with a synonym for a given word.\n\n3. In Section 3, the authors formally define the \"grammar\" of plausible prompt formats, thereby making the problem formulation more rigorous.", "weaknesses": "1. The paper primarily substantiates its core claim that \"Performance of large language models (LLMs) is highly sensitive to prompt formatting choices, particularly in few-shot settings,\" through experiments in classification tasks. However, the scope of the experiments does not extend to the frequently utilized capability of LLMs for long text generation. While short text generation tasks (such as identifying the second letter of a word, adding numbers, or responding with a synonym) are discussed in the appendix, these do not fully capture the important aspect of long text generation. Hence, I suggest that the authors either explicitly state that these findings are specifically under the context of classification tasks or conduct additional experiments on long-text generation to avoid any potential overclaim or misleading interpretation.\n\n2. For Figure 2, I recommend that the authors include the Pearson correlation coefficient directly on the figure for a more comprehensive representation of the data.\n\nOverall, I believe this paper studies an important question. If the authors can address my concerns, I would consider increasing my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)