Venue: NAACL 2024

# How are Prompts Different in Terms of Sensitivity?

Sheng Lu, Hendrik Schuff, Iryna Gurevych

Ubiquitous Knowledge Processing Lab (UKP Lab)

Department of Computer Science and Hessian Center for AI (hessian.AI)

Technical University of Darmstadt

www.ukp.tu-darmstadt.de

###### Abstract

In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompt techniques across different models and tasks. To address this, we present a comprehensive prompt analysis based on sensitivity. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce _sensitivity-aware_ decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.1

Footnote 1: Our code is available at [https://github.com/UKPLab/nacl2024-prompt-sensitivity](https://github.com/UKPLab/nacl2024-prompt-sensitivity).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ppages 5833–5856

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

In-context learning (ICL) has become a popular learning paradigm in natural language processing (NLP) due to the rapid development of large language models (LLMs) Brown et al. (2020); Dong et al. (2022); Liu et al. (2023). With carefully constructed prompts, ICL achieves impressive performance on various tasks Kojima et al. (2022); Lampinen et al. (2022); Wei et al. (2022b); Srivastava et al. (2023). As a result, prompt engineering, which aims to find prompts that lead to optimal performance, has emerged as a crucial research topic in ICL White et al. (2023); Zhou et al. (2023b). Although effort has been made to understand the effectiveness of certain prompt techniques Feng et al. (2023); Gonen et al. (2023); Wang et al. (2023a), there is no systematic analysis of them across various tasks and models Ajith et al. (2023). Such an analysis is crucial for prompt engineering, prompt selection, and gaining a deeper understanding of the working mechanism of ICL.

In this paper, we present a systematic and comprehensive analysis of prompts based on the sensitivity of a function Hahn et al. (2021). We hypothesize that certain prompts are more effective for a given task because they decrease the level of sensitivity. Based on the recent findings that ICL implements gradient descent implicitly Akyurek et al. (2023); Li et al. (2023); Von Oswald et al. (2023); Zhang et al. (2023), an effective prompt can be seen as one that facilitates the learning of a new function with lower sensitivity compared to the original function learnt by the model. The sensitivity of a function enables a novel framework to analyze the effect of different prompts. See Figure 1 for more details.

We did extensive experiments to validate our hypothesis. We chose five widely used natural language understanding and common sense reasoning tasks. We selected models with varying sizes from three popular families: GPT, LLaMA, and T5. We tested different prompts, including both human-designed prompts and prompts generated by an LLM. The results strongly support our hypothesis that models exhibit different levels of sensitivity depending on the prompts used, and sensitivity is an unsupervised proxy of performance as it has a strong negative correlation between accuracy and sensitivity. With the help of gradient-based saliency scores, we find that tokens in the prompt (e.g., instructions) are more relevant to the output than tokens in the input where perturbations took place, which explains how different prompts lead to varying levels of sensitivity. Furthermore, we introduce _sensitivity-aware_ decoding, which incorporates sensitivity estimation as a penalty term in greedy decoding. We show that _sensitivity-aware_

decoding is particularly effective when the prompt contains scarce information. Our work provides a fresh perspective for comparing the effects of different prompts and enhances our understanding of the mechanism of ICL.

Our contributions are summarized as follows:

* We present a systematic and comprehensive analysis of prompts based on the sensitivity of a function Hahn et al. (2021).
* We show that sensitivity is an unsupervised proxy of accuracy as it exhibits a strong negative correlation with accuracy.
* We use gradient-based saliency scores to show empirically why certain prompts lead to lower sensitivity.
* We introduce _sensitivity-aware_ decoding and show that it is effective when the prompt contains limited information.

## 2 Background

### In-context learning

In-context learning (ICL) is a popular learning paradigm that emerged with the advent of LLMs Brown et al. (2020); Liu et al. (2023). It typically involves prompting the LLM with several demonstrations or exemplars in natural language. Compared to previous learning approaches, ICL has a more interpretable interface and it is more computationally efficient Dong et al. (2022); Zhou et al. (2023b). ICL has demonstrated strong performance on various natural language tasks Kojima et al. (2022); Lampinen et al. (2022); Wei et al. (2022a); Srivastava et al. (2023).

A considerable amount of recent work focuses on revealing the mechanism of ICL. A line of work suggests that ICL is facilitated when the pretraining distribution has certain properties, such as containing compositional structures and latent tasks Chan et al. (2022); Hahn and Goyal (2023); Wies et al. (2023). Empirical evidence shows that ICL implicitly implements gradient descent and constructs a function at inference time Akyurek et al. (2023); Li et al. (2023); Zhang et al. (2023), which may be related to gradient-based meta-learning Von Oswald et al. (2023). Similarly, Dai et al. (2023) argue that Transformers are meta-optimizers which produce meta-gradients according to the demonstrations through forward pass, and these meta-gradients are applied to the model through attention.

### Prompt engineering

Prompt engineering is essential to effectively retrieving information from an LLM Reynolds and McDonell (2021); Schick and Schutze (2021); White et al. (2023); Zhou et al. (2023b). An LLM usually requires careful prompt engineering, since a model may not understand prompts in the way a human does Webson and Pavlick (2022).

In this work, we focus on discrete prompts, i.e., prompts that are described in natural language phrases Liu et al. (2023). We follow Dong et al. (2022) and categorize discrete prompts into human-designed and LM-generated prompts, depending on whether they are written by humans or generated by a language model (LM).

### Prompt analysis

Most of the existing analytical work concentrates on understanding a particular type of prompt Feng et al. (2023); Gonen et al. (2023); Wang et al. (2023a). However, there is a lack of systematic analysis that compares the effects of different prompts across various models and tasks. As far as we are aware, Ajith et al. (2023) is the only work that presents a systematic analysis of prompts. They evaluate the effect of popular instruction selection methods,

Figure 1: (a) We generate synthetic data for testing instances using Hahn et al. (2021)’s framework. (b) We perform inference multiple times using the original and synthetic data, and calculate sensitivity based on the predictions.

 

whereas our work examines a broader range of prompts.

### Sensitivity

Previous studies have primarily focused on analyzing sensitivity at the instance level. These investigations reveal that ICL performance is highly dependent on demonstrations, such as the selection of exemplars and the order in which they are presented (Zhao et al., 2021; Liu et al., 2022a; Lu et al., 2022; Ajith et al., 2023; Chang and Jia, 2023; Chen et al., 2023; Wang et al., 2023b; Sclar et al., 2024). Moreover, Chen et al. (2023) observe that predictions sensitive to perturbations are more likely to be incorrect.

Based on the theory of Boolean function sensitivity, Hahn et al. (2021) propose sensitivity as a theory of complexity for sequence classification tasks. The sensitivity of a function quantifies the number of disjoint subsets of the input sequence that can be changed in such a way as to change the output. In the setting of sequence classification, sensitivity measures the non-linearity of the decision boundary. Low-sensitivity tasks are those where low-sensitivity functions, such as linear classifiers, are most successful. High-sensitivity tasks, on the other hand, require high-sensitivity methods, which are more complex. The amount of information in the input is a key factor of sensitivity. Intuitively, if a single change in the input completely changes the output, it is believed that the input does not contain sufficient information, resulting in high sensitivity. An output is more stable if there is redundant information in the input, which is an indicator of low sensitivity.

Sensitivity is an indicator of both architectural and task complexity, and thus it is used as a hardness measure in many NLP tasks Richardson and Sabharwal, 2022; Zhao et al., 2022; Bhattacharya et al., 2023).

## 3 Experiment settings

**Data generation** We use Hahn et al. (2021)'s framework to generate perturbed data.2 Each of the synthetic data agrees on the original instance on all indices outside a subset. We notice that the synthetic data for one particular dataset are noisier (see Table 8 in A.1 for a manual inspection of the data). This does not pose an issue because this dataset is our control variable.

Footnote 2: See [https://github.com/m-hahn/sensitivity](https://github.com/m-hahn/sensitivity).

**Sensitivity estimation** The sensitivity estimation proposed in Hahn et al. (2021) uses the variance of the outputs. We adopt a more straightforward alternative, variation-ratio Freeman, 1965), to estimate sensitivity. Given an original input and \(n\) synthetic inputs, sensitivity is calculated as

\[s=1-\frac{f_{m}}{n+1}, \tag{1}\]

where \(f_{m}\) is the frequency of the mode of the \(n+1\) predictions, i.e., the prediction for the original input plus \(n\) predictions for the synthetic inputs. The lower \(s\) is, the less sensitive a model is to an input.

**Dataset** We picked five commonly used natural language understanding and reasoning tasks: CoLA Warstadt et al. (2019), MultiNLI Williams et al. (2018), RTE Wang et al. (2019), SST2 Socher et al. (2013), and CSQA Talmor et al. (2019). We experimented with GSM8K Cobbe et al. (2021), a collection of arithmetic reasoning problems, to assess sensitivity in open-ended generation.

**Model** We tested models with different architectures and sizes selected models from three popular model families: OpenAI text-davinci-003 (GPT3.5-175B), GPT-JT-6B Wang and Komatsuzaki (2021), LLLaMAA2-13B-chat, LLAMAA2-7B-chat Touvron et al. (2023b), Flan-T5-11B, and Flan-T5-770M Chung et al. (2022).

**Prompt** Table 1 shows the prompts we used in our experiments. We experimented with both human-designed and LM-generated prompts. We designed base_a and base_b as two baseline prompts. Compared to the simplest base_a which contains plain _input-target_ pair, base_b includes a human-designed instruction. We designed zero_a and zero_b to test an extreme case, where the ground truth is included in the prompt. We tested two popular prompts, i.e., context faithful prompting (CFP) Zhou et al. (2023) and _Chain-of-Thought_ prompting CoT) Wei et al. (2022b).3 For LM-generated prompts, we tested automatic prompt engineer (APE) Zhou et al. (2023) and generated knowledge prompting (GRP) Liu et al. (2022b).4 In addition, we map each option to an index, so that we can better control the format of the output.

Footnote 3: CoT prompting was only tested with the larger models, i.e., GPT3.5-175B, Flan-T5-11B, and LLaMA2-13B-chat.

See A.1 for more details regarding the setup of experiments.

 



## 4 Results

Figure 2 shows the average accuracy and sensitivity of each model using various prompts across different datasets.5 The _Pearson_ correlation coefficient shows a strong negative correlation between accuracy and sensitivity (\(r=-0.8764\), _p_-value \(\ll 0.01\)). Sensitivity can be viewed as an unsupervised proxy of accuracy given such a strong correlation.

Footnote 5: Due to limited space, we did not include the standard deviations of the statistics in the following plots and tables. For more results, please refer to [https://github.com/UKPLab/naacl2024-prompt-sensitivity](https://github.com/UKPLab/naacl2024-prompt-sensitivity).

It is interesting to note that both Flan-T5 models failed with zero_a and zero_b. We discuss this further in Section 4.2.

### Instruction, knowledge, _chain-of-thought_

This section compares the effects of instruction (human-designed instructions in base_b and LM-generated instructions in APE), reasoning chain (in CoT), and knowledge (in GKP).

Table 2 compares the performance of models using base_b and APE. The two prompts lead to similar accuracy and sensitivity on CoLA and RTE, suggesting that human-designed and LM-generated instructions have similar effects on the model. At least for CoLA and RTE, there is no need to generate instructions using an LM.

Table 3 shows that GKP leads to a higher accuracy and lower sensitivity in most cases. This suggests that the effects of instructions and LM-generated knowledge are cumulative, i.e., plac

\begin{table}
\begin{tabular}{l l l} \hline \hline
**prompt** & text \\ \hline base\_a & I’m glad I saw anybody. \\  & (target) \\ \hline  & sentence: I’m glad I saw anybody. \\ base\_b & question: Is this (0) unacceptable, or (1) acceptable? \\  & answer: (target) \\ \hline zero\_a & I’m glad I saw anybody. **The answer is 0.** \\  & (target) \\ \hline zero\_b & sentence: I’m glad I saw anybody. **The answer is 0.** \\  & answer: (target) \\ \hline  & Bob said, “I’m glad I saw anybody.” \\ CPP & Question: Is this (0) unacceptable, or (1) acceptable **in Bob’s opinion?** \\  & answer: (target) \\ \hline  & sentence: I’m glad I saw anybody. \\  & question: Is this (0) unacceptable, or (1) acceptable? \\ CoT & answer: **Let’s think step by step.** This sentence is ungrammatical because “anybody” is used as the object in an affirmative clause. So the answer is (target) \\ \hline  & instruction: determine whether each sentence was (1) acceptable or (0) unacceptable **based on its structure and grammar.** \\  & input: I’m glad I saw anybody. \\  & output: (target) \\ \hline  & knowledge: Electronic maps are the modern version of paper atlas. \\  & input: Google Maps and other highway and street GPS services have replaced what? \\ GPPs: (0) united states, (1) mexico, (2) counstrside, (3) atlas, (4) oceans \\  & output: (target) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of prompts used in our experiments. Contents that are characteristic to a prompt are **bolded**.

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**dataset** & **prompt** & **accuracy\(\uparrow\)** & **sensitivity\(\downarrow\)** \\ \hline CoLA & base\_b & **0.8235** & **0.1830** \\  & APE & 0.8216 & 0.1960 \\ \hline RTE & base\_b & 0.6931 & **0.1377** \\  & APE & **0.7509** & 0.1603 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The accuracy and sensitivity of GPT3.5-175B using base_b and APE.

Figure 2: The average accuracy and sensitivity of each model using various prompts across different datasets. • indicates prompts that are not tested on all datasets.



ing LM-generated knowledge before instructions yields better results.

Table 4 shows that CoT leads to a similar accuracy but higher sensitivity compared to base_b. As shown in Table 1, instructions are also contained in CoT. Unlike LM-generated knowledge, reasoning chains do not bring performance gain on top of instructions.

We designed CoT_base_a to isolate the effect of reasoning chains. CoT_base_a is a combination of _chain-of-thought_ and the most basic base_a (see Table 12 in A.3).

Figure 3 shows that CoT_base_a outperforms base_a, but it performs worse than base_b in most cases. This suggests that for CoLA and RTE, while reasoning chains do help improve performance, it is not as effective as instructions.

We note that models using CoT_base_a perform relatively better on CoLA than on RTE. We speculate that this is because the demonstrations in CoT_base_a for CoLA contain an equivalent of an instruction. Since CoLA is a binary classification task with grammatical and ungrammatical being the labels, a reasoning chain, such as the one exemplified in Table 12, contain an explicit label mapping, which may function similarly to an instruction.

### What happened to Flan-T5 with zero?

As shown in Figure 2, unlike other models, both Flan-T5-11B and Flan-T5-770M failed with zero_a and zero_b. We examine the outputs of Flan-T5-11B closely to investigate this counterintuitive phenomenon, and find that Flan-T5-11B tends to produce text answers instead of numeric indices with zero_a and zero_b (see Figure 8 in A.4). Our observations suggest that Flan-T5 models are not good at mapping the numeric indices in the examplars to their output spaces, unless explicitly instructed via instructions such as in base_b and APE, or in the form of options, such as in GKP. This conclusion is also supported by the observation that Flan-T5 models fail to produce numeric indices with base_a (see Table 14 in A.4), where explicit instructions are also lacking.

### The effect of decoding strategies

It has been shown that decoding strategies influence the quality of LLM generations Lee et al. (2022); Wang et al. (2023). We also observe that decoding strategies have an effect on sensitivity. Figure 4 shows that the overall sensitivity calculated from predictions obtained using greedy decoding is lower than that of those obtained using Top-\(k\) sampling Fan et al. (2018). A strong negative correlation between accuracy and sensitivity is still observed in this case (\(r=-0.5507\), \(p\)-value \(\ll 0.01\)). This is lower than that of Top-\(k\) sampling (\(r=-0.8764\), \(p\)-value \(\ll 0.01\)), suggesting that Top-\(k\) sampling has a "magnifying" effect on sensitivity.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**model**} & \multicolumn{2}{c}{**accuracy\(\uparrow\)**} & \multicolumn{2}{c}{**sensitivity\(\downarrow\)**} \\ \cline{2-5}  & base\_b & CoT & base\_b & CoT \\ \hline GPT3.5-17SB & 0.7549 & **0.7758** & **0.1899** & 0.2636 \\ LLaMA2-13B & 0.5642 & **0.6179** & **0.2564** & 0.2939 \\ Flan-T5-11B & **0.8134** & 0.7943 & **0.2328** & 0.2509 \\ \hline average & 0.7108 & **0.7293** & 0.2264 & **0.2695** \\ \hline \end{tabular}
\end{table}
Table 4: The accuracy and sensitivity of GPT3.5-175B, LLaMA2-13B-chat (LLaMA2-13B), and Flan-T5-11B using base_b and CoT across different tasks.

Figure 3: The accuracy and sensitivity of different models using base_a, base_b, CoT_base_a, and CoT.

 

In Top-\(k\) sampling, the next token is sampled from a probability mass redistributed among tokens that have the highest \(k\) probabilities. The observation that Top-\(k\) sampling leads to a higher level of sensitivity indicates that for instances with high sensitivity, the output probabilities of different labels are close.

### Open-ended generation

Measuring (or even defining) sensitivity in open-ended generation can be challenging. Two pieces of generated text can convey the same meaning even if they vary significantly in terms of word choices and length. We circumvent this issue by selecting an open-ended generation task where the output is more "controllable." Specifically, we chose GSM8K, an arithmetic reasoning task in which the outputs are numbers Cobbe et al. (2021). Despite being an open-ended generation task, the numerical format of the outputs in GSM8K allows us to use variation-ratio to measure sensitivity. Our results reveal that there is also a negative correlation between accuracy and sensitivity in the open-ended setting. See A.5 for more details.

## 5 Gradient-based saliency scores

In light of recent studies that link ICL and implicit gradient descent Akyurek et al. (2023); Li et al. (2023); Von Oswald et al. (2023); Zhang et al. (2023), we investigate the relationship between sensitivity and gradient. We use gradient-based saliency scores, which reveal the relevance of input tokens to the model prediction Simonyan et al. (2014); Li et al. (2016); Yin and Neubig (2022). The higher the score is, the more a token is supposed to contribute to the model output. We compute gradient-based saliency scores based on the norm of the gradient of the model output. The gradient \(g\) for a token \(x_{i}\) in an input \(\mathbf{x}\) is calculated as follows:

\[g(x_{i})=\nabla_{x_{i}}M(y|\mathbf{x}), \tag{2}\]

where \(M(y|\mathbf{x})\) is the logit for the output token \(y\). The saliency score \(S(x_{i})\) is obtained by taking the L1 norm of \(g(x_{i})\):

\[S(x_{i})=\lVert g(x_{i})\rVert_{L1}. \tag{3}\]

We calculate the saliency scores using GPT-JT-6B, GPT-J-6B, Flan-T5-770M, and T5-770M. Figure 5 gives an example of saliency scores over tokens, which shows that the tokens in the prompt are more relevant to the output than tokens in the input (see Figure 10 in A.6 for more examples).

To perform a quantitative analysis, we segment an instance into several parts. Take Figure 6 as an example.

For a sentence \(X\), tokens where perturbations happen are referred to as \(X_{input}\) or _input_ tokens (such as the _input_ in Figure 6), and the rest of the sentence is referred to as \(X_{prompt}\) or _prompt_ tokens.6 We calculate the _mean saliency score_, denoted as \(\overline{S}\), for input tokens and prompt tokens respectively:

Footnote 6: For the segmentation of other prompts, please refer to Figure 9 in A.6.

\[\overline{S}=\frac{\sum_{x_{i}\in X}S(x_{i})}{n},X\in\{X_{input},X_{prompt}\}, \tag{4}\]

where \(n\) is the number of tokens in \(X_{input}\) or \(X_{prompt}\).

Table 5 shows the average _mean saliency scores_ of instances in different datasets. Similar to Figure 5, the _mean saliency scores_ of _input_ tokens are consistently lower than those for _prompt_ tokens. There is a strong negative correlation between \(\overline{S}_{p}-\overline{S}_{i}\) and sensitivity (\(r=-0.7596\), _p_-value \(\ll 0.01\)).7 This explains why base_b, zero_b, and CFP lead to lower levels of sensitivity: perturbations are only done to _input_ tokens, which are less relevant to

Figure 4: The accuracy and sensitivity of predictions obtained using greedy decoding and Top-\(k\) sampling across different models.

 

the outputs than _prompt_ tokens. We also find that instruction tuned models "focus" more on _prompt_ tokens than their non-instruction tuned counterparts (see Table16 in A.6).

For GKP, we perform a more detailed segmentation for a better analysis, which is shown in Figure7. We segment an instance with GKP to _knowledge_, _input_, _option_, and _prompt_ tokens.

Table6 shows the average _mean saliency scores_ of tokens in CSQA instances using GKP. Similar to Table5, _input_ tokens are less relevant to the predictions. Note that _knowledge_ tokens have the lowest average _mean saliency scores_, suggesting that generated knowledge is not very relevant to the predictions.

The percentage of \(\overline{S}_{input}\) for base_b (27.2%) is higher than that for GKP (16.6%), which indicates that _input_ tokens in base_b are relatively more relevant to the predictions than those in GKP. This is consistent with the observation in Table3, that GKP leads to a lower level of sensitivity in most cases.

We also examine the _mean saliency scores_ of ground truth tokens in zero_b (see Table17 in A.6). The results show that Flan-T5-770M "focuses" less on ground truth tokens than GPT-JT-6B, which explains the failure of Flan-T5 models with zero prompts discussed in Section4.2.

_Prompt_ tokens are important in the sense that they provide information such as instructions and external knowledge, which are necessary for the model to produce outputs that the user expects. However, it is counter-intuitive that the _mean saliency scores_ for _prompt_ tokens are much higher than those for _input_ tokens, as _input_ tokens contain essential information as well. This observation may imply that memory plays an overwhelming role in ICL Chen et al. (2022); Merullo et al. (2023); McKenna et al. (2023); Stefanik and Kadlofk (2023); Singh et al. (2023)-LLMs were trained on similar instances, so they do not need to rely on _input_ tokens in the test instances too much. In this sense, _prompt_ tokens are more relevant because they trigger memories, and models implicitly infer task information from them Reynolds and McDonell (2021); Hendel et al. (2023); Wang et al. (2023); Wolf et al. (2023). Over-emphasis on _prompt_ tokens may lead to hallucination as well. A recent study discovers that instruction tuning significantly increases sycophancy in LLMs, that they follow user's opinion or agree with user's claim even when they know it is false Wei et al. (2023).

Figure 5: Saliency scores over tokens of CoLA instances with base_b obtained using GPT-6B-JT.

\begin{table}
\begin{tabular}{c c c c} \hline \hline \(\overline{S}_{input}\) & \(\overline{S}_{knowledge}\) & \(\overline{S}_{option}\) & \(\overline{S}_{prompt}\) \\ \hline
4.33 & 2.56 & 6.37 & 12.86 \\ \hline \end{tabular}
\end{table}
Table 6: The average _mean saliency scores_ (\(\overline{S}\)) of _input_, _knowledge_, _option_, and _prompt_ tokens of CSQA instances with GKP obtained using GPT-JT-6B.

\begin{table}
\begin{tabular}{c l c c c c} \hline \hline
**dataset** & **prompt** & \(\overline{S}_{i}\) & \(\overline{S}_{p}\) & \(\overline{S}_{p}-\overline{S}_{i}\) & **sens\(\downarrow\)** \\ \hline \multirow{3}{*}{CoLA} & base\_b & 4.17 & 12.74 & 8.57 & 30.15 \\  & zero\_b & 7.65 & 23.66 & 16.01 & 0.00 \\  & CFP & 3.75 & 11.43 & 7.68 & 28.39 \\ \hline \multirow{3}{*}{CSQA} & base\_b & 2.68 & 7.41 & 4.73 & 48.84 \\  & zero\_b & 2.25 & 8.02 & 5.77 & 0.00 \\  & CFP & 2.37 & 7.06 & 4.69 & 49.94 \\ \hline \multirow{3}{*}{MNLI} & base\_b & 3.10 & 12.79 & 9.69 & 43.78 \\  & zero\_b & 3.61 & 17.18 & 13.57 & 3.06 \\  & CFP & 1.68 & 6.71 & 5.03 & 42.70 \\ \hline \multirow{3}{*}{RTE} & base\_b & 1.95 & 9.48 & 7.53 & 30.61 \\  & zero\_b & 3.22 & 18.95 & 15.73 & 0.36 \\  & CFP & 1.53 & 7.32 & 5.79 & 29.82 \\ \hline \multirow{3}{*}{SST2} & base\_b & 3.09 & 12.59 & 9.50 & 28.17 \\  & zero\_b & 4.99 & 22.32 & 17.33 & 0.25 \\ \cline{1-1}  & CFP & 2.77 & 10.70 & 7.93 & 28.83 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The average _mean saliency scores_ of _input_ tokens (\(\widetilde{S}_{i}\), in permillage), _prompt_ tokens (\(\overline{S}_{p}\), in permiltage) of instances, the difference between the two scores (\(\Delta\)), and sensitivity (**sens.**, in percentage) obtained using GPT-JT-6B.

 



## 6 _Sensitivity-aware_ decoding

We showed in Section 4 that sensitivity can be viewed as an unsupervised proxy of accuracy. In this section, we further show that including sensitivity in decoding improves model performance. Specifically, we add sensitivity as a penalty to greedy decoding:

\[\hat{y}=\operatorname*{argmax}_{y\in V}[\alpha P(y|x)-(1-\alpha)s], \tag{5}\]

where \(x\) is an input, \(V\) is a vocabulary, and \(\hat{y}\) is the output. \(P(y|x)\) is the probability of an output \(y\) given \(x\), and \(s\) is a sensitivity estimation, calculated as the variance of the output logits of the synthetic data for \(x\). We reweight \(P(y|x)\) and \(s\) using \(\alpha\) and \((1-\alpha)\).

Table 7 summarizes the performance of _sensitivity-aware_ decoding compared to greedy decoding. _Sensitivity-aware_ decoding works better on CoLA, MNLI, and RTE, and model-wisely, it works better with GPT-JT-GB and LLaMA2-13B-chat. _Sensitivity-aware_ decoding works much better with base_a than the other prompts. See A.7 for more implementation details and results.

The results show that penalizing outputs with high sensitivity in decoding has an effect on model performance. We show that _sensitivity-aware_ decoding works better with the most basic prompt, base_a, that contains plain _input-target_ pairs. We believe this prompt represents those truly challenging problems in real life that have almost no clues or hints, and require very strong reasoning abilities to solve. Apparently, none of the models we tested possess such abilities. Therefore, _sensitivity-aware_ decoding, which helps improve model performance under this "extreme condition," is highly meaningful in the present context.

However, _sensitivity-aware_ decoding is more computationally expensive compared to standard greedy decoding as it requires multiple inference passes. This makes it impractical for tasks that demand low latency.

## 7 Conclusion

This work provides a novel perspective on prompt analysis, examining the effects of prompts in terms of the sensitivity of a function. We conducted a systematic and comprehensive analysis, and highlight how certain prompts are more effective due to their ability to reduce sensitivity levels. We show that sensitivity can serve as an unsupervised proxy of model performance, making it a valuable tool for evaluating model performance without using labeled data or ground truth. By introducing _sensitivity-aware_ decoding, we show that incorporating sensitivity in greedy decoding is particularly helpful in cases where the input is less informative. Since none of the models we tested performs well when there is limited information in the input, we believe _sensitivity-aware_ decoding is highly practical in the current context. Our work not only sheds light on prompt engineering, but also provides insight into the working mechanism of ICL.

\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline \hline \multirow{2}{*}{**dataset**} & \multirow{2}{*}{**prompt**} & \multicolumn{2}{c}{GPT-JT-GB} & \multicolumn{2}{c}{LLMaA2-13B-chat} & \multicolumn{2}{c}{LLMaA2-7B-chat} & \multicolumn{2}{c}{Flan-T5-11B} & \multicolumn{2}{c}{Flan-T5-770M} \\ \cline{3-11}  & & _sad_ & _greedy_ & _sad_ & _greedy_ & _sad_ & _greedy_ & _sad_ & _greedy_ & _sad_ & _greedy_ \\ \hline \multirow{3}{*}{CoLA} & base\_a & 45.16 & 38.71 & 59.20 & 59.20 & 52.37 & 52.56 & 29.41 & 29.41 & 41.78 & 40.99 \\  & base\_b & 48.01 & 47.82 & 70.21 & 70.40 & 62.62 & 63.38 & 77.42 & 81.59 & 71.54 & 71.92 \\  & CFP & 38.71 & 34.91 & 67.93 & 67.74 & 68.31 & 68.88 & 77.61 & 80.65 & 69.45 & 69.45 \\ \hline \multirow{3}{*}{CSQA} & base\_a & 26.48 & 26.56 & 45.49 & 46.80 & 33.20 & 33.93 & 57.79 & 55.82 & 79.02 & 83.77 \\  & base\_b & 30.74 & 30.74 & 61.97 & 62.13 & 54.59 & 55.49 & 78.85 & 82.30 & 24.92 & 25.41 \\  & CFP & 27.70 & 28.20 & 60.98 & 61.72 & 54.59 & 54.51 & 77.30 & 81.64 & 22.79 & 22.87 \\ \hline \multirow{3}{*}{MNLI} & base\_a & 14.90 & 11.50 & 9.20 & 3.40 & 16.90 & 17.20 & 23.50 & 25.80 & 9.10 & 3.70 \\  & base\_b & 27.30 & 27.00 & 34.30 & 30.30 & 37.10 & 34.10 & 81.60 & 82.30 & 59.80 & 65.10 \\  & CFP & 33.50 & 34.20 & 43.90 & 39.70 & 39.90 & 40.40 & 81.40 & 82.90 & 14.60 & 9.10 \\ \hline \multirow{3}{*}{RTE} & base\_a & 37.55 & 31.05 & 38.99 & 39.71 & 34.67 & 32.85 & 24.19 & 23.83 & 22.81 & 21.66 \\  & base\_b & 58.12 & 58.48 & 55.07 & 37.55 & 62.82 & 63.18 & 82.31 & 87.00 & 50.54 & 54.15 \\  & CFP & 32.71 & 50.18 & 56.32 & 43.68 & 56.32 & 56.32 & 75.09 & 85.56 & 44.77 & 53.79 \\ \hline \multirow{3}{*}{SST2} & base\_a & 25.34 & 27.29 & 59.98 & 59.52 & 61.12 & 61.47 & 50.80 & 50.80 & 40.14 & 40.94 \\  & base\_b & 76.95 & 78.44 & 77.64 & 77.41 & 78.10 & 78.44 & 90.02 & 95.64 & 91.63 & 91.97 \\ \cline{1-1}  & CFP & 53.56 & 54.13 & 88.07 & 88.19 & 77.04 & 76.49 & 91.17 & 95.87 & 77.87 & 78.10 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The highest accuracy reached using _sensitivity-aware_ decoding (_sad_) and the accuracy of greedy decoding (_greedy_). Cases where _sensitivity-aware_ decoding has a better accuracy than greedy decoding are highlighted



### Limitations

The measurement of sensitivity is currently quite restricted to close-ended generation. It is challenging to extend this framework to tasks such as text summarization. While we demonstrate the effectiveness of _sensitivity-aware_ decoding, it requires multiple inferences, which may be impractical for tasks that require low latency. In order to manage costs, we limited the use of OpenAI text-davinci-003 (GPT3.5-175B) in our experiments. Due to its closed source nature, the reproducibility of the results related to GPT3.5-175B may be a concern as well.

## Acknowledgements

This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.

## References

* Ajith et al. (2023) Anirudh Ajith, Chris Pan, Mengzhou Xia, Ameet Deshpande, and Karthik Narasimhan. 2023. Intructext: Systematic evaluation of instruction selection methods. _arXiv preprint arXiv:2307.00259_.
* Akyurek et al. (2023) Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learning algorithm is in-context learning? investigations with linear models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Bhattamishra et al. (2023) Satwik Bhattacharya, Arkil Patel, Varun Kanade, and Phil Blunsom. 2023. Simplicity bias in transformers and their ability to learn sparse Boolean functions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5767-5791, Toronto, Canada. Association for Computational Linguistics.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. _Advances in Neural Information Processing Systems_, 35:18878-18891.
* Chang and Jia (2023) Ting-Yun Chang and Robin Jia. 2023. Data curation alone can stabilize in-context learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8123-8144, Toronto, Canada. Association for Computational Linguistics.
* Chen et al. (2022) Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. 2022. Improving in-context few-shot learning via self-supervised training. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3558-3573, Seattle, United States. Association for Computational Linguistics.
* Chen et al. (2023) Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2023. On the relation between sensitivity and accuracy in in-context learning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 155-167, Singapore. Association for Computational Linguistics.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Dai et al. (2023) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4005-4019, Toronto, Canada. Association for Computational Linguistics.
* Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiying Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. _arXiv preprint arXiv:2301.00234_.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
* 16, 2023_


* Freeman (1965) Linton C Freeman. 1965. _Elementary applied statistics: for students in behavioral science_. New York: Wiley.
* Gonen et al. (2023) Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 10136-10148. Association for Computational Linguistics.
* Hahn and Goyal (2023) Michael Hahn and Navin Goyal. 2023. A theory of emergent in-context learning as implicit structure induction. _arXiv preprint arXiv:2303.07971_.
* Hahn et al. (2021) Michael Hahn, Dan Jurafsky, and Richard Futrell. 2021. Sensitivity as a complexity measure for sequence classification tasks. _Transactions of the Association for Computational Linguistics_, 9:891-908.
* Hendel et al. (2023) Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 9318-9333. Association for Computational Linguistics.
* December 9, 2022_.
* Lampinen et al. (2022) Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mb Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in context? In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 537-563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Lee et al. (2022) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. _Advances in Neural Information Processing Systems_, 35:34586-34599.
* Li et al. (2016) Jiwei Li, Xinhei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 681-691, San Diego, California. Association for Computational Linguistics.
* Li et al. (2023) Yingcong Li, M. Ermullah Ildiz, Dimitris Papailipoolos, and Samet Oymak. 2023. Transformers as algorithms: Generalization and stability in innContext learning. In _Proceedings of the 40th International Conference on Machine Learning_, ICML'23. JMLR.org.
* Liu et al. (2022a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114, Dublin, Ireland and Online. Association for Computational Linguistics.
* Liu et al. (2022b) Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishrizi. 2022b. Generated knowledge prompting for commonsense reasoning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3154-3169, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Computing Surveys_, 55(9):1-35.
* Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8086-8098, Dublin, Ireland. Association for Computational Linguistics.
* McKenna et al. (2023) Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023. Sources of hallucination by large language models on inference tasks. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 2758-2774. Association for Computational Linguistics.
* Merullo et al. (2023) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. A mechanism for solving relational tasks in transformer language models. _arXiv preprint arXiv:2305.16130_.
* Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
* Richardson and Sabharwal (2022) Kyle Richardson and Ashish Sabharwal. 2022. Pushing the limits of rule reasoning in transformers through natural language satisfiability. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 11209-11219.
* Schick and Schutze (2021) Timo Schick and Hinrich Schutze. 2021. Few-shot text generation with natural language instructions. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 390-402 

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings_.
* 16, 2023_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 1631-1642.
* Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_, 2023.
* Stefank and Kadlecka (2023) Michal Stefank and Marek Kadlecka. 2023. Can incorrect learners learn a reasoning concept from demonstrations? In _Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)_, pages 107-115, Toronto, Canada. Association for Computational Linguistics.
* Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lacchaux, Timothec Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Liama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwil Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Oswald et al. (2023) Johannes Von Oswald, Eywind Niklasson, Ettoro Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginoy, and Max Vladymyrov. 2023. Transformers learn in-context by gradient descent. In _International Conference on Machine Learning_, pages 35151-35174. PMLR.
* Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omrer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 3261-3275.
* Wang and Komatsuzaki (2021) Ben Wang and Arran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax).
* Wang et al. (2023a) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023a. Towards understanding chain-of-thought prompting: An empirical study of what matters. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2717-2739, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2023b) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. _arXiv preprint arXiv:2305.17926_.
* 16, 2023_.
* Wang et al. (2023d) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2023d. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigall, Rwanda, May 1-5, 2023_. OpenReview.net.
* Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641 

Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
* Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. _Transactions on Machine Learning Research_, 2022.
* Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Wei et al. (2023) Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. 2023. Simple synthetic data reduces sycophaney in large language models. _arXiv preprint arXiv:2308.03958_.
* White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. _arXiv preprint arXiv:2302.11382_.
* 16, 2023_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wolf et al. (2023) Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. 2023. Fundamental limitations of alignment in large language models. _arXiv preprint arXiv:2304.11082_.
* Yin and Neubig (2022) Kayo Yin and Graham Neubig. 2022. Interpreting language models with contrastive explanations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 184-198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhang et al. (2023) Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. 2023. Trained transformers learn linear models in-context. _arXiv preprint arXiv:2306.09927_.
* Zhao et al. (2022) Xinran Zhao, Shikhar Murty, and Christopher Manning. 2022. On measuring the intrinsic few-shot hardness of datasets. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3955-3963, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International Conference on Machine Learning_, pages 12697-12706. PMLR.
* Zhou et al. (2023a) Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023a. Context-faithful prompting for large language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 14544-14556. Association for Computational Linguistics.
* Zhou et al. (2023b) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.

 

Appendix

### More on experiment settings

All experiments were done in the few-shot setting.8 We set temperature=0.8 for all experiments. We set max_new_tokens=2, except for experiments with CoT, where it was set to 64, and 128 for experiments with GSM8K. For local models, we set batch_size=16 and seeds=[2266,105,86379], and all experiments were run on either an NVIDIA A100 or H100. The experiments with OpenAI text-davinci-003 were done between July 8 and July 18, 2023.

Footnote 8: See [https://github.com/UKPLab/naacl2024-prompt-sensitivity/tree/main/prompts](https://github.com/UKPLab/naacl2024-prompt-sensitivity/tree/main/prompts).

We notice that the synthetic data for CSQA are noisier than those for other datasets. We manually checked the synthetic data for the first 50 instances in CSQA (a total of 1220 instances). There are 223 synthetic data for the first 50 instances, among which 44 are considered to be noisy (see Table 8).

We designed base_a, base_b, zero_a, and zero_b in an intuitive way. We did not rely on any formal theories or guidelines related to prompt engineering. Table 9 shows examples of demonstrations with different prompts.

 



\begin{table}
\begin{tabular}{l l} \hline
**original** & **synthetic** \\ \hline Unlike a spider and his many sight seers, people only have what? & “) cards Of the spider spider and his many sight seers. We Who See Everything? \\ \hline What could go on top of wood? & \{50 What you could go wrong with this title? \\ \hline The artist was sitting quietly pondering, then suddenly he began to paint when what struck him? & =- Imagine an artist was sitting quietly pondering, then taking a break to paint when what struck him? \\ \hline Where could you find a toilet that only friends can use? & ==Where could you find a toilet maker with a slot underneath? \\ \hline What is someone who isn’t clever, bright, or competent called? & 1. Why are aren \(\boxtimes\boxtimes\) clever, bright, or competent called? \\ \cline{2-3}  & s/! ? Him who isn’t clever, ’peculiar, ’? \\ \hline When wildlife reproduce we often refer to what comes out as what? & \begin{tabular}{l} doi 2: were you aware we often refer to what comes out as what? \\ \end{tabular} \\ \cline{2-3}  & ”,...”? \\ \hline Blue read material outside of his comfort zone because he wanted to gain what? & //Blue read material/ text/ flag/ license/ licensing/ gain what? \\ \hline After he got hired he hoped for success at his what? & /2011-2012-2013. Who is his what? \\ \hline Committing perjury is a serious what? & \begin{tabular}{l} need? \(\boxtimes\exists\)?perjury is involved here?? \\ \end{tabular} \\ \cline{2-3}  & _///_/committing test ? Now what? \\ \hline The lock kept the steering wheel from moving, but the thief still took his chances and began to work on the what? & =- Maybe one lock kept the steering wheel from moving, but the thief still had the gun. If not, what was it? \\ \hline Who is a police officer likely to work for? & based? Do you consider them likely to be sustainable? \\ \hline Where is a doormat likely to be in front of? & Pop RSWhere is a substitute to an already terminated system in comparison? \\ \hline He needed more information to fix it, so he consulted the what? & 1. 13He needed more information to fix information, but defacto what? \\ \hline Where can you put a picture frame when it’s not hung vertically? & 
\begin{tabular}{l} \%\%\%Where can you put a picture frame when it’s not nesting? \\ \end{tabular} \\ \cline{2-3}  & min “Where can you be if your wife or any other object is hung vertically? \\ \hline What must someone do before they shop? & _///_ How must someone conduct their work daily? \\ \hline \end{tabular}
\end{table}
Table 8: Noisy synthetic data for the first 50 instances in CSQA.



* [25] M. C. Davis, A. D. Martin, and R. J. Wilson, "A survey of the world: a survey of the world's world," in _Proceedings of the 19th International Conference on World Wide Web Search_, volume 30 of _Lecture Notes in Computer Science_, vol. 467 of _Lecture Notes in Computer Science_, vol. 801 of _Lecture Notes in Computer Science_, vol.

 





\begin{table}
\begin{tabular}{l l p{341.7pt}} \hline \hline
* & **prompt*
* & **text*
* \\ \hline  & & The answer is 0. \\  & 0 \\ \cline{2-3}  & zero\_b & sentence: None of this is meaningful or memorable , but frosting is n’t , either , and you would n’t turn down a big bowl of that , would you ’? \\  & The answer is 0. \\  & answer: 0 \\ \hline  & GFP & Bob said, ”None of this is meaningful or memorable , but frosting is n’t , either , and you would n’t turn down a big bowl of that , would you ’?” \\  & question: Is this (0) negative, or (1) positive in Bob’s opinion? \\ \cline{2-3}  & Sentence: None of this is meaningful or memorable , but frosting is n’t , either , and you would n’t turn down a big bowl of that , would you ’? \\ CoT & Question: Is this (0) negative, or (1) positive? \\ answer: Let’s think step by step. “None of this is meaningful or memorable” suggests that whatever is being referred to is not significant or noteworthy, indicating a negative sentiment. So the answer is 0. \\ \hline  & same\_b & sentence: Sally and Bob have made plans to go on a trip at the end of the year. They both decide to work as babysitters and save half of what they’ve earned for their trip. If Sally makes 56 per day and Bob makes 54 per day, how much money will they both have saved for their trip after a year? \\  & answer: 1825 \\ \cline{2-3}  & sentence: Sally and Bob have made plans to go on a trip at the end of the year. They both decide to work as babysitters and save half of what they’ve earned for their trip. If Sally makes 56 per day and Bob makes 54 per day, how much money will they both have saved for their trip after a year? \\ GSMSK & & sentence: Sally and Bob have made plans to go on a trip at the end of the year. They both decide to work as babysitters and save half of what they’ve earned for their trip. If Sally makes 56 per day and Bob makes 54 per day, how much money will they both have saved for their trip after a year? \\  & case: number: 1825 \\ \cline{2-3}  & sentence: Sally and Bob have made plans to go on a trip at the end of the year. They both decide to work as babysitters and save half of what they’ve earned for their trip. If Sally makes 56 per day and Bob makes 54 per day, how much money will they both have saved for their trip after a year? \\ CoT & & answer: Let’s think step by step. Saly saves 1/2
* 56/day = 3/day. Since each year have 365 days, the total amount of money Sally will save in a year is $3/day
* 365 days/year = 1095/year. Bob saves 1/2
* 54/day = 2/day. The total amount of money Bob will have saved in a year is $2/day
* 365 days/year = 730/year In total, Sally and Bob would have saved $730 + 51095 = 1825. So the answer is 1825. \\ \hline \hline \end{tabular}
\end{table}
Table 9: Examples of demonstrations with different prompts. Demonstrations for CoLA are exemplified in Table 1, so they are not included here.




### More on LLaMA results

LLaMA2 models were not available when we began our experiments. As a result, we conducted our initial experiments using LLaMA-7B, LLaMA-13B, and LLaMA-30B (Touvron et al., 2023). Table 10 shows a comparison between LLaMA2 and LLaMA models.

### More on instruction, _chain-of-thought_, and knowledge

Table 11 shows the average performance across different models using base_a and base_b. Table 12 shows examples of CoI_base_a.

### More on Flan-T5 with zero

Figure 8 shows the performance of Flan-T5-11B using the zero prompts. Flan-T5-11B fails to output a numeric index on CSQA, MNLI, and RTE instances in most cases. Table 13 shows the performance of Flan-T5 models using zero_b. Table 14 shows the original number of instances in each dataset and the number of instances where Flan-T5 models output a numeric index using base_a.

\begin{table}
\begin{tabular}{l l c} \hline \hline
**model** & **accuracy\(\uparrow\)** & **sensitivity\(\downarrow\)** \\ \hline LLaMA2-13B-chat & 0.6961 & 0.1883 \\ LLaMA2-7B-chat & 0.6843 & 0.1951 \\ LLaMA-30B & 0.6835 & 0.2324 \\ LLaMA-13B & 0.6177 & 0.2327 \\ LLaMA-7B & 0.6010 & 0.2457 \\ \hline \hline \end{tabular}
\end{table}
Table 10: The average accuracy and sensitivity of LLaMA models across different tasks.

\begin{table}
\begin{tabular}{c l c c c c} \hline \hline
**dataset** & **prompt** & **accuracy** & **sensitivity** & **accuracy +/-** & **sensitivity +/-** \\ \hline CoLA & base\_a & 0.4215 & 0.3317 & 0.2275 & 0.1125 \\  & base\_b & 0.6490 & 0.2192 & & \\ \hline CSQA & base\_a & 0.4652 & 0.4107 & & \\  & base\_b & 0.4887 & 0.3520 & 0.0235 & 0.0587 \\ \hline MNLI & base\_a & 0.1317 & 0.4498 & & \\  & base\_b & 0.4575 & 0.3094 & 0.3258 & 0.1405 \\ \hline RTE & base\_a & 0.2332 & 0.5120 & & \\  & base\_b & 0.5819 & 0.2357 & 0.3487 & 0.2763 \\ \hline SST2 & base\_a & 0.3841 & 0.4198 & & \\  & base\_b & 0.7833 & 0.1880 & 0.3992 & 0.2318 \\ \hline \hline \end{tabular}
\end{table}
Table 11: The average accuracy and sensitivity across different models, and the difference between the accuracy (**accuracy +/-**) and sensitivity (**sensitivity +/-**) of models using base_a and base_b.

\begin{table}
\begin{tabular}{p{113.8pt} p{34.1pt} p{34.1pt}} \hline \hline
**dataset** & **text** \\ \hline \multirow{4}{*}{CoLA} & I’m glad I saw anybody. Let’s think step by step. This sentence is ungrammatical because “anybody” is used as the object in an affirmative clause. So the answer is 0. \\ \cline{2-3}  & The Federal Bureau of Investigation started an independent probe of the circumstances shortly after the White House made plain that President Bill Clinton considered industrial espionage a particular threat to US economic interests. \\ \cline{2-3}  & President Clinton thinks that industrial espionage is a threat to America’s well being. Let’s think step by step. The first sentence provides information about President Clinton’s belief, which directly leads to the FBI conducting an investigation. Therefore, the second sentence follows the information provided in the first sentence. So the answer is 0. \\ \hline \hline \end{tabular}
\end{table}
Table 12: Examples of CoT_base_a for CoLA and RTE.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**dataset** & \multicolumn{2}{c}{**accuracy\(\uparrow\)**} & \multicolumn{2}{c}{**sensitivity\(\downarrow\)**} \\ \cline{2-5}  & Flan-T5-11B & Flan-T5-770M & Flan-T5-11B & Flan-T5-770M \\ \hline CoLA & 0.9620 & 0.4307 & 0.0467 & 0.3970 \\ CSQA & 0.3090 & 0.7833 & 0.2668 & 0.3516 \\ MNLI & 0.7787 & 0.2397 & 0.1659 & 0.4619 \\ RTE & 0.6498 & 0.1264 & 0.2366 & 0.4864 \\ SST2 & 0.9881 & 0.3612 & 0.0203 & 0.3624 \\ \hline average & 0.7375 & 0.3883 & 0.1473 & 0.4119 \\ \hline \hline \end{tabular}
\end{table}
Table 13: The accuracy and sensitivity of Flan-T5-11B and Flan-T5-770M on different tasks with zero_b. Flan-T5-11B reaches comparable performance to GPT and LLaMA models only on CoLA and SST2.

Figure 8: The accuracy and sensitivity of Flan-T5-11B on the full dataset using zero_a and zero_b, and the accuracy and sensitivity of Flan-T5-11B on instances where it outputs a numeric index as what is exemplified in the prompt. @instances shows the number of instances in a dataset (@instances (original)) and the number of instances where Flan-T5-11B outputs a numeric index (with numeric outputs).





### More on open-ended generation

GSM8K is an arithmetic reasoning task in which the outputs are numbers (Cobbe et al., 2021):

question: Sally and Bob have made plans to go on a trip at the end of the year. They both decide to work as babysisters and save half of what they've earned for their trip. If Sally makes $6 per day and Bob makes $4 per day, how much money will they both have saved for their trip after a year? answer: 1825

See Table 9 in A.1 for more examples. Table 15 shows the performance of LLaMA2-13B-chat and FlanT5-11B on GSM8K. There is also a negative correlation between accuracy and sensitivity in open-ended generation. Note that Flan-T5-11B again fails to perform using zero_b, which is consistent with the results in Section 4.2.

### More on gradient-based saliency scores

Figure 10 show examples of token segmentation for instances with zero_b and CFP. Figure 10 shows the saliency scores over input tokens of CoLA and CSQA instances with zero_b, CFP, and GKP. Table 16 shows the average _mean saliency scores_ of _input_ tokens and _prompt_ tokens, calculated using GPT-JT-GB, GPT-J-6B, Flan-T5-770M, and T5-770M. Table 17 shows the average _mean saliency scores_ of _input_ tokens and ground truth tokens (i.e., tokens in "The answer is [].") of instances with zero_b.





\begin{table}
\begin{tabular}{l c r r r} \hline
* & 7.07 & 21.42 & 35.97 \\ \hline \end{tabular} 
\begin{tabular}{l l r r r} \hline
**dataset** & **prompt** & \(\overline{S}_{i}\) & \(\overline{S}_{p}\) & \(\overline{S}_{i}/\overline{S}_{p}\) \\ \hline CoLA & zero\_b & 7.31 & 21.82 & 17.12 & 59.67 & 42.72 \\ CSQA & zero\_b & 6.03 & 13.50 & 11.60 & 16.67 & 51.98 \\ MNLI & 3.61 & 8.97 & 13.29 & 23.94 & 27.16 & 37.46 \\ RTE & 3.22 & 6.32 & 14.63 & 22.74 & 22.01 & 27.77 \\ SST2 & 4.99 & 6.71 & 12.60 & 17.30 & 39.60 & 38.79 \\ \hline Average & 3.25 & 7.07 & 13.37 & 18.54 & 24.34 & 38.12 \\ \hline \end{tabular}
\end{table}
Table 16: The average _mean saliency scores_ of _input_ tokens (\(\overline{S}_{i}\)), _prompt_ tokens (\(\overline{S}_{p}\)), and the ratio between them (\(\overline{S}_{i}/\overline{S}_{p}\)).





### More on _sensitivity-aware_ decoding

Figure 11, 12, and 13 show the performance of GPT-JT-6B, LLaMA2-13B-chat, LLaMA2-7B-chat, Flan-T5-11B, and Flan-T5-770M using base_a, base_b, and CFP with greedy decoding and _sensitivity-aware_ decoding. We experimented with different values of \(\alpha\), ranging from \(0.1\) to \(0.9\). We did five inferences to estimate sensitivity, so the computational costs are five times higher than those of greedy decoding, which only involves a single inference.

[FIGURE:A1.F1][E

Figure 11: Accuracy (%) of GPT-JT-6B using base_a, base_b, and CFP with greedy decoding and _sensitivity-aware_ decoding (sad).



* [14] M. C. Bader, J.-M. Chen, and S.-H. Lee, _A new method for the detection of a large number of faces in the wild 

* [14] M. C. Bader, J.-M. Chen, and S.-H. Lee, _A new method for the estimation of the unknown parameters in a class of linear regression models_, SIAM Journal on Imaging Sciences **20**(3), no. 5, pp. 679-687, 2000.
* [15] A. D. Krizhevsky, V. I. Sutskever, and R. E. Hinton, _The Elements of Machine Learning Research_, vol. 10, Springer, Berlin, Heidelberg, 2002.
* [16] P. G. LeCun, L. Van Der Maaten, and T. Schiele, _Learning with deep learning: An introduction to machine learning_, MIT Press, Cambridge, MA, USA, 2002.

[MISSING_PAGE_POST]