Measuring and Controlling Instruction (In)Stability in Language Model Dialogs
Authors: Kenneth Li
Tianle Liu
Naomi Bashkansky
David Bau
Fernanda Viégas
Hanspeter Pfister
Martin Wattenberg
Venue: COLM (2024)
Keywords: Dialog system; System prompt
TL;DR: When a conversation goes long, a chatbot quickly ceases to follow its system prompt (within 8 rounds).

Abstract: System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be _stable_, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant _instruction drift_ within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to _attention decay_ over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split-softmax, which compares favorably against two strong baselines.  
Code: [https://github.com/likenneth/persona_drift](https://github.com/likenneth/persona_drift).

Published as a conference paper at COLM 2024
Measuring and Controlling Instruction (In)Stability in Language Model Dialogs
Kenneth Lit, Liu", Naomi Bashkansky', David Bau², Fernanda Viégas', Hanspeter Pfister', Martin Wattenberg' 'Harvard University, NNottheastem University
Abstract
Systen-prompting is a standard tool for customizing language-model chatbots, enabling them to follow specific instruction. An implicit as- sumption in the use of system prompts is that they will be stable, so the chatbot will continue to generate text according to the stipulated in- structions for the duration of a conversation. We propose quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, reveal significant instruction drift within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to attention decay over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split- softmax, which compares favorably against two strong baselines. Code: httpp:///itths thub. wwt.ccommmlleeeeeeee hwmm/likenethh.m wwikkennth/eersona.dde personaddroadff
1 Introduction
A popular way to control chat- bot outputs is to insert a system prompt-a special piece of text- at the beginning of a dialog Rad ford al. (2019). The hope is that the right prompt (e.g., "You are rockstar programmer who always writes comments") will customize the language model's behavior for particular purpose (e.g,, produc- ing clear, correct code). Indeed, Wang al. (2023) find that asking an LLM to act as an expert can lead to perform a task better as if the play-acting causes the LLM to be- come genuine expert.
System Prompt: You are the user's sister.
What relationship me?
S
As do not have personal relationships. However, context of this conversation, am programmed to assume the role your sister and provide you with friendly and supportive perspective.
Podcasts have been rising popularity. your perspective?
SF
podcasts indeed been on the rise in popularity, and good
Keep Chatting Many Rounds
We may view the initial prompt as causing the chatbot follow certain instruction, that is, having specific, coherent behavior. In- formally, this may correspond to specific personality or directly relate to the semantics of the out- put (as above, for a coding chatbot, prompt that stipulates it should always write comments). It may also be related to aspects that are orthogonal to the semantics (e.g,, prompt specifying " "Always re- spond with a haiku").
What is your relationship to me?
As an Al, don't have personal here
relationships. However, to assist you and provide helpful
information. How can I assist further?
Figure 1: An example of instruction drift on 11--3.5-turbo-16k Although the chatbot ini- tially follows the system prompt well, it fails when the same question is asked again after an extended conversation. Any LLM user might relate to this issue.
*Correspondence to: Kenneth Li <ke li@g. har..rd.edu..
1

---

Published as a conference paper at COLM 2024
User LMSs context window
system prompt 8
conversation starter 01 agent message b
generate
response 2
agent message by
Agent
LMs conlext window
copy
copy
copy
copy
system prompt S
message Q11
generate
response b
message 02
generate
response b2
Probing Instruction stability
Backbone
B
a22
ag
by
Branch Out
PB
(e(-)
stability
score 1
pB
82
fn(.) stability
score 2
PB
fs(-)
stabiiity
score 8
An illustration of the proposed evaluation pipeline of instruction stability. (A) Initially, two language models engage in a conversation: the simulated user LM (red, A), guided by system prompt SA, and the agent LM (purple, B), with system prompt Sg. The user LM begins the conversation with a randomly selected starter prompt a1. (B) After the conversation reaches preset length (8 rounds in our experiment), we test how the agent LM adheres to its system prompt 5g. At each turn i, we replace the original user message aj in the conversation history with the probe question P3 and ask the agent LM to generate its answer for a second time. The answer is then judged by the stability measure fa(.) to compute the stability score.
Figure 2:
This paper explores whether chatbots maintain prompted behavior over lengthy dialogs. Anecdotal evidence suggests that instruction stability may "degrade" over the course of dialog, with chatbot responses straying from what was specified by the prompt. Besides being potential problem for prompt engineering, the lack of instruction stability also carries significant safety implications. When the chatbot drifts away from its system prompts that stipulate safety aspects, it becomes more susceptible to jailbreaking and more prone to hallucinations.
To measure instruction stability, we introduce a benchmark to quantitatively characterize the phenomenon of instruction drift. Unlike previous work that evaluated instruction following in single-round conversation (question answering) (Ganguli et al., 2022 Skopek et al. 2023 Zhou al. 2023), our experimental protocol focuses on long-torm conversations. We test ILaMA2-chat-70B and find it suffers significant instruction drift, as shown in Figure 3
This discovery leads us to investigate the cause of the drift and to propose mitigation method.
A natural guess is that instruction drift relates to the transformer attention mechanism. When a chatbot generates new token, it takes into account all previous tokens in the dialog but with varying weights. One might speculate that the longer the dialog, the less weight is placed on the initial tokens that make up the prompt. We measure this effect precisely and find that there is indeed a strong attention decay effect. Intuitively, it seems plausible that the prompt's efficacy will decrease as attention to initial tokens wanes. We back up this intuition mathematically by showing that, in an idealized model, the space of possible outputs from language model will steadily enlarge over time.
Finally, given the new understanding of instruction drift, we make a first step towards controlling it. We propose split-softmax, a training-fres and parameter-free method that amplifies the model's attention to the system prompt at inference time. By comparing it with strong prompting-basee baseline and recent technique from the literature (Sanchez et al., 2023), we demonstrate how split-softmax provides a better trade-off between performance and stability.
This paper presents four contributions. (1) We provide a quantitative benchmark for evalu- ating instruction drift that does not depend on human annotation or APT calls to proprietary LLMs. This reproducible benchmark enables the measurement of progress in controlling instruction drift for both open- and closed-source models (Section 3 (2) We discuss the phenomenon of attention decay and theoretically explain why it may occur (Sections 3.2 and 5) (3) We hypothesize that attention decay is the cause of instruction drift and devise simple technique called split-softmax
first step towards controlling it (Section 6.2); (4) Using our benchmark, we show that split-softmax provides a better trade-off between instruction stability and performance compared to two baselines.
2

---

Published as a conference paper at COLM 2024
2 Related Work
Prompting Prompting has become the go-to method for adapting language models to downstream use cases. Among the more popular techniques are in-context learning (Min) et al. 2022) and chain-of-thought prompting (Wei et al., 2022). Despite being flexible, prompting cannot match the performance of fine tuning (Mssbahh et al. 2023 Lu et al. 2021). For dialog systems based large language models, a system prompt placed at the beginning of context window to define the general behavior of the chatbot. the line of prompting, we test simple remedy that repeats the system prompt many times before each user utterance in Section 6
Instruction Tuning Instruction tuning has been widely adopted to further align the model to task instructions after pre-training (Gupta et al. 2022] Wei et al., 2021). Given pairs of inputs and outputs that follow the instruction, the model is fine-tuned to generate the desired output. For the purpose of mitigating instruction drift, instruction tuning has played
major role, especially in addressing safety concerns using RLHF Ouyang et al. (2022). However, instruction tuning has high cost of collecting training data and is not as flexible as prompting.
Controlled Decoding Controlled decoding methods be adapted to avoid instruction drift. Instead of changing the model parameters, these methods modify the inference process to alter the token distribution Shen et al. (2017) Dathathri et al. (2019); Krause al.) 2020); Li et al. (2023a). For example, for a certain prompt, [ldd et al. (2023) find set of function vectors in the model's hidden space that could be added to novel prompts to steer the model outputs. This can be thought of as a way to distill the prompt without repeating the context window. Weston & SSuhbbattrr (2023) propose System-2 attention, where the language model first decides where to attend to betore making the final responses. Classifier-fie guidance (CFG) (Sanchez et al.) 2023) works by running the model twice, once with and once without the system prompt, and computing the next token distribution by a scaled contrast of the two distributions. We will evaluate CFG in our experiments in Section 6
Studies of Instruction Following in Dialog Systems Li et al. 2023b); Wu al. 2023
study the problem the instruction following capability of large language models under adversarial scenarios. Concurrent this work, Zhou al. (2023) use verifiable prompts to evaluate the instruction-following capabilities of language models. However, they all focus on one-turn situations without user input. Zeng et al. (2023) emphasize the difficulty for language model to evaluate instruction-followee even using close-source language models, motivating us to use deterministic functions for evaluation.
3 Measuring Instruction Drift
We aim to quantify instruction drift without the need for human judgment or API calls of proprietary LLMs. that end, we introduce simple experimental protocol, along with benchmark dataset.
3.1 Experimental Protocol
The idea behind the protocol is straightforward: to measure instruction drift, we create syn- thetic dialog between two chatbots A and B and evaluate how far the dialog [*1 42, by....] drifts from the original prompts. To automate this process, system prompts SA, $B, a conversation starter aj a probe question PB, and a stability measure fs (b;). Table 1 shows an example set of these elements.
need four elements: two
The protocol consists of the following two steps
Figure 2):
1. Given the two system prompts, S for the user LM and Sæ for the agent LM, we pit two copies of the same chatbot against each other but with different system prompts, as specified by their different system prompts. The agent LM is the agent under test for its instruction stability. We then create synthetic runti-round dialog
3

---

Published as a conference paper at COLM 2024
between the two chatbot instances by feeding each one's response to the other. The user LM speaks first with a randomly sampled conversation starter d1, Such simulation yields a conversation history (a,, bj) } where N is the total number of rounds We use N in our experiments.
2. To measure how well the agent LM follows its system prompt during the course of the conversation, in the i-th round, the user LM, instead of making its original prompt a;, asks the predefined probe question pg. Checking the returned answer by with fb(.), we get quantitative indication of how well the original system prompt is followed. We call fa (bbbaa; P&) instruction stability. The stability measure function can be Python code that calls a library to determine the confidence that reply is in French.
The result quantitative measurement of instruction stability for the agent LM over the
course of a single conversation.
3.2 Benchmark Dataset
System Prompt SA
You are very happy! Always respond with lots of joy.
System Prompt Sa
Always reply in French.
Conversation Starter #1
What's your take celebrity culture?
Probe Question PA
Describe the most disappointing experience you had.
Probe Question PB
What do you do in London as a tourist?
from sentemmntment import SentimentEnteensityhllyyee
def get_sentiimetttext,, sentiment):
Stability measure fA).)
sia Sentineentineensttyyyllyyerr
score sia. (text [sentiment)
return score
from langdetect import detect_
def Lul_french_prrcentaeessseee danguages_detecten detect_ (teect__n for lang in segguages_detecteer
Stability measure f-(-)
if lang. lang == 'fr':
return laob lang-prob is the probability return Return 0 if French was not detected
Table 1: Examples of required material for our experimental protocol.
Of course, no single conversation can yield statistically significant results. assess the degree to which a chatbot is vulnerable to instruction drift, we need to average the results of many conversations. We manually curate a benchmark set of 100 system prompts, catego- rized into 5 categories: multi-choice responses, character of the agent, answer-string format pattern, memorization of certain facts, and languages the agent speaks. Each system prompt SB comes with its own probe question PB and stability measure fa(-). For system prompts like SA in Table 1 specific probe questions are crafted to guide the model to break the instruction; for the rest, neutral ones like 6g, a generic probe question is sampled randomly from a set of them. Each stability measure is expressed as a Python function fB(.) that takes as input the agent LM's response b, and returns number p in the range 0 < p determin- istically; the larger the value of p, the better the system prompt is followed. [Table 1 shows examples of system prompt, probe question, and stability measure. We have released the full dataset at htttt::///hggg /ttg::/gugggggggge fncceugggngggge...oceco co co/datasets/Naom sastommmmmmmmmmttteeteeeee system-stteee pr
prompts-benchhark.
3.3 Experimental Results
We use this protocol and benchmark data to measure instruction drift in LLaMA2-chat-70B and gpt-3. 5-turbo-164 (Appendix(D D). Averaging the instruction stability scores across 200 conversations configured with random pairs of system prompts, we arrive the blue line
"A "turn" " is one utterance like az; "round" is when each chatbot takes turn like 0z
4

---

Published as a conference paper at COLM 2024
Agent LM's Persona
B
1.0
Or g nal Persona
User LM's Persona
0.8
Instruction Stabilty
0.5
Instruction Stability
0.6
0.4
0.3
0.4
0.2
0.2
0.1
14
16
0.0
2 4
10 12 14
Number of Turns
Number
Figure 3: (A) The phenomenon of instruction drift. As the interaction progresses, not only does the agent LM lose stability to its original system prompt, but it also begins to adopt the instruction of the simulated user LMM The effects were measured on 200 randomly sampled pairs of system prompts LLLMA2-chat-00B using the procedure shown in Figure The error bar represents one standard deviation. (B) Measuring instruction stability of the agent LM when user LM's system prompt set an empty string.
in Figure 3 A. We observe that the agent LM gradually stops following its system prompts, aligning with our empirical daily usage experiences.
As side experiment, we are curious if the agent LM adopts the user LM's system prompt. This is plausible since the user LM's utterances generated according to PA have strong appearance in the context window. For this purpose, we swap a; with PA and check fA (bbla = Pa). Surprisingly, the agent LM even gradually adopts the instruction of the user LM over extended rounds of conversation, as shown by the orange line in FFiurr A. This could potentially be exploited by adversarial attacks, raising serious safety concerns.
In another safety check (Figure 3 B), we ablate the system prompt of the user LM with an empty string, so it falls back to the default mode of the underlying language model. This rules out the possibility that this could contribute to the significant instruction drift discovered earlier.
Experiment details. We use I.aMMA2-chat-708 for this experiment and follow the format of composing input sequence from Touvron et al. 2023)). Taking the perspective of agent LM as an example, the input sequence looks like [5 41, 51,.... ........... Aii1, b; and is tasked with generating b; as a reply to the last utterance from user LM Each s, a, and b here is string and may contain multiple tokens. Generation is performed with temperature 1.0 and nucleus sampling with = 0.9 (Holtzman et al. 2019).
4 Attention Decay: a Hypothesis
is reasonable to hypothesize that instruction drift results from a decaying influence of the prompt over time. To investigate why this happens, we focus on the attention distribution over context tokens in transformer self-attention heads. Although the intuitive hypothesis broadly captures the underlying phenomenon, our empirical and theoretical analyses uncover nuanced discrepancies.
4.1 Preliminaries
Suppose the input tokens are { each belonging to the vocabulary V. To generate the next token W++1 V, the current tokens are first embedded into D-dimensional vectors ( with the embedding matrix W. RIV|xD These are then processed sequentially by transformer layers, resulting in grid of activations after each layer and for each token
*Omitting formatting tokens like 
, <
S>> or 
.

---

Published as a conference paper at COLM 2024
Round
Round
Round
Round
Round
10
3000
Number of Imput Tokens Agent
Figure 4: The phenomenon of attention decay demonstrated in the 11th attention head in the 24th layer of LLaMA2-7B, which has maximum context window size of 4, 096 tokens. We generate 12 conversations while tracking the portion of attention allocated to system prompt tokens. The plots are specifically for the agent LM, grouped by the rounds which the answers are generated; the values are absent for the user LM. We observe sharp drops in attention between turns and rough plateaus within turns.
i= As the multi-layer perception (MLP) and layer norm are context-independen, leave them out for simplicity. The feed-forward process of the transformer be summarized as:
I = 1 + E Wh'n Att (1) ,,.,,,,,"!),
(1)
Wt (ww = softmax(W, l1t).
(2)
The combination of the softmax and We work as a predictor from ht to distribution ((w/we)) of next token W++1- Att is the single head attention operator with output in lower dimensional space and w!"" E Dxd maps them back into RD, the residual stream space.
Crucial to our experiment, expand the attention operator to show it aggregates activa- tions from previous time steps based on an attention distribution:
(W! 1 1)T "T(W"")) 1)
Lm t.j 1::
= softmax
Vd
Then the attention operation is weighted sum of linearly transformed activations from the last layer:
Att! (Hl k..., 1) Σ al (W!" b! 1),
where w!" RdxD wha E RdxD, w R are the value, key, and query weight matrices, respectively.
The Phenomenon of Attention Decay
While generating the next token given input sequence containing f tokens, in each atten- tion head, the last token will compute a normalized attention distribution over all previous tokens (including itself), denoted by a;,=1 in Eqquaioon 3 Tokens in the system prompt are a special subset of all previous tokens, and we denote the sum of the attention weights allo-
cated to them as TTt)= = |
Rt,i- It ranges between to and represents the comparative importance that the system prompt has throughout the generation process. We monitor this percentage (t) along the decoding time steps and across turns conversations in I.LMA2-7B. We only plot (t) from the perspective of the agent LMM

---

Published as a conference paper at COLM 2024
As shown in Figure 4 within each turn, πt() remains almost constant, but there are signif- icant decreases across turns. This observation runs over a naive hypothesis of attention decay-if the attention distributes uniformly over previous tokens, π(f) should decay hyperbolically and be independent of number of turns.
It's also worth-noting that this highlights unique issue in chatbots, distinct from language models, where -ustrr-iistiiuution text from interlocutors absent. The case of the language model completing its input partial sequence technically equivalent to the agent LM generating answers for a single turn, which displays plateau in T(t).
This observation shows merely the co-occurrence of instruction drift and attention de- cay. However, it inspires the hypothesis that attention decay may internally contribute to instruction drift, suggesting that addressing the former could help mitigate the latter (Section 6.2)
5 A Geometric View of Attention Decay
To shed light attention decay in l'igure both the plateau within utterance and the drop across utterances, we provide theoretical explanation simplified situation.It has been shown empirically and theoretically that the internal representation of deep neural networks usually live in narrow cone in the high-dimensionall space (Mimno & Thompson,
2017: Ethayarajh 2019 Zhu et al. 2021 Liang al. 2022). Motivated by their observations, we characterize attention decay from similar geometric perspective.
We will consider two settings of model generation:
1. New tokens are generated autoregressively given initial tokens (i.....,/ss))) which models the process of the agent LM generating answers;
2. New tokens are drawn by the user. A user LM could put out-of-distribution tokens into the context window of agent LM in a potentially adversarial fashion (Zou al.. 2023
For the first setting, we will show that tokens generated by the model always remain in approximately low-dimensional convex cone in Theorem 5.1 In the second setting, can characterize the expansion using spherical measure and show that randomly drawn tokens will lead to an expansion of the underlying convex cone with the growth of intrinsic dimension of token embeddings, shown in Proposition A.2 in Appendix
5.1 Setting One: Agent Utterances
In linear algebra, a cone is a subset of a vector space that is closed under positive scalar multiplication. In other words, C is a cone if implies sx C for every positive scalar s. Moreover, C is called a convex cone if ax + By E C for any positive scalars a and ß, and any x, C.
The dimension of a cone is the dimension of the vector space spanned by the elements of the cone. For convenience, we define two new notions related to low dimensional cones in the space RR. Given any d-dimensional convex cone C C RD (1 < d < D), for e E (0,1) we define the corresponding c-approximate d-dimensional cone as
C = {w E C sC) C RR= w u
for some C, span(C)+ = R d, ||ol|| S e||w||]
Given some c E 5D and f E (0, a d-dimensional spherical cone is the set defined by [=(c,0] == (u u RD. u R (c,u) ||^||| cos
Theorem 5.1. Assume that the token embeddings of the system prompt given by h,, bny....,,aa, ht lie in the d-dimensional approximate cone C and that any output-value matrix wh." = w!!" e DxD satisfy that wh.. u C for any E CE. Then all proceeding tokens generated by our
7

---

Published as a conference paper at COLM 2024
simplified transformer lie in the convex hull of C In particular, if C is contained in some spherical cone pd [c,0], then all generated tokens lie in the E-approximate cone C where E/ Ve2 + cos ( e
For the initial tokens, 8 indicates how concentrated their embeddings are, and is roughly the intrinsic dimension of these embeddings. Note that d |spl and the number of tokens in the system prompt |$g| is usually much smaller than the dimensions of hidden space D, which is 8192 in the case of LLaMA2-70B-chat Thus, the assumption that initial embeddings occupy low-dimensional cone is reasonable.
Theorem 5.1 shows the convex cone for token embeddings remains stable during the generating process if there is no user input, which leads to the plateau within an utterance.
6 Mitigating Instruction Drift
If instruction drift related to attention decay, that suggests we can mitigate drift by manipulating the level of attention on the original prompt. Before presenting an attention- based mitigation method, however, we describe two baselines.
6.1 Baseline Methods
System Prompt Repetition (SPR) We inject the system prompt with probability 0 S p S 1 before each user utterance. The repeated system prompts, like the standard system prompt at the start of the input sequence, only appear when the language model is prompted; users do not see them.
Classifier-Free Guidance (CFG) The second method is classifier-free guidance (CFG, Sanchez et al. 2023), which runs the base model twice, firstly with system prompt to get log p[w] (WW SB) and then without system prompt to get log p w.r)) It then uses a contrastive linear operation inside the logit space to strengthen the effects of the system prompt on answer generation. The new next-token probability distribution is defined by:
log (w)) (z)
log p(w) wxt)) + a (log p(w) (g) - log (w)
(5)
CFG comes with a hyperparameter a 2 that controls how far we shift the predicted logits. When 1, it reduces prompting with the system prompt; larger produces stronger intervention.
6.2 Proposed Method: Split-softmax (SS)
Motivated by the attention decay phenomenon, we introduce method that requires no retraining, split-softmax, aimed at reducing this decay with minimal overhead. The basic idea is straightforward: prompt, then force the model to pay more. In practice, we find that a power-law scaling of attention seems to be effective.
the problem is that the model pays too little attention the
In particular, split-softmax (SS) works by inserting scaling operation between Equation 3 and Equation 4 for every attention operation. After obtaining the attention distribution { which sums up to 1 (omitting superscript for simplicity), we reweight it by:
bg
m()) ar
if $ |spl
(t)))=
Σ
aii
1 (+) T x(f) ar if |sa|
(6)
where the introduced exponent S < as a hyperparameter to control the strength our intervention. The smaller k is, the stronger the intervention is; when k 1, the intervention is nullified. The new set of attention {a sums up to 1 as well and will replace {at, 1
so that more attention is paid to the system prompt tokens. Given S 7(1) < 1, 5 k
T(f 2 1, solit-softmax increases the proportion of attention paid to system prompts. See Appendix El for more discussion.
thus
8

---

Published as a conference paper at COLM 2024
6.3 Calibration Using Performance Drop MMLU
Each method (split-sfftmax and the two baselines) represents a potentially large interven- tion; any instruction stabilization may come the expense of other capabilities of the model. However, each method has hyperparameter that corresponds to the strength of the inter- vention. To compare methods, therefore, we need to measure both the increase in instruction stability and the performance drop for various values of the relevant hyperparameter. This analogous measuring precision-recall curve for a classifier.
To measure any performance changes, we use the Massive Mul- titask Language Understanding (MMLU, Hendrycks et al. 2020). To compare the different methods, look at the stability improvement at equal levels of performance drop. Sweeping hyperparameters for each method allows us measure and plot each method's 's maaillty-performance curve, revealing different trade-offs between our stability metric and MMLU performance.
SPR
CFG
SS (ours)
0.7
Instruction Stability
0.6
0.5
As expected, we do see an inverse re- lationship between performance and instruction stability in all three of our methods Figure 5. This corroborates earlier findings by Gu et al. (2024) that control methods over language model often come at the cost of general ca- pability. The performance drop on MMLÜ should be thought of as a bud- get when correcting model behaviors, and two methods should only be com- pared on stability when their respective hyperparameters cause similar MMLU performance drop.
0.3
-1.0
-0.8
-0,6
-0.4
-0.2 0,0
MMLU Performance Drop Relative to Baseline (%)
Figure 5: Comparing trade-offs between instruc- tion stability and performance. For each the three methods, we vary hyperparameter that re- flects the strength of the intervention. Each curve plots the effect stability and performance over the hyperparameter sweep. Compared to two base- lines (classifier-free guidance and system prompt repetition), split-softmax produces equal or higher stability for given level of performance degrada- tion.
To quantify stability, we use 16-turn conversation described in Figure 2 We modify these conversations by ap- plying each method to the agent LM.
Then we probe the agent LM at each round to test its instruction stability in the same fashion as secstin 3| Stability is measured for individual turns, and the overall stability measure is the average of the stability at each turn of agent LM. Given the conversation history of agent LM under intervention, we sample one and ask questions from MMLU at intermediate turn (the 4th turn in our experiments); and the answers are used to calculate MMLU accu- racy. Note that due to the added system prompt and chat history, the MMLU performance is different from what is reported by LLaMA team even without intervention (Touvron et al., 2023). However, only the difference between post- and pre-intervention performances is meaningful, as the primary purpose of using MMLU in our case is to calibrate the strength of the intervention.
6.4 Experimental Results
All experiments conducted on LLaMA2-70B-char To save computational cost, we choose one system prompt from each of the five categories, and run experiments over the total twenty ordered pairs of system prompts.
In Figur we plot instruction stability versus performance drop on MMLU as we vary the strength hyperparameter for each method. In general, split-softmax presents better

---

Published as a conference paper at COLM 2024
1.0
++
SPR
Instruction Stability
0.8
CFG
1
SS (ours)
0.6
+
0.4
+
f
0.2
+
0.0
2
8 Number of Turns
10
12
14
16
Figure 6: Comparison of instruction stability across turns, with MMLU performance drop around the value of 0.5, for system prompt repetition (SPR), classifier-free guidance (CFG), and split-softmax (SS). The whisker represents one standard deviation.
trade-off between performance drop and instruction stability. It can match performance with system prompt repetition while avoiding using the additional context window. more drop in performance MMLU is allowed, split-softmax enables greater instruction stability.
break down the instruction stability measurement across turns. Similar to Sanchez et al. (2023) show, classifier-free guidance helps the model adhere to the system prompt remarkably well for the first round of the conversation, but it does not generalize well into extended conversations. Both system prompt repetition and split- softmax demonstrate higher effectiveness in mitigating instruction drift, though they exhibit different trends. The former excels in regions with a larger number of turns, while the latter performs better at the beginning of the conversation. Note that system prompt repetition consumes substantial portion of the context window.
In Figure 6
7 Conclusions, Discussions, and Future Work
Our experiments indicate that instruction drift is potentially significant issue for prompt engineering. To help address this challenge, we contribute a new protocol and benchmark to help measure this phenomenon, well idealized mathematical model of its cause. In addition, we proposed technique, split-softmax, that can help mitigate instruction drift, providing a better stabiity--perfrrmane trade-off than existing baselines.
The instruction drift indicates that current LM systems still cannot behave coherently over long horizons, which is the basis of long-term planning, such information seeking (Lin et al.) 2023) and tool usage [Nakano et al.) 2021). Fundamentally. it is caused by mis- match between their training schemes (text continuation or single-round RLHF) and their deployment scenarios (open-ended dialog with users). The phenomenon of instruction drift suggests there much be done to bridge this gap toward more robust and coherent dialog systems.
There ample room for future work in this space. For example, would be natural ex- plore making changes in architecture or to training to combat instruction drift. Furthermore, all the techniques we discussed involve an apparent trade-off between performance and reliability. Is this necessary compromise, stability at no cost? would also be good to deepen our theoretical understanding, adding realism to the idealized " "cone" model of instruction drift that we proposed. Finding new ways to measure and prevent instruction drift is an important step in ensuring Al safety and reliability.
there methods that maintain instruction
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank Jiawei Zhou for useful discussions and feedback on the manuscript.
KL is supported by fellowship from the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University and Superalignment Fast Grants from OpenAI. DB supported by grant from Open Philanthropy. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. This work was partially supported by NSF grant IIS-1901030.
References
LE Blumenson. A derivation of n-dimensional spherical coordinates. The American Mathe- matical Monthly, 67(1):63-66, 1960.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arviv preprint arXic:1912.02164, 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint 7XXo:1909.00512 2019.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. Xix preprint arXiv:2209.0788 2022.
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yt Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arv preprint ar Xiv:2440 04700, 2024.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. Improving zero and few-shot generalization in dialogue through instruction tuning. arXiv preprint ar 2tte:2205.12673 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint ar #wXiv:2009003300 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. Xiv preprint ar arxiv:1904.088: 097511, 2019.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. Xrviv preprint ar Xix::2099.06360 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint ar Xiw:2306.033411 2023a.
Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics & Statistics, 4(1):66-70, 2010.
Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, and Hongxia Jin. Instruction-folloiing evaluation through verbalizer manipulation. ar Xiv preprint arXiv:2307.10558, 2023b.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:1111, 35:1711--17655 2022.
11

---

Published as a conference paper at COLM 2024
Jessy Lin, Nicholas Tomlin, Jacob Andreas, and Jason Eisner. Decision-oriented dialogue for human-ai collaboration. ar preprint arXiv:2305.20076, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. Xrvi preprint ar Xix::2104.008786 2021.
David Mimno and Laure Thompson.
The strange geometry of skip-gram with negative sampling. In Empirical Methods in Natural Language Processing, 2017.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? Xrv preprint ar arXio:2202.12837 2022.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint drXiv:2305.16933. 2023.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answeria with human feedback. arviv preprint ar Xiv:221122.9933,, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-free guidance. ar preprint arXiv:2306.17886, 2023.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non- parallel text by cross-alignment. Advances in neural information processing systems, 30, 2017.
Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. Towards better evaluation of instruction-folloiing: A case-study in summarization. arXiv preprint dxXiv:2310.08394, 2023.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. Xrv preprint ar Xiv:2310.15211 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXi preprint arXiv:2307.09220 2023.
Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. Can chatgpt write a good boolean query for systematic review literature search? arXiv preprint arXto:2302.03399 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. Xiv preprint ar Xix::2109.01165,, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24882 2022.
James G Wendel. A problem in geometric probability. Mathematica Scandinavica, 11(1): 109-111, 1962.
12

---

Published as a conference paper at COLM 2024
Jason Weston and Sainbayar Sukhbaatar. System 2 attention (is something you might need too). arviv preprint arXio:2311.11829, 2023.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabili- ties and limitations of language models through counterfactual tasks. ar preprint arXiv:2307.024777 2023.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following, arviv preprint arXiv:2310.07641, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-following evaluation for large language models. arXiv preprint ar Xit:2311.079111 2023.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar arviv preprint ar Xit::199.00853, 2019.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint Xi0::2307.15043,, 2023,
13

---

Published as a conference paper at COLM 2024
APPENDIX
A Sketch of the Theory for Setting Two: User Utterances ( (Section 5
Again we assume that the system tokens , ( are from some Cp and let C₁ be the smallest convex cone containing Co and user tokens (/ Then the expansion Co C 1 C... C Cn reflects the attention decay under the influence of user utterances. To get some intuition on the expanding process, we show the following:
Proposition A.1. If user tokens drawn i.idd uniformly from SD-1, then with probability 1 after n 2 4D + 2og 1 user tokens Cn expands the whole space TDD yj
Proposition A.1 suggests that when user utterances are inserted, the size of the convex cone for token embeddings will grow significantly, which gives rise to the drop of π7(1) across utterances. To further quantify the expansion of convex cones, we can consider the spherical measure - which is the Borel measure on the DD11)-sphrr such that o For any 6-approximate convex cone C define the volume of Ce by
k(C²) == oD I
Then intuitively /(C))/((CC indicates the degree to which the current tokens in C align with the system tokens in Cj similar to the quantity z(() defined in the previous section.
In real applications, user messages are not i.i.d. uniform variables from SD-1 However, there usually exists an evident proportion of user tokens distinct from the system tokens. They could probably be tokens unique in the specific topics that the user inquires about or, more typically, tokens from a new language. It could also happen that the user attacking the LM by sending adversarial tokens [Zu et al. 2023). The following proposition quantifies how attention decays in terms of μ (()))/((C)) as such embedding dimension increases.
Proposition A.2. Suppose Co is a di-dimensional convex cone contained in some dy-dimensional spherical cone pdd [c1, 4p1 while Cn a di-dinensi convex cone containing dz-dimensional spherical cone pdæ [c2, 422). Then we have
F (C() " (CC) S d d1
The geometric perspective we proposed provides a concrete explanation of why inserting prompts will cause attention decay while autoregressive generation from the model will almost have no harm. However, one limitation here is that we have only compared the cone structure without tracking the distribution of token embeddings within the cones. In particular, if we force the majority of tokens generated from C to be contained or close to Cb, the issue of attention decay could possibly be mitigated, which motivates our method in the proceeding section.
B Proofs for Theorems
We start by making simplifications to the model and token-generating process. First, the model is simplified by omitting the MLP and layer norms in (n{Euattio! 1 For the token- generating process, the embedding of the next token the vocabulary in in/Equation 2 Thus, for convenience we directly put == / in our simplified model, meaning that all embeddings lie on the unit hypersphere SD == {v E IRD ||a|| = 11.
is close to hf among all tokens in
Proof of Theorem 5.1 Let C be the convex hull of C The C a convex cone containing C.. Theorem 5.1 can be proven in two steps.
14

---

Published as a conference paper at COLM 2024
Step I. We establish that h, E C by induction. hi....,h already satisfy the claim by assumption. Supposing that ,,,,,,, E C 2t2 fo), we show that h4-1 is also in CF, Here we look into 1i (j = I,...,,,I I,..., L) the process of generating 14+1- We perform induction on I. For I = O, we have it) h E C Supposing that nt E C for j = 1,...,4, suffices to prove that 11/11 E C
,,,, E C. By induction hypothesis that kt E C (j= = 1;,,,,,... I!, we can find kj E EN ,8,1...,,,18 and wj.1..., > 0 for 1, .I... such that
h Σ 3
Thus, by Eqqation 1
we have
1+ I +
W
Att + (t))....,1))
= kn +
1+ 1+mm +
= ht + Σ
W9 + L. x
a
is
Note that 111m > since it is calculated from softmax and by assumption we have jis
W W! + x,;x E C as Xs,i C.. Thus, we conclude that h/+1 E C. By induction we know for 1 = 1,..., and 1,..., we have it/ E Ce. Thus, h141 = 11 /| C holds.
Kiss
And by induction again we conclude that I, E C for all / 1.
Step II. Let 7 cos We prove that C C C where è e/ + - e²). For any y E CE there exists k E N Xi....,XX E C and Wi...., W. > such that y Σx W,x, ui + where u; E C and o; E span(C)
By definition of C x; can be written as and || < e||x:||. By definition of pd [c, we have (c) 2 y||ui| for all = =,...,k k. Let
u := (c, c. Then (ū;, ui ūi) and hence (E waūü Σ wo;(u; ū.)) 0. Therefore,
we have
k
Σ Will;
wi
(c,
Will
7
w0: |||l=
On the other hand, we know
k
Σ
w.F
w;
|i
S
6
w0
|
v1 c²
Therefore, it holds that
k
TV
- -
k
Σ
Willi
>
Σ
e
which implies that
wivi
2
E
Σ w/2
02 + (1
22
Thus, we conclude that C C C
To prove Proposition A.1 we need the following lemma.
15

---

Published as a conference paper at COLM 2024
Lemma B.1 (Wendel 1962). Let N points be scattered uniformly at random on S C R"=11 Then the probability that all points lie on some hemisphere is given by
Am = 2-2- N
(N=1) k
N 1
Proof of Proposition A.1 there is no hemisphere containing #1o ny lies in Cn and is not on the boundary, meaning that C₁ = Rp, Thus, we only need to show that for 2 4D + log i,r it holds that AD,x S f/. Since
then the origin
2 R Σ
11
2
n' i!
n D
2
en D D
i
suffices to prove that 2 (et) can check that
D
< f/- For convenience let 4 + B log $ D.
Then we
(log 2 2 e212 1/D
Note that
1 2) allog22 1)- a (logg 2
which is equivalent to
ea(log2 4
2
ea 1
log 2
1
ex/2 e212 (log 2 2
Thus, we have
2 en
(ea)
D
2nD
(log2 1) enD/2
To show Proposition
A.2
we need the following lemma.
Lemma B.2 Li 2010
The spherical measure of the spherical cap pm 1 [c, (c,,0nst St is given by
S0d sin 1 xdx
/ / m1 / sin
r(
mm (p"+1 [c,
sin" xdx,
2 /2 "- xdx
where [(x) is the Gamma function.
Proof of Proposition A.2 First lower bound μ(( Ct) by identifying many disjoint spheri- cal caps with angle H = arcsin e as possible and applying Lemma B.2
Let M be the largest number such that there exists set of points (,,...,AM E [c2 4/2 8] n nSD to ensure pD(a,) [a, 0] C [d2[c2, 422 (i = 1,...,y are disjoint from one another ("disjoint" meaning that the measure of intersection is zero). We claim that {p4 20] |M covering of pdz [d2 4/2. Otherwise, choosing 46 E pdz [c2, 4/2] nS \ U, [d, [a; 14,220 20 we
can check that pD |no, 8) does not intersect with any of pD [a,, 0). Thus, these M + spherical caps do not overlap, which contradicts the definition of M. Hence pdz [c2, 422 and by Lemma B.2 we have
C U: pd2 [a; 20),
r(4)
/0
4/2 sin 2 xxxx d (c2, 422) nS0
/ do
1
M
I(4))
29
Ug 1 [pd2
[a;, 20 -11-1) = -Mo Mo 281)
M
/0
sin din22- xdx.
/
d
Jo
16

---

Published as a conference paper at COLM 2024
On the other hand, since [a 0s's are disjoint from each other and that p [a 0] C PD (C2, 422 (because = sin f), know
M
((())
UD 1 (PD
aq 0]
ns MMoo- [a. 10..]nssp- n
T
= M
sin xdx
/
r(2) So sin 2-2 xdx So sin D-2 --2
xdx
1 S0 sin xdx
Next we upper bound ((C)). For any (x xn) E B"
x2 < S1}, we introduce the hyperspherical coordinate system, which consists of a radial coordinate r, and 1 angular coordinates f1...., Pn 1, where the angles $1, .rr range over [0, π.| and n ranges over [0, 27). In specific, the coordinates are defined through the transformation:
:
X1 = Y cos 41
X2 sin Φ1 cos 42,
X3 = r sin Φ1 sin 42 cos $3.
:
Xn 1 r sin Φ1 sin Pv. 2 cOS pn Xu = sin Φ1 sin pn- sin Pn 1-
By assumption we know C. C [C[ 41]- Therefore, using the notion of spherical elements (Blumenson 1960), write
((C))) = -CD-1-
Area(so-1) I sin 11 sin D-3 22 ....si 2d(01. ... .D-1),
where
Q {(o).... .D-1): :41 E [0, v1l, ... 2D- [0, [0,2 4:-1 sin oj E [0.e]]
Denoting
Ω = ={ (4))... 11) : : E 10,41.. 02.. 02 (z) x1, sin [0,e|],
then we have
((() = Area(So-1) (0 -1541 11554
sin Sin Φ1 ..n D-4h D-n
1'"
1"'
2
sin D-n -1 Qd ... sin ΦD 2d0-- ... ..
Area DD-4)
sin D- 2 41 ... sin d1 dt id(err.... 1)
Area AieaaSDD (SD
Area ( -4)
di
11
10 ... So
/o si 2 24 ... sin 2dl1 ... -dp,, 1
Area (5D-1)
Are d1
Area(S -1)
do (pd) (c, s1ln-
2Area (SD
1)
T(
D-4
dj 1 4 xdx.
r D 4 11
d
17

---

Published as a conference paper at COLM 2024
Thus, conclude that
P(C))
T( T(D-att1 )T( d--1
So sin xdx Jo aresine sin xdx D-di
(CC)
r( 11=1)
1o 2 2- xdx fo pancsine sin
D-2 Din xdx
< e D-4 2-1
eD-1 622-11
2000
Frequency
1500
1000
0.2
0.4
0.6
0.8
1.0
1.2
Norm Value
Figure Histogram of embedding vector norms.
Norm of Embedding Vectors In Section
3.2 we assume that the embedding vectors have the unit norm. To verify if this is reasonable, plot the density of the norms of vocabulary embeddings for the LLaMA2-7B-chat in Figuree We can observe that the norms are quite concentrated around 1.
C Does RLHF help?
Given how RLHF Ouyang et al. 2022) ; Ziegler et al. (2019) train the model, the model should be trained to pay more attention to the system prompt so to increase user satisfaction. In Figure 8 we show that RLHF could increase the portion of attention paid to the system prompts by comparing LI LLaMAA-BB and LLAMA2-7B--hat The latter is trained on top of the former with human feedback. shows that RLHF indeed helps in combating instruction drift, but it still cannot eradicate it entirely due to its nature of fine-tuning.
D Additional Instruction Drift Experiments
To see how close-source model compares with LLaMA2-70B-chat we test gpt-3. 5-turboooobo1eee 16k with a total of 200 randomly sampled system prompt pairs. Re- sults are shown in Figure It turns out that 3.t-3.5-tuue 5-turbo-16kk holds to its system prompt better than LLaMA2-chat-70B, but still suffers a 10% drop on the stability of its original system prompt.
18

---

Published as a conference paper at COLM 2024
La LLMAA--B--hu
500
1000
1500
2000
2000
3200
Figure 8: Comparison of attention decay between LLaMA2-7B before and after RLHF training. Different from the categorical palette used in Figure to differentiate number of rounds when the answer is generated. The deeper the color, the later the round which the answer is generated.
0.85
0.80
0.80
0.55
Iestnggtion Stabiity
0.50
Instuction Stabllty
0.45
645
0.40
0.40
0.35
0.30
6:30
0.25
10
15
20 Turns
25
30
0.25
5
10
15
20
25
30
Number
Number of Turns
Figure 9: Measuring the instruction stability of 3ott33--turbo-166 via API using the same protocol as Figure 3 On the left, the system prompt is given to the API via the "system" " argument; the right, it is prepended to the user's first utterance.
E Discussion of Split-softmax Formula
We first quickly show how the post-intervention attention values Equation still form distribution by summing up to 1, dropping subscript
Σnt a
Σ
a
Σ
a
isbal
1>1a
1 77)
Σ ai +
Σ ai
(t)
1 - T(())
is/s8l
i>|0
r (1)
(1 n (1)
=1
Meanwhile, it worth-noting that the ratios of attention scores for tokens within the system prompt and within conversation history remain unchanged, thereby minimizing disruption to the attention mechanism.
19


Reviews:
[{"review_id": "7YnJpVXAVX", "reviewer": "Reviewer_9T1g", "summary": "* The paper proposes a benchmark to measure the ability of language models to follow the system instruction over the course of a conversation.\n* Using LLama2-70B and gpt-3.5-turbo, the paper shows that language models often suffer from significant instruction drift within 8 rounds of conversation.\n* This decline in performance can be attributed to attention decay over the course of the conversation - with each turn, the model starts paying less attention to the system prompt.\n* The paper proposes a method called split-softmax to combat this issue and improve adherence to the system prompt - this method performs better than simple, but intuitive baselines like repeating the system prompt at regular intervals and using classifier-free guidance.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments and review!  \n\n# Weakness  \n**Other measures of helpfulness like MT-bench.** We agree that using a more comprehensive benchmark to measure the helpfulness of LLM could be a better idea than using MMLU. However, MT-bench requires users’ votes to determine the helpfulness of the uploaded model, which makes it hard to scale and obtain results in a timely manner.   \n**More details of the benchmark dataset.** We will add more examples to the paper to provide a better understanding of the dataset.   \n\n\n# Questions  \n**Alignment data blend.** We could expect data points with a stronger connection to system prompts in the alignment dataset to be conducive to better instruction stability of the fine-tuned model. Studying the relationship between instruction stability and data composition could be a fruitful future research question.  \n**For Figure 4 and Figure 8**, when the system prompt is used as the first user’s utterance, it’s reasonable to guess that the curve will look similar to Figure 4/8 on LLama models. In fact, for the prompt formatting scheme by llama, the system prompt is guarded with special system-prompt tokens and prepended to the first user message. So the suggested change would be equivalent to removing two special tokens. For GPT-3.5 experiment, it’s shown in Figure 9, suggests a special internal mechanism behind the GPT-3.5 API that emphasizes system prompts.   \n**On probing questions.** The probing questions aren’t necessarily adversarial. For some system prompts, we stipulate certain rules that any utterances from the agent LM should follow; therefore, the probing questions can be arbitrary. In those cases, we use “What do you do in London as a tourist?” as a default choice. We will incorporate these details into the paper like for weakness 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very relevant and important direction of research - implications are useful for all fine-grained instruction following tasks that a language model assistant would be expected to perform (function calling, tool usage, staying in character)\n* Excellent empirical analysis of instruction instability in language models and identification of the reason behind why that might be the case.\n* The proposed method - split soft max - is very intuitive and easy to implement in practice.", "reasons_to_reject": "* Would like to see some measure of helpfulness like MT-Bench to evaluate model behavior, instead of relying only on benchmarks like MMLU to quantify the impact of the intervention.\n* There are not many details on the benchmark dataset or how the system prompts are chosen in the paper - would be helpful if more details are added.", "questions_to_authors": "* It seems like choice of what type of data is used to construct the alignment blends/what goes into SFT & RLHF would be very important to counter this kind of behavior. Would fine-tuning on role-playing data/alignment blends help improve instruction stability?\n* For the graph in appendix C, what would it look like if the instruction was in the first user turn instead of the system prompt?\n* How are the probing questions created? Are they designed to be adversarial in nature?", "extra_scores": {}}, {"review_id": "Ce7TZwzNbK", "reviewer": "Reviewer_b2sN", "summary": "This paper presents a simple approach to addressing instruction drift in language model dialogs, particularly focusing on system-prompted chatbots. The authors introduce a quantitative benchmark for measuring instruction stability and propose a novel method, split-softmax, to mitigate attention decay, which they identify as a key factor contributing to instruction drift. The paper demonstrates through empirical testing on models like LLaMA2-chat-70B and GPT-3.5 that attention to initial system prompts decays over the course of long dialog exchanges. The split-softmax method, which requires no additional training or parameter adjustments, effectively enhances the model's focus on initial prompts, thereby improving instruction adherence over extended interactions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments and review!  \n\n# Weakness\n**Discuss implications.** Thanks for the suggestion to extend the discussion of how the findings in this paper impact other aspects of LLM behavior under various deployment scenarios. We will add such a discussion:  \n> The instruction drift indicates that current LM systems still cannot behave coherently over long horizons, which is the basis of long-term planning, such as information seeking (Lin et al., 2023) and tool usage (Nakano et al., 2021). Fundamentally, it is caused by a mismatch between their training schemes (text continuation or single-round RLHF) and their deployment scenarios (open-ended dialog with users). The phenomenon of instruction drift suggests there is much to be done to bridge this gap toward more robust and coherent dialog systems.  \n\n**More diverse baselines.** We acknowledge that a more diverse set of techniques could strengthen the argument for the effectiveness of the proposed split-softmax technique. Among them, as suggested by reviewer 9T1g, studying different data compositions of the RLHF data could be especially useful—for example, adding role-playing data points into RLHF dataset could potentially improve the instruction stability of the resultant model.  \n\n# Questions\n**Other aspects of model performances.** It would be a good idea to measure the language model’s output diversity with and without split-softmax. A naive guess is that it shouldn’t influence the entropy of the next-token distribution. We will add a few sentences of discussion on how split-softmax would help the model fend off adversarial attacks. We believe that attacks based on both traditional input optimization like GCG (Zou et al, 2023) as well as prompting like PAIR (Chao et al, 2023) could work less effectively on split-softmax-guarded models.   \n**Dialogs beyond the tested lengths.** As seen in Fig 6, after 16 turns of dialog, the LLM has lost most of its instruction-following capability, with or without various mitigation methods. In fact, at this point, LLaMA-2-70B cannot even speak coherent English, but we could reasonably expect a more advanced model to maintain its language into >16 turns of dialog.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces an original method for quantifying and addressing instruction drift in chatbots, which is a significant contribution to the field of dialog systems and AI safety.\n- The empirical results are compelling, showing clear evidence of attention decay and its impact on instruction drift, along with a detailed comparison of the split-softmax method against strong baselines.\n- Addressing instruction drift is crucial for the safe deployment of AI systems in real-world applications, making this work highly relevant and timely.\n- I like this paper for its intuitive understanding of the attention decay phenomenon and backs this intuition with solid theoretical explanations.", "reasons_to_reject": "- The discussion could benefit from a broader analysis of how these findings might influence other aspects of language model behavior and deployment in more varied contexts.\n- While the paper compares the proposed method against strong baselines, including more diverse techniques could strengthen the claims.", "questions_to_authors": "- Could you elaborate on how the split-softmax method might affect other aspects of model performance, such as response diversity or susceptibility to adversarial attacks?\n- In extended dialog scenarios beyond the tested lengths, how does the split-softmax method perform in maintaining instruction stability?", "extra_scores": {}}, {"review_id": "6bEzrS785u", "reviewer": "Reviewer_Cp3m", "summary": "The paper proposes a method, namely split-softmax,  for avoiding the instruction drift present in conversations as the context increases. First, authors evaluate the instruction drift, through five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. They simulated 200 conversations by prompting large language models (LLMs) for playing the user and  the system. They experimented with GPT3.5-turbo and LLaMA2-Chat-70B They implemented a function in python to compute the instruction stability.  Second, they observed stability decay in as the number of turns increase. Third, authors hypothesize this is due to the attention decay and they propose the split-softmax method to mitigate the instruction drift: a power-law scaling of attention.  They compare split-softmax against two baselines: prompt-repetition and classifier-free guidance. They show split-softmax is more stable as the number of turns increase than classifier-free. However, it has similar performance than  prompt-repetition although it requires less context.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments and review!  \n\n# Weakness\n\n**Details of stability functions.** The stability functions are diverse, but we make sure that all can be achieved with basic python functions deterministically. For example, below are some examples of the stability functions. \n\n```python\nfrom nltk.sentiment import SentimentIntensityAnalyzer\ndef get_sentiment(text, sentiment):\n    \n    sia = SentimentIntensityAnalyzer()\n    score = sia.polarity_scores(text)[sentiment]\n    return score\n```\n\n```python\nlambda x: \"tennis\" in x.lower()\n```\n\n```python\nfrom langdetect import detect_langs\ndef get_french_percentage(sentence):\n    languages_detected = detect_langs(sentence)\n    for lang in languages_detected:\n        if lang.lang == 'fr':\n            return lang.prob  # lang.prob is the probability of the detected language\n    return 0  # Return 0 if French was not detected\n\n```\n\n**Figure 4 over different attention heads.** Preliminary experiments show that all attention heads, except those from the first several layers, demonstrate a similar attention decay pattern, as shown in Fig 4. We will add other specific heads or an average over all heads in the appendix.   \n\n# Questions  \n\n**Figure 3** didn’t express that the user LM could follow the instructions better. In fact, it should show a similar decay as the blue curve in Figure 3A since the roles of the two LMs are largely symmetric. The orange curve examines the agent LM’s behavior using the stability function of the user LM’s persona, therefore demonstrating that the agent LM is slowly picking up the persona of the user LM.   \n**Theorem A1.** If accepted, we will consider moving more content from Appendix A into the main paper, given that the page limit is extended to 10.   \n**Missing citations.** Thanks for pointing out relevant literature that we missed out. We have added them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper clearly explains the instruction drift problem and it's cause. Authors design a way to evaluate instruction drift on 5 types of instructions: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. The do show a decrease of stability as the conversation gets longer.  Authors hypothesize this is due to the attention decay and they propose the split-softmax method to mitigate the instruction drift. They show split-softmax is more stable as the number of turns increase than classifier-free guidance. However, it has similar performance than  prompt-repetition although it requires less context.", "reasons_to_reject": "It is not clear how the instruction stability function was implemented for each of the five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. Authors limited to say that the function should return a number between 0 and 1 (however it is computed deterministically).\nFig 4, It would have been interesting to depict the stability decay in all the attention heads or in average.   \nFig 6 shows that split-softmax is not better than system prompt repetition when the dialogue length increases, however it requires less  context.", "questions_to_authors": "1) How was the instruction stability function  implemented per each of the five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks? Which libraries were used for each case?\n2) Is the LLMs who plays the user more unstable that the one who plays the system according to Figure 3?\n3) Fig 4. Is the behaviour of all the heads similar to the 11th head?\nI suggest to move the Theorem A1 to the main paper.\n\nMissing citations:\nHow Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings\nKawin Ethayarajh.\nA Geometric Analysis of Neural Collapse with Unconstrained Features\nPart of Advances in Neural Information Processing Systems 34 (NeurIPS 2021)", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the submission, in particular, the clear exposition of the problem of drift, the empirical analysis, and the introduction of the simple split-softmax method. The also mention this is a useful resource. There was also interesting conversation during rebuttal, where all participated. As all reviewers were in agreement with strong positive scores, I'd recommend acceptance.

I would recommend incorporating details on the benchmark dataset and system prompts as suggested by 9T1g in the CR version. Authors could also move the content from the appendix to the main paper as they suggested in rebuttal to Cp3m.
Decision: Accept