Preprint

Measuring and Controlling Instruction (In)Stabilityin Language Model Dialogs
Kenneth Li1 ,∗ Tianle Liu1 , Naomi Bashkansky1 ,David Bau2 , Fernanda Viégas1 , Hanspeter Pfister1 , Martin Wattenberg1
1 Harvard University, 2 Northeastern University

arXiv:2402.10962v3 [cs.CL] 1 May 2024

Abstract
System-prompting is a standard tool for customizing language-model
chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be stable, so
the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative
benchmark to test this assumption, evaluating instruction stability via
self-chats between two instructed chatbots. Testing popular models like
LLaMA2-chat-70B and GPT-3.5, we reveal a significant instruction drift
within eight rounds of conversations. An empirical and theoretical analysisof this phenomenon suggests the transformer attention mechanism plays
a role, due to attention decay over long exchanges. To combat attention
decay and instruction drift, we propose a lightweight method called splitsoftmax, which compares favorably against two strong baselines. Code:https://github.com/likenneth/persona_drift.

1

Introduction

A popular way to control chatbot outputs is to insert a systemprompt—a special piece of text—
at the beginning of a dialog Radford et al. (2019). The hope is thatthe right prompt (e.g., “You are arockstar programmer who always
writes comments”) will customize
the language model’s behavior fora particular purpose (e.g., producing clear, correct code). Indeed,
Wang et al. (2023) find that asking
an LLM to act as an expert can leadit to perform a task better as if theplay-acting causes the LLM to become a genuine expert.
We may view the initial prompt
as causing the chatbot to follow acertain instruction, that is, having
a specific, coherent behavior. Informally, this may correspond toa specific personality or directly
relate to the semantics of the output (as above, for a coding chatbot,
a prompt that stipulates it should
always write comments). It may
also be related to aspects that areorthogonal to the semantics (e.g.,a prompt specifying “Always respond with a haiku”).

Figure 1: An example of instruction drift on
gpt-3.5-turbo-16k. Although the chatbot
initially follows the system prompt well, it
fails when the same question is asked again
after an extended conversation. Any LLM
user might relate to this issue.

* Correspondence to: Kenneth Li <ke li@g.harvard.edu>.

1

Preprint

1

User LM’s context windowAsystem prompt

conversation starter
copy

Agent LM’s context windowB2system prompt

agent message

user message

copy
generate

response

generate

response

agent message

copy
user message

copy
generate

response

Probing instruction stability
B

Backbone
Branch Out

stability

score 1stability

score 2stability

score 8Figure 2: An illustration of the proposed evaluation pipeline of instruction stability. (A)Initially, two language models engage in a conversation: the simulated user LM (red, A),guided by system prompt s A , and the agent LM (purple, B), with system prompt s B . The
user LM begins the conversation with a randomly selected starter prompt a1 . (B) After theconversation reaches a preset length (8 rounds in our experiment), we test how the agent
LM adheres to its system prompt s B . At each turn i, we replace the original user message ai
in the conversation history with the probe question p B and ask the agent LM to generate
its answer for a second time. The answer is then judged by the stability measure f B (·) tocompute the stability score.
This paper explores whether chatbots maintain prompted behavior over lengthy dialogs.
Anecdotal evidence suggests that instruction stability may “degrade” over the course of a
dialog, with chatbot responses straying from what was specified by the prompt. Besides
being a potential problem for prompt engineering, the lack of instruction stability also carries
significant safety implications. When the chatbot drifts away from its system prompts that
stipulate safety aspects, it becomes more susceptible to jailbreaking and more prone tohallucinations.
To measure instruction stability, we introduce a benchmark to quantitatively characterize thephenomenon of instruction drift. Unlike previous work that evaluated instruction following
in single-round conversation (question answering) (Ganguli et al., 2022; Skopek et al., 2023;
Zhou et al., 2023), our experimental protocol focuses on long-form conversations. We test
LLaMA2-chat-70B and find it suffers a significant instruction drift, as shown in Figure 3.
This discovery leads us to investigate the cause of the drift and to propose a mitigation
method.
A natural guess is that instruction drift relates to the transformer attention mechanism.
When a chatbot generates a new token, it takes into account all previous tokens in the dialog
but with varying weights. One might speculate that the longer the dialog, the less weightis placed on the initial tokens that make up the prompt. We measure this effect precisely
and find that there is indeed a strong attention decay effect. Intuitively, it seems plausiblethat the prompt’s efficacy will decrease as attention to initial tokens wanes. We back up
this intuition mathematically by showing that, in an idealized model, the space of possible
outputs from a language model will steadily enlarge over time.
Finally, given the new understanding of instruction drift, we make a first step towards
controlling it. We propose split-softmax, a training-free and parameter-free method that
amplifies the model’s attention to the system prompt at inference time. By comparing it witha strong prompting-based baseline and a recent technique from the literature (Sanchez et al.,2023), we demonstrate how split-softmax provides a better trade-off between performanceand stability.
This paper presents four contributions. (1) We provide a quantitative benchmark for evaluating instruction drift that does not depend on human annotation or API calls to proprietary
LLMs. This reproducible benchmark enables the measurement of progress in controlling
instruction drift for both open- and closed-source models (Section 3); (2) We discuss thephenomenon of attention decay and theoretically explain why it may occur (Sections 4
and 4.3); (3) We hypothesize that attention decay is the cause of instruction drift and devise
a simple technique called split-softmax as a first step towards controlling it (Section 5.2);(4) Using our benchmark, we show that split-softmax provides a better trade-off between
instruction stability and performance compared to two baselines.
2

Preprint

System Prompt s A
System Prompt s B
Conversation Starter a1
Probe Question p B

You are very happy! Always respond with lots of joy.
Always reply in French.
What’s your take on celebrity culture?
What do you do in London as a tourist?

Stability measure f B (·)
Table 1: Examples of required material for our experimental protocol.

2

Related Work

Prompting Prompting has become the go-to method for adapting language models to
downstream use cases. Among the more popular techniques are in-context learning (Minet al., 2022) and chain-of-thought prompting (Wei et al., 2022). Despite being flexible,
prompting cannot match the performance of fine tuning (Mosbach et al., 2023; Lu et al.,2021). For dialog systems based on large language models, a system prompt is placed at the
beginning of context window to define the general behavior of the chatbot. In the line ofprompting, we test a simple remedy that repeats the system prompt many times before eachuser utterance in Section 5.
Instruction Tuning Instruction tuning has been widely adopted to further align the model
to task instructions after pre-training (Gupta et al., 2022; Wei et al., 2021). Given pairsof inputs and outputs that follow the instruction, the model is fine-tuned to generate thedesired output. For the purpose of mitigating instruction drift, instruction tuning has playeda major role, especially in addressing safety concerns using RLHF Ouyang et al. (2022).
However, instruction tuning has a high cost of collecting training data and is not as flexible
as prompting.
Controlled Decoding Controlled decoding methods can be adapted to avoid instruction
drift. Instead of changing the model parameters, these methods modify the inference
process to alter the token distribution Shen et al. (2017); Dathathri et al. (2019); Krause et al.
(2020); Li et al. (2023a). For example, for a certain prompt, Todd et al. (2023) find a setof function vectors in the model’s hidden space that could be added to novel prompts tosteer the model outputs. This can be thought of as a way to distill the prompt without
repeating it in the context window. Weston & Sukhbaatar (2023) propose System-2 attention,
where the language model first decides where to attend to before making the final responses.
Classifier-free guidance (CFG) (Sanchez et al., 2023) works by running the model twice,once with and once without the system prompt, and computing the next token distribution
by a scaled contrast of the two distributions. We will evaluate CFG in our experiments
in Section 5.
Studies of Instruction Following in Dialog Systems Li et al. (2023b); Wu et al. (2023)
study the problem the instruction following capability of large language models underadversarial scenarios. Concurrent to this work, Zhou et al. (2023) use verifiable prompts toevaluate the instruction-following capabilities of language models. However, they all focus
on one-turn situations without user input. Zeng et al. (2023) emphasize the difficulty for
language model to evaluate instruction-following even using close-source language models,
motivating us to use deterministic functions for evaluation.

3

Measuring Instruction Drift

We aim to quantify instruction drift without the need for human judgment or API calls ofproprietary LLMs. To that end, we introduce a simple experimental protocol, along with abenchmark dataset.
3.1

Experimental Protocol

The idea behind the protocol is straightforward: to measure instruction drift, we create a synthetic dialog between two chatbots A and B and evaluate how far the dialog [ a1 , b1 , a2 , b2 , ...]
3

Preprint

drifts from the original prompts. To automate this process, we need four elements: twosystem prompts s A , s B , a conversation starter a1 , a probe question p B , and a stability
measure f B (bi ). Table 1 shows an example set of these elements.
The protocol consists of the following two steps ( Figure 2):
1. Given the two system prompts, s A for the user LM and s B for the agent LM, we
pit two copies of the same chatbot against each other but with different systemprompts, as specified by their different system prompts. The agent LM is the agent
under test for its instruction stability. We then create a synthetic multi-round dialog
between the two chatbot instances by feeding each one’s response to the other.
The user LM speaks first with a randomly sampled conversation starter a1 . Suchsimulation yields a conversation history {( ai , bi )}iN=1 , where N is the total numberof rounds* . We use N = 8 in our experiments.
2. To measure how well the agent LM follows its system prompt during the courseof the conversation, in the i-th round, the user LM, instead of making its original
prompt ai , asks the predefined probe question p B . Checking the returned answer bi′
with f B (·), we get a quantitative indication of how well the original system prompts B is followed. We call f B (bi′ | ai = p B ) instruction stability. The stability measurefunction can be Python code that calls a library to determine the confidence that areply is in French.
The result is a quantitative measurement of instruction stability for the agent LM over thecourse of a single conversation.
3.2

Benchmark Dataset

Of course, no single conversation can yield statistically significant results. To assess thedegree to which a chatbot is vulnerable to instruction drift, we need to average the resultsof many conversations. We manually curate a benchmark set of 100 system prompts,
categorized into 5 categories: multi-choice responses, character of the agent, answer-string
format pattern, memorization of certain facts, and languages the agent speaks. Each systemprompt s B comes with its own probe question p B and stability measure f B (·), expressed
as a Python function. Each stability measure f B (·) takes as input the agent LM’s responsebi and returns a number p in the range 0 ≤ p ≤ 1 deterministically; the larger the valueof p, the better the system prompt is followed. Table 1 shows one such triplet of systemprompt, probe question, and stability measure. We will release the full dataset as well as theconversation starters we use.
3.3

Experimental Results

We use this protocol and benchmark data to measure instruction drift in LLaMA2-chat-70B
and gpt-3.5-turbo-16k (Appendix D). Averaging the instruction stability scores across 200
conversations configured with random pairs of system prompts, we arrive at the blue line
in Figure 3 A. We observe that the agent LM gradually stops following its system prompts,aligning with our empirical daily usage experiences.
As a side experiment, we are curious if the agent LM adopts the user LM’s system prompt.
This is plausible since the user LM’s utterances generated according to p A have a strong
appearance in the context window. For this purpose, we swap ai with p A and checkf A (bi′ | ai = p A ). Surprisingly, the agent LM even gradually adopts the instruction of the user
LM over extended rounds of conversation, as shown by the orange line in Figure 3 A. This
could potentially be exploited by adversarial attacks, raising serious safety concerns.
In another safety check (Figure 3 B), we ablate the system prompt of the user LM with
an empty string, so it falls back to the default mode of the underlying language model.
This rules out the possibility that this could contribute to the significant instruction drift
discovered earlier.
*A “turn” is one utterance like a2 ; a “round” is when each chatbot takes a turn like a2 , b2

4

PreprintA

Instruction Stability

Instruction Stability

B

Figure 3: (A) The phenomenon of instruction drift. As the interaction progresses, not only
does the agent LM lose stability to its original system prompt, but it also begins to adopt the
instruction of the simulated user LM. The effects were measured on 200 randomly sampledpairs of system prompts on LLaMA2-chat-70B using the procedure shown in Figure 2. The
error bar represents one standard deviation. (B) Measuring instruction stability of the agent
LM when user LM’s system prompt is set to an empty string.
Experiment details. We use LLaMA2-chat-70B for this experiment and follow the formatof composing input sequence from Touvron et al. (2023). Taking the perspective of agent
LM as an example, the input sequence looks like [s B , a1 , b1 , . . . , ai−1 , bi−1 , ai ], and it is tasked
with generating bi as a reply to the last utterance from user LM.† Each s, a, and b here is astring and may contain multiple tokens. Generation is performed with temperature 1.0 andnucleus sampling with p = 0.9 (Holtzman et al., 2019).

4

Attention Decay: a Hypothesis

It is reasonable to hypothesize that instruction drift results from a decaying influence of theprompt over time. To investigate why this happens, we focus on the attention distribution
over context tokens in transformer self-attention heads. Although the intuitive hypothesis
broadly captures the underlying phenomenon, our empirical and theoretical analyses
uncover nuanced discrepancies.

Figure 4: The phenomenon of attention decay demonstrated in the 11th attention head in
the 24th layer of LLaMA2-7B, which has a maximum context window size of 4, 096 tokens.
We generate 12 conversations while tracking the portion of attention allocated to systemprompt tokens. The plots are specifically for the agent LM, grouped by the rounds in which
the answers are generated; the values are absent for the user LM. We observe sharp drops inattention between turns and rough plateaus within turns.
† Omitting formatting tokens like 
, 
 or 
.

5

Preprint

4.1

Preliminaries

Suppose the input tokens are {wi }it=1 , each belonging to the vocabulary V. To generate thenext token wt+1 ∈ V, the current tokens are first embedded into D-dimensional vectors{h0i }it=1 with the embedding matrix We ∈ R|V |× D . These are then processed sequentially by
L transformer layers, resulting in a grid of activations after each layer and for each token{hil }it,L=1,l =1 . As the multi-layer perception (MLP) and layer norm are context-independent,
we leave them out for simplicity. The feed-forward process of the transformer can besummarized as:Hhil = hil −1 + ∑ Wol,m Attl,m (h1l −1 , . . . , hil −1 ),(1)

wt+1 ∼ p(w|w≤t ) = softmax(We htL ).

(2)

m =1

The combination of the softmax and We work as a predictor from htL to distribution p(w|w≤t )of next token wt+1 . Attl,m is the single head attention operator with output in a lower
dimensional space and Wol,m ∈ RD×d maps them back into RD , the residual stream space.
Crucial to our experiment, we expand the attention operator to show it aggregates activations from previous time steps based on an attention distribution:!
l −1 ⊤(Wkl,m h1:t
) (Wql,m hlt−1 )l,m√αt,j=1:t = softmax.
(3)dThen the attention operation is a weighted sum of linearly transformed activations from thelast layer:tAttl,m (h1l −1 , . . . , hlt−1 ) = ∑ αl,m
Wvl,m hlj−1 ,
t,j

(4)

j =1

where Wvl,m ∈ Rd× D , Wkl,m ∈ Rd× D , Wql,m ∈ Rd× D are the value, key, and query weightmatrices, respectively.
4.2

The Phenomenon of Attention Decay

While generating the next token given an input sequence containing t tokens, in each attention head, the last token will compute a normalized attention distribution over all previous
tokens (including itself), denoted by αt,i=1:t in Equation 3. Tokens in the system prompt area special subset of all previous tokens, and we denote the sum of the attention weights allo|s |cated to them as π (t) = ∑i=B1 αt,i . It ranges between 0 to 1 and represents the comparative
importance that the system prompt has throughout the generation process. We monitor
this percentage π (t) along the decoding time steps t and across turns of conversations in
LLaMA2-7B. We only plot π (t) from the perspective of the agent LM.
As shown in Figure 4, within each turn, π (t) remains almost constant, but there are significant decreases across turns. This observation runs our a naive hypothesis of attention
decay—if the attention distributes uniformly over previous tokens, π (t) should decay
hyperbolically and be independent of number of turns.
It’s also worth-noting that this highlights a unique issue in chatbots, distinct from language
models, where out-of-distribution text from interlocutors is absent. The case of the language
model completing its input partial sequence is technically equivalent to the agent LM
generating answers for a single turn, which displays a plateau in π (t).
This observation shows merely the co-occurrence of instruction drift and attention decay. However, it inspires the hypothesis that attention decay may internally contributeto instruction drift, suggesting that addressing the former could help mitigate the latter(Section 5.2).
6

Preprint

4.3

A Geometric View of Attention Decay

To shed light on attention decay in Figure 4, both the plateau within utterance and the drop
across utterances, we provide a theoretical explanation in a simplified situation. Liang et al.
(2022) show empirically and theoretically that the internal representation of deep neuralnetworks usually live in a narrow cone in the high-dimensional space. Motivated by theirobservations, we characterize attention decay from a similar geometric perspective.
We will consider two settings of model generation:
1. New tokens are generated autoregressively given initial tokens h1 , . . . , h|sB | , whichmodels the process of the agent LM generating answers;
2. New tokens are drawn by the user. A user LM could put out-of-distribution tokens
into the context window of agent LM in a potentially adversarial fashion (Zou et al.,2023).
For the first setting, we will show that tokens generated by the model always remain in an
approximately low-dimensional convex cone in Theorem A.1. In the second setting, we
can characterize the expansion using spherical measure and show that randomly drawn
tokens will lead to an expansion of the underlying convex cone with the growth of intrinsic
dimension of token embeddings, as shown in Proposition A.3. More details in Appendix A.

5

Mitigating Instruction Drift

If instruction drift is related to attention decay, that suggests we can mitigate drift bymanipulating the level of attention on the original prompt. Before presenting an attentionbased mitigation method, however, we describe two baselines.
5.1

Baseline Methods

System Prompt Repetition (SPR) We inject the system prompt with probability 0 ≤ p ≤ 1
before each user utterance. The repeated system prompts, like the standard system prompt
at the start of the input sequence, only appear when the language model is prompted; users
do not see them.
Classifier-Free Guidance (CFG) The second method is classifier-free guidance(CFG, Sanchez et al., 2023), which runs the base model twice, firstly with system prompt
to get log p(w|w≤t , s B ) and then without system prompt to get log p(w|w≤t ). It then uses
a contrastive linear operation inside the logit space to strengthen the effects of the systemprompt on answer generation. The new next-token probability distribution is defined by:log p̂(w|w≤t , s B ) = log p(w|w≤t ) + α(log p(w|w≤t , s B ) − log p(w|w≤t )).

(5)

CFG comes with a hyperparameter α ≥ 1 that controls how far we shift the predicted logits.
When α = 1, it reduces to prompting with the system prompt; larger α produces stronger
intervention.
5.2

Proposed Method: Split-softmax (SS)

Motivated by the attention decay phenomenon, we introduce a method that requires noretraining, split-softmax, aimed at reducing this decay with minimal overhead. The basic
idea is straightforward: if the problem is that the model pays too little attention to theprompt, then force the model to pay more. In practice, we find that a power-law scaling ofattention seems to be effective.
In particular, split-softmax (SS) works by inserting a scaling operation between Equation 3
and Equation 4 for every attention operation. After obtaining the attention distribution
7

Preprint{αt,i }it=1 which sums up to 1 (omitting superscript for simplicity), we reweight it by: k|s B |
 π (t) αt,i
if i ≤ |s B |π (t) = ∑ αt,i , α′t,i = 1π−(πt)k (t),αt,i if i > |s B |
i =1

(6)

1− π ( t )where the introduced exponent 0 ≤ k ≤ 1 as a hyperparameter to control the strength of ourintervention. The smaller k is, the stronger the intervention is; when k = 1, the intervention
is nullified. The new set of attention {α′t,i }it=1 sums up to 1 as well and will replace {αt,i }it=1
so that more attention is paid to the system prompt tokens. Given 0 ≤ π (t) ≤ 1, 0 ≤ k ≤ 1
π k (t)thus π (t) ≥ 1, split-softmax increases the proportion of attention paid to system prompts.
See Appendix E for more discussion.
5.3

Calibration Using Performance Drop on MMLU

Each method (split-softmax and thetwo baselines) represents a potentiallylarge intervention; any instruction stabilization may come at the expense ofother capabilities of the model. However, each method has a hyperparameter that corresponds to the strength of
the intervention. To compare methods,
therefore, we need to measure both theincrease in instruction stability and theperformance drop for various valuesof the relevant hyperparameter. Thisis analogous to measuring a precisionrecall curve for a classifier.
Tomeasureany
performancechanges, we use the Massive Multitask
Language
Understanding(MMLU, Hendrycks et al., 2020). To
compare the different methods, lookat the stability improvement at equallevels of performance drop. Sweeping
hyperparameters for each method
allows us to measure and plot eachmethod’s stability-performance curve,
revealing different trade-offs between
our stability metric and MMLU
performance.

Figure 5: Comparing trade-offs between instruction stability and performance. For each of thethree methods, we vary a hyperparameter that reflects the strength of the intervention. Each curveplots the effect on stability and performance overthe hyperparameter sweep. Compared to two baselines (classifier-free guidance and system prompt
repetition), split-softmax produces equal or higher
As expected, we do see an inverse restability for a given level of performance degradalationship between performance andtion.
instruction stability in all three of our
methods Figure 5. This corroborates
earlier findings by Gu et al. (2024) that control methods over language model often come at
the cost of general capability. The performance drop on MMLU should be thought of as abudget when correcting model behaviors, and two methods should only be compared onstability when their respective hyperparameters cause similar MMLU performance drop.
To quantify stability, we use a 16-turn conversation as described in Figure 2. We modify
these conversations by applying each method to the agent LM. Then we probe the agent
LM at each round to test its instruction stability in the same fashion as section 3. Stability ismeasured for individual turns, and the overall stability measure is the average of the stability
at each turn of agent LM. Given the conversation history of agent LM under intervention,
we sample one and ask questions from MMLU at an intermediate turn (the 4th turn in our
8Preprint

Figure 6: Comparison of instruction stability across turns, with MMLU performance droparound the value of 0.5, for system prompt repetition (SPR), classifier-free guidance (CFG),
and split-softmax (SS). The whisker represents one standard deviation.
experiments); and the answers are used to calculate MMLU accuracy. Note that due tothe added system prompt and chat history, the MMLU performance is different from what
is reported by LLaMA team even without intervention (Touvron et al., 2023). However,
only the difference between post- and pre-intervention performances is meaningful, as theprimary purpose of using MMLU in our case is to calibrate the strength of the intervention.
5.4

Experimental Results

All experiments are conducted on LLaMA2-70B-chat. To save computational cost, we choose
one system prompt from each of the five categories, and run experiments over the total
twenty ordered pairs of system prompts.
In Figure 5 we plot instruction stability versus performance drop on MMLU as we vary
the strength hyperparameter for each method. In general, split-softmax presents a better
trade-off between performance drop and instruction stability. It can match performancewith system prompt repetition while avoiding using the additional context window. If
more drop in performance on MMLU is allowed, split-softmax enables greater instruction
stability.
In Figure 6, we break down the instruction stability measurement across turns. Similar
to what Sanchez et al. (2023) show, classifier-free guidance helps the model adhere to thesystem prompt remarkably well for the first round of the conversation, but it does not
generalize well into extended conversations. Both system prompt repetition and splitsoftmax demonstrate higher effectiveness in mitigating instruction drift, though they exhibit
different trends. The former excels in regions with a larger number of turns, while the latter
performs better at the beginning of the conversation. Note that system prompt repetition
consumes a substantial portion of the context window.

6

Conclusions and Future WorkOur experiments indicate that instruction drift is a potentially significant issue for prompt
engineering. To help address this challenge, we contribute a new protocol and benchmark
to help measure this phenomenon, as well as an idealized mathematical model of its cause.
In addition, we proposed a technique, split-softmax, that can help mitigate instruction drift,
providing a better stability-performance trade-off than two existing baselines.
There is ample room for future work in this space. For example, it would be natural to explore making changes in architecture or to training to combat instruction drift. Furthermore,
all the techniques we discussed involve an apparent trade-off between performance andreliability. Is this a necessary compromise, or are there methods that maintain instruction
stability at no cost? It would also be good to deepen our theoretical understanding, adding
realism to the idealized “cone” model of instruction drift that we proposed. Finding new
ways to measure and prevent instruction drift is an important step in ensuring AI safety
and reliability.
9Preprint

Acknowledgments
We thank Jiawei Zhou for useful discussions and feedback on the manuscript.
KL is supported by a fellowship from the Kempner Institute for the Study of Naturaland Artificial Intelligence at Harvard University. DB is supported by a grant from OpenPhilanthropy. This work has been made possible in part by a gift from the Chan Zuckerberg
Initiative Foundation to establish the Kempner Institute for the Study of Natural andArtificial Intelligence. This work was partially supported by NSF grant IIS-1901030.

References
LE Blumenson. A derivation of n-dimensional spherical coordinates. The American Mathematical Monthly, 67(1):63–66, 1960.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino,
Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach tocontrolled text generation. arXiv preprint arXiv:1912.02164, 2019.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath,
Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language
models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint
arXiv:2209.07858, 2022.
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, andNanyun Peng. Model editing can hurt general abilities of large language models. arXiv
preprint arXiv:2401.04700, 2024.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P
Bigham. Improving zero and few-shot generalization in dialogue through instruction
tuning. arXiv preprint arXiv:2205.12673, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, andJacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint
arXiv:2009.03300, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neuraltext degeneration. arXiv preprint arXiv:1904.09751, 2019.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty,
Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided
sequence generation. arXiv preprint arXiv:2009.06367, 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.
Inference-time intervention: Eliciting truthful answers from a language model. arXiv
preprint arXiv:2306.03341, 2023a.
Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian
Journal of Mathematics & Statistics, 4(1):66–70, 2010.
Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, and Hongxia
Jin. Instruction-following evaluation through verbalizer manipulation. arXiv preprint
arXiv:2307.10558, 2023b.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mindthe gap: Understanding the modality gap in multi-modal contrastive representation
learning. Advances in Neural Information Processing Systems, 35:17612–17625, 2022.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantasticallyordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.
arXiv preprint arXiv:2104.08786, 2021.
10Preprint

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi,
and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context
learning work? arXiv preprint arXiv:2202.12837, 2022.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Fewshot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint
arXiv:2305.16938, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin,
Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language
models to follow instructions with human feedback. Advances in Neural Information
Processing Systems, 35:27730–27744, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint
arXiv:2306.17806, 2023.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from nonparallel text by cross-alignment. Advances in neural information processing systems, 30,
2017.
Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. Towards better
evaluation of instruction-following: A case-study in summarization. arXiv preprint
arXiv:2310.08394, 2023.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David
Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei,
Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2:
Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. Can chatgpt write a goodboolean query for systematic review literature search? arXiv preprint arXiv:2302.03495,
2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan
Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.
arXiv preprint arXiv:2109.01652, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language
models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
James G Wendel. A problem in geometric probability. Mathematica Scandinavica, 11(1):109–111, 1962.
Jason Weston and Sainbayar Sukhbaatar. System 2 attention (is something you might needtoo). arXiv preprint arXiv:2311.11829, 2023.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung
Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint
arXiv:2307.02477, 2023.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluatinglarge language models at evaluating instruction following. arXiv preprint arXiv:2310.07641,
2023.
11

Preprint

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny
Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv
preprint arXiv:2311.07911, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei,
Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable
adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

12

PreprintA

Sketch of the Theory in Section 4.3

A.1

Setting One: Agent Utterances

In linear algebra, a cone is a subset of a vector space that is closed under positive scalarmultiplication. In other words, C is a cone if x ∈ C implies sx ∈ C for every positive scalar s.
Moreover, C is called a convex cone if αx + βy ∈ C for any positive scalars α and β, and any
x, y ∈ C.
The dimension of a cone is the dimension of the vector space spanned by the elements ofthe cone. For convenience, we define two new notions related to low dimensional cones inthe space RD . Given any d-dimensional convex cone C ⊂ RD (1 ≤ d ≤ D), for ϵ ∈ (0, 1) we
define the corresponding ϵ-approximate d-dimensional cone as
C ϵ := {w ∈ C ⊕ span(C )⊥ ⊂ RD : w = u + v∼ RD−d , ∥v∥ ≤ ϵ∥w∥}.
for some u ∈ C, v ∈ span(C )⊥ =Given some c ∈ SD−1 and θ ∈ (0, π/2), a d-dimensional spherical cone is the set defined by
Pd [c, θ ] := {u ∈ U ⊂ RD : U ∼= Rd , ⟨c, u⟩ ≥ ∥u∥ cos θ }.
Theorem A.1. Assume that the token embeddings of the system prompt given by h1 , . . . , h|sB | lie inl,mthe d-dimensional approximate cone C ϵ , and that any output-value matrix Wov= Wol,m Wvl,m ∈l,mRD× D satisfy that Wovu ∈ C ϵ for any u ∈ C ϵ . Then all proceeding tokens generated by oursimplified transformer lie in the convex hull of C ϵ . In particular, if C ϵ is contained in somespherical
cone Pd [c, θ ] , then all generated tokens lie in the ϵ̃-approximate cone C ϵ̃ where ϵ̃ =p2ϵ/ ϵ + cos2 θ (1 − ϵ2 ).

For the initial tokens, θ indicates how concentrated their embeddings are, and d is roughly
the intrinsic dimension of these embeddings. Note that d ≤ |s B | and the number of tokens
in the system prompt |s B | is usually much smaller than the dimensions of hidden space D,
which is 8192 in the case of LLaMA2-70B-chat. Thus, the assumption that initial embeddings
occupy a low-dimensional cone is reasonable.
Theorem A.1 shows the convex cone for token embeddings remains stable during thegenerating process if there is no user input, which leads to the plateau within an utterance.
A.2

Setting Two: User Utterances

Again we assume that the system tokens h1 , . . . , h∥sB ∥ are from some C0ϵ , and let Cn be
the smallest convex cone containing C0 and user tokens { h|sB |+i }in=1 . Then the expansion
C0 ⊂ C1 ⊂ · · · ⊂ Cn reflects the attention decay under the influence of user utterances. To
get some intuition on the expanding process, we show the following:
Proposition A.2. If user tokens are drawn i.i.d. uniformly from SD−1 , then with probability 1 − ηafter n ≥ 4D + 2 log η1 user tokens Cn expands to the whole space RD .
Proposition A.2 suggests that when user utterances are inserted, the size of the convex cone
for token embeddings will grow significantly, which gives rise to the drop of π (t) acrossutterances. To further quantify the expansion of convex cones, we can consider the sphericalmeasure σD−1 , which is the Borel measure on the ( D − 1)-sphere such that σD−1 (SD−1 ) = 1.
For any ϵ-approximate convex cone C ϵ , define the volume of C ϵ byµ(C ϵ ) := σD−1 (C ϵ ∩ SD−1 ).
Then intuitively µ(C0ϵ )/µ(Cnϵ ) indicates the degree to which the current tokens in Cnϵ alignwith the system tokens in C0ϵ , similar to the quantity π (t) defined in the previous section.
In real applications, user messages are not i.i.d. uniform variables from SD−1 . However,there usually exists an evident proportion of user tokens distinct from the system tokens.
13

Preprint

They could probably be tokens unique in the specific topics that the user inquires about or,more typically, tokens from a new language. It could also happen that the user is attacking
the LM by sending adversarial tokens (Zou et al., 2023). The following proposition quantifies
how attention decays in terms of µ(C0ϵ )/µ(Cnϵ ) as such embedding dimension increases.
Proposition A.3. Suppose C0 is a d1 -dimensional convex cone contained in some d1 -dimensional
spherical cone Pd1 [c1 , ψ1 ] while Cn is a d2 -dimensional convex cone containing a d2 -dimensional
spherical cone Pd2 [c2 , ψ2 ]. Then we haveµ(C0ϵ )
≲ ϵ d2 − d1 .
µ(Cnϵ )The geometric perspective we proposed provides a concrete explanation of why insertinguser prompts will cause attention decay while autoregressive generation from the model
will almost have no harm. However, one limitation here is that we have only compared thecone structure without tracking the distribution of token embeddings within the cones. In
particular, if we force the majority of tokens generated from Cnϵ to be contained or close toC0ϵ , the issue of attention decay could possibly be mitigated, which motivates our method in
the proceeding section.

B

Proofs for Appendix A

We start by making simplifications to the model and token-generating process. First, themodel is simplified by omitting the MLP and layer norms as in Equation 1. For the tokengenerating process, the embedding of the next token ht+1 is close to htL among all tokens in
the vocabulary in Equation 2. Thus, for convenience we directly put ht+1 := htL /∥htL ∥ in oursimplified model, meaning that all embeddings lie on the unit hypersphere SD−1 := {v ∈R D : ∥ v ∥ = 1}.
Proof of Theorem A.1. Let C ϵ be the convex hull of C ϵ . The C ϵ is a convex cone containing C ϵ .
Theorem A.1 can be proven in two steps.
Step I. We establish that ht ∈ C ϵ by induction. h1 , . . . , ht0 already satisfy the claim byassumption. Supposing that h1 , . . . , ht ∈ C ϵ (t ≥ t0 ), we show that ht+1 is also in C ϵ . Here
we look into hlj (j = 1, . . . , t, l = 1, . . . , L) in the process of generating ht+1 . We perform
induction on l. For l = 0, we have hlj = h j ∈ C ϵ . Supposing that hlj ∈ C ϵ for j = 1, . . . , t, it
suffices to prove that hlj+1 ∈ C ϵ .
By induction hypothesis that hlj ∈ C ϵ (j = 1, . . . , t) we can find k j ∈ N+ , x j,1 , . . . , x j,k j ∈ C ϵ ,and w j,1 , . . . , w j,k j > 0 for j = 1, . . . , t such that
kj

hlj =

∑ w j,i x j,i .

i =1Thus, by Equation 1 we haveH

hlj+1 = hlj + ∑ Wol +1,m Attl +1,m (h1l , . . . , hlj )
m =1
H

j

= hlj + ∑ ∑ αlj,s+1,m Wol +1,m Wvl +1,m hls
m =1 s =1
Hjks= hlj + ∑ ∑ ∑ αlj,s+1,m ws,i Wol +1,m Wvl +1,m xs,i .
m =1 s =1 i =1

14

Preprint

Note that αlj,s+1,m > 0 since it is calculated from softmax and by assumption we have
Wol +1,m Wvl +1,m xi,s ∈ C ϵ as xs,i ∈ C ϵ . Thus, we conclude that hlj+1 ∈ C ϵ . By induction weknow for l = 1, . . . , L and j = 1, . . . , t we have hlj ∈ C ϵ . Thus, ht+1 = htL /∥ htL ∥ ∈ C ϵ holds.
And by induction again we conclude that ht ∈ C ϵ for all t ≥ 1.
p
Step II. Let γ = cos θ. We prove that C ϵ ⊂ C ϵ̃ where ϵ̃ = ϵ/ ϵ2 + γ2 (1 − ϵ2 ). For anyy ∈ C ϵ , there exists k ∈ N+ , x1 , . . . , xk ∈ C ϵ , and w1 , . . . , wk > 0 such that y = ∑ik=1 wi xi .
By definition of C ϵ , xi can be written as xi = ui + vi where ui ∈ C and vi ∈ span(C )⊥and ∥vi ∥ ≤ ϵ∥ xi ∥. By definition of Pd [c, θ ] we have ⟨c, ui ⟩ ≥ γ∥ui ∥ for all i = 1, . . . , k. Letũi := ⟨c, ui ⟩c. Then ⟨ũi , ui − ũi ⟩ = 0 and hence ⟨∑ik=1 wi ũi , ∑ik=1 wi (ui − ũi )⟩ = 0. Therefore,
we have
E
k
k
k D
k
k
w
u
≥
w
ũ
=
c,
w
u
≥
γ
i
i
i
i
i
i
∑
∑
∑ ∑
∑ wi ∥ u i ∥.
i =1

i =1

i =1

i =1

i =1

On the other hand, we know
k

k

i =1

i =1

k

ϵ

∑ wi v i ≤ ∑ wi ∥ v i ∥ ≤ √1 − ϵ2 ∑ wi ∥ u i ∥.

Therefore, it holds thatk

∑ wi u i

i =1

i =1

√
γ 1 − ϵ2≥ϵk∑ wi v i ,i =1

which implies thatkϵk

∑ wi v i ≥ p ϵ2 + γ2 (1 − ϵ2 ) ∑ wi x i .
i =1

i =1

Thus, we conclude that C ϵ ⊂ C ϵ̃ .
To prove Proposition A.2 we need the following lemma.
Lemma B.1 (Wendel, 1962). Let N points be scattered uniformly at random on Sm ⊂ Rm+1 . Thenthe probability that all points lie on some hemisphere is given by

m 
N−1
− N +1
am,N = 2.
∑kk =0Proof of Proposition A.2. If there is no hemisphere containing ht0 +1 , . . . , ht0 +n , then the origin
lies in Cn and is not on the boundary, meaning that Cn = RD . Thus, we only need to showthat for n ≥ 4D + log η1 , it holds that a D,n ≤ η. SinceD 

2− n ∑

i =0

n
i



It suffices to prove that 2−n
can check thatD en  DDni
D!  n i= 2− n ∑≤ 2− n
.
i!
i! D
D
i =0
i =0

≤ 2− n ∑
en D< η. For convenience let α := 4 + D2 log η1 ≤ Dn . Then weD
log 2 −1  α/2
1 1/De>.
2
ηNote that1eα(log 2− 2 )−1 ≥ α log 2 −1
,2

which is equivalent to1eα ≤eα(log 2− 2 )
2α
=
.
1
log 2 − 2eα/2 log 2 − 2115

Preprint

Thus, we have2− n

 en  DD≤(eα) D
1
≤
< η.
D
αD
2
log 2 − 12 eαD/2

To show Proposition A.3 we need the following lemma.
Lemma B.2 (Li, 2010). The spherical measure of the spherical cap Pm+1 [c, θ ] ∩ Sm is given byR θ m −1
Z θ
sin
xdx
Γ ( m +1 )m +1m
σm ( P[c, θ ] ∩ S ) = R 0π/2
sinm−1 xdx,
= √ 2mπΓ( 2 ) 02
sinm−1 xdx
0where Γ( x ) is the Gamma function.
Proof of Proposition A.3. First we lower bound µ(Cnϵ ) by identifying as many disjoint spherical caps with angle θ := arcsin ϵ as possible and applying Lemma B.2.
Let M be the largest number such that there exists a set of points a1 , . . . , a M ∈ Pd2 [c2 , ψ2 −θ ] ∩ SD−1 to ensure P D [ ai , θ ] ⊂ Pd2 [c2 , ψ2 ] (i = 1, . . . , M) are disjoint from one anotherM(“disjoint” meaning that the measure of intersection is zero). We claim that Pd2 [ ai , 2θ ] i=1
S
is a covering of Pd2 [c2 , ψ2 ]. Otherwise, choosing a0 ∈ Pd2 [c2 , ψ2 ] ∩ SD−1 \ i Pd2 [ ai , 2θ ] wecan check that P D [ a0 , θ ] does not intersect with any of P D [ ai , θ ]. Thus, these M + 1 sphericalS
caps do not overlap, which contradicts the definition of M. Hence Pd2 [c2 , ψ2 ] ⊂ i Pd2 [ ai , 2θ ],and by Lemma B.2 we haveΓ( d22 )Z ψ2

sind2 −2 xdx = σd2 −1 ( Pd2 [c2 , ψ2 ] ∩ SD−1 )d2 −10πΓ( 2 )Z 2θM
Γ( d22 )sind2 −2 xdx.
≤
σd2 −1 ( Pd2 [ ai , 2θ ] ∩ SD−1 ) = Mσd2 −1 ( Pd2 [ ai , 2θ ]) = M √
d2 −1
0
πΓ
(
)
i =1
2

√

∑

On the other hand, since P D [ ai , θ ]’s are disjoint from each other and that P D [ ai , θ ] ⊂P D [c2 , ψ2 ] (because ϵ = sin θ), we knowMµ(Cnϵ ) ≥ ∑ σD−1 ( P D [ ai , θ ] ∩ SD−1 ) = MσD−1 ( P D [ ai , θ ] ∩ SD−1 )i =1

Γ( D2 )

Z θ

sinD−2 xdxπΓ( D2−1 ) 0RψRθΓ( D2 ) 0 2 sind2 −2 xdx 0 sinD−2 xdx.
≥
R 2θ d −22
Γ( D2−1 )
sin
xdx
0

= M√

Next we upper bound µ(C0ϵ ). For any ( x1 , · · · , xn ) ∈ Bn := {( x1 , . . . , xn ) : ∑in=1 xi2 ≤ 1}, we
introduce the hyperspherical coordinate system, which consists of a radial coordinate r, andn − 1 angular coordinates ϕ1 , . . . , ϕn−1 , where the angles ϕ1 , · · · , ϕn−2 range over [0, π ] and
ϕn−1 ranges over [0, 2π ). In specific, the coordinates are defined through the transformation:
x1 = r cos ϕ1 ,x2 = r sin ϕ1 cos ϕ2 ,x3 = r sin ϕ1 sin ϕ2 cos ϕ3 ,..
.
xn−1 = r sin ϕ1 · · · sin ϕn−2 cos ϕn−1 ,xn = r sin ϕ1 · · · sin ϕn−2 sin ϕn−1 .
16Preprint

By assumption we know C0 ⊂ P D [c1 , ψ1 ]. Therefore, using the notion of spherical elements(Blumenson, 1960), we can writeµ(C0ϵ ) = σD−1 (C0ϵ ∩ SD−1 ) =1Area(SD−1 )ZΩsinD−2 ϕ1 sinD−3 ϕ2 · · · sin ϕD−2 d(ϕ1 , . . . , ϕD−1 ),

where
n
o
d −1
Ω = (ϕ1 , · · · , ϕD−1 ) : ϕ1 ∈ [0, ψ1 ], ϕ2 , . . . , ϕD−2 ∈ [0, π ], ϕD−1 ∈ [0, 2π ], ∏ j=1 1 sin ϕj ∈ [0, ϵ] .
Denotingnod −1Ω1 = (ϕ1 , · · · , ϕd1 −1 ) : ϕ1 ∈ [0, ψ1 ], ϕ2 , . . . , ϕd1 −1 ∈ [0, π ], ∏ j=1 1 sin ϕj ∈ [0, ϵ] ,then we haveµ(C0ϵ ) =1
Area(SD−1 )
Z π

···

≤
=
=

(ϕ1 ,...,ϕd1 −1 )∈Ω1

Z π Z 2π

0

=

Z

0
0
Z
D
−
d
1
)
Area(S

Area(SD−1 )sinD−2 ϕ1 · · · sinD−d1 ϕd1 −1 d(ϕ1 , . . . , ϕd1 −1 )sinD−d1 −1 ϕd1 · · · sin ϕD−2 dϕd1 · · · dϕD−1
sinD−2 ϕ1 · · · sinD−d1 ϕd1 −1 d(ϕ1 , . . . , ϕd1 −1 )(ϕ1 ,...,ϕd1 −1 )∈Ω1

Area(SD−d1 ) D−d1ϵArea(SD−1 )Z ψ Z π
1

···

Z π

0
0
D
−
d
d
−
1
1
1
Area(S)Area(S)

2Area(SD−1 )Γ( D2 )

ϵ D − d1
Γ( D−d21 +1 )Γ( d12−1 )0sind1 −2 ϕ1 · · · sin ϕd1 −2 dϕ1 · · · dϕd1 −1

σd1 −1 ( Pd1 [c1 , ψ1 ] ∩ SD−1 )

Z ψ

1

0

sind1 −2 xdx.

Thus, we conclude thatRψ
R 2 arcsin ϵ d −2
sin 2 xdx D−dΓ( D−d21 +1 )Γ( d12−1 ) 0 1 sind1 −2 xdx 0µ(C0ϵ )
1
≤
ϵ
R
R
ψ2
arcsin ϵµ(Cnϵ )
Γ( D2−1 )sind2 −2 xdx
sinD−2 xdx
00

ϵ d2 −1≲ ϵ D − d1 D −1 = ϵ d2 − d1 .
ϵFigure 7: Histogram of embedding vector norms.

17Preprint

Norm of Embedding Vectors In Section 4, we assume that the embedding vectors havethe unit norm. To verify if this is reasonable, we plot the density of the norms of vocabularyembeddings for the LLaMA2-7B-chat in Figure 7. We can observe that the norms are quite
concentrated around 1.

C

Does RLHF help?

Given how RLHF Ouyang et al. (2022); Ziegler et al. (2019) train the model, the model
should be trained to pay more attention to the system prompt so to increase user satisfaction.
In Figure 8, we show that RLHF could increase the portion of attention paid to the systemprompts by comparing LLaMA2-7B and LLaMA2-7B-chat. The latter is trained on top of theformer with human feedback. It shows that RLHF indeed helps in combating instruction
drift, but it still cannot eradicate it entirely due to its nature of fine-tuning.

Figure 8: Comparison of attention decay between LLaMA2-7B before and after RLHF
training. Different from the categorical palette used in Figure 4 to differentiate number ofrounds when the answer is generated. The deeper the color, the later the round in which theanswer is generated.

D

Additional Instruction Drift Experiments

To see how close-source model compares with LLaMA2-70B-chat, we test
gpt-3.5-turbo-16k with a total of 200 randomly sampled system prompt pairs.
Results are shown in Figure 9. It turns out that gpt-3.5-turbo-16k holds to its systemprompt better than LLaMA2-chat-70B, but still suffers a 10% drop on the stability of its
original system prompt.

E

Discussion of Split-softmax Formula

We first quickly show how the post-intervention attention values in Equation 6 still form a
distribution by summing up to 1, dropping subscript t:∑ αi′ = ∑ αi′ + ∑ αi′i ≤|s B |i >|s B |π k (t)1 − π k (t)
αi +
∑
∑ αi
π (t) i≤|s |1 − π (t) i>|
sB |
B


= π k (t) + 1 − π k (t)

=

=1
18

Preprint

Figure 9: Measuring the instruction stability of gpt-3.5-turbo-16k via API using the same
protocol as Figure 3. On the left, the system prompt is given to the API via the “system”
argument; on the right, it is prepended to the user’s first utterance.
Meanwhile, it is worth-noting that the ratios of attention scores for tokens within the systemprompt and within conversation history remain unchanged, thereby minimizing disruption
to the attention mechanism.

19