Instruction Position Matters in Sequence Generation with Large Language Models (2024)

Abstract: Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. The fine-tuning data is generally a sequential concatenation of a specific task instruction, an input sentence, and the corresponding response. Considering the locality of self-attention modeling in LLMs, these models face the risk of instruction forgetting when generating responses for long input sentences. To mitigate this issue, we propose to enhance the instruction-following capability of LLMs by relocating the position of task instructions after the input sentences. Theoretical analysis suggests that our straightforward method can alter the model’s learning focus, thereby emphasizing the training of instructionfollowing capabilities. Concurrently, experimental results demonstrate that our approach consistently outperforms traditional settings across various model scales (1B / 7B / 13B) and different sequence generation tasks (translation and summarization), without any additional data or annotation costs. Notably, our method significantly improves the zero-shot performance on conditional sequence generation, e.g., up to 9.7 BLEU points on WMT zero-shot translation tasks. Further analysis reveals that our method can substantially enhance the model’s instruction-following ability by 1x compared to the traditional approach.

heading: 1 Introduction
text: In recent years, there has been a rapid emergence of large language models (LLMs) like ChatGPT and GPT-41, which have shown promising performance in various traditional natural language processing tasks (Wang et al., 2023; Jiao et al., 2023b; Kasneci et al., 2023; Hill-Yardin et al., 2023; Šlapeta, 2023; Aydın and Karaarslan, 2023). Meanwhile, there
* Corresponding author. 1https://chat.openai.com/chat
is also a growing interest in open source mediumsized language models, such as the LLaMA model with 13 billion parameters (Touvron et al., 2023) and the BLOOMZ language model with 7.1 billion parameters (Muennighoff et al., 2022), to meet research and hardware deployment requirements.
To align the outputs of language models with human intentions and unlock their full potential, InstructGPT (Ouyang et al., 2022) constructs a small amount of instruction-following data for finetuning LLMs and conducts reinforcement learning to align the model with human preferences. This approach to instruction following has gained widespread attention from both the academic and industrial communities (Brooks et al., 2023; Chung et al., 2022; Wei et al., 2022; Ahn et al., 2022; Wei et al., 2021; Aher et al., 2023).
Generally, the instruction-following data consists of three parts. Taking the machine translation task as an example, these parts include a specific task instruction (e.g., "Please translate the following paragraph from English to French"), an input sentence (the English sentence to be translated), and the final response (the corresponding French translation). Since most large language models are
11652
based on the decoder-only structure of the Transformer (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020), with a training objective of predicting the next token. Generally, these three parts of the instruction-following data are sequentially concatenated into a long nature sentence as the training data for language models. Given that the self-attention mechanism in Transformer decoders tends to focus more on nearby words i.e., the locality of self-attention modeling (Beltagy et al., 2020; Kitaev et al., 2019; Voita et al., 2019), there is a considerable risk of instruction forgetting when predicting responses for long input sentences. For example, when performing long text summarization tasks, the input sentence may contain thousands of tokens. Consequently, the model may be at risk of forgetting the initial task instruction when predicting responses, leading to the generation of responses that do not fully comply with the user’s intent. In this paper, we refer to this risk as the instruction forgetting issue.
To alleviate the above issue for LLMs during instruction fine-tuning, we first observe that the relative position of the input sentence and the task instruction is crucial. Therefore, we propose a simple and straightforward solution, namely, placing the task instruction at the end of the input sentence (referred to as ‘Post-Ins’). In this way, when the model predicts the final response, it naturally attends to the nearest preceding sequence, which is just the task instruction indicating what content should be generated next. For comparison, we refer to the data format in existing studies where the task instruction is concatenated to the front of the input sentence as Pre-Instruction (abbreviated as ‘Pre-Ins’).
To verify whether Post-Ins improves the instruction-following ability of language models and alleviates the instruction forgetting issue on long sentences compared to Pre-Ins, we first analyze the conditional probability characteristics of the models under both data formats with the trinomial Bayes formula. Through appropriate assumptions and formula derivations, we draw the following conclusions: (1) Pre-Ins tends to model a reverse conditional probability (e.g., reverse translation probability), emphasizing the coverage of the input sentence while insufficiently modeling the task instruction. (2) Post-Ins is more inclined to model a conditional probability about the task instruction (e.g., predicting the task instruction given inputs and outputs), emphasizing the modeling of
task instruction-following ability. In addition to the theoretical analysis, we conduct extensive experiments based on two widely used large language models, LLaMA and BLOOMZ, with various parameter sizes ranging from 1.7 billion to 13 billion. We select two common sequence generation tasks as specific downstream tasks, namely, machine translation and long text summarization. The experimental results show that Post-Ins consistently outperforms Pre-Ins across various settings without using any additional supervised data. Furthermore, due to the superior modeling ability of task instruction, Post-Ins exhibits stronger task instruction generalization capabilities, resulting in significant performance gains in zero-shot translation tasks (e.g., up to a 9.7 BLEU score improvement). Further analysis reveals that our method can substantially enhance the model’s instruction-following ability by 1x compared to the traditional approach, and about 4x improvements on specific instructions.
Our contribution can be summarized as follows: 2
• We show that the position of task instruction is a key factor to conduct instruction fine-tuning with LLMs, and propose to relocate the task instruction after the input sequence (i.e., PostIns), which could significantly enhance the instruction-following ability of LLMs by up to 4x improvements.
• Both our theoretical and experimental analyses demonstrate that Post-Ins pays more attentions on the model’s instruction-following capabilities, yielding consistent performance improvements across two common sequence generation tasks. (e.g., up to 9.7 BLEU and 3.5 ROUGE improvements on machine translation and text summarization respectively.)

heading: 2 Related Work
text: InstructGPT (Ouyang et al., 2022) is the first to unveil the immense potential of instruction learning. Remarkably, an InstructGPT model with 1.3 billion parameters can outperform the 175B GPT3, despite having 100x fewer parameters. Then Stanford releases the Alpaca instruction-following dataset (Taori et al., 2023), which is constructed by the self-instruction data generation pipline (Wang
2Codes and data are at https://github.com/Adaxry/ Post-Instruction
et al., 2022). In the field of machine translation, Parrot (Jiao et al., 2023a) builds contrastive and errorguided instructions to align the translation results of LLMs with humman preferences. Subsequently, Zeng et al. (2023) further extends the error-guided instructions with token-level Direct Preference Optimization (Rafailov et al., 2023). To better transfer the capabilities of sequence generation of LLMs, BayLing (Zhang et al., 2023b) proposes to conduct interactive translation task through instructing fine-tuning. Although the aforementioned methods have made considerable progress, we argue that the Pre-Ins data format ultilized in existing studies face the potential risk of instruction forgetting, what we aims to address in this paper.

heading: 3 Approach

heading: 3.1 Definition
text: The standard instruction-following data format consists of three components: a specific task instruction z, an input sentence x, and the corresponding response y. Taking the machine translation task as an example, z is a specific task instruction that directs the model to translate from the source language into the target language, while x and y are respectively the source input sentence and target translation. The z, x and y are sequentially concatenated into a long sequence, which is then fed into the LLMs for training in a teacher-forcing mode3. We provide a specific example in the Pre-ins format, as shown in Figure 2a.
Considering the nature of sequence generation tasks, the input part (z) often tends to be lengthy,
3Following existing studies (Taori et al., 2023; Jiao et al., 2023a), the cross-entropy loss is calculated merely on y, while x and z only participate in the forward encoding process.
such as translating an entire article or generating a summary of a paragraph. After applying the fine-grained tokenization, it can result in a long sequence of tokens for training. Generally, the mainstream LLMs are based the decoder-only architecture of Transformer, where the self-attention tends to pay more attention on nearby tokens. Therefore, in the case of long sequences as mentioned above, there is a significant risk that the model may forget the frontmost task instruction in the Pre-Ins data format, yiedling responses that do not follow the task instruction.

heading: 3.2 Preliminary Observations on Pre-Ins
text: To verify whether the Pre-Ins data format suffers from the above issue of instruction forgetting on long input sentences, we conduct preliminary experiments on the machine translation task (detailed experimental setups are at Section 4.1). We divide the training data into multiple groups based on the length range of the source text, ensuring that the total number of tokens in each group of training samples is approximately the same. Similarly, we also select corresponding test sets for different length ranges. Results on BLOOMZ are ploted in Table 1. We measure the degree of completion on a specific translation task instruction. Specifically, we ask the model to complete the task of "translating a sentence into a translation in a zero-shot direction," and then use the language identification tool lingua-py4to determine whether the model has correctly translated the current sentence into the target language. The zero-shot directions we chose are translation directions that the model has not seen during the supervised fine-tuning phase, specifi-
4https://github.com/pemistahl/lingua-py
cally Chinese-to-Japanese, Chinese-to-Ukrainian, English-to-Japanese, and English-to-Ukrainian. In addition, we concatenate the source and target sides of the above translation direction data to synthesize long sentence pairs. Since the above data has contextual relationships, sentence-level concatenation is coherent and meaningful. Under different sentence lengths, we observe the completion of the Pre-Instruction and Post-Instruction translation tasks, i.e., the accuracy of correctly translating into the target language is shown in Table 1.
Based on the above experimental results, it can be seen that at different lengths, the task instruction completion rate of Post-Instruction is significantly higher than that of Pre-Instruction, and the performance decreases relatively gently with the increments in sentence length, indicating that PostInstruction can alleviate the existing instructionforgetting problem of Pre-Instruction. Furthermore, the instruction-forgetting problem of PreInstruction becomes more severe as the sentence lengthens. For instance, when the average sentence length exceeds 90, the accuracy drops from 8% to 2%. These observations indicate that the existing Pre-Ins data format has limited ability to follow the instructions, especially when the input sentence z is long. Pre-Ins exhibits a risk of instruction forgetting, resulting in outputs that are not faithful to the user’s intent.

heading: 3.3 Post-Instruction
text: To address the above issue of instruction forgetting, we propose a simple and straightforward solution, namely, relocating the task instruction z after the input sentence x. As a result, the model can perceive the specific task instructions more closely when generating responses, regardless of the length of the input sentence. We refer this data format as Post-Instruction (Post-Ins), and provide a Post-Ins formatted example in Figure 2b.
Formally, the Post-Ins format of data encourages the LLMs to model the following conditional probability p(y|x, z). Here, we can decompose the above formula using the trinomial Bayes’ theorem
as follows:
p(y|x, z) = p(y) · p(x|y) · p(z|y, x) p(x) · p(z|x) (1)
where p(inp|res) represents the probability of the input given the response.
Given that the task instruction z is not involved in the training loss, we can simply treat its predicted probability p(z|x) as a constant, We and get the following form:
p(y|x, z) ≈ p(y)︸︷︷︸ fluency · p(z|y, x)︸ ︷︷ ︸ instruct · p(x|y)/p(x)︸ ︷︷ ︸ irreducible
(2)
where p(y) denotes the modeling probability of the target response, which guarantees the fluency of the model in predicting the response. The irreducible item represents the ratio of conditional probability to unconditional probability. On the other hand, p(z|y, x) represents the probability of the model determining which task instruction is currently being executed given the input x and response y. This can ensure that the model has a strong perception of the requirements of the task instruction.

heading: 3.4 Post-Instruction versus Pre-Instruction
text: As a comparison, we have also conducted a similar theoretical analysis for Pre-Ins, and ultimately obtain the following formula:
p(y|z, x) = p(y) · p(z|y) · p(x|y, z) p(z) · p(x|z)
= p(y)︸︷︷︸ fluency · p(x|y, z)︸ ︷︷ ︸ coverage · 1/p(x/z)︸ ︷︷ ︸ irreducible
(3)
Similar to Post-Ins, Pre-Ins also includes a component responsible for modeling the fluency of the response, denoted as p(y). The key difference lies in that the Pre-Ins emphasizes modeling the probability of the input given the instruction and response, namely, p(x|y, z), which is similar to modeling the coverage in translation tasks (Tu et al., 2016). Such modeling approach may be suitable for a single task or a small number of tasks, as the model can memorize these few task instructions through supervised fine-tuning.
However, LLMs inherently have strong fundamental capabilities that can naturally be applied to various sequence generation tasks. When modeling multiple sequence generation tasks simultaneously (such as multiple translation directions), Pre-Ins may suffer from instruction forgetting and produce low-quality responses that do not follow instructions due to the lack of task instruction modeling. In contrast, Post-Ins, with its preference for directly modeling task instructions as shown in Equation (2), can easily handle various sequence generation tasks and has good transferability for task instructions. We experimentally verify the stronger instruction transferability of Post-Ins compared to Pre-Ins in zero-shot translation tasks in section 5.1. Furthermore, we analyze the modeling preferences of the two data formats from the perspective of attention distribution and observe that the observations are consistent with our above theoretical analysis in section 5.3.

heading: 4 Experiments and Evaluations

heading: 4.1 Datasets
text: Alpaca. The Alpaca dataset, released by Stanford (Taori et al., 2023), is widely used for the instruction-following tasks. The data format follows the aforementioned Pre-Ins format, consisting of three parts: instruction, input, and output. We adjust the positions of the instruction and input, yielding a Post-Ins formated Alpaca. We apply this Post-Ins formatted Alpaca dataset on Post-Ins experiments, while the other experiments are still conducted on the original Alpaca dataset.
WMT Datasets. We use the WMT development sets from 2017 to 2020 as high-quality translation training data, following existing settings (Jiao et al., 2023a; Zeng et al., 2023). For translation directions with multiple references, we duplicate the source side and then match them with the corresponding translations to form multiple translation sentence pairs. Finally, we obtain a collection of 51k sentence pairs for instruction fine-tuning. To facilitate comparison, we follow the settings of existing methods (Jiao et al., 2023a; Zeng et al., 2023) and fine-tune LLMs on data for three languages and four translation directions: Chinese-toEnglish, English-to-Chinese, German-to-English, and English-to-German. The test sets for these four directions in WMT-2022 are used to evaluate translation performance, while the remaining directions, such as French-to-German or Russian-to-English,
are used to evaluate the zero-shot performance of the models. Furthermore, considering the similarity in data distribution over years in the WMT dataset (Barrault et al., 2020; Zeng et al., 2021), we also conduct evaluation and validation on another test set, namely the FLORES-200 benchmark.
Multidimensional Quality Metrics (MQM). The MQM dataset is based on the outputs of top systems from the WMT 2020 shared task, which provides error analysis of above translations annotated by professional translators. We follow the preprocessing scripts of existing studies and finally obtain a same sized training set with 99k examples (Jiao et al., 2023a; Zeng et al., 2023). In this paper, MQM is only used for translation task.
CNN/DailyMail. The popular CNN/DailyMail Dataset (See et al., 2017) is a collection of Englishlanguage news articles, comprising slightly more than 300k unique articles authored by journalists from CNN and the Daily Mail. The average sentence length of the source text of these data is approximately 665 words, or about a thousand tokens, which served as a widely used benchmark for long text summarization (Tang et al., 2023; Zhang et al., 2023a; Lin et al., 2023). We follow the pre-processing and post-processing scripts of existing studies (Qi et al., 2020). We use the CNN/DailyMail dataset only for the text summarization task and conduct the evaluation on the standard test set with 11,490 samples.

heading: 4.2 Evaluation
text: Inference Settings. For all tasks, we set the batch size to 1 during inference to avoid the effect of padding side (e.g, BLOOMZ applies left-padding mode, while LLaMA uses right-padding mode when batching the input data). As for the decoding strategies, we apply the beam search for all tasks, and set beam size to 4 for machine translation. While for the text summarization task, we have to decrease the beam size to 2, as encoding the long input sentences will consume a large portion of GPU memory.
Metrics For the machine translation task, we use SacreBLEU5 to calculate the BLEU scores. Given the limitations of N-gram-based metrics to measure semantic similarity, we also calculate the popular neural-based metric, namely COMET226. We use
5https://github.com/mjpost/sacrebleu 6https://github.com/Unbabel/COMET
the paired bootstrap resampling methods (Koehn, 2004) to compute the statistical significance of translation results. For the text summarization task, we report the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L following existing studies (Tang et al., 2023; Qi et al., 2020).

heading: 5 Main Results and Analysis
text: In this section, we first list the detailed experimental results of both the machine translation and text summarization tasks in Section 5.1 and Section 5.2. Subsequently, we show the analysis of the distributions of self-attention in Section 5.3, and human evaluation results in Appendix A.

heading: 5.1 Results of Machine Translation
text: Supervised Translation. Table 2 presents experimental results on WMT22. Our proposed method, Post-Ins, consistently outperforms Pre-Ins in most of translation directions over different model sizes of BLOOMZ (from 1.7B to 7.1B). Specifically, LLaMA-7B achieves a remarkable increase of +6.67 BLEU and +4.96 in COMET22 in En⇒Zh translation. Table 3 showcases the performance of
our method on the Flores-200 test set. Our Post-Ins outperforms Pre-Ins in 13 out of 16 settings, with maximum improvements reaching +7.20 BLEU and +6.16 COMET22 score in En⇒Zh.
Zero-Shot Translation. Furthermore, we observe significant improvements in zero-shot translation in the post-ins mode. Table 4 reports the results of different instruction modes in the WMT22 zero-shot test set. In terms of BLOOMZ, there is an impressive increase of +8.8 in De⇒Fr translation, with an average improvement of +1.4 BLEU. For LLaMA-7B, an average improvement of +2.1 BLEU is achieved, and LLaMA-13B exhibits an average improvement of +1.6 BLEU. Notably, LLaMA-13B showcases the highest improvement of +9.7 BLEU in De⇒Fr translation. Overall, the consistent improvements of Post-Ins over Pre-Ins indicate that Post-Ins exhibit stronger instruction generalization capabilities, being able to generate responses effectively even for task instructions it has never encountered during fine-tuning.

heading: 5.2 Results of Text Summarization
text: To further validate whether Post-Ins can effectively alleviate the issue of instruction forgetting, we perform experiments on tasks such as the long text summarization task, where the average length of input tokens is over 1,000. Table 5 presents the experimental results for the text summarization task on CNN / DailyMail, where we report the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L. It is evident that all models achieved significant performance improvements of up to +3.49 in BLOOMZ3B when utilizing the Post-Ins approach. The superior performance on the text summarization task demonstrates the effectiveness of Post-Ins on han-
dling long inputs.

heading: 5.3 Distributions of Self-attention
text: Given that the distribution of self-attention can explain the behavior of the Transformer model to some extent (Hao et al., 2021; Mahmood et al., 2021; Dai et al., 2021; Braşoveanu and Andonie, 2020), we plot the heatmap of self-attention for models trained with Pre-Ins and Post-Ins in Figure 3. We take BLOOMZ-7.1B as the base model and conduct forward propagation on the training samples of machine translation to obtain the attention scores. To mitigate the impact of fluctuations of multi-head attention and various layers, we average the scores of all heads over different layers to obtain the final score. We reach the following observations:
• A greater concentration of attention scores is observed at the beginning of sentences and along the diagonal positions of the attention matrix, which is consistent with existing conclusions (Liu et al., 2023).
• As shown in the lower right part of Figure 3b, Post-Ins pays more attentions on the task instruction when generating the response, while Pre-Ins mainly foucses on the source input and pays weak attentions on instruction as shown in the upper left part of Figure 3a.
• After instruction fine-tuning on the machine translation data, models learn latent word alignment information on both data formats. That is, models tend to allocate more attention to aligned parts of the source when generating responses word by word, which is similar to the conclusions of the traditional encoderdecoder structure in the field of machine translation (Bahdanau et al., 2014; Lample et al., 2017).
In summary, through the visualization of the selfattention heatmap, we observe that Post-Ins naturally tends to emphasize the task instruction, which is relatively overlooked in Pre-Ins. This finding is consistent with the theoretical analysis and conclusions presented earlier in Section 3.4.

heading: 5.4 Instruction-Following Evaluation
text: Inspired by IFEval (Zhou et al., 2023), We construct a dataset to evaluate the model’s ability to follow instructions. Specifically, we constructed fine-grained translation instruction compliance data based on the WMT-2022 Chinese-to-English and English-to-Chinese test sets. This involves five task instructions: (1) Please translate the following sentence into Chinese (or English), requiring the prefix of the translation to be [Prefix_Placeholder]; (2) Please translate the following sentence into Chinese (or English), requiring the suffix of the translation to be [Suffix_Placeholder]; (3) Please translate the following sentence into Chinese (or English), requiring the length of the translation to be greater than or equal to [Len_Min_Placeholder] and less than [Len_Max_Placeholder]; (4) Please translate the following sentence into Chinese (or English), requiring the translation to contain [NE_Placeholder]; (5) Please translate the ollowing sentence into [Language_Placeholder]. The xxx_Placeholder in the above instructions will be replaced with the corresponding content according to the golden references and the corresponding instruction requirements, with each type of task instruction containing approximately 300 test examples on average. For the last task instruction, [Language_Placeholder], we used the target languages of two zero-shot translation directions, namely Japanese and Ukrainian.
In terms of evaluation, for the first four task instructions, we can evaluate through simple rule matching. As for the last task requiring a switch of
target language, we judge whether the task instruction is correctly followed and executed by using the language identification tool lingua-py. Each type of instruction has 300 instances. We conduct experiments based on LLaMA-7B, and list results in the Figure 4. We observe that Post-Ins consistently outperforms Pre-Ins in terms of instruction-following ability, especially for switching target languages, with about 4x improvement. Overall, Post-Ins improves instruction-following ability by about 1x over the traditional Pre-Ins.

heading: 6 Conclusion
text: This paper highlights the importance of task instruction positioning in the instruction fine-tuning process of LLMs for conditional sequence generation tasks. We propose a simple yet effective method, Post-Ins, which relocates the task instruction after the input sequence to enhance the instructionfollowing ability of LLMs. Our theoretical analysis and experimental results demonstrate that Post-Ins effectively shifts the model’s learning focus, leading to improved performance across various model scales and different tasks, such as machine translation and long text summarization. Notably, our method significantly boosts zero-shot performance without additional data or annotation costs.
Limitation
The advantages of Post-Ins are likely to be more pronounced with longer text data (such as paragraph data), and we plan to explore its performance on extended text in future work. The effectiveness of Post-Ins on larger scale models also warrants further investigation.

heading: A Human Analysis
text: We employ two linguistics professionals to evaluate the translation task from English to Chinese. Specifically, the annotators are requested to judge whether the model’s output faithfully follow to the translation instructions and the source input.
If translation hallucinations occurred, i.e., the response contains content that is not present in the source sentence, or if the translation task is not effectively executed, the label was marked as ‘0’; otherwise, it is marked as ‘1’. The manual annotation results on 1,000 samples show that the hallucination rate for Pre-Ins is 4.8%, while 1.7% for Post-Ins. We also provide several examples in Figure 5. In summary, Post-Ins can enhance the model’s instruction-following capability and effectively reduce the proportion of prediction hallucinations.

heading: Future Work
text: Limitation
The advantages of Post-Ins are likely to be more pronounced with longer text data (such as paragraph data), and we plan to explore its performance on extended text in future work. The effectiveness of Post-Ins on larger scale models also warrants further investigation