Instruction Tuning With Loss Over Instructions

Authors: zhengyan  shi
adam x yang
bin  wu
laurence  aitchison
emine  yilmaz
aldo  lipani
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:05:22

Abstract: Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, Instruction Modelling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (*e.g.,* MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (*e.g.,* MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing \ours as a replacement for the current instruction tuning process.
Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.
Our code is available at https://github.com/ZhengxiangShi/InstructionModelling.

Main paper: 
 related work: 
 Instruction Tuning. LMs can better align with user intents through fine-tuning on datasets consisting of instructions and human-written completions [3, 44] . Early studies mainly focus on NLP tasks, showing that fine-tuning with various NLP datasets trained with instruction output pairs improves cross-task generalisation [1, 33, 43, 44, 51, 54, 65, 66] . Recent works explore the creation of instruction tuning datasets by LMs themselves [63, 25, 69, 36] or through crowdsourcing approaches [13, 78] . Such instruction-tuning phrase [30, 53, 59, 26, 73] enables LLMs to generalise beyond instructions in the training set, largely enhancing their practical utility.Data Selection for Instruction Tuning. Research on instruction tuning for LMs presents diverging perspectives on the optimal data scale for supervised fine-tuning. A prevailing view recommends fine-tuning on expansive datasets to enhance LM performance across various NLP tasks, thereby improving zero-shot and few-shot learning capabilities [1, 33, 44, 65, 43, 42, 51, 57, 64, 67] . For example, Flan V2 comprises over a million question-answer pairs from diverse NLP sources [14] , and Natural Instructions features 61 distinct tasks and 193k task instances [43] . Conversely, another research trajectory prioritises data quality over quantity [20, 70, 40, 32] . The Superficial Alignment Hypothesis (SAH) [78] advocates for using smaller, high-quality datasets, arguing that LMs primarily acquire their capabilities during the pretraining phase and thus require only minimal data for effective instruction tuning. For instance, LIMA [78] employs a carefully curated set of 1k diverse prompts to generate stylistically consistent responses, aimed at creating a helpful AI assistant. AlpaGasus [11] and LESS [68] employ methods to select high-quality data based on LLM-generated judgements and gradient signals. However, both views agree on the importance of (1) the quality of pre-trained base LMs and (2) the diversity and quality of the IT data.Regularisation Through Language Modelling Objectives. Pretraining data and language modelling objectives have been used as a regularisation technique in fine-tuning LMs. In particular, [15, 39] fine-tunes LMs on labelled data, with unsupervised learning on unlabelled data for auxiliary tasks as regularisation. [44] mixes the alignment objective with the next token prediction objective using pretraining data to mitigate alignment tax in reinforcement learning from human feedback (RLHF). [22] adopts the masked language objective on the pretraining or downstream task corpus to preserve pre-trained features, and shows improvements in calibration and accuracy. [29] investigates the effect of incorporating instruction loss weighting on instruction tuning, suggesting that the instruction loss ratio is an important hyperparameter when fine-tuning short-completion data but is irrelevant when using long-completion data. In this work, we propose a broader guideline that does not introduce new hyperparameters but focuses on when and how to include loss over instruction effectively. We refer to our approach as INSTRUCTION MODELLING because it combines elements of both language modelling and instruction tuning. 
our approach: 
 In this section, we first revisit the background of INSTRUCTION TUNING (IT) and then introduce our proposed method, INSTRUCTION MODELLING (IM).Instruction Tuning. In instruction tuning, each input is a concatenation of an instruction I and a completion C. Let I be the instruction sequence {I 1 , I 2 , . . . , I m } and C be the completion (output) sequence {C 1 , C 2 , . . . , C n }, where I may include special prompt template tokens (such as "
" and "
"). The total input sequence x is {I 1 , I 2 , . . . , I m , C 1 , C 2 , . . . , C n }. The model predicts each token in C given all the previous tokens in I and C up to that point:EQUATIONThe loss function, L, for instruction tuning is the negative log-likelihood of the completions given the instructions, expressed as follows:L = -log P (C 1 , C 2 , . . . , C n |I 1 , I 2 , . . . , I m ) = - n j=1 log P (C j |I 1 , I 2 , . . . , I m , C 1 , C 2 , . . . , C j-1 )(2) This approach aims to optimise the predictions for the completion sequence C, using the instruction sequence I as contextual information.Our Approach: Instruction Modelling. Our approach, instruction modelling, is an expansion of instruction tuning by incorporating loss calculation for both the instruction and the completion tokens, except it omits any special prompt template tokens. The model is trained to predict both the instruction and completion parts of x but excludes tokens that are part of prompt templates (denoted as T ). For simplicity, we consider that these template tokens are not part of I or C. The model predicts the next token given all previous tokens (both instructions and completions up to that point):EQUATIONThe loss function, L, for instruction modelling calculates the negative log-likelihood for both instruction and completion tokens, excluding any prompt template tokens. It is computed as follows:EQUATIONwhere 1(x t / ∈ T ) is an indicator function that is 1 if x t is not a template token and 0 otherwise. This ensures that the loss is computed only over the meaningful tokens, not over the static template tokens. Our approach allows the model to improve its understanding of both the instructions and the completions while being sensitive to the context provided by both segments of the input sequence. 
experiments and results: 
 In this section, we evaluate the effectiveness of our proposed method INSTRUCTION MODELLING (IM) by comparing it with INSTRUCTION TUNING (IT) and other baselines on various datasets.Instruction Tuning Datasets. We assess our method, IM, across various instruction tuning datasets, detailed as follows: (1) Stanford Alpaca [61] Evaluation Benchmarks. Our study conducts a comprehensive analysis of 21 NLP datasets, focusing on a suite of canonical NLP benchmarks and their capacity for open-ended language generation. For canonical NLP benchmarks, the evaluation is organised into six categories (18 tasks in total): (1) Language Understanding and Knowledge includes MMLU [24] , PIQA [6] ,OpenbookQA [41] , HellaSwag [74] , and LAMBADA [45] ; (2) Multilinguality contains LAMBADA Multilingual [45] , WMT 2014 [8] , and WMT 2016 [52] ; (3) Commonsense Reasoning features Winograd schema challenge (WSC) [35] , WinoGrande [50] , AI2 Reasoning Challenge (ARC) [16] ,and CoQA [49] ; (4) Math and Coding Reasoning includes GSM8K [17] , and HumanEval [12] ; (5) Safety and Helpfulness comprises TruthfulQA [38] , ToxiGen [21] , and Hendrycks Ethics [23] . ( 6) Big Bench Hard (BBH) dataset [60] is included to assess models. Our models are also tested for their open-ended text generation capabilities using model-based evaluations, specifically through MT-Bench [77] , AlpacaEval 1.0 and 2.0 [37] , where the AlpacaEval 1.0 compares the model outputs against the text_davinci_003 evaluated by GPT-4 and the AlpacaEval 2.0 compares the model outputs against GPT-4 outputs evaluated by GPT-4 Turbo. See evaluation details in Appendix §B.All Comparison Approaches. In our study, we mainly conduct experiment using the LLAMA-2-7B-BASE and LLAMA-2-13B-BASE [62] , and the OPT-6.7B [75] models. We report model performance trained on LLAMA-2-7B-BASE if not specified. We compare with NEFTUNE [31] as the baseline, which adds noise to the embedding during the instruction tuning to increase the robustness of instruction-tuned models. See hyperparameter and implementation details in Appendix §C.In this section, we first evaluate the model performance of our approach and baselines across various tasks. Then we investigate the key factors that contribute to the effectiveness of our approach. Below we will discuss our findings in detail.#1: Our approach IM can improve the performance of instruction tuning on various NLP tasks and open-ended generation benchmarks. Figure 1 provides a summary of the model's performance across both traditional NLP tasks and the AlpacaEval 1.0 benchmark. Table 1 offers a detailed breakdown of experimental results for traditional NLP tasks across six categories, as well as performance on additional benchmarks for open-ended generation (i.e., MT-Bench and AlpacaEval). The experimental results show that our approach (IM) can improve the performance of instruction tuning on various NLP tasks and open-ended generation benchmarks. Specifically, on the Alpagasus Dolly 3k dataset, IM improves the overall mean score of NLP tasks to 48.95, an increase of 2.37 points from the baseline. Similarly, on the Alpagasus Dolly 9k dataset, we observe an improvement of 2.46 points in the mean NLP score. These improvements are mirrored in the LLM-based evaluations. Specifically, IM raises scores on the AlpacaEval 1.0 benchmark, achieving approximately a ten-point increase on the Alpagasus Dolly 9k dataset and doubling the performance on datasets such as Less MMLU Chat and Less Tydiqa. However, the extent of improvement varies across different datasets. For example, the LIMA dataset shows more modest gains, prompting our further analysis to understand the factors influencing the effectiveness of IM.#2: Our approach IM is especially beneficial for datasets characterised by lengthy instructions or prompts paired with comparably brief outputs. To better understand the impact factors on the effectiveness of IM, we extend our experiments to more instruction-tuning datasets, such as Science Literature, Code Alpaca and Tulu V2. Interestingly, as shown in Figure 2 Left, we find that IM is particularly effective in scenarios where datasets characterised by lengthy instructions and shorter outputs, such as Less MMLU Chat and Less BBH ICL. For example, in datasets like Less MMLU Chat and Less Tydiqa, IM shows remarkable efficacy. In contrast, the Tulu V2 dataset, with an instruction to output length ratio of about 0.5, benefits less compared to the Science Literature dataset, which has a much higher ratio of 24.7. We hypothesise that this trend can be attributed to the tendency of language models trained on datasets with shorter outputs to overfit. In cases where the instructions are longer, IM acts as an effective form of regularisation, mitigating this issue. For further details on the experimental setup, refer to the Appendix in §C. 2 Left. We ensure that our samples maintain an instruction-to-output length ratio of around 10. As all these samples are from Tulu V2, we can assume they are from the same distribution. As shown in Figure 2 Right, IM demonstrates substantial performance improvements on the AlpacaEval 1.0 as the number of training examples decreases. This suggests that IM could be particularly valuable for developing robust models in resource-constrained scenarios or under the SAH. For further details on the experimental setup, please refer to the Appendix in §C.This section explores the underlying interpretation behind the effectiveness of our approach. Our experimental results demonstrate that IM can alleviate the overfitting problem of Instruction Tuning. Below we will discuss our findings in detail.#1. Train and test loss analysis. Figure 3 clearly illustrates the effectiveness of our approach IM in mitigating overfitting issues compared to IT. For both IM and IT, here we only compute the loss over the output part. In the training loss distribution for the LIMA dataset, IM exhibits a slightly higher mean loss of 1. RLHF causes an Alignment Tax on the NLP tasks [3, 44] . In this study, we observe that instruction tuning can sometimes lead to diminished model capabilities in some areas, such as multilinguality and commonsense reasoning. To this end, we further explore the impact of instruction tuning on the performance of NLP tasks. Figure 4 illustrates that our approach IM generally has a lower instruction tuning tax compared to IT, suggesting better robustness under low-resource settings. We provide additional experiments for win rates across epochs in Appendix §E.#4. Can we simply use KL divergence loss as regularisation for instruction tuning? In this analysis, we demonstrate that the application of KL divergence loss in instruction tuning, which is widely used as regularisation for aligning LMs [3, 44, 72] , cannot easily address the overfitting issue of instruction tuning. Table 3 offers a detailed comparison across various NLP benchmarks and open-ended language generation tasks, particularly using AlpacaEval 2.0, with models trained with and without KL divergence loss. Our findings are as follows: (1) Incorporating KL Loss reduces overfitting and reduces the performance degradation on traditional NLP tasks. For example, on the #1. The advantage of our proposed method persists with different language models and sizes. As shown in Figure 5 , our analysis demonstrates that our proposed method IM consistently outperforms the IT across different models and sizes, including OPT-6.7B and LLAMA-2-13B-BASE, on 18 traditional NLP tasks and the AlpacaEval 1.0 benchmark. These findings underline the effectiveness of our approach irrespective of the underlying language model or its scale. #2. Relationship between the model output length and the win rate. In this analysis, we explore the potential connection between win rates on the AlpacaEval and the increased output length [37, 76, 31] . As shown in Figure 6 , our result reveals that our approach IM does not necessarily generate longer outputs than IT across different data utilisation levels from the Tulu V2 dataset. Specifically, the output lengths for both approaches are similar despite varying levels of data utilisation. Furthermore, IM consistently outperforms the IT, suggesting that improvements in performance as measured by win rates on the AlpacaEval 1.0 are not dependent on the output length. We provide additional analysis on other instruction tuning datasets under the SAH in Appendix §G. #3. Our proposed method IM could further improve the model performance with NEFTUNE. generally further improves the win rates in AlpacaEval 1.0, showing notable improvements in several datasets such as a 13.31% increase on Less Tydiqa and a 12.55% boost on Alpagasus Alpaca 5k (in absolute). However, this combination leads to a performance drop in certain contexts, such as a lower performance on NLP tasks on Less MMLU Chat and Less BBH ICL. This indicates that while NEFTUNE may enhance model robustness under certain conditions, its benefits are context-dependent, highlighting the need for the careful application of NEFTUNE when used in conjunction with IM to optimise effectiveness across diverse evaluation settings. 
epilogue: 
 Conclusion. Our study proposes INSTRUCTION MODELLING, which trains LMs with loss over instructions rather than outputs only. Our experimental evaluations demonstrate that our approach largely improves the performance of LMs on both NLP tasks and open-ended generation benchmarks in some scenarios, especially under the Superficial Alignment Hypothesis and low-resource setting where minimal training data is used for instruction tuning. Our analysis has shed light on two key factors that influence the effectiveness of our approach, (1) the ratio between instruction and output lengths, and (2) the quantity of training data, providing practical insights for optimising instructionbased training methods. Additionally, our analysis reveals the mechanisms behind the effectiveness of IM, particularly its ability to reduce overfitting, showing that applying instruction losses in some scenarios can lead to more robust and adaptable LMs.Limitations and Broader Impact. Here we discuss some potential limitations and the broader impact of our work. Several limitations are outlined as follows: (1) The success of our approach relies on the quality and diversity of the instructions and prompts in the training datasets. Poorly defined or ambiguous instructions may undermine the effectiveness of IM, leading to sub-optimal performance; and (2) It is crucial to ensure that the instructions are ethically sound and free from harmful or biased content. Training on inappropriate or toxic instructions may result in undesirable outputs. Previous works [5, 7, 4] have extensively discussed the risks and potential harms associated with LMs, including the amplification of undesirable biases learned from training data [5, 2, 9] . Our work has the potential to positively impact the community by helping to mitigate overfitting, resulting in models that are more robust and generalise better to new data, especially in low-resource scenarios. This can enhance the reliability and trustworthiness of AI systems in real-world applications. We use the open-source repositories, LM-Evaluation Harness3 and Huggingface Dataset4 as the evaluation tools. We describe our evaluation setup below: MMLU. We evaluate the model using the dataset at the huggingface dataset 5 . We follow the protocol outlined in HuggingFace Open LLM Leaderboard 6 . The evaluation uses multiple-choice questions formatted as the question followed by four choices (A, B, C, D) and prompting for an answer. We calculate the mean accuracy (acc) across test examples.BBH. The model evaluation utilises the dataset at the huggingface dataset 7 , specifically tested on the 'test' split without the use of few-shot examples. We follow the setup in previous works [30, 60] . The evaluation metric is the exact match score, averaged (mean) to assess performance. Generation is constrained to a maximum of 1024 tokens, with termination upon encountering specific delimiters such as "
", "Q", or double newlines. The generation is greedy decoding (temperature set to 0.0) and does not use sampling. Answer extraction employs regex patterns to identify responses immediately following "the answer is" and captures only the first occurrence.GSM8K. We evaluate using the dataset at the huggingface dataset 8 , focusing on arithmetic problem-solving in the 'test' split. We follow the HuggingFace Open LLM Leaderboard to 8 few-shot examples. Exact match is the chosen metric, with case insensitivity and select regex-based filtering of common punctuation and formatting characters to ensure precise validation of numerical answers. The primary focus is on extracting and comparing the final numerical answer to the model's output using a strict regex-based match setup.HumanEval. We evaluate using the dataset and the evaluation code from the previous work [30] .We report the performance of the pass@1. We perform the decoding using two different temperatures, 0.1 and 0.7. We report the better pass@1 from these two decoding results.ARC. The evaluation setup for the dataset at the huggingface dataset 9 utilises a multiple-choice format. We follow the HuggingFace Open LLM Leaderboard to 25 few-shot examples. The performance metric used is mean normalised accuracy (acc_norm).CoQA. We conduct the model evaluation on the dataset at the huggingface dataset 10 . We follow the HuggingFace Open LLM Leaderboard to 0 few-shot examples. The output generation terminates upon encountering a new line followed by "Q:". The mean F1 score is used as the evaluation metric.PIQA. Evaluation on the dataset at the huggingface dataset 11 involves a multiple-choice. The evaluation incorporates 10 few-shot examples, according to the LIMIT [32] . Performance is measured using the mean normalized accuracy (acc_norm).OpenBookQA. The dataset at the huggingface dataset 12 is evaluated in a multiple-choice format.The mean normalized accuracy (acc_norm) is used as the evaluation metric.LAMBADA. The evaluation of the model on the dataset at the huggingface dataset 13 is performed using a loglikelihood output type. The mean accuracy is used as the evaluation metric.HellaSwag. In the 'hellaswag' dataset at the huggingface dataset 14 , model evaluation is conducted using a multiple-choice format. We follow the HuggingFace Open LLM Leaderboard to 10 few-shot examples. The mean normalized accuracy (acc_norm) is used as the evaluation metric.The Winograd Schema Challenge. The evaluation is conducted using a multiple-choice format on the 'test' split at the huggingface dataset 15 . The mean accuracy is used as the evaluation metric.Winogrande. The 'winogrande' dataset is assessed using a multiple-choice format at the huggingface dataset 16 . We follow the HuggingFace Open LLM Leaderboard to 5 few-shot examples. The mean accuracy is used as the evaluation metric.LAMBADA. For this dataset, evaluation is conducted using the loglikelihood output type on the 'test' split at the huggingface dataset 17 . This variant focuses on predicting the last word of text passages in English. The mean accuracy is used as the evaluation metric.Translation Benchmarks WMT. The evaluation of the translation capabilities is performed on the WMT 2014 18 and WMT 2016 19 datasets at the huggingface dataset. Here we use the 'ter' score as the evaluation metric.TruthfulQA. We use the dataset at the huggingface dataset 20 . We follow the setup at the Hug-gingFace Open LLM Leaderboard using the 6 few-shot examples. The mean accuracy is used as the evaluation metric.ToxiGen. We use the dataset at the huggingface dataset 21 . The task is assessed using a multiplechoice framework to evaluate the model's capability to identify hateful content in text statements.The mean accuracy is used as the evaluation metric.Hendrycks Ethics. We use the dataset at the huggingface dataset 22 , with a multiple-choice format.The model aims to detect whether described actions in various contexts are ethically wrong. The prompt format integrates a specific scenario followed by a structured question: "Is this wrong?" and then prompts for an answer with options 'no' or 'yes'. The mean accuracy is used as the evaluation metric.Experimental Design for Figure 2 Left. Here we present a detailed experimental design for Figure 2 Left. We perform experiments on a variety of datasets, including ,000 examples in the Tulu V2 dataset, we create subsets of data ranging from as few as 1,000 to as many as 35,000 examples. These subsets were selected randomly, ensuring a representative mix across different scales. We adhered to a fixed instruction-to-output length ratio of approximately 10 to maintain consistency in training conditions across all samples. We train the LLAMA-2-7B-BASE on all these subsets and evaluate them respectively. Gradient accumulation steps Calculated dynamicallyImplementation Details. In our study, we fine-tune the LLaMA-2-7B, LLaMA-2-13B and OPT-6.7 model using four A100 80G GPUs, with a per-GPU batch size of 1 and a total batch size of 128, employing a learning rate of 2e-5. Training typically proceeds for 2 epochs with a maximum sequence length of 2048 tokens. We utilise gradient accumulation, calculated to effectively distribute training steps across the available hardware, resulting in larger batch sizes despite hardware limitations. We employ mixed precision (bf16), linear learning rate scheduling with a warm-up ratio of 0.03, and a weight decay of 0. To optimise our training, we use DeepSpeed with a stage 3 configuration without offloading. Our setup also includes the usage of Flash Attention [19] and slow tokenization to enhance training efficiency and compatibility. Our code is implemented using Open-Instruct 23 , Pytorch 24 and Huggingface 25 . Table 6 lists the hyperparameters. Test loss distribution for each example between IM and IT on the Tulu V2 dataset, using a 10% sampled data. Mean losses are marked by dashed lines. For both IM and IT, here we only compute the loss over the output part. IM has a higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared to IT. For both IM and IT, here we only compute the loss over the output part. IM has a higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared to IT.In this section, we provide additional experiments regarding training and testing loss distributions. Figure 8 focuses on the Alpagasus Dolly 3k and Tulu V2 datasets, displaying how IM tends to exhibit higher training losses yet achieves lower test losses compared to IT. Similarly, Figure 9 compares these methods on the Less MMLU Chat and Tulu V2 datasets under analogous conditions.Figure 10 illustrates the comparative analysis of AlpacaEval 1.0 scores across different epochs for two datasets, LIMA and Alpagasus Dolly 9k datasets. We evaluate the performance of IM and IT over different numbers of epochs. IM consistently surpasses IT in performance on the Alpagasus Dolly 9k dataset, while the performance of both approaches is comparable on the LIMA dataset.In this section, we first briefly introduce the Kullback-Leibler (KL) divergence and then introduce the experimental details. Future work can investigate the effectiveness of parameter-efficient fine-tuning approaches as the regularisation [80, 55, 28] .Kullback-Leibler Divergence. Kullback-Leibler (KL) divergence is commonly employed as a regularisation method in the fine-tuning of LMs, helping to mitigate overfitting by constraining the fine-tuned model to remain similar to the pre-trained model [44] . Specifically, the KL divergence is added to the fine-tuning objective as a per-token regularisation term between the fine-tuned LM π θ (x), and the pre-trained LM, π pre (x). For supervised fine-tuning with the next token prediction loss, the training objective incorporating KL divergence is computed as follows:EQUATIONwhere λ is a regularisation parameter that balances the loss due to the next token prediction and the KL divergence, and π(x t |x 0:t-1 ) represents the next token distribution of the fine-tuned or pre-trained LM conditioned on the preceding context. Ablation study on the effect of λ. In Table 3 , we set the value of the λ as 1.0. Here we provide additional experiments with different values of λ. Table 7 presents the model performance on the NLP tasks and AlpacaEval 2.0. This aligns our observations in §4.3. 9k, LIMA, and Less Tydiqa. Each line represents the average output length of a model, with epochs ranging from 2 to 10, and is accompanied by error bars that denote the normalised standard deviation (10%) of the output lengths. Our experimental results show that our approach IM does not consistently increase the output length and that win rates are not necessarily associated with the length of the output.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report error bars for some results (see Figure 6 ). For other experiments, error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our research conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g. if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to §5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not address or focus on topics such as data security, privacy, or ethical considerations regarding the release of sensitive technology.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators and original owners of the assets used in the paper are properly credited. This is achieved by acknowledging their contributions directly within the text, referencing their original works, and by including citations to their publications or sources.For example, we acknowledge the contribution of other open-source repositories in our repository at https://github.com/ZhengxiangShi/InstructionModelling.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] https://github.com/princeton-nlp/LESShttps://github.com/gpt4life/alpagasushttps://github.com/EleutherAI/lm-evaluation-harnesshttps://huggingface.co/docs/datasetshttps://huggingface.co/datasets/hails/mmlu_no_trainhttps://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboardhttps://huggingface.co/datasets/lukaemon/bbhhttps://huggingface.co/datasets/gsm8khttps://huggingface.co/datasets/EleutherAI/hendrycks_ethicshttps://github.com/allenai/open-instructhttps://pytorch.org/https://huggingface.co/ 
: 
 Figure 1 : Performance differences between INSTRUCTION TUNING (IT) and our proposed method INSTRUCTION MODELLING (IM) trained on 7 instruction tuning datasets. These datasets contain prompts and responses but do not contain preference pairs. Specifically, we use the Less datasets [68] and Alpagasus datasets [11] , which are subsets of Flan V2 [14] , Dolly [18] , and Stanford Alpaca [61] to ensure good performance. We also report the results on the LIMA dataset. (Left) The mean performance across 18 traditional NLP tasks (see §4.1 for details). (Right) The win rate on the AlpacaEval 1.0 benchmark [37] . Please refer to §4.2 for details.38th Conference on Neural Information Processing Systems (NeurIPS 2024). Llama-7B trained on different dataset sizes Linear Fit: y = -39.37x + 431. 74 Figure 2 : (Left) Performance improvement, achieved by our approach INSTRUCTION MODELLING (IM) compared to INSTRUCTION TUNING (IT) on the AlpacaEval 1.0, against the ratio between average instruction length and average output length in instruction tuning datasets (training size noted in parentheses). We highlight several representative instruction tuning datasets in yellow. Our analysis suggests that IM is especially beneficial for datasets characterized by lengthy instructions or prompts paired with comparably brief outputs, such as Code Alpaca [10] and Less MMLU Chat [68] . (Right) Performance improvement achieved by our approach IM over IT on the AlpacaEval 1.0 against the number of training examples in instruction tuning datasets. Here we maintain a fixed ratio between instruction and output length of 10. This analysis suggests that IM is particularly effective under the low-resource setting or Superficial Alignment Hypothesis. Please refer to §4.2 for details.Language models (LMs) are trained to predict the next token on massive corpora, enabling them to learn general-purpose representations transferable to various language understanding or generation tasks [7, 47, 48, 56, 62] . However, it does not align LMs to act in accordance with the user's intentions [34] . To enable this transfer, various methods for aligning language models [3, 14, 44, 73, 58, 79] have thus been proposed, one of which is instruction tuning (IT) [36, 65, 69] . Recent study [78] proposes Superficial Alignment Hypothesis (SAH): A model's knowledge and capabilities are learnt almost entirely during pretraining, only minimal instruction tuning data is required to enable highquality outputs in the desired output style. Existing works [1, 43, 44, 51, 65, 69, 36] mainly perform instruction tuning by focusing the loss computation solely on the output segments.In this work, we demonstrate that in many scenarios, incorporating the loss computation for instructions or prompts, which we refer to as INSTRUCTION MODELLING (IM) (see §3), could substantially improve the performance of instruction tuning on both various NLP tasks (e.g., MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g., MT-Bench and AlpacaEval), as shown in Figure 1 . Remarkably, in the most favourable case, our proposed method IM boosts performance on AlpacaEval 1.0 by over 100%. As illustrated in Figure 2 , our study further identifies two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length (see Figure 2 Left). Our analysis shows that our approach IM is especially beneficial for datasets characterised by lengthy instructions or prompts paired with comparably brief outputs, such as Code Alpaca [13] and Less MMLU Chat [68] ; (2) The number of training examples (see Figure 2 Right). We demonstrate that our approach IM performs better under the SAH, where a small amount of training examples are available (see §4.2).In summary, the main contributions of this paper are:• We propose INSTRUCTION MODELLING (IM), aiming to enhance both the instructionfollowing and general performance on NLP tasks of LMs. Through extensive experiments across 21 benchmarks, we demonstrate that, in many scenarios, IM substantially improves the performance of LMs trained on various instruction tuning datasets, particularly notable in the AlpacaEval 1.0 benchmark where it boosts scores by over 100%.• Our study identifies key factors influencing the effectiveness of IM, including the ratio between instruction length and output length and the number of training examples. We are not proposing IM as a replacement for current instruction tuning processes. Rather, we provide empirical guidance for fine-tuning LMs, especially under low-resource scenarios.• We provide underlying mechanisms that make IM effective, specifically how it mitigates overfitting, thereby enhancing the LMs' performance across various tasks. 
 
 Appendix:  


Decision: This paper proposes instruction modeling for training language models, which applies a function to the instruction and prompt part rather than solely to the output part. Experiments show that instruction modeling can improve model performance on both NLP tasks and open-ended generation benchmarks, with some key factors found that influence the effectiveness of the approach: the ratio between instruction and output lengths, and the quantity of training data. 

Some reviewers raised concerns regarding to the limited model exploration and lacking the explaination of why applying loss to instructions works. The authors made clarifications in their rebuttal with some extra results, and all reviewers vote positive to this work. Overall, the proposed method is quite simple and scalable, and the results are impressive. This method would be of great interest to practitioners in the community.  Therefore, I recommend an "acceptance".

Reviews: review: paper_summary: paper_summary: In this work, authors propose to use instruction modeling (using loss over the full instruction-output pair) instead of just instruction tuning (using loss over the output given the instruction) as a method for supervised finetuning on LLMs. The authors demonstrate consistent gains over multiple benchmarks using this simple technique and hypothesize that the gains occur due to reduced overfitting to instruction tuning dataset.
strength_weakness: strength_weakness: 1. Limited Models Explored: While the authors do a good job of experimenting with multiple instruction datasets, the experiments are only done with LLama-2 (7B and 13B) and OPT-7B models. This forgoes a plethora of open source models, experiments on which would have greatly strengthen the paper. Given that these LLMs are all from the Meta family of LLMs -- this raises some suspicion as to the generality of the proposed technique when different architectures or modeling assumptions are made.
questions: questions: 1. Do you have any results on other LLM families: Phi, MPT or Gemma?
limitations: limitations: Limitations are adequately addressed.
paper_summary: paper_summary: This paper proposes Instruction Modeling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. The method is found to be effective on NLP tasks and open-ended generation benchmarks. This paper found two key factors that influence the effectiveness of the approach: (1) the ratio between instruction and output lengths, and (2) the quantity of training data. There are also additional analysis that shows IM can reduce overfitting.
strength_weakness: strength_weakness: A part I feel that's missing in the paper is analysis on the **quality of the instruction part**. One reason that people didn't calculate loss over the instruction part is that the instruction part is usually noisy and not well-structured, e.g, ShareGPT has a lot of user shared low-quality instructions. The paper should analyze how the quality of the instruction part affects the performance of the model. Intuitively, if the model is also learning from the noisy instruction part, it's possible that the model will learn undesirable patterns from the instruction part, but I don't see this being discussed in the paper.

Moreover, the number of LMs being finetuned and tested is relatively small. It will be better if more families of LMs and larger models (eg, 60-70B params) are tested.
questions: questions: 1. Do you have any analysis on the quality of the instruction part? How does the quality of the instruction part affect the performance of the model? See "Weaknesses" part.

2. I might have missed this one in the paper, but how do you pack multiple samples during finetuning? Do you concatenate samples to form a fixed length and then trucate them? Or do you add padding tokens?

3. Do you have any scenarios where the IM is significantly worse than the baseline? If so, can you provide some examples? I want to know under what conditions IM is not effective and should be avoided.

4. Do you think IM can be used as a drop-in replacement for the current finetuning process? Or do you think it should be used in conjunction with the current finetuning process?
limitations: limitations: Limitations are being discussed in the paper.
paper_summary: paper_summary: The propose that when updating models using instruction tuning, the models should also be updated based on loss on the instruction itself.

This is a simple change that is un-intuitive, so the successful results are impressive.
strength_weakness: strength_weakness: It is unclear what the paper considers an "instruction", there should be some examples of it. At one point it says that "static parts" of the instruction, like a "<user>:" token, are masked from the lost. Does that mean the rest of the instruction is all examples? An instruction like "Tell me the sentiment of this text" is going to be static across examples.

When looking at table 1, although their IM methods wins in terms of average performance, there are many datasets where it performs worse. This is glossed over in their prose.

Their "Loss analysis" experiments are not convincing as they don't fully capture the changes to the model that occur in each setting. By only checking the loss on the continuations, they don't capture how their IM model might be more overfit to the instructions themselves. For example, if you looked that the train loss for the whole Instruction + continuation for each models, the IM model would mostly likely be far lower than the loss for the instruction tuned model. It makes sense that the IM model is going to be worse on the continuations after the same amount of training as updates to get better as the instruction itself will cause conflicts. Discarding the models performance on the Instructions creates a self-fullfilling prophecy of the loss being higher when only the continuations are considered.
questions: questions: What is considered an "instruction" when training? The text mentions that "static" tokens like "<user>:" are masked out, but many instructions are static, i.e. "Tell me if the second sentence entails the first". If instructions are non-static are they mostly example like in In-Context-Learning?

They mention that IM is most effective in settings where the instructions are long. Is it possible that most of this gain is for essentially continuing pre-training on in-domain data a la https://arxiv.org/abs/2004.10964

Similarly, do you think the success in the SAH settings is possibly an artifact of getting a model that is just trained on a lot more data? How many more tokens does an IM model see than in IT model in that setting? Is there a way to hold the number of tokens seen constant? For example by training the IM model on even fewer examples to show that the gain really is from the training on the instruction?
limitations: limitations: yes
paper_summary: paper_summary: This paper proposes a new method called Instruction Modelling (IM) for training language models, which applies a loss function to both the instruction and output parts of training data, rather than just the output. Through experiments on diverse benchmarks, the authors show that IM can improve model performance on both NLP tasks and open-ended generation benchmarks compared to standard Instruction Tuning (IT). The effectiveness of IM is influenced by two key factors: the ratio between instruction length and output length in the training data, and the number of training examples. IM is particularly beneficial for datasets with long instructions paired with brief outputs, or when using a small amount of training data. The authors hypothesize that IM's improvements stem from reducing overfitting during instruction tuning.
strength_weakness: strength_weakness: - Similar ideas and conclusions have been proposed by previous work [1].
- While the paper presents empirical results showing the effectiveness of Instruction Modelling (IM), it lacks a strong foundation explaining why applying loss to instructions works. The authors hypothesize that IM reduces overfitting, but a more rigorous theoretical analysis could provide deeper insights into the mechanism behind IM's success.
- The experiments primarily use LLaMA-2 and OPT models. While these are significant models, the paper doesn't explore how IM performs across a wider range of model architectures or sizes, e.g., whether the conclusion also holds for a 34B or a 70B model.

[1] Instruction Fine-Tuning: Does Prompt Loss Matter? https://arxiv.org/pdf/2401.13586v2
questions: questions: - Are there any possible experiments that you can do to further explain why applying loss to instruction works? 
- Do you think when we have a significantly larger size of instruction tuning data, the conclusion still holds? For example, we see the recent release of Llama-3 models, which adopted 10+ million instructions. And which factor is more important: the ratio between output and the instruction length or the number of instruction tuning samples?
limitations: limitations: Yes. But the authors do not have a mandatory paper checklist.