Instruction Tuning With Loss Over Instructions
Authors: Zhengyan Shi
Adam X. Yang
Bin Wu
Laurence Aitchison
Emine Yilmaz
Aldo Lipani
Venue: NeurIPS 2024 poster (2024)
Keywords: Instruction Tuning
Language Models
Low-Resource Methods for NLP
Primary area: natural_language_processing
TL;DR: We show that in many scenarios, applying loss to instructions rather than outputs only, which we refer to as Instruction Modelling, could largely improve the performance of instruction tuning on both various NLP and open-ended generation benchmarks.

Abstract: Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, Instruction Modelling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (*e.g.,* MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (*e.g.,* MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing \ours as a replacement for the current instruction tuning process.
Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.
Our code is available at https://github.com/ZhengxiangShi/InstructionModelling.

Instruction Tuning With Loss Over Instructions
Zhengyan Shi' Laurence Aitchison? Universsity College London
Adam X. Yang²
Bin Wu'
Emine Yilmaz'
Aldo Lipani'
University of Bristol
(zhengxiang. shi 19, bin. wu. 23, emine. yilmaz, aldo aippnnieuuc........e. ac. {adam. yang, laurence aitchison)@bristo.l uk
Abstract
Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, propose simple yet effective method, INSTRUCTION MODELLING (IM), which trains LMs by applying a loss function the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (e.g., MMLU, TruthfulQA, and HvmanEval) and open-ended generation benchmarks (e.g. MT- Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, under the Superficial Alignment Hypoth- esis (SAH) where small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can attributed reduced overfitting instruction tuning datasets. It worth not- ing that we are not proposing IM as replacement for the current instruction tuning process. Instead, our work aims to provide practical guidance for instruc- tion tuning LMs, especially in low-resource scenarios. Our code is available at https: http//ggtttubb.t. com com http//henngxannnnie http://///ntttttttoooooollllala.
Introduction
AlpacaEval 1.0 Win Rate 19)
across 18 NLP taaks (16)
Mean Performanc
AQUARIIA
LOVA
CMA
BAM
Daaly
Figure 1: Performance differences between INSTRUCTION TUNING (IT) and our proposed method
INSTRUCTION MODELLING (IM) trained on instruction tuning datasets. These datasets contain prompts and responses but do not contain preference pairs. Specifically, we use the Less datasets [68] and Alpagasus datasets [11], which are subsets Flan V2 [14], Dolly [18], and Stanford Alpaca [61] to ensure good performance. We also report the results the LIMA dataset. (Left) The mean performance across 18 traditional NLP tasks (see $4.1 for details). (Right) The win rate on the AlpacaEval 1.0 benchmark [37]. Please refer to $4.2 for details.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Improsmaatt on Alpccaaar (86)
120
Liama-78 trained differant y-
HHUU-
Look
160
trained difierett
m-39.37x
(33.5K)
140
x
80
Tydlip
60-
Stanford
Codl 120%)
17.933
40-
ShuracPT (52R)
80
x
20
60
(14)
40
TuN Va
(30K)
20
20
Instructio Longth Average Output Longth Training [Log Scale)
Number of Training Exampies Scale)
Figure 2: (Left) Performance improvement, achieved by our approach INSTRUCTION MODELLING (IM) compared to INSTRUCTION TUNING (IT) on the AlpacaEval 1.0, against the ratio between average instruction length and average output length in instruction tuning datasets (training size noted in parentheses). We highlight several representative instruction tuning datasets in yellow. Our analysis suggests that IM is especially beneficial for datasets characterized by lengthy instructions or prompts paired with comparably brief outputs, such as Code Alpaca [10] and Less MMLU Chat [68]. (Right) Performance improvement achieved by our approach IM over IT on the AlpacaEval 1.0 against the number of training examples in instruction tuning datasets. Here we maintain a fixed ratio between instruction and output length of 10. This analysis suggests that IM is particularly effective under the low-resource setting or Superficial Aligmment Hypothesis. Please refer to $4.2 for details.
Language models (LMs) are trained to predict the next token massive corpora, enabling them to learn general-purpose representations transferable to various language understanding or generation tasks [7. 47, 48, 56, 62]. However, it does not align LMs to act in accordance with the user's intentions [34]. To enable this transfer, various methods for aligning language models [3, 14, 44, 73, 58, 79] have thus been proposed, one of which is instruction tuning (IT) [36, 65, 69]. Recent study [78] proposes Superficial Alignment Hypothesis (SAH): A model's s knowledge and capabilities are learnt almost entirely during pretraining, only minimal instruction tuning data is required enable high- quality outputs in the desired output style. Existing works [1, 43, 44, 51, 65, 69, 36] mainly perform instruction tuning by focusing the loss computation solely on the output segments.
In this work, we demonstrate that in many scenarios, incorporating the loss computation for instruc- tions or prompts, which we refer as INSTRUCTION MODELLING (IM) (see $3), could substantially improve the performance of instruction tuning on both various NLP tasks (e.g.. MMLU, U, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g.. MTT-Bench and AlpacaEval), as shown in Figure 1. Remarkably, in the most favourable case, our proposed method IM boosts performance on AlpacaEval 1.0 by over 100%. As illustrated in Figure our study further identifies two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length (see Figure 2 Left). Our analysis shows that our approach IM is especially beneficial for datasets characterised by lengthy instructions or prompts paired with comparably brief outputs, such as Code Alpaca [13] and Less MMLU Chat [68]: (2) The number of training examples (see Figure 2 Right). We demonstrate that our approach IM performs better under the SAH, where a small amount of training examples are available (see $4.2).
Recent works [27, 31, 44, 71, 73] suggest that LMs can quickly memorise training examples even after seeing them just once. We hypothesise that the improvement stems from reducing instruction tuning's 's tendency to overfit, particularly under limited training resource conditions: Instruction tuning on brief outputs or small amount of data can potentially lead to rapid overfitting. To substantiate our hypothesis, our analysis shows that (1) IM cxhibits higher training losses but lower test losses on new instruction tuning data; (2) The outputs generated by IM have a lower similarity to the training examples compared to those from IT, as indicated by BLEU scores; and (3) IM leads to less performance degrade on NLP tasks across training epochs (see $4.3). Additionally, our study reveals that this overfitting cannot be effectively addressed by applying Kullback-Leible (KL) divergence for regularisation [3, 44], as it compromises the model's 's ability to follow instructions. Our further analysis reveals that the advantages of IM persist across different LMs and model sizes, and that IM could be effectively combined with the previous approach (i.e., NEFTUNE [31].. Meanwhile, we investigate the relationship between output length and win rate for our approach (see $4.4).

---

In summary, the main contributions of this paper are:
We propose INSTRUCTION MODELLING (IM), aiming to enhance both the instruction- following and general performance on NLP tasks of LMs. Through extensive experiments across 21 benchmarks, we demonstrate that, in many scenarios, IM substantially improves the performance of LMs trained various instruction tuning datasets, particularly notable in the AlpacaEval 1.0 benchmark where it boosts scores by over 100%.
Our study identifies key factors influencing the effectiveness of IM, including the ratio between instruction length and output length and the number of training examples. We are not proposing IM as a replacement for current instruction tuning processes. Rather, we provide empirical guidance for fine-tuning LMs, especially under low-resource scenarios.
We provide underlying mechanisms that make IM effective, specifically how it mitigates overfitting, thereby enhancing the LMs' performance across various tasks.
2 Related Work
Instruction Tuning. LMs can better align with user intents through fine-tuning on datasets con- sisting of instructions and human- written completions [3, 44]. Early studies mainly focus on NLP tasks, showing that fine-tuning with various NLP datasets trained with instruction output pairs im- proves cross-task generalisation [1, 33, 43, 44, 51, 54, 65, 66]. Recent works explore the creation of instruction tuning datasets by LMs themselves [63, 25, 69, 36] or through crowdsourcing approaches [13, 78). Such instruction-tunn -uning phrase [30, 53, 59, 26, 73] enables L.MSs to generalise beyond instructions the training set, largely enhancing their practical utility.
Data Selection for Instruction Tuning. Research on instruction tuning for LMs presents diverging perspectives on the optimal data scale for supervised fine-tuning. prevailing view recommends fine-tuning expansive datasets to enhance LM performance across various NLP tasks, thereby improving zero-shot and few-shot learning capabilities [1, 33, 44, 65, 43, 42, 51, 57, 64, 67]. For example, Flan V2 comprises and Natural Instructions features 61 distinct tasks and 193k task instances [43]. Conversely,
million -anssiion-nnwerr pairs from diverse NLP sources [14],
another research trajectory prioritises data quality over quantity [20, 70, 40, 32]. The Superficial Aligmment Hypothesis (SAH) [78] advocates for using smaller, high-quality datasets, arguing that LMs primarily acquire their capabilities during the pretraining phase and thus require only minimal data for effective instruction tuning. For instance, LIMA [78] employs a carefully curated set of 1k diverse prompts generate stylistically consistent responses, aimed creating a helpful AI assistant. AlpaGasus [1 I] and LESS [68] employ methods to select high-quality data based on LLM--generaeed judgements and gradient signals. However, both views agree on the importance of (1) the quality of pre-trained base LMs and (2) the diversity and quality of the IT data.
Regularisation Through Language Modelling Objectives. Pretraining data and language mod- elling objectives have been used as a regularisation technique in fine-tuning LMs. In particular, [15, 39] fine-tunes LMs on labelled data, with unsupervised learning on unlabelled data for auxiliary tasks as regularisation. [44] mixes the alignment objective with the next token prediction objective using pretraining data to mitigate alignment tax in reinforcement learning from human feedback (RLHF). [22] adopts the masked language objective the pretraining downstream task corpus to preserve pre-trained features, and shows improvements in calibration and accuracy. [29] investi- gates the effect of incorporating instruction loss weighting instruction tuning, suggesting that the instruction loss ratio is an important hyperparameter when fine- -uninu comt--com data but is irrelevant when using compleotletion data. In this work, propose a broader guideline that does not introduce new hyperparameters but focuses on when and how to include loss over instruction effectively. We refer to our approach as INSTRUCTION MODELLING because it combines elements of both language modelling and instruction tuning.
3 Our Approach
In this section, we first revisit the background of INSTRUCTION TUNING (IT) and then introduce our proposed method, INSTRUCTION MODELLING (IM).
3

---

Instruction Tuning, In instruction tuning, each input is concatenation of an instruction I and completion C. Let I be the instruction sequence {41, 12....... ................................ I and be the completion (output) se- quence (C1, .2...., J. ,..,,.,,,,, where / may include special prompt template tokens (such as "< user I>"
and "<| assi aesistannlss 1>"). The total input sequence x (71, /2,.... I C1, c.,..., Cn) The model predicts each token in C' given all the previous tokens in and C' up to that point:
P(C1, C,l/4,, .., mm) - II P(C Imm C1. C2....,, C3-1)
(I)
j=1
The loss function, L, for instruction tuning is the negative log- ligelikel of the completions given the instructions, expressed as follows:
== log P(C1, C.Ci:C 1. .2) =
Σ
log P(C 12. ................................ Im C1, C2, Cj-1)
==1
(2)
This approach aims optimise the predictions for the completion sequence C, using the instruction sequence I as contextual information.
Instruction Modelling. Our approach, instruction modelling, is an expansion of instruction tuning by incorporating loss calculation for both the instruction and the completion tokens, except omits any special prompt template tokens. The model is trained to predict both the instruction and completion parts of x but excludes tokens that are part of prompt templates (denoted as 7"). For simplicity, we consider that these template tokens are not part of I or C. The model predicts the next token given all previous tokens (both instructions and completions up to that point);
Our Approach:
m++
P(x) = P(I,, 12, Lmn C1, C2...., Cr) = II 1=1
P(x:/æ1, 22,,,
The loss function, f. for instruction modelling calculates the negative log- ligliiklihood for both instruc- tion and completion tokens, excluding any prompt template tokens. It is computed as follows:
+ log P(ra| 42, .... 11- 1xx $ T),
L -=
where (xx & T) is indicator function that is if xr is not template token and O otherwise. This ensures that the loss is computed only over the meaningful tokens, not over the static template tokens. Our approach allows the model to improve its understanding of both the instructions and the completions while being sensitive to the context provided by both segments of the input sequence.
4
Experiments and Results
In this section, we evaluate the effectiveness of our proposed method INSTRUCTION MODELLING (IM) by comparing it with INSTRUCTION TUNING (IT) and other baselines on various datasets.
4.1 Experimental Setup
Instruction Tuning Datasets. We assess our method, IM, across various instruction tuning datasets,
detailed as follows: (1) Stanford Alpaca [61] (52 002 examples); (2) Dolly [18] 011 ex-
amples); (3) Sharegpt [13] (50 000 examples); (4) Code Alpaca [10] (20 022 examples): (5) Science Literature [30] (7 544 examples); (6) WizardLM [69] (30 000 examples); (7) Tulu V2 [30] (326 181 examples). Additionally, we incorporate instruction tuning datasets under the low-resource setting SAH: (8) LIMA [78] (1 030 examples); (9) Less' [68], where high-quality instruction tuning data are selected from Flan V2 and Dolly. Here, we use the Less MMLU Chat (13 533 examples), Less BBH ICL (13 533 examples), and Less Tydiqa (13 533 examples); (10) Alpagasus² 2 [11], which offers three subsets: Alpagasus Dolly 3k (2 996 examples), Alpagasus Dolly 9k (9 229 examples) selected from Dolly, and Alpagasus Alpaca 5k (5 305 examples) selected from Stanford Alpaca. See dataset details and statistical analysis in Appendix $A.
http::///thhabbbcccc/pnnceeeee/s Phttps://githin wtw.aauw..oomgptttiltecc wtw.:/hbb.coogggllllf//a/aeeeeee

---

Table 1: Performance comparisons using 7 instruction tuning datasets with the -LAMA-2-7B on 6 categories of 18 traditional NLP tasks and 3 open-ended benchmarks with LLM as judgements. "IT"" refers to INSTRUCTION TUNING. "IM" refers to INSTRUCTION MODELLING. Green and red arrows indicate performance changes against the baseline (IT).
NLP Benchmarks
LLMM-based Evaluaaton
Method
Understanding Knowledge
Malti- linguality
Commonsence Reasoning
Madh Reasoning
BBL
Safety & Helpfulness
Mean
MI-Bench
AlpucaEval
1,0
AlpacaEval
2.0
LEAMA 2- 2-BAst
63.91
61.99
75.86
13.32
38,55 38.50
42.05
69.32
1.16
0.01
0.01
LLAAMA-2-CHA
63,42
55.15
70.26
15.33
36,92
51.79
49.15
6.63
79.04
6.48
Alpaca, 305 triining esamples)
T1
64.9%
57.24
66.06
8,93
20.50
47.74
3,62
16.29
2.46
NEFTUNE
63.18
50.38
66,05
10.24
29.53
45.45
3.50
21.57
2,37.83
IM (ouns)
64.01
56.63
72.47
11.58
35.52
64.62
47,471218
3.48
19.52
3.29 5.55
Alpogarus Dolly 996 training examples)
IT
X1
$7.46
46.55
4:23
2.00
NEFTUNE
63.90
57.79
44.30
4.421119
14.0
2.05
IM
63.66
57.47
73.24
37.48
4.06
15.1
2.44
Dolly 19,229 training examples)
IT
64.10
56.62
62.70
7.96
32.19
42.65
45.54
4:33
20.54
2.28
NETTUNE
64.20
56.69
09:51
8.99
33.91
42.62
31 11.611330
2.84'1.5
IM (OuIS)
64.67
55.32
74.87
12.50
36.09
43.96
48.000
4.55
30.77
2.67-
Leau Tydiqu (13,535 briining cxamples))
IT
64.01
$6.51
$5.09
45.21
4.08
5.12
NEUTUNE
64.03
$5.00
4.191111
8,35
2,58'
(ouns)
64.28
56.00
70
4.3811
10.10
Laxs VALD
mriining examples)
IT
64.74
57.42
62.94
9.53
38.13
55.35
47.18
4.43
1.30
NEETUNE
65.21
57.43
63.14
9.45
35.80
55.32
74
1.06,114
IM (ours)
63.95
56.34
64.76
12.52
26.94
52.55
43 47,84Tu
4.547
9.78
1.93
Lann HON TCL, (13.5% riiining exmmples)
TT
63.83 83
42.07
366 20
03
(ours)
64.14
56.72
13.56
49.15
5.03112
44.15
3.56
SIMA
examples)
II
63.92
58.29
71.96
16.01
29.27
45.24
$5.79
4.77
33.06
2.58
10 spoch NUTTUNE
63.66
57.67
73.03
15.95
38.77
43.14
48, 70
4.791
30,5112.5
2.43,a
IM (ours)
64.49
58.21
75.35
17.06
38.84
43.45
49 1S5
4.
32, $2.94
2.47.a11
Evaluation Benchmarks. Our study conducts a comprehensive analysis of 21 NLP datasets, focusing on a suite of canonical NLP benchmarks and their capacity for open-ended language generation. For canonical NLP benchmarks, the evaluation is organised into six categories (18
tasks in total): (1) Language Understanding and Knowledge includes MMLU [24], PIQA [6],
OpenbookQA [41], HellaSwag [74], and LAMBADA [45]; (2) Multilinguality contains LAMBADA
Multilingual [45], WMT 2014 [8], and WMT 2016 [52]: (3) Commonsense Reasoning features
Winograd schema challenge (WSC) [35], WinoGrande [50]. A12 Reasoning Challenge (ARC) [16],
and CoQA [49]; (4) Math and Coding Reasoning includes GSM8K [17], and HumanEval [12]; (5)
Safety and Helpfulness comprises TruthfulQA [38], ToxiGen [21], and Hendrycks Ethics [23]. (6)
Big Bench Hard (BBH) dataset [60] is included assess models. Our models are also tested for their open-ended text generation capabilities using model-based evaluations, specifically through MT-Bench [77], AlpacaEvall 1.0 and 2.0 [37], where the AlpacaEval 1.0 compares the model outputs against the text_deaeeeeee dwviiciiiini.... __00 evaluated by GPT-4 and the AlpacaEval 2.0 compares the model outputs against GPT-4 outputs evaluated by GPT-4 Turbo. See evaluation details in Appendix $B,
All Comparison Approaches. In our study, we mainly conduct experiment using the LLAMA- 2-7B-BASE and 1L--MA22-13B-B- performance trained on LLAMA-2-7BB 2-B-BBBS if not specified. We compare with NEFTUNE [31] as the baseline, which adds noise to the embedding during the instruction tuning to increase the robustness of instruction- tuned models. See hyperparameter and implementation details in Appendix SC.
[62], and the OPT-6.7P [75] models. We report model
4.2
Main Results
In this section, we first evaluate the model performance of our approach and baselines across various tasks. Then we investigate the key factors that contribute to the effectiveness of our approach. Below we will discuss our findings in detail.
#1: Our approach IM can improve the performance of instruction tuning on various NLP tasks and open-ended generation benchmarks. Figure/ provides summary of the model's performance across both traditional NLP tasks and the AlpacaEval 1.0 benchmark. Table 1 offers
5

---

a detailed breakdown of experimental results for traditional NLP tasks across six categories, as well as performance on additional benchmarks for open-ended generation (i.e., MTTBeench and AlpacaBvva). The experimental results show that our approach (IM) can improve the performance of instruction tuning on various NL.P tasks and open-ended generation benchmarks. Specifically, on the Alpagasus Dolly 3k dataset, IM improves the overall mean score of NLP tasks to 48.95, an increase of 2.37 points from the baseline. Similarly, on the Alpagasus Dolly 9k dataset, we observe an improvement of 2.46 points in the mean NLP score. These improvements are mirrored in the LLM-based evaluations. Specifically, IM raises scores on the AlpacaEval 1.0 benchmark, achieving approximately ten-point increase the Alpagasus Dolly 9k dataset and doubling the performance on datasets such as Less MMLU Chat and Less Tydiqa However, the extent of improvement varies across different datasets. For example, the LIMA dataset shows more modest gains, prompting our further analysis to understand the factors influencing the effectiveness of IM.
#2: Our approach IM is especially beneficial for datasets characterised by lengthy instructions or prompts paired with comparably brief outputs. To better understand the impact factors on the effectiveness of IM. we extend our experiments to more instruction-tuninn datasets, such as Science Literature, Code Alpaca and Tulu V2. Interestingly, as shown in Figure 2 Left, we find that IM is particularly effective in scenarios where datasets characterised by lengthy instructions and shorter outputs, such as Less MMLU Chat and Less BBH ICL. For example, in datasets like Less MMLU Chat and Less Tydiqa, IM shows remarkable efficacy. In contrast, the Tulu V2 dataset, with an instruction to output length ratio of about 0.5, benefits less compared to the Science Literature dataset, which has a much higher ratio of 24.7. We hypothesise that this trend can be attributed to the tendency of language models trained on datasets with shorter outputs to overfit. cases where the instructions are longer, IM acts as an effective form of regularisation, mitigating this issue. For further details on the experimental setup, refer to the Appendix &C.
#3: Our approach IM performs better with fewer training examples. We find that another important factor in the effectiveness of IM is the quantity of training examples. Specifically, design additional experiments by sampling different numbers examples from the Tulu V2 datasets, which contain about 320k training examples and achieve modest improvement compared to other datasets in Figure 2 Left. We ensure that our samples maintain an of around 10. As all these samples are from Tulu V2, we can assume they are from the same distribution. As shown in Figure 2 Right, IM demonstrates substantial performance improvements on the AlaacaEvaal 1.0 the number of training examples decreases. This suggests that IM could be particularly valuable for developing robust models in resource-constrained scenarios or under the SAH. For further details on the experimental setup, please refer to the Appendix in SC.
length ratio
4.3 Instruction Modelling Mitigates Overfitting of Instruction Tuning
This section explores the underlying interpretation behind the effectiveness of our approach. Our experimental results demonstrate that IM can alleviate the overfitting problem of Instruction Tuning. Below we will discuss our findings in detail.
#1. Train and test loss analysis. Figure 3 clearly illustrates the effectiveness of our approach IM in mitigating overfitting issues compared IT. For both IM and IT, here only compute the loss over the output part. In the training loss distribution for the LIMA dataset, IM exhibits a slightly higher mean loss of 1.45 compared to 1.37 for IT, suggesting that IM does not overfit to the training data as much as IT does. This is further corroborated in the test loss distribution on the Tulu V2 dataset (using a 10% randomly sampled data set), where IM demonstrates a lower mean test loss of 1.17 compared to 1.32 for IT. This indicates that IM maintains better generalisation to new data, emphasising the model's 's capability learn effectively without fitting excessively to training examples. For more examples, see Appendix $D.
#2. BLEU score analysis. Here we generate outputs using the instructions from the training examples via greedy decoding, and then compare the generated outputs with the ground truth outputs in training examples and report the results. We use the BLEU score (up to n-gram order 4) [46] to measure the similarity between outputs, where higher score on outputs indicates a higher overlap with training examples. As shown in Table 2, outputs generated by IM consistently have lower BLEU

---

186
Mean Losc:
Mean
5000
Mean Losc:
3.27
2.45
1,37
3.32
IT
IM (Ours)
6000
MM (Du rs]
120
1.00
2000
60-
40-
1000
26
6.c
0.5
1.9
2.0
3.5
0.0
1.5
2.5
Training loss on the LIMA Dataset (1k Examples)
on the Dataset (32k Examples)
Figure 3: (Left) Training loss distribution for each example between our approach INSTRUCTION MODELLING (IM) and INSTRUCTION TUNING (IT) on the LIMA dataset. (Right) Test loss distribu- tion for each example between IM and IT on the Tulu V2 dataset, using a 10% randomly sampled data for efficacy. Mean losses are marked by dashed lines. For both IM and IT, here we only compute the loss over the output part. IM has a higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared to IT. See Appendix $D for more examples.
Table 2: Average BLEU Score comparison of IM and IT, where a lower score indicates less overfitting, Green and red arrows indicate performance changes against the baseline (IT).
Less Tydiqa
Lesa
Less
Alpagasus Alpaca 5k
Alpagaaus Dolly 9k
Alpagaaus Dolly 3k
LIXA
MMLU Chat
BBH ICL
IT
18.15
69.21
72.43 69.2013.23
60.96
72.26
61.76 60.61 (1.15
60.99 59.04]1.95
IM (ours)
17.30
65.631/358
53.94/7m
70.50
scores than those generated by IT. This suggests that IM produces outputs have less overlap with the ground truth outputs in training examples, indicating less overfitting.
Alpagesus
LESS
3000 HUP bo00 186)
48.5
tours)
65.5
68.0
65.0
67.5
45.5
46
67.0
45.0
Spocs
Spocs
Epoch
Spoch
Epoch
Figure 4: Mean performance on 18 NLP tasks over epochs using LLAMA-2-7 2-7------BB--EEEE This analysis suggests that IM experiences lower instruction tuning tax compared to IT.
#3. Instruction Tuning Tax on the NLP tasks. Previous works show that training LMs with RLHF causes an Alignment Tax on the NLP tasks [3, 44]. In this study, we observe that instruction tuning can sometimes lead to diminished model capabilities in some areas, such as multilinguality and commonsense reasoning. To this end, we further explore the impact of instruction tuning on the performance of NLP tasks. Figure 4 illustrates that our approach IM generally has lower instruction tnning tax compared to IT, suggesting better robustness under low-resource settings. We provide additional experiments for win rates across epochs Appendix $E.
#4. Can we simply use KL divergence loss as regularisation for instruction tuning? In this analysis, we demonstrate that the application of KL divergence loss in instruction tuning, which is widely used as regularisation for aligning LMs [3, 44, 72], cannot easily address the overfitting issue of instruction tuning. Table 3 offers a detailed comparison across various NLP benchmarks and open-ended language generation tasks, particularly using AlpacaEval 2.0, with models trained with and without KI. divergence loss. Our findings are as follows: (1) Incorporating KI. Loss reduces overfitting and reduces the performance degradation traditional NLP tasks. For example, on the
7

---

Table 3: Performance on 18 NLP benchmarks and AlpacaEval 2.0. Green and red arrows indicate performance changes against the baseline (LLAMA-2-7B -BBBB)). This analysis suggests that while applying KL. Loss in the instruction tuning helps mitigate performance degradation in NLP tasks, it substantially harms the model performance in open-ended generation tasks.
LIMA (IK)
ALPAGASUS DOLLY (9K)
LLAMA-2-7B-BASE
IT w/o KL Loss
IT w/ KL Loss
IT w/o KL Loss
IT w/ KL Loss
NLP Tasks AlpacaFval 2.0
49.32 0.01
48.79
49. 26.066...6
45.54 2.28
49.31,001 0.04
2.58
0.067005
Dolly dataset, incorporating KL Divergence Loss leads to less instruction tuning tax in NLP tasks, with scores rising from 45.54 to 49.31. (2) KL Loss detrimentally affects the model's instructions following abilities. For example, the LIMA dataset, we observe substantial decrease in AlpacaEval 2.0 from 2.58 to 0.06. For additional experiments and implementation details, see Appendix $F.
4.4 Further Analysis
Mean Performante (N)
OP IT
DPT-5.78 IT
64.
02.5
a1r0ss NLP (50)
Lama-113B IT 38
Lbama-138 IT
Menn performence
Figure Comparison of INSTRUCTION TUNING (IT) and INSTRUCTION MODELLING (IM) methods using OPT-6.7B (Top Row) and LLAMA-2-138 13B-BASE BASE (Bottom Row) trained on 7 instruction tuming datasets. (Left) The mean performance across 18 NLP tasks. (Right) The win rate on the AlpacaEval 1.0 benchmark.
#1. The advantage of our proposed method persists with different language models and sizes. As shown in Figure 5, our analysis demonstrates that our proposed method IM consistently outperforms the IT across different models and sizes, including OPT-6.7B and LL.AMA-2- 13B- BASE, on 18 traditional NL.P tasks and the AlpacaEval 1.0 benchmark. These findings underline the effectiveness of our approach irrespective of the underlying language model or its scale.
#2. Relationship between the model output length and the win rate. In this analysis. we explore the potential connection between win rates on the AlpacaEval and the increased output length [37, 76, 31]. As shown in Figure 6, our result reveals that our approach IM does not necessarily generate longer outputs than IT across different data utilisation levels from the Tulu dataset. Specifically, the output lengths for both approaches are similar despite varying levels of data utilisation. Furthermore, IM consistently outperforms the IT, suggesting that improvements performance as measured by win rates the AlpacaEval 1.0 are not dependent the output length. We provide additional analysis on other instruction tuning datasets under the SAH in Appendix $G.
#3. Our proposed method IM could further improve the model performance with NEFTUNE. Table demonstrates the combined effects of our proposed method IM and NEFTUNE on performance across various NLP tasks and the AlpacaEval 1.0 benchmark. The integration of NEFTUNE with IM
8

---

190
AlpacaEvol 1.0 Vin Rate (%)
590
IM (ours)
(ours)
Output Lengtn
10% (32k)
20% (655)
50% (1633)
100% (326k)
10% (32k)
20% (65k) 50% (1633)
100% (326k)
(%)
Utilization
(86)
Figure 6: (Left) Output length comparison between our approach INSTRUCTION MODELLING (IM) and INSTRUCTION TUNING (IT) across various data utilisation levels from the Tulu V2 dataset, as evaluated on the AlpacaEval dataset. (Right) Performance comparison (measured by win rate) between IM and IT the AlpacaEval 1.0 across various data utilisation levels from the Tulu V2 dataset. This analysis suggests that the improvement provided by IM is not necessarily associated with the increased output lengths. See more length analysis in Appendix $G.
Table 4: Performance comparison of IM and IM +NEFTUNE AlpacaEval 1.0 and various NLP benchmarks. Green and red arrows indicate performance changes against the baseline (IM). This analysis shows that adding NEFTUNE to IM could further improve model performance.
Less Tydiqa
Lees MMLU Chat
Less
Alpagasus Alpaca 5k
Alpagasus Dolly 9%
Alpagasus Dolly 3k
LIMA
ICL
AlpacaEval 1.0 Win Rate
IM
32.94
10.10
9.78
44.15
19.52
30.77
15.11
IM +NEFTUNE
30.77(a17
23.41
12.451267
48.2514.10
32.07
38.2817.51
23.35
Mean Performance Across NLP Tasks
IM
49.60
48.70
47.84
49.15
47,47
48.00
48.95
IM +NEFTUNE
49.47.,013
49.447u74
47.73/a11
48.62.ju.53
48.70
48.63
49.54
generally further improves the win rates in AlpacaEval 1.0, showing notable improvements in several datasets such as a 13.31% increase on Less Tydiqa and 12.55% boost on Alpagasus Alpaca 5k (in absolute). However, this combination leads to performance drop in certain contexts, such as lower performance on NI.P tasks on Less MMLU Chat and Less BBH ICL. This indicates that while NEFTUNE may enhance model robustness under certain conditions, its benefits are context-dependen, highlighting the need for the careful application of NEFTUNE when used in conjunction with IM to optimise effectiveness across diverse evaluation settings.
5 Epilogue
Conclusion. Our study proposes INSTRUCTION MODELLING, which trains LMs with loss over instructions rather than outputs only. Our experimental evaluations demonstrate that approach largely improves the performance of LMs on both NLP tasks and open-ended generation benchmarks in some scenarios, especially under the Superficial Alignment Hypothesis and low-resource setting where minimal training data is used for instruction tuning. Our analysis has shed light on two key factors that influence the effectiveness of our approach, (1) the ratio between instruction and output lengths, and (2) the quantity of training data, providing practical insights for optimising instruction- based training methods. Additionally, our analysis reveals the mechanisms behind the effectiveness of IM, particularly its ability to reduce overfitting, showing that applying instruction losses in some scenarios can lead to more robust and adaptable LMs.
Limitations and Broader Impact. Here we discuss some potential limitations and the broader impact of our work. Several limitations are outlined as follows: (1) The success of our approach relies on the quality and diversity of the instructions and prompts in the training datasets. Poorly defined or ambiguous instructions may undermine the effectiveness of IM, leading to sub-optimal performance; and (2) It is crucial to ensure that the instructions are ethically sound and free from

---

harmful or biased content. Training on inappropriate or toxic instructions may result in undesirable outputs. Previous works [5, 4] have extensively discussed the risks and potential harms associated with LMs, including the amplification of undesirable biases learned from training data [5, 2, 9.. Our work has the potential to positively impact the community by helping to mitigate overfitting, resulting in models that are more robust and generalise better to new data, especially in low-resource scenarios. This can enhance the reliability and trustworthiness of Al systems in real-world applications.
Acknowledgments and Disclosure of Funding
The authors express their gratitude to the NeurIPS reviewers and area chairs for their insightful discussions. Zhengyan is funded by the Research Studentship from University College London (UCL). Bin is funded by the Bloomberg Fellowship. The authors gratefully appreciate the generous support from OpenAl for providing API credits through the OpenAl API Researcher Access Program.
References
[1] Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862, 2022.
[4] Emily M. Bender, Timnit Gebru, Angelina McMillan- -aMii, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[5] Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online, 2020. Association for Computational Linguisties.
[6] Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florin, Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[8] Chris Callison-Burchh Philipp Koehn, Christof Monz, and Josh Schroeder. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop Statis- tical Machine Translation, pages 1-28, Athens, Greece, 2009. Association for Computational Linguistics.
10

---

[9] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
[10] Sahil Chaudhary. Code alpaca: An instruction-follovii llama model for code generation. https://gihubb.co//shil2222
14/odeealpac, 2023.
[11] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang. Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vi- jay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca model with fewer data. In The Twelfth International Conference on Learning Representations. 2024.
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ArXiv preprint, abs/2 107.03747 2021.
[13] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
[14] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Maric Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-fiiettuee language models. Journal of Machine Learning Research, 25(70): 53, 2024.
[15] Kevin Clark, Minh- Haah-TThag Luong, Christopher D. Manning, and Quoc Le. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1925, Brussels, Belgium, 2018. Association for Computational Linguistics.
[16] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/ (oss/883300557, 2018
[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[18] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm, 2023.
[19] Tri Dao. Flashattention-22 2: Faster attention with better parallelism and work partitioning, 2023.
[20] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. ArXiv preprint, abs/2306.1 1164,, 2023.
[21] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Ka- mar. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3309-3326, Dublin, Ireland, 2022, Association for Computational Linguistics.
11

---

[22] Guande He, Jianfei Chen, and Jun Zhu. Preserving pre-trained features helps calibrate fine-tuned language models. ArXiv preprint, abs/2305. 19249, 2023.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning Al with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[25] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409-14428, Toronto, Canada, 2023. Association for Computational Linguistics.
[26] Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. In The Twelfth International Conference on Learning Representations, 2024.
[27] Jeremy Howard and Jonathan Whitaker. Can Ilms learn from a single example?, 2023.
(28) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[29] Mathew Huerta-Enochian Instruction fine-tuning: Does prompt loss matter?, 2024.
[30] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradcep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate:
Enhancing Im adaptation with tulu 2, 2023.
[31] Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu. Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations, 2024.
[32] Aditi Jha, Sam Havens, Jeremey Dohmann, Alex Trott, and Jacob Portes. Limit: Less is more for instruction tuning across evaluation paradigms. ArXiv preprint, abs/2311.13133 2023.
[33] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online, 2020. Association for Computational Linguistics.
[34] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. ArXiv preprint, abs/ 17.0077,, 2018.
[35] Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012, Proceedings of the International Conference on Knowledge Representation and Reasoning, pages 552-561. Institute of Electrical and Electronics Engineers Inc., 2012. 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012: ; Conference date: 10-06-2012 Through 14-06-2012.
[36] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. The Twelfih International Conference Learning Representations, 2024.
12

---

[37] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cnstructiio-followwie models, 2023.
[38] Stephanic Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguistics.
[39] Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-Garcfa, Naoto Usuyama, Sheng Zhang, Tristan Naumann, Aditya Nori, Hoifung Poon, Javier Alvarez-Valle Ozan Oktay, and Stephanie L. Hyland. Compositional zero-shot domain transfer with text-to-text models. Transactions of the Association for Computational Linguistics, 11:1097-1113, 2023.
[40] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for aligmment? a comprehensive study of automatic data selection in instruction tuning. ArXiv preprini, abs/2312 15685, 2023.
[41] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381 2381-2391, Brussels, Belgium, 2018. Association for Computational Linguisties.
[42] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCI.: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791-2809, Seattle, United States, 2022. Association for Computational Linguisties.
[43] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task gener- alization via natural language crowdsoureing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland, 2022. Association for Computational Linguistics.
[44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkín, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[45] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534, Berlin, Germany, 2016. Association for Computational Linguistics.
[46] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8). 2019.
[48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 11:140:11440:7. 2020.
[49] Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266. 2019.
13

---

[50] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula. and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Shiroy-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020.
[51] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Mattco Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[52] Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 371-376, Berlin, Germany, 2016. Association for Computational Linguistics.
[53] Zhengxiang Shi, Yue Feng. and Aldo Lipani. Learning to execute actions or ask clarification questions. In Findings of the Association for Computational Linguisties: NAACI. 2022, pages 2060-2070, Seattle, United States, 2022. Association for Computational Linguistics.
[54] Zhengxiang Shi and Aldo Lipani. Don't stop pretraining? make prompt-based fine-tuning powerful learner. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[55] Zhengxiang Shi and Aldo Lipani. DePT: Decomposed prompt tuning for parameter-effin efficceent fine-tuning. In The Twelfth International Conference on Learning Representations, 2024.
[56) Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, and Yunlong Jiao. Rethinking semi-supervised learning with language models. In Findings of ACL 2023, Toronto, Canada, 2023. Association for Computational Linguisties.
[57] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11321-11329, Jun. 2022.
[58] Zhengyan Shi, Sander Land, Acyr Locatelli, Matthieu Geist, and Max Bartolo. Understanding likelihood over-optimisation in direct alignment algorithms, 2024.
[59] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. ArXiv preprint, abs/2400.066619 2024.
[60] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, pages 13003-13051, Toronto, Canada, 2023. Association for Computational Lin- guistics.
[61] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooinng Ilama model, 2023.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/22007.09288, 2023.
14

---

[63] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Selff Aligning language models with self-generated instruc- tions. In Anna Rogers, Jordan Boyd-Grabber, and Naoaki Okazaki, editors, Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13484-13508, 13508, Toronto, Canada, 2023. Association for Computational Linguistics.
[64] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabahii, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampar, Siddhartha Mishra, Sujan Reddy A, Generalization via declarative instructions on 1600+ NLPP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguisties.
Sumanta Patro, Tanay Dixit, and Xudong Shen. Cupper-Nauuralnnttuctrro
[65] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. Finetuned language models are zero-shot learners. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[66] Bin Wu, Jinyuan Fang, Xiangxiang Zeng, Shangsong Liang, and Qiang Zhang. Adaptive compositional continual meta-learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 37358-37378. PMLR, 23-29 Jul 2023.
[67] Bin Wu, Zaiqiao Meng, Qiang Zhang, and Shangsong Liang. Meta-learning helps personalized product search. In Proceedings of the ACM Web Conference 2022, WWW '22, page 2277-2287, New York, NY, USA, 2022. Association for Computing Machinery.
[68] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for instruction tuning, 2024.
[69] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
[70] Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. Rethinking the instruction quality: Lift is what you need, 2023.
[71] Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling LLM under token-crisis. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
[72] Adam X. Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, and Laurence Aitchison. Bayesian reward models for I1m alignment, 2024.
[73] Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In ICLR, 2024.
[74] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, 2019. Association for Computational Linguistics.
[75] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.01068. 2022.
15

---

[76] Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, 2024.
[77] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging LLMMaa--ajuddee with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[78] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[79] Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024.
[80] Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. AutoPEFT: Automatic Con- figuration Search for Parameter-Efficient Fine-Tuning. Transactions of the Association for Computational Linguistics, 12:525-542, 2024.
16

---

Appendix Overview
The appendix is structured follows:
Appendix SA datasets.
provides a brief description (with statistical summaries) for instruction tuning
Appendix $B
provides details of evaluation benchmarks and settings.
provides the experimental setting, implementation details and hyperparameters for all comparison methods used in our experiments.
Appendix $C
provides the supplementary experimental results to investigate the effect of our approach on training and testing losses.
Appendix $D
provides the supplementary experimental results to investigate the relationship between the win rate the AlpacaEvval 1.0 and the number of epochs.
Appendix $E
Appendix $F provides the mathematical formula for the Kullback-Lciberr (KL) divergence used in paper.
Appendix $G provides the supplementary experimental results to investigate the relationship between the output length and the number of epochs.
A Instruction Tuning Dataset
In this work, we use 13 popular datasets from previous instruction tuning research. For the WizardLM, Sharegpt, Science Literature, and Code Alpaca datasets, directly use the subset provided by the previous work [30]. Refer to the dataset statistics Table 5. addition, we provide an analysis of the output length distribution for LIMA, Alpagasus Dolly 3k, Alpagasus Dolly 9k, Alpagasus A Alpaca 5k, Less MMLU Chat, Less Tydiqa, and Less ICL datasets, as shown in Figure 7.
Table 5: Statistical summary for various instruction tuning datasets. The table includes sample sizes, the average total length of instructions and outputs, the average output length, and the average instruction length with their standard deviations, and ratio calculations.
Output
Outpot
Output/Instruction
Instruction/Output
LINA
1,030
184.47
112.75
191.31
11.72
79.28
10.6124
0.09 12
Less NNLO Chat
13,533
225.19
8.24
16.42
216.95
301.64
0.0380
26.3316
Leas Tydiqn
13,533
172.44
25.13
42.60
147.3 31
235.37
0.1706
5.862
13,533
262.00
200.60
0.3063
Alpagaaus Delly
2,996
Alpagauus Bolly 9k
9.229
73.40
56.62
48.91
16.79
11.33
3.3727
0.2965
Alpagaaus Alpada 5k
5,305
48.29
90.81
34.44
17.48
12.45
1.7631
0.5672
Talu V2
326.181
541.16
343.56
575.32
197.60
345,
1.7387
0.5751
32,618
Talu V2 (50%)
163,090
515.63
3110.67
$71.06
174.97
343.45
19470
0.5136
Talu V2 (20%)
63.236
504.56
336.89
$62.46
167.68 68
331.24
2,0992
0.4977
NizaräLM
30,000
350.05
258 35
182.98
91.71
86.19
2,8170
0.3550
2144 24
4.0695
0.2457
Sciance Literature
Stanford Alpaca
$2,002
63.77
45.18
44.97
18.59
12.42
2,4302
0.4115
Coda Alpaca
20,022
49.74
27.40
27.35
22. 34
10.67
1.2262
0.8156
Evaluation Datasets and Details
We use the open-source repositories, LM-Evaluation Harness and Huggingface Dataset as the evaluation tools. We describe our evaluation setup below:
"attps://uugginggacco.o http://g.......cooo/////tttaaaaa
17

---

Lima Data
Dally (3k from Alpagasus)
Dolly (9k from Alpagasus)
Alpaca (from Sk Alpagasus)
dveroge Lenggn 462
160
Astrage Length: 25.28
295
deciogg Langin 26,M
TT1
Ascrage tonpth: at
Nadian Langth: 260.02
Mode
140
ar Lengt:
175-
Note sough
Nadian Largt:
350
Median Langt::
Frequenicy
Fresuencyy
11
Frequancy
125-
190
Freeuenn
250
300
832-
200-
01
3530
Output Length
Output
Output Length
Output Langth
FLAN V2
Dolly (lyDQAA)
12 FLAN V2
Dolly (MMLU Chat)
FLAN V2
Dolly (BBH ICL
300
10.
307
Mode Length: 2.00
Hode Langth: 2.20
Noze Jangm:
Frequency
Frequency srrcyy
Fred
120
207
121
.0:
30
19
121
184
UP
Output Length
Output
Output Length
Figure 7: Distribution of output lengths of instruction tuning datasets. This figure presents histograms for the distribution of output lengths across seven datasets, including LIMA, Alpagasus Dolly 3k, Alpagasus Dolly 9k, Alpagasus Alpaca 5k, Less MMLU Chat, Less Tydiqa, and Less BBH ICL. Each subplot displays the frequency of output lengths with key statistical indicators: the average (red dashed line), median (green dashed line), and mode (blue dashed line) of each dataset. The last three subplots employ logarithmic scale both axes to better illustrate data spread.
We evaluate the model using the dataset at the huggingface dataset We follow the protocol outlined in HuggingFace Open LLM Leaderboard 6 The evaluation uses multiple-choice questions formatted as the question followed by four choices (A, B, C, D) and prompting for an answer. We calculate the mean accuracy (acc) across test examples.
MMLU.
BBH. The model evaluation utilises the dataset at the huggingface dataset ", sseeifically tested on the 'test' split without the use of few-shot examples. We follow the setup in previous works [30, 60]. The evaluation metric is the exact match score, averaged (mean) to assess performance. Generation is constrained to maximum of 1024 tokens, with termination upon encountering specific delimiters such as "
", "Q", or double newlines. The generation is greedy decoding (temperature set to 0.0) and does not use sampling. Answer extraction employs regex patterns to identify responses immediately following "the answer is" and captures only the first occurrence.
GSMSK. We evaluate using the dataset at the huggingface dataset 8 focusing on arithmetic problem-solving in the 'test' split. We follow the HuggingFace Open LLM Leaderboard to 8 few-shot examples. Exact match is the chosen metric, with case insensitivity and select regex- -bsed filtering of common punctuation and formatting characters to ensure precise validation of numerical answers. The primary focus is on extracting and comparing the final numerical answer to the model's 's output using a strict regex- -ased match setup.
Evaa We evaluate using the dataset and the evaluation code from the previous work [30]. We report the performance of the pass@1. I. We perform the decoding using two different temperatures, 0.1 and 0.7. We report the better pass@1 from these two decoding results.
train
httph//gw....e.oooooeeeee//////a /otpl//////ee____/eee larpee__l___eeerrrii leaderboard
7 hnttp::/huggingafaeec//////aeaee http:///g....eee.aaaaaaaaaaaaaaa
hattpe://hugginggacc.oo http://cccc.co/taatttt///g/aa/aa
18

---

The evaluation setup for the dataset at the huggingface dataset 9 uuiilises multiple-choiee format. We follow the HuggingFace Open LLM Leaderboard to 25 few-shot examples. The performance metric used is mean normalised accuracy (acc_norm).
ARC.
CoQA. We conduct the model evaluation on the dataset at the huggingface dataset 10, We follow the HuggingFace Open LLM Leaderboard to 0 few-shot examples. The output generation terminates upon encountering a new line followed by "Q:". The mean Fl score is used as the evaluation metric.
PIQA. Evaluation the dataset the huggingface dataset || involves multiple-choice. The evaluation incorporates 10 few-shot examples, according to the LIMIT [32]. Performance measured using the mean normalized accuracy (acc_norm).
The dataset at the huggingface dataset 12 is evaluated in a multiple-choice format. The mean normalized accuracy (acc_norm) is used as the evaluation metrie.
OpenBookQA.
The evaluation of the model on the dataset at the huggingface dataset 13 is performed using a loglikelihood output type. The mean accuracy used as the evaluation metric.
LAMBADA.
HellaSwag. In the 'hellaswag" dataset at the huggingface dataset model evaluation is conducted using a multiple-choice format. We follow the HuggingFace Open LLM Leaderboard to 10 few-shot examples. The mean normalized accuracy (acc_ porm_m is used as the evaluation metric.
The Winograd Schema Challenge. The evaluation is conducted using a multiple-choice format on the 'test" split at the huggingface dataset 15, The mean accuracy is used as the evaluation metric.
Winogrande. The 'winogrande" dataset is assessed using a multiple-choice format at the hugging- face dataset 16, We follow the HuggingFace Open LLM Leaderboard few-shot examples. The mean accuracy is used as the evaluation metric.
LAMBADA. For this dataset, evaluation is conducted using the loglikelihood output type on the 'test" split the huggingface dataset !7. This variant focuses predicting the last word of text passages in English. The mean accuracy is used as the evaluation metric.
Translation Benchmarks WMT. The evaluation of the translation capabilities is performed on the WMT 2 and WMT 2016 datasets at the huggingface dataset. Here we use the 'ter' score as the evaluation metric.
TruthfulQA. We use the dataset at the huggingface dataset 20 We follow the setup at the Hug- gingFace Open LLM Leaderboard using the few-shot examples. The mean accuracy used as the evaluation metric.
ToxiGen. use the dataset at the huggingface dataset 21 The task is assessed using multiple- choice framework to evaluate the model's capability identify hateful content in text statements. The mean accuracy is used as the evaluation metric.
19

---

Hendrycks Ethics. We use the dataset at the huggingface dataset 22, with multiple-choice format. The model aims to detect whether described actions in various contexts are ethically wrong. The prompt format integrates a specific scenario followed by structured question: "Is this wrong?" and then prompts for an answer with options 'no" or "yes". The mean accuracy is used as the evaluation metric.
C Implementation Details
Experimental Design for Figure 2 Left. Here we present a detailed experimental design for variety of datasets, including LIMA, Alpagasus
Figure 2 Left. We perform experiments Dolly 3k. Alpagasus Dolly 9k, Alpagasus Alpaca 5k, Less MMLU Chat, Less Tydiqa, Less BBH ICL, Tulu V2, Code Alpaca. Stanford Alpaca, Science Literature, WizardLM, and Sharegpt. Furthermore, to evaluate the effectiveness of IM on datasets with different instruction- to-output length ratios, we select three subsets from Tulu V2. Each subset contains 3,000 training examples, with instruction-to- -atructiontto-ouuuut length ratios of approximately 5, 10, and 15, respectively.
Experimental Design for Figure 2 Right. Here we provide a detailed experimental design for Figure 2 Right. We strategically sampled varying sizes of training examples from the Tulu v2 dataset to investigate the effectiveness of IM with different sizes of training examples. Starting with approximately 320,000 examples in the Tulu V2 dataset, we create subsets of data ranging from as few as 1,000 to as many as 35,000 examples. These subsets were selected randomly, ensuring a representative mix across different scales. We adhered to fixed instruction-tor to-tutput length ratio of approximately 10 to maintain consistency in training conditions across all samples. We train the CA-MA-2--B-BAA on all these subsets and evaluate them respectively.
Table 6: Hyperparameters and configurations for supervised fine-tuning.
Hyperparameter
Assignment
GPUs
2 or 4 A100 80G GPUs, 2 48G A6000 GPUs
Batch size per GPU
Total hatch size
128
Number of epochs
2, 3, or 10
Maximum sequence length
2048
Learning rate
x 10-5
Learning rate optimizer
AdamW
Adam epsilon
1e-6
Adam beta weights
0.9, 0.98
Learning rate scheduler
Linear with warmup
Warmup proportion
0.03
Weight decay
Mixed precision
bfl6
Gradient accumulation steps
Calculated dynamically
Implementation Details. In our study, we fine-tune the ILLMA-2-7B, LLaMA-2-13B and OPT-6.7 model using four 1100 80G GPUs, with a per-GPU batch size of 1 and a total batch size of 128, employing learning rate of 2e-5. Training typically proceeds for 2 epochs with maximum sequence length of 2048 tokens. We utilise gradient accumulation, calculated to effectively distribute training steps across the available hardware, resulting in larger batch sizes despite hardware limitations. We employ mixed precision (bf16), linear learning rate scheduling with a warm-up ratio of 0.03, and a weight decay of O. To optimise our training. we use DeepSpeed with stage 3 configuration without offloading. Our setup also includes the usage of Flash Attention [19] and slow tokenization
htttps://huggiingaace httppp///aaggiiggooooooosststttr
20

---

to enhance training efficiency and compatibility. Our code is implemented using Pytorch 24 and Huggingface Table lists the hyperparameters.
D Train and Test Loss
Moan Loss:
Moan Loodi
Mean Loskig Mean Lossi
800
IM (Ours)
(Ours)
4000
400
2000
1000
0.0
0.5
3.0
3.5
2.5
2.5
0.5
1.5
1.5
2.0
2.5
3.0
3.5
Training loss on the Alpagesus Dolly 3k detaset
Test Loss on the Tiltu dletasee
Figure 8: (Left) Training loss distribution for each example between our approach INSTRUCTION
MODELLING (IM) and INSTRUCTION TUNING (IT) the Alpagasus Dolly 3k dataset. (Right) Test loss distribution for each example between IM and IT on the Tulu V2 dataset, using a 10% sampled data. Mean losses are marked by dashed lines. For both IM and IT, here we only compute the loss over the output part. IM has higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared IT.
Mocan Loser
10000
Moan Lossi
Moan Losa:
10000
Moan Losa:
IM (Durs)
8000
IM (Ours)
6000
Dount
0000
Count
4000
4000
2000
8.0
1.0
1.4
0.0
1.5
2.0
2.5
Taining loss on the Less MMLU-CHAT dataset
Tist Loss on the Tulu dataset
Figure 9: (Left) Training loss distribution for each example between approach INSTRUCTION MODELLING (IM) and INSTRUCTION TUNING (IT) on the Less MMLU Chat dataset. (Right) Test loss distribution for each example between IM and IT on the Tulu V2 dataset, using 10% sampled data. Mean losses are marked by dashed lines. For both IM and IT, here we only compute the loss over the output part. IM has higher train loss with lower test loss, suggesting that IM effectively mitigates the overfitting issues compared to IT.
In this section, we provide additional experiments regarding training and testing loss distributions. Figure 8 focuses on the Alpagasus Dolly 3k and Tulu V2 datasets, displaying how IM tends to exhibit higher training losses yet achieves lower test losses compared IT. Similarly, Figure compares these methods the Less MMLU Chat and Tulu v2 datasets under analogous conditions.
E
The impact of Epochs on the Win Rate
Figure 10 illustrates the comparative analysis of AlpacaEval 1.0 scores across different epochs for two datasets, LIMA and Alpagasus Dolly 9k datasets. We evaluate the performance of IM and IT
2https://github. http:://ggthuu....maaaaaa www./////aa/appeeaeneeeeeea insttrrr hhatpe://ppttorch.ogg
25 "atrpes//huggingaace.oo
21

---

(a) Lima Dataset
(b) Alpagasus 9k Dolly Dataset
AlpacaEval 1.0 Win Rate (96
35.0
(Ours)
32.5
AlpacaEval Win
30.0
27.5
35
22.5
30
20.0
4
6 Epoch
8
10
2
4
10
Epach
Figure 10:
AlpacaEval 1.0 performance trends for IM and IT approaches the LIMA and Alpagasus Dolly 9k datasets across different epochs.
different numbers of epochs. IM consistently surpasses IT in performance the Alpagasus Dolly 9k dataset, while the performance both approaches comparable on the LIMA dataset.
Applying KL Divergence Loss for Instruction Tuning
In this section, first briefly introduce the Kullback-Leibler (KL) divergence and then introduce the experimental details. Future work can investigate the effectiveness of parameter-efficien fine-tuning approaches as the regularisation [80, 55, 28].
Kullback-Leible Divergence. regularisation method in the fine-tuning of LMs, helping to mitigate overfitting by constraining the fine-tuned model to remain similar to the pre-trained model [44]. Specifically, the KI. divergence added the fine-tuning objective as per-token regularisation term between the fine-tuned LM (x)),, and the pre-trained LM, T (z). For supervised fine-tuning with the next token prediction loss, the training objective incorporating KL divergence is computed as follows:
Beilbackk-eibbers (KL) divergence is commonly employed as
(0) EEAD
logg (o + Σ
(5)
where À is regularisation parameter that balances the loss due to the next token prediction and the KL divergence, and i (=a.-1) represents the next token distribution of the fine-tuned or pre-traee pre-trained LM conditioned on the preceding context.
Table 7: Performance on 18 NI.P tasks and AlpacaEval 2.0, with various values of X, trained on the (LLAMA-2-74) 2-7B-BASE)
NLP Tasks
AlpacuEval 2.0
LLAMA-2-78 1-AMA-2-7B-BASE BASE
49.32
0.01
0.01
2.58
À 0.1
48.77
2.44
49.26
0.06
Ablation study on the effect of X. In Table 3, we set the value of the as 1.0. Here we provide additional experiments with different values of X. Table presents the model performance on the NLP tasks and AlpacaEval 2.0. This aligns our observations in $4.3.
G The impact of Epochs on Output Lengths
Figure 11 illustrates the average output length of various models across different epochs. We report the output length on four different datasets, including Alpagasus Dolly 3k, Alpagasus Dolly
22

---

403.
10.00
103.0
252.
06.0
203.0
10407
Figure 11: Comparative analysis of output lengths for IM and IT across different epochs Alpagasus Dolly 3k, Alpagasus Dolly 9k, LIMA, and Less Tydiqa datasets.
9k, LIMA, and Less Tydiqa Each line represents the average output length of model, with epochs ranging from 2 to 10, and is accompanied by error bars that denote the normalised standard deviation (10%) of the output lengths. Our experimental results show that approach IM does not consistently increase the output length and that win rates are not necessarily associated with the length of the output.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the introduction, we explicitly detail our contributions and provide hyper- links that direct readers the corresponding sections of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
Please refer to $5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Appendix $A for training data details, Appendix $B for evaluation details, Appendix $C for implementation details. We also provide all the codes at https: / //t/ thub. www.coo///aaaee Somm/Zhegxxii angShi, instructions to reproduce our work, including details about the coding environment, data, training procedures, evaluation, and analysis.
with comprehensive
25

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Please refer to Appendix $A for training data details, Appendix $B for evaluation details, Appendix $C for implementation details. We also provide all the codes at /ttps://githhr gigghu........ www.coo///aaa. Zhengxi angShi/ InstructionModelllii instructions to reproduce our work, including details about the coding environment, data, training procedures, evaluation, and analysis.
with comprehensive
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
cablic/guides htt/////eemmiiiioooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wtwpp.cccppubaaaa http//ubllclggggeeeeeeeeee Sodeede//ooeemmmmiii.c wewteesuumsssiinnne. ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
26

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Appendix $A for training data details, Appendix $B for evaluation details, Appendix $C for implementation details. We also provide all the codes at with comprehensive
https: hhtbbb.....t com/ Zhengxi angShi instructionlooelllait instructions to reproduce our work, including details about the coding environment, data, training procedures, evaluation, and analysis.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars for some results (see Figure 6). For other experiments, error bars are not reported because would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
27

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please refer to Appendix &C for implementation details, including GPUs used in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neuut http::::////upp.////e/ cc wwwbllc//tthissuieeeeee
Answer: [Yes]
Justification:
Our research conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g. if there is special consider- ation due laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to $5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
28

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper does not address or focus on topics such as data security, privacy, or ethical considerations regarding the release of sensitive technology.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators and original owners of the assets used in the paper are properly credited. This is achieved by acknowledging their contributions directly within the text, referencing their original works, and by including citations their publications sources. For example, we acknowledge the contribution of other open-source repositories in our repository at https: / //githug. com www.coo///////. comm/Zhengxiangn
Justification:
Instructioooooellllli
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
29

---

Justification: We will open-source our repository at the GitHub repository. Please check our code at https: github. www//henngiiagaaeeeeecceeeeeeeee http://www...nggiiii/nttttttttna
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "2aqBpawV9u", "reviewer": "Reviewer_z7Cy", "summary": "In this work, authors propose to use instruction modeling (using loss over the full instruction-output pair) instead of just instruction tuning (using loss over the output given the instruction) as a method for supervised finetuning on LLMs. The authors demonstrate consistent gains over multiple benchmarks using this simple technique and hypothesize that the gains occur due to reduced overfitting to instruction tuning dataset.", "questions": "1. Do you have any results on other LLM families: Phi, MPT or Gemma?", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is quite simple and scalable as such would be of great interest to practitioners in the community.\n2. The authors try to characterize the gains and attribute it to the extra signal obtained in the case of short outputs as well as to reduced overfitting of the approach. Both characterizations seem intuitive in explaining the results obtained. Further, the authors show that proposed method is complementary to NEFTUNE, another recent method which results in consistent empirical gains.", "weaknesses": "1. Limited Models Explored: While the authors do a good job of experimenting with multiple instruction datasets, the experiments are only done with LLama-2 (7B and 13B) and OPT-7B models. This forgoes a plethora of open source models, experiments on which would have greatly strengthen the paper. Given that these LLMs are all from the Meta family of LLMs -- this raises some suspicion as to the generality of the proposed technique when different architectures or modeling assumptions are made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort and time of the reviewer (z7Cy). We are thrilled to receive positive feedback on the simplicity and scalability of our method, and the recognition of our efforts to characterise the gains. The reviewer's acknowledgement of our intuitive explanations for the results and the complementarity of our method with recent approaches like NEFTUNE is particularly gratifying. We would like to address the reviewer's valuable feedback as follows:\n\n\n> This forgoes a plethora of open source models, experiments on which would have greatly strengthen the paper. Given that these LLMs are all from the Meta family of LLMs -- this raises some suspicion as to the generality of the proposed technique when different architectures or modelling assumptions are made. (......) Do you have any results on other LLM families: Phi, MPT or Gemma?\n\nWe appreciate the reviewer's concern about the generalizability of our results across different model families. In response to the reviewer's suggestion, we conducted additional experiments on additional models including Phi and Gemma, which show qualitatively similar results (see Table 2 in the rebuttal pdf). Our results show that our findings still hold with different language models. \n\nThe meta family of LLMs are very standard transformers [1]. Indeed, the designers note that they tried to avoid innovating on the model architecture [2]. For example, the Llama model family has very standard features such as RoPE embeddings [3]. As such, we would not expect Llama family models to behave differently from other models.\n\n### Reference:\n\n[1] Llama 2: Open Foundation and Fine-Tuned Chat Models.\n\n[2] The Llama 3 Herd of Models.\n\n[3] RoFormer: Enhanced Transformer with Rotary Position Embedding", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zwpBGTdYyJ", "reviewer": "Reviewer_b1op", "summary": "This paper proposes Instruction Modeling (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. The method is found to be effective on NLP tasks and open-ended generation benchmarks. This paper found two key factors that influence the effectiveness of the approach: (1) the ratio between instruction and output lengths, and (2) the quantity of training data. There are also additional analysis that shows IM can reduce overfitting.", "questions": "1. Do you have any analysis on the quality of the instruction part? How does the quality of the instruction part affect the performance of the model? See \"Weaknesses\" part.\n\n2. I might have missed this one in the paper, but how do you pack multiple samples during finetuning? Do you concatenate samples to form a fixed length and then trucate them? Or do you add padding tokens?\n\n3. Do you have any scenarios where the IM is significantly worse than the baseline? If so, can you provide some examples? I want to know under what conditions IM is not effective and should be avoided.\n\n4. Do you think IM can be used as a drop-in replacement for the current finetuning process? Or do you think it should be used in conjunction with the current finetuning process?", "limitations": "Limitations are being discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Generally the paper is well written and covers a wide range of experiments. It forms a comprehensive study on whether insturction tuning should calculate loss over the instruction part. Actually, it's a little surprise to me that calculating loss over the instruction part can improve the performance of the model, and the reasoning behind this explained by the paper is insightful and interesting. Overall I think:\n\n1. The finding is interesting and novel. This is in contrast to the common practice of calculating loss over the output part only. I think it's going to have broad impact on how people finetune LMs in the future if all the claims are true.\n\n2. The experiments and evaluations are comprehensive. THe paper covers a wide range of instruction datasets and quite a few widely used benchmarks. \n\n3. The paper has a lot of details, in both the main body and the appendix. The appendix is very detailed and informative, which is good for reproducibility.", "weaknesses": "A part I feel that's missing in the paper is analysis on the **quality of the instruction part**. One reason that people didn't calculate loss over the instruction part is that the instruction part is usually noisy and not well-structured, e.g, ShareGPT has a lot of user shared low-quality instructions. The paper should analyze how the quality of the instruction part affects the performance of the model. Intuitively, if the model is also learning from the noisy instruction part, it's possible that the model will learn undesirable patterns from the instruction part, but I don't see this being discussed in the paper.\n\nMoreover, the number of LMs being finetuned and tested is relatively small. It will be better if more families of LMs and larger models (eg, 60-70B params) are tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort and time of the reviewer (b1op). We are grateful for the positive feedback on our paper's comprehensiveness, novelty, potential impact, extensive experiments, and detailed presentation. We are particularly pleased that our well-written reasoning and thorough appendix were recognised. We would like to address the reviewer's valuable feedback as follows:\n\n> A part I feel that's missing in the paper is analysis on the quality of the instruction part. \n\nWe thank the reviewer for this insightful comment. In response to this feedback, we have conducted additional experiments to address this aspect of our work.\n\nWe acknowledge that it is not trivial to define what constitutes high-quality data, and it is particularly challenging to judge the quality of instructions objectively. Given these difficulties, we have chosen to follow Alpagasus \\[1\\] in ICLR 2024 and LESS \\[2\\] in ICML 2024 where the overall quality of instructions and completions has been evaluated.\n\nSpecifically, we compare high-quality data subsets identified in previous work \\[1,2\\] with randomly selected data subsets of the same size from the same underlying source datasets. We utilised four datasets:\n\n- Less Tydiqa (13 533 examples), selected from the source datasets, Flan V2 and Dolly  \n- Alpagasus Dolly 3k (2 996 examples), selected from the source dataset Dolly  \n- Alpagasus Dolly 9k (9 229 examples), selected from the source dataset Dolly  \n- Alpagasus Alpaca 5k (5 305 examples), selected from the source dataset Alpaca\n\nAs shown in Table 1 in our attached PDF file, our results show that our Instruction Modeling (IM) approach performs better than baselines on both high-quality and randomly selected datasets. This finding suggests that IM is robust across various data qualities. We will include these results in our revised paper.\n\n> Moreover, the number of LMs being finetuned and tested is relatively small. It will be better if more families of LMs and larger models (eg, 60-70B params) are tested.\n\nWe conducted additional experiments on additional model families including Phi and Gemma, which show qualitatively similar results (see Table 2 in the rebuttal pdf).  However, we were not able to finetune 60-70b parameter due to computational restrictions.  We will add a note on this point to the limitations section.\n\n> I might have missed this one in the paper, but how do you pack multiple samples during finetuning? Do you concatenate samples to form a fixed length and then truncate them? Or do you add padding tokens?\n\nWe appreciate the opportunity to clarify this point. We follow the standard training paradigm of instruction tuning, where we do not pack training examples. Instead, we add the padding token to the maximum sequence length for each training example. We will make this clearer in our revised paper.\n\n> Do you have any scenarios where the IM is significantly worse than the baseline? If so, can you provide some examples? I want to know under what conditions IM is not effective and should be avoided.\n\nWe appreciate the reviewer's interest in the potential limitations of our method. In our extensive experiments, we have not observed scenarios where IM is significantly worse than the baseline. We have observed that when two specific scenarios are not met—namely, (1) limited instruction tuning data and (2) short completions—it generally does not significantly impact performance whether the prompt loss is included during training, leaving the performance more empirical. We acknowledge that there might be specific edge cases or scenarios not covered in our current experiments. For example, if instruction parts in training examples include more toxic or harmful content, understanding how our approach affects model performance in such situations remains an area for future research.\n\n> Do you think IM can be used as a drop-in replacement for the current finetuning process? Or do you think it should be used in conjunction with the current finetuning process?\n\nWe thank the reviewer for this question as it allows us to clarify our position. We are not proposing IM as a complete replacement for current fine-tuning processes. Rather, our key message is that whether masking user prompts during instruction tuning could be more empirical. We find that the effectiveness of masking user prompts can vary depending on factors such as the amount of instruction tuning data available and the length of completions. In some cases, including the prompt loss during training might be advantageous. We appreciate the opportunity to clarify this point and will ensure it is clearly stated in our revised paper.\n\n### Reference: \n\n\\[1\\] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR 2024\\.\n\n\\[2\\] LESS: Selecting Influential Data for Targeted Instruction Tuning, ICML 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LReCpVLRup", "reviewer": "Reviewer_kinw", "summary": "The propose that when updating models using instruction tuning, the models should also be updated based on loss on the instruction itself.\n\nThis is a simple change that is un-intuitive, so the successful results are impressive.", "questions": "What is considered an \"instruction\" when training? The text mentions that \"static\" tokens like \"<user>:\" are masked out, but many instructions are static, i.e. \"Tell me if the second sentence entails the first\". If instructions are non-static are they mostly example like in In-Context-Learning?\n\nThey mention that IM is most effective in settings where the instructions are long. Is it possible that most of this gain is for essentially continuing pre-training on in-domain data a la https://arxiv.org/abs/2004.10964\n\nSimilarly, do you think the success in the SAH settings is possibly an artifact of getting a model that is just trained on a lot more data? How many more tokens does an IM model see than in IT model in that setting? Is there a way to hold the number of tokens seen constant? For example by training the IM model on even fewer examples to show that the gain really is from the training on the instruction?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "They include experiments from many different datasets and look at multiple different LLMs. And there results generally hold across them.", "weaknesses": "It is unclear what the paper considers an \"instruction\", there should be some examples of it. At one point it says that \"static parts\" of the instruction, like a \"<user>:\" token, are masked from the lost. Does that mean the rest of the instruction is all examples? An instruction like \"Tell me the sentiment of this text\" is going to be static across examples.\n\nWhen looking at table 1, although their IM methods wins in terms of average performance, there are many datasets where it performs worse. This is glossed over in their prose.\n\nTheir \"Loss analysis\" experiments are not convincing as they don't fully capture the changes to the model that occur in each setting. By only checking the loss on the continuations, they don't capture how their IM model might be more overfit to the instructions themselves. For example, if you looked that the train loss for the whole Instruction + continuation for each models, the IM model would mostly likely be far lower than the loss for the instruction tuned model. It makes sense that the IM model is going to be worse on the continuations after the same amount of training as updates to get better as the instruction itself will cause conflicts. Discarding the models performance on the Instructions creates a self-fullfilling prophecy of the loss being higher when only the continuations are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are pleased to receive positive feedback on the extensive experiments and the consistency of our results across diverse settings. We would like to address the reviewer's valuable feedback as follows:\n\n> It is unclear what the paper considers an \"instruction\", there should be some examples of it. At one point it says that \"static parts\" of the instruction, like a \"<user>:\" token, are masked from the lost. Does that mean the rest of the instruction is all examples? An instruction like \"Tell me the sentiment of this text\" is going to be static across examples.\n\nIn our work, the instructions contain the following parts: \n- Static templates. These are pre-defined templates randomly selected (please refer to Lines 4-38 in our code repository file `src/instruction_encode_templates.py` (Please find the link in our abstract).   \n- Dynamic instructions from the user. These are the actual user-provided instructions contained in the “instruction” key of the examples.  \n- Formatting tokens. We consistently mask out formatting tokens like \"\\<user\\>:\" to concentrate on the content.\n```json\n{\n  \"instruction\": \"In 1994 , the Polish ambassador to South Africa , Mr Durrant , presented the `` Warsaw Cross of Insurrection to SCieniuch 's widow .\\nIn 1994 , the Polish ambassador to South Africa , Mr Scieniuch , presented the `` Warsaw Cross of Insurrection to Durrant 's widow .\\n\\nAre these two sentences paraphrases of each other?\\nOptions are:\\n 1). no.\\n 2). yes.\\n\",\n  \"completion\": \"1)\",\n},\n{\n  \"instruction\": \"Give three tips for staying healthy.\\n\\n\",\n  \"completion\": \"1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \\n2. Exercise regularly to keep your body active and strong. \\n3. Get enough sleep and maintain a consistent sleep schedule.\",\n}\n```\nWe will add these examples in the revised paper. \n\n> When looking at table 1, although their IM methods wins in terms of average performance, there are many datasets where it performs worse. This is glossed over in their prose.\n\nWe will modify the text to acknowledge this point. We also respectfully present a different perspective: it is not trivial to discern a clear trend or pattern of how different methods affect model performance through a single or small number of tasks. This is precisely why our work includes 23 NLP benchmarks from 6 different categories and 3 LLM-based evaluations. Our analysis aims to provide a holistic view of how our approach and baselines impact downstream task performance. The observed performance variations across datasets offer valuable insights for future research. \n\n> Their \"Loss analysis\" experiments are not convincing as they don't fully capture the changes to the model that occur in each setting. \n\nOur loss analysis focuses on the loss of continuations, as including the loss over the instruction part for our approach could lead to an unfair comparison. In our paper, we also conducted a BLEU Score analysis to investigate the potential overfitting issue. This analysis compares the overlap between model outputs and the ground truth outputs in training examples. Our results show that our approach produces outputs with less overlap with the ground truth outputs in training examples, indicating less overfitting compared to the baseline models.\n\n> They mention that IM is most effective in settings where the instructions are long. Is it possible that most of this gain is for essentially continuing pre-training on in-domain data a la https://arxiv.org/abs/2004.10964\n\nWe thank the reviewer for this insightful question. Our approach differs from continued pre-training in several aspects:\n- Our approach exposes the model to more loss signals under limited resources, whereas continued pre-training leverages **large-scale** unlabeled text to learn in-domain knowledge.  \n- Our method does not require an additional source of data and can be applied almost for free within existing finetuning pipelines, and we can therefore expect to see wide use.  \n- Continued pre-training typically targets performance improvement in specific domains or tasks. In contrast, our approach demonstrates effectiveness across a broad spectrum of 23 NLP benchmarks and 3 LLM-based evaluations, indicating greater generalizability.  \n- Our method specifically targets the relationship between instructions and their corresponding outputs, a focus not present in traditional continued pre-training approaches.\n\nWe will clarify these differences in our revised version.\n\n> Similarly, do you think the success in the SAH settings is possibly an artifact of getting a model that is just trained on a lot more data? How many more tokens does an IM model see than in IT model in that setting? Is there a way to hold the number of tokens seen constant? For example by training the IM model on even fewer examples to show that the gain really is from the training on the instruction?\n\nWe would like to argue that the increased exposure to tokens in our approach is not a limitation, but rather a key advantage of our method. This aligns with observations in other areas of LLM research. For instance, Yi Tay discussed in his blog [1] that one potential reason decoder models outperform encoder-only models is due to greater \"loss exposure\" in the next token prediction objective compared to the denoising objective.\n\nSimilarly, in our approach, exposing the model to more loss signals through instruction modelling is beneficial. This is particularly useful in scenarios with limited instruction tuning resources. By including loss on the prompt during training, we leverage more of the available data, potentially leading to more robust and effective models.\n\n### Reference:\n[1] https://www.yitay.net/blog/model-architecture-blogpost-encoders-prefixlm-denoising", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UnwaWitZJ1", "reviewer": "Reviewer_vkoa", "summary": "This paper proposes a new method called Instruction Modelling (IM) for training language models, which applies a loss function to both the instruction and output parts of training data, rather than just the output. Through experiments on diverse benchmarks, the authors show that IM can improve model performance on both NLP tasks and open-ended generation benchmarks compared to standard Instruction Tuning (IT). The effectiveness of IM is influenced by two key factors: the ratio between instruction length and output length in the training data, and the number of training examples. IM is particularly beneficial for datasets with long instructions paired with brief outputs, or when using a small amount of training data. The authors hypothesize that IM's improvements stem from reducing overfitting during instruction tuning.", "questions": "- Are there any possible experiments that you can do to further explain why applying loss to instruction works? \n- Do you think when we have a significantly larger size of instruction tuning data, the conclusion still holds? For example, we see the recent release of Llama-3 models, which adopted 10+ million instructions. And which factor is more important: the ratio between output and the instruction length or the number of instruction tuning samples?", "limitations": "Yes. But the authors do not have a mandatory paper checklist.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper revisits the fundamental approach to instruction tuning, which typically involves calculating loss only on the output portion of the data. By proposing Instruction Modelling (IM), which applies loss to both the instruction and output parts, the authors challenge this standard practice. This fresh perspective on a widely-used technique is a key strength of the paper. \n- The authors conduct extensive experiments across many diverse benchmarks, demonstrating the broad applicability and effectiveness of their proposed method.\n- The paper identifies and analyzes two crucial factors influencing IM's effectiveness: the ratio between instruction length and output length in training data, and the number of training examples. This analysis provides valuable insights for practitioners on when and how to best apply the IM approach, particularly in low-resource scenarios.", "weaknesses": "- Similar ideas and conclusions have been proposed by previous work [1].\n- While the paper presents empirical results showing the effectiveness of Instruction Modelling (IM), it lacks a strong foundation explaining why applying loss to instructions works. The authors hypothesize that IM reduces overfitting, but a more rigorous theoretical analysis could provide deeper insights into the mechanism behind IM's success.\n- The experiments primarily use LLaMA-2 and OPT models. While these are significant models, the paper doesn't explore how IM performs across a wider range of model architectures or sizes, e.g., whether the conclusion also holds for a 34B or a 70B model.\n\n[1] Instruction Fine-Tuning: Does Prompt Loss Matter? https://arxiv.org/pdf/2401.13586v2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are grateful for the reviewer's positive feedback: the fresh perspective, the extensive experiments, and the valuable analysis of crucial factors influencing IM's effectiveness. We would like to address the reviewer's valuable feedback as follows:\n\n> Limitations: But the authors do not have a mandatory paper checklist.\n\nWe have the checklist from the Page 23 to 29 in our main PDF file. We recognize that it may not have been easily noticeable, and we will ensure that in the revised version.\n\n> Similar ideas and conclusions have been proposed by previous work \\[1\\].\n\nWe appreciate the reviewer bringing attention to \\[1\\]. As noted in our related work section, we have already discussed this paper. While \\[1\\] introduces a hyperparameter to control the degree of instruction loss during training and investigates its impact, our work proposes a broader guideline. Instead of introducing new hyperparameters, we focus on when and how to include loss over instruction effectively and explore the underlying mechanisms. Our approach provides a more practical and widely applicable framework.\n\n> While the paper presents empirical results showing the effectiveness of Instruction Modelling (IM), it lacks a strong foundation explaining why applying loss to instructions works. The authors hypothesize that IM reduces overfitting, but a more rigorous theoretical analysis could provide deeper insights into the mechanism behind IM's success.\n\nWe appreciate the reviewer's feedback. While we acknowledge the importance of rigorous theoretical analysis, it falls beyond the scope of our current work. Our paper aims to provide general, empirical insights for instruction tuning. Our key message is that the efficacy of masking user prompts during instruction tuning is an empirical question. We find that including prompt loss during training can be particularly advantageous when the number of instruction tuning data is limited and completions are short.\n\nWe recognise the challenges in rigorously proving improvements, especially given the varied knowledge carried by different pre-training models and instruction tuning datasets. A comprehensive theoretical analysis would need to account for these complex, interacting factors. Our empirical findings provide insights for future theoretical investigations into instruction tuning.\n\n> The experiments primarily use LLaMA-2 and OPT models. While these are significant models, the paper doesn't explore how IM performs across a wider range of model architectures or sizes.\n\nWe conducted additional experiments on additional models including Phi and Gemma, which show qualitatively similar results (see Table 2 in the rebuttal pdf).  However, we were not able to finetune 60-70b parameter due to computational restrictions.  We will add a note on this point to the limitations section.\n\n> Are there any possible experiments that you can do to further explain why applying loss to instruction works?\n\nWe appreciate the reviewer's insightful question. While our current study provides empirical evidence for reducing overfitting through loss and BLEU analysis, we agree that further analyses could offer deeper insights into the underlying mechanisms. Several factors may affect the model's effectiveness, including assessing whether IM improves the factual correctness of responses or reduces toxic content in model outputs, as well as testing if IM enhances robustness to out-of-distribution (OOD) instructions.\n\nTo address this aspect, we conducted additional experiments comparing high-quality datasets \\[2, 3\\] with randomly selected datasets of the same size from the same source datasets. As shown in Table 1 of our attached PDF file, our results indicate that our approach generally performs better even when trained on randomly selected datasets. This finding suggests that our approach is robust across various data qualities and may be particularly beneficial when working with diverse, non-curated datasets. We will include these results in our revised paper.\n\n> Do you think when we have a significantly larger size of instruction tuning data, the conclusion still holds? For example, we see the recent release of Llama-3 models, which adopted 10+ million instructions. And which factor is more important: the ratio between output and the instruction length or the number of instruction tuning samples?\n\nWe thank the reviewer for raising these important questions. We would like to clarify that our approach is not intended to replace instruction tuning in all scenarios; rather, we propose that the decision to mask user prompts during this process should be empirically driven. In scenarios with limited instruction tuning data and short completions, incorporating prompt loss during training can be particularly beneficial. \n\nRegarding the relative importance of factors, we view the ratio between output and instruction length and the number of instruction tuning samples as fundamentally equivalent, both representing constraints on instruction tuning resources. Our research demonstrates that exposing the model to additional loss signals through instruction modelling—specifically by including loss on the prompt during training—can lead to more robust and effective models. This approach maximizes the utility of limited resources, potentially enhancing model performance across various tasks.\n\n### Reference:  \n\\[1\\] Instruction Fine-Tuning: Does Prompt Loss Matter? Arxiv 2024\\.\n\n\\[2\\] LESS: Selecting Influential Data for Targeted Instruction Tuning, ICML 2024\n\n\\[3\\] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR 2024\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their thoughtful feedback and valuable suggestions. We are pleased that our work has been recognised for its novelty (`b1op`,`vkoa`), simplicity and scalability (`z7Cy`), extensive experiments (`b1op`, `kinw`, `vkoa`), and potential impact (b1op). Reviewers also appreciated our intuitive explanations for the results (`z7Cy`,`vkoa`), the complementarity of our method with recent approaches like Neftune (`z7Cy`), our well-written reasoning and thorough appendix (`b1op`), and the consistency of our results across diverse settings (`kinw`). In response to the reviewers' comments, we have conducted additional experiments and analyses, which we believe significantly strengthen our paper. Key updates include:

1. **Additional experiments**: We have tested our approach on additional model families, including Gemma (2B) and Phi-1.5 (1.3B) (`z7Cy`, `b1op`, `vkoa`). We also conducted experiments comparing high-quality curated datasets with randomly selected subsets, showing that our approach is robust across various data qualities (`b1op`, `vkoa`).

2. **Clarification on instruction definition**: We have provided detailed examples of what constitutes an "instruction" in our work, including static templates, dynamic user instructions, and formatting tokens (`kinw`).

3. **Loss analysis and overfitting**: We elaborated on our loss analysis experiments and BLEU score analysis to address concerns about potential overfitting analysis (`kinw`).

4. **Comparison with continued pre-training**: We clarified the differences between our approach and continued pre-training, highlighting the unique aspects of IM (`kinw`).

5. **Limitations**: We have clarified the location of our paper checklist (`vkoa`) and added notes on computational restrictions preventing experiments with very large models (60-70B parameters) (`b1op`, `vkoa`).

We believe these additions and clarifications address the main concerns raised by the reviewers and improve the overall quality and impact of our paper. We will incorporate these changes in our revised version.

Meta-review: This paper proposes instruction modeling for training language models, which applies a function to the instruction and prompt part rather than solely to the output part. Experiments show that instruction modeling can improve model performance on both NLP tasks and open-ended generation benchmarks, with some key factors found that influence the effectiveness of the approach: the ratio between instruction and output lengths, and the quantity of training data. 

Some reviewers raised concerns regarding to the limited model exploration and lacking the explaination of why applying loss to instructions works. The authors made clarifications in their rebuttal with some extra results, and all reviewers vote positive to this work. Overall, the proposed method is quite simple and scalable, and the results are impressive. This method would be of great interest to practitioners in the community.  Therefore, I recommend an "acceptance".
Decision: Accept (poster)