Instruction Following without Instruction Tuning

Authors: John Hewitt, Nelson F. Liu, Christopher D Manning, Percy Liang
Primary area: foundation or frontier models, including LLMs
Keywords: instruction tuning, instruction following, ablation, rule-based

Abstract:
Adaptations that seem deficient compared to instruction-tuning (finetuning on instruction-response pairs) can still implicitly yield instruction-following language models. We show that training solely on responses, without any corresponding
instructions, yields instruction following. This suggests that instruction-tuning just needs to teach the desired distribution of responses. However, we then show that one can finetune on instruction-response pairs from a distribution unlike the test distribution of instructions, like just poetry, or just math, and still yield a model that broadly follows instructions. Instead of acting, e.g., just as math models, these single-task models sometimes behave more as general-purpose chatbots for, e.g., non-math instructions. To begin to explain this implicit instruction tuning, we hypothesize that simple changes to a language model’s distribution yield instruction following. We support this by hand-writing a rule-based adapter that yields instruction-following behavior in language models. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words’ probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.

Decision: reject
Reviewer scores: 3, 5, 10, 6

Reviews and discussion:
<thread>
creation date:
1729089336182

thread_type:
official_review

<thread_object>
creator: reviewer_pxjk

<content>
summary:
This paper ran a comprehensive ablation study on instruction tuning, testing out several dimensions of the original instruction tuning design --- (1) tune conditionally on prompt? (2) tune over all tasks? (3) even tuning? The results show that current instruction tuning is kind of an overkill, thus suggesting finding simpler and more principled post-training techniques.

strengths:
+ The object of study lies at the central place of post-training technique. I find the selected problem important and fundamental. 
+ The article provides new insights into the post-training process, along with literature like LIMA, that post-training process could be more lightweight than we assume currently. The caveat in the conclusion is also important and worth highlighting that both adding AND removing behaviors picked up in pretraining is difficult.

weaknesses:
+ Experiments are well designed and insightful. Even though, I wonder whether the inclusion of other instruction following benchmarks could add to the solidness of the results, e.g. MT Bench, or Arena hard.
+ Worth noting that none of the ablated methods out-performs original instruction tuning. Therefore, this paper is not contributing new techniques to the field, as intended. Maybe introducing another dimension of measurement could be useful, e.g., language distribution drift to pretrained model, or alignment tax as measured by drop of performance on capability benchmarks.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1729304679646

thread_type:
official_review

<thread_object>
creator: reviewer_en6v

<content>
summary:
This paper conducts an exploration of instruction tuning of large language models (LLMs) in three progressively deeper layers. Firstly, the authors present what appears to be a radical conclusion—that fine-tuning solely on the answers while disregarding the instructions can significantly enhance the model's ability to follow instructions. Following this surprising conclusion, the authors further emphasize that the effectiveness of this "response tuning" method has no necessary correlation with the diversity of responses. In certain tasks, even employing extremely narrow, single-task responses for fine-tuning can improve the model's instruction-following performance. Finally, the authors provide their ultimate conclusion: this implicit instruction tuning method merely requires a slight adjustment in the model’s parameter distribution. The authors developed a rule-based language model and found that minimal alterations to the model’s output distribution can achieve remarkable improvements in instruction-following capabilities.

strengths:
This is undeniably an exemplary piece of work. The authors courageously put forward a new paradigm for instruction tuning and meticulously present their arguments in a step-by-step manner.

From a conceptual perspective, the conclusions of this paper have significantly transformed my understanding of instruction following. The authors demonstrate that instruction following can largely be accomplished by directly optimizing the response, and that these responses need not overly emphasize diversity.

From an experimental standpoint, I highly appreciate the extensive experiments carried out by the authors. Naturally, I will add some of my personal viewpoints in the "weaknesses" section.

Regarding the writing, I must apologize that I did not find the abstract particularly fluent to read. However, the instruction section is exceptionally well-written. The step-by-step, question-and-answer format is particularly captivating.

weaknesses:
I have several minor suggestions regarding this paper. However, these suggestions by no means diminish its outstanding merits.

Firstly, as previously mentioned, the experimental section requires further consideration. At this year's International Conference on Machine Learning (ICML), Allen Zeyuan Zhu presented his work titled "The Pythics of Language Models." In this work, he maintained extremely strict control over variables from the pretraining phase, ensuring that there was no overlap or interference between the pretraining data and the data used for fine-tuning. Given that the conclusions of this paper are similarly groundbreaking, I suggest that, if resources permit, the authors could consider controlling variables from the pretraining phase. This would alleviate concerns about potential interference from the pretraining data of the LLaMA model affecting the conclusions. Naturally, pretraining costs are extremely high, and I do not consider this a shortcoming on the part of the authors. Nevertheless, conducting experiments using a wider range of base models might render the conclusions more robust.

Additionally, in the section on single-task tuning, although the conclusions are quite remarkable, I believe that the discussion of two exceptional datasets (one of which pertains to chess) could be enhanced.

Finally, I found the abstract to be less engaging than the introduction.

questions:
There are no specific questions. Kindly refer to the weakness. Additionally, ICLR does not have a best paper nomination. However, I consider this work worthy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730274543860

thread_type:
official_review

<thread_object>
creator: reviewer_aakp

<content>
summary:
This paper proposes that the instruction-following ability can be obtained without explicitly using instruction. It presents findings that models can follow instructions when trained only on response data, without explicit pairing with instructions. Additionally, single-task finetuning—training on narrow data domains like poetry or coding—also elicits broad instruction-following behavior across different tasks, such as recipe generation. The study also introduces a rule-based approach with simple modifications, showing that small changes to a language model’s distribution can yield instruction-following capabilities. This work challenges traditional views on instruction tuning, suggesting that implicit methods can lead to general instruction-following, raising questions about the necessity of explicit instruction tuning in developing adaptable, instruction-capable language models​

strengths:
1. The phenomenon proposed by this paper, Instruction Following without Instruction Tuning, is novel and interesting. 
2. This paper is well-written, easy to follow, and explicitly illustrates the points that might lead to misunderstanding, e.g. the definition of instruction-following behavior in line 183; the use of templates in line 178; and the potential rephrasing in line 280.

weaknesses:
1. This paper tries to reveal a transformative finding in the area of Instruction Tuning, while the experiments conducted in this paper are too limited to support the finding: 
(1) Only 2 LLMs are experimented, and all of them are slightly out-dated. Although as mentioned in line 101, the authors use these 2 LLMs because it is common to include instruction-tuning data during pretraining in modern days, only presenting results on 2 LLMs is still so limited. It gives the impression that your finding is only useful for slightly earlier LLMs. 
(2) For the response-tuning setting, only the LIMA dataset is utilized, which is also really limited. How can we know whether the instruction-following ability you mentioned can only be derived from the LIMA dataset? The LIMA dataset has some unique characteristics, e.g. all of them are long and deal with complex instructions. What if standard instruction data is utilized? What if the datasets used in single-task fine-tuning settings are used in response tuning?     
(3) Only the AlpacaEval metric is used to judge if the model has the instruction-following ability, which is not enough and is potentially biased. 

2. Related to the previous point, I think this phenomenon (response tuning to get instruction-following ability) might probably only work for some types of tasks. For example, if you only finetune the response on your Chess task, I don’t think the LLM can obtain a reasonable instruction-following ability. Thus more detailed experiments and discussion should be provided. 

3. The definition of instruction-following behavior in line 183 is not convincing to me and your experiments of rule-based response adapter actually deepened my suspect: If doing Slowly upweight EOS, Uniform token changes, and Encourage word diversity leads to better win rates, how can you ensure the gain on win rates is caused by real instruction-following ability or the drawbacks of this evaluation metric, which will give higher scores as long as the output is long and diverse. I think more controlled experiments and discussions are required for this part.

questions:
1. More response examples should be provided for all the settings including response tuning, single-task tuning, and rule-based tuning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605786022

thread_type:
official_review

<thread_object>
creator: reviewer_uwbn

<content>
summary:
This paper focuses on exploration of approaches to enable LLM’s instruction following capabilities without instruction tuning. It discusses three different approaches including training solely on responses, instruction-response training on narrow-domain data, and changing a model’s distribution to yield instruction following. Though studying an interesting direction not widely adopted by main-stream approaches, the paper still has multiple drawbacks.

strengths:
The phenomenon of instruction following without LLM post-training is observed in practice. It would be interesting to find the root cause or theoretical foundation of such phenomenon.

weaknesses:
1. All three approaches explored in this paper perform significantly worse than their comparable instruction-tuned versions in terms of win rates against the instruction-tuned models. It is unclear about the motivation of this work from the practical aspect.

2. Besides missing practical motivation, the paper lacks theoretical analysis or parameter-level insight on why response-only tuning, narrow-domain instruction tuning, or distribution perturbation of certain tokens helps on instruction following as claimed in the paper. Some related work discussing how instructing tuning modifies LLMs could be referred to for this line of study.

[1] Wu et al., “From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning”, NAACL, 2024.

3. It is still unclear whether the instruction following capability has already been partially obtained through pretraining. Indeed, the experiments with OLMo, in which no instruction-tuning data was intentionally included in its pretraining, show similar conclusions as the experiments with Llama-2. However, existing pretraining corpus could include instruction tuning components from contents like human dialogues from novels, question-answer pairs from websites, etc. It requires careful investigation on the roles of pretraining, instruction tuning, as well as preference learning to reach persuasive conclusions about LLM’s instruction following capability.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731908676299

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to all reviewers!

comment:
We thank all the reviewers for their work and evaluations. We’ve responded inline, but here are some key takeaways:

 - **Our goal is insight through ablations.** We believe the value of this work is in coming to a deeper understanding of instruction following in language models, through ablations – of the need for instructions (response-tuning,) of the need for a broad distribution of instruction-response pairs (single-task finetuning) and finally of any adaptation beyond a few simple changes (rule-based adapter.) As ablations, we do not expect any of these models to perform better than instruction-tuning, but instead we provide more insight into what is important about instruction-tuning.
 - **Additional samples.** For every model we trained, we have added roughly a full page worth of instruction and predicted response pairs – 9 samples per model – to our appendix, to give a fuller qualitative view of instruction following behavior. We believe these samples show how distinctive instruction following behavior is from the base Llama and Olmo behaviors.
 - **Additional Model.** We hear reviewers’ concerns about using slightly older models (Llama-2-7B, Olmo-7B). We maintain that this is a good decision to avoid midtrained models (models instruction tuned during pretraining, as is common now.) However, we’ve extended our experiments to Gemma-2-9B, to give a new model (and a third model family.) We find that indeed, the base Gemma-2-9B model performs better than Llama-2-7B and Olmo-7B. As a result, it performs better than some, but not all, of the single-task finetuned Gemma models. Overall, the conclusions of the results are similar. Fascinatingly, after changing the token IDs of our rule-based adapter from the Llama ids to Gemma (and increasing the weights), the adapter also yields instruction-following in Gemma, showing great generalization.
 - **New Evaluations.** We hear the reviewers’ concerns about our AlpacaEval-based evaluations. We believe them to be valid, especially since none of our methods, e.g., lead to very long responses (in fact, the base models’ responses are almost always longer.) However, we have added IFEval and MTBench evaluations. Overall, models perform relatively poorly on IFEval relative to instruction tuning (showing that they are not as good at capturing details), and MTBench gives qualitatively similar results to AlpacaEval for Llama. For Olmo and Gemma, only GSM outperforms the base model on MTBench, but for Olmo in particular, looking at the samples (in our appendix,) we see the AlpacaEval evaluations as more indicative of the relative quality of response to the base models.
 - **Additional Instruction-Tuning Dataset** We hear reviewers’ concerns about using LIMA as our instruction-tuning dataset and ran additional experiments using Tulu-v2 data, finding similar conclusions to the LIMA experiments.

Overall, we believe these new experiments complement the existing ablations and evaluations we submitted in the initial draft, strengthening the paper even further.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732478662515

thread_type:
public_comment

<thread_object>
creator: ~pinzhen_chen1

<content>
title:
Question: difference between pre-training and response tuning?

comment:
Thanks for your work! After reading the paper, I am curious about the difference between response tuning (Sec 4.1) and the LLM pre-training stage. In terms of the training objective, they seem equivalent. If so, I then wonder why the former can lead to better instruction following but not the latter. Is it mostly due to the style/domain/format of the (response) data? A paradox here is that an arbitrary (pre-training) text can be a suitable response for some instructions, making LLM pre-training itself "response tuning".
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper notes that you can achieve traditional instruction tuning effects by just tuning on responses and then provides a 3 rule method to induce instruction following abilities without actually instruction tuning. The effects of how instruction tuning abilities happen and their interplay with pretraining dynamics are of high interest to the entire LLM community at large and actionable insights into this topic would be high impact.

The paper claims overall to make generalizable observations about instruction vs response tuning on base models ("Our goal is insight through ablations") but unfortunately I agree with the reviewers who think that the results from the models provided show no clear insights. Normally I wouldn't be too concerned with the choice of models but in this case they are very relevant as noted by reviewer uWBN.  Olmo is the only model they can guarantee did not have instruction tuned data / was midtrained but nothing is mentioned to be best of my knowledge in the Olmo paper about how they filtered for instruction data in the pretraining corpus. This places their fundamental claim on shaky grounds. The three rules of the rule based approach are also seemingly arbitrary and not fully justified with respect to the claims originally made. The primary eval, AlpacaEval, is also not particularly reliable as noted by reviewer aAKp - being entirely GPT based (but I do appreciate the examples shown which qualitatively show the differences between the various models). 

The paper has a lot of potential and I believe with some (significant) revisions it will have high impact but it is not suitable for publication in its current form.