Response Tuning: Aligning Large Language Models without Instruction

Authors: Seokhyun An, Hyounghun Kim
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Instruction Tuning, LLM Safety

Abstract:
Instruction tuning—supervised fine-tuning using instruction-response pairs—is a foundational step in transitioning pre-trained Large Language Models (LLMs) into helpful and safe chat assistants. Our hypothesis is that establishing an adequate output distribution can enable such a transition given the capabilities inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT), which eliminates the instruction-conditioning step in instruction tuning and solely focuses on response distribution supervision. Our experiments demonstrate that RT models, trained only using responses, can effectively respond to a wide range of instructions and exhibit helpfulness comparable to that of their instruction-tuned counterparts. Furthermore, we observe that controlling the training response distribution can substantially improve their user preference or elicit target behaviors such as refusing assistance for unsafe queries. Our findings illuminate the role of establishing an adequate output distribution in alignment, highlighting the potential of the extensive inherent capabilities of pre-trained LLMs.

Decision: reject
Reviewer scores: 5, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730329863592

thread_type:
official_review

<thread_object>
creator: reviewer_krvs

<content>
summary:
Based on the hypothesis that language models learn to follow instructions during instruction tuning mainly by learning the output space, the paper presents experiments where models are trained only on responses ("Response Tuning" or RT) instead of conditioning them on instructions (the usual "Instruction Tuning" or IT).

RT on various Llama and Gemma models with data from Alpaca, Dolly, and LIMA performs comparable to IT on instruction following (AlpacaEval) and several core capability evaluations (MMLU, OpenbookQA, Hellaswag, ARC, GSM8K, and PIQA).

To further demonstrate the importance of the quality of the responses, the responses from the datasets above are refined using a larger LM, and training on this refined dataset improves AlpacaEval scores in both IT and RT setups.

When trained on LM responses corresponding to safety related refusals, though IT performs better than RT initially, with increasing amounts of data, performance of RT models eventually matches IT.

In an ICL (instead of finetuning) setup as well, it is shown that LMs can be aligned with human preferences using response demonstrations alone, with response-only ICL yielding comparable performance to paired-demonstration ICL in one evaluation setting (https://allenai.github.io/re-align/just_eval.html).

strengths:
The main finding in this paper is interesting. It provides an insight into how LMs learn to follow instructions. Future work may be able to leverage this insight to improve instruction tuning procedures.

weaknesses:
## Practical utility
The paper does not demonstrate the practical utility of the finding. It would be helpful in the following cases:

1. Obtaining instruction-response pairs is significantly harder than obtaining responses alone.
2. Training on responses alone is more effective than training on instruction-response pairs.

1 is not true in any of the experimental settings considered in the paper, and 2 is not true in any of the results, at least not significantly.

Suggestions for addressing this concern:

- There may be tasks where 1 is true. Code generation (from natural language instructions) is a good example. Does RT on code, which is much easier to obtain than paired NL-code examples do as well as IT? More generally, tasks with structured outputs that are easy to synthetically generate are suitable here.
- Tasks where the model performance is sensitive to the superficial features in the instructions would be helpful to demonstrate 2. Does RT make the model more robust to variations in instructions than IT in such cases?

## Role of the response refinement experiments
These experiments generally show the importance of the quality of the responses, but it is unclear how this is related to the main hypothesis about conditioning on instructions being unnecessary. Refinement generally helps in both IT and RT and RT settings. It would be more helpful to directly compare IT and RT models with and without response refinement. Does refinement increase the winrate of RT models against IT models?

questions:
- What is the performance of the base models (Llama 3.1 8B and Gemma 2 9B) on the tasks shown in Table 1?
- (More of a suggestion) It would be easier to interpret the results in Table 3 if all the models (IT and RT with and without response refinement) are evaluated against a fixed reference model for AlpacaEval.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639023078

thread_type:
official_review

<thread_object>
creator: reviewer_jxf8

<content>
summary:
The article suggests that the model's instruction-following capability can be acquired by fine-tuning using with the response data, and this hypothesis is validated through experiments. Furthermore, the article demonstrates enhanced preference alignment and safety by controlling the response distribution.

strengths:
1. The article is well-written and highly readable.
2. It presents an interesting hypothesis and proposes a simple yet effective method.
3. Experiments were conducted across various benchmarks.

weaknesses:
1. The pre-training data for some models in the experiments not publicly available, and models like Gemma-2 already possess some instruction-following capability. This suggests that the pre-training data may inherently include instruction-response pairs, potentially skewing the experimental results and affecting accuracy.
2. Experimental results show that the performance of Response Tuning still falls short of Instruction Tuning, particularly in human and GPT-4 evaluations. Although this is an interesting finding, deliberately constructing a response set without instructions for training seems neither effective nor efficient. Therefore, the significance of this method for advancing research remains questionable.

questions:
1. Can you conduct similar experiments using models with publicly available pre-training data, such as OLMo, TinyLlama, and Pythia, to validate the conclusion of your article?
2. In Table 1, why does Instruction Tuning perform better on several benchmarks but have a lower overall performance compared to Response Tuning? How is the overall performance in Table 1 calculated? For instance, the arithmetic averages for Response Tuning on these benchmarks are 54.05 and 61.08, so why are they reported as 58.42 and 65.62?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679652938

thread_type:
official_review

<thread_object>
creator: reviewer_6b1i

<content>
summary:
This paper demonstrates that only fine-tuning base models on the responses in instruction datasets (response tuning, or RT) often results in comparable performance to fine-tuning on the responses *conditioned on the instructions* (traditional instruction tuning, or IT) without losing any core model capabilities. This supports the authors' hypothesis that models have already learned instruction following ability during pretraining, and instruction tuning thus mainly facilitates the model placing higher probabilities on tokens associated with helpful responses at inference time, or what the authors call "aligning the training response distribution". The authors measure response quality using 4 metrics: Likert-style individual response acceptability ratings + pairwise response preference selection using both human annotators and GPT4-as-a-judge. They find results to be consistent across 3 models from different families and on 3 instruction-tuning datasets, and also find that a larger Gemma model closes the gap between RT and IT more than a smaller one.

The authors further investigate whether using an LLM to refine the response quality along dimensions such as clarity, structure, and tone before training improves model response quality (as judged by GPT4), and find a positive result.

Finally, they extend their experimental setting to refusal, and show that only training on responses from refusal/safety datasets (and not the unsafe queries the responses are associated with) also yields comparable performance, without causing the model to over-refuse. This is particularly interesting, because it supports the hypothesis that models learn to identify unsafe queries during pretraining, but need tuning to learn the "language" of refusal.

strengths:
- Originality: the paper takes a creative approach to an underexplored research question.
- Quality: The paper includes quite extensive experiments to answer 3 main RQs and is not lacking on content. The authors include multiple datasets and models in their analysis. I appreciate that they use both human and model evaluation that is generally well-controlled for biases (e.g., not using the same model/family to evaluate another model); given that both evaluation methods have known shortcomings, their results seem more robust due to the extensive effort given to evaluation.
- Clarity: The paper is generally well-written. Given there are a large amount of experimental results, the authors make good use of the Appendix for additional details while also making sure the main paper is readable and self-contained. I appreciate the delineation of the paper into 3 main sections corresponding to the 3 RQs of interest. Generally comprehensive lit review.
- Significance: The paper provides a useful datapoint an interesting insight to ongoing discussion about what models learn during instruction tuning. While the 2nd RQ is a bit orthogonal to the paper's narrative, the 1st and 3rd RQs are interesting and impactful. I would like to see it presented at the conference.

weaknesses:
Main:
- My main qualm with the paper is the authors use the word "comparable" liberally when describing the performance of RT and IT tuning. However, in most cases, it is consistently the case that IT outperforms RT, often by what looks like rather substantial margins (e.g., Figures 2 and 4). I agree with the authors that it's really interesting that RT can make up the majority of the gap between the untuned and IT models, but I think the language can be toned down a bit and more discussion given to what performance aspects cannot be solely learned from RT alone. Alternatively, the authors could provide a quantitative definition of what they consider comparable performance.
Similarly, some of the differences in core capabilities (Table 1) seem meaningful (such as +7 on HellaSwag for Gemma 2 and -7 on GSM8k for Llama 3.1), but these are mostly chalked up to being typical or expected noise. It would be valuable to include a line for the untuned/base model in this table as a comparison point.
- The URIAL paper should certainly be discussed in the related work section as it asks a similar RQ and has similar conclusions to this paper (though different methods).
- The authors make a key assumption about the data they instruction- and response-tune on: that it hasn't been included in the pretraining data of the models in a meaningful way. This assumption seems to be supported by the poor performance of the untuned models in, e.g., Figure 2, but it should be stated more concretely. 
- Practical impact: from an analysis/model understanding perspective, the paper is strong. The authors do not explain, however, what impacts only training on responses could have from a practical angle. Presumably, it is not cheaper to collect the response data, since human or synthetic data generation pipelines would still require the instructions or refusal query to be present. It could be faster to train the model when one does not have to run inference on the instruction tokens, though the loss computation is the same so this may be minor. I would appreciate some discussion about this, and/or quantitative #s on, e.g., finetuning speed, in the paper.

More minor:
- the authors use a number of terms in their main claims that are a bit vague and deserve proper definition, such as "output/response space", "training response distribution", "behavioral guide" and "inherent attributes" of the response space. I got the gist of what they were saying, but I don't think that should be assumed, and it would be good to be more consistent w/ terminology and more clearly define it.
- the paper could benefit if the authors added some analysis on models with open training data (such as Olmo or Pythia)-- this would allow them to perform a leakage analysis and understand what types of/how much instruction data is present in the pretraining corpus.
- Comparing Figs 3 and 6 it appears that both URIAL and URIAL-R are competitive with RT/IT, which raises the question of **why are we instruction-tuning in the first place?** I would appreciate some clarification of this in the rebuttal.

questions:
Notes and minor other points:
- How are you testing the models' core capabilities in Table 1-- prompt format, instructions in prompt, greedy decoding, few-shot examples?
- typos line 044: "omits"; line 054 missing "we"; line 082: "guides"; line 092: "focuses"; line 405: missing "the"; line 406: extra "the"
- it would be good to link to the plots for the other models in the Appendix in the caption of each Figure, for reader's easy reference and cross-model comparison.
- related work: would be good to cite https://aclanthology.org/2020.emnlp-main.105/ for cross-task generalization as this was the earliest (?) work
- The authors should mention this concurrent work in a camera-ready: https://arxiv.org/abs/2409.14254
- I didn't find Figs 3/6 to be particularly useful and I am not sure what the 1-5 metric represented is. At a minimum it would be good to make the graphs represent the same metrics so that Figs 3 and 6 can be directly compared. 
- Using the word "significant" to describe results (lines 312, 403) implies you've done statistical testing
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705961622

thread_type:
official_review

<thread_object>
creator: reviewer_afzu

<content>
summary:
The main question this paper tries to answer is "how much do we lose by tuning just on the responses (response tuning) instead of instruction-response pairs (instruction tuning)". The authors use existing IT datasets and remove instructions from them to get datasets for response tuning. They fine-tune a variety of base-llms (not instruction tuned) and compare IT vs RT. They evaluate instructability with human evaluation, GPT-4 based response quality evaluation and pairwise assessment. They also evaluate core capabilities across standard benchmarks like MMLU, OpenbookQA, HellaSwag, ARC, GSM 8K, PIQA. Across a wide range of experiments, they show that response tuning comes close to instruction tuning. Additionally, they show the application of this method for preference alignment and safety alignment. Finally, they show that a similar trend holds for in-context learning, where using just responses doesn't degrade performance significantly compared to using instruction-response pairs.

strengths:
* originality - The authors ask a simple question that hasn't been considered from a scientific point of view. "Can we use good responses instead of instruction-response pairs"
* quality - The authors run a wide variety of experiments to make their claim stronger and well substantiated. 
* clarity - The paper is well-written and easy to read. In most places all the necessary details and citations are provided
* significance - There is the scientific question on the source of gains due to instruction tuning. From a practical perspective, this raises the question whether, as a community, we should invest more energy looking at instruction-free response datasets which are free of instruction biases.

weaknesses:
There are two main concerns regarding the soundness of these experiments
## Leaking instructions to response
The datasets in section 4, 5 and 6 are collected as instruction - response pairs. The data generation process may induce a leak from the instruction to the response. For e.g. if the instruction was "Write an article on the role of dinosaurs in prehistoric ecology", and the response was "Dinosaurs played a significant role in prehistoric ecology ...", the instruction can be inferred based on the response. In a trivial situation, for an instruction "Follow the rules x, y, z", the response could be "following the rules x, y, z, here's the response ...". To argue that one needs only the response (and not the instruction), the authors would need to show that the instruction cannot be trivially inferred from the response. Alternatively, an experiment training on high quality responses gathered without explicit instructions could also prove the point. 

## Training setup (Section 4.1)
The authors use PEFT (QLoRA) but don't give any references that this is a standard setup for instruction tuning. It is possible that a less expressive training scheme (PEFT vs full fine-tuning) suppresses the utility of having instructions in the training data. Alternatively, if they could show that their PEFT-trained IT models are comparable to the best IT models on these datasets, that could alleviate this concern too.

questions:
My two main questions with potential fixes are indicated in the weaknesses section. Convincing experimental results for the two weaknesses will change my stance on soundness and the overall rating.   

Some suggestions that aren't critical to soundness
* "establishing an output space" is less standard than "establishing an output distribution". The authors do use "distribution" at various places in the paper, so it isn't clear why not use the same standard term everywhere (including the abstract)
* The authors use certain terms that aren't well-defined. 073 - "contextual refusals", 
* line 311 - "some fluctuations" - why are the fluctuations attributed to fine-tuning? if there's a good reason, it should be stated, if not, it is sufficient to state that the numbers are comparable. 
* line 333 - what does the up arrow mean?
* line 428 - what exactly is meant by "pattern matching"?
- Figure 4b - IT win-rate is 60% compared to RT win rate of 40%. This is a big difference but there is no discussion of this negative result in the text. 

The authors consistently use the narrative that "capabilities required to behave as a chat assistant are largely inherent in pre-trained LLMs and can be activated without explicit supervision from instruction-response pairs" (lines 096, 515 and perhaps more). However, there is no direct proof for the "capabilities being inherent in pre-trained LLMs". One can argue that training on responses adds these capabilities that aren't inherent, just like instruction-response pairs are supposedly adding these capabilities. This claim is unnecessary and unsubstantiated. This claim is unnecessarily distracting and I would urge the authors to stay focused on the central claim of response tuning being at-par with instruction tuning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732317793011

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers (1/2)

comment:
We sincerely appreciate the reviewers for their constructive comments and valuable suggestions, which have greatly helped us improve our manuscript. Below, we address common concerns and questions raised and summarize the revisions implemented in response.

**1. Scope and goal of our study**

We would like to clarify the scope and goal of our study. The primary aim of our work is to contribute insights into the mechanisms by which language models develop the capabilities needed to handle instructions helpfully and safely. RT serves as a tool in our study, intentionally designed as a simple ablation method of IT. As highlighted by Reviewer 6b1i and krvS, the value of our work lies in the experimental findings and insights derived from them, rather than RT being proposed as a standalone method. We believe these findings are valuable and worth sharing.

**2. Practicality of RT**

While the primary focus of our study is to deepen the understanding, we recognize the importance of dicussing the practical aspects of RT to stimulate future exploration.

**2.1. RT for domains where instruction-response pairs are hard to obtain**

Reviewer krvS raised an important question about whether RT could be effective in domains where instruction-response pairs are challenging to gather, yet specific response formats are necessary—such as in code generation. To investigate this, we fine-tuned the Llama-3.1-8B model using 1,024 pure Python code snippets as RT training data. These snippets were sourced from the StarCoder IT dataset [1], but we only used code devoid of comments or docstrings to simulate data that is easier to collect. We then evaluated the model’s ability to follow code generation instructions using the HumanEvalPack benchmark [2]. The results demonstrate that the RT model, trained solely on pure code, performs comparably to the IT model in adhering to code generation instructions.

|  | IT  | RT  |
| --- | --- | --- |
| Pass@10 | 0.45 | 0.40 |

**2.2. Reduced computational cost**

In addition, as noted by reviewer 6b1i, RT generally requires fewer tokens, which results in a reduced number of floating-point operations compared to IT, thereby leading to shorter training times. For instance, the number of floating-point operations performed during fine-tuning Llama-3.1-8B is shown below:

|  | IT | RT |
| --- | --- | --- |
| Alpaca | 7.39e+18 | 6.87e+18 (-8%) |
| Dolly | 3.73e+18 | 1.31e+18 (-65%) |
| LIMA | 6.78e+17 | 5.09e+17 (-25%) |

These findings have been incorporated into the revised manuscript (Appendix A and E).

**3. Additional analysis of pairwise evaluations**

Reviewers 6b1i, JXF8, and krvS suggested further discussion of the pairwise evaluation results. We agree that analyzing these results adds depth and value to our work. In response, we conducted a qualitative analysis of cases where outputs from IT models are preferred over those of RT models finds that IT models often produce outputs with a style more closely aligned with the given instructions. This suggests that the explicit instruction-response mapping aids the model in selecting a response format more closely tailored to the instruction from the established response distribution. These have been incorporated into the revised manuscript (line 303), and we have adjusted the tone of our discussion to provide a more balanced perspective, as recommended by Reviewer 6b1i.

**4. Results for the open-data models**

To address suggestions from reviewers 6b1i, JXF8, and krvS, we also fine-tuned the OlMo 7B model using the LIMA dataset, along with 500 safety examples used only for safety evaluation. The results for this open-data model are consistent with the trends observed in our paper.

|  | AlpacaEval (vs IT win-rate) | AdvBench Refusal Rate |
| --- | --- | --- |
|  OLMo IT | - | 0.97 |
| OLMo RT | 37 % | 0.68 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes and studies response tuning, where an LM is aligned to become an instruction following model via just being finetuned on responses. The paper finds that in many cases, tuning on responses only can result in performant instruction following models.

On the positive side the paper presents an interesting approach that sometimes works well. On the negative side, there were some concerns with regard to the practicality of the method, the underperformance of the method vs. vanilla instruction tuning, as well as the experimental setup/datasets. In my view I do not think the results are interesting/significant enough to overcome the impracticality of the method, so I am recommending that this paper not be accepted.