Instruction Following without Instruction Tuning

## Abstract

Abstract Instruction tuning commonly means finetuning a language model on instruction-response pairs.
We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning .
We first find that instruction-response pairs are not necessary: training solely on responses , without any corresponding instructions, yields instruction following.
This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses.
However, we then find it’s not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation.
In particular, when instructions are very different from those in the narrow finetuning domain, models’ responses do not adhere to the style of the finetuning domain.
To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model’s distribution yield instruction following.
We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model.
The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words’ probabilities.
In summary, adaptations made without being designed to yield instruction following can do so implicitly .

## 1 Introduction

Instruction tuning, finetuning on a broad distribution of responses (e.g., Tiramisu is made by…) conditioned on instructions (e.g., Give me a recipe for tiramisu), yields instruction following from language models for a wide range of instructions (Ouyang et al., ).
Prior work has shown that instruction tuning is sample-efficient, requiring as few as 1000 broad-domain instruction-response pairs (Zhou et al., ) or a carefully crafted prompt and few-shot instruction-response examples (Lin et al., ).
We take this a step further, exploring the idea that instruction following can be yielded from language models even implicitly, i.e., through methods not explictly designed to do so.
We discover two forms of adaptation that perform implicit instruction tuning, being seemingly deficient compared to explicit instruction tuning: (1) response tuning, training on only responses, and (2) single-task finetuning, training on data from a narrow domain of goals, like poetry generation.

We first demonstrate that response tuning, training on responses alone without conditioning on their instructions—is sufficient to yield instruction following (Section 4).
In particular, using the LIMA dataset (Zhou et al., ) for tuning, and evaluating on AlpacaEval 2, response-tuned models win roughly 43% of the time against a comparable instruction-tuned model, where equal performance would correspond to a 50% win rate.
Response tuning provides no explicit information about the mapping from instructions to responses, only information about the distribution of desired responses.
This suggests that an instruction-response mapping may be learned during pretraining, but all desirable responses are too low-probability to be generated.
We support this, finding that pretrained models rank an instruction’s real response higher than a random other instruction’s response at the same rate as instruction-tuned models.

Figure: Figure 1: Instruction tuning trains a language model on responses conditioned on instructions. We find that (1) response tuning (estimating the probability of responses with no instructions), (2) single-task finetuning (e.g., code or poetry generation), and even (3) a simple rule-based adapter all yield language models with general instruction-following behavior.
Refer to caption: x1.png

From the response tuning results, it seems that the crucial part of instruction tuning is to teach the distribution of desirable responses.
However, we find that this is not crucial either.
Finetuning on single-task, narrow-domain data, like mapping English requests to Python snippets (MBPP; Austin et al., ), or generating poetry from poem titles, also yields broad instruction-following behavior (Section 5).
That is, despite training only to generate only Python code or only poetry, models generate biographies or recipes when instructed.
For example, a poetry-tuned Llama-2-7B wins 23.7% in AlpacaEval 2 head-to-head against an instruction-tuned Llama-2-7B, whereas the base model wins 2.4% of the time.
Qualitatively, we find that single-task finetuned models only adhere to the finetuning distribution on instructions similar to those finetuned on, and exhibit general instruction following behavior for other instructions.

These two results motivate us to ask why instruction following is yielded by such a wide range of adaptations.
While Lin et al. () showed that relatively few token decisions change between pretraining and instruction following, it is not obvious that it’s simple to determine which tokens to change.
We hypothesize that very simple changes in conditional distributions can cause a language model to follow instructions.
We validate this by hand-writing a rule-based language model with three rules that, when taken in a product with a pretrained language model, causes instruction following (Section 6).
Our three rules are: slowly increasing the probability of ending the sequence, uniformly modifying 15 tokens’ likelihoods (e.g., I, <, should), and penalizing token repetition.
Our rule-based product-of-experts with Llama-2-7B wins 24.4% against instruction-tuned Llama-2-7B, compared to the base model’s win rate of 2.4%.

In summary, our results show that adaptation methods not intended to yield instruction following may yet do so implicitly.
When adapting language models for specific purposes, we might not expect that models would behave as general-purpose instruction following models on instructions dissimilar from those tuned on, but our results suggest that this will often be the case.
Our rule-based adapter experiments begin to explain this, showing that very simple changes to a model’s distribution—not obviously tied to following instructions—also yield instruction following.(^1^11Our code and data are available at [https://github.com/john-hewitt/implicit-ins](https://github.com/john-hewitt/implicit-ins).)

### 1.1 A note on pretraining data

One critical question in the interpretation of our results is whether language models were instruction-tuned during pretraining, as is becoming common (Bi et al., ; Dominguez-Olmedo et al., ).
If this is the case, our results become less surprising.
To mitigate this, we experiment with two open-weights language models: Llama-2-7B and OLMo-7B-Feb2024.
The Llama-2-7B model is stronger than the OLMo-7B-Feb2024 model, but we have no guarantee against intentional instruction tuning.
OLMo-7B-Feb2024 is weaker, but no instruction-tuning data was intentionally included in its pretraining.(^2^22Public communication. [https://github.com/allenai/dolma/issues/177](https://github.com/allenai/dolma/issues/177).)
These two models’ results lead to similar conclusions in our experiments.

## 2 Related Work

#### Designing instruction tuning datasets.

A few years before instruction tuning became common, the idea of a single model performing a range of tasks was explored by DecaNLP (McCann et al., ).
In the LLM era, stronger pretrained models led to more adoption.
In this era, early datasets also transformed a wide range of existing NLP tasks into a single common format (Mishra et al., ; Wang et al., ; Wei et al., ; Sanh et al., ).
The introduction of ChatGPT in October of 2022 shifted focus onto the construction of instruction tuning datasets that reflect goals users might be interested in, as opposed to NLP tasks.
Some are relatively small, with the intuition to teach nothing fundamentally new during instruction tuning (Taori et al., ; Zhou et al., ).
Others, which tend to be more performant, can be quite large and contain reasoning chains and other behaviors thought to be rare in pretraining (Ivison et al., ; Wang et al., ; ; Yu et al., ); this has included considerable non-academic open efforts (Teknium, ).

#### Ablation studies on instruction tuning.

Our work contributes to a set of discoveries concerning how little it takes to get language models to follow instructions.
Taori et al. () showed that finetuning a Llama model on 52,000 instruction-response pairs caused it to follow instructions to an extent that was surprising at the time.
Zhou et al. () showed that—with careful selection—one could use as little as 1,000 instruction-response pairs and achieve strong results.
This strongly suggested that one does not need to cover all types of instructions during instruction tuning.
Lin et al. () then showed that, with careful prompting and in-context few-shot examples, a handful of examples could also lead to instruction following.
The most similar work to ours is Kung & Peng (). They take the NaturalInstructions (Mishra et al., ) style of instruction, which includes a task description, an in-context example, and the data for the current example.
They then remove either the task description or the in-context example (leaving the other), finding that performance degrades less than expected in either case.
In the direction of more supervision, Shi et al. () find that training to maximize the joint likelihood of instruction and response improves over optimizing for the conditional likelihood of the response.

#### Out-of-distribution generalization.

Discussing out-of-distribution generalization is difficult to do precisely in the world of language pretraining, especially when we don’t know what data went into the pretraining. Furthermore, it’s often difficult to reason about the coverage of datasets with 10+ trillion tokens (Section 1.1).
In computer vision and vision-language models, various studies have shown that pretraining is the singularly important component in robustness to distribution shift (Carmon et al., ; Miller et al., ; Awadalla et al., ).
In language as well, pretraining on diverse data is known to lead to better out-of-distribution performance (relative to the finetuning dataset) (Hendrycks et al., ).
When we finetune, for example, on only poetry and test on recipes, this is out-of-distribution to the finetuning, but not to the pretraining.

## 3 Experiment Setting

A neural language model $p_{\theta}(\mathbf{x})$ is a distribution over strings $\mathbf{x}\in\mathcal{V}^{*}$, where $\mathcal{V}$ is a finite vocabulary, and $\theta$ are learnable parameters in the neural network.
Language models are trained to minimize the cross-entropy loss with a large corpus of text.
These pretrained, or base, models serve as starting points, but they are typically adapted to directly respond to user queries.

#### Instruction tuning.

Instruction tuning finetunes the parameters $\theta$ of a language model to adapt its behavior to respond to queries with relevant, helpful answers. Given a set of samples $D_{\text{ins}}=\{\text{instruction}_{i},\text{response}_{i}\}_{i=1}^{k}$, of instructions and corresponding responses (each of which in $\mathcal{V}^{*}$), instruction tuning optimizes for

$$ $\displaystyle\min_{\theta}\ \ \frac{1}{k}\sum_{i=1}^{k}-\log p_{\theta}(\text{response}_{i}\mid\text{instruction}_{i}).$ (1) $$

#### Instruction formatting.

In language model practice, the distinction between the instruction and the response—and thus what to compute the loss over or generate—is specified through formatting tokens in the input.
We use the Tulu formatting of Wang et al. (); Ivison et al. (); we present it here because instruction formatting may matter for how easy it is to yield instruction following behavior from language models:

Here, BOS and EOS are the model-specified special beginning- and end-of-sequence tokens.
Though the semantics of the tags are suggestive of instruction-following, we keep this formatting to study the setting used in practice.
In Appendix B, we test whether the semantics of the tags affect the results in this paper, finding that non-semantic tags like 
 and 
 do not change our conclusions.

#### Defining instruction-following behavior.

In this work we draw a distinction between instruction-following behavior and non-instruction-following behavior.
In reality there is a spectrum of better and worse responses, and no single boundary.
To provide some level of systematicity, we use the following evaluation setting:

We want to avoid trying to standardize how good a response must be in order to be considered instruction-following.
Further, we would like to compare various methods’ effects on language model behavior compared to doing standard instruction tuning.
We measure the length-controlled head-to-head win rate of each of our models against a comparable instruction-tuned model according to the AlpacaEval LLM-as-a-judge framework.
We find in practice that these win rates concentrate around low single digits for the sort of outputs of base models, and are higher, e.g., $>10\%$, for models with reasonable responses.

Greedily decoding from a language model lets us only observe the locally most likely decisions.
We greedily decode from models to observe when instruction following responses are (locally) the most likely continuations according to the model.

## 4 Response Tuning Yields Instruction Following

In this section, we explore *response tuning*, finetuning models on just responses–without any corresponding instructions.
One intuition for the effect of instruction-tuning compared to pretraining is that it teaches a language model that (1) the {instruction} part of the input is an instruction to be followed, not a pattern to be continued.
For example, for a prefix Give me a recipe for tiramisu, the model should not continue with Give me a recipe for cake. And (2), it should teach that strings like the {response} part of the input are considered desirable responses for that instruction.
Instruction tuning is just conditional probability estimation, $p(y\mid x)$.
Response tuning, on the other hand, can only directly teach the marginal distribution of desirable responses, not their relationships to instructions: we optimize only for $p(y)$.
So, we examine whether response tuning implicitly tunes for a good estimate of $p(y\mid x)$ when we only intentionally trained for $p(y)$.

### 4.1 Response Tuning

#### Method.

Given a set of samples $D_{\text{ins}}=\{\text{instruction}_{i},\text{response}_{i}\}_{i=1}^{k}$, of instructions and corresponding responses (each of which in $\mathcal{V}^{*}$), response tuning replaces the instruction string with the empty string, uses the same formatting we show in Section 3, and optimizes

$$ $\displaystyle\min_{\theta}\ \ \frac{1}{k}\sum_{i=1}^{k}-\log p_{\theta}(\text{response}_{i}\mid\text{[empty string]}).$ (2) $$

#### Experiments.

We now compare instruction tuning to response tuning.
For our adaptation dataset, we use LIMA (Zhou et al., ), which has 1,030 training examples.
For our base pretrained models, we use the Llama-2-7B and OLMo-7B-Feb2024 language models.
We finetune all parameters of the models.
For hyperparameter selection, we use the AlpacaEval win rate against GPT-3.5-turbo on a held-out validation set that we developed for this paper (and will release).
Our validation set, written partly by hand and partly by GPT-4, has various knowledge, translation, and administrative instructions, like Plan me a two day vacation to the virtual world of Code Lyoko; more details are in Appendix D.
For each method, we train for 5, 7, 10, 15, or 20 epochs.
After hyperparameter selection, we report averages and standard deviations over 5 training runs.
We report the length-controlled win rate of base and response-tuned models against instruction-tuned models on the AlpacaEval test set.
More hyperparameter details are provided in Appendix A.

Figure: Figure 2: Responses from response tuning, instruction tuning, and the base Llama-2-7B model.
Refer to caption: x2.png

**Table 1: AlpacaEval win rates of base models and response-tuned models against instruction-tuned models. Standard deviation is reported after the $\pm$. Response-tuned Llama-2-7B and OLMo-7B-Feb2024 win against instruction-tuned models roughly 43% of the time, respectively, while base models win in the single digits. A win rate of 50% would denote equal-quality models.**
| Model | Tuning | Win Rate vs. Instruction Tuning |
| --- | --- | --- |
| Llama-2-7B | None (Base) | 2.4% $\pm$ 0.14% |
| Response Tuning | 43.3% $\pm$ 1.1% |  |
| OLMo-7B-Feb2024 | None (Base) | 4.7% $\pm$ 0.57% |
| Response Tuning | 43.7% $\pm$ 1.7% |  |

#### Results.

We find that response-tuned Llama-2-7B models achieve on average a 43.3% win rate against instruction-tuned Llama-2-7B models, compared to a $2.4\%$ win rate for the base model against instruction-tuned models.
For OLMo-7B-Feb2024, response-tuned models win 43.7% of the time against instruction-tuned models, compared to 4.7% for the base model.
We provide an example from a response-tuned, instruction-tuned, and base Llama-2-7B model in Figure 2.

The behavior of response-tuned models is much closer to instruction-tuned models than that of base models for both the Llama-2-7B and OLMo-7B-Feb2024 base models.
Instruction tuning consistently outperforms response tuning, but not massively.
So, there is something to be gained from specifying instructions during adaptation, but it is not crucial in yielding a baseline level of instruction-following behavior.

It is possible that the success of response tuning is from responses often starting by rephrasing the instruction, thus making the mapping simpler. In Appendix C, we estimate that a bit less than $10\%$ of LIMA responses begin with rephrasing. We run instruction and response tuning on a transformed version of LIMA with rephrasing approximately removed, finding a win rate of 43.3% for response tuning on OLMo-7B-Feb2024, almost identical to the result here. We conclude that instruction rephrasing is not key to response tuning’s success.

### 4.2 The Response Ranking Capability

The success of response tuning suggests that we don’t need to teach base models an explicit mapping from instructions to responses.
Under what conditions is this true, yet base models do not follow instructions?
One possibility is that base models can rank a desired response for an instruction higher than a desired response for another instructions, but scores a string that is not a desired response at all higher than both.

To explore this, we propose the response ranking capability:
assigning a higher likelihood to the right response for an instruction than to a desirable response for a random other instruction.
For independent instruction-response pairs $(\text{instruction},\text{response})\sim D$ and $(\text{instruction}^{\prime},\text{response}^{\prime})\sim D$, and a model $p_{\theta}$, the response ranking capability holds if

$$ $\displaystyle p_{\theta}(\text{response}\mid\text{instruction})>p_{\theta}(\text{response}^{\prime}\mid\text{instruction}).$ (3) $$

Since both probabilities might be small, the response-ranking capability can hold even for models that don’t follow instructions.
With response tuning—increasing the probability of desirable responses—models with the response-ranking capability for many instructions may generate desirable responses.
For the Alpaca training set, we compute the likelihood with which the response-ranking capability holds for pairs of instructions for a pretrained, LIMA instruction-tuned, and response-tuned model.
Our results show that the capability holds for pretrained models to a similar extent as for instruction-tuned models (Table 2).

**Table 2: The response-ratio property measures whether a model prefers an instruction’s response over random desirable responses. This property holds in pretrained language models at least as well as in instruction-tuned models.**
|  | $\mathbb{E}[$Response Ranking Capability$]$ |  |
| --- | --- | --- |
|  | Base Models | Instruction-Tuned Models |
| Llama-2-7B | 80.4% | 77.4% |
| OLMo-7B-Feb2024 | 74.5% | 74.3% |

## 5 Single-Task Finetuning Yields Instruction Following

The success of response tuning suggests that models need not learn the instruction-response mapping, so maybe they just need to learn the distribution of desirable responses.
To test this, we can train on a distribution of responses that is bad for most responses—like poems, or the English-and-math derivations of the Grade School Math dataset (Cobbe et al., )—and see if models yet generate desirable responses for other kinds of questions.
We call training on a narrow domain of problem—like generating poems or recipes from titles, or completing math word problems—single-task finetuning.
Intuitively, single-task finetuning might lead models to either (1) exhibit the task behavior for any input, like generating math, or (2) “break” and output unpredictable low-quality outputs.
Single-task finetuning is usually not explicitly intended to yield broad instruction following behavior from base models, so if it does, it does so implicitly.

### 5.1 Single-Task Finetuning

#### Method.

This method is identical to instruction tuning (Section 3), except the distribution of inputs and outputs is changed. We again use the same formatting shown in Section 3.

#### Data.

Figure: Figure 3: Examples from each of the five single-task finetuning datasets. At the left of each dataset is the input that is conditioned on, and at the right is the output that is learned.
Refer to caption: x3.png

We use five single-task text-to-text datasets to test a range of behaviors and text properties.
We use the MBPP English-to-Python dataset, which contains 374 short requests for Python code and the corresponding code (Austin et al., ).
We use 1000 examples from the Grade School Math 8K (Cobbe et al., ) training dataset, which consist of mathematics word problems, and a corresponding English-and-math derivation that arrives at the answer to the problem (GSM).
We format 1000 recipe strings from the Kaggle Food Recipes structured recipe dataset (Recipes), where the input is Recipe for X where X is the name of the dish, and each output is a recipe starting with a bulleted list of ingredients, followed by the instructions.(^3^33Drawn from [https://huggingface.co/datasets/Hieu-Pham/kaggle_food_recipes](https://huggingface.co/datasets/Hieu-Pham/kaggle_food_recipes), which drew from [https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images?resource=download](https://www.kaggle.com/datasets/pes12017000148/food-ingredients-and-recipe-dataset-with-images?resource=download).)
We format a dataset of 571 poems (Poetry), wherein the input is the string Write a poem called X, where X is replaced with the poem name, and the output is the poem.(^4^44Drawn from [https://huggingface.co/datasets/merve/poetry](https://huggingface.co/datasets/merve/poetry).)
We format a dataset of 1000 chess games in PGN notation (Chess), wherein the input is the string containing the ELO (player quality) ratings for two players, and the output is the moves played in a game between those players.(^5^55Drawn from [https://huggingface.co/datasets/patrickfrank1/chess-pgn-games](https://huggingface.co/datasets/patrickfrank1/chess-pgn-games).)
See Figure 3 for an input-output example from each dataset.

**Table 3: AlpacaEval win rates of single-task finetuned models against instruction-tuned models on five datasets. A diverse set of single-task finetuning, from code generation to poetry, elicits substantial instruction following in language models. Standard deviation reported after $\pm$.**
| Tuning | Win Rate vs. Instruction Tuning |  |
| --- | --- | --- |
|  | Llama-2-7B | OLMo-7B-Feb2024 |
| None (Base) | 2.4% $\pm$ 0.14% | 4.7% $\pm$ 0.57% |
| MBPP | 16.9% $\pm$ 0.70% | 10.4% $\pm$ 1.0% |
| GSM | 23.7% $\pm$ 0.74% | 30.3% $\pm$ 0.6% |
| Poetry | 22.9% $\pm$ 0.97% | 21.9% $\pm$ 0.48% |
| Recipes | 14.6% $\pm$ 0.81% | 21.5% $\pm$ 0.86% |
| Chess | 2.1% $\pm$ 0.36% | 6.3% $\pm$ 1.1% |

The five datasets are qualitatively different. For example, the poems in the poetry dataset are famous (like the Shakespeare example in Figure 3), and so may have been memorized by the language model already.
The recipes all start with a hyphenated ingredient list.
Chess games start largely in one of a very small number of chess moves (e.g., “1. e4”).

#### Experiments.

We compare single-task finetuning against LIMA instruction tuning.
For each dataset, we use the same hyperparameter selection process as in Section 4, sweeping over five choices for the number of training epochs.
As before, AlpacaEval win rates and standard deviations are computed over 5 independent training runs of both instruction-tuned and single-task models.

Figure: Figure 4: Responses generated by single-task finetuned models for each of our five datasets. MBPP trains only on python snippet generation, GSM on math word problems, Poetry on poetry generation, Recipe on recipe generation, and Chess on chess game generation. Yet, except for Chess, the responses deviate from the single-task behavior towards reasonable responses.
Refer to caption: x4.png

#### Results.

We find that finetuning both Llama-2-7B and OLMo-7B-Feb2024 on each single-task finetuning dataset except Chess lead to general instruction following behavior, with substantially higher win rates against the instruction-tuned model (Table 3) than the base model achieves.

For both OLMo-7B-Feb2024 and Llama-2-7B, finetuning on the GSM dataset leads to the highest AlpacaEval win rates.
We provide examples of model outputs in Figure 4.
We note, for example, that the Recipe-tuned example in Figure 3 starts with a list-formatting hyphen, like all of our recipes do, yet then proceeds to provide a somewhat coherent answer.
For Chess, we find that almost all outputs are just chess games.
We speculate that this is due in part to the very low entropy of the beginning sequences of chess games.

### 5.2 Model Response adherence to finetuning constraints depends on instruction similarity to tuning instructions

Figure: Figure 5: For a GSM-finetuned model, the similarity between a test-time instruction to the instructions in the GSM dataset (x-axis) plotted against the similarity between the model’s generated response to GSM responses (minus the similarity of that response to LIMA broad responses). On the left, an example of an average-similarity instruction; note that the response is unlike GSM formatting, except for the telltale ####1, which is how GSM formats its final answer. On the right, a very high-similarity instruction leads to GSM-like behavior.
Refer to caption: x5.png

Each of our single-task finetuning datasets implicitly specify constraints—well-formed Python code for MBPP, poetry formatting—that are not met by the model’s responses to other instructions, like for recipes.
This is not necessarily bad—we don’t necessarily want recipes in poetry formatting—but it’s a striking instance of how finetuning for a desired behavior can cause neither (1) the finetuned behavior, nor (2) the base model’s behavior, but instead (3) a general instruction following behavior similar to neither the base behavior nor the finetuning task.

We study how similarity of an instruction to the GSM instructions relates to the similarity of the model response to GSM responses versus general responses, expecting that at least the math-like instructions lead to GSM-like responses from the model.
Using Nomic embedding (Nussbaum et al., ) cosine similarities, we plot the similarity-to-GSM of each LIMA instruction against the similarity-to-GSM-over-LIMA of a GSM-tuned model’s response to that instruction.
The result is in Figure 5.
We note qualitatively that for instructions that are very GSM-like, the model outputs adhere to the GSM style and particular mathematical notation it uses.
For most instructions, however, we note that the outputs are only subtly affected by GSM: they have the ubiquitous GSM sequence-ending style of ending with four hashes and an integer answer, e.g., ####1.

## 6 A 3-Rule Adapter for Instruction Following.

In this section, we provide steps towards understanding the surprising commonness of instruction following behavior.
One appealing intuition is that the difference between a pretrained model’s distribution and a corresponding instruction-following distribution is simple.
Simple conditions should be relatively easy to meet, and might be met by a variety of disparate adaptation methods.

We provide direct evidence for the simplicity of a mapping from pretrained to instruction-following distribution by hand-writing a rule-based mapping with three rules.
Prior work has shown that changes in distribution are somewhat sparse in token space: in a desirable response, Lin et al. () found that 77.7% of token decisions would also have been made by a base language model.
Still, 22.3% of tokens is not negligible, and even if it were, sparsity in the token space does not imply that it is simple to determine which tokens should change.
For intuition, a weak chess engine may agree with a strong engine on, say, 95% of moves, but determining which 5% should be changed and how to change them could be very complicated.

### 6.1 The Rule-Based Response-Adapter

#### A product of distributions.

To adapt a pretrained language model to follow instructions via a rule-based adapter, we choose our resulting model to have the form of a local product of distributions.
For a word $w\in\mathcal{V}$ and prefix ${\bm{x}}\in\mathcal{V}^{*}$, a base model $p_{\text{base}}$, and our rule-based adapter language model $p_{\text{rules}}$, the final distribution $p_{a}$ is:

$$ $\displaystyle p_{a}(w\mid{\bm{x}})=p_{\text{base}}(w\mid{\bm{x}})p_{\text{rules}}(w\mid{\bm{x}})/Z({\bm{x}}),$ (4) $$

where the normalization term is $Z({\bm{x}})=\sum_{w\in\mathcal{V}}p_{\text{base}}(w\mid{\bm{x}})p_{\text{rules}}(w\mid{\bm{x}})$.

Intuitively, a product of distributions is useful because it computes a soft AND function of the tokens that are likely under each distribution (as opposed to, e.g., a soft OR if one were to average the distributions).
Put another way, it allows our rules to change the probabilities of the base model by multiplicative factors.

#### The rules.

Each of our rules determines a score for each vocabulary item $w\in\mathcal{V}$; let $r(w,{\bm{x}})$ be the sum of all rules’ scores for $w$.
For example, for the vocabulary item #, it might score $+4$.
Our rules distribution $p_{\text{rules}}(\cdot\mid{\bm{x}})$ is defined by computing the softmax over the vector of scores for all vocabulary items:

$$ $\displaystyle p_{\text{rules}}(\cdot\mid{\bm{x}})=\text{softmax}\left(\left[r(w^{(1)},{\bm{x}});\cdots;r(w^{(|\mathcal{V}|)},{\bm{x}})\right]\right)$ (5) $$

Our rules are as follows (with corresponding Python code in Listing 1, and weights in Table 8):

Our first rule is to increase the score of the EOS token linearly with the response length, to favor shorter responses.

Our second rule is to uniformly change the probabilities of 15 words in the vocabulary at every token decision.
For example, we massively reduce the probability of repeating tokens from the formatting, like the left angle bracket, or words “I” or “We” or “Should”, which we found base models use to erroneously refuse to respond. The full list is found at Table 8.

We compute the set of all tokens generated so far in the response, and add a penalty to generating any of them again.

#### Experiments.

We compare our rule-based ensemble with instruction tuning.
We heuristically tuned the rule set and corresponding rule weights on our separate validation set.
As before, we compute the AlpacaEval win rate on the AlpacaEval test set against our LIMA instruction-tuned models.
Since there’s no training for our rule-based model, we compute the average win rate and standard deviations of the one rule-based model over the 5 instruction-tuned model seeds.

**Table 4: Instruction-tuning win rates of a rule-based product Llama-2-7B model against the LIMA instruction-tuned Llama-2-7B model, and ablations of each of our three rules. Standard deviation reported after $\pm$.**
| Model | Rule-Based Model | Win Rate vs. Instruction Tuning |
| --- | --- | --- |
| Llama-2-7B | None (Base) | 2.4% $\pm$ 0.14% |
| All Rules | 24.4% $\pm$ 0.40% |  |
| - EOS Rule (Rule 1) | 10.4% $\pm$ 0.30% |  |
| - Diversity Rule (Rule 3) | 14.3% $\pm$ 0.58% |  |
| - uniform token changes (Rule 2) | 16.3% $\pm$ 0.25% |  |

Figure: Figure 6: Responses from our rule-based language model product.
Refer to caption: x6.png

#### Results.

We find that our 3-rule model achieves win-rates against the LIMA-tuned Llama-2-7B model of 24.4%, roughly on par with the win rate of the best single-task finetuning setting.
Our ablations show concretely that each of the three rules is necessary for achieving the win rate, at least in the context of the other two.
We provide examples of the rule-based product model’s output in Figure 6.
The answers are coherent, but only the response to the kickball question really provides a good answer; we did not cherry-pick, and many other responses are more reasonable.

## 7 Conclusion

Instruction following is not only possible without explicit instruction tuning (or careful prompting), it is in some sense easy to stumble upon because some adaptations implicitly instruction-tune.
Models already encode instruction-response mappings, since training on responses alone leads to instruction following.
Training on, e.g., code, or math, or poetry, does not make language models generate those things when asked for, say, a recipe.
This is great in some sense; language models continue to show astounding out-of-distribution performance.
Except, in another sense, this shows it’s perhaps surprisingly hard to get language models to change their behavior in general, since they are so prone to just following instructions outside the distribution of finetuning.

As a practical consequence, if a practitioner deploys an language model adapted to some specific task, they should not assume that the model will exhibit that tasks’ behavior on inputs dissimilar to those trained on.
Instead, they should put it through testing and safety trials as if they were releasing a general-purpose chatbot, since the finetuning may implicitly instruction-tune the model.

## Appendix A Finetuning and hyperparameter details

In this section, we describe hyperparameter choices and implementation details.
In all experiments, we use core libraries, e.g., NumPy (Harris et al., ), PyTorch (Paszke et al., ), and Huggingface Transformers (Wolf et al., ).
We’re also indebted to the OpenInstruct repository (Wang et al., ; Ivison et al., ), which we extended for this work.

For all Llama-2-7B models, after manual search, we choose $10^{-5}$ as a learning rate, and for all OLMo-7B-Feb2024 models, we choose $3*10^{-6}$.
We use the Adam optimizer Kingma & Ba ().
We use a cosine annealing rate to 0 learning rate, and 10% of the training consists of a linear warmup.
When we sweep over epochs, we always take the final epoch, so the warmup and cosine decay have always completed for each model we evaluated.
We fix a batch size of 64 across all experiments.
We run experiments across A100 and A6000 NVIDIA GPU machines.

## Appendix B Formatting Tags Ablation

One possible explanation for the success of response tuning in eliciting instruction-following behavior is that the semantics of instruction following are already encoded in the formatting tags 
 and 
 (see Section 3).
To test this, we use the tags 
 and 
 instead, and compare response-tuned Llama-2-7B models with instruction-tuned Llama-2-7B models.
We run exactly the experiment in Section 4.1, including re-running hyperparameter tuning, and averaging AlpacaEval scores across 5 independent pairs of model training runs.

With the new A/B tags, we find that response tuning wins over instruction tuning 41.8% of the time, compared to 43.3% of the time for the original user/assistant tags.
So, it’s possible that the semantics of the tags are somewhat useful, but not the cause of the improvement in win rate over the base model’s 2.4%.
The results are compared in Table 5.

**Table 5: AlpacaEval win rates of instruction-tuned vs. response-tuned and base models. Standard deviation is reported by the error bars. Response-tuned Llama-2-7B and OLMo-7B-Feb2024 win against instruction-tuned models 43% and 36% of the time, respectively, while base models win in the single digits.**
| Model | Tuning | Win Rate vs. Instruction Tuning | Std. Dev |
| --- | --- | --- | --- |
| Llama-2-7B | None (Base) | 2.4% | 0.14% |
| Response Tuning | 43.3% | 1.1% |  |
|  | Response Tuning (A/B tags) | 41.8% | 0.84% |

## Appendix C Instruction Rephrasing Removal Ablation

**Table 6: Examples of responses detected to start with rephrasing the instruction, and the transformed examples that are intended to no longer start with rephrasing the instruction.**
| Instruction | Original Response | Rephrased Response |
| --- | --- | --- |
| What are the major concerns about planting trees to create carbon offsets? | Here are some common concerns about using tree planting projects to create carbon offsets:<br>* The scheme actually has a net positive effect, both in the short and long-term.[…truncated] | * The scheme actually has a net positive effect, both in the short and long-term. […truncated] |
| How does immersion passively help with learning a language? | There are a few reasons why immersion helps you learn a language without you actively trying:<br>* You encounter the language more often. When you are immersed in a language, you don’t go to class and do homework for a few hours a week, but rather use the language the entire day.[…truncated] | * You encounter the language more often. When you are immersed in a language, you don’t go to class and do homework for a few hours a week, but rather use the language the entire day.[…truncated] |
| Reasons for being vegetarian or vegan other than ethical reasons? | There are multiple reasons why people are/become Vegetarian/Vegan apart from ethics, including but not limited to<br>* Religion/Culture - most of the worlds Vegetarians reside in India, mainly due to culture/personal beliefs but often interlinked with some of their major religions[…truncated] | * Religion/Culture - most of the world’s Vegetarians reside in India, mainly due to culture/personal beliefs but often interlinked with some of their major religions[…truncated] |

In this section, we detail an auxiliary experiment for response tuning in which we attempt to control for the effect of some responses beginning by rephrasing the instruction.
Intuitively a response starting by rephrasing the instruction provides some supervision as to how to map from something like an instruction (the rephrasing) to the response.
At test time when presented with an instruction, the model might start by rephrasing, and then be able to continue with the response.

To attempt to control for this, we first prompt GPT-4 to detect which responses in LIMA start by rephrasing the instruction.
Out of 1030 examples, 99 are labeled as starting with rephrasing.
A few examples are provided in Table 6.
For these 99 examples, we then ask GPT-4 to re-write the response without starting with the rephrasing.
A few examples of rephrasings are provided in Table 6.
While there’s no guarantee that these 99 are exactly all of the responses that begin with rephrasing, we looked at some of the decisions and rephrasings and found them reasonable.

We response-tune and instruction-tune OLMo-7B-Feb2024 using this transformed dataset and the hyperparameters from Section 4.
We then run the AlpacaEval head-to-head comparison for these models.
Intuitively, if the win rate for response tuning vs. instruction tuning are similar for this removed-rephrasing dataset to the original LIMA dataset, it is unlikely that instruction rephrasing is the cause of the success of response tuning.

Across five seeds, the average win rate of the OLMo-7B-Feb2024 model response tuned on the no-rephrasing LIMA dataset against the OLMo-7B-Feb2024 model instruction tuned on the no-rephrasing LIMA dataset is 43.3%, with a standard deviation of 5.3%.
Although the standard deviation is curiously higher, the average win rate of 43.3% is comparable to the OLMo-7B-Feb2024 response tuning win rate in Section 4 of 43.7%.
Hence, we conclude that rephrasing is likely not the primary cause of the success of response tuning, with the caveat that we have no guarantee that our transformed LIMA dataset gets rid of all instruction rephrasing.

## Appendix D Details on our Validation Set

**Table 7: Example instructions generated by GPT-4 for our validation set.**
| Example Instruction from Validation Set |
| --- |
| What is 17 multiplied by 24, divided by 8, and then subtract 3? |
| Summarize the story of ’Cinderella’ in one sentence. |
| Given the list of numbers [34, 7, 23, 32, 5, 62], find the median. |
| Explain the significance of the theory of relativity in modern physics. |
| Write a persuasive paragraph on why electric cars are better for the environment than gasoline cars. |
| Describe the process of photosynthesis in detail, including the chemical equations involved. |
| Design a marketing campaign for a new eco-friendly product, including target audience, key message, promotional strategies, and budget allocation. |
| Translate the following legal document excerpt into German: ’The party of the first part agrees to indemnify and hold harmless the party of the second part from any and all liabilities, damages, and losses arising out of or in connection with this agreement.’ |
| Write a scientific research proposal on the effects of microplastics on marine life, including hypothesis, methodology, and expected outcomes. |

To follow machine learning model development guidelines, we only run evaluations on the AlpacaEval test set after hyperparameter selection.
We perform model selection and development on a separate validation set that we constructed partly by hand (without intentional reference to the AlpacaEval test set) and partly using GPT-4 as a co-generator.

Our validation set, which has 56 instructions, tests for a variety of behaviors. Here are some examples we wrote:

> What is Samyang Buldak ramen? Give the Penn Treebank constintuency parse for the sentence “The chef who ran to the store was out of food.” Give a python function that sorts a list by absolute value of difference from 10.

In a single interactive session with GPT-4, we iteratively requested instructions for testing chatbots, starting with simple questions for “weak” chatbots and progressively asking for harder questions for “stronger” chatbots.
Our intuition was to develop a small, cheap-to-evaluate set that would yet provide signal in distinguishing model quality across a range of model sizes and performances.
It is unclear the extent to which this goal was accomplished beyond what a simple set of instructions would provide, but we did qualitatively find that the AlpacaEval results on this small validation set correlated reasonably well with intuitions when looking manually at responses.
The instructions suggested by GPT-4 include outlining requests, translations of varying lengths and languages, among other things.
We provide a few of these instructions in Table 7.

## Appendix E Rule-Based Adapter Details

In this section we provide more details on our rule-based adapter language model.
In Table 8, we provide a list of details about the scores for each word under each of our rules.
In Listing 8, we write Python code implementing the forward pass of our language model implemented by the rules to demonstrate the simplicity.
Some of the lines are to, e.g., format the output distribution properly to be combined with a HuggingFace Transformers (Wolf et al., ) language model.

**Table 8: Rules and scores for our rule-based adapter.**
| Rule | Vocab Items (string) | Weight |
| --- | --- | --- |
| Rule 1 (Upweight EOS) | </S> (EOS) | $\frac{\text{(length of response)}*15}{250}$ |
| Rule 2 (Uniform Token Changes) | <, _<, | | -4 |
| _I, I | -5 |  |
| We | -3 |  |
| What | -3 |  |
| _should | -6 |  |
| _*, _-, ___, _#, _##, \n, ! | +1 |  |
| Rule 3 (Penalize Used Words) | $\{x\in\mathcal{V}\mid x\in\text{(response so far)}\}$ | -1.5 |

Figure: Listing 1: Python Code for Rule-Based Language Model.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: None.
soundness: 2
strengths: The phenomenon of instruction following without LLM post-training is observed in practice. It would be interesting to find the root cause or theoretical foundation of such phenomenon.
summary: This paper focuses on exploration of approaches to enable LLM’s instruction following capabilities without instruction tuning. It discusses three different approaches including training solely on responses, instruction-response training on narrow-domain data, and changing a model’s distribution to yield instruction following. Though studying an interesting direction not widely adopted by main-stream approaches, the paper still has multiple drawbacks.
weaknesses: 1. All three approaches explored in this paper perform significantly worse than their comparable instruction-tuned versions in terms of win rates against the instruction-tuned models. It is unclear about the motivation of this work from the practical aspect.

2. Besides missing practical motivation, the paper lacks theoretical analysis or parameter-level insight on why response-only tuning, narrow-domain instruction tuning, or distribution perturbation of certain tokens helps on instruction following as claimed in the paper. Some related work discussing how instructing tuning modifies LLMs could be referred to for this line of study.

[1] Wu et al., “From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning”, NAACL, 2024.

3. It is still unclear whether the instruction following capability has already been partially obtained through pretraining. Indeed, the experiments with OLMo, in which no instruction-tuning data was intentionally included in its pretraining, show similar conclusions as the experiments with Llama-2. However, existing pretraining corpus could include instruction tuning components from contents like human dialogues from novels, question-answer pairs from websites, etc. It requires careful investigation on the roles of pretraining, instruction tuning, as well as preference learning to reach persuasive conclusions about LLM’s instruction following capability.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. More response examples should be provided for all the settings including response tuning, single-task tuning, and rule-based tuning.
soundness: 3
strengths: 1. The phenomenon proposed by this paper, Instruction Following without Instruction Tuning, is novel and interesting. 
2. This paper is well-written, easy to follow, and explicitly illustrates the points that might lead to misunderstanding, e.g. the definition of instruction-following behavior in line 183; the use of templates in line 178; and the potential rephrasing in line 280.
summary: This paper proposes that the instruction-following ability can be obtained without explicitly using instruction. It presents findings that models can follow instructions when trained only on response data, without explicit pairing with instructions. Additionally, single-task finetuning—training on narrow data domains like poetry or coding—also elicits broad instruction-following behavior across different tasks, such as recipe generation. The study also introduces a rule-based approach with simple modifications, showing that small changes to a language model’s distribution can yield instruction-following capabilities. This work challenges traditional views on instruction tuning, suggesting that implicit methods can lead to general instruction-following, raising questions about the necessity of explicit instruction tuning in developing adaptable, instruction-capable language models​
weaknesses: 1. This paper tries to reveal a transformative finding in the area of Instruction Tuning, while the experiments conducted in this paper are too limited to support the finding: 
(1) Only 2 LLMs are experimented, and all of them are slightly out-dated. Although as mentioned in line 101, the authors use these 2 LLMs because it is common to include instruction-tuning data during pretraining in modern days, only presenting results on 2 LLMs is still so limited. It gives the impression that your finding is only useful for slightly earlier LLMs. 
(2) For the response-tuning setting, only the LIMA dataset is utilized, which is also really limited. How can we know whether the instruction-following ability you mentioned can only be derived from the LIMA dataset? The LIMA dataset has some unique characteristics, e.g. all of them are long and deal with complex instructions. What if standard instruction data is utilized? What if the datasets used in single-task fine-tuning settings are used in response tuning?     
(3) Only the AlpacaEval metric is used to judge if the model has the instruction-following ability, which is not enough and is potentially biased. 

2. Related to the previous point, I think this phenomenon (response tuning to get instruction-following ability) might probably only work for some types of tasks. For example, if you only finetune the response on your Chess task, I don’t think the LLM can obtain a reasonable instruction-following ability. Thus more detailed experiments and discussion should be provided. 

3. The definition of instruction-following behavior in line 183 is not convincing to me and your experiments of rule-based response adapter actually deepened my suspect: If doing Slowly upweight EOS, Uniform token changes, and Encourage word diversity leads to better win rates, how can you ensure the gain on win rates is caused by real instruction-following ability or the drawbacks of this evaluation metric, which will give higher scores as long as the output is long and diverse. I think more controlled experiments and discussions are required for this part.

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 3
questions: There are no specific questions. Kindly refer to the weakness. Additionally, ICLR does not have a best paper nomination. However, I consider this work worthy.
soundness: 4
strengths: This is undeniably an exemplary piece of work. The authors courageously put forward a new paradigm for instruction tuning and meticulously present their arguments in a step-by-step manner.

From a conceptual perspective, the conclusions of this paper have significantly transformed my understanding of instruction following. The authors demonstrate that instruction following can largely be accomplished by directly optimizing the response, and that these responses need not overly emphasize diversity.

From an experimental standpoint, I highly appreciate the extensive experiments carried out by the authors. Naturally, I will add some of my personal viewpoints in the "weaknesses" section.

Regarding the writing, I must apologize that I did not find the abstract particularly fluent to read. However, the instruction section is exceptionally well-written. The step-by-step, question-and-answer format is particularly captivating.
summary: This paper conducts an exploration of instruction tuning of large language models (LLMs) in three progressively deeper layers. Firstly, the authors present what appears to be a radical conclusion—that fine-tuning solely on the answers while disregarding the instructions can significantly enhance the model's ability to follow instructions. Following this surprising conclusion, the authors further emphasize that the effectiveness of this "response tuning" method has no necessary correlation with the diversity of responses. In certain tasks, even employing extremely narrow, single-task responses for fine-tuning can improve the model's instruction-following performance. Finally, the authors provide their ultimate conclusion: this implicit instruction tuning method merely requires a slight adjustment in the model’s parameter distribution. The authors developed a rule-based language model and found that minimal alterations to the model’s output distribution can achieve remarkable improvements in instruction-following capabilities.
weaknesses: I have several minor suggestions regarding this paper. However, these suggestions by no means diminish its outstanding merits.

Firstly, as previously mentioned, the experimental section requires further consideration. At this year's International Conference on Machine Learning (ICML), Allen Zeyuan Zhu presented his work titled "The Pythics of Language Models." In this work, he maintained extremely strict control over variables from the pretraining phase, ensuring that there was no overlap or interference between the pretraining data and the data used for fine-tuning. Given that the conclusions of this paper are similarly groundbreaking, I suggest that, if resources permit, the authors could consider controlling variables from the pretraining phase. This would alleviate concerns about potential interference from the pretraining data of the LLaMA model affecting the conclusions. Naturally, pretraining costs are extremely high, and I do not consider this a shortcoming on the part of the authors. Nevertheless, conducting experiments using a wider range of base models might render the conclusions more robust.

Additionally, in the section on single-task tuning, although the conclusions are quite remarkable, I believe that the discussion of two exceptional datasets (one of which pertains to chess) could be enhanced.

Finally, I found the abstract to be less engaging than the introduction.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: NA
soundness: 4
strengths: + The object of study lies at the central place of post-training technique. I find the selected problem important and fundamental. 
+ The article provides new insights into the post-training process, along with literature like LIMA, that post-training process could be more lightweight than we assume currently. The caveat in the conclusion is also important and worth highlighting that both adding AND removing behaviors picked up in pretraining is difficult.
summary: This paper ran a comprehensive ablation study on instruction tuning, testing out several dimensions of the original instruction tuning design --- (1) tune conditionally on prompt? (2) tune over all tasks? (3) even tuning? The results show that current instruction tuning is kind of an overkill, thus suggesting finding simpler and more principled post-training techniques.
weaknesses: + Experiments are well designed and insightful. Even though, I wonder whether the inclusion of other instruction following benchmarks could add to the solidness of the results, e.g. MT Bench, or Arena hard.
+ Worth noting that none of the ablated methods out-performs original instruction tuning. Therefore, this paper is not contributing new techniques to the field, as intended. Maybe introducing another dimension of measurement could be useful, e.g., language distribution drift to pretrained model, or alignment tax as measured by drop of performance on capability benchmarks.

Meta-review: The paper notes that you can achieve traditional instruction tuning effects by just tuning on responses and then provides a 3 rule method to induce instruction following abilities without actually instruction tuning. The effects of how instruction tuning abilities happen and their interplay with pretraining dynamics are of high interest to the entire LLM community at large and actionable insights into this topic would be high impact.

The paper claims overall to make generalizable observations about instruction vs response tuning on base models ("Our goal is insight through ablations") but unfortunately I agree with the reviewers who think that the results from the models provided show no clear insights. Normally I wouldn't be too concerned with the choice of models but in this case they are very relevant as noted by reviewer uWBN.  Olmo is the only model they can guarantee did not have instruction tuned data / was midtrained but nothing is mentioned to be best of my knowledge in the Olmo paper about how they filtered for instruction data in the pretraining corpus. This places their fundamental claim on shaky grounds. The three rules of the rule based approach are also seemingly arbitrary and not fully justified with respect to the claims originally made. The primary eval, AlpacaEval, is also not particularly reliable as noted by reviewer aAKp - being entirely GPT based (but I do appreciate the examples shown which qualitatively show the differences between the various models). 

The paper has a lot of potential and I believe with some (significant) revisions it will have high impact but it is not suitable for publication in its current form.

Decision: Reject