Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation (2024)

Abstract: We introduce Bonito, an open-source model for conditional task generation that converts unannotated text into task-specific training datasets for instruction tuning. We aim to enable zeroshot task adaptation of large language models on users’ specialized, private data. We train Bonito by fine-tuning a pretrained large language model on a new large-scale dataset with 1.65M examples created by remixing existing instruction tuning datasets into metatemplates. The meta-templates for a dataset produce training examples where the input is the unannotated text and the task attribute and the output consists of the instruction and the response. We use Bonito to generate synthetic tasks for seven datasets from specialized domains with unannotated text across three task types—yes-no question answering, extractive question answering, and natural language inference—and adapt language models. We show that Bonito significantly improves the average performance of pretrained and instruction tuned models over the de facto self supervised baseline. For example, adapting MistralInstruct-v2 and instruction tuned variants of Mistral and Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1 points whereas the next word prediction objective undoes some of the benefits of instruction tuning and reduces the average performance by 0.8 F1 points. We conduct additional experiments with Bonito to understand the effects of the domain, the size of the training set, and the choice of alternative synthetic task generators. Overall, we show that learning with synthetic instruction tuning datasets is an effective way to adapt language models to new domains. The model, dataset, and code are available at https://github.com/BatsResearch/bonito.

heading: 1 Introduction
text: Large language models show remarkable zero-shot capabilities by simply learning to predict the next
token at scale (Brown et al., 2020; Touvron et al., 2023). By fine-tuning these models on instruction tuning datasets containing many tasks—each comprising an input instruction and a desired response—the model generally improves in its ability to respond to unseen instructions. However, this generalization is still limited by the qualities of the instruction tuning dataset. Existing datasets like Public Pool of Prompts (P3) (Bach et al., 2022), Natural Instructions (Mishra et al., 2022; Wang et al., 2022), and Dolly-v2 (Conover et al., 2023) focus on text from the Web and classic natural language tasks so that they can serve a wide range of use cases, i.e., they are a one-size-fits-all approach. On the other hand, tasks in areas like biomedical and legal domains require specialized, often implicit, domain knowledge. We study how to adapt language models to follow instructions in specialized domains without annotated data.
The ability to follow task-specific instructions in specialized domains is important for bringing the benefits of large language models to a wider range of users. Recent evaluations—including evaluations of proprietary models—show that they often significantly underperform specialized models (Kocoń et al., 2023; Shen et al., 2023; Ziems et al., 2023), particularly in domains requiring subject matter expertise. This motivates us to investigate effective ways to provide domain knowledge to large language models.
Self supervision in the form of next word prediction on the target corpus is a simple way to teach language models about new domains (Gururangan et al., 2020). However, this approach requires an enormous amount of training to achieve strong performance (Chen et al., 2023). Further, we find that self supervision can undo the benefits of instruction tuning (see Section 5.3). Alternatively, continued training of models with instructions from specialized domains significantly improves performance (Scialom et al., 2022; Shi and Lipani, 2023;
1 12585
Yunxiang et al., 2023; Deng et al., 2023; Singhal et al., 2023a; Wu et al., 2024). However, they need to repeat the time-consuming and labor-intensive process of annotating a domain-specific dataset. Furthermore, collecting instructions in specialized domains is very expensive because they are annotated by domain experts such as scientists and researchers (Thulke et al., 2024). In this work, we automate the creation of instruction tuning datasets in specialized domains.
We create Bonito, an open-source model to convert unannotated text from specialized domains into task-specific training datasets for instruction tuning (Figure 1). We call this problem conditional task generation. Our key idea is to make a new largescale dataset called Conditional Task Generation with Attributes (CTGA), to train Bonito, by reorganizing existing instruction tuning datasets (see Figure 2). Instruction tuning datasets like P3 (Bach et al., 2022) exist as templates that convert semistructured examples of natural language tasks into a fully prompted format, in which both the input and the desired response are text strings. We focus on a subset of the templates in P3 that require a context or a passage to complete the task. For example, a context could be a paragraph that contains a fact or that contains the answer to a question. Then, we remix these templates to create the meta-templates. Each meta-template for a dataset produces training examples in which the input is context and a task attribute such as yes-no question answering, and the output is the entire task: the instruction (including
the context) and the desired response. In this way, we can easily create abundant, diverse examples to train Bonito. After training Bonito, we can use new unannotated text from the target domain as the context to generate task-specific synthetic datasets and train specialized language models.
Bonito significantly improves over self supervision on zero-shot task adaptation of pretrained and instruction tuned models. We use Bonito to generate instruction tuning data for seven datasets across three task types—yes-no question answering (PubMedQA and Privacy Policy QA), extractive question answering (SQuADShifts-NYT, Amazon, and Reddit), and natural language inference (ContractNLI and Vitamin C)—and adapt language models. Our results show that Bonito improved Mistral-7B by 34.7 F1 points and Llama 2 7B by 31.6 F1 points over the self supervised baseline, next word prediction objective. We also consider a more practical setting where we further train Mistral-7B-Instruct-v0.2 and instruction tuned variants of Mistral-7B and Llama 2 7B trained on the T0 split of the P3 dataset. Our results show that Bonito outperforms the strong zero-shot baseline performance by an average of 22.1 F1 points across all the models. On the other hand, we find that self supervision undoes some of the benefits of instruction tuning, i.e., it leads to catastrophic forgetting, resulting in a drop in performance by an average of 0.8 F1 points across all models. Our analysis of Bonito shows that even task specialized models can be further improved by simply learning
2
on Bonito generated tasks (see Section 6.1). We also find that training with more synthetic instructions on datasets like PubMedQA and Vitamin C improves model performance the most compared to other datasets (see Section 6.2). We perform additional experiments by prompting off-the-shelf open-source models like Zephyr-7B-β and Mistral7B-Instruct-v0.2 and GPT-4 to generate tasks and find they can often improve the pretrained models but still struggle to increase model performance further when they are instruction tuned (see Section 7). Finally, our human evaluation of Bonito-generated tasks shows that Bonito and GPT-4o generate the same answer on 71 to 77 percent of the tasks. This indicates that Bonito generates high-quality tasks. However, there is room for improvement in generation quality to increase downstream model performance.
In summary, our main contributions are:
• We introduce Bonito, an open-source model for conditional task generation model to convert the user’s unannotated text into taskspecific instruction tuning datasets.
• Our experiments on zero-shot task adaptation on seven datasets across three task types show that Bonito improves over the self supervised baseline by an average of 33.1 F1 points on the pretrained models and 22.9 F1 points on the instruction tuned models.
• We analyze the effect of the domain, training size, and the choice of alternative task generators highlighting the benefits and limitations of Bonito.

heading: 2 Zero-Shot Task Adaptation
text: We describe the problem of zero-shot task adaptation. We have a language model, either pretrained via self supervision or further fine-tuned on a training mixture like P3 (Bach et al., 2022), along with a corpus of unannotated text from the target domain. We also know the target task type e.g., extractive question answering, natural language inference, etc. If the target task type has a fixed set of labels such as “yes” or “no” in yes-no question answering, we assume access to the label space. Our goal is to adapt the language model to follow task instructions in the target domain without human annotations, achieving zero-shot task adaptation.

heading: 3 Related Work
text: Instruction Tuning Multitask instruction tuning of language models dramatically improves their ability to follow instructions and generalize to new unseen tasks (Sanh et al., 2022; Wei et al., 2022; Mishra et al., 2022; Longpre et al., 2023; Chung et al., 2022; Zhou et al., 2023; Li et al., 2023). Typically, pretrained models are trained to follow instructions on large-scale training mixtures such as P3 (Bach et al., 2022) and the FLAN collection (Longpre et al., 2023). In this work, we use P3 to create meta-templates and train Bonito to generate NLP tasks in specialized domains.
Domain Adaptation Several works have adapted large language models to tasks in specialized domains (Gururangan et al., 2020; Yunxiang et al., 2023; Cui et al., 2023; Wu et al., 2023). Several works (Gu et al., 2021; Chen et al., 2023) show that self supervision or continuing the pretraining objective of the pretrained language model on the target domain corpus improves downstream performance. In this work, we find that self supervision improves the performance of pretrained models but hurts the performance of instruction tuned models (Section 5).
Recent work has adapted language models by training on large-scale in-domain datasets(Parmar et al., 2022; Gupta et al., 2022; Singhal et al., 2023b; Deng et al., 2023) or with a few examples from domain-specific tasks (Singhal et al., 2023a). However, annotating training datasets for new domains is labor-intensive and expensive (Thulke et al., 2024). We focus on generating training datasets in specialized domains and adapting language models without annotations.
Zero-shot task adaptation is closely related to unsupervised domain adaptation (Ganin and Lempitsky, 2015). In unsupervised domain adaptation, a trained model is used to generate pseudo-labels for the target unlabeled data and then trained on these labels. Naive pseudo-labeling cannot be applied to this work since we consider tasks like question answering and natural language inference tasks that require a question or a hypothesis before producing an answer in natural language. Further, popular techniques used in unsupervised domain adaptation such as choosing top-K confident classes (Huang et al., 2022; Menghini et al., 2023) cannot be easily adapted to NLP tasks as there may not be an explicit notion of classes.
There is a growing interest in using retrieval
3
augmented generation (RAG) for domain-specific question answering (Lewis et al., 2020; Karpukhin et al., 2020; Siriwardhana et al., 2023; Zhang et al., 2024). In a RAG pipeline, given a question, the most relevant documents are retrieved before accurately producing an answer with a language model. Our work compliments the RAG pipeline as we assume access to the gold documents or paragraphs from specialized domains and improve the language model’s ability to answer the questions.
Task Generation Task generation is a fastgrowing area of research to adapt large language models to follow instructions (Wang et al., 2023; Taori et al., 2023; Honovich et al., 2023; Köksal et al., 2023; Liu et al., 2024). They typically condition GPT on a set of seed task demonstrations and generate new synthetic tasks (Wang et al., 2023; Honovich et al., 2023). However, task generation conditioned on the user’s unannotated text has mostly been ignored. Additionally, generating with API-based models is expensive and cannot be used for proprietary or private research data (Köksal et al., 2023). On the other hand, Bonito is an open-source model that can be used to create tasks with the user’s unannotated text without additional API costs.
Recently, Li et al. (2023) proposed to learn a backtranslation model, similar to Bonito, to iteratively grow and refine their instruction tuning dataset (Gulcehre et al., 2023). However, they focus on generating instructions conditioned on the unannotated text from a web corpus for long-form conversational data where the answer to the instruction is the unannotated text. In contrast, we focus on generating NLP tasks conditioned on a task type and unannotated text from a specialized domain. Further, we consider tasks such as question answering and natural language inference that require a question or a hypothesis before generating the appropriate answer.
Concurrent to this work, Yehudai et al. (2024) use in-context learning with Falcon-40B and Llama-65B to generate “grounded tasks” to adapt smaller models like FLAN-T5-XL (3B). These grounded tasks are similar to conditional tasks, except the instructions do not necessarily refer directly to the user’s text. They might only be based on it, such as asking an open-ended question based on the original text. Our work goes further in several ways. First, we study how to create an open-source model for conditional task generation,
as opposed to relying on prompting alone. Second, Bonito has only 7B parameters and we show that it creates data that can improve instruction tuned models of the same size and outperform even larger models like Flan-T5-XXL (11B) (see Appendix D). Third, we evaluate tasks with precise correct/incorrect answers, such as yes-no question answering and natural language inference, as opposed to tasks evaluated with similarity metrics.
Knowledge Distillation Knowledge distillation is a well-studied area (Hinton et al., 2015; Sanh et al., 2019; He et al., 2020). Typically, smaller models learn from the outputs of a larger model. Most recently, API-based models have been used to generate tasks and distilled into smaller models to mimic the abilities of the API-based models (Peng et al., 2023; Gudibande et al., 2023). In this work, we use Bonito to generate tasks based on the user’s context and distill them into pretrained and instruction tuned models of the same size for zero-shot task adaptation (see Section 5).
Question Generation Several works have been proposed in question generation over the years (Mitkov and Ha, 2003; Pan et al., 2020; Lewis et al., 2021; Ushio et al., 2023). They often use heuristics such as templates (Mitkov and Ha, 2003), named entity recognition (Lewis et al., 2021), and semantic graphs (Pan et al., 2020). In our work, we train a language model without relying on task-specific heuristics. Ushio et al. (2023) is closely related to our work as they train a unified model to generate extractive questions and answers, but only focus on adapting small pretrained language models like T5-Large (770M). In contrast, Bonito can generate tasks beyond extractive question answering and enable zero-shot task adaptation on several task types with large models like Llama 2 7B and Mistral-7B.

heading: 4 Bonito: Learning to Generate Tasks
text: We describe the steps to create the Conditional Task Generation with Attributes (CTGA) dataset and train Bonito. Then, we briefly describe the procedure to create synthetic instruction tuning datasets with the target unannotated texts to adapt language models.
Key Properties We outline the key properties that we desire in our conditional task generation model: (1) given a corpus containing articles and paragraphs, the model should take the text as input
4
and generate high-quality tasks that require minimal cleaning or post-processing, (2) the model should adhere to the task type like extractive question answering or natural language inference task, and (3) the model should generate diverse tasks for the exact text with varying styles.

heading: 4.1 Creating Bonito: Dataset and Training
text: To create a model that generates tasks conditioned on text, we create a new training dataset: Conditional Task Generation with Attributes (CTGA). The dataset contains 1.65 million examples derived from P3 (Bach et al., 2022) by annotating 323 prompt templates from 38 datasets with 16 task types (see Appendix H). Then, we train a pretrained large language model on this training dataset to create Bonito.
Constructing the Dataset Figure 2 shows the process of constructing the Conditional Task Generation with Attributes (CTGA) dataset. First, we identify datasets from P3 (Bach et al., 2022) that require a passage or a context to complete the task. For example, SQuAD (Rajpurkar et al., 2016) requires a context to answer extractive question answering tasks, whereas CommonSenseQA (Talmor et al., 2019) asks a multiple choice question without providing any relevant text. We identify a total of 38 datasets to be included in CTGA. For each dataset, we also collect the Jinja1 templates from P3. Next, we remix the Jinja templates to create meta-templates. A meta-template is a Jinja template that includes the task attribute or the task type and the key for the context column in the input and the Jinja template for the instruction-response pair in the output with a placeholder {{context}} to
1https://jinja.palletsprojects.com/en/3.1.x/
avoid repeating the context. Since Jinja templates from P3 do not include a task type, we manually annotate them with a target task type such as yes-no question answering (see Appendix H for details). Overall, we get 323 meta-templates spanning 16 task types (See Table 13 for the list of task types). Finally, we apply the meta-templates to all the examples in a dataset to create the CTGA dataset, i.e., we replace the keys for the columns in the Jinja templates with corresponding key-value pairs from the examples. If the dataset has multiple metatemplates, we uniformly sample one meta-template per example. We limit the total number of examples per dataset to 100,000. The final training dataset is used to train Bonito.
Training the Bonito Model We train Bonito by fine-tuning Mistral-7B, an open-source decoder language model (Jiang et al., 2023), on the CTGA dataset. The model is trained by optimizing the cross entropy loss over the output tokens. We include all the hyperparameters and training details in Appendix F.1.

heading: 4.2 Adapting Models with Bonito
text: We use Bonito to create synthetic instruction tuning datasets for the target unannotated texts. Then, the target language model is adapted by training on the synthetic dataset to get the specialized language model.
Generating the Synthetic Dataset Figure 1 shows the inference with Bonito to generate the synthetic instruction tuning dataset. The unannotated text and the task type are passed to the Bonito model to get the synthetic instruction-response pairs. The process is repeated for all the unannotated text to get the training dataset. The generated
5
pairs are then post-processed into a standardized instruction-response format for instruction tuning. In each generation, we replace {{context}} with the corresponding unannotated text from the input. If the generated output is not parsable due to missing 
, we filter them out.
Adapting the Target Model We train the target language model on the synthetic instruction tuning dataset containing instruction-response pairs. The model is trained using a cross entropy loss over the response tokens. Additional details in Section 5.1.

heading: 5 Experiments

heading: 5.1 Experiment Setup
text: Target Tasks and Datasets We consider three target tasks: yes-no question answering (YNQA), extractive question answering (ExQA), and natural language inference (NLI). Table 1 shows the seven datasets across three task types and the number of unannotated text in each dataset. We use the unannotated text from the datasets to train the specialized language models. For yes-no question answering, we choose PubMedQA (Jin et al., 2019) and Privacy Policy QA (Ravichander et al., 2019). For extractive question answering, we choose the SquadShifts dataset (Miller et al., 2020) that includes splits for the New York Times (NYT), Amazon, and Reddit. Finally, for the NLI task, we choose Contract-NLI (Koreeda and Manning, 2021) and Vitamin C (Schuster et al., 2021). We provide additional details in Appendix A.
In our experiments, we focus on tasks such as question answering and natural language inference that require us to generate a question or hypothesis and an answer. Prior work generates synthetic data for tasks like summarization that do not warrant a specialized task generation model (Yehudai et al., 2024). Other work focuses on generating instructions (Li et al., 2023; Köksal et al., 2023) for
long-form text generation tasks where the solution to the instruction is the unannotated text. While these long-form generative tasks are useful for applications such as code generation, domains like biomedical and legal that we consider might benefit more from traditional predictive tasks (Miller, 2024).
Baselines We consider two key baselines: zeroshot and self supervised baseline. For the zero-shot baseline, we prompt the model and run the evaluation without using any of the unannotated text from the target task (None). For the self supervised baseline, we use task-adaptive pretraining (TAPT) (Gururangan et al., 2020). The learning objective is to continue to the pretraining objective on the unannotated text in the downstream dataset. In our experiments, we use the next word prediction learning objective to fine-tune Mistral-7B and Llama 2 7B models.
Synthetic Task Generation As described in Section 4.1, we prompt Bonito with the unannotated texts and the target task type to generate the instruction tuning data. We use nucleus sampling (Holtzman et al., 2020) with a top P value of 0.95 and a temperature of 0.5, and a maximum sequence length of 256 in the vLLM framework (Kwon et al., 2023).
Models We adapt two pretrained large language models: Mistral-7B (Jiang et al., 2023) and Llama 2 7B (Touvron et al., 2023). They are decoder language models trained with the next word prediction objective on trillions of tokens. Both these models have around 7 billion parameters, with slightly different architectures optimized for sequence length and inference. For more details, see Touvron et al. (2023) and Jiang et al. (2023).
We also consider a more practical setting where we further adapt instruction tuned models to the target task. We first consider an off-the-shelf instruction tuned model: Mistral-7B-Instruct-v0.2. This model based on Mistral-7B achieves comparable performance to Llama 2 13B Chat on the MT-Bench (Zheng et al., 2023). In addition, we train Mistral-7B and Llama 2 models on the T0 split from the P3 dataset (Bach et al., 2022) and adapt them to the target tasks. We call these models Mistral-7BP3 and Llama 2P3. For the instruction tuning details, see Appendix F.2
Training Details We fine-tune the language models on the supervision sources—TAPT, and
6
Bonito—using Q-LoRA (Dettmers et al., 2023). When further adapting Mistral-7BP3 and Llama 2 7BP3, we fine-tune the same Q-LoRA adapter on the supervision sources instead of merging and reinitializing the adapters. We train all the models for 1 epoch. If the dataset size is greater than 160,000 examples, then we train for 10,000 steps. We use the same hyperparameter values from Dettmers et al. (2023) to avoid additional hyperparameter tuning. Depending on the dataset, training on four GPUs takes 25 minutes to 17 hours. For more additional details, see Appendix F.5.
Evaluation We evaluate the performance of the models on the test splits of the target datasets (see Table 6 in Appendix A). To prevent “prompt hacking”, following Sanh et al. (2022), we first write five prompt templates for the target datasets and then benchmark the model performance. See Appendix I for all the prompts used in our experiments. We follow standard evaluation practices and report the F1 score for all the datasets. Following Radford et al. (2019), we evaluate yes-no question answering and NLI using ranked classification, i.e., we generate the loglikelihood of all the choices and choose the sequence with the highest loglikelihood as the prediction. Following Rajpurkar et al.
(2016), we evaluate models on extractive question answering by computing the SQuAD F1 score on the generated output. During evaluation, we use greedy decoding to generate the output from the model and then calculate the SQuAD F1 score for the dataset.

heading: 5.2 Adapting Pretrained Models
text: Table 2 shows that adapting pretrained models with synthetic instruction tuning data generated from Bonito significantly outperforms zero-shot and TAPT. Bonito improves over the zero-shot performance by an average of 37.7 F1 points across Mistral-7B and Llama 2. Although TAPT shows a nominal improvement of only 4.5 F1 points on average, we find that Bonito outperforms TAPT by an average of 33.3 F1 points across both models. This result strengthens our main claim that synthetic instruction tuning data is a much better way of providing domain knowledge compared to self supervision. Finally, we observe that the Mistral-7B shows significantly greater improvement in performance compared to Llama 2 7B suggesting that stronger pretrained models might respond better to synthetic instructions.
7

heading: 5.3 Adapting Instruction Tuned Models
text: Table 3 shows that Bonito improves instruction tuned models by an average of 22.1 F1 points whereas TAPT reduces the average performance by 0.8 F1 points. This is because self supervision with TAPT interferes with prior instruction tuning and leads to catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017). In contrast, adapting instruction tuned models with Bonito-generated tasks further improves performance in specialized domains. We also observe that Bonito addresses the task-specific deficiencies and improves the instruction tuned models. For example, we find that Bonito significantly improves Mistral-7B-Instructv0.2 performance on extractive question answering as it typically generates chat-like responses for questions. Finally, adapting instruction tuned variants of Mistral-7B and Llama 2 7B achieves a higher F1 score than adapting the pretrained models (Table 2).

heading: 6 Analysis

heading: 6.1 Impact of Domain Knowledge
text: Here we ask a key question: are we improving the language model by learning about the domain or are we distilling instructing tuning data from a stronger to a weaker model? To answer this question, we train task-specialized instruction tuned models and then further train them on synthetic tasks generated from Bonito for the target unannotated texts. We create the task-specialized training dataset by selecting the instructions in CTGA with the target task type. We train two task-specialized models in the standard instruction-response format: Mistral-7B-Instruct-v0.2special and Mistral7Bspecial. We also train a task-specialized Bonito special on the same task-specific dataset. See Appendix F.3 for training details.
Table 4 shows that further training on synthetic instructions can improve performance suggesting
that the model benefits from the unnannotated text from the specialized domain. We find that training on Bonito tasks either slightly improves or matches the performance of task-specialized models on average. When we train on Bonito special tasks, we further improve task-specialized Mistral-7B-Instructv0.2 by 0.5 F1 points and Mistral-7B and 2.5 F1 points. We see that the model performance often reduces on extractive QA. We suspect that the model performance has saturated due to the presence of SQuAD in the task-specialized training dataset. To further improve on extractive question answering, we could benefit from having access to a few examples from the target dataset. Finally, we almost always improve performance on Vitamin C and PubMedQA datasets highlighting the importance of training on more task samples (see Section 6.2).

heading: 6.2 Effect of the Training Dataset Size
text: Here we study the effect of the size of the training dataset. In particular, we study how Mistral7B performance varies when trained on different quantities of synthetic instruction tuning data for PubMedQA and Vitamin C. Figure 3 shows that training on more steps typically improves perfor-
8
mance. We find that Bonito on PubMedQA reaches the peak performance of 47.1 F1 points after 10,000 steps but the F1 can fluctuate when trained for fewer steps. In contrast, we find that Bonito gets the highest performance of 73.3 F1 points after 2,500 points and gradually diminishes the performance to 71.7 F1 points. Finally, if available, we suggest using a validation set to select the best-performing model checkpoint.

heading: 6.3 Human Evaluation: Agreement with GPT-4o
text: We manually evaluate Bonito tasks by comparing the answers generated by Bonito and GPT-4o.
Setup We sample 100 unique instructions each from Bonito for PubMedQA, SQuADShifts Reddit, and ContractNLI. Next, we prompt GPT-4o with instructions generated by Bonito to produce an answer. We prefix the instructions with a simple format prompt to produce answers in the desired format with GPT-4o. Finally, we ask humans if GPT-4o and Bonito produce the same (including paraphrased) answers for the instructions. For reproducibility, we use GPT-4o-2024-05-13. We separately ask the first three authors of the paper to compare the responses from both models. We choose the final agreement if two or more annotators agree on either a match or no match.
Results Table 5 shows that Bonito and GPT-4o produce the same answer for Bonito tasks 71 to 76 percent of the time across three datasets, with high inter-annotator agreement. Each dataset reveals different patterns of disagreement. In PubMedQA, GPT-4o generates the response as “unanswerable” when the answer is not present in the passage, whereas Bonito produces either “yes” or “no”, or “true” or “false”. In SQuADShifts Reddit, Bonito almost always extracts the answer from the paragraph, whereas GPT-4o can generate answers with additional text that may not be present in the paragraph. In ContractNLI, both models can pro-
duce plausible answers. In one example, Bonito generates the question, ‘Does this imply that “This document is confidential information”? Yes, no, or maybe?’. Bonito answers “yes”, whereas GPT-4o produces the answer “maybe”. In such cases, we annotate the responses as no match, reducing the agreement.
Our analysis shows that Bonito generates highquality tasks with accurate answers. However, there is still room to improve the quality of the tasks. Improving the task quality could further increase the downstream model performance. Therefore, we believe that research on conditional task generation is an important direction for future work.

heading: 7 Additional Experiments
text: We briefly describe additional experiments included in Appendix B and C.
In Appendix B, we generate synthetic tasks by prompting Mistral-7B-Instruct-v0.2 and Zephyr7B-β. Our results show that the synthetic tasks from Mistral-7B-Instruct-v0.2 and Zephyr-7B-β improve the average performance of Mistral-7B but decrease significantly when adapting MistralP3. This indicates that we require high-quality synthetic tasks to increase the performance of strong instruction-tuned models. In Appendix C, we generate synthetic tasks with GPT-4 for Privacy Policy QA, SQuADShifts Reddit, and ContractNLI. Our results show that GPT-4 improves MistralP3 on Privacy Policy QA and ContractNLI but slightly reduces performance on SQuADShifts Reddit. Finally, we analyze the generated tasks and identify common issues with both open-source models and GPT-4, such as the distribution of the label space and “chatty” responses, which potentially lead to the drop in performance.

heading: 8 Conclusion
text: We present Bonito, an open-source model for conditional task generation that converts unannotated text into instruction tuning datasets. We show that training with synthetic instruction tuning datasets in specialized domains is a strong alternative to self supervision. Our experiments demonstrate that Bonito-generated instructions improve pretrained and instruction tuned models on zero-shot task adaptation. Overall, Bonito enables practitioners to adapt large language models to tasks on their data without annotations.
9

heading: Acknowledgements
text: We appreciate Yeganeh Kordi, Zheng-Xin Yong, Aidan LaBella, and members of the Brown SuperLab for their thoughtful comments and feedback on our draft. We gratefully acknowledge support from Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, a company that provides software and services for data-centric artificial intelligence.

heading: Limitations
text: Our work relies on the availability of large amounts of unannotated text. If only a small quantity of unannotated text is present, the target language model, after adaptation, may experience a drop in performance. While we demonstrate positive improvements on pretrained and instruction-tuned models, our observations are limited to the three task types considered in our experiments.

heading: Potential Risks
text: Bonito poses risks similar to those of any large language model. For example, our model could be used to generate factually incorrect datasets in specialized domains. Our model can exhibit the biases and stereotypes of the base model, Mistral-7B, even after extensive supervised fine-tuning. Finally, our model does not include safety training and can potentially generate harmful content.

heading: A Datasets
text: We briefly describe the datasets used in our experiments. We get all the datasets from the datasets library (Lhoest et al., 2021). For all the datasets, we consider five prompt templates (see Appendix I). Table 6 shows the statistics for the test splits in the evaluation datasets. Below we include details about the evaluation datasets:
• PubMedQA (Jin et al., 2019): The dataset asks questions about PubMed abstracts that can be answered with yes, no, or maybe. We use the abstracts without the questions as unannotated text for adaptation. During the evaluation, we provide the PubMed abstract along with the question to the model.
• Privacy Policy QA (Ravichander et al., 2019): The dataset consists of paragraphs from privacy policies with corresponding questions. The task involves determining the relevance of each question, formatted as a yes-no questionanswering task. We use the processed test split in Privacy Policy QA from Guha et al. (2023) as unannotated text.
• SquadShifts (Miller et al., 2020): The dataset is designed to test the robustness of extractive question answering models. We use three of the four test sets in our work — New York Times articles, Reddit posts, and Amazon product reviews. During training, we use the articles or context from the test set without the questions and generate extractive question answering tasks with Bonito. During evaluation, we evaluate the same test set with the questions in the dataset.
• ContractNLI (Koreeda and Manning, 2021): ContractNLI is a natural language inference task to aid contract review. Given a hypothesis about a clause in a contract, the model predicts if the hypothesis is supported, refuted, or not mentioned.
• Vitamin C (Schuster et al., 2021): This dataset focuses on fact verification in Wikipedia framed as a natural language inference task. Each example consists of an evidence text from Wikipedia and a corresponding fact. The model is asked to indicate whether the fact is supported, refuted, or neutral.

heading: Dataset # Classes # Test Examples

heading: B Generating Tasks with Open-Source Models
text: We use Mistral-Instruct-v0.2 and Zephyr-β, two popular openly available models, to generate instruction tuning data. Then, we adapt pretrained Mistral-7B and Mistral-7B-P3 on the generated tasks.

heading: B.1 Generating Synthetic Datasets
text: Here we describe the process of creating synthetic datasets with Mistral-Instruct-v0.2 and Zephyr-β. We prompt these models to generate questions or hypotheses for the target unannotated text. Table 7 shows the prompts we used to generate the tasks. Creating these prompts required a tremendous amount of prompt engineering as they strug-
15
gled to follow the prompt format (Xia et al., 2024). We first generate the question or the hypothesis and then re-prompt the model to produce the answer. For question answering tasks, we prepend the question as the prompt followed by the unannotated text to generate the output. For the NLI datasets, we use five prompt templates from the ANLI dataset in Bach et al. (2022) and plug in the hypothesis and the unannotated text as the input to the model to generate the answer. We use the same input and output to adapt the pretrained and instruction tuned models. For all the generations, we use a top-P of 0.95, temperature of 0.5, and maximum token length of 256.

heading: B.2 Results
text: Table 8 shows results for zero-shot task adaptation with openly available models. We see that both Mistral-7B-Instruct-v0.2 and Zephyr-7B-β improve performance over the pretrained Mistral-7B but find that they severely hurt average performance compared to Mistral-7BP3.
We suspect that the drop in performance is due to issues related to the generated tasks. For extractive question answering, we find that Mistral7b-Instruct-v0.2 and Zephyr-β often generate questions with multiple sub-questions that cannot be easily answered by extracting words from the context. Furthermore, the responses are “chatty” which might not be appropriate for extractive question answering. We also observe that the generated questions are often “positive”, i.e., they usually have “yes” or “true” as the answer. For example, 68% of the questions generated by Zephyr-β for PubMedQA have answers starting with “yes” or “true” but only 5% of the questions have answers that start with “no” or “false”. We observe a similar “positive” bias in the hypotheses generated for natural language inference datasets.

heading: C Generating Tasks with GPT-4
text: Here we use GPT-4 to generate tasks to adapt Mistral-7BP3.

heading: C.1 Generating Synthetic Datasets
text: We prompt GPT-4 to generate tasks for Privacy Policy QA, SQuADShifts Reddit, and Contract NLI. For simplicity, we use the same prompts from Appendix B.1 to generate questions and hypotheses (see Table 7). For Privacy Policy QA, we add a simple instruction prefix to answer the question with yes or no along with the question and the context to generate the answer. For extractive question answering, we add the prefix "Extract the exact words from the paragraph for the question. If the question is not answerable, say N/A." before the question and the context and produce the answer. We use a simpler prefix "Answer the following question." when training the downstream model on SQuADShifts Reddit. Finally, for ContractNLI, we use the same prompts from Appendix B.1 to generate answers. For all the generations, we use gpt-4-0613 with a maximum token length of 256, top-P of 0.95, and temperature of 0.5.

heading: C.2 Results
text: Table 9 shows that tasks generated by GPT-4 improve performance over Mistral-7BP3 on Privacy Policy QA and ContractNLI but slightly reduce performance on SQuADShifts Reddit. While GPT-4 is
16
a much better task generator than the open-source models, we find that GPT-4 also suffers from a similar issue. For example, ContractNLI often has a positive hypothesis and PrivacyQA has a question with the answer yes. While GPT-4 follows the instruction to generate exactly one question for the paragraph, we find that it produces slightly longer answers to the question. The SQuAD metric penalizes if there unwanted tokens in the answers. Finally, the cost of generating tasks with GPT-4 makes it prohibitively expensive to generate tasks for larger datasets like PubMedQA and Vitamin C.

heading: D Bonito vs. FLAN
text: We evaluate the zero-shot performance of FLANT5-XXL (11B) and FLAN-T5-XL (3B) models (Longpre et al., 2023) on the target datasets used in our experiments. Table 10 shows that Mistral-7B-Instruct-v0.2 and MistralP3 with Bonito-generated tasks improves over FLAN-T5XXL (11B) by 2.7 F1 points and 4.0 F1 points. Our results also show that Mistral-7B-Instruct-v0.2 and MistralP3 with Bonito outperforms FLAN-T5-XL (3B) by 6.4 F1 points and 7.7 F1 points.

heading: E Bonito with Smaller Models
text: We report an additional comparison with Bonito trained on Pythia (2.8B) (Biderman et al., 2023). We follow the same experimental setup used in Section 5.1.
Results Table 11 shows that Bonito improves Pythia (2.8B) by an average of 30.3 F1 points across all the datasets. We observe that Pythia (2.8B) with Bonito performs better than Mistral with TAPT and Llama 2 with TAPT despite being twice as small (See Table 2). These results show that Bonito can be used to create small but powerful specialized language models.

heading: F Training Details
text: Here we provide training details for models used in the paper.

heading: F.1 Training Bonito
text: We train Mistral-7B on the conditional task generation with attributes (CTGA) dataset. From the training set, we uniformly sample 10,000 examples as the validation set to monitor the loss. The rest of the dataset is used for training Bonito. We train the model using Q-LoRA (Dettmers et al., 2023) by optimizing the cross entropy loss over the output tokens. The model is trained for 100,000 steps. The training takes about 4 days on four GPUs to complete. We include all the hyperparameters in Appendix F.5.
The same training recipe can be used to train other existing language models such as Falcon (Almazrouei et al., 2023), Pythia (Biderman et al., 2023), and RedPajama (Together, 2023). While models such as Llama 2 (Touvron et al., 2023) can be trained on CTGA, their license prohibits the use of the output to enhance any other large language model.
F.2 Instruction Tuned Models
Here we describe the procedure to train Mistral7BP3 and Llama 2 7BP3. We use the processed T0 dataset from Muennighoff et al. (2022). Since the dataset is large, we uniformly sample 1.6 million input-output examples and train the language model on them. Following Dettmers et al. (2023), we train the model for 10,000 steps with Q-LoRA and optimize the cross entropy loss over the output tokens. The training takes about 10 hours on four GPUs to complete. For the rest of the hyperparameters, see Appendix F.5.

heading: F.3 Training Task-Specialized Models
text: To train the task-specialized Mistral-7B-Instructv0.2special and Mistral-7Bspecial, we create a taskspecific dataset by filtering out task types from the
17
CTGA dataset. We selected datasets containing templates that correspond to three task types: yesno question answering, extractive question answering, and natural language inference. The datasets have a total of 130,703 examples for yes-no question answering, 378,167 examples for extractive question answering, and 100,250 examples for natural language inference.
To train the task-specialized Bonito special, we convert the same task templates into meta templates. Then, we use the meta templates to generate the dataset to train the model.
For fairness, we use the same hyperparameters to train task-specialized Bonito and the task-specialized Mistral-7B-Instruct-v0.2special and Mistral-7Bspecial models. Since the datasets have significantly fewer examples than CTGA, we train these models for at most 10,000 steps. If the training mixture has less than 160,000 examples, we train the Bonito model for 1 epoch. The training on four GPUs takes about 4 to 10 hours. For the rest of the hyperparameters, see Appendix F.5.

heading: Task type # Examples

heading: F.4 Software and Hardware Details
text: Our codebase is built using the transformers (Wolf et al., 2019) library in PyTorch (Paszke et al., 2019). We train all the models in a distributed multi-GPU environment using DeepSpeed (Rasley et al., 2020). We use the distributed data parallel in DeepSpeed to increase the effective batch size during training. For training and evaluation, we use the following GPUs depending on their availability on our compute cluster: NVIDIA GeForce RTX 3090, NVIDIA RTX A5500, NVIDIA RTX A6000, NVIDIA RTX A5000, and NVIDIA A40.

heading: F.5 Hyperparameters
text: Throughout our fine-tuning experiments, unless otherwise mentioned, we use the hyperparameters from Dettmers et al. (2023). Table 12 shows the hyperparameters in our experiments. We use gradient accumulation to achieve the effective batch size of 16. We also use gradient checkpointing to train large models like Llama 2 7B and Mistral-7B.
18

heading: G Use of AI Assistants
text: Our work used AI Assistants such as ChatGPT and Grammarly for spell-checking and fixing minor grammatical mistakes. We also use GitHub CoPilot in VSCode to write our codebase.

heading: H Conditional Task Generation with Attributes: Datasets and Tasks
text: Table 13 shows the task distribution of the conditional task generation with attributes dataset. Table 14 lists all the datasets along with the task types in the dataset. The dataset includes 16 task types across 38 datasets. The task types are summarization, sentiment analysis, multiple-choice question answering, extractive question answering, topic classification, natural language inference, question generation, text generation, question answering without choices, paraphrase identification, sentence completion, yes-no question answering, word sense disambiguation, paraphrase generation, textual entailment, and coreference resolution. The difference between extractive question answering and question answering without choices is that in extractive question answering the target answer is present in the context whereas in question answering without choices, that always is not the case.

heading: I Prompts for Evaluation

heading: I.1 PubmedQA
text: Dataset from Jin et al. (2019):
• Input
Given a passage: {{ context.contexts | join(" ") }}

heading: Answer the question: {{question}}
text: Summarize the above answer as YES, NO, or MAYBE?
Target
{{final_decision}}
Answer Choices
yes ||| no ||| maybe
• Input
I'm a doctor and I want to answer the question "{{question}}" using The following passage:
{{ context.contexts | join(" ") }}
Summarize the above answer as YES, NO, or MAYBE?
Target
{{final_decision}}
Answer Choices
yes ||| no ||| maybe
• Input
What is the answer to the question "{{question}}" based on The following passage:
{{ context.contexts | join(" ") }}
Summarize the above answer as YES, NO, or MAYBE?
Target
{{final_decision}}
Answer Choices
yes ||| no ||| maybe
• Input
Please answer the question "{{question}}" using The following passage:
{{ context.contexts | join(" ") }}
Summarize the above answer as YES, NO, or MAYBE?
Target
{{final_decision}}
Answer Choices
yes ||| no ||| maybe
• Input
19
Given the following passage, answer the question: "{{question}}"
Passage: {{ context.contexts | join(" ") }}
Summarize the above answer as YES, NO, or MAYBE?
Target
{{final_decision}}
Answer Choices
yes ||| no ||| maybe

heading: I.2 Privacy Policy QA
text: Dataset from Ravichander et al. (2019).
• Input
Given the context, is this related to the question? Context: {{text}} Question: {{question}}
Target
{{answer}}
Answer Choices
Relevant|||Irrelevant
• Input
Is this question "{{question}}" related to this context "{{text}}"?
Target
{% if answer == "Relevant" %} Yes {% else %} No {% endif %}
Answer Choices
Yes|||No
• Input
Can this "{{text}}" help answer this question "{{question}}"?
Target
{% if answer == "Relevant" %} Yes {% else %} No {% endif %}
Answer Choices
Yes|||No
• Input
As a lawyer, can you answer the question given the context? Question: {{question}} Context:{{text}}
Target
{% if answer == "Relevant" %} Yes {% else %} No {% endif %}
Answer Choices
Yes|||No
• Input

heading: Question:{{question}} Context:{{text}}

heading: Is the question related to the context?
text: Target
{% if answer == "Relevant" %} Yes {% else %} No {% endif %}
Answer Choices

heading: Yes|||No

heading: I.3 SQuADShifts
text: Dataset from Miller et al. (2020).
I.3.1 NYT • Input
20
After reading the following paragraph, please answer this question: {{question}}
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I'm working on the final exam for my class and am trying to figure out the answer to the question "{{question}}" I found the following info on New York Times and I think it has the answer. Can you tell me the answer?
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I've always wondered: {{question}}

heading: I searched New York Times and this is what I
text: found. What's the answer?
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
{{context}}
With the help of the passage, please answer the following question: {{question}}
Target
{{answers["text"]|choice}}
• Input
{{["Question", "Problem"] | choice}} {{range(1, 12) | choice}}: {{question}}

heading: Hint: {{context}}
text: Target
{{answers["text"] | most_frequent | choice}}
I.3.2 Amazon • Input
After reading the following paragraph, please answer this question: {{question}}
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I'm working on the final exam for my class and am trying to figure out the answer to the question "{{question}}" I found the following info on Amazon and I think it has the answer. Can you tell me the answer?
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I've always wondered: {{question}}

heading: I searched Amazon and this is what I found.
text: What's the answer?
{{context}}
Target
21
{{answers['text'] | most_frequent | choice}}
• Input
{{context}}
With the help of the passage, please answer the following question: {{question}}
Target
{{answers["text"]|choice}}
• Input
{{["Question", "Problem"] | choice}} {{range(1, 12) | choice}}: {{question}}

heading: Hint: {{context}}
text: Target
{{answers["text"] | most_frequent | choice}}
I.3.3 Reddit • Input
After reading the following paragraph, please answer this question: {{question}}
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I'm working on the final exam for my class and am trying to figure out the answer to the question "{{question}}" I found the following info on Reddit and I think it has the answer. Can you tell me the answer?
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
I've always wondered: {{question}}

heading: I searched Reddit and this is what I found.
text: What's the answer?
{{context}}
Target
{{answers['text'] | most_frequent | choice}}
• Input
{{context}}
With the help of the passage, please answer the following question: {{question}}
Target
{{answers["text"]|choice}}
• Input
{{["Question", "Problem"] | choice}} {{range(1, 12) | choice}}: {{question}}

heading: Hint: {{context}}
text: Target
{{answers["text"] | most_frequent | choice}}

heading: I.4 ContractNLI
text: Dataset from Koreeda and Manning (2021).
• Input
Suppose {{premise}} Can we infer that "{{hypothesis}}"? yes, no or maybe?
Target
22
{{answer_choices[label]}}
Answer Choices
No ||| Yes ||| Maybe
• Input
{{premise}}
Question: Does this imply that "{{hypothesis}}"? yes, no or maybe?
Target
{{answer_choices[label]}}
Answer Choices
No ||| Yes ||| Maybe
• Input
Take the following as truth: {{premise}} Then the following statement: "{{hypothesis}}" is {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target
{{answer_choices[label]}}
Answer Choices
False ||| True ||| Inconclusive
• Input
{{premise}} Based on that information, is the claim: "{{hypothesis}}" {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target
{{ answer_choices[label]}}
Answer Choices
False ||| True ||| Inconclusive
• Input
{{premise}} Based on the previous passage, is it true that "{{hypothesis}}"? Yes, no, or maybe?
Target
{{ answer_choices[label] }}
Answer Choices

heading: No ||| Yes ||| Maybe
text: I.5 Vitamin C
Dataset from Schuster et al. (2021).
• Input
Suppose {{evidence}} Can we infer that "{{claim}}"? yes, no or maybe?
Target
{% if label == "REFUTES" %} No {% elif label == "SUPPORTS" %} Yes {% else %} Maybe {% endif %}
Answer Choices
No ||| Yes ||| Maybe
• Input
{{evidence}}
Question: Does this imply that "{{claim}}"? yes, no or maybe?
Target
{% if label == "REFUTES" %} No {% elif label == "SUPPORTS" %} Yes {% else %} Maybe {% endif %}
Answer Choices
23
No ||| Yes ||| Maybe
• Input
Take the following as truth: {{evidence}} Then the following statement: "{{claim}}" is {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target
{% if label == "REFUTES" %} False {% elif label == "SUPPORTS" %} True {% else %} Inconclusive {% endif %}
Answer Choices
False ||| True ||| Inconclusive
• Input
{{evidence}} Based on that information, is the claim: "{{claim}}" {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target
{% if label == "REFUTES" %} False {% elif label == "SUPPORTS" %} True {% else %} Inconclusive {% endif %}
Answer Choices
False ||| True ||| Inconclusive
• Input
{{evidence}} Based on the previous passage, is it true that "{{claim}}"? Yes, no, or maybe?
Target
{% if label == "REFUTES" %} No {% elif label == "SUPPORTS" %} Yes {% else %} Maybe {% endif %}
Answer Choices

heading: No ||| Yes ||| Maybe

heading: J Qualitatitve Examples
text: Table 16 shows Bonito-generated tasks for the PubMedQA, SQuADShifts Amazon, and ContractNLI.
24
25
26
27

heading: Future Work
text: Therefore, we believe that research on conditional task generation is an important direction for future work