Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates
Authors: Avanika Narayan
Mayee F Chen
Kush Bhatia
Christopher Re
Venue: COLM (2024)
Keywords: instruction-tuning, programmatic data, large language models, LLMs
TL;DR: Cookbook is a framework for generating training data using simple Python functions over random tokens that improves LLM generative abilities across a range of tasks.

Abstract: Fine-tuning large language models (LLMs) on instruction datasets is a common way to improve their generative capabilities. 
However, instruction datasets can be expensive and time-consuming to manually curate, and while LLM-generated data is less labor-intensive, it may violate user privacy agreements or terms of service of LLM providers. Therefore, we seek a way of constructing instruction datasets with samples that are not generated by humans or LLMs but still improve LLM generative capabilities. In this work, we introduce Cookbook, a framework that programmatically generates training data consisting of simple patterns over random tokens, resulting in a scalable, cost-effective approach that avoids legal and privacy issues. First, Cookbook uses a template---a data generating Python function---to produce training data that encourages the model to learn an explicit pattern-based rule that corresponds to a desired task.  We find that fine-tuning on Cookbook-generated data is able to improve performance on its corresponding task by up to 52.7 accuracy points. Second, since instruction datasets improve performance on multiple downstream tasks simultaneously, Cookbook algorithmically learns how to mix data from various templates to optimize performance on multiple tasks. On the standard multi-task GPT4ALL evaluation suite, Mistral-7B fine-tuned using a Cookbook-generated dataset attains the best accuracy on average compared to other 7B parameter instruction-tuned models and is the best performing model on 3 out of 8 tasks. Finally, we analyze when and why Cookbook improves performance and present a metric that allows us to verify that the improvement is largely explained by the model’s generations adhering better to template rules.

Published as a conference paper at COLM 2024
COOKBOOK: A framework for improving LLM generative abilities via programmatic data generating templates
Avanika Narayan', Mayee F. Chen', Kush Bhatia Christopher Ré
Department of Computer Science
Stanford University Stanford, CA 94305, USA
(avanikan,mf mfchen, kushb, chr i edu
Abstract
Fine-tuning large language models (LLMs) on instruction datasets is com- mon way to improve their generative capabilities. However, instruction datasets can be expensive and time-consuming to manually curate, and while LLM-generated data is less labor-intensive it may violate user pri- vacy agreements or terms of service of LLM providers. Therefore, we seek way of constructing instruction datasets with samples that are not gener- ated by humans or L.MSs still improve LLM generative capabilities. In this work, introduce COOKBOOK, framework that programmatically generates training data consisting of simple patterns over random tokens, resulting in scalable, cost-effective approach that avoids legal and pri- vacy issues. First, COOKBOOK uses a template-a data generating Python function-to produce training data that encourages the model to learn an explicit pattern-baser rule that corresponds to desired task. We find that fine-tuning on CooKBoOK-generatee data is able to improve perfor- mance its corresponding task by up to 52.7 accuracy points. Second, since instruction datasets improve performance on multiple downstream tasks simultaneously, COOKBOOK algorithmically learns how to mix data from various templates to optimize performance on multiple tasks. On the standard multi-task GPT4ALL evaluation suite, Mistral-7B fine-tuned using -eoeeoo--generated dataset attains the best accuracy on average compared to other 7B parameter instruction-tuned models and is the best performing model on out of 8 tasks. Finally, we analyze when and why СоOKBOOK improves performance and present metric that allows to verify that the improvement is largely explained by the model's generations adhering better to template rules.
1 Introduction
Fine-tuning large language models (LLMs) on instruction datasets (Lian et al., 2023a; Long- pre et al., 2023; Achiam et al., 2023; "Teknium", 2023), can significantly improve LLM generative capabilities. However, these datasets can be time-consuming and expensive to curate. Moreover, recent alternatives such user chat logs (e.g., shareGPT) and LLM- generated data (Lian et al., 2023a; "Teknium" 2023) are less labor-intensive but can still be costly and may violate user privacy and the terms of service of LLM providers. Programmat-- ically generated data, where samples are generated by programs rather than by humans or LLMs, a potential alternative that more scalable, cost-effective, and avoids privacy and legal issues. Examples of programmatic data include data used in Dyck languages, sequence copying, and bit parity tasks (Jelassi et al., 2024; Hahn, 2020). However, seguuae unclear to what extent programmatic data can be used in place of standard instruction datasets to improve model capabilities; existing work focuses on using it for understanding how models learn specific patterns and for improving pre-training, which is still followed by fine-tuning the downstream task (Nanda al., 2023; Wu et al., 2022). This raises the question: is
'Equal contribution.

---

Published as a conference paper at COLM 2024
1. Template construction
Mixing
data
Natural language task
Rule Identity and extract
doc_OA..templltee
Question
Use the decument perwwer the following quastion.
Occument: the sun la ligg toot buil in the dids Quistion what lig, list
sky?
mult_choice,. coult_chhios.temppleee template()
LLM
data
Template
matching template))
Jemplatell
sandam sampletokknn
question document get, spantapon_Jength) Jength)
3. Template alignment statistic
subeir Joc document los qquesion)
answer = documentfsuser loc substr Ice len/question) +
instruction Use the document to enswer the following question. retum I' Tonstruction)
Sumpluul Lanolle
Document: (document)
tack samples
pie pemplas
Question: (question)
Documentt THE SUn 1a Degght ball in sky. Arsevar: the
(anlwar))
Temppile digement Terphitakkgrmmn stanatic
0+0
Progremmatic training data
Question:
LLM
Antree the sun
doc_QA_ te_emppatee alignment score s)
canu qp aan
Figure 1: COOKBOOK, (1) Templates approximate given task's "rule" and generate data consisting of patterns over random tokens. (2) (remplate-gnneaatee data can be mixed to improve multiple capabilities. (3) The template alignment statistic measures the extent to which the rule learned by the template responsible for improving L.M performance.
possible programmaticaaly generate an instruction dataset whose performance is competitine with human or LLM-generated instruction datasets on multiple downstream generative tasks?
There are two key considerations for programmatically generating a performant instruction dataset.
First, how do we design programmatic data that teaches the model rules correspond- ing to complex generative tasks that instruction datasets typically improve on? It is straightforward to construct programmatic data that teaches simple patterns like copy- ing, but doing the same for more complex rules is challenging. For example, consider the document-based question answering (QA) task, where model must answer a question about document. One underlying rule for this task is to identify the relevant subpassage and exiract an answer; it is unclear how to translate this rule into programmatic data.
Second, since instruction datasets improve on many tasks, how do we mix programmatic data corresponding to different rules optimize performance of multiple downstream tasks simultaneously? Recent work has shown that data mixture proportions can signifi- cantly impact downstream performance (Albalak et al., 2024). Since naive trial-and-erros is costly and has large search space, we need principled mixing approach.
We propose COOKBOOK, a framework that explores the use of templates cemplates-siplle Python functions-to produce programmatic training data that improves LLM performance by encouraging the model to learn explicit pattern-based rules that approximate desired gener- ative task capabilities. Each template is designed for given natural language (NL) task and has two key properties: 1) it invokes task-specific data generating functions, which specify how inputs and outputs are generated in order approximate task rule, and 2) it operates over a random token space, which not only avoids privacy and legal concerns but also allows us to express a complex rule as pattern over the tokens. While templates are task-specific, they are composed using simple list shuffling, replacement, and concatenation-aan we show that their construction can be either manually specified or automatically generated using models such as GPT-4. To more clearly understand our templates, consider a rule for document QA (Figure 1) which to identify the text in the document that pertains to the question and extract the answer from nearby. Our template for document QA constructs the document as a random sequence uniformly sampled over the token vocabulary. To instantiate the identify and extract rule, our template samples the question as a small span of the document token sequence and defines the answer the span of tokens surrounding the question.
saatttions-mmminnng span selection,
2

---

Published as a conference paper at COLM 2024
Next, to improve performance many downstream tasks simultaneously, discuss how to construct our instruction dataset as a mixture of samples generated from multiple templates. We propose a simple mixing algorithm over COOKBOOK templates, which we call COOKBOOK-MIX, Given COOKBOOK templates that generate data for several tasks, this algorithm first finetunes the base model on data generated by each template. Then, we use weak supervision (WS) methods (Ratner et al., 2019) to estimate the accuracy of each fine-tuned model on each downstream task without requiring labeled evaluation data. The proportions are set as the softmax of the estimated accuracies, and our instruction dataset generated according these proportions. Notably, these proportions are theoretically optimal under linearity assumption on the accuracies, which we empirically validate.
Empirically, COOKBOOK programmatically generates instruction data that improves LLM performance on generative tasks, unveiling an exciting finding: we can improve a model's natural language capahilities via standard superoised fine-tuning on patterned sequences of random tokens. We find that Mistral fine-tuned on BOODKK00K--MX--geeraaae
data outperforms Mis- tral and Llama 2 fine-tuned on existing instruction datasets (e.g,, OpenOrea, OpenHermes, Capybara) on average the multi-task GPT4ALL evaluation suite, and, an individual task level, is the best performing model on 3 out of 8 tasks (top-1 for win-rate). We further find that fine-tuning performance of the corresponding NL task across out of 7 downstream tasks spanninnnnnnnggnnnn document QA, retrieval, commonsense QA, entity matching, and entity disambiguation- on GPT-NEO-1.3B. In particular, COOKBOOK outperforms the base model by up to 52.7 points, confirming the efficacy of COOKBOOK at both the instruction dataset level and more fine-grained single-task level.
data generated from individual COOKBOOK templates improves
Finally, we analyze when and why COOKBOOK can improve model performance. First, we define template alignment scorers, which quantify adherence of a NL task sample to a template rule; for instance, in document QA (Figure 1) we score samples based on how close the answer and question words are in the document (i.e., the identify and extract rule). Using these scorers, we develop the template alignment statistic, which empirically confirms that the models improvement is largely due to better adherence to template rules. Second, we study the role of the template's data generating function, random tokens, and the base model to better understand when training on COOKBOOK data generalizes to NL tasks.
Related work
We present an abbreviated related work and provide a full treatment in Appendix C. Instruction datasets range from those that are manually curated (Wang et al., 2022b; Longpre et al., 2023) to those that are generated by LLMs (Taori et al., 2023; et al., 2023a; Lian et al., 2023a). Programmatically generated data is an alternative to human and LLM-generated datasets, and has been explored as a replacement for manually labeled data (Ratner et al., 2020; Zhang et al., 2022a). Programmatic data is also used in pre-training, incorporating latent structures similar to those in natural language (Wu et al., 2022; Papadimitriou & Jurafsky, 2020; Krishna et al., 2021) and for understanding models (Nanda et al., 2023; Zhang et al., 2022b). For multi-task performance, recent data mixing algorithms set proportions based on online model performance, largely for pre-training (Chen et al., 2023; Xie al., 2023).
3 COOKBOOK
We set up the problem of constructing a template for single task and mixing template- generated data for multiple tasks in Section 3.1. We then describe our framework, COOK- BOOK, of task-specific templates for generating programmatic data that can improve model abilities on generative tasks (Section 3.2). In Section 3.3, we present COOKBOOK-MIX, which mixes data from several templates to improve performance on multiple tasks.
3

---

Published as a conference paper at COLM 2024
commoneense htt.r//eenseeeeeonnnnnnooooooooo ceesoning/overlag sent, tl, c2 randon.sanple sanple ids) imputsi
int)i
natching(aoise: float)1
el = randon. wawgsettteeeeennaaaaaa
sent get and
inpute:
al el + randon. somple(sent, k=overiap_ len) c2 randon, sample(token_ ids, kroverlap__es len) choices (el, c2] sshuffie)
e2 rel.seplc) if rand. 0.5 eise (.replace(Kernoiee)
outputs: use sent and choices to get anewer idx argmax [sent n choices[0), choices onawer choices ans [dx]
outpota: and get: ansver 'yes" it el n e2 (1-noise) else "no"
instruction "Select correct choice."
instruction "Deternine whether Product and Product
returo 1
(instruction)
f
{sent]
(instruction)
Choices:
Product A1 (el)
(choices[0
Product (e2)
{choices [1])
Question: Are Product A and Product. the same? Anewer: (answer)
(anewer)
Figure Example templates (pseudocode). Templates construct the inputs, outputs, and then return formatted sample. (Left) template for commonsense reasoning which generates two answer choices, where one choice (the answer) has a greater token overlap to the sentence. (Right) template for entity matching, which generates two entities which are labeled a match if their overlap exceeds threshold.
3.1 Setup
Constructing templates (single task). We are given natural language (NL) generative task as input. Samples of T have several components. Let E I be the NL instruction for the task, x E X denote its inputs, and y E ) denote the task output. Let fmtr () be the task formatter that formats the text sample using x and y, prepending Using document QA as an example, answer, and fint((x, y, could be f"Use the document to answer the following question Document: [x1}. Question: (x2). Answer: (y)". Given this information about T, construct a template GT, a Python function that generates samples. The goal is to design the template such that the generated samples can be used to fine-tune model that does well on 7. More precisely, we fine-tune a base model f X' + y on n samples generated from GT to yield fGr,a, and our goal for fGr, perform well on T.
x2) where is the document and X2 is the question, y is the
Mixing template-generated data (multi-task). We have as input templates Gr = {Gr, GT, generated using COOKBOOK for tasks TI,..., Tj. We use these templates to generate n samples of programmatic data with mixture proportions p = [PI...,,pi] E A!, the probability simplex. Define fGn.up as the base model f fine-tuned on this mixture, with np; samples generated by each GT:- Suppose there m downstream evaluation tasks Teval [Tqual,.... Teaa) (note that they may be different from the tasks that the templates were constructed for), and let acc(fcr,mp. Tyyall be the accuracy of fGr.p Teval Our goal is to determine the that maximizes average accuracy, maximize 1 Σ acc(fGr, Tymval Tywal). PC
3.2 COOKBOOK templates
We first describe the high-level process by which template generates samples. We present examples of COOKBOOK templates in section 3.2.1, explaining how they are composed using common operators. In section 3.2.2, we explore ways to automate template creation.
Data generation process The data generation process involves (1) constructing the inputs, 2, and (2) constructing the output based &. Inputs are categorized into a parent input and child inputs; the parent input is a sequence of randomly sampled tokens, and the child inputs are constructed from the parent input. The output constructed from the parent and child inputs using a data generating function that approximates the task rule. The sample is then formatted as fint((E
3.2.1 Example templates
We present example templates from three generative task families, described below.
4

---

Published as a conference paper at COLM 2024
Selection tasks involve choosing one of the given inputs (answer choices) as output. Examples: entity disambiguation, commonsense reasoning QA, and multiple choice QA.
Search tasks require extracting from the input. Examples: document QA and retrieval.
Comparison tasks involve outputting "yes" /"nn" based on the relationship among inputs. Example: entity matching.
For each example template, we describe its inputs (parent, child) and outputs and present corresponding pseudocode. Across the tasks, identify five operators for generating inputs: random. sample(), random. www.whhhffeeee get_span() (extract a random span from sequence of tokens), replace (k) (replace tokens in a sequence with probability k), and list concatenation (+). All task templates (7 total) can be found in Appendix B.
Document QA (Search) The document QA tasks involves answering a question over provided textual context.
Template pseudocode: See "doc_QA _ee_eelltte" in Figure 1.
document is the parent input, and question is the child input. document is gener- ated as sequence of randomly sampled tokens from the token vocabulary. question is generated as subspan of document.
Inputs:
Outputs: answer is the output which is generated by locating the question in the document and returning the tokens within k indices of the location.
Commonsense Reasoning (Selection) The task of commonsense reasoning QA is to select the answer choice which best completes the sentence. For example, given sentence of "to eat more sweets should" and two choices of (1) "drink water" and (2) "eat more candy", the correct answer choice is "eat more candy".
rraaionnnnnseeeaaninnnnnn Template pseudocode: See "commonsense reasoning" in left of Figure 2.
Inputs: sent is the parent input, and c1, c2 are the child inputs. sent is generated
sequence of randomly sampled tokens from the token vocabulary. Both c1 and c2 are generated sequences of randomly sampled tokens from the token vocabulary. For c2 we append additional tokens which is a sequence of tokens sampled from sent. choices the list containing c1 and c2 which is randomly shuffled.
Outputs: answer is the index of choices which has the greatest overlap with sent.
Entity Matching (Comparison) The task of entity matching is determine whether two entities are equivalent.
Template pseudocode: See "matching" in right of Figure 2.
Inputs: The first entity e1 is the parent input, and the second entity e2 is the child sequence of randomly sampled tokens from the token vocabulary. With 50% probability, e2 is set to e1 with a small amount of tokens replaced (as determined by the noise threshold ); otherwise, e2 is generated as sequence of randomly sampled tokens from the token vocabulary of the same length as el.
input. e1 is generated
Outputs: answer is determined by the amount of overlap between e1 and e2. If the amount overlap above threshold, the output "yes". If not, is "no".
Note that across all templates, the task rules are approximated by translating them into patterns that involve token overlap; that is, outputs are constructed from inputs based on token overlap among the inputs. However, we do not claim that the token overlap and the five operators above are necessary components of a template; in section B.7 we present poetry generation task and template, which does not invoke most of these components.
3.2.2 Automating template creation
We introduce method for automatically generating COOKBOOK templates using GPT- We do so by prompting the model with a description of the data generation process (see Section 3) alongside two in-context examples mapping task description to data generating template (see Appendix E for prompt details). We evaluate the performance of GPT-NEO-1 1.3B finetuned data generated from these templates which we call
5

---

Published as a conference paper at COLM 2024
COOKBOOK-NEO-AUTO - finding that performance of COOKBOO--NEO-AUTO is within 0.2% manually curated templates (see Appendix E for details).
3.3
CoOKBOOK-MIX: mixing template-generated data
Given multiple templates and downstream tasks, we study how to set template proportions to construct training dataset that improves model performance across all the downstream tasks. First, we impose a simple linearity assumption on model accuracy and derive the op- timal data proportions p" that maximize average downstream accuracy of the COOKBOOK- tuned model. This approach requires evaluating models on labeled data, so we present an extension where we can approximate p" using unlabeled data via a latent variable model inspired by weak supervision (Ratner et al., 2019). Our full algorithm, COOKBOOK-MIX, is provided as Algorithm in Appendix D.1.
Optimal template-generater data proportions. Our objective is to maximize fGy,up" 's average downstream accuracy, maximize D acc(fGr.mp. Tyyal).
To solve this problem,
PELl
we impose a simple linear assumption: that acc(fGr.pp Tyval) E p.acc(for,, T
for all] j E [m]. That is, the accuracy of model trained a weighted mixture is equal to the weighted average of models trained on each template, which we empirically assess in
Appendix D.2. We also add an entropy term H(p) = -E== p; log with a weight 1j 2 0 to control how close to uniform p should be. Our optimization problem is now
maximize AEdi
E ==
p,aco(for.m Tgyal + HH()
(1)
m
Solving this optimization problem yields the following solution. Proposition 1. Define A E R 1m where Aij = acc(fGr, Teval), ). Let 0; exp( my -j= Aij)
for for
1 6; for all E [|].
all i E [I]. Then, the p* that maximizes (1) satisfies p E
See Appendix D.3 for the proof. The computation of p" is straightforward. We train one model per template, för,a for each E [[]. We compute the average accuracy across the m downstream tasks per model, and then compute the softmax over these accuracies to get the proportions p* to determine how many samples are needed from each template.
Extension to evaluation data without ground-truth outputs. Note that computing acc(fcT, Tyval) requires evaluating on dataset with ground-truth outputs. However, in practice datasets are not often annotated with outputs, which requires us to estimate the accuracy of the model. Since we have multiple indívidual models, we can frame them as noisy "voters" that make predictions on the true label and cast accuracy estimation as weak supervision problem, a line of work that estimates labels given an unlabeled dataset and several heuristic voters using latent variable models. We describe this extension of our approach, which uses the Me'lal weak supervision algorithm (Ratner et al., 2019) in Appendix D.1.
3.4 Training on CookBoOk-data
All models trained on CooKBOOK-template data were finetuned by running hyper- parameter search, sweeping across learning rate ([4e - 06, 5e - 06, Se - 06, 5e 05, 8e 05]), batch size (18, 16, 32, 64)) and total training steps (1100, 200, 300, 400 and 500]). In finetuning CookBoOk-models no training samples were repeated.
4 Experimental evaluations
We evaluate the empirical performance of COOKBOOK in multi-task evaluation setting (i.e., against standard instruction datasets on many downstream tasks) and in single-task evaluation setting on NL tasks that our COOKBOOK templates correspond to. In Appendix F, we show that COOKBOOK can be extended to creative tasks (i.e., poetry generation) going beyond select, search, and compare tasks.
6

---

Published as a conference paper at COLM 2024
Model
arc_c
boolg
hellaswag
lambada
openbookqa
winogrande
average
LLAMA-2-7B
46.25
74.58
77.74
73.99
73.92
44.20
79.11
69.14
67.61
LLAMA-2-7BNNH
49.74
76.09
77.72
72.99
79.76
70.01
69.05
MISTRAL-7B
54.10
79.50
83.49
51.12
75.59
44.40
82.05
73.88
71.76
MISTRAI-7B-ORCA
72.37
73.24
72.28
MISTRAL-7B-OH
59.98
81.65
86.73
81.77
73.90
44.20
82.70
73.56
73.06
COOXBOOK-LLAMA
48.04
76,77
79.20
76.04
77.10
43.40
78.56
69,30
68.55
COOKBOOK-MIST
57.76
Table 1: Performance on GPT4ALL benchmark. We denote our CoOKBOOK-MIX-tume MISTRAL-7B model as COOKBOOK-MIST Averaged across tasks, COOKBOOK-MIST has the best accuracy. For baseline datasets, "NH" denotes NousHermes, "OH" is Open-Hermes, and "ORCA" is OpenOrca.
4.1 Multi-task Evaluation
We evaluate if the COOKBOOK framework can improve multiple generative capabilities at once. Here, we compare COOKBOOK-MIJ to SoTA instruction datasets (e.g,, OpenOrca, OpenHermes).
Our method. We fine-tune two pre-trained base models, LLAMA-2-7B (Touvron et al., 2023) and MISTRAL-7B (fiang et al., 2023). The set of templates consider is: MATCHING, ENTITY- DISAMBIGUATION, MULTI-CHOICE-QQ A, and COMMONSENSE-SELEES for their constructions). We fine-tune each base model these templates using the data proportions obtained using COOKBOOK-MIX as described in Section 3.3.
(refer to Appendix
Evaluation datasets. We evaluate on the standard GPT4ALL (NomicAl) benchmark consid- ered by many models (e.g., Mamba (Gu & Dao, 2023), Cerebras-GPTT (Dey et al., 2023)). We report average zero-shot accuracy for all tasks using the standard EleutherAl LM Evaluation Harness library (LM eval) (Gao et al., 2023). The LM eval harness provides fixed set of prompt formats for all tasks, generating accuracy and standard deviation metrics.
Baselines. As baselines we consider the base LLAMA-2-7B and MISTRAL-7B models. Additionally, we compare against the current SoTA open-source instruction-tuned versions of these models (OpenHermes--isttree et al., 2023b)) and the closed source versions (Nous-Llama-24) (Research, 2023b), Nous- Capybara-7B (Research, 2023a)).
(Teknium, 2023), Distral-7B-OpenOOe (Lian
Results. Results for our multi-task evaluations can be found Table 1, with full results in Table 5. Averaging across tasks we find that (1) COOKBOOK-MIL improves performance across model variants-Llanm CooKBOOK-MIx (which abbreviate COOKBOOK-MISTH on the benchmark suite and also outperforms other mixtures of CоOKBOOK templates, such uniform mixture (Table 6), and (3) COOKBOOK-MISTRA. is the best performing model on 3/8 tasks.
and Mistral (see Table 1), (2) MISTRAL-7B fine-tuned with is the best fine-tuned model
4.2 Single-task Evaluation
This section evaluates single-task performance of task-specific COOKBOOK templates, offer- ing more fine-grained study of the templates themselves without mixing.
Dataset
NEO-BASE
NEO-FEW
COOKBOOK-NEO
TYDIQA
21.8 0.49
14.0
41.9
SQUAD
14.5 4
50.4
60.5 : 0.68
PIQA
0.0
48.5
52.7 : 0.43
MS_MAN MS_MARCO
12.6 1.08
1..7
18.6 + 0.16
WINOGRANDE
3.9
64.6
54.3 0.87
BEER
26.7
26.7
66.6
ITUNES-AMAZON AMAZON
39.7 0.0
63.1
69.6 = 0.0
Table 2: Single-task evaluations. We report accuracy for all datasets except BEER and ITUNES- AMAZON (Fl-score). CookBook-tuned models exhibit improved performance over the base model.
7

---

Published as a conference paper at COLM 2024
Our method. For each evaluation task, we generate data from the associated COOKBOOK template (see Table for the mapping between task and template). We fine-tune two base pre-trained models, GPT-NEo-1.3B (Gao et al., 2020) and CEREBRAS-GPT-L.3 (Dey et al., 2023), using data generated from the template. In our data generating process, a new batch is sampled at each step. Across tasks, fine-tune models over average of 4.2K datapoints. We evaluate the fine-tuned model's zero-shot performance.
Evaluation datasets. For our evaluations, we consider 7 tasks across selection (PIQA, WINOGRANDE), search (TYDIQA, SQUAD, MS MS_MARCO) and comparison (BEER, ITUNES- AMAZON) categories. For all tasks excluding BEER and ITUNES-AMAZON for which we report Fl-score, we report accuracy. Table 7 in Appendix G.5.2 contains dataset statistics.
Baselines. We compare our CoOKBOOK-tunee models to the base model via two baselines: zero-shot and few-shot (where k=3) using natural language task samples as in-context examples. Zero-shot involves directly evaluating the base model's ability on given task, while few-shot primes the base model with task-specific formatting and context.
Results. Table 2 shows the results for GPT-NEo-1.3B. We defer results for CEREBRAS- GPT-1.3B to Table 9 in Appendix G.5.2. Our results show that (1) COOKBOOK-NFO out- performs the shro and ICL performance on 6/7 tasks, (2) performance of COOKBOOK models can be 52.7 points better than base zero-shot performance, suggesting that COOK- -ook-generated data can help the model learn a task it was previously unable to do, (3) similar trends can be seen across model families (see Appendix G.5.2).
5 Analysis of COOKBOOK
First, we propose and measure the template alignment statistic on several tasks to validate that the rules taught by templates are responsible for improved task performance. Then, we analyze the key components of COOKBOOK to better understand its effectiveness, finding that 1) the extent of NL knowledge the base model has impacts COOKBOOK's performance; 2) training on data from one COOKBOOK template can improve on multiple tasks; 3) rules, not only the instruction formatting of samples, are necessary to the performance of COOKBOOK.
5.1 Template Alignment Framework
We introduce the template alignment scorer, which measures how similar a NL sample and template are. We use this to define the template alignment statistic, which measures how better adherence to the template's 's rule is responsible for improved performance on the task.
Template Alignment Scorer Given task T and its corresponding template GT con- structed in Section 3, we propose a template alignment scorer X x ) -> [0, 1]. The template-specific scorer captures how well the input sample's relationship between x and y follows that of Gy's 's data generating functions. We provide three examples below:
Document QA: the template selects random span of document as the question and defines the answer as the surrounding span. SG, scores a sample by finding where the answer located in the document and computing the fraction of words in the question that are near the answer (Fig 1). Note that template-generate samples have score of 1.
Commonsense Reasoning:
the template generates two answer choices, with only one choice containing tokens from the sentence. SG, scores sample by finding the words that are unique to the two choices, measuring the minimum normalized embedding distance between each choice's unique words and the words in the sentence, and computing the absolute value of the difference between the two choices' distances to the input sentence. This quantity is large if one answer choice has much higher word similarity to the input sentence than the other. Note that when we use the hamming distance, semplate-ggneratee samples have a score of 1.
the template first generates one entity randomly, and then generates the second entity to have some overlap with the first one. SG, scores sample by computing the word overlap between the two entities. Note that cemplate-generated samples have score equal to 1 noise when the entities are equivalent.
Entity Matching:
8

---

Published as a conference paper at COLM 2024
Template Alignment Statistic Next, we use the alignment scorer to relate model per-
formance to template rule adherence. Recall the base model f X + y and the COOK- BOOK-tuned model fc,.-- Given NL. task samples Dr, let Dj (Gr) = {xx,)) E Dr f(x) + y, fo = 1/} be the set of samples that f gets incorrect and fGr, gets correct, and let Pr (GT) = yx E Dr: f(x) # y, f.) & y) be the set of samples that both f and fGr, get incorrect. We now define the template alignment statistic, which characterizes how different the template alignment scores are on samples that the COOKBOOK-tune -tuned model gets correct versus incorrect.
xxx,y) E DF and ST = (x, yx, E Dy (GT)} be the alignment scores over Df (GT) and D7 (GT), respectively. Define F+,G [0, - [0, as the empirical CDF over S (GT) and similarly define FT,G, Then, the template alignment statistic for task T, template GT is
Definition 1. Let ST (GT) = (x,
A(T,Gr = -sup ,,,,,, (9) F,,, (s)).
(2)
sE(0,1]
Note that (T, is the total variation distance between the empirical distributions of the template alignment scores for D (GT) and DF (GT), as well as the Kolmogorov-Smiing test statistic for the two-sample hypothesis test on whether the scores for these two subsets of DD come from the same distribution. This statistic is thus bounded between and 1. A large value for this statistic suggests that there is significant difference between improved and non-improved samples in terms of how they adhere to template rules.
Template Gr / Task 7 PIQA/ / COMMONSENSB-SELECT
TEMPLATE-MISTRAL A(T, (0.867, 7.1 x 46) (1.0, 0.0025)
DBLP-ACM MATCHING
TYDIQA DOCUMENT-QA
(0.615, 0.013)
Table 3: Template alignment statistics. Template alignment statistics for COOKBOOK-tung -ttuned models.
Values are the (template alignment statistic, p-value). CooKBooK-tunee models have learned the rule captured by the synthetic.
Results. We compute the template alignment statistic for the MIsTRAL-7B+template models the tasks TYDIQA, PIQA and DBLP-ACM using the DOCUMENT-QA, COMMONSENSE: SELECT and MATCHING templates respectively. Note that for PIQA, we use Sentence-BERT embeddings (Reimers & Gurevych, 2019) to compute embedding distance between the words in the answer choices and the words in the goal. We find that the template alignment statistic for each task (Table 3) is statistically significant (< 0.05), suggesting that rules taught by the templates are responsible for improvement on downstream NL tasks.
5.2 Understanding CоOKBOOK random tokens and rules
When do random tokens work? We hypothesize that fine-tuning on random tokens helps learn NL task when the model already has sufficient NL capabilities. To test this, we evaluate COOKBOOK on PYTHIА-1B checkpoints (log scale from 0 to 144K) as the base models, where later model checkpoints have more sufficient NL capabilities. We do this on PIQA, ITUNES--AAZOOOO there exists checkpoint at which COOKBOOK starts to help performance significantly, before which it has little effect (see Figure 4 in Appendix H). This suggests that the generalization from random tokens to NL is dependent on the base model's NL capabilities.
SQUAD, WINOGRANDE, and their respective templates, finding that
Do rules taught over random tokens result in less overfitting? We compare the overfitting of rules taught over random tokens versus rules taught over NL tokens. We fine-tune models on the COOKBOOK MATCHING template and on the NL matching task itself (ITUNES- AMAZON), and evaluate them on ITUNES-AMAZON and two other tasks, PIQA and TYDIQA (see Table 11). We observe that COOKBOOK-tuning actually improves base model perfor- mance across tasks beyond matching (up to 5 points), whereas NL-tuning worsens base performance (by up to 7 points) for all tasks outside of matching. This suggests that skills

---

Published as a conference paper at COLM 2024
taught in the random token space help generalize to multiple tasks and overfit less, offering a potential explanation for how COOKBOOK-M MIX outperforms standard instruction datasets.
Do we need data generating functions: Are random tokens all need? We empirically inspect the importance of the template rules vs. the task format (fmty) in improving downstream performance by replacing our rule-based generated input and outputs in fmtr() () with random tokens without any structured patterns. Our results show that finetuning on data without rules leads to worsened performance accuracy points (see Table 12 in Appendix H).
average performance drop of 18.3
Discussion
In this work, tackle the challenge of building instruction datasets for improving genera- tive abilities. Via COOKBOOK, we show that it is possible to programmatically generate data that teaches task-specific rules. Moreover, we show how such programmatic data be mixed to improve performance on many tasks at once. Finally, we propose method for measuring whether model has indeed learned task rule, and whether learning that rule improves downstream task performance. In future work, we seek to further explore how to use programmatic data generation to enable self-improving systems.
7 Acknowledgements
We thank Michael Zhang, Neel Guha, Michael Wornow, Ben Spector, Silas Alberti, Jordan Juravsky, Eric Nguyen, and Alyssa Unell for their feedback and discussion.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 Wardwaree-Aarrd CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2- 0184 (Long-context) and W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under Nos. N000142312633 (Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), Knight-Hennessy Scholarship, NSF Graduate Research Fellowship and members of the Stanford DAWN project: Meta, Google, and VMWare. The U.S. Government is authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
10

---

Published as a conference paper at COLM 2024
References
Winogrande: An adversarial winograd schema challenge at scale. 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xrv preprint ar arXii:2303.08774, 2023.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training, 2023.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall efficient language models. Xiv e-prints, pp. arXiv-2312, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arxiv preprint ar axXiv:2204.05862, 2022.
Kush Bhatia, Avanika Narayan, Christopher De Sa, and Christopher Ré. Tart: A plug-and- play transformer module for task-agnostic reasoning. arXiv preprint ar Xii:23306007536 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Arlificial Intelligence, 2020.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, pp. 2, 2023.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 36000-36040, Cur- ran Associates, Inc., 2023. URL
cttpsiipppccpppppeeeeeeeeeeeeeee neurips cc/ cctccc/cpppcoocccc paper_ www.fflllpppppee /paper/ pdf.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXio preprini ar Xrio:22100.14116 2022.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/ no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. #rXie:1803.05457ll 2018.
Luigi Daniele and Suphavadeeprasit
Synthetically generated diverse multi-turn conversations for effecient Ilm training. arXiv preprint ar arxiiiccomnng soon), 2023.
URL https::/huggiigfface co/ datasatsttLDmmnm LDDDr//apybbar..
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained the cerebras wafer-scale cluster. ar Xiv preprint ari02:30440002 2023.
11

---

Published as a conference paper at COLM 2024
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference Learning Representations, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of 2020.
diverse text for language modeling, ar Xrv preprint ar Xrx::2210100000
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo. org/records///// http///oordd//25655553 10256836.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint xr1o::231.00075,, 2023.
Michael Hahn. Theoretical Limitations of Self-Attention in Neural Sequence Models. Trans- actions of the Association for Computattonnll Linguistics, 8:156-171, 01 2020. ISSN 2307-387X. doi: 10.1162 //cl 00306. URL http::://oi...ooo wwwpoi..rrr..... 1e111/1accaaa ataprr//o...///aaaaaaaa_ a_00006.
Marti A. Hearst. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference Computational Linguistics, 1992. URL
https://aclantholl.e wrg/C92-2000 2082.
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arviv preprint ar Xri::2310.06825, 2023.
Kundan Krishna, Jeffrey P Bigham, and Zachary C Lipton. Does pretraining for summariza- tion require knowledge transfer? In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3178-3189, 2021.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023a.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2023b.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:: /ttgss////gggggaa//ee So/Open--Orca/penorree 2023a.
Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Mistralorca: Mistral-7b model instruct-tuned on filtered openorcavl gpt-4 2023b.
dataset. https: http::::/hhgginnaaaaee cot cor//pee--rrrr//issraaa--------
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. Xiv preprint Xrx:::2301 13688, 2023.
12

---

Published as a conference paper at COLM 2024
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arviv preprint (iXi::23060086663 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Represenlations, 2023. URL httt::://operevvei// Coperreview.nett/ooummiiii net/forum?id= 9XFShDPmdW.
Nihal V. Nayak, Yiyang Nan, Avi Trost, and Stephen H. Bach. Learning to generate instruc- tion tuning datasets for zero-shot task adaptation, 2024.
NomicAL Gpt4all performance benchmark. https: https://gpt4alle 10/ intex.ddxxx.... html.
Isabel Papadimitriou and Dan Jurafsky. Learning Music Helps You Read: Using transfer to study linguistic structure in language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6829-6833 Online, November 2020. Association for Computational Linguistics. doi: 10.18653 /v1 10200eemll-maiin aclanthology. org 2rg/0020.. emnlp-main.555 554.
URL https://
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolu- tional language models. ar Xiv preprint ar Xiv:22302. 10866, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Tlya Sutskever. Language models are unsupervised multitask learners. 2019.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly, 2016
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty- First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Sym- posium Educational Advances in Artificial Intelligence,
MAAT11//IAATIS SES/19/EAAA. EAAII1. AAAI Press, 2019. ISBN 978-1-57735-809 -09-9. doi: 10.1609 a0000//aaa.v3301.33330 https://doi, org/10. /609/ aaai. v33101 33014763.
URL
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal, 29 (2-3):709-730, 2020.
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3532-3542, Minneapolis, Minnesota, June 2019. Association for URL https: / aclanthology.
Computational Linguistics. doi: 00.18633vv1/N19
org/N19-1356.
13

---

Published as a conference paper at COLM 2024
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-
IJCNLP). Association for Computational Linguistics, 2019. doi: 00.1653/vv1dd11-4411.
URL pottt://ddx/ wtt..r////...11////////./ 1410.
Nous Research. Nous-capybara77--1119 2023a.
URL https://huggingface.co
Bangs/eeearrth/ouu-----/aaarrr-- Hugging Face Model.
Nous Research. Onoosseeearcch-laaa-22775 2023b. URL https://huggingfacee
NousReearrchLLLama-222h5 Hugging Face Model.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing 2023.
Ilama model. https://gl thub. httub.ccm//ttttttt
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially Accessed: 2023-05-05.
usable Ilms, 2023. URL www. www.mooaicml..ommbblo/////o
"Teknium". Openhermes. https///wwgggggeeeeee//aaaaasaae
2023.
Teknium. 0penhermes22.5mmsttal--l 2023. URL https://hugginnffccc..o/etiiiii OpenHermes- 5-Mistral-78 2attiiernnes--.MMstaaallle-
Hugging Face Model.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXiv:2307.092883 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, Generalization via declarative instructions
David Stap, et al. Superrnaturaainstruuction 1600+ nlp tasks. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022b.
Yuhuai Wu, Felix Li, and Percy S Liang. Insights into pre-training via simpler synthetic tasks. Advances Neural Information Processing Systems, 35:21844-21857, 2022.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining, 2023.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision, 2021.
Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey programmatic weak supervision, 2022a.
Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: synthetic reasoning task, 2022b.
14

---

Published as a conference paper at COLM 2024
Appendix
We provide an overview of the sections of the Appendix:
Appendix A: we provide a glossary of the variables and symbols used in this paper.
Appendix B: we present templates corresponding generative tasks as well as sample data point generated from each template.
Appendix C: we provide a full related work.
Appendix D: we provide more details on COOKBOOK-MIX, including the formal algo- rithm, experiments to verify the linearity assumption, and proof of Proposition 1.
Appendix E: we explain how to use GPT-4 to automatically generate COOKBOOK tem- plates and evaluate these generated templates.
Appendix F: demonstrate how the COOKBOOK framework can be extended more creative poetry generation task.
Appendix G: we provide additional details on the experiments Section 4.
Appendix H: we provide additional experiments and details for understanding COOK- BOOK.
A Notation
The glossary is given in Table 4 below.
Symbol
Used for
Natural language (NL) task that we aim construct template for improving. Natural language instruction E task
x
Inputs x e X for task T. Outputy y for task T.
fmt()
Function that formats instruction, inputs and output for task T into fmt((x, y, Data generating template corresponding to task T.
Gr
f
Base model f : > y whose generative capabilities we aim to improve. Number of samples generated from templates used for fine-tuning, f- Base model fine-tuned A set of templates G [Gr, Gr, used input for the data mixing multi-task evaluation setting.
n
fora
samples generated by template G7.
GT
Mixture proportions p {PI,..., P1) / over data generated from GT- Base model f fine-tuned и samples, with пp; samples from each GT,- A set of downstream tasks Teval {Teval ...., Teval) aim to improve performance on.
p
Sorap
ace(formp Toval)
The accuracy of fGrmp on Tyyvll
Weight //
entropy term H(p) in (1) controlling how close uniform p should be.
SGr
Template alignment scorer SG, X' x y [0, 1] that measures how much a NL sample from I acheres to the rule of template Gy.
A(T,Gr)
Template alignmment statistic for task T, template GT which measures the total variation distance in template alignment scores for samples of that the model fine- tunettneddde on data from GT gets correct versus incorrect (see def. 1).
Table 4: Glossary of variables and symbols used in this paper.
15

---

Published as a conference paper at COLM 2024
B Templates
We present seven templates (MATCHING, MULTI-CHOICE-QA, DOCUMENT-QA, ENTITY- DISAMBIGUATION, COMMONSENSE-SELLCT,,, TOKEN-RETRIEVAL, and POETRY GENERATION) and sample data point generated from each.
B.1 Matching
Data generating template:
def matching (noise:: int):
import random
e1 random. (ooplentoke__dde k=1)
# inputs: use e1 to get e2
e2 el replace (k=1) if random.. random() 0.5 else el replace (k=noise
# outputs: use e1 and e2 to get output answer 'yes" if (ee( set()2)) (1 noise) len(e1) else 'no'
instruction "Determine whether product and product are the same An"
return **""
(instruction) Product A: [e1}
Product B; [e2]
Question: Are Product and Product B the same?
Answer: (answer)"**
Sample data point:
Determine whether product A and product B are the same Product A: occur Competing Tobuiltembean Hollywood met
Product B: occur Competing TObuiltembean Hollywood met
Question: Are Product and Product the same?
Answer: yes
B.2 Multi-Choice QA
Data generating template:
def multi_choice (overlap_len: int): question random. sample (token_ids)
(c1, c2, c3, c4, c5) random. sample (token_ids, k=5)
inputs: use question to get correct choice c5 c5 question. sample (k=overlap_len) + c5[: ooverllleepppee
outputs: use question, [c] choices [c1, c2, c3, c4, c5]. shuffle()
c5]) get the answer
ans_idx = argmax ([question \epp choices[e],... <> [4]])
question \eap choices
answer choices[[ns_iddk
instruction "Answer the question. An" return f**n [instruction]
Question: [question]
16

---

Published as a conference paper at COLM 2024
Choices:
{choices [0]}
{choices [1]}
[choices [2]}
[[choices [3]]
[choices [4]]
Answer: (answer)"M*
Sample data point:
Answer the question.
Why lake Shares wildly Gandhi rers ademic AES 1995 ports?
Choices:
poke installment
ORS> unmanned Slave ellar Mart English
visions AES wildly lakeexper Gamer Gate ports ademicE
Haibeh Dom
aiming 462 adultery Greenberg collar
Answer: visions AES wildly lakeexper Gamer Gate ports ademicE
B.3 Document QA
Data generating template:
def (emmllatemminsseeee int, max_slen: int):
document random.. sample ( token_ids) #input source
inputs: document get question span_length random... choice question document. get_span (k=span_ get indexes
choice (min_slen, max_slen)
outputs: document and question to get the answer document (document. intersect (question)) loc (document.. < intersect (questioon))))))) 3]
instruction "Use the document to the question. An"
return ("((instruction)
Document: (document)
Question: [question)
Answer: [answer)"
Sample data point:
Use the document answer the following question. Document: Destiny Ricardo Bundle swarm trips spilling crews trips Python disliked absorption phon Fallen Mour Wales parameter
Question: trips spilling
Answer: estiny Ricardo Bundle swarm trips spilling crews trips Python
B.4 Entity Disambiguation
Data generating template:
17

---

Published as a conference paper at COLM 2024
def sentence_1, sentence_2 random. sample (token_ids)
(e__spanni int):
# inputs: sentence_1 to set sentence_2. el and e2 sl, sentence_ get_ (kae__sa___eeen el, s1[0], s2[0]
if random.. (and 0.5 sentence_2 +=[ <blank>'] s1[1:])
else:
sentence_2 2 +=[ <blank>> s2[1:])
outputs: use (question, sentence_1, sentence_2) get answer ("hia(<<laakkk support Loc(sentence_1. intersect (support) - 1]
span_loc answer = sentence_ 1sstance_ 1]
instruction "Select the choice which best completes the <BLANK >. An" return f f"*" (instructionn)
Choices:
-{e1)
[e2]
Answer: (answer)""*
Sample data point:
Select the phrase which best fills in the <BLANK>
Sentence: ranc Islands solid illustrates horsepower furry Pay early Santiago +)scan output. ographies deals privatization <BLANK> Santiago *csc output.
Choices:
Islands
early
Answer: early
B.5 Commonsense Selection
Data generating template:
def commonsense (overlapp_ee len: int): question random.. sample (token_ids) c1 = random. sample (token_ids) c2 random. sample
inputs: use question to set and c2
c1 = c1 question. sample (k=overlap_len) c2 = c2 random. waw.pestteealeneteeiea (token_ k=overlap_len) choices [c1, c2]. shuffle()
# outputs: use (question, [cl, ans_idx argmax ([ (question. intersect (chocccss0))))))))) len (question.
c5]) to get the answer
<> intersect (choices answer choices[ans_idke
instruction "Answer the question, " return f**n [instruction]
Question: [question]
18

---

Published as a conference paper at COLM 2024
Choices:
{choices [0]}
[choices [1]}
Answer: {answer)""
Sample datapoint:
Select the choice which best completes the sentence. midst Gloss Quant Nest engaging Soul Customer
Choices:
Zack extraordinarily willingly Gene Quant engaging rest
Zack extraordinarily willingly Gene With No DXwaters shone
Answer: Zack extraordinarily willingly Gene Quant engaging rest
B.6 Token Retrieval
Data generating template:
def token_retrieval (overlap_ int):
docs []
in range (10):
docs. append (random.. sample(token)) (token_id)
# inputs: docs to set question
idx random. randint 10) randomly sample an index from 0...9 doc_with_answer docs [idx] question doc_with_answer sample
#outputs: use (question, docs) get the answer answer_idk argmax [question intersect ..... question. intersect - docs[9])
answer = docs [answer_idx]
instruction "Use the documents to answer the question, An" return f**"
(instruction)
[docs[o]) [docs[1])
(docs [9]} Question: (question) Answer: (answer)""*
Sample datapoint:
Use the documents to answer the question. Document 0: Def culture Luc writers takingo adjustment
Document saturally bites song house speak
Document 2: BEATE level arbitrator Layer eminent substant
Document 3: spend conesz identification unincorporateddemand
Document 4: incident existent Back Fellow Turk peacea Father
Document 5: citations Cove solel Nick cards removing comprise
Document 6: quilt sun instructed facil enacted council confess
Document 7: cs probably 59 specify Page harris famine pumps
Document equal Originally quick Adjust hearted OCK beat
Document 9: Base Sept 168 <</resh59MMr tang moss mash pain
Question: 593Mr tang moss mash pain Dir
Answer: Base Sept 168 *</resh593Mr tang moss mash pain
19

---

Published as a conference paper at COLM 2024
B.7 Poetry Generation
??) Data generating template:
def Thyme_scheme (rhyme_dict: dict):
# inputs
topic random. sample (token_idss k=1)
rhyme_scheme__,, rhyme_scheme_ rhyme_dict. ettpeeeeittsammpeeeeee.. (k=2)
inputs: use (topic, rhyme_scheme_A. rhyme_ scheme_B) get lines lines []
for idx in range (5)
r_word rhyme_ scheme_L_. sample() if idx & 2 0 else thyme_scheme___
sample()
line = random.. (tmkee(tokee_ids) line insert (topic) r_word lines append
line =
instruction "Write a five line poem with an ABABA rhyme scheme about"
return f"n"
(instruction) (topic) (lines[0]]
(lines[1]] [1])
(lines [4])""
Sample datapoint:
Write a five line poem with a ABABA rhyme structure about abi abi abilities Ches 76 Purabul befriend abi agre Chung Rapt unfit hate redditt item Struabi pre transcend
Pedro Rescueabi 1080 vines296speed ledet
continents anticip EMP texts sigabi trend
C Complete Related Works
There is large body of work which studies instruction-tuning dataset creation; program- matic data for approximating rules, pre-training, and understanding: and data mixing. Here review some of them which relate most closely with our work.
Instruction tuning Instruction tuning datasets seek to improve the ability of LLMs to follow instructions (i.e., "Generate summary of the following news article"). These datasets consist of input and output pairs, where the input is composed of an instruction with some context and the output is the "gold" generation. Early instruction tuning datasets utilized manual data curation to construct these <instruction, input, output> triples for tasks such as summarization (Bai et al., 2022). In effort to scale the instruction tuning dataset curation process, new datasets emerged which prepend natural language instructions to standard NLP tasks where the input-output pairs previously exist, such as Supernatural Instructions (Mishra et al., 2022; Wang et al., 2022b), the Flan Collection (Longpre et al., 2023), and in Chung al. (2022). More recently, instruction tuning datasets have emerged which use existing LLMs (e.g,, GPT-4 Achiam et al. (2023)) to generate the data, such as OpenOrca, Alpaca, Wizardcoder, Camel, Amplify-instruct Self-instruct, and GLAN (Lian et al., 2023a; Taori et al., 2023; Luo et al., 2023; Li et al., 2023a; Daniele & Suphavadeeprasit, 2023; Wang et al., 2022a; Li et al., 2024). In this approach, instructions are fed to the LLM (i.e., "Generate list of 5 animals") and the model generates output. While this approach sidesteps manual data curation efforts, it suffer from privacy and terms of service violations. COOKBOOK circumvents the need for both manual model-gennrated data. Some recent
20

---

Published as a conference paper at COLM 2024
approaches also train their own models to generate instruction data, such as instruction backtranslation and Bonito (Li et al., 2023b; Nayak et al., 2024). While these can avoid legal issues, it can still be computationally expensive to use LLM to generate a large corpus of instruction data. In contrast, COOKBOOK's templates generate data without the inference costs of using an LLM.
Programmatic data for approximating rules: Programmatic data generation has been primarily studied for the automated labeling of datasets (Ratner et al., 2020; 2016; Hearst, 1992), where heuristics are used to approximate the true labeling rule. For example, a heuristic for Youtube spam comment classification is check the word "subscribe" is in the comment (Zhang al., 2021). These works provide programmatic labels for classification tasks given an unlabeled dataset containing inputs only, and they find that training on data with these programmatic labels can produce models that do well on the given task. However, these approaches do not directly extend our setting, which requires us construct entire samples for generative tasks-bott programmatic inputs and outputs, which are often open-ended generations (e.g. document QA).
Programmatic data for improved training Prior work has shown the value of synthetic, token-level tasks such as set operations as an alternate to natural language data for LLM pre-training (Wu et al., 2022). Similarly, other work has demonstrated the efficacy of data with latent structure (e.g,, music) in the pretraining phrase (Papadimitriou & Jurafsky, 2020) well as for tasks such summarization (Krishna et al., 2021). These works are similar to COOKBOOK in that they utilize non-language based inputs to improve LLM perfor- mance, but unlike COOKBOOK, which directly evaluates the model trained on programmatic data, they all require some fine-tuning on natural language data afterwards. Additionally, programmatic data has be shown improve classification abilities by using it to train synthetic reasoning module and compose it with a base LLM (Bhatia et al., 2023); however, this approach does not apply to generative tasks.
Programmatic data for LLM understanding Recent works have explore the use of synthetic tasks for understanding the underlying mechanisms of LLMs (Bricken al., 2023; Nanda et al., 2022; Ravfogel et al., 2019; Zhang et al., 2022b) as well for understanding and improving architectures (Fu et al., 2022; Arora et al., 2023; Gu Dao, 2023; Poli et al., 2023). Unlike these prior works which seek to use non-language based tasks to understand LLMs, we use such
to improve LLM capabilities.
Data mixing An important step in curating training data is to determine the mixture over several given groups of data (e-g., domains such as CommonCrawl, ArXiv, Wikipedia, and Github). Data sampling proportions are key factor in model performance, with many widely-used LLMs having seemingly arbitrary mixture proportions (Touvron et al., 2023; Team, 2023). Since expensive use trial-and-error to explore the search space of mixture proportions, some recent works have proposed algorithms for data mixing, where proportions are learned based on model performance throughout training (Xie et al., 2023; Albalak et al., 2023). Data mixing scaling laws have also been proposed (Ye et al., 2024). These approaches primarily focus on the pre-training setting and only hold when the training groups of data are the same as the evaluation groups of data, which is not true in our setting of instruction datasets. Skill-It (Chen et al., 2023) proposes an online algorithm for setting proportions when the training tasks are not equal to the evaluation tasks; we note that CookBooK-MIX can be considered a non-online instantiation of their approach.
D COOKBOOK-MIX additional details
First, we describe the full algorithm for mixing data from templates, with or without labeled evaluation data. Then, we demonstrate that the linear assumption on accuracies empirically holds. Finally, we provide a proof of Proposition 1, the optimal expression for p*.
D.1 COOKBOOK-MIX Template Mixing Algorithm
Algorithm presents our approach for how to compute template data proportions. It computes 9 according to Proposition 1; namely, CookRook-tuned models one per
21

---

Published as a conference paper at COLM 2024
Algorithm 1 COOKBOOK-MIX.
1: Input: / templates GT, base model f, mi downstream task datasets Deval Deval ...., p Deval, number of training samples, y entropy parameter
2: Fine-tune f on и samples generated from GT, to get CookBOoK-tuned model for, for
all templates Gr, CT-
3: if Deval has fround-truth outputs then
4: Evaluate each model fGr,a on Deval to get A.j acc(f6r, Teval) for all [m], i e
[1].
5: else
Set Âg ESTIMATEACCS(fG, .r.... fGr, Deval), weak supervision-based
6:
method for estimating accuracy without ground-truth outputs, for all j E [m], E []].
7: end if
8: Compute 0; = exp my - Ai) for all i E [1]
1
9: Calculate sampling proportion vector p. where p;
a for all [T]-
10: return p. f is then fine-tuned on samples from templates for with proportions p.
Algorithm 2 ESTIMATEACCS.
1: Input: CookBBOK-tuned models ("voters") A; := for, : X y for E []], dataset
without ground-truth outputs, unlabeled evaluation dataset D {x²)}=1-
2: Get predictiinns (A1 (x))) across CookBook-tuned models for each xj E DD
[1], forming votes dataset D, E ynxl_
3: Estimate accuracy â Pr(A; = y) for all i E [4] by using Algorithm from Ratner
et al. (2019) noisy votes Dr.
4: return 4
template GT; -re trained, and then evaluated on all m downstream tasks. Then, the average accuracy across tasks for each CooKBOOK-tuned model computed, and softmax (with temperature 1 1/1 depending on the entropy regularization in (1)) is performed over these average accuracies get p.
When evaluating each COOKBOOK- model, computing accuracy on each downstream task is straightforward ground-truth outputs are available for the task. However, they are unavailable but the outputs are discrete (e.g,, answer choices in PIQA or "yes" AMAZON,, we can use techniques from weak supervision to estimate
or "no" accuracies (line 6 of Algorithm 1). Weak supervision involves producing programmatically labeled data by modeling several weaker "voters" using a latent variable probabilistic graphical model over their votes. In particular, weak supervision algorithms fit the latent variable model using predictions from the voters, estimating the voter accuracies. Then, they aggregate the votes based on the estimated accuracies to get a programmatic label. We utilize a popular weak supervision algorithm from Ratner et al. (2019) to get an estimated accuracy for each COOKBOOK-turn -uneed model in Algorithm 2. In particular, we compute votes dataset D E ymxl by using each COOKBOOK -uuned model to produce predictions on the downstream dataset. This votes dataset is converted into numbers (e.g., "yes" becomes 1 and "no" becomes 1) and is passed in as input to Algorithm of Ratner et al. (2019). This algorithm uses conditional independence assumption, which implies particular sparsity pattern in the inverse covariance matrix of D,, to create a system of equations that is solved with SGD yield estimated accuracies.
D.2 Assessing the Linearity Assumption Empirically
To see if the linearity assumption applies on real data, we consider two templates Gr (GT, G7 and compare the performance of individual CookRook-tuned models,
22

---

Published as a conference paper at COLM 2024
Namber template pairs accttiGr, Gr,)) [acotd(Gr)), ace(((G; Mixture lacet& Gr,)) avglaso(8G; a
Number of tampleet gains tont 6)
Number evaludioo tades
25
30
25
3
20
15-
10-
are challenge
anc may
hoolq
hellsnwag
lambadaa opunal
pipa
winegrande
15 20
25
Dowastream lask
Figure 3: Evaluating if the linearity assumption for data proportions holds empirically. Left: measure the interpolation property, how often the mixture model has accuracy in between the individual COOKBOOK-tung
models' accuracies. Right: we measure the mixture model deviation, the absolute difference between the mixture model's accuracy and the average of the individual COOKBOOK-tune K-tuned models' accuracies. Measurements are made across 6 pairs of templates (over the 4 templates used in Section 4.1), GPT4ALL evaluation
evaluattnol tasks, and the MISTRAL-7B base model.
for, and for.. versus model fine-tuned on uniform mixture over the two templates, fG/.n(1/21/2), which we call the mixture model. We re-use the setting described Section 4.1 (4 templates, 8 tasks from GPT4ALL, fine-tuning the MISTRAL-7B model). We measure two quantities, described below.
first, we examine how often the mixed model's ac- curacy interpolates between the individual CooKBOOK-tuned model accuracies, acc(fcqm(1/21/z)1T Teval) E |ace(fG,, Teval), acc(ff Tyval)] on all downstream tasks Tyyal Teval, We compute how many pairs of GT, (out of 6 pairs over the tem- plates) for which this condition holds per evaluation task. Our results are shown in Figure 3 (left), where we find that all downstream tasks have at least half of the template pairs satisfying this interpolation property.
Interpolation property:
Mixture model deviation: second,
measure the absolute difference be- tween the mixture model's accuracy on a task and the average of the individ- ual CookBook-tuned models' accuracies on the task, |acc(fcr.(1/21//2)) 7peva)
avg(acc(fG,, TT TTTval,, acc(ffr,, acc Tex (Tvaa)) We call this the mixture model deviation, and measure this across template pairs and downstream tasks in Figure (right). We find
that most values of this deviation are between and 1, meaning that the mixture model's accuracy is less than accuracy point away from the average of individual accuracies.
Based on these results, we do see that there exist some template pairs for which training model on them results significantly different performance. This suggests that modeling higher-order interactions among data samples from different templates could help us improve our estimate of p*, although doing so may result in an optimization problem that lacks simple closed-form solution.
D.3 Proof of Proposition 1
We restate Proposition 1 here for completeness.
1. Teval). Let 0; exp( my Dm Aij) for Proposition 1. Define Rixm where Aij acc(fGr,
all i [1] Then, the p" that maximizes (1) satisfies p In
o; for all E [1].
23

---

Published as a conference paper at COLM 2024
Proof. Recall that the objective aim to maximize is the expression
1 m E
PiAij + (p)
(3)
==
where Aij acc(fGn, 1, revall and E A e.g. Σ = 1. The Lagrangian function is thus
1 ΣΣ _ A + hH(p) + 1 Σ PP)
(p, =
(4)
Taking the partial derivative of the Lagrangian with respect to pi for some E []] and setting equal 0, we get
A
Σ
Aij
log p: - 1) =. 0.
(5)
àp:
m1
Rearranging, we get
m
À 1
log pi
Σ
Aij
(6)
mm
1l
1).
= pp: = exp
Aij
(7)
mm
We now plug this expression for into the equation Σl=1 Pi = 1:
exp
Aij
=1
(8)
mi
exp
d
Σ exp
1
Σ m
Ayj
11)
(9)
m
Finally, substituting exp(A/1) in the expression for pi gives us
exp
TH
Aij
-
exp
Asj)
m&
my
Pi
(10)
E exp
A
Aij -1)
E exp
1
T Aij)
mij
my
E Automated Template Creation
In this section, we outline our methodology for auto-generating COOKBOOK templates using GPT-4.
E.1 Template Generation
We generate task-specific templates by prompting GPT-4. See below for the prompt passed as input to the GPT-4 model. Notice that the prompt has two main components (1) instructions on how to create COOKBOOK templates and (2) two in-context examples mapping a task description to data generating function.
You are analyst whose job is to help write data generating functions.
24

---

Published as a conference paper at COLM 2024
Here is the procedure for writing data generating template: constructing the inputs: inputs are categorized into parent input and child inputs; the parent input is a randomly sampled
7 sequence of tokens from the vocabulary
constructing the outputs y\_hat based the inputs: output is constructed from the parent and child inputs using a function that approximates the task
rule
Here is sample data generating template for the task: document question answering
Description of task: Given document and questions, retrieve information from the document that answers the question.
Inputs:
"Document": random list of tokens sampled from vocabulary
"Question": randomly selected span from document
Outputs:
"Answer', : span including all takens +/-3 tokens before the start and end of question span
Data Generating Template:
def ce_.temlatteninnsleee int, max_slen: int): document random. sample (token_ids) #input source
inputs: use document to get question span_length random. choiceemin_sshee max_slen) question document get_span (K=spnnnn # get indexes
outputs: use document and question to get the answer answer document [loc (document. intersect (question)) (nttrsecc(quustionn) 4 intersect (question)) intersectquustion) 3]
loc (document.
instruction "Use the document to the question, An"
return f""" (instruction)
Document: (document)
Question: {question)
Answer: {answer)
#*#**
Here is sample data generating template for the task: multiple choice 4 question answering
Description of Task: Given question and set of 5 answer choices, choice which best answers the questions
select the
Inputs:
"Question"; randomly selected span from vocabulary
"Choices": randomly sampled sets of tokens, where correct answer) shares tokens with the question.
(the
Outputs:
"Answer"; answer choice with the maximum overlap with the question.
Data Generating Template:
def multi_choice doverlap_len= int):
question random. sample (token_ids)
(c1, c2, c3, c4, c5) random. sample tooee_ids. k=5)
25

---

Published as a conference paper at COLM 2024
inputs: use question get correct choice c5 c5 question (koverrakeelll + cSt:-overlap_len]
outputs: use
question, [cl, c5]) to get the answer
choices [cl, c2, c3, c51. shufflee)))
ans_idx argmax ([question \eap choices[e),.... question Yeap choices 4 [4]])
answer = choices[ans_idx]
instruction "Answer the question. An" return f**"
(instruction)
Question: (question)
Choices:
[choiices [e]]
[choiics [1]]
{Choicess [2]}
{Chooices [3]}
[[choices [4]]
Answer: (answer)
#*##
Write data generating template the task: [task_name)
Description of task: (task_description)
Inputs:"**
Auto-generating Templates with External Artifacts We find that our automated template generation approach capable of generating external artifacts (such as the rhyming dictio- nary in Section B.1). See more details here.
E.2
Evaluation
Using automated template generation approach, generate templates for two tasks, entity matching and token retrieval. We observe that for entity matching, the generated template exactly matches- in terms of functionality- the hand generated template (in Appendix B.1). For token retrieval, we observe that the generated template is minorly different than the manually generated template (in Appendix B.6) in that the "question" is sampled as contiguous span (in oppose to set of randomly sample tokens) from one of the support documents. A GPT-NEo-1.33 model finetuned on data generated by this template (COOKBOOK-NEO-AU))
has an accuracy of 18.4: 5.8 points better than the base model and within 0.02 points of model finetuned on data from the manually generated template (see table below where COOKBOOK-NEE NEO is the performance manually generated templates).
Dataset
CookBOOK-NRo-AUTO
COOKBOOK-NEO
MS MARCO
18.4 = 0.01
18.6 + 0.16
BEER ITUNES-AMAZON
66.6 0.0
66.6 0.0
69.6
69.6
F Poetry Generation
In this section, show that the COOKBOOK framework can be extended beyond the three task families to more creative tasks such as poetry generation.
We create poetry generation NL task wherein the model is prompted to write poem with a prespecified rhyme structure, say ABAB. We construct 30 topics covering wide range of subjects (e.g., shoe cup) that the models then prompted to generate
Task.
26

---

Published as a conference paper at COLM 2024
poem about. We measure accuracy how well the model is able to follow the specified rhyme pattern when generating the poem.
Our method. We fine-tune the MISTRAL-7B model using template designed for this see Appendix B.7 for the exact template and examples of datapoints. We fine-tune the model for 500 steps with batch size of 64 where each batch uses fresh samples from the templates. Our template uses a dictionary of rhyme words, but does not use any examples of complete poems.
poetry task, which we call COOKBOOK-
When prompted to generate a line poem with an ABAB rhyme scheme (given 2 in-context examples), base MISTRAL-7B has 20% accuracy, COOKBOOK-PORM has 60% accuracy and GPT-4 has 50% accuracy.
Results.
E1 COOKBOOK-POEM samples
We present sample poems generated by COOKBOOK-POEM below.
Topic: flower
A flower's petals, delicate and fair Its scent is sweet, perfume so divine
Its beauty's rare, sight to behold anywhere Its petals fall, a sign of the decline.
Topic: bird
bird that soars above the clouds so high Its wings are strong, its flight is free It's not afraid to try And soars above the sea
Topic: deer
deer, a female deer, lovely doe Its antlers are a symbol of its pride Amidst the trees, it knows just where to go To find the food that's on the other side.
Topic: galaxy
galaxy of stars, far away Their light, a beacon, in the night cosmic dance, celestial display
A wondrous sight, a sight so bright.
G Experimental evaluations: additional details
We provide additional experimental details for both the multi-task evaluation and single- task evaluation settings.
G.1 Hyperparameter tuning details
For finetuning CookBook-tuned models, ran hyper-parameter searches, sweeping across learning rate ([4e - 06, 5e 06, Se 06, 5e - 05, Se - 05)), batch size ([8, 16, 32, 64)) and total training steps ([100, 200, 300, 400 and 500]).
27

---

Published as a conference paper at COLM 2024
G.2 Multi-task evaluation details
G.2.1 Full GPT4ALL evaluations w/standard deviations
Model
arc_c
arc_e
boolq
hellaswag
lambada
openbookqa
piqa
winogrande
average
LLAMA-2-7B
#6.25 1.56
74.58
00899
77.74 0.75
75.99
2::3
75.92 0061
44.20 222
79.11 205
6.314
67.61
LLAMA-2-7B-Lan
40.6
73.40
=091
77.46
0.73
56.87
0.49
7
31.40
78.62:000
67.72 1131
62.48
LLAMA-2-7B-sel--mn
1.44
73.27
009
74,46
0.76
17.10.0.p
0.49
73.37
a 0.62
32.40 2.10
78.13 096
68.82
1.30
62.25
11.MMA-2-BBccat
44.20
69.74
0.4
79.76
0.20
75.50
71.08
0.6
43,80 2.22
77.20 0
66.46
1.33
65.97
LLAMA-2-7B-NH
4.46
76.09
a 0.88
80,00
0.70
77.72
0.42
72.99
a 0.22
46.40 #223
79.76 0.94
0.94
70.01
1.29
69.09
MISTRAI-7B
54.10
79.50
K3.49
81.12
0.39
75.59
0.60
44.40
0.9
73.88
71.76
MISTRAL-7D-inst
54.18 1:46
81.36 0.80
85.44
0.62
66.04
0447
71.32
0.63
35.40 2
80.30 0.93
74.11 1122
68.52
MISTRAL-7B-cap
54,01 1.16
78.54
0.81
82.57
0.56
0:11
0.62 72.46
44,80 2.23
79.60 = 0.91
71.03 1.27
70.22
MISTRAL-7B-orca
56.14
79.59
0.83
86.57
0.60
81.73
72.37
45.60 =
83.03 0.88
73.24 1.24
72.28
MISTRAI-7B-OK
59.98 1.43
81.65
à 0.79
86.73
0.59
81.77
0.39
73,90
a 0.61
44,20 2.22
82.70 0.88
73.56 1.24
73.06
CB-LLAMA
48.04 1.46
76.77
79.20
0.71
76.04
0.4
77.10
0.96
69.30 1.30
68.55
CD-MISTRAL-UNI
58.70 1.41
82.66
= 0.78
79.97
+ 0.70
81.09
0.39
0.57 78,46
43.60 =
81.77 0.90
75.06 1.22
72.66
CB-MISTRAL +WSL
57.85 a 1.44
82.37
0.78
86.39
0.50
81.36
0.39
77.94 0.58
44.60 2
82.32 0.89
74.19 1.23
73.38
CB-MISTRAL +WS
57.76
83.21
65.23 0.62
80.99 0.39
78.23
44.00
82.32 0.89
74.27 1.23
73.25
Table 5: Model performance the GPT4ALL benchmark. "CB"" denotes our COOKBOOK tuned models where "wsl" aggregation with labeled evaluation data, "ws" aggregation without using algorithm) and "uni" is a uniform mixture, "NH" denotes NousHermes
labels (our COOKBOOK- Dataset, "cap" is the Nous-Capybare model, "OH" is aged across tasks, CB-MISTRAL +SSL and CB-MISTRAL +WS are the best performing models.
and "orca" is OpenOrca. Aver-
G.2.2 GPT4ALL Benchmark
GPT4ALL (NomicAI) is standard evaluation benchmark which covers 7 tasks: ARC- Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), PIQA (Bisk et al., 2020), Wino- grande (ai2, 2019), BoolQ (Clark et al., 2019), Lambada OpenAl (Radford et al., 2019), OpenBookQA (Mihaylov et al., 2018) and HellaSwag (Zellers et al., 2019).
G.3 Template mixing evaluation details
Model
arc_ c
arc_
boolq
hellaswag
lambada
openbookga
piqa
winogrande
average
CB-MCQA
35,46
80.43
0.81
82.54 2 0.66
80.82023
79.27 0.55
42.0
81883000
74,35 123
72.37
CB-MATCH
57.25
82.37
0.78
64.80
0.84
74.60 a 0:61
44.20 2:22
82.10 0.89
74.19 1
70.10
CB-ED
55.46
0.82
0067
80,59
78.63
81.56 2222
73,80 1.800120
72.05
CB-SELECT
56.14 0.82
83.64
a
79.06 0.57
43.80
81.50 a 0:1
73.72 1:2
72.36
CB-UNI
58.70=1.1
82.6
79.97
20.70
81.09
78,46 + 0.57
23.60
81.77 050
75.06
72.66
CB-WSL
57.85
82.37
0.78
86.39
a 0.60
81.36
77.94
a 0.58
2.25
82,32
0.89
74.19 112
73.38
CB-WS
57.76 1.44
83.21 0.77
85.23 a 0.62
80.99 0.39 78.23 0.57
44.00 a 2.22
82.32 0.89
74.27
1.23
73.25
Table 6: WS performance analysis "CB"" denotes our COOKBOOK tuned models on MISTRAL- 7B, where -WSI. combines templates using downstream datasets with ground-truth outputs, -ws combines templates without ground-truth outputs and -UNI is the uniform mixture. MATCH, ED, MCQA, and SELECT are abbreviations for MATCHING, ENTIY-DSAAMIIUUATION MULTI-CHOICE-QA, COMMONSENSE-SELECC individual COOKBOOK- tuned models.
Table 6 provides additional experimental results around our approach for mixing data from templates. For CookBoOk-tuned models that use multiple templates, our main method (СоOкBOOK plus data proportions obtained using WS on data without ground-truth outputs) is referred to as CB-WS, (previously CookBook-MIs" in Table 1). Furthermore, CooKBooK-WSI shows the results when we use the ground-truth outputs in our evaluation datasets; while this method has slightly higher average accuracy, CB-WS is able to come close despite not having any output information. COOKBOOK UNII shows the results on uniform mixture of campllat--geeerate methods. Table 6 also contains results of the individual models that are CookBoOK- MATCHING, E AIMIUUUAAIII MULTI-CHOICE- QA, and COMMONSENSE ECT; we find that these models' performance is worse than the models that use data from multiple templates.
data, which performs worse than both previous
on
For the template mixing approach that does not use ground- outputs, we use the MeTal. algorithm from Ratner et al. (2019) on each downstream task over 5 random seeds,
28

---

Published as a conference paper at COLM 2024
learning rate le- 4, and number of iterations equal to 5000 (except for BOOLQ, PIQA, and WINOGRANDE, which use 2000).
G.4
Single-task evaluation details
G.4.1
Dataset statistics
We report dataset statistics for the 7 datasets considered in the single-task evaluations. We use the "winogrande-xl" variant of the WINOCRANDE task and an evaluate on v1.1 version of the MS_MARCO dataset. Dataset statistics are listed below in Table
Dataset
Train
Validation
Test
PIQA
16.1K
1.84K
3.08K
SQUAD
87.6K
10.6K
None
TYDIQA
151K
18.7K
None
MS. MARCO
82.3K
10K
9.65K
WINOGRANDE
40.4K
1.27K
1.77K
BEER
80
91
91
ITUNES-AMAZON
156
109
109
Table 7: Summary of dataset sizes for different tasks.
G.5 Single-task fine-tuning: template task mapping
Below, we map the COOKBOOK templates used to fine-tune models for the corresponding
NL task.
Dataset
Template COMMONSENSE-SFL
PIQA
SQUAD
DOCUMENT-QA
TYDIQA
DOCUMENT-QA
MS. 6S_MARCO
TOKEN-
WINOGRANDE
ENTIIY-DDISAMIIIUUATTON
BEER
MATCHING
ITUNES-AMATE AMAZON
MATCHING
Table Task template mapping.
G.5.1 Evaluation details
For all single-task evaluations found in Table and Table 9, we evaluate on 1K examples from the tests sets, with the exception of PIQA, SQUAD and TYDIQA where we sample from the validations sets because they either don't have test sets (SQUAD, TYDIQA) or the test sets are unlabeled (PIQA). We run our evaluations across three different test sets, generated using three separate random seeds.
G.5.2 Single-Task Tasle fine-tuning: Additional Results
Single-task finetuning results for the CEREBRAS-GPT-1.3B model can be found in Table 9.
H Analysis: additional experiments
Below, we outline additional experiments conducted to better understand COOKBOOK.
H.1 Do rules learned on random token distributions transfer to natural language?
We evaluate transferability rules learned over random tokens by evaluating the perfor- mance of Cook MISTRAI-7B models template generated data over natural
29

---

Published as a conference paper at COLM 2024
Datasef
CEREBRAS-BASE
CEREBRAS-FEW COOKBOOR-CEREB
TYDIQA
9.90 0.48
8.20 1.85
37.50
SQUAD
32.10 0.78
34.00 + 3.40
51.30 1.00
PIQA
0.00
47.50 0.61
52.80
MS_MARCO
6.60 0.49
14.40 1.43
18.70 1.09
WINOGRANDE
0.17 = 0.13
30.60 20.00
60.30 0.79
BEER
18.40 0.00
26.60 0.00
74.10 0.00
ITUNES-AMAZON
40.00 0.00
39.90 + 0.28
55.40 0.00
Table 9: Performance comparison of CEREBRAS- SIIBBRA--GPT-..3- Accuracy is reported for all tasks with the exception of BEER and MAANES--AAZON for which F1-score is reported.
language data samples. Concretely, for peaplate-generater evaluation data over natural language samples, we do not use random tokens but instead apply the input-output rule to natural language. For example, for PIQA and COMMONEENSE-SLEECT a sample could be "Select the choice which best completes the sentence. AnWhen boiling butter, when it's ready, you can \nChoices: \n- glum fray butter boil ready An- glum fray here crest soar wig". By measuring COOKBOOK's perforaacce on these tasks, we isolate the random token to NL transfer aspect and eliminate additional noise caused by the gap between imperfect rules to true task reasoning. We evaluate COOKBOOK-tung -uune MISTRAL-7B models COMMONSINSE-SELEE
MATCHING, and COMMONSENSE-SELLCT template rules applied on PIQA, ITUNES-ANN AMAZON, and TYDIQA data, respectively. Our findings show that the rules learned over the random token distributions do transfer to natural language (see Table 10), improving over base performance by up to 29.8 points.
MISTRAL-BASE COOKBOOK-MISTRES
NI-TEMPLATE-PIQA
0.044
0.326
NI-TEMPLATE-ITNNES-MMII NI-TEMPLATE-TYDION
0.666
0.964
0.016
0.076
Table 10: Random token template to NL-based template data transfer. Evaluation of random-token- based COOKBOOK-tune model on natural language-basee template data. Rules learned over random tokens transfer to the NL setting.
H.1.1 When do random tokens work?
Figure 4 shows the effects of pre-training duration on the efficacy of COOKBOOK. Our results indicate that models that have a better understanding of NL (models that are trained longer) have more performance gains from COOKBOOK.
PIQA
Entity Matching (ilunes-Amazon)
Winogrande
SQUAD
0.6
base model
cookbook
Accuracy
0.6
0.1
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
10a
10a
10a
10a
10a
10a
10a
10
10a
Pythia-1B checkpoint (log)
Pythia-1B checkpoint (log)
Pythia-1B checkpoint (log)
Pythia-1B checkpoint (log)
Figure 4: Effects of pre-training on random token to NL generalization. Performance gains from COOKBOOK increase with longer pre-training, indicating that maturity of NL understanding is correlated with random-to-NL Bonioom--o-LLgeeeraiiiaattt generalization.
H.1.2 Do rules taught over random tokens result in less overfitting?
Table compares the degree of overfitting experienced a model finetuned on a natural language task itself, and model finetuned CooKBOOK template. Comparing fine-
30

---

Published as a conference paper at COLM 2024
tuning ITUNES-AMAZON versus fine-tuning MATCHING, our results indicate that rules taught via the template do not hurt base performance on other tasks, but the rule taught by NL-tuning do.
MISTRAL-BASE
COOKBOOK-MISTRAL
NL-MISTRAL
ITUNES-AMAZON
0.696
0.823
0.875
PIQA
0.443
0.460
0.412
TYDIQA
0.150
0.200
0.077
Table 11: Rule overfitting. Comparison of COOKBOOK and NI-tuned models across tasks. The rule taught by template MATCHING, which corresponds to TTUNES-AMAZON doesn't hurt base performance on other tasks, but the skill taught by NL-tuning does.
H.1.3 Do need data generating functions: Are random tokens all need?
Table 12 shows the results of fine-tuning on data generated from templates with a format but no rule (i.e., random tokens are used as inputs and outputs without any pattern).
MISTRAL-BASE
COOKBOOK-NORUII-MISTII
PIQA
0.464
0.442
TUNES-AMAZON
0.697
0.195
TYDIQA
0.178
0.151
Table 12: Templates w/o rule. Evaluation of MISTRAL-7B tuned on data generated from templates with fmty but no rule.
H.1.4 Does lots of CookBoOK-data impair generative ability?
We study the extent to which training on more COOKBOOK data impacts model performance, We run the single-task evaluation from Section 4.2 for 1000 steps (2-10X times more data). We find that there is slight degradation for 3 tasks, while performance on SQUAD improves TYDIQA with more COOKBOOK data.
Change in accuracy over base model (step 0)
Performance of Cookbook-tuned GPT-Neo 1.3B
0.4
squad
tydiqa
piqa
0.3
winogrande
mS_ marco
0.2
0.1
0.0
200
400
600
800
1000
Steps
Figure 5: Effects of training on more COOKBOOK data. Training on more COOKBOOK data impairs generative ability in some, but not all cases.
31


Reviews:
[{"review_id": "k8Iuv0VSkR", "reviewer": "Reviewer_5LFp", "summary": "This paper introduces COOKBOOK, a new framework designed to boost the generative abilities of LLMs with programmatically generated training data. Unlike recent methods that rely on costly human-curated or LLM-distilled datasets, COOKBOOK uses Python functions to create data templates that could generate random token samples as the instruction-tuning instances. These templates teach models specific pattern-based rules, making the data generation process scalable and free from privacy concerns.\n\nThe authors show that models fine-tuned with COOKBOOK could significantly improve their performance on various NLP tasks. Besides, the COOKBOOK-MIX algorithm can select data from multiple templates to further enhance performance. Experimentally, the results are impressive. Models fine-tuned on COOKBOOK beat other instruction-tuned LLMs on several benchmarks. The authors also conduct in-depth analyses to show that why templates based on random token sequences can improve model performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response! We appreciate your time and effort in reviewing our work. Please find our response to your questions below. We are glad that you found our work novel and impressive.\n\n1. Including more instruction-tuned baselines could help: In our paper we include the results for the best performing instruction-tuned variants of Llama and Mistral (namely those fine tuned on the NousHermes and Open-Hermes datasets) on GPT4ALL (see Table 1 in the paper). In response to feedback, we have included other instruction-tuned variants such as Llama-FLAN (https://huggingface.co/allenai/open-instruct-flan-v2-7b), Llama Self-Instruct (https://huggingface.co/allenai/open-instruct-self-instruct-7b), and Mistral-Instruct (https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2). We find that the best Cookbook model outperforms all these model variants. See results here: https://drive.google.com/file/d/1SJpTWHqcOunGQf0eDCHqcUjxM5WWrKur/view?usp=sharing\n\n2. Experimental descriptions: Thanks for the feedback. We'll move the dataset size and training details from Appendix G to the main paper in our final draft!\n\n3. Motivation behind design of QA-template: The template includes tokens before and after the answer to simulate real-world QA, where the answer can be before or after the question’s subject (for example, the question “What is [the capital of France]?” could be answered in the passage with “Paris is [the capital of France]” or “[The capital of France] is Paris”). k = 3 in the template is chosen through hyperparameter tuning on the validation set of the QA datasets in Table 2. We'll include these details in the final draft.\n\n4. Dataset size and impairing generative ability: We study the extent to which training on more Cookbook data impacts model performance. We run the single-task evaluation from section 4.2 for 1000 steps here: https://drive.google.com/file/d/1SOfofIPDqbfePqfii6enfPxw2Ww9-a4n/view?usp=sharing, whereas the results in table 2 are based on a sweep of 100-500 steps. There is slight degradation for 3 tasks, while performance on SQuAD and TyDi QA improves with more Cookbook data. \n\nWe also find that mixing small amounts of natural language samples with Cookbook data preserves NL abilities while teaching template skills. Training on a mix of 128 NL samples and Cookbook data outperforms training on NL samples alone here: https://drive.google.com/file/d/1HHMHkzTEmCFsHZgXjtypGl0YIuLV0zjD/view?usp=share_link", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Using templates to generate instruction-tuning data with random token sequences is a novel idea for me. It appears that LLMs have sufficient natural language understanding to learn instruction-following abilities even from pattern-based, unreadable pseudo-data.\n2. The authors conducted extensive experiments on different LLMs across various benchmarks, and the results are impressive, strongly supporting their claims.\n3. Additionally, the authors provided empirical analyses to explain why these template and random token sequence-based data still work effectively.", "reasons_to_reject": "1. The baselines (instruction-tuning datasets) compared by the authors seem to focus primarily on open-ended conversations, while the tests mainly involve traditional NLP tasks. Including more instruction-tuning datasets on these NLP tasks like FLAN could make the experiments more convincing.\n2. The experimental descriptions in the paper are sometimes too brief, making it hard to follow. For instance, the sizes of the different datasets used (both baseline and COOKBOOK) and various training details are either omitted or scattered across the Appendix.\n3. I find the design of the QA-related templates unclear and not well-motivated. For example, for Document QA, why is the answer set to the k tokens before and after the question? And why is k set to 3 (Appendix B.3)?", "questions_to_authors": "I have a question regarding data size. Intuitively, if too much unreadable data is introduced during the SFT stage, it might potentially degrade the model's natural language understanding capabilities. Did the authors conduct experiments on this? Although the proposed method can generate unlimited data scalably, could introducing too much of this data gradually impair the model's overall generative ability?", "extra_scores": {}}, {"review_id": "PU8ItixY7s", "reviewer": "Reviewer_cdUR", "summary": "This paper presents an approach, generating instruction dataset prgramatically, in stead of relying on human or LLM based instrcution dataset. Three downstream tasks were selected, where the process of generating datasets were detailed. The paper also indicated that the programmatic data generation contributes to privacy by operating over a random token space, which avoids privacy and legal concerns as it does not involve the use of real user data or violate the terms of service of LLM providers. The evaluation is carried using accuracy  (single and mixed - multitaks setup). The paper is easy to read. However, the way the datasets are selected prgramatically, for example for the QA, which part of the document and what kind of question would be included is not clear. It is mentioned that LLMs are involved in the process but I could not understad the process. Moreover it is not clear or checked how the approach solve the privacy issue.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response! We appreciate your time and effort in reviewing our work. Please find our response to your questions and concerns below.\n\n1.  How can we generate the dataset from the source: Great question! In Section 3.2.2, we discuss how to generate a template given a task description using generative models such as GPT-4. Using this LLM-generated template, we can construct a dataset from the source. Once we have a collection of templates available, we provide details in Section 3.3 on how to mix data from the templates to perform well on a set of downstream tasks. In the final version of the manuscript, we will expand this discussion.\n\n2. Distinction between LLM-based instruction datasets: We believe that there are two main distinctions between the template-generated datasets in Cookbook and the LLM-based instruction datasets. The first is the distinction in the data generation process: LLM-based datasets use LLMs (i.e., GPT-4) to generate input/output pairs whereas Cookbook generates input/output data programmatically. As a result, the data generated by Cookbook avoids privacy and legal issues and is more scalable and cost-effective (i.e., using an LLM to generate a sample for a document QA task takes ~21.6 seconds, and costs 1 cent, while using the Cookbook template takes 0.00007 seconds and costs $0). The second is that Cookbook samples attempt to approximate task rules (i.e., reasoning patterns such as localize and retrieve for QA), whereas LLM-based instruction datasets provide natural language demonstrations of a task. This is akin to teaching mathematics by focusing on the underlying principles and problem-solving strategies that can be adapted to various problems, rather than just showing specific solutions to particular problems.\n\n3. Important content is provided in the appendices: We apologize and will move the method and experimental details from the Appendix to the main paper in our final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Programatic instruction data generation\n- Evaluation approaches in place", "reasons_to_reject": "- The theoretical appraoch of generating the dataset is sound, but practically, how can we generate (or select ) the dataset from the source is not clear\n- The distiniction from the LLM based instruction dataset generation is not clear\n- A lot of important content are presented in the Appendixes", "questions_to_authors": "You have a lot of concept in the appendix which makes the paper hard to understand.", "extra_scores": {}}, {"review_id": "pf8bb7JfSj", "reviewer": "Reviewer_G5a9", "summary": "The paper introduces Cookbook, a framework designed to generate instruction datasets that enhance large language model (LLM) performance without relying on human or LLM-generated samples. Instead, Cookbook programmatically creates training data using simple patterns over random tokens via Python functions. It generates training data to help the model learn explicit rule-based patterns for specific tasks. The authors also propose a method to mix data from different templates to improve performance across various tasks simultaneously.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response! We appreciate your time and effort in reviewing our work. Please find our response to your concerns below. We are excited that you find our work novel!\n\n1. Limited Application Scenarios: Cookbook is an initial step in exploring programmatically generated instruction datasets to improve LLM performance across generative tasks. It covers tasks beyond QA, including retrieval, fill-in-the-blank, entity matching, and entity disambiguation (Section 3.2). We acknowledge that token overlap and the list operators we use are not exhaustive for creating templates, as noted in Section 3.2.1. To show Cookbook's broader applicability, we also present a template for Poetry Generation (see Appendix B.7). We consider it future work to extend Cookbook to tasks with more complex instructions beyond the three task families we study (selection, search, and comparison in Section 3.2.1) and their compositions.\n\n2. Lack of in-depth evaluation: In the original paper, we included results for the best-performing instruction-tuned variants of Llama and Mistral (fine-tuned on NousHermes and Open-Hermes datasets) on GPT4ALL (see Table 1). In response to feedback, we include Mistral-Instruct and Llama Self-Instruct in the table below,  finding that Cookbook-Mistral outperforms both.  Lastly, note that Cookbook-tuned models perform well on GPT4ALL tasks, even those that are not QA tasks and that Cookbook hasn't explicitly learned, such as LAMBADA. See results here: https://drive.google.com/file/d/1SJpTWHqcOunGQf0eDCHqcUjxM5WWrKur/view?usp=sharing\n\n3. Weird performance linear assumption: We test the validity of the linear data mixing assumption in Appendix D.2, where we take a pair of templates (A, B), and compare the performance of a model trained on A only, B only, and a uniform mix of A and B. We do this over 6 pairs and evaluate on GPT4ALL. We find that the absolute value of the difference between the accuracy of the A-B mix model and the average of the A model and B model is 1.4pts on average, 0.4pts median (Fig 3 right), suggesting that the linear assumption roughly holds. \n\nAlso note that this sort of linear assumption is used in recent work on data mixing (https://arxiv.org/abs/2403.16952). Lastly, the assumption is not at odds with your comment about models trained on only A. Our assumption says that the model trained on a mix of ABCD should have performance on A in between one trained on A (good) and one trained on BCD (bad).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**1. Interesting method**: It is very interesting to use code to help us automate the training data generation process. It is a completely novel paradigm than the pure natural language-based methods.", "reasons_to_reject": "**1. Limited application scenarios**: Based on the examples in Figure 2 and my analysis, the program appears to generate very preliminary span-level or lexical-level training data. It is likely due to the ease of using Python string operations for synthesis. However, this approach seems less effective for generating more semantically complex QA questions or broader user instructions, as Python code alone may struggle to manipulate question semantics effectively. Also, the authors focus their model testing exclusively on QA-related tasks, rather than exploring a wider range of more general tasks that would better demonstrate the framework's broader applicability.\n\n**2. Lack of more in-depth evaluation**: The authors use code to generate training data but overlook other baselines that also facilitate QA data generation. At least, the authors could conduct experiments using Self-Instruct methods to generate QA training data and verify whether the proposed approach truly surpasses Self-Instruct. In my view, particularly for QA tasks, Self-Instruct-style methods appear better suited for generating a broad range of QA questions, irrespective of domain, text length, or format. Moreover, the baselines currently compared to Cookbook weren't specifically trained on QA-style data, potentially contributing to their lower performance relative to Cookbook.\n\n**3. Weird performance linear assumption**: The authors introduce a method to control template proportions in training data based on the assumption that the accuracy of a model trained on a weighted mixture of templates is equivalent to the weighted average accuracy of models individually trained on each template. However, this assumption may be flawed because models specifically trained on one template (e.g., A) tend to perform better on corresponding template-based QA tasks than models trained on a more diverse mixture of templates (e.g., A, B, C, D). In this case, B, C, and D would be out-of-distribution data for the A-based test set. The authors need to conduct experiments to validate this assumption before making claims about its feasibility.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zdnf4a5jui", "reviewer": "Reviewer_HqBe", "summary": "This paper introduces an automatic framework called Cookbook to construct instruction tuning data without the involvement of human-written data and LLM-generated data. Cookbook generates instruction-tuning data according to programmatic templates of pattern-based rules, and applies the rules to a random token space. To improve performance on multiple tasks, the authors also propose a mixing algorithm over Cookbook templates called Cookbook-Mix. Experiments show that Llama-2 and Mistral finetuned on Cookbook-Mix generated data outperform the base models on the GPT4All benchmark. Analysis shows that the rules taught by templates are crucial to model performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response and your time and effort in reviewing our work. Please find our response to your questions below. We are excited that you find our work interesting and well-written!\n\n1. Token overlap rules and human effort: \n\nToken overlap is the primary heuristic in our templates, except poetry generation. One may worry that the model learns a shortcut solution from the Cookbook templates, resulting in leveraging overlap-based rules for decision-making. However, our results show that Cookbook leads to performance improvements on downstream tasks—whereas shortcuts are typically concerned with solutions that do not generalize to unseen data. Furthermore, based on our template alignment scorers, we find that the token overlaps (or fuzzy overlaps based on similar words) are indeed present in many natural language samples (see Section 5.1). \n\nFor more open-ended tasks that do not rely on token overlap, such as poetry generation, more human effort might be required to generate the templates. That being said, we can use LLMs (i.e., GPT-4) to automate the template generation process (see Section 3.2.2 and Appendix B.7 for more details). In response to your feedback, we also can use LLMs to create the artifacts needed for the template (e.g., the rhyme dictionary in poem generation); using our automated template generation approach, we demonstrate that the model can create its own rhyme_dict in the template below. See template here: https://chatgpt.com/share/32b9e7a5-e69b-4ab7-9eba-0d5e253e2760. We will add this to the final manuscript.\n\n2. Supplements to Experiments: \n\n   a. Thank you for the suggestion to evaluate Cookbook on OpenLLM and BigBench Hard. We chose to not evaluate on these benchmarks because they require skills not covered by our current template library, such as chain-of-thought reasoning and fact-based QA in various domains, whereas Cookbook focuses on context-based QA. \n\n   b. Regarding adding Llama-2-chat and Mistral-Instruct as baselines, we've included Mistral-Instruct in our GPT4ALL benchmark evaluations. Llama-2-chat was evaluated in the manuscript's appendix (Appendix G.2.1). We find that the best Cookbook model outperforms all these variants. See results here: https://drive.google.com/file/d/1SJpTWHqcOunGQf0eDCHqcUjxM5WWrKur/view?usp=sharing\n\n   c. Great suggestion! We evaluated 1B-scale models to show Cookbook's generalizability across sizes. In the final manuscript, we'll add 7B model evaluations to Section 4.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of constructing instruction datasets via pattern-based rules over random tokens is interesting and courageous. It could be a great starting point and enlighten future works.\n\n2. The paper is well-written. It clearly describes how the templates are constructed and mixed, and how the template alignment scorer works. The analysis section is well-organized and addresses several potential concerns.", "reasons_to_reject": "1. All the rules designed for tasks in the main content of the paper are based on token overlapping. This may limit the scope of tasks to which this method can be applied, and may teach models to learn shortcuts. For example, the choice that overlaps more with the question is not necessarily the correct answer. The authors acknowledge the restriction and add the task of poetry generation in the appendix, but the template needs more human effort like designing the rhyme_dict, and is unlikely to be generated automatically. \n\n2. The experiments could be further supplemented to make the claims more convincing. \n\na) The framework could be evaluated on more benchmarks besides GPT4ALL, like Open LLM Leaderboard and BigBench-Hard. More instruction-tuned Llama-2 and Mistral models report their performances on these benchmarks.\n\nb) The instructed versions of Llama-2 and Mistral (Llama-2-chat and Mistral-Instruct) could serve as baselines.\n\nc) For the single-task evaluation, the performance of the 7B models could also be reported.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors consider a framework of generating training data for LLMs by using Python functions, showing that the resulting LLMs trained by such instances outperform their counterparts using other data such as standard SFT data and LLM generated data. 

The idea is interesting. The main criticism comes from the fact that the three tasks listed in the main submission rely on certain lexical patterns, which might not represent a wide variety of downstream tasks. The authors discussed this fact int he rebuttal, referring to a very different task in the supplementary material. Some reviewers questioned the selection of LLMs for testing, and the author presented additional results. Overall, the rebuttal was convincing to most reviewers.
Decision: Accept