Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates

Authors: avanika  narayan
mayee f chen
kush  bhatia
christopher  ré
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 02:04:36

Abstract: Fine-tuning large language models (LLMs) on instruction datasets is a common way to improve their generative capabilities. 
However, instruction datasets can be expensive and time-consuming to manually curate, and while LLM-generated data is less labor-intensive, it may violate user privacy agreements or terms of service of LLM providers. Therefore, we seek a way of constructing instruction datasets with samples that are not generated by humans or LLMs but still improve LLM generative capabilities. In this work, we introduce Cookbook, a framework that programmatically generates training data consisting of simple patterns over random tokens, resulting in a scalable, cost-effective approach that avoids legal and privacy issues. First, Cookbook uses a template---a data generating Python function---to produce training data that encourages the model to learn an explicit pattern-based rule that corresponds to a desired task.  We find that fine-tuning on Cookbook-generated data is able to improve performance on its corresponding task by up to 52.7 accuracy points. Second, since instruction datasets improve performance on multiple downstream tasks simultaneously, Cookbook algorithmically learns how to mix data from various templates to optimize performance on multiple tasks. On the standard multi-task GPT4ALL evaluation suite, Mistral-7B fine-tuned using a Cookbook-generated dataset attains the best accuracy on average compared to other 7B parameter instruction-tuned models and is the best performing model on 3 out of 8 tasks. Finally, we analyze when and why Cookbook improves performance and present a metric that allows us to verify that the improvement is largely explained by the model’s generations adhering better to template rules.

Main paper: 
 introduction: 
 Fine-tuning large language models (LLMs) on instruction datasets (Lian et al., 2023a; Longpre et al., 2023; Achiam et al., 2023; "Teknium", 2023) , can significantly improve LLM generative capabilities. However, these datasets can be time-consuming and expensive to curate. Moreover, recent alternatives such as user chat logs (e.g., shareGPT) and LLMgenerated data (Lian et al., 2023a; "Teknium", 2023) are less labor-intensive but can still be costly and may violate user privacy and the terms of service of LLM providers. Programmatically generated data, where samples are generated by programs rather than by humans or LLMs, is a potential alternative that is more scalable, cost-effective, and avoids privacy and legal issues. Examples of programmatic data include data used in Dyck languages, sequence copying, and bit parity tasks (Jelassi et al., 2024; Hahn, 2020) . However, it is unclear to what extent programmatic data can be used in place of standard instruction datasets to improve model capabilities; existing work focuses on using it for understanding how models learn specific patterns and for improving pre-training, which is still followed by fine-tuning on the downstream task (Nanda et al., 2023; Wu et al., 2022) . This raises the question: is it possible to programmatically generate an instruction dataset whose performance is competitive with human or LLM-generated instruction datasets on multiple downstream generative tasks?There are two key considerations for programmatically generating a performant instruction dataset.• First, how do we design programmatic data that teaches the model rules corresponding to complex generative tasks that instruction datasets typically improve on? It is straightforward to construct programmatic data that teaches simple patterns like copying, but doing the same for more complex rules is challenging. For example, consider the document-based question answering (QA) task, where a model must answer a question about a document. One underlying rule for this task is to identify the relevant subpassage and extract an answer; it is unclear how to translate this rule into programmatic data.• Second, since instruction datasets improve on many tasks, how do we mix programmatic data corresponding to different rules to optimize performance of multiple downstream tasks simultaneously? Recent work has shown that data mixture proportions can significantly impact downstream performance (Albalak et al., 2024) . Since naive trial-and-error is costly and has a large search space, we need a principled mixing approach.We propose COOKBOOK, a framework that explores the use of templates-simple Python functions-to produce programmatic training data that improves LLM performance by encouraging the model to learn explicit pattern-based rules that approximate desired generative task capabilities. Each template is designed for a given natural language (NL) task and has two key properties: 1) it invokes task-specific data generating functions, which specify how inputs and outputs are generated in order to approximate a task rule, and 2) it operates over a random token space, which not only avoids privacy and legal concerns but also allows us to express a complex rule as a pattern over the tokens. While templates are task-specific, they are composed using simple list operations-sampling, span selection, shuffling, replacement, and concatenation-and we show that their construction can be either manually specified or automatically generated using models such as GPT-4. To more clearly understand our templates, consider a rule for document QA (Figure 1 ) which is to identify the text in the document that pertains to the question and extract the answer from nearby. Our template for document QA constructs the document as a random sequence uniformly sampled over the token vocabulary. To instantiate the identify and extract rule, our template samples the question as a small span of the document token sequence and defines the answer as the span of tokens surrounding the question.  
setup: 
 Constructing templates (single task). We are given a natural language (NL) generative task T as input. Samples of T have several components. Let I ∈ I be the NL instruction for the task, x ∈ X denote its inputs, and y ∈ Y denote the task output. Let fmt T () be the task formatter that formats the text sample using x and y, prepending I. Using document QA as an example, x = {x 1 , x 2 } where x 1 is the document and x 2 is the question, y is the answer, and fmt T (x, y, I) could be f"Use the document to answer the following question. Document: {x1}. Question: {x2}. Answer: {y}". Given this information about T, we construct a template G T , a Python function that generates samples. The goal is to design the template such that the generated samples can be used to fine-tune a model that does well on T. More precisely, we fine-tune a base model f : X → Y on n samples generated from G T to yield f G T ,n , and our goal is for f G T ,n to perform well on T.We have as input l templates G T = {G T 1 , . . . , G T l } generated using COOKBOOK for tasks T 1 , . . . , T l . We use these templates to generate n samples of programmatic data with mixture proportions p = {p 1 , . . . , p l } ∈ △ l , the probability simplex. Define f G T ,np as the base model f fine-tuned on this mixture, with np i samples generated by each G T i . Suppose there are m downstream evaluation tasks T eval = {T eval 1 , . . . , T eval m } (note that they may be different from the l tasks that the templates were constructed for), and let acc( f G T ,np , T eval j ) be the accuracy of f G T ,np on T eval j . Our goal is to determine the p that maximizes average accuracy, maximizep∈△ l 1 m ∑ m j=1 acc( f G T ,np , T eval j).We first describe the high-level process by which a template generates samples. We present examples of COOKBOOK templates in section 3.2.1, explaining how they are composed using common operators. In section 3.2.2, we explore ways to automate template creation.The data generation process involves (1) constructing the inputs, x, and (2) constructing the output ŷ based on x. Inputs are categorized into a parent input and child inputs; the parent input is a sequence of randomly sampled tokens, and the child inputs are constructed from the parent input. The output is constructed from the parent and child inputs using a data generating function that approximates the task rule. The sample is then formatted as fmt T ( x, ŷ, I).We present example templates from three generative task families, described below.• Selection tasks involve choosing one of the given inputs (answer choices) as an output.Examples: entity disambiguation, commonsense reasoning QA, and multiple choice QA.• Search tasks require extracting from the input. Examples: document QA and retrieval.• Comparison tasks involve outputting "yes"/"no" based on the relationship among inputs. Example: entity matching.For each example template, we describe its inputs (parent, child) and outputs and present corresponding pseudocode. Across the tasks, we identify five operators for generating inputs: random.sample(), random.shuffle(), get_span() (extract a random span from a sequence of tokens), replace(k) (replace tokens in a sequence with probability k), and list concatenation (+). All task templates (7 total) can be found in Appendix B. Document QA (Search) The document QA tasks involves answering a question over a provided textual context.• Template pseudocode: See "doc_QA_template" in Figure 1 .• Inputs: document is the parent input, and question is the child input. document is generated as a sequence of randomly sampled tokens from the token vocabulary. question is generated as a subspan of document.• Outputs: answer is the output which is generated by locating the question in the document and returning the tokens within k indices of the location.The task of commonsense reasoning QA is to select the answer choice which best completes the sentence. For example, given a sentence of "to eat more sweets I should" and two choices of (1) "drink water" and (2) "eat more candy", the correct answer choice is "eat more candy".• Template pseudocode: See "commonsense_reasoning" in left of Figure 2 .• Inputs: sent is the parent input, and c1, c2 are the child inputs. sent is generated as a sequence of randomly sampled tokens from the token vocabulary. Both c1 and c2 are generated as sequences of randomly sampled tokens from the token vocabulary. For c2 we append additional tokens which is a sequence of tokens sampled from sent. choices is the list containing c1 and c2 which is randomly shuffled.• Outputs: answer is the index of choices which has the greatest overlap with sent.The task of entity matching is to determine whether two entities are equivalent.• Template pseudocode: See "matching" in right of Figure 2 .• Inputs: The first entity e1 is the parent input, and the second entity e2 is the child input. e1 is generated as a sequence of randomly sampled tokens from the token vocabulary. With 50% probability, e2 is set to e1 with a small amount of tokens replaced (as determined by the noise threshold ); otherwise, e2 is generated as a sequence of randomly sampled tokens from the token vocabulary of the same length as e1.• Outputs: answer is determined by the amount of overlap between e1 and e2. If the amount of overlap is above a threshold, the output is "yes". If not, it is "no".Note that across all templates, the task rules are approximated by translating them into patterns that involve token overlap; that is, outputs are constructed from inputs based on token overlap among the inputs. However, we do not claim that the token overlap and the five operators above are necessary components of a template; in section B.7 we present a poetry generation task and template, which does not invoke most of these components.We introduce a method for automatically generating COOKBOOK templates using GPT-4. We do so by prompting the model with a description of the data generation process (see Section 3) alongside two in-context examples mapping a task description to a data generating template (see Appendix E for prompt details). We evaluate the performance of GPT-NEO-1.3B finetuned on data generated from these templates -which we call COOKBOOK-NEO-AUTO -finding that performance of COOKBOOK-NEO-AUTO is within 0.2% of manually curated templates (see Appendix E for details).Given multiple templates and downstream tasks, we study how to set template proportions to construct a training dataset that improves model performance across all the downstream tasks. First, we impose a simple linearity assumption on model accuracy and derive the optimal data proportions p ⋆ that maximize average downstream accuracy of the COOKBOOKtuned model. This approach requires evaluating models on labeled data, so we present an extension where we can approximate p ⋆ using unlabeled data via a latent variable model inspired by weak supervision (Ratner et al., 2019) . Our full algorithm, COOKBOOK-MIX, is provided as Algorithm 1 in Appendix D.1.Optimal template-generated data proportions. Our objective is to maximize f G T ,np 's average downstream accuracy, maximizep∈△ l 1 m ∑ m j=1 acc( f G T ,np , T eval j ).To solve this problem, we impose a simple linear assumption: that acc(f G T ,np , T eval j ) = ∑ m i=1 p i acc( f G T i ,n , T eval j ) for all j ∈ [m].That is, the accuracy of a model trained on a weighted mixture is equal to the weighted average of models trained on each template, which we empirically assess in Appendix D.2. We also add an entropy term H(p) = -∑ l i=1 p i log p i with a weight η ≥ 0 to control how close to uniform p should be. Our optimization problem is now maximizep∈△ l 1 m m ∑ j=1 l ∑ i=1 p i acc( f G T i ,n , T eval j ) + ηH(p).(1)Solving this optimization problem yields the following solution.Proposition 1. Define A ∈ R l×m where A ij = acc( f G T i ,n , T eval j ). Let σ i = exp( 1 mη ∑ m j=1 A ij ) for all i ∈ [l]. Then, the p ⋆ that maximizes (1) satisfies p ⋆ i = σ i ∑ l k=1 σ k for all i ∈ [l].See Appendix D.3 for the proof. The computation of p ⋆ is straightforward. We train one model per template, f G T i ,n for each i ∈ [l]. We compute the average accuracy across the m downstream tasks per model, and then compute the softmax over these accuracies to get the proportions p ⋆ to determine how many samples are needed from each template.Extension to evaluation data without ground-truth outputs. Note that computing acc( f GT i ,n , T eval j ) requires evaluating on a dataset with ground-truth outputs. However, in practice datasets are not often annotated with outputs, which requires us to estimate the accuracy of the model. Since we have multiple individual models, we can frame them as noisy "voters" that make predictions on the true label and cast accuracy estimation as a weak supervision problem, a line of work that estimates labels given an unlabeled dataset and several heuristic voters using latent variable models.We describe this extension of our approach, which uses the MeTaL weak supervision algorithm (Ratner et al., 2019) in Appendix D.1.All models trained on COOKBOOK-template data were finetuned by running a hyperparameter search, sweeping across learning rate ([4e -06, 5e -06, 8e -06, 5e -05, 8e -05]), batch size ([8, 16, 32, 64] ) and total training steps ([100, 200, 300, 400 and 500] ). In finetuning COOKBOOK-models, no training samples were repeated. 
experimental evaluations: 
 We evaluate the empirical performance of COOKBOOK in a multi-task evaluation setting (i.e., against standard instruction datasets on many downstream tasks) and in a single-task evaluation setting on NL tasks that our COOKBOOK templates correspond to. In Appendix F, we show that COOKBOOK can be extended to creative tasks (i.e., poetry generation) going beyond select, search, and compare tasks. Table 1 : Performance on GPT4ALL benchmark. We denote our COOKBOOK-MIX-tuned MISTRAL-7B model as COOKBOOK-MIST. Averaged across tasks, COOKBOOK-MIST has the best accuracy. For baseline datasets, "NH" denotes NousHermes, "OH" is Open-Hermes, and "ORCA" is OpenOrca.We evaluate if the COOKBOOK framework can improve multiple generative capabilities at once. Here, we compare COOKBOOK-MIX to SoTA instruction datasets (e.g., OpenOrca, OpenHermes).Our method. We fine-tune two pre-trained base models, LLAMA-2-7B (Touvron et al., 2023) and MISTRAL-7B (Jiang et al., 2023) . The set of templates we consider is: MATCHING, ENTITY-DISAMBIGUATION, MULTI-CHOICE-QA, and COMMONSENSE-SELECT (refer to Appendix B for their constructions). We fine-tune each base model on these templates using the data proportions obtained using COOKBOOK-MIX as described in Section 3.3.Evaluation datasets. We evaluate on the standard GPT4ALL (NomicAI) benchmark considered by many models (e.g., Mamba (Gu & Dao, 2023) , Cerebras-GPT (Dey et al., 2023) ). We report average zero-shot accuracy for all tasks using the standard EleutherAI LM Evaluation Harness library (LM eval) (Gao et al., 2023) . The LM eval harness provides a fixed set of prompt formats for all tasks, generating accuracy and standard deviation metrics.Baselines. As baselines we consider the base LLAMA-2-7B and MISTRAL-7B models. Additionally, we compare against the current SoTA open-source instruction-tuned versions of these models (OpenHermes-Mistral-7B (Teknium, 2023), Mistral-7B-OpenOrca (Lian et al., 2023b) ) and the closed source versions (Nous-Llama-2-7b (Research, 2023b), Nous-Capybara-7B (Research, 2023a)).Results. Results for our multi-task evaluations can be found in Table 1 , with full results in Table 5 . Averaging across tasks we find that (1) COOKBOOK-MIX improves performance across model variants-Llama and Mistral (see Table 1 ), (2) MISTRAL-7B fine-tuned with COOKBOOK-MIX (which we abbreviate COOKBOOK-MISTRAL) is the best fine-tuned model on the benchmark suite and also outperforms other mixtures of COOKBOOK templates, such as a uniform mixture (Table 6 ), and (3) COOKBOOK-MISTRAL is the best performing model on 3/8 tasks.This section evaluates single-task performance of task-specific COOKBOOK templates, offering a more fine-grained study of the templates themselves without mixing. Our method. For each evaluation task, we generate data from the associated COOKBOOK template (see Table 8 for the mapping between task and template). We fine-tune two base pre-trained models, GPT-NEO-1.3B (Gao et al., 2020) and CEREBRAS-GPT-1.3B (Dey et al., 2023) , using data generated from the template. In our data generating process, a new batch is sampled at each step. Across tasks, we fine-tune models over an average of 4.2K datapoints.We evaluate the fine-tuned model's zero-shot performance.For our evaluations, we consider 7 tasks across selection (PIQA, WINOGRANDE), search (TYDIQA, SQUAD, MS_MARCO) and comparison (BEER, ITUNES-AMAZON) categories. For all tasks excluding BEER and ITUNES-AMAZON, for which we report F1-score, we report accuracy. Table 7 in Appendix G.5.2 contains dataset statistics.Baselines. We compare our COOKBOOK-tuned models to the base model via two baselines: zero-shot and few-shot (where k=3) using natural language task samples as in-context examples. Zero-shot involves directly evaluating the base model's ability on a given task, while few-shot primes the base model with task-specific formatting and context.Table 2 shows the results for GPT-NEO-1.3B. We defer results for CEREBRAS-GPT-1.3B to Table 9 in Appendix G.5.2. Our results show that (1) COOKBOOK-NEO outperforms the zero-shot and ICL performance on 6/7 tasks, (2) performance of COOKBOOK models can be 52.7 points better than base zero-shot performance, suggesting that COOK-BOOK-generated data can help the model learn a task it was previously unable to do, (3) similar trends can be seen across model families (see Appendix G.5.2). 
analysis of cookbook: 
 First, we propose and measure the template alignment statistic on several tasks to validate that the rules taught by templates are responsible for improved task performance. Then, we analyze the key components of COOKBOOK to better understand its effectiveness, finding that 1) the extent of NL knowledge the base model has impacts COOKBOOK's performance; 2) training on data from one COOKBOOK template can improve on multiple tasks; 3) rules, not only the instruction formatting of samples, are necessary to the performance of COOKBOOK.We introduce the template alignment scorer, which measures how similar a NL sample and a template are. We use this to define the template alignment statistic, which measures how better adherence to the template's rule is responsible for improved performance on the task.Template Alignment Scorer Given a task T and its corresponding template G T as constructed in Section 3, we propose a template alignment scorer s G T : X × Y → [0, 1]. The template-specific scorer captures how well the input sample's relationship between x and y follows that of G T 's data generating functions. We provide three examples below:• Document QA: the template selects a random span of a document as the question and defines the answer as the surrounding span. s G T scores a sample by finding where the answer is located in the document and computing the fraction of words in the question that are near the answer (Fig 1 ). Note that template-generated samples have a score of 1.• Commonsense Reasoning: the template generates two answer choices, with only one choice containing tokens from the sentence. s G T scores a sample by finding the words that are unique to the two choices, measuring the minimum normalized embedding distance between each choice's unique words and the words in the sentence, and computing the absolute value of the difference between the two choices' distances to the input sentence. This quantity is large if one answer choice has much higher word similarity to the input sentence than the other. Note that when we use the hamming distance, template-generated samples have a score of 1.• Entity Matching: the template first generates one entity randomly, and then generates the second entity to have some overlap with the first one. s G T scores a sample by computing the word overlap between the two entities. Note that template-generated samples have a score equal to 1 -noise when the entities are equivalent. (G T ) = {(x, y) ∈ D T : f (x) ̸ = y, f G T ,n (x) = y}be the set of samples that f gets incorrect and f G T ,n gets correct, and letEQUATION) ̸ = y} be the set of samples that both f and f G T ,n get incorrect. We now define the template alignment statistic, which characterizes how different the template alignment scores are on samples that the COOKBOOK-tuned model gets correct versus incorrect.Definition 1. Let S + T (G T ) = {s G T (x, y) ∀(x, y) ∈ D + T (G T )} and S - T (G T ) = {s G T (x, y) ∀(x, y) ∈ D - T (G T )} be the alignment scores over D + T (G T ) and D - T (G T ), respectively. Define F + T,G T : [0, 1] → [0, 1] as the empirical CDF over S + T (G T ) and similarly define F - T,G T. Then, the template alignment statistic for task T, template G T isA(T, G T ) = sup s∈[0,1] |F + T,G T (s) -F - T,G T (s)|.(2)Note that A(T, G T ) is the total variation distance between the empirical distributions of the template alignment scores for D + T (G T ) and D - T (G T ), as well as the Kolmogorov-Smirnov test statistic for the two-sample hypothesis test on whether the scores for these two subsets of D T come from the same distribution. This statistic is thus bounded between 0 and 1. A large value for this statistic suggests that there is significant difference between improved and non-improved samples in terms of how they adhere to template rules.Template G T / Task T TEMPLATE-MISTRAL A(T, G T ) PIQA / COMMONSENSE-SELECT (0.867, 7.1 × 10 -46 ) DBLP-ACM / MATCHING(1.0, 0.0025) ) TYDIQA / DOCUMENT-QA (0.615, 0.013) Results. We compute the template alignment statistic for the MISTRAL-7B-template models on the tasks TYDIQA, PIQA and DBLP-ACM using the DOCUMENT-QA, COMMONSENSE-SELECT and MATCHING templates respectively. Note that for PIQA, we use Sentence-BERT embeddings (Reimers & Gurevych, 2019) to compute embedding distance between the words in the answer choices and the words in the goal. We find that the template alignment statistic for each task (Table 3 ) is statistically significant (< 0.05), suggesting that rules taught by the templates are responsible for improvement on downstream NL tasks.When do random tokens work? We hypothesize that fine-tuning on random tokens helps learn a NL task when the model already has sufficient NL capabilities. To test this, we evaluate COOKBOOK on PYTHIA-1B checkpoints (log scale from 0 to 144K) as the base models, where later model checkpoints have more sufficient NL capabilities. We do this on PIQA, ITUNES-AMAZON, SQUAD, WINOGRANDE, and their respective templates, finding that there exists a checkpoint at which COOKBOOK starts to help performance significantly, before which it has little effect (see Figure 4 in Appendix H). This suggests that the generalization from random tokens to NL is dependent on the base model's NL capabilities.We compare the overfitting of rules taught over random tokens versus rules taught over NL tokens. We fine-tune models on the COOKBOOK MATCHING template and on the NL matching task itself (ITUNES-AMAZON), and evaluate them on ITUNES-AMAZON and two other tasks, PIQA and TYDIQA (see Table 11 ). We observe that COOKBOOK-tuning actually improves base model performance across tasks beyond matching (up to 5 points), whereas NL-tuning worsens base performance (by up to 7 points) for all tasks outside of matching. This suggests that skills taught in the random token space help generalize to multiple tasks and overfit less, offering a potential explanation for how COOKBOOK-MIX outperforms standard instruction datasets.Are random tokens all we need? We empirically inspect the importance of the template rules vs. the task format (fmt T ) in improving downstream performance by replacing our rule-based generated input and outputs in fmt T () with random tokens without any structured patterns. Our results show that finetuning on data without rules leads to worsened performance -an average performance drop of 18.3 accuracy points (see Table 12 in Appendix H). 
discussion: 
 In this work, we tackle the challenge of building instruction datasets for improving generative abilities. Via COOKBOOK, we show that it is possible to programmatically generate data that teaches task-specific rules. Moreover, we show how such programmatic data can be mixed to improve performance on many tasks at once. Finally, we propose a method for measuring whether a model has indeed learned a task rule, and whether learning that rule improves downstream task performance. In future work, we seek to further explore how to use programmatic data generation to enable self-improving systems.T Natural language (NL) task that we aim to construct a template for improving.Natural language instruction I ∈ I for task T. Template alignment statistic for task T, template G T which measures the total variation distance in template alignment scores for samples of T that the model fine-tuned on data from G T gets correct versus incorrect (see def. 1). We present seven templates (MATCHING, MULTI-CHOICE-QA, DOCUMENT-QA, ENTITY-DISAMBIGUATION, COMMONSENSE-SELECT, TOKEN-RETRIEVAL, and POETRY GENERATION) and a sample data point generated from each.Data generating template: Question: 593Mr tang moss mash pain Dir Answer: Base Sept 168 '</resh593Mr tang moss mash pain approaches also train their own models to generate instruction data, such as instruction backtranslation and Bonito (Li et al., 2023b; Nayak et al., 2024) . While these can avoid legal issues, it can still be computationally expensive to use an LLM to generate a large corpus of instruction data. In contrast, COOKBOOK's templates generate data without the inference costs of using an LLM.Programmatic data for approximating rules: Programmatic data generation has been primarily studied for the automated labeling of datasets (Ratner et al., 2020; 2016; Hearst, 1992) , where heuristics are used to approximate the true labeling rule. For example, a heuristic for Youtube spam comment classification is to check if the word "subscribe" is in the comment (Zhang et al., 2021) . These works provide programmatic labels for classification tasks given an unlabeled dataset containing inputs only, and they find that training on data with these programmatic labels can produce models that do well on the given task. However, these approaches do not directly extend to our setting, which requires us to construct entire samples for generative tasks-both programmatic inputs and outputs, which are often open-ended generations (e.g. document QA).Prior work has shown the value of synthetic, token-level tasks such as set operations as an alternate to natural language data for LLM pre-training (Wu et al., 2022) . Similarly, other work has demonstrated the efficacy of data with latent structure (e.g., music) in the pretraining phrase (Papadimitriou & Jurafsky, 2020) as well as for tasks such as summarization (Krishna et al., 2021) . These works are similar to COOKBOOK in that they utilize non-language based inputs to improve LLM performance, but unlike COOKBOOK, which directly evaluates the model trained on programmatic data, they all require some fine-tuning on natural language data afterwards. Additionally, programmatic data has be shown to improve classification abilities by using it to train a synthetic reasoning module and compose it with a base LLM (Bhatia et al., 2023) ; however, this approach does not apply to generative tasks.Recent works have explore the use of synthetic tasks for understanding the underlying mechanisms of LLMs (Bricken et al., 2023; Nanda et al., 2022; Ravfogel et al., 2019; Zhang et al., 2022b) as well as for understanding and improving architectures (Fu et al., 2022; Arora et al., 2023; Gu & Dao, 2023; Poli et al., 2023) . Unlike these prior works which seek to use non-language based tasks to understand LLMs, we use such data to improve LLM capabilities.Data mixing An important step in curating training data is to determine the mixture over several given groups of data (e.g., domains such as CommonCrawl, ArXiv, Wikipedia, and Github). Data sampling proportions are a key factor in model performance, with many widely-used LLMs having seemingly arbitrary mixture proportions (Touvron et al., 2023; Team, 2023) . Since it is expensive to use trial-and-error to explore the search space of mixture proportions, some recent works have proposed algorithms for data mixing, where proportions are learned based on model performance throughout training (Xie et al., 2023; Albalak et al., 2023) . Data mixing scaling laws have also been proposed (Ye et al., 2024) . These approaches primarily focus on the pre-training setting and only hold when the training groups of data are the same as the evaluation groups of data, which is not true in our setting of instruction datasets. Skill-It (Chen et al., 2023) proposes an online algorithm for setting proportions when the training tasks are not equal to the evaluation tasks; we note that COOKBOOK-MIX can be considered a non-online instantiation of their approach.First, we describe the full algorithm for mixing data from templates, with or without labeled evaluation data. Then, we demonstrate that the linear assumption on accuracies empirically holds. Finally, we provide a proof of Proposition 1, the optimal expression for p ⋆ .Algorithm 1 presents our approach for how to compute template data proportions. It computes p according to Proposition 1; namely, l COOKBOOK-tuned models-one per Algorithm 1 COOKBOOK-MIX.1: Input: l templates G T , base model f , m downstream task datasets D eval = D eval 1 , . . . , D eval m , n number of training samples, η entropy parameter 2: Fine-tune f on n samples generated from G T i to get COOKBOOK-tuned model f G T i ,n for all templates G T i ∈ G T . 3: if D eval has ground-truth outputs then4: Evaluate each model f G T i ,n on D eval to get Âij = acc( f G T i ,n , T eval j ) for all j ∈ [m], i ∈ [l]. 5: else 6: Set Âij = ESTIMATEACCS( f G T 1 ,n , . . . , f G T l ,n , D eval j), a weak supervision-based method for estimating accuracy without ground-truth outputs, for all j ∈[m], i ∈ [l]. 7: end if 8: Compute σ i = exp( 1 mη ∑ m j=1 Âij ) for all i ∈ [l]. 9: Calculate sampling proportion vector p, where pi= σ i ∑ l k=1 σ k for all i ∈ [l].10: return p. f is then fine-tuned on n samples from templates f G T with proportions p.Algorithm 2 ESTIMATEACCS.1: Input: l COOKBOOK-tuned models ("voters")λ i := f G T i ,n : X → Y for i ∈ [l], dataset without ground-truth outputs, unlabeled evaluation dataset D = {x j } n j=1 . 2: Get predictions {λ 1 (x j ), . . . λ l (x j )} across COOKBOOK-tuned models for each x j ∈ D, i ∈[l], forming votes dataset D λ ∈ Y n×l . 3: Estimate accuracy âi ≈ Pr(λ i (x) = y) for all i ∈ [l] by using Algorithm 1 from Ratner et al. (2019) on noisy votes D λ . 4: return { â1 , . . . , âl }.template G T i -are trained, and then evaluated on all m downstream tasks. Then, the average accuracy across tasks for each COOKBOOK-tuned model is computed, and a softmax (with temperature 1/η depending on the entropy regularization in (1)) is performed over these average accuracies to get p.When evaluating each COOKBOOK-tuned model, computing accuracy on each downstream task is straightforward if ground-truth outputs are available for the task. However, if they are unavailable but the outputs are discrete (e.g., answer choices in PIQA or "yes" or "no" in ITUNES-AMAZON), we can use techniques from weak supervision to estimate accuracies (line 6 of Algorithm 1). Weak supervision involves producing programmatically labeled data by modeling several weaker "voters" using a latent variable probabilistic graphical model over their votes. In particular, weak supervision algorithms fit the latent variable model using predictions from the voters, estimating the voter accuracies. Then, they aggregate the votes based on the estimated accuracies to get a programmatic label. We utilize a popular weak supervision algorithm from Ratner et al. (2019) to get an estimated accuracy for each COOKBOOK-tuned model in Algorithm 2. In particular, we compute a votes dataset D λ ∈ Y m×l by using each COOKBOOK-tuned model to produce predictions on the downstream dataset. This votes dataset is converted into numbers (e.g., "yes" becomes 1 and "no" becomes -1) and is passed in as input to Algorithm 1 of Ratner et al. (2019) . This algorithm uses a conditional independence assumption, which implies a particular sparsity pattern in the inverse covariance matrix of D λ , to create a system of equations that is solved with SGD to yield estimated accuracies.To see if the linearity assumption applies on real data, we consider two templates f G T 1 ,n and f G T 2 ,n , versus a model fine-tuned on a uniform mixture over the two templates, f G T ,n[1/2,1/2] , which we call the mixture model. We re-use the setting described in Section 4.1 (4 templates, 8 tasks from GPT4ALL, fine-tuning the MISTRAL-7B model). We measure two quantities, described below.G T = {G T 1 , G T 2 },• Interpolation property: first, we examine how often the mixed model's accuracy interpolates between the individual COOKBOOK-tuned model accuracies, acc(f G T ,n[1/2,1/2] , T eval j ) ∈ [acc( f G T 1 ,n , T eval j ), acc( f G T 2 ,n , T eval j)] on all downstream tasks T eval j ∈ T eval . We compute how many pairs of G T 1 , G T 2 (out of 6 pairs over the 4 templates) for which this condition holds per evaluation task. Our results are shown in Figure 3 (left), where we find that all downstream tasks have at least half of the template pairs satisfying this interpolation property.• Mixture model deviation: second, we measure the absolute difference between the mixture model's accuracy on a task and the average of the individual COOKBOOK-tuned models' accuracies on the task, |acc(f G T ,n[1/2,1/2] , T eval j ) - avg(acc( f G T 1 ,n , T eval j ), acc( f G T 2 ,n , T eval j ))|.We call this the mixture model deviation, and measure this across template pairs and downstream tasks in Figure 3 (right). We find that most values of this deviation are between 0 and 1, meaning that the mixture model's accuracy is less than 1 accuracy point away from the average of individual accuracies.Based on these results, we do see that there exist some template pairs for which training a model on them results in significantly different performance. This suggests that modeling higher-order interactions among data samples from different templates could help us improve our estimate of p ⋆ , although doing so may result in an optimization problem that lacks a simple closed-form solution.We restate Proposition 1 here for completeness.Proposition 1. Define A ∈ R l×m where A ij = acc( f G T i ,n , T eval j ). Let σ i = exp( 1 mη ∑ m j=1 A ij ) for all i ∈ [l]. Then, the p ⋆ that maximizes (1) satisfies p ⋆ i = σ i ∑ l k=1 σ k for all i ∈ [l].Proof. Recall that the objective we aim to maximize is the expressionEQUATION)whereA ij = acc( f G T i ,n , T eval j) and p ∈ △ l , e.g. ∑ l i=1 p i = 1. The Lagrangian function is thusEQUATIONTaking the partial derivative of the Lagrangian with respect to p i for some i ∈ [l] and setting equal to 0, we getEQUATIONRearranging, we getEQUATIONWe now plug this expression for p i into the equation ∑ l i=1 p i = 1:EQUATIONFinally, substituting exp(λ/η) in the expression for p i gives usEQUATIONIn this section, we outline our methodology for auto-generating COOKBOOK templates using GPT-4.We generate task-specific templates by prompting GPT-4. See below for the prompt passed as input to the GPT-4 model. Notice that the prompt has two main components (1) instructions on how to create COOKBOOK templates and (2) two in-context examples mapping a task description to a data generating function.You are an analyst whose job is to help write data generating functions .Here is the procedure for writing a data generating template : * constructing the inputs : inputs are categorized into a parent input and child inputs ; the parent input is a randomly sampled → sequence of tokens from the vocabulary * constructing the outputs y\ _hat based on the inputs : output is constructed from the parent and child inputs using a function that → approximates the task rule Here is a sample data generating template for the task : document question → answering Description of task : Given a document and a questions , retrieve → information from the document that answers the question .Inputs :---`Document `: random list of tokens sampled from vocabulary ---`Question `: randomly selected span from document Outputs :---`Answer `: span including all tokens +/ -3 tokens before the start and → end of question span Data Generating Template :def qa_template ( min_slen : int , max_slen : int ): document = random . sample ( token_ids ) # input source Auto-generating Templates with External Artifacts We find that our automated template generation approach is capable of generating external artifacts (such as the rhyming dictionary in Section B.1). See more details here.Using our automated template generation approach, we generate templates for two tasks, entity matching and token retrieval. We observe that for entity matching, the generated template exactly matches-in terms of functionality-the hand generated template (in Appendix B.1). For token retrieval, we observe that the generated template is minorly different than the manually generated template (in Appendix B.6) in that the "question" is sampled as a contiguous span (in oppose to a set of randomly sample tokens) from one of the support documents. A GPT-NEO-1.3B model finetuned on data generated by this template (COOKBOOK-NEO-AUTO), has an accuracy of 18.4: 5.8 points better than the base model and within 0.02 points of a model finetuned on data from the manually generated template (see table below where COOKBOOK-NEO is the performance on manually generated templates). In this section, we show that the COOKBOOK framework can be extended beyond the three task families to more creative tasks such as poetry generation.Task. We create a poetry generation NL task wherein the model is prompted to write a poem with a prespecified rhyme structure, say ABAB. We construct 30 topics covering a wide range of subjects (e.g., shoe or cup) that the models are then prompted to generate a poem about. We measure accuracy as how well the model is able to follow the specified rhyme pattern when generating the poem.Our method. We fine-tune the MISTRAL-7B model using a template designed for this poetry task, which we call COOKBOOK-POEM; see Appendix B.7 for the exact template and examples of datapoints. We fine-tune the model for 500 steps with a batch size of 64 where each batch uses fresh samples from the templates. Our template uses a dictionary of rhyme words, but does not use any examples of complete poems.Results. When prompted to generate a 4 line poem with an ABAB rhyme scheme (given 2 in-context examples), base MISTRAL-7B has 20% accuracy, COOKBOOK-POEM has 60% accuracy and GPT-4 has 50% accuracy.We present sample poems generated by COOKBOOK-POEM below.Topic: flower A flower's petals, so delicate and fair Its scent is sweet, a perfume so divine Its beauty's rare, a sight to behold anywhere Its petals fall, a sign of the decline.A bird that soars above the clouds so high Its wings are strong, its flight is free It's not afraid to try And soars above the sea Topic: deer A deer, a female deer, a lovely doe Its antlers are a symbol of its pride Amidst the trees, it knows just where to go To find the food that's on the other side.A galaxy of stars, so far away Their light, a beacon, in the night A cosmic dance, a celestial display A wondrous sight, a sight so bright.We provide additional experimental details for both the multi-task evaluation and singletask evaluation settings.For finetuning our COOKBOOK-tuned models, we ran a hyper-parameter searches, sweeping across learning rate ([4e -06, 5e -06, 8e -06, 5e -05, 8e -05]), batch size ([8, 16, 32, 64] ) and total training steps ([100, 200, 300, 400 and 500] ). Table 5 : Model performance on the GPT4ALL benchmark. "CB*" denotes our COOKBOOK tuned models where "wsl" is aggregation with labeled evaluation data, "ws" is aggregation without using labels (our COOKBOOK-MIX algorithm) and "uni" is a uniform mixture, "NH" denotes NousHermes Dataset, "cap" is the Nous-Capybara model, "OH" is Open-Hermes, and "orca" is OpenOrca. Averaged across tasks, CB-MISTRAL +WSL and CB-MISTRAL +WS are the best performing models.GPT4ALL (NomicAI) is a standard evaluation benchmark which covers 7 tasks: ARC-Easy (Clark et al., 2018) , ARC-Challenge (Clark et al., 2018) , PIQA (Bisk et al., 2020) , Winogrande (ai2, 2019), BoolQ (Clark et al., 2019) , Lambada OpenAI (Radford et al., 2019) , OpenBookQA (Mihaylov et al., 2018) and HellaSwag (Zellers et al., 2019) . Table 6 provides additional experimental results around our approach for mixing data from templates. For COOKBOOK-tuned models that use multiple templates, our main method (COOKBOOK plus data proportions obtained using WS on data without ground-truth outputs) is referred to as CB-WS, (previously COOKBOOK-MIST in Table 1 ). Furthermore, COOKBOOK-WSL shows the results when we use the ground-truth outputs in our evaluation datasets; while this method has slightly higher average accuracy, CB-WS is able to come close despite not having any output information. COOKBOOK-UNI shows the results on a uniform mixture of template-generated data, which performs worse than both previous methods. For all single-task evaluations found in Table 2 and Table 9 , we evaluate on 1K examples from the tests sets, with the exception of PIQA, SQUAD and TYDIQA where we sample from the validations sets because they either don't have test sets (SQUAD, TYDIQA) or the test sets are unlabeled (PIQA). We run our evaluations across three different test sets, generated using three separate random seeds.Single-task finetuning results for the CEREBRAS-GPT-1.3B model can be found in Table 9 .Below, we outline additional experiments conducted to better understand COOKBOOK.We evaluate transferability of rules learned over random tokens by evaluating the perfor- language data samples. Concretely, for template-generated evaluation data over natural language samples, we do not use random tokens but instead apply the input-output rule to natural language. For example, for PIQA and COMMONSENSE-SELECT, a sample could be "Select the choice which best completes the sentence.\nWhen boiling butter, when it's ready, you can \nChoices: \n-glum fray butter boil ready \n-glum fray here crest soar wig". By measuring COOKBOOK's performance on these tasks, we isolate the random token to NL transfer aspect and eliminate additional noise caused by the gap between imperfect rules to true task reasoning. We evaluate COOKBOOK-tuned MISTRAL-7B models on COMMONSENSE-SELECT, MATCHING, and COMMONSENSE-SELECT template rules applied on PIQA, ITUNES-AMAZON, and TYDIQA data, respectively. Our findings show that the rules learned over the random token distributions do transfer to natural language (see Table 10 ), improving over base performance by up to 29.8 points. Figure 4 shows the effects of pre-training duration on the efficacy of COOKBOOK. Our results indicate that models that have a better understanding of NL (models that are trained longer) have more performance gains from COOKBOOK. Table 12 shows the results of fine-tuning on a data generated from templates with a format but no rule (i.e., random tokens are used as inputs and outputs without any pattern). We study the extent to which training on more COOKBOOK data impacts model performance.We run the single-task evaluation from Section 4.2 for 1000 steps (2-10X times more data). We find that there is slight degradation for 3 tasks, while performance on SQUAD improves TYDIQA with more COOKBOOK data.  
 
 Appendix:  


Decision: The authors consider a framework of generating training data for LLMs by using Python functions, showing that the resulting LLMs trained by such instances outperform their counterparts using other data such as standard SFT data and LLM generated data. 

The idea is interesting. The main criticism comes from the fact that the three tasks listed in the main submission rely on certain lexical patterns, which might not represent a wide variety of downstream tasks. The authors discussed this fact int he rebuttal, referring to a very different task in the supplementary material. Some reviewers questioned the selection of LLMs for testing, and the author presented additional results. Overall, the rebuttal was convincing to most reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduces COOKBOOK, a new framework designed to boost the generative abilities of LLMs with programmatically generated training data. Unlike recent methods that rely on costly human-curated or LLM-distilled datasets, COOKBOOK uses Python functions to create data templates that could generate random token samples as the instruction-tuning instances. These templates teach models specific pattern-based rules, making the data generation process scalable and free from privacy concerns.

The authors show that models fine-tuned with COOKBOOK could significantly improve their performance on various NLP tasks. Besides, the COOKBOOK-MIX algorithm can select data from multiple templates to further enhance performance. Experimentally, the results are impressive. Models fine-tuned on COOKBOOK beat other instruction-tuned LLMs on several benchmarks. The authors also conduct in-depth analyses to show that why templates based on random token sequences can improve model performance.
strength_weakness: strength_weakness: 1. The baselines (instruction-tuning datasets) compared by the authors seem to focus primarily on open-ended conversations, while the tests mainly involve traditional NLP tasks. Including more instruction-tuning datasets on these NLP tasks like FLAN could make the experiments more convincing.
2. The experimental descriptions in the paper are sometimes too brief, making it hard to follow. For instance, the sizes of the different datasets used (both baseline and COOKBOOK) and various training details are either omitted or scattered across the Appendix.
3. I find the design of the QA-related templates unclear and not well-motivated. For example, for Document QA, why is the answer set to the k tokens before and after the question? And why is k set to 3 (Appendix B.3)?
questions: questions: I have a question regarding data size. Intuitively, if too much unreadable data is introduced during the SFT stage, it might potentially degrade the model's natural language understanding capabilities. Did the authors conduct experiments on this? Although the proposed method can generate unlimited data scalably, could introducing too much of this data gradually impair the model's overall generative ability?
paper_summary: paper_summary: This paper presents an approach, generating instruction dataset prgramatically, in stead of relying on human or LLM based instrcution dataset. Three downstream tasks were selected, where the process of generating datasets were detailed. The paper also indicated that the programmatic data generation contributes to privacy by operating over a random token space, which avoids privacy and legal concerns as it does not involve the use of real user data or violate the terms of service of LLM providers. The evaluation is carried using accuracy  (single and mixed - multitaks setup). The paper is easy to read. However, the way the datasets are selected prgramatically, for example for the QA, which part of the document and what kind of question would be included is not clear. It is mentioned that LLMs are involved in the process but I could not understad the process. Moreover it is not clear or checked how the approach solve the privacy issue.
strength_weakness: strength_weakness: - The theoretical appraoch of generating the dataset is sound, but practically, how can we generate (or select ) the dataset from the source is not clear
- The distiniction from the LLM based instruction dataset generation is not clear
- A lot of important content are presented in the Appendixes
questions: questions: You have a lot of concept in the appendix which makes the paper hard to understand.
paper_summary: paper_summary: The paper introduces Cookbook, a framework designed to generate instruction datasets that enhance large language model (LLM) performance without relying on human or LLM-generated samples. Instead, Cookbook programmatically creates training data using simple patterns over random tokens via Python functions. It generates training data to help the model learn explicit rule-based patterns for specific tasks. The authors also propose a method to mix data from different templates to improve performance across various tasks simultaneously.
strength_weakness: strength_weakness: **1. Limited application scenarios**: Based on the examples in Figure 2 and my analysis, the program appears to generate very preliminary span-level or lexical-level training data. It is likely due to the ease of using Python string operations for synthesis. However, this approach seems less effective for generating more semantically complex QA questions or broader user instructions, as Python code alone may struggle to manipulate question semantics effectively. Also, the authors focus their model testing exclusively on QA-related tasks, rather than exploring a wider range of more general tasks that would better demonstrate the framework's broader applicability.

**2. Lack of more in-depth evaluation**: The authors use code to generate training data but overlook other baselines that also facilitate QA data generation. At least, the authors could conduct experiments using Self-Instruct methods to generate QA training data and verify whether the proposed approach truly surpasses Self-Instruct. In my view, particularly for QA tasks, Self-Instruct-style methods appear better suited for generating a broad range of QA questions, irrespective of domain, text length, or format. Moreover, the baselines currently compared to Cookbook weren't specifically trained on QA-style data, potentially contributing to their lower performance relative to Cookbook.

**3. Weird performance linear assumption**: The authors introduce a method to control template proportions in training data based on the assumption that the accuracy of a model trained on a weighted mixture of templates is equivalent to the weighted average accuracy of models individually trained on each template. However, this assumption may be flawed because models specifically trained on one template (e.g., A) tend to perform better on corresponding template-based QA tasks than models trained on a more diverse mixture of templates (e.g., A, B, C, D). In this case, B, C, and D would be out-of-distribution data for the A-based test set. The authors need to conduct experiments to validate this assumption before making claims about its feasibility.
paper_summary: paper_summary: This paper introduces an automatic framework called Cookbook to construct instruction tuning data without the involvement of human-written data and LLM-generated data. Cookbook generates instruction-tuning data according to programmatic templates of pattern-based rules, and applies the rules to a random token space. To improve performance on multiple tasks, the authors also propose a mixing algorithm over Cookbook templates called Cookbook-Mix. Experiments show that Llama-2 and Mistral finetuned on Cookbook-Mix generated data outperform the base models on the GPT4All benchmark. Analysis shows that the rules taught by templates are crucial to model performance.
strength_weakness: strength_weakness: 1. All the rules designed for tasks in the main content of the paper are based on token overlapping. This may limit the scope of tasks to which this method can be applied, and may teach models to learn shortcuts. For example, the choice that overlaps more with the question is not necessarily the correct answer. The authors acknowledge the restriction and add the task of poetry generation in the appendix, but the template needs more human effort like designing the rhyme_dict, and is unlikely to be generated automatically. 

2. The experiments could be further supplemented to make the claims more convincing. 

a) The framework could be evaluated on more benchmarks besides GPT4ALL, like Open LLM Leaderboard and BigBench-Hard. More instruction-tuned Llama-2 and Mistral models report their performances on these benchmarks.

b) The instructed versions of Llama-2 and Mistral (Llama-2-chat and Mistral-Instruct) could serve as baselines.

c) For the single-task evaluation, the performance of the 7B models could also be reported.

Comments: title: Response to the authors
comment: Thank you for replying to my concerns!

It addresses part of my questions. I'm more convinced by the strong performance and some empirical validation for the assumptions. Therefore, I will raise my score.
Dear Reviewer 5LFp,

We appreciate your valuable feedback and suggestions. Thank you for raising your score. Please let us know if you have additional questions!
Dear Reviewer G5a9,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we provide clarifications on the broader application of Cookbook (demonstrating its ability to expand to creative tasks such as poetry generation), expand our evaluations to include Mistral-Instruct and Self-Instruct (showing that Cookbook outperforms both of these), and provide empirical evidence as to why our linear assumption holds.

As the discussion period ends on Thursday, we are happy to answer any additional questions you may have until then. If our response and additional results have addressed your questions and concerns, we would be very grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.
Dear Reviewer HqBe,

We greatly appreciate your valuable feedback and suggestions! To address your concerns we provide empirical evidence as to why the heuristics used in our templates lead to generalized reasoning vs. learning shortcuts, demonstrate how our approach for automated template generation can expand to more complex tasks (i.e., templates for poetry generation that require generating rhyming dictionaries), and expand our experimental baselines to include Llama-2-chat and Mistral-Instruct (which the best Cookbook model outperforms).

As the discussion period ends on Thursday, we are happy to answer any additional questions you may have until then. If our response and additional results have addressed your questions and concerns, we would be very grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly
Dear reviewer cdUR,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have provided clarifications as to how we generate Cookbook templates in an automated manner and how we select templates for downstream tasks. We have also provided more details on the key distinctions between Cookbook and LLM-based datasets, demonstrating empirically that Cookbook is more scalable and cost-effective.

As the discussion period ends on Thursday, we are happy to answer any additional questions you may have until then. If our response and additional results have addressed your questions and concerns, we would be very grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.
Thank you for replying to my concerns and adding the experiments.

However, I may have a different view of shortcuts. Although shortcuts may help models perform well in benchmarks, they are spurious correlations or unwanted biases in the benchmarks. Models that rely on shortcuts may show overestimated capabilities, and could be easily attacked.

[1] Friedman, Dan, Alexander Wettig, and Danqi Chen. "Finding Dataset Shortcuts with Grammar Induction." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[2] Sun, Zechen, et al. "Exploring and Mitigating Shortcut Learning for Generative Large Language Models." Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024.