Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024)

Abstract: Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, highquality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is laborintensive. Recent work has studied promptdriven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, DataTune, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of languagebased tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49% and improves over existing methods that use synthetic or retrieved training data by 34%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community.1

heading: 1 Introduction
text: The major bottleneck in modern AI research is data. Despite the paradigm-shifting developments of pretraining and prompting, the recipe for achieving peak performance on any particular task has hardly changed: obtain large amounts of high-quality training data and fine-tune your model. This is particularly valuable for developing models with fewer than 3 billion parameters; in this regime,
∗equal contribution. 1https://github.com/neulab/prompt2model
supervised finetuning can be significantly more effective than in-context learning (Mosbach et al., 2023). However, this recipe can be challenging for specialized or novel tasks where task-specific annotated data is limited.
Prior works have proposed strategies for finetuning in this low-resource setting. The most intuitive way is obtain labeled data through manual curation (Callison-Burch, 2009; Zhang et al., 2022), which assumes access to domain experts and may require considerable financial resources to com-
6453
pensate annotators fairly (Huang et al., 2023). An increasingly popular alternative to large-scale manual annotation is to synthetically produce datasets using existing large models (Wang et al., 2022; Peng et al., 2023; Tang et al., 2023). We will refer to this method as synthetic data generation. Unfortunately, directly generating synthetic datasets that simultaneously have high correctness and sufficient diversity is difficult. Models trained on synthetic datasets are typically significantly worse than those trained on manually curated data when standardizing for dataset size, suggesting that current synthetic dataset generation methods still leave significant room for improvement in dataset quality (Ding et al., 2023).
On the other hand, cross-task transfer entirely sidesteps the need for strictly in-domain data by training models either in a multi-task fashion on a wide range of datasets (Sanh et al., 2021) or on taskspecific datasets closest to the new target task (Vu et al., 2020). We focus on the latter, which we refer to as existing data. Recent methods such as Prompt2Model (Viswanathan et al., 2023b) combine both of the above methods, showing the additive benefits of synthetic data generation and existing datasets for finetuning in few-shot settings. However, they report that even for the machine reading question answering dataset of SQuAD (Rajpurkar et al., 2016), a dataset considered today to be largely “solved”, training on existing data alone is significantly worse than training on manually or synthetically created data.
To overcome the limitations of these approaches, we introduce DataTune, a system that automatically repurposes public datasets for new tasks. DataTune identifies the most relevant public datasets and uses a large language model to transform them into a format aligned with the target task’s needs. For instance, for a task requiring descriptions of Python code in English (as shown in Figure 1), DataTune finds a dataset with programming questions and code solutions (along with other, irrelevant data columns). It transforms the retrieved dataset by using the code solutions as input and generating a synthetic description of the code as output. This approach maintains the original dataset’s diversity while matching the task specification, boosting performance by 22 points over baselines. We refer to this synthetic adaptation of publicly available datasets as Dataset Transformation.
We evaluate the effectiveness of DataTune on six
challenging language-based tasks from the BIGBench benchmark (BIG Bench Authors, 2023), which are designed to gauge the system’s performance across diverse NLP task categories. When compared to few-shot prompting of the same base model (Mistral-7B) without fine-tuning, DataTuneenhanced models improve by an average of 5.2 points on five tasks, demonstrating its value for domain-specific fine-tuning. DataTune also can be used additively with existing synthetic dataset generation approaches, yielding an 8-point improvement over the few-shot prompting baseline. Comparing DataTune with existing methods of synthetic data generation, we find that DataTune often produces more difficult and diverse examples, and on a small sample of data we observe that these benefits do not come at the expense of data correctness.

heading: 2 Problem Setup
text: We study the problem of how to automatically retrieve and transform existing datasets to prepare a fine-tuning dataset for a new task. In our problem setting, a user specifies the task of interest with a textual prompt T (optionally containing a few demonstration examples). We assume access to a large, diverse collection of labeled datasets D, a large language model that can be prompted, and a small language model M that we can fine-tune.
The goal here is to automatically generate a synthetic training dataset D′ which, after finetuning M on D′, will improve M ’s ability to satisfy the task specification. For each task T in a known set of task descriptions T , we can measure our progress towards this goal by evaluating the trained model MT against labeled data obtained for task T .

heading: 3 Methods
text: DataTune focuses on selecting and transforming a dataset to align it with a specific task. First, it finds relevant datasets from D that are good candidates for further transformation, through dataset retrieval and reranking. Then, it performs data transformation (i.e., synthetically modify each entries in the selected dataset) to create a new dataset D′ that better aligns with the task requirements.

heading: 3.1 Dataset Retrieval
text: The first phase of DataTune involves identifying relevant datasets for our task from a large repository of existing datasets. We confine our dataset space
D to the HuggingFace Hub (Lhoest et al., 2021), which consists of over 75,000 datasets.
To efficiently complete this task, we use a dual-stage retrieval approach. We first retrieve a set of documents using DataFinder (Viswanathan et al., 2023a), a bi-encoder retriever specifically trained for retrieving datasets from natural language queries using textual descriptions of each dataset. We then rerank these datasets to increase the likelihood that our selected dataset can be effectively transformed for the target task. Inspired by Sun et al. (2023), who showed that LLMs can effectively rerank documents, we similarly use LLMs for reranking datasets. While we use only descriptions of the dataset to generate our initial candidate datasets, this is often inadequate for localizing to the best dataset for a given task. For instance, for a math-based task which have multiple choice questions (MCQs), the dataset descriptions of both GSM8K2 and math_qa3 make them valid choices, but it is only when we look at the schema and sample rows of math_qa (which has multiple choice options) do we see that it is a better choice for our task than GSM8K (which has open-ended questions). Thus we also provide additional dataset attributes such as its schema and a small sampling of rows, to aid with the reranking. The reranking step concludes with the name (and any versions) of the chosen dataset DR. We include the reranking prompt we used in Appendix. A.1. Additionally, to improve the reliability of dataset chosen, we employ self-consistency decoding (Wang et al., 2023), where we run the reranker multiple times, and choose the most frequently re-
2https://huggingface.co/datasets/gsm8k 3https://huggingface.co/datasets/math_qa
turned dataset. It is also possible for no suitable dataset being found, acknowledging that not every task has a relevant dataset.

heading: 3.2 Dataset Transformation
text: Having selected an appropriate dataset DR from D, we now focus on tailoring it to the task requirements to create D′. This transformation process can include modifications ranging from adjusting the input/output format to more closely match that of the few shot examples to more substantial changes like generating new fields that the present dataset may not currently have.
To accomplish this complex task, we split it into two primary steps. First, the Planning Module generate a multi-step plan illustrating the sequence of transformations needed to convert examples from DR into the desired output. Next, the Execution Module executes the plan on each datapoint in DR to create our resulting synthetic training dataset D′.
Empirically, we notice that the Planning Module is more effective when given descriptions of the source dataset (DR) and target task (T ) that are detailed and specific. As such, we further implement two more scaffolding modules — a Schema Selector that removes irrelevant columns from dataset DR (providing a clearer source description) and a Task Expansion Module that enriches task descriptions T with requirements (giving a better target specification).
Below, we describe the four modules in their execution sequence (examples in Figure 2).
Task Expansion The brevity of task descriptions often hampers the creation of an effective plan, and often the subtleties of the task are more readily
grasped through the examination of examples.4 To address this issue, we implemented an intermediate step wherein the LLM is utilized to scrutinize both the task description and provided examples and generate an expanded version of the task description. This enhanced task specification helps in devising a more detailed and actionable plan, encompassing more explicit steps tailored to the task at hand. The task expansion prompt is linked in the Appendix A.6.
Schema Selection The schema selection component is designed by instructing LLMs to identify the most pertinent columns within a dataset for a given task. For example, for the task of code comment generation, where we have a dataset of internet code, the code snippet column is extremely useful, whereas the URL column code would not be useful. To identify these relevant columns, we provide the LLM with detailed information, including the task description T , the chosen dataset DR, the names of existing columns in DR, and samples of dataset rows from DR. The LLM is then tasked with identifying which columns are relevant to the specific task T . This approach ensures a targeted selection
4This is akin to requirement elicitation in Software Engineering, where a deeper understanding of requirements is achieved through further analysis (Lamsweerde, 2009).
of dataset features that are directly applicable to the task requirements, optimizing the dataset DR for the intended application. The schema selection prompt is linked in the Appendix A.2.
Planning Module Now, the LLM generates a comprehensive plan to adapt each data point from the retrieved dataset DR to the task requirements. This plan is a series of concrete, concise steps. These steps may include combining data fields, elaborating on existing fields, or creating new fields derived from the existing data or excluding the data sample altogether if it is irrelevant. The LLM is provided with the expanded task description, along with the optimized dataset, complete with its description and sample rows. A sample plan is shown in Figure 3. We are only required to create a plan once per task, since the same task-level plan will get executed for each row of the retrieved dataset. The planning module prompt is linked in the Appendix A.3.
Execution Module The execution of the transformation plan for each dataset sample from DR is done using an LLM. The LLM receives three key pieces of information for each row of the dataset: the row itself, detailed specifications of the input task, and the transformation plan formulated earlier. Conditioned on these pieces of information, the
model responds by producing an adjusted dataset row which, hopefully, meets the requirements of the input task. The execution module prompt is linked in Appendix A.4. The adjusted dataset rows form our synthetic dataset D′ used for fine-tuning.

heading: 3.3 Using Multiple Datasets
text: The transformation process of data points may result in a significant number of them getting filtered out. Additionally many datasets may be small to begin with. Both these factors can result in a reduced quantity of transformed data. To balance quantity and quality, we adopt a strategy of transforming multiple highly ranked datasets until we reach our desired dataset set. Furthermore, if a considerable proportion of data sample transformations fail (e.g. noisy data samples in the dataset) for a specific dataset, we opt to exclude it from consideration.

heading: 4 Experimental Setup
text: Evaluation procedure We evaluate DataTune and related baselines over 6 tasks from the BIG-Bench benchmark (BIG Bench Authors, 2023), which we list in Table 1. BIG-Bench focuses on challenging tasks believed to be beyond the capabilities of current language models, covering a variety of task categories. We choose tasks to simulate real-world examples, which span over question answering, logical reasoning, coding, math, causal reasoning, multilinguality, and domain specific tasks. For each task, we create 3000 data points through DataTune, and apply the training procedure highlighted earlier. We evaluate all models for two-shot performance, as per the BIG-Bench testing suite.5
Methods We compare the effectiveness of various dataset collection strategies, by using the resulting data to finetune the same base model,
5https://github.com/google/BIG-bench
Mistral-7B (Jiang et al., 2023), a leading opensource LLM within its size category. As shown in Table 2, this includes: (1) retrieving existing data using dense retrieval (Viswanathan et al., 2023a), (2) generating synthetic data, (3) Prompt2Model (Viswanathan et al., 2023b) (a stateof-the-art method that combines retrieving existing data and synthetic data generation), (4) our DataTune approach, and (5) a combination of DataTune and synthetic data generation, which represents an integration of all the existing methods.
We also include two prompting baselines: (6) few-shot prompting on the base model Mistral-7B, and (7) GPT-3.5-Turbo, a significantly larger model, which we include as a robust benchmark due to its extensive capabilities.
Dataset Creation Setup Our initial retrieval step using DataFinder retrieves 25 candidate datasets that are processed for reranking. We transform upto 4 datasets per task until we meet our desired set of 3000 data points. The LLM used for all components is GPT-4-Turbo, except for the final execution step, which uses GPT 3.5-Turbo.6 The cost for each task is detailed in Appendix B.
Training Setup We used the Mistral-7B model, following the approach of Jiang et al. (2023), and applied QLoRA (Dettmers et al., 2023) for fine-tuning. The process was carried out over 3 epochs. We select parameters by running 4 runs across two sets of hyperparameters over two values( learning rate: 5e−5 and 1e−4, and QLoRA’s α parameter between 16 and 24 and choose the run with the lowest validation loss at any point. We used the AdamW optimizer (Loshchilov and Hutter, 2017) and set QLoRA r = 8. We conduct our training on 2 NVIDIA RTX A6000 GPUs.
6Our dataset transformation method requires making an LLM query for each instance in the dataset. Therefore, the decision to use GPT-3.5-turbo was made for budgetary reasons.
Metrics Following BIG Bench Authors (2023), we use a normalized aggregate score evaluation that normalizes a multiple choice grade score, such that a score of 0 implies random chance performance, and 100 implies human expert performance. Additionally, a score less than 0 indicates performance worse than random chance.

heading: 5 Results and Analysis

heading: 5.1 Performance Comparison
text: DataTune consistently outperforms few-shot prompting, as well as existing individual data collection methods. From Table 2, we see that fine-tuning our base model on DataTune outperforms the base Mistral-7B model by 6.4 points on average, improving over it in five out of six tasks. We also show that DataTune provides an average improvement of 11 and 2.9 points over fine-tuning on existing data and synthetically generated data respectively.
DataTune’s transformation is complementary to synthetic data generation. It is noteworthy that the combination of DataTune and synthetically generated data results in a marked performance increase. This synergistic improvement yields an overall average score of 44.5 on the BIG-Bench tasks we consider. We provide a more detailed analysis of the synergy between DataTune and existing synthetic dataset generation methods in Section 5.5.
Our system (DataTune + Synthetic Data Generation) outperforms SOTA baselines like Prompt2Model. In order to make a fair comparison with baselines such as Prompt2Model that fine-tune on both existing data and synthetically generated data, we define our system as the base Mistral-7B model fine-tuned on DataTune-created data and synthetically generated data. Our analysis in Table 2 highlights significant differences in performance between our system and Prompt2Model, with our system demonstrating a notable advantage over Prompt2Model five out of six tasks, with an average improvement of 8.3 points. These findings underscore the effectiveness of our system to create quality datasets for fine-tuning across a wide range of tasks and domains.

heading: 5.2 DataTune Impact on Dataset Diversity
text: Sampling diverse yet high-quality examples from a language model is a challenging task. Prior work has shown that the correctness of a synthetic dataset sampled from a language model is inversely corre-
lated with its diversity (Ye et al., 2022). Generating synthetic data directly from a language model (using a method like Prompt2Model) often contains near-duplicate examples.
Does transforming existing datasets reduce the incidence of duplicate examples? Using ROUGEL to determine lexical uniqueness (Lin, 2004; Wang et al., 2022), we determine a sentence s in dataset D′ to be unique up to threshold7 T if maxs′∈D′\{s}ROUGE(s, s′) < T . In Figure 4, we observe that over 50% of examples generated synthetically are near-duplicates for 3 of 5 tasks; in each of those cases, using DataTune instead effectively eliminates this problem.
We observe similar trends with lexical diversity across all dataset creation methods. In Table 3, we measure the number of unique bigrams per generated input example. DataTune significantly increases the number unique bigrams per example (and moderately increases the length of each example) on 3 of 5 datasets.
According to both measures, dataset diversity decreases on Temporal Sequences and Medical Questions in Russian, which are also the two tasks where DataTune fails to improve over training on fully synthetic data. We discuss these two tasks in Section 5.5 and Limitations, respectively.
7We set the ROUGE-L threshold to 0.8 for Code Line Descriptions, where examples are Python snippets, 0.9 for Temporal Sequences, where examples are long English texts, and 0.7 for the other datasets.

heading: 5.3 DataTune Generates Harder Examples Compared to Synthetic Data
text: Synthetic datasets sampled directly from a language model tend to overrepresent with easyto-solve examples (Xu et al., 2023). To test our ability to overcome this issue, we used gpt-3.5-turbo to estimate the difficulty of each example generated from each method (DataTune, synthetic generation, and manual cura-
tion) for four tasks. We wrote a custom difficulty estimation prompt (including in-context examples) for each task; in all prompts, we specify that the LLM should rate the example on a scale of 1 to 5. We outline an example of this prompt in Appendix A.6. As illustrated in Figure 5, we observe that the data from DataTune exhibits a higher level of difficulty compared to synthetically generated data for all datasets other than Medical Questions in Russian. We attribute this increase in difficulty to the added knowledge introduced by the existing dataset selected for transformation, which is often
more nuanced and, consequently, more challenging. In contrast, unconditional generation from LLMs tends to produce examples that are likely to have a high prior (prompt-independent) probability under the language model. The exception to this effect, Medical Question in Russian, can be attributed to the failure of our approach’s transformation plan to generate data in the correct language for this task. We provide more details of this failure mode in Limitations.

heading: 5.4 Does generating harder examples lead to lower label accuracy?
text: We evaluated the label accuracy of generated data for the Code Line Description task by comparing DataTune against synthetic dataset generation. Manually annotating a random sample of 300 data points from each generation method for the Code Line Description Task, we found the label accuracy from DataTune is 88%, compared to 86.6% for synthetic data generation. This comparison suggests that DataTune can produce datasets that are comparable in accuracy to purely-synthetic datasets, despite DataTune generating significantly more diverse and challenging examples on this task.

heading: 5.5 Transformed Data Can Be Complementary to Synthetic Data
text: In two cases where DataTune fails to improve over a synthetic data baseline, Temporal Sequences and Elementary Math, our combined DataTune + Synthetic system still outperforms all other comparable (Mistral-based) baselines. We observe these two approaches to dataset generation can be complementary to each other, yielding additive improvements when combined.
For a concrete example of this, we can visually observe the two-dimensional semantic distribution of questions generated for the Elementary Math task via DataTune, Synthetic Generation, and from the gold dataset in Figure 6. We encoded each question using MiniLM v2 (Wang et al., 2021) via sentence-transformers (Reimers and Gurevych, 2019), then projected each embedding into a 2D space using t-SNE (van der Maaten and Hinton, 2008).
Visually, we observe that the embedding clusters of questions generated via DataTune and Synthetic Generation appear to be largely disjoint. This supports our hypothesis that these two methods systematically cover different distributions of the target
task space, and therefore combining the examples from each method can lead to a synergistic effect.

heading: 5.6 DataTune can be Robust to Data Scarcity
text: Sometimes there are tasks where dataset deemed to be the most relevant by the dataset retrieval system is still not sufficiently relevant. In existing dataset retrieval-based systems where retrieved datasets are used to automatically train a model (such as Prompt2Model), the unrelated training signals triggered by training on an irrelevant dataset can lead to worse performance. In our system, we find that the use of the Planning module can avoid such negative interference. When there are no relevant training datasets available (such as in the "Temporal Sequences" task), we find that the planning module essentially generates new data without conditioning on irrelevant datasets at all.

heading: 6 Conclusion and Future Work
text: We presented DataTune, an improved approach for enhancing automatic dataset generation by transforming existing labeled datasets. Our method significantly outperforms existing automatic dataset generation techniques on several challenging tasks from BIG-Bench. When used in conjunction with existing synthetic dataset generation methods, it achieve superior performance compared to other few-shot learning methods on the base model we used (Mistral-7B). Our analysis reveals that
DataTune not only creates more diverse and accurate datasets but also increases their complexity for fine-tuning. An important direction is whether transformation-based dataset creation methods like DataTune can still be effective when examples are retrieved from the open web rather than from the collections of manually-curated datasets we consider in our work. Another important direction for future work will be to generate code to execute a transformation plan (rather than querying an LLM for each instance of data). Both of these future directions would improve the accessibility and scalability of our suggested approach.
Limitations
We identify four key limitations in our system:
1. LLM Query Cost: Our dataset transformation component requires running the Execution Module on each row of each dataset we wish to transform. Given that our Execution Module prompts a large language model, the number of LLM queries scales linearly with the amount of data to be transformed. This could be cost-prohibitive for transforming very large datasets. This LLM usage requirement could also exclude members of the research community without consistent LLM access from benefiting from our work.
2. Dependence on the Planning Module: Our dataset transformation system relies heavily on the Planning Module to produce clear and comprehensive instructions for the Execution Module to enact. Our Planning Module operates by prompting a large language model. Given that prompted LLMs can behave unpredictably in changes in prompts (Sclar et al., 2023), an important limitation of our method is that its success depends on a large language model following a prompt correctly. We see this failure with Medical Questions in Russian, where the agent is supposed to translate a Russian dataset into English to facilitate the generation of question-answer pairs. When running our benchmarking, the Planning Agent failed to translate the generated question-answer pairs back to Russian. This resulted in a training dataset that, despite being conceptually close, was practically far from correct.
3. Handling Non-English Data: Our transformation agent’s ability to process non-English data tasks is substantially compromised, frequently altering example tasks instead of the actual data. This deficiency is primarily due to the reliance on models like GPT 3.5, which have been extensively trained on English data, thereby diminishing their proficiency with other languages.
4. Dependence on Instruction-Following LLMs: The system’s execution component depends on Large Language Models (LLMs) that are specifically designed to adhere to instructions. We have identified discrepancies in performance among LLMs tailored for instruction-based tasks versus those developed for conversational purposes. This limitation confines our system to using only a narrow selection of LLMs that demonstrate the ability to follow instructions accurately.
Ethical Considerations
DataTune could make it easier for the general public to build custom language models. The broadening of open-ended technology induces ethical concerns, similar to the issues with open-source deepfake libraries described by Widder et al. (2022). While DataTune has potential for misuse, this is likely no greater than the potential harms presented by the underlying open-source large language models. By making it easier to build task-specific language models, we hope that these risks are balanced by the benefits of making NLP models accessible to those outside the NLP community or those without the resources to manually collect labeled data. We aim to be transparent in our documentation about the potential limitations of the system.
The platform is particularly valuable for individuals outside the NLP community who can benefit from using NLP models in their work but lack the specialized knowledge to develop these tools independently. The decision to open-source DataTune invites community contributions, emphasizing a collaborative approach to improve and expand the tool’s capabilities. This strategy not only enhances the system’s utility but also aligns with a broader goal of increasing the accessibility of NLP innovations and fostering a more inclusive technological environment.

heading: A Prompt Appendix
text: A.1 Reranking Prompt
Your objective is to choose the most relevant dataset for a given a task (and few examples of the task). For each dataset, you will be provided with the dataset description, and tags related to the dataset. Please return the most relevant dataset, e.g., squad The following is the task {{instruction}} and these are some examples of the same: {{examples}} There are {{num}} datasets available for this task. {{datasets}} The name of the most relevant dataset for this task is:
where each dataset in {{datasets}} is defined as:
{counter}{{dataset_name}}:Description-{{dataset_description}}. This dataset has the following tags: {{tags}}
A.2 Schema Selection Prompt
Your objective is to carefully analyze the task and the dataset mentioned, and decide whether the columns are relevant input, relevant output, irrelevant for the given task, or if it is ambiguous. There should be at most one output column. It is possible to have no relevant columns, in which case return the input and output column as empty lists. Answer in a json format, with the following keys: input, output, irrelevant, ambiguous. {{INCONTEXT_EXAMPLES}} After seeing these examples with the required columns, please provide the relevant columns for this context:
You are tasked with the following process. {{instruction}} For this task, you will use the {{dataset_name}} dataset from HuggingFace. Dataset Description: {{dataset_description}} A sample data instance from this is as follows. {{sample_row}}. This dataset has the following columns: {{dataset_columns}} Required Columns :
A.3 Planning Module Prompt
You are a Planning Agent. You create a plan to transform data samples from their existing format into the required format for a given task. ————————————————- Here are some examples for your reference. {{in_context_examples}} ———————————————— Now do the following task: Task Description: {{task_description}} Task Examples: example Here are samples from a potentially relevant dataset for the task above. Notice how the format below is not as required by the task above. Dataset Samples: {{dataset_row}} Carefully analyze the ‘Task Description‘ and the ‘Task Examples‘. Propose a higher-level plan to convert data from the Dataset Sample to data in the required format task examples. Your plan should be a list of sequential steps that can be taken to perform the data transformation. You don’t need to use all columns, as the dataset may not be fully relevant. Keep steps as simple, explicit and concise as possible. Each step in the plan may take any of the following actions: 1. Generate new columns as required by the task, and save them 2. Expand on a particular column to make it something more relevant to the task and save it 3. Combine multiple columns from the dataset 4. Choose columns that will form "input" 5. After the input field is created, carefully analyze it to choose/generate the output field 6. Ignore a data sample because it is not all relevant and return null for them.
Return only the plan.
A.4 Execution Module Prompt
You are a Data Transforming Agent. Your job is to transform data from a given format to the required format. Following are the detailed instructions for the same: 1. Read the ‘Task Description‘. 2. An example of the input and output looks like for the task is shown in ‘Task Examples‘ 3. The sample to be transformed is in ‘Data Sample‘. 4. Read the data transformation plan carefully that will help you convert the ‘Data Sample‘ into the required format. This should be relevant and intune to the ‘Task Description‘ 5. Perform the plan step by step and explain your thinking. 6. End your response with the transformed sample as a JSON response with exactly 2 fields: "input" and "output". ————————————————- Here are some examples for your reference. {{incontext_examples}} ———————————————— Now do the following task: Task Description: {{task_description}} Task Examples: {{sample}} {{plan}} Dataset Sample: {{dataset_row}} Think step by step through the plan to convert the above ‘Dataset Sample‘ and show your working. End your response as a JSON with exactly two fields: "input", and "output" Response:
A.5 Task Expansion Prompt
Carefully analyse the task description and examples of the task, and explain the task to give a clearer description. Do not explain each example, but rather capture the general trends. Also place special focus on the format of the input/output examples. ————————————————-
Task Description: {task description} Task Examples: {examples}
A.6 Difficulty estimation prompt: Code Line Descriptions example
We are building a dataset for automatically describing code (in words). Evaluate and rate the difficulty and complexity of describing the following code lines. You should give an overall score on a scale of 1 to 5, where a higher score indicates higher difficulty. You must just give a score without any other reasons. Here’s the grading scale:

heading: 1: Very easy. Anyone who understands the programming language could describe this almost instantly

heading: 2: Easy. Anyone who understands the programming language could describe this with a bit of thought

heading: 3. Neutral. Most non-expert people who understand the programming language would be able to describe this, but it might take time for them to understand the code

heading: 4. Hard. It would require at least a minute for a non-expert person who understand the programming lanugage to understand and describe this code.

heading: 5. Very hard. Most non-experts would make a mistake when trying to describe this code in a fixed timeframe. Professional programmers would have an easier time.
text: Your answer shoud be a single number, 1 through 5, with nothing else in your response.
{Incontext examples with code and difficulty}
{Input Code}

heading: B Cost Estimation
text: In our pipeline, we utilize GPT-4 for most of the processes, but for the execution module, we switch to GPT-3.5-turbo. This is because the execution module requires a single API call per data sample, while the rest of the pipeline handles fewer than 20-30 requests. For a dataset with 3000 rows, the cost amounts to approximately $9 per task. Practitioners can further reduce costs by opting for more affordable LLM APIs, such as Claude 3 “Haiku,” using locally-hosted LLMs like Mistral 7B, or combining LLM queries to transform multiple examples in a single call.

heading: Future Work
text: Another important direction for future work will be to generate code to execute a transformation plan (rather than querying an LLM for each instance of data). Both of these future directions would improve the accessibility and scalability of our suggested approach. Limitations
We identify four key limitations in our system:
1. LLM Query Cost: Our dataset transformation component requires running the Execution Module on each row of each dataset we wish to transform. Given that our Execution Module prompts a large language model, the number of LLM queries scales linearly with the amount of data to be transformed. This could be cost-prohibitive for transforming very large datasets. This LLM usage requirement could also exclude members of the research community without consistent LLM access from benefiting from our work. 2. Dependence on the Planning Module: Our dataset transformation system relies heavily on the Planning Module to produce clear and comprehensive instructions for the Execution Module to enact. Our Planning Module operates by prompting a large language model. Given that prompted LLMs can behave unpredictably in changes in prompts (Sclar et al. , 2023), an important limitation of our method is that its success depends on a large language model following a prompt correctly. We see this failure with Medical Questions in Russian, where the agent is supposed to translate a Russian dataset into English to facilitate the generation of question-answer pairs. When running our benchmarking, the Planning Agent failed to translate the generated question-answer pairs back to Russian. This resulted in a training dataset that, despite being conceptually close, was practically far from correct. 3. Handling Non-English Data: Our transformation agent’s ability to process non-English data tasks is substantially compromised, frequently altering example tasks instead of the actual data. This deficiency is primarily due to the reliance on models like GPT 3. 5, which have been extensively trained on English data, thereby diminishing their proficiency with other languages. 4. Dependence on Instruction-Following LLMs: The system’s execution component depends on Large Language Models (LLMs) that are specifically designed to adhere to instructions. We have identified discrepancies in performance among LLMs tailored for instruction-based tasks versus those developed for conversational purposes. This limitation confines our system to using only a narrow selection of LLMs that demonstrate the ability to follow instructions accurately. Ethical Considerations
DataTune could make it easier for the general public to build custom language models. The broadening of open-ended technology induces ethical concerns, similar to the issues with open-source deepfake libraries described by Widder et al. (2022). While DataTune has potential for misuse, this is likely no greater than the potential harms presented by the underlying open-source large language models. By making it easier to build task-specific language models, we hope that these risks are balanced by the benefits of making NLP models accessible to those outside the NLP community or those without the resources to manually collect labeled data. We aim to be transparent in our documentation about the potential limitations of the system. The platform is particularly valuable for individuals outside the NLP community who can benefit from using NLP models in their work but lack the specialized knowledge to develop these tools independently. The decision to open-source DataTune invites community contributions, emphasizing a collaborative approach to improve and expand the tool’s capabilities. This strategy not only enhances the system’s utility but also aligns with a broader goal of increasing the accessibility of NLP innovations and fostering a more inclusive technological environment