DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
Authors: Zaid Khan
Elias Stengel-Eskin
Jaemin Cho
Mohit Bansal
Venue: ICLR 2025 Spotlight (2025)
Keywords: iterative data generation
llm agent
lifelong learning
Primary area: foundation or frontier models, including LLMs

Abstract: The process of creating training data to teach models is currently driven by humans, who manually analyze model weaknesses and plan how to create data that improves a student model. Recent approaches using large language models (LLMs) as annotators reduce human annotation effort, but still require humans to interpret feedback from evaluations and control the LLM to produce data the student needs. Automating this labor-intensive process by creating autonomous data generation agents – or teachers – is desirable, but requires environments that can simulate the feedback-driven, iterative, closed loop of data creation. To enable rapid and scalable testing for such agents and their modules, we introduce DataEnvGym, a testbed of teacher environments for data generation agents. DataEnvGym frames data generation as a sequential decision-making task, involving an agent consisting of a data generation policy (which generates a plan for creating training data) and a data generation engine (which transforms the plan into data), inside an environment that provides feedback from a student. The agent’s end goal is to improve student model performance. Students are iteratively trained and evaluated on generated data, with their feedback (in the form of errors or weak skills) being reported to the agent after each iteration. As a general-purpose testbed, DataEnvGym includes multiple instantiations of teacher environments across three levels of structure in the state representation and action space, with varying levels of scaffolding support. More structured environments are based on automatically-inferred skills and offer a higher degree of interpretability and control over the curriculum. We support developing and testing data generation agents in four diverse tasks covering text, images, and actions (mathematics, programming, visual question answering, and tool-use) and test multiple student and teacher models. We find that example agents in our teaching environments can iteratively improve students across diverse tasks and settings. Moreover, we show that environments can teach different skill levels and can be used to test variants of key modules, pointing to directions of future work in improving data generation agents, engines, and feedback mechanisms. Project page: https://DataEnvGym.github.io.

Published as a conference paper at ICLR 2025
DATAENVGYM: DATA GENERATION AGENTS IN TEACHER ENVIRONMENTS WITH STUDENT FEEDBACK
Zaid Khan
Elias Stengel-Eskin
Jaemin Cho
Mohit Bansal
UNC Chapel Hill
(za idkhan, esteng, jmincho, mbansal [es. unc edu
ABSTRACT
The process of creating training data to teach models is currently driven by humans, who manually analyze model weaknesses and plan how to create data that improves a student model. Recent approaches using large language models (LLMs) annotators reduce human annotation effort, but still require humans to interpret feedback from evaluations and control the LLM to produce data the student needs. Automating this inten-ie process by creating autonomous data generation agents teachers is desirable, but requires environments that simulate the feedback-driven, iterative, closed loop of data creation. To enable rapid and scalable testing for such agents and their modules, introduce DATA ENTGYN, testbed of teacher environments for data generation agents. DATAENvGYN frames sequential cerision-making task, involving an agent consisting of a data generation policy (which generates plan for creating training data) and a data generation engine (which transforms the plan into data), inside an environment that provides feedback from student. The agent's end goal is to improve student model performance. Students are iteratively trained and evaluated on generated data, with their feedback (in the form of errors or weak skills) being reported to the testbed, DATAEN includes
data generation
agent after each iteration. As a general-purpoot multiple instantiations of teacher environments across three levels of structure in the state representation and action space, with varying levels of scaffolding support. based automatically-inferee skills and offer
More structured environments a higher degree of interpretability and control over the curriculum. We support developing and testing data generation agents in four diverse tasks covering text, images, and actions (mathematics, programming, visual question answering. and tool-use) and test multiple student and teacher models. We find that example agents in our teaching environments can iteratively improve students across diverse tasks and settings. Moreover, we show that environments can teach different skill levels and can be used to test variants of key modules, pointing to directions of future work in improving data generation agents, engines, and feedback mechanisms. Project page: https:/ /Dataanveyymmm...aaeaa/aa github. io.
INTRODUCTION
Improving an already-trained model by creating additional training data that is targeted towards current model weaknesses is an important and frequent task for researchers and engineers. For example, past work in instruction tuning and aligmment has found that models can be improved with additional task-specific training examples (Touvron et al., 2023; Ding et al., 2023; Zhou et al., 2024; Chia et al., 2024; Wang et al., 2023b; Shimabucoro et al., 2024). However, the current model improvement process is largely driven by humans, who try to identify the weaknesses of the model based on evaluations, use intuition and heuristics to create data to target weaknesses, train an updated model the data, and revise the data based how the new model performs (Iyer et al., 2022; Longpre et al., 2023; Shao et al., 2024). The labor and repetition involved this process strongly motivate the creation of data generation agents that can automate the process of creating synthetic training data to teach student models. However, there are no simulators or environments to serve testbed for the development of such automated teacher agents,
to evaluate the

---

Published as a conference paper at ICLR 2025
Environment
Agent
(a) Evaluation
Student Model Performance #;
(b) Data Generation Policy
Az Eval(
P: w(p(as)
Updated Student Model Mig-1
DataEnwGym Cycle
Data Generation Plan
Pt.
(d) Training
Training Data
(c) Data Generation Engine
mell
Train(m,, de)
Engine(pi)
Figure 1: Overview of DATAENVGYM. novel testbed for data generation agents. The environment (left) consists of evaluation (a) and training (d) of the student model. The data generation agent (right) takes state encoding the current student model's performance and provides training data to improve the student model, by first creating a plan through the data generation policy (b), then executing the plan via the data generation engine (c).
effectiveness of the different components required to automate such an iterative data generation and model improvement process.
We propose DATAENVGYM, a testbed or gym of parameterizable teacher environments for developing autonomous data generation agents (i.e. teachers), whose goal is to iteratively improve student models by generating targeted training data conditioned on representations of the student's weaknesses (e.g. errors, inferred weak skills). We frame this task as an iterative interaction between a teacher agent and a student model in which the teacher agent creates training data that improves the student model. In the same way that game environments can be used to evaluate game-playing agents in conventional reinforcement learning (RL), DATAENVGYS Sec. 2.1) allow to test data generation agents for a given student model. In these environments, the data generation agent (teacher), performs multiple rounds of data creation and receives feedback from the student after each iteration in the form of student performance, which is the teacher's 's reward signal. We provide modules for data generation, training, and evaluation, with final performance being measured by the improvement the student model. An overview of the DATAENvGYM can be seen in Fig. 1. First, the environment provides the agent with state st, containing information about the errors of current student model my (ef. Fig. (a)). Then, the agent's data generation policy predicts actions that constitute a plan for generating training data (cf. Fig. ((b)): Pt ~ i TP (st). Next, the agent's 's data generation engine executes the plan create training data de (cf. Fig. ((c)). The created datapoints are then used to train updated student model (cf. Fig. I(d)): M4+11 Train(m;; de). The updated student model is re-evaluated to produce the next iteration's state 81+1 and provide feedback to the agent (in the form of student performance). DATAENYGYM is designed in a generalizable way to support data creation for diverse agents across multiple tasks, covering multimodal (visual question answering) and text-only (mathematics and programming) tasks.
modular environments (cf.
DATAENVGYM's modular design enables many possible instantiations of data generation environ- ments. We provide three implementations of DATAENVGYM environments along with the agent modules required for each. These differ in the state representations and action spaces they provide the agents, and they range from open-ended (generating data directly from per-example model predictions) to more structured (generating data based on a skill-based hierarchical representation of intermediate model progress). First, the OPEN-ENDED environment (cf. Fig. 2(a)), the state representation is an unstructured list of the student model's errors. The action space in this envi- ronment is also unstructured (i.e. open-ended), with an action consisting of generating particular set of datapoints; i.e., the agent infers directly from the errors what type of data would help the model and then directly generates that data. This contrasts with human developers, who typically use more skill-directed approach, breaking performance down into skill-specific metrics to decide where to add data. Skill- development has three distinct advantages: it provides the agent with structured ways of controlling the data generation process, it makes the process more interpretable by organizing data generation around easy-to- sooy-tt-grspp skills, and it enables human-model interoperability and curriculum control, where human model can specify skills for the model to improve
2

---

Published as a conference paper at ICLR 2025
Based on these advantages, we argue that structrdr agents may be preferable necessary in some cases. Therefore, DATAENVGYM also supports saiel-based teaching and learning. Specifically, DATAENYGYM includes the SKILL-LIST environment (ef. Fig. 2(b)), in which a skill discovery module first automatically infers human-interpretablee skills from the training data using a large language model (LLM).. This produces more structured state representation (i.e., report of skill- specific performance), and makes the agent's 's task more interpretable, as it has explicit feedback on which skills the student model struggling, Like the OPEN-ENDED environment, the SKILL-LIST environment asks agents to directly generate data. While the SKILL-LIST environment provides more structure and interpretability to the agent than the OPEN-ENDED environment by adding skills the state representation, both have granular action spaces with a high degree of freedom. Thus, while the SKILL-LLIST input space is more structured and interpretable, its output space is not. To give the agent more structured output, we also include an environment in which the action space is structured into fixed, pearser-grained actions. To this end, in the SKILL- -RREE environment (cf. Fig. 2(c)), we abstract skills into a tree-based representation called a skill forest. Here, skills are organized hierarchically into skill trees, with parent skills as root nodes and subskills as child nodes (see Fig. for example). This hierarchical framing allows new, more granular subskills to be discovered and simplifies the agent's 's task into a binary choice between two actions: the explore action, which grows skill tree by adding new subskills, and the exploit action, which rebalances the skill tree to allocate more data to existing subskills. This split is designed to help the agent prioritize important skills (by generating more data for skills that have helped improve performance in the past) while also balancing competing pressures for breadth and depth in the skill hierarchy (by adding new subskills and broadening the skill tree). Our DATAENVGYM testhed not only provides default implementations for all these environments and components, but also makes it easy to test alternate implementations; for example, an improved skill discovery implementation or an alternate data structure can easily be plugged in and tested based on downstream student performance. A summary of the input and action spaces for these environments is shown in Fig. 3.
We benchmark several baseline agents as examples (data generation policies combined with data gen- cration engines) in AATAENyGYM 's teaching environments, across different domains (mathematics, programming, visual question answering) and on different student and teacher models. Generally, we find that the example agents we provide already improve student performance when models are trained in DATA ENTYY'ss teaching environments; after training, students see consistent improvement when compared to their starting point (i.e.. before training in the environment). Across environments, students improve by an average of 4.43% (absolute accuracy) on GQA, 4.82% on MATH, 1.80% on LiveCodeBenchh 15.17% on NaturalBench, and 20.81% on MnMs. Moreover, we find that our example agents can make use of student feedback to help iteratively improve the student: we compare baseline agent policies that make use of student feedback states ("With State" policies) to ones that do not ("No State" policies), finding that conditioning on the feedback state key to successful data generation, with "With State" policies outperforming "No State" by 3.5% in OPEN-ENDIE 2.05% in SKILL-LIST, and 1.08% in SKILL-TREE. We also show that some environments make improving students more challenging for teachers, based on how flexible versus how controllable (and interpretable) the curriculum needs to be. Moreover, we show that environments can teach different skills (based on skill frequency and question difficulty) and can be used to test variants of key modules, e.g., the skill discovery module. Lastly, we provide qualitative examples of student model predictions before and after our training. Overall, DATA ENVGYM is seneral-purpose testbed for developing and evaluating data generation agents, engines, and feedback mechanisms, laying the foundation for future improvements these key elements of automated model improvement.
2 DATAENVGYM ENVIRONMENTS AND AGENTS
We provide three categories of (environment, agent) pairs in DATAENVGYM with multiple levels of structure given to data generation agent, corresponding to different levels of interpretability. Agents are composed of two modules: the data generation policy æ (which creates a data generation plan Pt π [pr (s:)) and the data generation engine Engine (which executes the plan to produce training data dN = Engine meinneepp ); cf. Sec. 2.2.2), Both the policy and plan can change depending on the environment the agent is in, as the environment provides the agent with affordances that define the agent's 's action space. The environments encapsulate several modules, including a student model m,, trainer (which trains student model given the generated training data, m++1 = Traí n(m,,d))
3

---

Published as a conference paper at ICLR 2025
only f
Skill Discovery
only (
Skill Discovery
State
Student Porformance (List of errors)
Student Performance apkil-apacific errors)
Skill List
Student Performance ( -Skillspecific errors)
Skill Tree
Data Generation Palicy ps ~ x(pylss)
Agent
Every
Organization (Explore Exploit)
Action
Data Generation Plan
Data Generation Engine
de Engize(pp))
More Training
Train Student Model
Train de)
Environment
Updated Student Model m
Evaluate Student Model
[val(m
Next State #1-1
Student Performence (List of erroms)
Student Performance (Skll-specific errors)
Skill List
Student Parformance (Skill-spes Eskillspeciffc errors)
Skill Tree
Open-ended Environment
(b) Skill-List Environment
(c) Skill-Tree Environment
Illustration of the three example instances of DATAENVGYM environments described in Sec. 2. In the (a) EPEN-ENDED environment, the state is represented list of per-example accuracies, and the data generation policy directly creates a data generation plan from them. In the (b) SKILL-LIST environment, the state is represented model performance; its data generation plan allows the policy to prioritize weak skills. In the (c) SKILL-TREE environment, the state is represented as forest of skill trees containing skill-subskill relational information, and its data generation policy chooses between two actions for each skill: explore (grow skill tree) and exploir (rebalance skill tree).
Figure 2:
categorized list of skills and per-skill student
and an evaluator (which evaluates the updated student model and outputs its performance, 81+1 Eval 1 (m++11: cf. Sec. 2.1.1):
As summarized in Table some environments have additional modules for generating skill-specifft training examples via automatic skill discovery (Sec. 2.1.2) and organization (Sec. 2.1.3). Skill- based structures give agents three distinct advantages: first, they provide the agent with affordances to control how targeted or diverse the data generates (i.e. knobs that adjust to what degree the data addresses single skill vs. broadly improves variety of skills). Secondly, when the skills are interpretable to people, skill-based agents provide human-model interoperability and human-in influence the data generation process (e.g. by specifying skills
the-loop control, where humans to improve) or can in fact step in for the agent (and vice versa) whenever needed. Finally, having skill-based improvements allows for interpretable agent behavior; user can observe for which skills data is being generated and where training is most effective. We provide all of these components for three different tasks: mathematics, visual question answering (VQA), and programming.
(1) ENNDDE Environment. The OPEN-ENDED environment, shown in Fig. 2(a), provides the simplest state structure to the data generation agent, and is the least constrained of the environments. State representation: The state is represented as a list of evaluated predictions from the student. The agent must infer from this list what kind of data would best help the student, mapping directly from errors in the list to desired future datapoints. Action space: The action space that the OPEN-ENDED environment affords directly specifies the datapoints to generate, i.e. the agent expected to directly generate specs for every datapoint, without any auxiliary actions to structure the generation process.
(2) SKI.II-LIST Environment. The SKILII-LIST environment (shown in Fig. 2(b)) requires the teacher, i.c. the data generation agent, to teach specific set of skills. State representation: The SKILL-LIST environment induces skills needed for the task on the training data (see Sec. 2.1.2) and reports student performance each of these skills. The input the agent policy is list of evaluated predictions partitioned by skills. This informs the agent about what mistakes are being made on questions requiring the skill. Action space: The action space is shared with OPEN-ENDED.
(3) SKILL-TREE Environment. The SKILL-TRES environment, shown in Fig. 2(c), disentangles data generation from data control, adding structure the action space s.t. its policy longer
4

---

Published as a conference paper at ICLR 2025
Table 1: Summary of baseline environments for DATAENVGYM, with different components that determine how to generate training examples for each iteration.
Trainer/Evaluttor (See. 2.1.1)
Skill Discovery (Sec. 2.1.2)
Skill Organization
Environments
(Sec. 2.1.3)
OPEN-ENDED
SKILL-LIST
SKILL-TRREE
directly generates data specifications but simply dictates how much data is to be generated and for which subskills. This constrains the action space and provides the agent with additional scaffolding. State representation:
The surface form of the state representation is shared with the SKILL--LSST environment. However, the SKILL-TRES environment also maintains an underlying skill forest composed of skill frees, where each tree is a hierarchical representation of skill and its subskills (see Sec. 2.1.3 for details, see Fig. 3 for an example). Thus, while the input is similar (skill names and the student's performance each skill) the actual skills differ from those give in the SKILL-LIST environment, which does not have any hierarchy. Action space: The SKILL- affords the agent a more structured action space. each iteration, rather than directly generating data, the agent chooses, for each skill (and its corresponding skill tree) to either exploit t the existing skill set by rebalancing the skill tree for an existing skill, i.e., re-allocating the data budget to its subskills, to explore, which grows the skill tree by creating new subskills. The action is applied to the skill tree and produces a new skill tree that has either had new subskills added or had the amount of data allocated to each subskill changed. The data generation engine then consumes each skill tree and produces the planned amount of training data for each subskill within each skill tree.
environment
Below, describe the constituent modules (Sec. 2.1) and the data generation agent (Sec. 2.2) that are instantiated in DATAEN
2.1
ENVIRONMENT MODULES
2.1.1 TRAINER AND EVALUATOR
Given training data d, from the data generation engine, the trainer performs training run (i.e., certain number training steps on the dataset) updating the student model: m = Train(m,, de). Then, the evaluator tests the student model and outputs its performance: 8+41 Eval l (m++1). See Appendix C.1 for implementation details.
2.1.2 SKILL DISCOVERY
SKILL-LIST and SKILL-TRES environments have a skill discovery module that takes a set of training samples d, and returns a set of skills that would be needed to solve these examples; in the beginning of training / = 0, the environments use the skill discovery module to discover a set of skills over the validation set. Alternatively, the environments can be parameterized by set of user-specified target skills. The skill discovery module will assign discovered skill label to each evaluated student prediction in the validation set. The list of skills and evaluated student predictions are consumed directly by the SKILL-LIST environment. In the SKILL-TRED environment, skills are used as input by the skill organization module.
Baseline implementation. module employs two-stage approach, following Didolkar et al. (2024). First, we assign a specific skill to each instance of task, using template that asks the LLM to identify the high-level skill required to solve the question. Second, we aggregate these skills into categories using another template that asks the LLM to group similar skills.
To discover the skills, our baseline instantiation of the skill discovery
In the SKILL-TREE environment, we use the same process to propose subskills for existing skills in hierarchical fashion. For example, given the skill "Algebra", subskills might be "Solving Linear Equations" or "Polynomial Factorization". Implemennatiionllly we provide the LLM with a skill and existing subskills for that skill. It is instructed to propose subskills that are unique and belong the given skill. We can repeat this process again on top an existing skill to induce a set of subskills for each skill, needed. The L.M prompts are shown in Appendix C.2.
5

---

Published as a conference paper at ICLR 2025
Explore
Explore
Exploit
Skills
Algebra
Algebra
Algebre
Algebra
Subskilla
Lineer Equations
Polynomial Factorituation
Lineer Equations
Polynomial Factorbation
Fractione
Linwar Equations
Polynomial Foctorization
Frections
10
10
10
10
15
10
30
O (-15)
(+10)
(+10)
(+15)
(+20)
Figure 3: Example skill tree updates over time for MATH task's "Algebra" skill in the SKILL-TREE environment. Starting from empty single node, the data generation policy (Sec. 2.2.1) iteratively chooses actions between explore (grow skill tree) and exploit (rebalance skill tree). Then the skill organization module (Sec. 2.1.3) accordingly adds/removes subskills and re-allocates the training data for each subskill.
2.1.3 SKILL ORGANIZATION
To solve complex problems by adaptively growing the set of skills the student can perform, is natural to organize skills into
kind of hierarchy. In the SKILL-TRREE environment, the skill organization module takes as inputs set of skills, and outputs a forest of "skill-trees", an organized hierarchical structure that encodes skills and stores their metadata (e.g., how much data is allocated to each skill). This is the state St in the SKILL-TREE environment. Fig. 3 shows an example skill tree.
Baseline implementation. In the 3KILL-TREF environment, the skill forest captures the student's proficiency at increasing levels of granularity, with the root of each tree corresponding to high- level skill domain and the children corresponding to subskills. Each tree in the forest contains key information about subskills, including the amount of training data allocated to each subskill (i.e., the data allocation) and the student's 's performance on the training split for each subskill. Note that the specific implementation of the skill forest is left to the user; DATAENVGYn provides default version of the skill forest, but other implementations can plugged in.
2.2
DATA GENERATION AGENT MODULES
2.2.
DATA GENERATION POLICY
The data generation policy takes as input the student performance state 9; (list of per-example errors for OPEN-ENDED environment, skill-specific eπrоrs and the skill list for SKILL-LIST environment, and skill-specific errors and the skill tree for SKILL-TRRER environment), and outputs as an action the data generation plan (the inputs to data generation engine): pt x (Prisz) (.). In the OPEN-ENDER and SKILL-LIST environments, the data generation plans are lists of specifications for training data, one for each training datum to be produced. The training datum is rendered formatted into the appropriate format for instruction finetuning by the data generation engine. In the SKILL-TREE environment, shape the action space and provide two discrete actions: explore and exploit; note that further actions can easily be added. Explose ore actions grow the skill tree by adding subskills. Explos ceploit actions change the allocation of data for existing subskills.
Baseline implementation. We drive the policies for the OPEN-ENDER and SKILL-LIST environ- ments with an I.LM by giving the verbalized the state the I.IM and prompting it to produce the corresponding actions. For the SKILL-TREE environment, we implement a policy that grows the skill tree fixed size and while maintaining uniform data distribution by sequencing explore and exploit actions. Details can be found in Appendix C.6.
2.2.2 DATA GENERATION ENGINE
The data generation engine's 's role to generate training examples based on the data generation plan from the policy: de Eng (p. The training examples will be used to teach the student. Because each environment affords the agent with a different action space, the data generation engines corresponding them also differ. Specifically, for the OPEN-ENDED and SKILL-LIST environments, the data generation engine receives actions in the form of datapoints to generate (since the policy's 's action space is unstructured) and formats the appropriate examples (e.g. for GQA, it generates images using a T2I model). The OPEN-ENDED and SKILL-LIST generators have access to the task and a list of examples render into training data. For the SKI.II-TRFE environment, where the action space is

---

Published as a conference paper at ICLR 2025
fexplore, exploit), the data generation engine must first interpret these actions. Each action triggers a modification to the skill tree. An explore action invokes a subskill discovery pipeline to grow the skill tree by adding subskills. When emitting exploit action, the agent has to specify how to change the data allocation, or budget, for each subskill; executing the action means adjusting the budget stored in the skill tree accordingly. Finally, the data generation engine consumes the skill tree and generates the planned amount of data for each subskill.
Baseline implementation. For all tasks (mathematics, VQA, and programming), we generate training data using an LLM (GPT-4o). For mathematics problems, we generate problems using an LLM, where each problem consists of a question, a step-by-step solution, and final answer. For VQA tasks, we first use an LLM to generate image descriptions aligned with the task/Skill/subskill given as input. We then employ a text-to-image model to convert these descriptions into images. Then, the LLM is instructed to generate specified number of unique questions for the given task/skill/subskill. For programming. we generate data in two stages. We generate a problem and starter code given subskill information and detailed instructions about the expected format, and then solve it with an independent LLM call. We provide details on the generators for each environment in Appendix C.3 and show generated training examples for each task in Appendix D.
3 EXPERIMENTS
We experiment with DATAENVGYM environments in four domains: visual question answering. mathematics, programming and tool-use. For visual question answering, we use GQA (Hudson & Manning, 2019) and NaturalBench (Li et al., 2024); for mathematics, we use MATH (Hendrycks et al., 2021); for programming, we use LiveCodeBench (Jain et al., 2024); for tool-use, we use MnMs (Ma et al., 2024). For most experiments (reported in Sec. 3.1), we start from instruction-tuned models rather than base models because we believe it is a more realistic and challenging setting since starting from instruction-tume
models is standard for applications, and these models have undergone post-training on large amounts of task-specific data. For GQA and NaturalBench, we use PaliGemma- 3b-pt-224 (Beyer et al., 2024) as our student model and we use GPT-4o OpenAl (2024) the teacher agent policy, augmented with SDXL Turbo (Sauer et al., 2023) for T21 generation. For MATH, we use ( use Llama-3-88 3-nss-----B-Isstruett Instruct (Llama Team, 2024) as a student. For all domains, we generate data with GPT-4o. Note that the student models we use are typically already proficient the target task and thus are difficult to improve. For each domain, we choose the student model to satisfy the following criteria: 1) the student should be strong enough to perform the given task (e.g.. LiveCodeBench too challenging for a Gemma2-2B student). 2) The student should not have been heavily post-trained s.t. further improvements are unlikely (e.g., Llama3-8B has been extensively trained for math and further improvements with additional training unlikely, regardless of the data generation agent). Details can be found in Appendix C.7 (validation and test splits) and Appendix C,3 (data generation). Experiments on tool-use (MnMs) and additional multimodal experiments (NaturalBench) are detailed in Appendix C.
(Gemma Team, 2024) as a student. For LiveCodeBench and MnMs, we
We train all models for fixed number of steps and terminate episodes after fixed number of iterations, and use validation accuracy to select the training dataset iteration corresponding to the highest student accuracy. For each environment and domain, we report two values: first, we report the increase in student performance achieved by the baseline implementations of the teacher policy described in Sec. 2.2.1: this policy takes in the state st, i.e., it is given by x (pp.)s:). We refer this setting as With State. Second, we report the increase in student performance achieved by the same policy without conditioning on the state information, i.e. sampling an action from x (prl-)) without conditioning on st. For the OPEN-ENDED environment policy, we replace the list of student errors with random train samples. For the SKILL-LIST policy, we do the same. For the SKILL-TREE policy, we take explore and exploit actions with equal probability. We refer to this setting as No State.
3.1 PRIMARY RESULTS: VQA, MATHEMATICS, PROGRAMMING
Tab. 2 presents results on example instantiations of environments within DATAENVGYM. Here, we compare students before and after a multi-step trajectory of training across environments, with different data generation policies. For each setting, we report the relative gain or loss (in blue) compared to student model before training in DATAENVGYM. Note also that the models used here are already instruction-tumi tuned on large datasets (including tpsk-sp datasets), making obtaining further improvements particularly challenging.
7

---

Published as a conference paper at ICLR 2025
Table 2: Agents in DATAENVGYM'S environments are able to improve students across tasks and teaching environments. "Note that No State is the same for these environments only differ in their state representation, their No State policies (which do not condition on the state) are identical.
and SKILL-LIST because
GQA (PaliGemma 3B)
MATH (Gemma2 2B)
LiveCodeBench (Llama3 8B)
Setting/Env.
Avg. Improvement
Before teaching
44.18
15.78
16.50
OPEN-ENTE ENDED environment
+No State" =(p(-))
43.48 (-0.70%)
19.78 (+4.00%)
16.50 (-0.00%)
(+1.10%)
+With State (P)((s))
47.90 (+3.72%)
23.44 (47.66%%
18.91 (+2.41%)
(+4.60%)
SKILL-LIST environment
+No State" =(p(l-)
43.48 (-0.70%)
19.78 (+4.00%)
16.50 (-0.00%)
(+1.10%)
+With State (p)())))
48. 18 (+4.00%)
19.48 (+3.70%)
18.25 (+1.75%)
(+3.15%)
SKILL-TREE environment
+No State (ppl-)
49.53 (+5.35%)
17.15 (+1.37%)
16.50 (-0.00%)
(+2.24%)
+With State π7 (Ppe)a) (p)
49.76 (+5.58%)
18.90 (+3.12%)
17.75 (+ (.25%
(+3.32%)
Accuracy Difft (Miter -eforel
xn. Anelysis & Manip,
Exn, Anelysis Manip.
0.075
0.000
Logs and Exps.
grob. & Comb.
Froh. & Comb.
Logs and Exps. Cale. & Optimg
0.025
Cale. Optim Seqs.
Geries & Seqs.
0.000
g710.
Trio.
-0.050
2.8
3.2
3.4
3.6
3.8
42
0.002
0.004
5.006
0. 0.008
0.010
(a) Average Question Level
(b) Skill Rarity (1/Freq.)
Figure 4: Per-skill accuracy improvements of Gemma-2B trained on MATH in the SKILL-TREE environment, as a function of (a) question difficulty and (b) skill rarity (inverse of frequency) in the training data. In both cases, the biggest performance increases occur in the middle range.
LLM policies can make use of state information to provide better training data for the student. Students trained in the "No State" setting generally perform worse than those trained in the "With State" setting. This is true across environments, with the largest difference (3.5%) for the OPEN- ENDED environment and the smallest difference (1.08%) for the SKIIII-TREE. environment. On LiveCodeBench, policies without state information are not able to improve the student at all, whereas on MATH, a policy without state information still able to improve a student in all environments. The support provided to the teacher by the SKILL-TREE environment is particularly robust for GQA, where a policy without state information reaches almost identical performance to a policy with state information. However, absent SREE's structured actions, removing state information actually hurts performance on GQA, with slight drops from the baseline for the "No State" setting on OPEN-ENDED and -KILL-LIST environments. For both these environments, "With State" improves the student model. Taken together, these results highlight the importance of the state information.
Teaching is easier in some environments than others. With a fixed student and task (i.e. looking at "With State" entry across the columns of Tab. 2), teachers typically elicit the highest student performance in the unconstrained OPEN-ENDED environments, where they are not required teach specific set of skills. However, there may be domain specific effects here as the teachers in the SKILL-TREE environment perform the best on the multimodal GQA dataset (+5.58%), whereas this is reversed for MATH, where teachers in the OPEN-ENDED environment perform the best (+7.66%)
These difference may relate to the level of constraint imposed: in the ENEN-ENDED environment, the teacher can produce any data without any constraints, while in the skill-structured environments, we require the teacher to improve the student along specified skills. This may be more difficult task, as it may impose suboptimal constraints on the teacher, i.e., the teacher may have difficulty teaching the specified skills, whereas in the unconstrained OPEN-ENDED environment, the teacher may be implicitly able to identify and teach the skills it is best at tcaching. However, unconstrained teaching (which has no clear link between errors and generated data) may not always be useful in practice, e.g. when user wants to control the training and make targeted improvements certain skills.
8

---

Published as a conference paper at ICLR 2025
3.2 ANALYSIS: DIFFICULTY/RARITY, TRAINING DYNAMICS, SKILL DISCOVERY QUALITY
Iterative training dynamics. In Fig. 5, we plot the change in the student model's performance on the validation set through- out a full run in DATAEN- VGYM on each task and for each environment. Each ex- periment truncated once the performance saturates (consis- tently fails to increase for mul- tiple iterations). We use the "With State" baseline agents for each environment, and use the same models as in Tab. 2. Fig. 5 shows other words, the baseline
Talk: GQA
Tesk: LiveColeBechh
Talk MATH
0.5
1:0
0.9
0.18
0.35
0.16
0.1-
OPEN HK0TL 12ST
0:45-
0.12
0.40
0.10
Figure 5: Training dynamics across three tasks. Each line is split into solid segment terminating in vertical line (the maximum performance achieved by the student) and dashed line showing the effect of continued training beyond this point.
that the students generally improve across iterations. agents do uncover new datapoints that further improve the student at each iteration. Moreover, the behavior of agents trained in the three environments differs across tasks: on GQA, SKIFL-TRREF improves for more iterations than the other environments, while on MATH it reaches plateau and on LiveCodeBench it is truncated after two rounds of data generation.
Impact of skill discovery quality. In Tab. 3, we show the result of training students us- ing data generated by SKILL-LIST environ- ments with different skill discovery modules. For each domain, we determine a set of ora- cle skills. For GQA, the oracle skills are the suman-annotaal skills. Because MATH does not have cuman-annooatee skills, we approx- imate oracle skills by running the skill discov- ery module the test data, thereby creating skills from privileged information. In both settings the oracle skills allow the teacher produce better data and improve student per- formance. The increases from oracle skills are higher for GQA than MATH, possibly due to the fact that the MATH skills still rely on the same skill discovery module as the inferred skills. This promising signal, as it indicates that further performance improvements can be obtained by improving the skill discovery module. These results also highlight the utility of DATAEN in allowing us to swap in different components: by creating modular framework for developing data generation agents, we enable future work to test modules and components in the framework using student performance as a metric.
Table 3: DATAENVGYM allows us to test various implementations of environment components. Here, we compare oracle vs. inferred skills for GQA and MATH. Better skills result in better teaching and thus an improved student.
Skill Type
GQA
MATH (Gemma2 2B)
(PaliGemma 3B)
Before teaching
44.18
15.78
SKILL-LIST +Oracle Skills
53.02 (+8.84%)
19.52 (+3.74%)
+Inferred Skills
48.18 (+4.00%)
18.25 (+2.47%)
Skill learning across rarity and difficulty levels. Tab. 2 shows that skill-based learning in the SKILL-TREE environment can improve overall performance of student models. Two core questions are (1) how interpretable these skills are and (2) how learning correlates with features like question average difficulty or skill frequency. In Fig. 4, we plot the accuracy improvement of a Gemma-2B student model after training in DATAENYGYM'S SKILL-TREE environment for MATH; most skills improve, some more than others. In Fig. 4(a) we plot improvement across the average question difficulty (provided by MATH a scale of 1 to 5). Training in DATA ENYGYM boosts student performance the most in the middle difficulty region (around 3.5). On the edges, we see smaller boosts, with close to 0 difference for Calculus and Optimization (high difficulty) and even decreases for Trigonometry (low difficulty). In Fig. 4(b) we compare performance to skill rarity (inverse frequency) in the training data. Here also, the least and most represented skills benefit. less. Taken together, the results in Fig. 4 suggest that there is a sweet-spot of difficulty and frequency. At the low end this could be due to saturation: easy and frequent skills benefit less from training because the model already performs well on them or has saturated. At the other end, difficult skills or very rare skills may be underrepresented teacher's 's training data or be harder to generate questions for, making learning less effective. Alternatively, the questions generated may be too hard for the student. In the middle, the teacher generates helpful examples, allowing the student to learn. Similar theories have been put forth for human learning, e.g., Vygotsky (1962)'s Zone of Proximal Development,

---

Published as a conference paper at ICLR 2025
where learning is most effective given problems slightly harder than those students could solve alone, but not so difficult that they would have no hope of solving them.
4 RELATED WORK
Training Environment Generation. In agent learning frameworks, designing training environ- ments usually becomes bottleneck, as it requires sophisticated human efforts (Park al., 2024). Unsupervised environment design (UED) (Dennis et al., 2020; Jiang et al., 2021; Parker-Holder al., 2022) explores progressively increased environment difficulty based on agent scores in simple games, Liesen et al. (2024) introduce a meta-learning approach to create learning environments for continuous control. In vision-languaee navigation (VLN), past work (Li et al., 2022: Li & Bansal, 2024; Wang et al., 2023c) propose augmenting the visual diversity of training environments with image generation models. Generation has been applied to game environments as well: Cobbe et al. (2020) generate a diverse set of game environments for training RL agents and measuring their generalization. Zala et al. (2024) continuously adapt game environments for training RL agents, using an LLM to generate different environments that teach core skills to the agent based on feedback from the agents' intermediate progress, which improves final performance of agents as well as learning efficiency. Yang et al. (2024) generate 3D embodied environments from uscr-specified prompts, generating rooms in different styles. Sudhakaran et al. (2023) and Todd et al. (2023) use LLMs for open-ended generation of game levels for tile-based games. OMNI (Zhang et al., 2023) uses LLMs for task selection in open-ended environments, while OMNI-Epic (Faldor et al., 2024) augments OMNI with the ability to generate new tasks in simulated robotics settings. Generally, past environment generation work has focused on environments like games or graphical simulations with predefined actions and skills. Moreover, past environment generation work has focused on developing students rather than improving the data generation process itself. In contrast, our work focuses on data generation agents, creating a testbed for teachers and treating students as part of the environment. Furthermore, our work introduces environments for data generation with automatic skill discovery in a diverse set of open-ended and more realistic settings such as mathematics, visual question answering, and programming.
Learning from Generated Data. DATAENYGYM involves transferring task knowledge from a teacher agent to student model in an effective way, based on the student model's feedback. Past work on knowledge transfer from one model to another has been centered around knowledge distillation, where outputs from larger models are used to train smaller ones (Hinton et al., 2015; Buciluă et al., 2006; Chen et al., 2020); unlike the process in DATAENVGYM, this process is typically not adaptive, relying on fixed datasets of inputs that are processed by the larger teacher model and used to train the student. In the context of LLMs, symbolic distillation (West et al., 2022) has become increasingly common; here, text is generated from a large model and used train smaller one, e.g., in instruction tuning (Wang et al., 2023a) or distilling chain-of-thouge from large proprietary models into smaller models (Magister et al., 2023; Shridhar et al., 2023; Fu et al., 2023; Ho et al., 2023; Saha et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023; Chen et al., 2024). The kind of teaching that data generation agents are expected to perform in ENYAENss environments differs from the knowledge distillation setting in that the inputs to the model themselves are model-generated (as opposed to being sourced from an existing dataset). Moreover, the inputs are dynamically determined based on the student model's 's feedback and errors, whereas in knowledge distillation they are determined by a fixed dataset or generated all at once. Note that DATAENYGYM is compatible with different methods of training the student (i.e., symbolic distillation, logit- based losses, etc.), and these can be swapped into our modular environments.
reasoning (Wei et al., 2022)
CONCLUSION
We propose DATAENVGYM, a testbed of teacher environments for developing modular data genera- tion agents (i.c., teachers) and environments. In DATAENGGYM. a teacher agent generates training data for a student model and receives feedback on the student's 's performance from the environment. We provide three environment instantiations with different levels of structure. Across four diverse domains (visual question answering, mathematics, programming, and tool use), we demonstrate that the example teachers we introduce can improve students in all DATAENVGYM environments. We analyze DATAENVGYM, including its training dynamics, performance across difficulty levels, and the impact of skill discovery module quality. We hope our work will foster future progress on data generation agents, engines, and feedback mechanisms by providing a testbed for evaluating and improving them.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We experiment with improving skills of student models in visual question answering. mathematics, and programming. In our DATAENVGYM task, data generation agents are expected to create training data; we implement this process using LLMs and vision-language models, which can be biased and reflect common stereotypes as well other negative information that is present in their training data (Weidinger et al., 2021; Birhane al., 2024). These problems impact all work secking to generate data via models or train LLMs, and merit further exploration.
REPRODUCIBILITY STATEMENT
We will publicly release our code and leaderboard. For all experiments, we use publicly available datasets and student models. We provide the detailed hyperparameters and additional method details in Appendix B.
ACKNOWLEDGMENTS
This work was supported by DARPA ECOLE Program No. Institute DRL-2112633 DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031. ARO Award W91 INF21 10220, ONR Grant N00014-23-1-2356 Microsoft Accelerate Foundation Models Research (AFMR) grant program, and a Bloomberg Data Science PhD Fellowship. The views contained in this article are those of the authors and not of the funding agency.
NSF-AI Engage
REFERENCES
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, volume 29, 2016.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Fisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL htttpp://aaaae /ttp://aaivvvoio .rgg abs/ /407 07726.
Abeba Birhane, Sanghyun Han, Vishnu Boddeti, Sasha Luccioni, al. Into the laion's den: Inves- tigating hate in multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizz Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
URL hwtps:ttp///cc...t.. /dl. cm org/ doi / /1 0. 1115/11111001
1150464,
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https: //openreview..o. net forum? warpummmiiiill Hdmmmmdd--1115111111 Ym.
Justin Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. In Forty-firs International Conference Machine Learning, 2024.
Ting Chen, Simon Korbblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255 2020. URL https: / arxiv. www.vorgg/aaae abs /00 10029.
11

---

Published as a conference paper at ICLR 2025
Yew Ken Chia, Pengfei Hong, Lidong Bing. and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuna large language models. In Proceedings of the First edition of the Workshop the Scaling Behavior of Large Language Models (SCALE-LLA) LLM 2024), pp. 25-64,
2024.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048-2055 PMLR, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:1 33:13049-440 3049--336911,1 2020.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving, May 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 3029-3057 2023.
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.
Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin George Konidaris, Nicolas Le Roux, Mare- Alexandre Côté, and Xingdi Yuan. Language-guise guided skill learning with temporal variational inference. In Forty-first International Conference on Machine Learning. 2024.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10421-10430. PMLR, 23-29 Jul 2023. URL https: / /tpppeedoeegggss////ee fu23d. html.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209008858. 2022.
Tong Gao, Shivang Singh, and Raymond Mooney. Towards automated error analysis: Learning characterize errors. In The International FLAIRS Conference Proceedings, volume 35, 2022.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL
https://ae //avivv//rvvvva//aa///e org/ abss/////////oooooooooo///saaasr 2408 00118.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14962. 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmurks Truck (Round 2), 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /503 02.33333,,, 2015. URL https: / /tt.x/ivvvv//.////////.////aaavv www.iv..rrgaaaeeeeee www./aa.//////eo//////////eaaee/ htt///bs//555 /0003 02531.
12

---

Published as a conference paper at ICLR 2025
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14852-14882, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi:10. 1.w.0...1111111 org/2023 aacalclllln long. 830.
att55//1/2233..looong...... URL https:: : //acllccccccccc antho logy.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 2rXiv:2212.120113 2022.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
Minqi Jiang, Michael Dennis, Jack Parker-Holle Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 14:1884-1897, 2021.
Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision- language models on natural adversarial samples. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. Advances in Neural Information Processing Systems, 36, 2024.
Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for dision-and-languae naviga- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15407-15417, 2022.
Jarek Liesen, Chris Lu, Andrei Lupu, Jakob N. Foerster, Henning Sprekeler, and Robert T. Lange. Discovering minimal reinforcement learning environments, 2024. URL URLAttts://arxiv.oor abs/ abss/206.. 12589.
Llama Team. The llaaa 3 herd of models, 2024. URL https://arrvv..or//abbb/// 21783.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648. PMLR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110:1711.0510,, 2017.
Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m's: A benchmark to evaluate tool-use for multi-step multi-modal tasks. EECV 2024, 2024.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 1773-1781, Toronto, Canada, July 2023, Association for Computational Linguistics. 101.10001633///2222 acl-short. 1. ogy- orr//0233 acl short. 151
URL https::///.............t. //acl anntan anttoaaogy
13

---

Published as a conference paper at ICLR 2025
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045, 2023. URL https:
//arxiv. aaargg/bb/// 2311 11045.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint 02707.
arXiv:2306.02707, 2023. URL https: pepttts://axiiie /tt///iiiiiv..//vi http:/pps//araair.rrrra/a///////
OpenAL. Hello gpt-4o, 2024. URL https:
Younghyo Park, Gabriel B. Margolis, and Pulkit Agrawal. Automatic Environment Shaping is the Next Frontier in RI., July 2024.
Jack Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pp. 17473-17498, PMLR, 2022.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03222 2022.
Seungeun Rho, Laura Smith, Tianyu Li, Sergey Levine, Xue Bin Peng, and Sehoon Ha. Language guided skill discovery. arXiv preprint arXiv:2406.06611 2024.
Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach weaker agents? teacher explanations improve students via personalization. In NeurlPS, 2023. URL https:
//
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023. URL hwtps:tt/////. /wwxxviivvvvv.////////a////aaav/ org/ h/s///b//20 1112.11.100000011
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1713-1726, 2022.
Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzich Fadace, and Sara Hooker. Llm see, Ilm do: Guiding data generation to target non-differentiable objectives. arXiv preprint arXiv:2407.0144 20224007.1490 2024.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: 2023,
pp. 7059-7073, 2023. URL https: / arxiv. org/ abs/ 00193.
Shyam Sudhakaran, Miguel González Duque Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 /2tpeer//2/////// 11888144588888 htw.:aleeeee-btttaaat--oCteeeeee wonfoooooeeoooe html.
16. 2023, 2023. URL http: : ptwppspppppppas//ppppeeeeaeaeaap nips 2c wwtpappp_aiiiiieeeeeeeeeeeeeeepe a
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An mnstruction-folloiinn llama model, 2023.
Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Phil Lopes, Filipe Luz, Antonios Liapis, and Henrik Engström (eds.), Proceedings of the 18th International Conference the Foundations of Digital Games, FDG 2023, Lisbon, Portugal, April 12-14, 2023, pp. 70:1-70:8, ACM, 2023. 358722
URL https: /tpp///....///// orr.oorg/1100
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXh::2307.09288. 2023.
Lev Vygotsky. Thought and Language. MIT Press, Cambridge, MA, 1962.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 13484-13500 13508, Toronto, Canada, July 2023a. Association for Computational Lin- 2ergg/22.332
guistics. doi: 10. 1885888112223332222a2a 754. URL httpss:///ccaaaeeeeeeetee acl-long. 754.
Zige Wang, Wanjun Zhong, Yufei Wang. Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang. and Qun Liu. Data management for training large language models: A survey, 2023b. URL https: //api.pppppp
wes.entccccclllaa ar. wwr./org//ooo www./orrgoooaeeeeooe Cor/orrrussTS TD: 266600669,
Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in usion-and-languuae navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12009-12020, 2023c.
C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of thouhhtt prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 012248242448311 2022. URL https: http://arxxivooo//e// abss/201..0 11903.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et Ethical and social risks of harm from language models. arXiv preprint arwiv:2112.044::0 2021. URL https: www.a.iivvvv......a.aa.a org. /bbs/ 2112. 04359.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (, Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 4602-4625, Seattle, United States, July 2022. Association for Computational Linguistics. 1oi: m41n.341. URL https: aclanthology...... org/ 2022 naacl 341.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguisties. URL https:://// www. ac web. org/ anthology 2020.000222222222 emnlp-demos demos. 6.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Errudite: Scalable, repro- ducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 747-763, 2019.
Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Ab- hinav Shrivastava, Furong Huang, Jordan L. Boyd-Graber, Tianyi Zhou, and Dinesh Manocha. Autohallusion: Automatic generation of hallucination benchmarks for vision-language models. ArXiv, abs/2406. 10900. 2024. URL https:/ /ppi. semant i iescholar. Drg/CorpusID: 270560551.
Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16227-16237, 2024.
15

---

Published as a conference paper at ICLR 2025
Abhay Zala, Jaemin Cho, Han Lin, Jachong Yoon, and Mohit Bansal. Envgen: Generating and adapting environments via lims for training embodied agents. In COLM, 2024.
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711 2023.
Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma,
Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. arXiv preprint arXiv:2406. 11775, 2024.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL hwwp:ttpp//. /tte:://arxive org/ abs 2tts/2400 2403.. 13372.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. Advances Neural Information Processing Systems, 36, 2024.
APPENDIX
In the appendix, we present additional related work (Appendix A), additional method details (Ap- pendix B), results on additional benchmarks (Appendix C) qualitative examples (Appendix D), experiments showing accuracy generated data (Appendix E) and ablations the relative impor- tance data vs training time (Appendix F).
A ADDITIONAL RELATED WORK
Skill Discovery. A line of work in reinforcement learning has studied unsupervised skill discovery, where agents learn diverse emergent behaviors without explicit rewards. The majority of this work helps agents discover new skills by maximizing the diversity of agent trajectories, such exploration- encouraging rewards (Gregor et al., 2016; Bellemare et al., 2016) and adding randomness during action sampling (Watkins, 1989; Burda al., 2019). However, such methods require long exploration steps, which is expensive if the cost for agent action is not negligible. Recent work has also proposed using the knowledge contained in pretrained language models to help skill discovery (Sharma et al., 2022; Rho et al., 2024; Fu et al., 2024) and to sample new skills (Didolkar et al., 2024). Developments and improvements to skill discovery are complementary to DATAENYGYM, where skill discovery is used in the SKILL-LIST and SKILL-TREE environments in Sec. 2.1.2) to dynamically extract human-interpretabble skills from data and to create student feedback. These skills help identify model weaknesses and condition the data generation process, and we find that improving skill discovery can improve student model performance (cf. Tab. 3), pointing to directions for future work. Moreover, by using skill discovery in its environments, DATAENVGYN agents, but also provides a concrete extrinsic evaluation for skill discovery, allowing competing skill discovery methods be evaluated and compared on the basis of how well they improve downstream agent performance.
only helps develop and test interpretable
Model Weakness Discovery. Testing trained machine learning model in different scenarios is crucial in mitigating unexpected malfunctions, so that developers can actively prevent such behaviors and finetune models on weak skills. Traditionally, model weaknesses have been identified by hiring human annotators and asking them to create adversarial inputs in different scenarios and check when model outputs are incorrect or undesired (Ganguli al., 2022). Recent work explores automatically finding model weaknesses by creating test cases in different scenarios, with either with pipelines consisting of traditional ML models (Gao et al., 2022; Wu et al., 2019) or LLMs (Perez et al., 2022; Zhang et al., 2024; Wu et al., 2024). These methods tend to focus on adversaria scenarios such as jailbreaking or redteaming, where a model is made more robust against an increasingly difficult adversary. Our framing is orthogonal: rather than focusing on defending against adversaries, in DATAENVGYM, students and teachers (data generation agents) are cooperative, working together to improve student performance.
16

---

Published as a conference paper at ICLR 2025
GgA Nalbibie
1Q4 Task BucaKes
Dudaadee snewn tn exch 900 Instercc
Crospkaed Yark Instaces
Teacher Apera
mage
Image Question
Studlent
Arssiee
Instruction 21 Prompt Resporeee
Do Stador
Studerk
Instruction Image Response
121 Hendering
Stedeat
Trakingg Iata.
Figure 6: A complete data generation loop for the multimodal domain in the OPEN-ENDED environ- ment. Note that the teacher agent is modular. For domains such as mathematics or code generation, data generation engine may not be required.
B ADDITIONAL METHOD DETAILS
In Fig. 6, we provide a more detailed walkthrough of the core environment loop. The example in Fig. depicts one experience in the OPEN-ENDES environment for the multimodal domain using the GQA dataset the target dataset. The loop is highly modular, and can be changed to target different data distribution by changing the dataset used for evaluation. The data generation policy and data generation engine can likewise be modified easily.
C ADDITIONAL MULTIMODAL AND TOOL-USE BENCHMARKS
In this section, show results additional challenging multimodal and tool-use tasks: Natural- Bench (Li et al., 2024) and Table and MnMs (Ma et al., 2024) in Table 5. MnMs and NaturalBench include fine-grained metrics other than accuracy, which enables generated by different methods affects student model. On both MnMs and NaturalBench, condition- ing on the task and errors of the student allows the teacher generate more effective training data (compare "With State" against "No State" in Table 4 and Table 5).
detailed view of how data
On NaturalBench, the training data generated when conditioning on the student's state more effective at improving the student on all metrics. The "Group Score" probes the consistency of model across four visual question answer pairs that test the same concept. The "Group Score" increase from generated data without conditioning on the state is zero, while conditioning on the state results in an increase the group score.
MnMs is tool-use benchmark that requires modified skill discovery mechanism, the standard pipeline applied to MnMs produces skills that are not tightly coupled to the underlying tools required. For MnMs, we consider each combination of tools "skill". This results in very large number of "skills" for tool-use. To generate data on a fixed budget, we must sample subset of skills (tool combinations) for which generate data. We experiment with several sampling methods, and find that the most effective sampling method to randomly sample a subset of skills for which the student does not have perfect accuracy.
17

---

Published as a conference paper at ICLR 2025
Table Performance metrics for NaturalBench (Li et al., 2024) before and after teaching using the
OPEN-ENDED and SKILI--LIST environments.
Question Score
Image Score
Binary Score
Group Score
Avg. Improvement
Before teaching
3.92
3.92
48.53
0.00
OPEN-ENDED environment
+With State r(pe(s))
26.00 (+22.08%)
29.00 (+25.08%)
61.00 (+12.47%)
6.00 (46.00%)
(+16.41%)
+No State x(pxl-)
11.00 (+7.08%)
9.00 (+5.08%)
51.50 (42.97%)
0.00 (+0.008)
(+3.78%)
SKIIL-LISTT
+With State s(p((s))
25.00 (821
26.00 (+22.08%)
59.50 (+10.77%)
2.00 (+2.00%)
(+14.03%)
4No State r(p.))
11.00 (47.08%)
9.00 (+5.08%)
51.50 (42.97%)
0.00 (+0.00%)
(43.78%)
Table 5: Performance metrics for MnMs (Ma et al., 2024) before and after teaching using the and SKILL-LIST environments.
Plan Accuracy
Tool Precision
Tool Recall
Tool
Avg. Gain
Before teaching
37.30
73.74
68.18
65.41
OPEN-ENDER environment
With State =(p]ls))
72.00 (434.70%)
87.32 (+13,98%)
82.62 (+14.44%)
82,25 (+16.84))
(+ (+19,99%)
+No State =(pe-)
62.13 (+24.83%)
87.94 (+ (+14.60%)
87.09 (+12,915) ()
80.51 (+ (+15.10%)
(+16.86%)
SKILL-LIST enviroument
+With State (
75.96 (+38. (+38.66%)
86.11 (+12.77%)
84.35 (+16.17%)
84.36 (+ (+18.95%)
(+21.64%)
+No State (Pp) )
62.13 (424.83%)
87.94 (+14.60%)
87.09 (+12.91%)
80,51 (+15.10%)
(+16.86%)
We refer the reader to the GitHub repository for complete details of prompts and hyperparameters.
C.1 TRAINING DETAILS
We use supervised finetuning for training using the Transformers (Wolf et al., 2020) library. We present data in an instruction-finen fintioiinnn format of Alpaca (Taori et al., 2023) with the standard language modeling loss. For evaluation, we use the standard training splits from the datasets we test on. More details the training process, including hyperparameters such as learning rates and optimizers, provided below.
C.1.1 GQA
We use the Transformers Wolf et al. (2020) library for training. We train PaliGemma-3b-pt-22 (Beyer et al., 2024) for 10 epochs using the Adam W (Loshchilov & Hutter, 2017) optimizer with warmup steps and a learning rate of 10S 5, weight decay of 10 6 using the BF16 datatype and batch size of 16. We apply LoRA (Hu et al., 2022) with a rank of 16 and an alpha of 32, no bias, and dropout of 0.05. We apply LoRA all linear layers.
C.1.2 LIVECODEBENCH AND MATH
We use Transformers (Wolf et al., 2020) and Llama-Factory (Zheng et al., 2024) libraries for training. We format all data in the Alpaca format (Taori et al., 2023) instruction-respons pairs. We use the Adam optimizer with batch size of 16 and cosine learning rate scheduler with a warmup ratio of 0.1 and train for epochs in the FP16 datatype. We apply LoRA to all linear layers with rank of 16 and an alpha of 32, no bias, and a dropout 0.05, We truncate all training examples to a maximum length of 1024 tokens.
C.2 LLM DETAILS
LLM configuration. We use 400p--4--202408-000 (OpenAI, 2024) for the following mod- ules: skill discovery (Sec. 2.1.2; in SKILL-LIST env), data generation policy (Sec. 2.2.1 in SKILL- LIST env), data generation engine (Sec. 2.2.2; in OPEN-ENDED, SKILL-LIST, and SKILL-TREE envs). We use temperature 0 and top-p of 1.0, which are default API settings. We use the Instructor library' to produce structured output from LLM calls and automatically parse LLM calls into Python objects.
httpp:: :/pp::///iia www..ccom//o/. jxn1 / rutttr
18

---

Published as a conference paper at ICLR 2025
Table 6: Token expenditure and GPU time per iteration for each environment and domain. domain.OPEN: ENDED environments are typically the cheapest. The most expensive domain is LiveCodeBench. although experiments in the OPEN-ENDEL for code cost less than $3 per run. Training time for most environments, even with single A6000 GPU, is typically less than 30 minutes.
Domain
Enviromment
Num Tokens
Cost (GPT-4o-mini)
Cost S (GPT-40)
GPU Minutes Iteration
Math
Open-Eine Ended
173234
0.10
1.73
24
Math
Skill-List
318528
0.19
3.19
24
Math
Skill-Tree
355033
0.21
3,55
16
Coding
Open-Ended
279304
0.17
2.79
16
Coding
Skill-List
497787
0.30
4.98
16
Coding
Skill-Tree
967610
0.58
9.68
16
Multimodal
Open- Ended
25073
0.02
0.25
37
Multimodal
Skill-List
82419
0.05
0.82
134
Multimodal
Skill-Tree
33991
0.02
0.34
78
Prompt Templates. We provide the LLM prompt templates for skill discovery (Fig. 13), data generation for GQA (Fig. 14) MATH (Fig. 15) LiveCodeBench (Fig. 16,), and data generation policy for OPEN-ENDED (Fig. 17) and SKILL-LIST (Fig. 18) environments.
C.3 DATA GENERATION DETAILS
For all tasks, we use validation accuracy to identify when to terminate an episode. An episode is terminated when when fixed number of iterations is reached, and the best student is selected from all students trained by the policy using validation accuracy.
We use stabil Ctabilittye litttiii/// //ss//lltturbbr with 4 inference steps to generate images at a resolution of 1024 1024. In the SKILL-LIST environment, budget after iterations, producing 7.5k data points. In the SKILL- TREEE environment, the baseline policy episode produced the top performing student after 20 steps 3k datapoints. In the OPEN- ENDED environment, the baseline policy episode produced the top performing student after 5 steps and 3k data points.
GQA.
baseline policy exhausted its data
MATH. For math, follow the termination criteria as for GQA. In the SKI.II-LIST environment, the baseline policy produces roughly 2. 1k datapoints and the its best student after 4 iterations. In the SKILL-TRES environment, the baseline policy produces 2.8k datapoints over 20 steps produce the best performing student. In the OPEN-ENDEE environment, the baseline policy requires 10 iterations and 752 datapoints to produce its best student.
LiveCodeBench. For LiveCodeBench, first generate problems, and then ask the LLM solve the problem. Termination criteria are the same as the other settings. In the OPEN-ENIE the baseline policy produces 1.6k datapoints and produces the best student after iterations. In the SKILL-LIST environment, the baseline policy produces 675 datapoints and produces the best student after 11 iterations. the SKILL-TRE environment, the baseline policy produces 3138 datapoints and produces the best student after 10 iterations.
environment,
C.4 RESOURCE COSTS
In Tab. 6, we tabulate the token costs and GPU time required for each environment and domain. The "Coding" domain and "Multimodal" domains are most expensive with respect to token expenditure and GPU time, respectively. Moreover, we show that we can reduce the overall cost by using different teacher model. Specifically, we conduct experiments using GPT-4o-mini (which is less expensive than GPT-4o) as the LLM powering the data generation agent, which we tabulate Tab. Here, continue expect performance improvements to remain positive but decrease as we reduce the size of the teacher model.
performance improvements
with the reduced model. In general,
19

---

Published as a conference paper at ICLR 2025
Table 7: Here, we replace GPT-4o inside the teacher agent with GPT-4o-mini to show that it is possible to use a less powerful but inexpensive teacher to conduct experiments.
GQA
MATH (Gemma2 2B)
LiveCodeBench (Ilama3 8B)
Setting/Env.
Avg. Improvement
(PaliGemma 3B)
Before teuching
44.18
15.78
16.50
OPEN-ENDED environment
+GPT-44-mini
45.51 (+1.33%)
15.78 (+0.00%)
17.50 (+1.00%)
(+0.77%)
+GPT-40
47.90 (+3.72%)
23.44 (+7.66%)
18.91 (+2.41%)
(+4.60%)
SKILL-LIST environment
+GPT-4o-mini
46.74 (+2.56%)
16.92 (+1.14%)
17.25 (+1 (+1.25%)
(+1.75%)
+GPT-40
48.18 (+4.00%)
19.48 (+3.70%)
18.25 (+1.75%)
(+3.15%)
SKILL-TREE environment +GPT-4oomini
48.83 (+4.65%)
16.06 (+0.28%)
17. 00 (+0.50%)
(+1.81%)
+GPT-40
49.76 (+5.58%)
18.90 (+3.12%)
17.75 (+1.25%)
(+3.32%)
C.5 SKILL FOREST DETAILS
The skill forest (used in SKILL- TRRE environment) is hierarchical structure representing skills and subskills across various domains. It models the student model's skill proficiency with increasing detail. Each tree in the forest corresponds to a high-level skill domain and contains the following key pieces of information for each subskill:
1. Data Allocation: The amount of training data allocated to each subskill.
2. Performance Training Data: The student model's performance on the training data for each subskill.
Skill Name: The name of the skill.
4. Subskills: A list of subskills for the skill, which starts out empty.
The forest evolves through two mechanisms.
1. Growing: Adding new subskills to the tree.
2. Rebalancing: Adjusting data allocation for existing subskills based on performance.
These actions conceptually correspond to exploring and exploiting the skill tree.
This structure allows to represent fine- mne-grained skills, allocate resources for data generation, track the student model's progress, and prioritize skills the data production process.
C.6 SKILL-TREE POLICY
We develop a policy as the baseline "With State" policy shown in Tab. 2. that aims to grow and balance skill tree. It operates in two phases: Growth Phase: The policy alternates between exploration and exploitation actions until predetermined maximum number of subskills reached. During exploration, new subskills are added to the tree. During exploitation, the policy resets data allocations to zero, preparing for the next exploration. Filling Phase: Once the maximum number of subskills is pure exploitation strategy. It caleulates and executes actions that
reached, the policy switches incrementally allocate data to each subskill, aiming to reach specified maximum amount of data per subskill. The policy respects maximum allocation limit per action and continues until all subskills have reached their data capacity.
C.7 VALIDATION AND TEST SPLITS
For GQA, we create validation and test splits by doing a balanced stratified sampling of the validation and testdev sets repeatedly. Specifically, sample 5 questions from each of the 100 question types in GQA, following Gupta & Kembhavi (2023). For MATH, we create a validation set by doing balanced stratified sampling of the test set across all levels and topies in MATH, selecting 50 from
20

---

Published as a conference paper at ICLR 2025
Skill Animal Identification
Skill Object Attributes
Subskill Identifying Animal Characteristics
Subskill Recognizing Animal Diet Patterns
Subskill Understanding Shape Transformstions
Subskill Identifying Color Texture
Q: ls the goat's beard long? No
Q: Do you see animals that eat both plans and moat? A:
Q: Doss this shape have more than sidee? A: Yes
Q: the material of the coat textured?
the gost's horn curved?
Q: there an animal that only leaves?
Q: the shape symmetrical? yes
Q: the door painted bright color? A:
Figure 7: A partial view of skill forest for GQA. Depicted are 2 out of 13 discovered skills. For each skill in the skill tree, we show subskills and 2 examples of generated data for that subskill. Note that all images are generated.
Skill Ma Moripuatiion
Skill Combinanaris
Subeil Ineese Funcsion
Sabakil Asymototic Schavior
Subsiil Permuiations
Subekill Stochasis Processcc
Tolnd the meaas a mnamaa
Svip ai
na dinaae couar ven pecple
Peotoem Eqpers Poisoor presee ploae1
Farts hambbaaee DCCAT"
sescerate mas sureaar bel aolt
anderrreeee (1)
Wes w andi Pavalaceia Li's 20
tea budt R
Figure 8: A partial view of skill forest for MATH. Depicted are out of 12 discovered skills. For each skill in the skill tree, we show 2 subskills and example of generated data for that subskill.
each group. We use the official test set for MATH. LiveCodeBench, we create a validation set by choosing all problems that are in the 2nd release but not in the 1st release our validation set, and use the entire 1st release as our test set. This results in a relatively small validation set of only 100 problems.
D QUALITATIVE EXAMPLES
Generated skill frees and examples. In Fig. 7, Fig. 8, and Fig. we show qualitative examples of skills and subskills discovered for GQA, MATH, and LiveCodeBench, respectively. For each subskill,
21

---

Published as a conference paper at ICLR 2025
SE Procerry
SII StIaA
Sdink
Denine
Totrk Soludd
The
sarple
Lood
Figure 9: A partial view of skill forest for LiveCodeBench. Depicted are 2 out of 15 discovered skills. For each skill in the skill tree, we show 2 subskills and examples of generated data for that subskill.
LiveCadeBench
GQA
MATH
Army Monipuaaiinn and Optimiatiion
Animal Hsed Fumitaro Recopriton Comparison
Algetrn Erparion Solking
Dynumi Pagronmning and Combé
Funaton Maripulation
rig Mariquiation
Natching
Geometry Spotial Trgonomcay
Game Stutugy and Theory
Sootal el Redatoratipa and Onject Posiiion
Graph and Traversal Alporihims
Miterial derdifcation and Comparson
Frobub By and Compinatario
Mocho natioal and Statisical Computation
Envionment Classiliaaion
Polynomia Analysis
Houritt Approaches
Recogmiion Detection
Frequency and Ceeummeea Analysta
Weether Condition itecngraton
Number Theery and Meculer Anthmerie
the Openations and Logie
Object Atributea
Manto and Vettr Calouaatoon
Giet and Mastic Prncessing
Contersoe image dinde-dreddng
Compae wee Namboen
Ladoogropic String Drder
Spectic Oijped Decognation
Loganthmme and Expenente
Vall
Becoggation
tommy
Probociletcc and imsoccho Sohetone
Figure 10: Skill lists for each domain. These were determined by the skill discovery module and used in the SKILL-TREE and SKILL-LIST environments.
22

---

Published as a conference paper at ICLR 2025
Skill Material Identification & Comparison
Skill
Environment Classification
Question
Question
What the material of the chair?
Does the forest look brown and tall?
Before Training (Wrong) PaliGemma- 3B: Wooden
Before Training (Wrong)
PaliGemma Yes
After Training (Correct) PaliGemma- 3B: Plastic
After Training (Correct) PaliGemma 38: No
Figure 11: Qualitative examples of how training generated data changes the response of PaliGemma-33 student.
also show example datapoints generated by the teacher model. Note that these datapoints generated in their entirety, including the images shown in Fig. 7.
Model predictions before and after training. In Fig. we show qualitative examples of how training on generated data changes the response of PaliGemma-3B student. The example on the left falls under the "Material Identification and Comparison" skill that was identified during skill discovery. Training on generated data leads to the model correctly identifying the material as "plastic". This may be a result of debiasing in terms of the possible materials for chairs in the synthetic data. On the right is another tricky example, where the initial answer could be result of the model confusing the foreground (a giraffe) - for which "brown and tall" true with the background. After training on synthetic data, this mistake is corrected.
E ACCURACY ON GENERATED DATA
We analyze the performance of the student training data generated by the teacher successive iterations. We plot this data in Fig. 12. Concretely, for the Math and Multimodal environments, plot the accuracy of student iteration on data generated in iteration is generated but it has not been seen by the student yet. The student improves on data generated by the teacher through successive iterations as seen in Fig. 12. This shows that the data produced by the teacher is consistent over iterations and the student is gradually improving at the target data distribution. However, the data distribution produced by the teacher complex enough that the improvement is neither monotonic or accomplished in single iteration.
1; thus, the data
F ABLATION: DATA VS TRAINING
To substantiate that claim student performance is increased due to added data points rather than insufficient training, we take subset of the data and increase the number epochs such that the student receives fraction of the added data, but equivalent number of epochs as training on the full data. For example, if a student normally trained for 10 epochs with 1000 generated training datapoints, take the data from the first data generation iteration (which might, for example, contain 200 training datapoints) and train an alternative student for 1000 x 10 50 epochs to isolate the effect of the generated training data the added training epochs. We show the results in Tab. 8. In each case,
200
23

---

Published as a conference paper at ICLR 2025
Acturacy on Generated Training Data
Asturacy 150
Figure 12: Over successive iterations, the student becomes more proficient at data generated by the teacher. Here, we plot the accuracy of student from iteration on data generated by the teacher in iteration n + 1, which has not been seen by the student.
Mutimodal (GQA)
MATH
Coding (1 (TiveCodeRennc)
Data
Epochs
Accuracy
Data
Epochs
Accuracy
Data
Epochs
Accuracy
Before Teaching Less Data Longer Training
44.18
15.78
16.50
20%
42.79
10%
13.98
20%
15.00
More Data / Standard Training
100%
3
47.90
100%
3
23.44
100%
3
18.91
Table 8: Training with less data but for more epochs produces significantly smaller improvements than training with more data for fewer epochs, showing that data is responsible for increased performance rather than more training.
training with less data but for more epochs produces substantially smaller improvements than training with more data for fewer epochs, showing that data is responsible for increased performance rather than more training. In fact, extending training without additional data typically hurts performance fresh data is essential. This highlights the importance of studying data generation as we do, data generation is one of the few ways of obtaining fresh data.
24

---

Published as a conference paper at ICLR 2025
Coeadder tus pastion coess losls cxsation with spec
requlied selse the caestin).
should sitit only. muttipla weeds descrize skille recuired cnttinn. sbould discriptiive
you ruteistla questioe, than 3055 underscere..
bustiee:: (lqueatien)
you cereeonzes
shrees Liee of eeille weatired by stuttnt_.eccrpptte 11 to selve qiections soen seple 31: :4- esttl -33
Sutuce the sander of intaus statta by groaping sinttar seills Into casegaciaa and gise descripcive rase to each category.
The caceporias shulld io anually exclasise.
The cacegeries sbould destripine the stilts they contain..
sruindeg will medil penformance akilis.
perforeate an tapie I1 saaks
besign Sateguries coltecting
canccaraariee is
31
Leat cetesetegeii Ii categarie.
furcatitnn calting
Dusper eit Limm. since each category naw follomd colon and taxt iepossenaatone akiits thet sategery. teglata
cuppicte "(astee), (alteee Inpromntiiinnne.uu
Hiro, cognicing.iid, bipcaapecis
wieie wiltin strel, winlre Justnss wi flmaue
inpoesentative
Peeence plass not scp the respesse sisttico teiple teckticss.
ix eedir 32
"Treesr tane ta to ieeitilyy akStt recatrad rnd to aslve questice thas tasts (f taste H.
for skstt ia scitils S) seotec X)
nut ALANS crezse sctll
that scstts zrovided.
geecies: (queeettas))
Figure 13: LLM prompt templates for skill discovery.
25

---

Published as a conference paper at ICLR 2025
You exconreecced machine leanning
create traiingg data model.
Here some examples of the quesiion the modil will snswering cornect resporole the
Na for esarpçle ccamples %)
instrution: example instruciion
response: (t example ground truth, label
% endfor 96)
We will looss on improving skills undermeatn the calaggay "/ subakii W*
You will propose hyporheses about what training data the model needs to improve "V suneell Jf-
The hypotheese will contein speciicators trining and generate those cationn, and then train the model the
The you produce schema:
USDN using the provided schoma.
desoriptons
reepond
"inage": The resporsse
When cratting
coneider tollowing:
similar the images should relevant the instracionn
the instructions
previded
complextty examptes provided
think about what ledee the modal might misding that would help answer the questioo correctly and craft triining data to that knowledge each response should be a logicalty comnect response to the instrction the context of the mmge cescrption
the traiiig data should be divense and holp the model improve on 'If subskill 3f
Producce ( num, num_dalk, sptccsppesssssseooesco 3) speciicctions on for saiiing data.
Figure 14: LLM prompt template for data generation GQA.
You esper
your
improving soking
generated
for data problems
generated data %%)
Problem: dasa.prde casa.problem l)
Chiin of Thought: dela.dhwin.c of thought
Final Answer data. waa.ttnaaaassea answer
ts andfor Si)
1% endit %%
Each problem ahould improve the student's etsilly to salve problems under the calegory of 1// subsiii gr
Each problem should require the cludent know the concopt of "E: suboiil IP"
The proclems you produce must vald JSON using the provided achoma.
tere are cescriptons
Reeds in ochemar
"yrotiem" The math probtem you want model acive. Ernure
LaTix that property eaceped representation etting Python.
thought step-ty-step explenation
solve probliem Ereare is
properly secaped represemation string
answer" answer to problem as a string. For example (orao) "omatray/1
seroence here, just
Prepose budget
Figure 15: LLM prompt template for data generation MATH.
26

---

Published as a conference paper at ICLR 2025
You expert Python sngineer and competilve You are herping junior engineer improve their coding skills.
tolor.
its lib, aeamples si)
are represontative examples of the aidd coding probleme the junior engineer faciing esample pecamppalll 16)
Proztom:
ecumplle
3)
example starter, 16)
excample starter,
encil Ni
(%% example solution 16)
Solution:
X ecumpie sctution y)
I's enci %dl
(%% enctor (g
%)
focusing
requiring exeegory
You will proposie
problema that reaure applying skills the callgory of 'I suboic y
The problems you propose should be such that solking them will help the jurior engineer improve thair skila in the canegory of subokil 3y:
Here are some gu gukdeeinnes:
the prociems should be similar cod ng problems on piotoomms like LectCodo, Codstorces, etc.
the protees should require apaiving oxits in "I subseil y solution
only propose problems YOU
problem,
instuction: comprete problien atalement that would found in prace like LeeiCode This will be alhoon werbetzn to the junior anginwer. This should include an example input curput and condise explanation for why the outbut correct.
Do not who "242 Question,, just ouiput the problem antammnnt
starter oode: The starter code to the protrem. Not all problems need starter code.
you are induuiin starter pyttonn
formettdd
AunctonWWhhhe wowningnu parameter, IntSome Type), parameter. AnotherType):
YOOR CODE
Keep YOUR CODE HERE" in the codl block the junior ongines know where to Bll the solutio.
You can change fundttonwinttee intessnnntuuhaame anything want
Don't forget to also shange the per ameter names somathing that makes serse for the probiem.
Propose more than num, spoos new problems..
[ rum. startee oode, probiems 30 should have sterker codle.
remaining ( specs num, starter oodo protrems
Figure 16: LLM prompt template for data generation LiveCodeBench
You are excerienced engineer and your rale is provide training data correct model.
The model was given the following instruction and respanded incomretly.
instrcctoon: (l voal task, emron task instance instuctioon I)
Model Hesponse: voa task, emror precicorr response It Response: vqa, teak, instance ground Jabal
training improve
be
you provide.
traiiing you produce
USON following
"nllene. condise desciiation
modil
give
"instrucion"
otion the instruction about.
respond
desoription "rospenso": The
instruction.
When crafting the training datta, considor the fallowing:
the instructions buuld be simiiar style, longth, and complestyy original instruction the mmaes should be relevant the instrucion
the responses sho uld dimilar in style, lnngt, and complexity the original responce thin about what knowilddge the model might missing that would help anower the question correctly, and craft training data to give that knowledge each response should be a logicalty cornect. response to the instuction the contest of the iage cesomption the trainig data shouls be diverse anough to helb the model generalize new examples
Produce no mone than &f num, Iraining deta Y training data examples.
Figure 17: LLM prompt template for generation policy for OPEN-ENDED environment.
27

---

Published as a conference paper at ICLR 2025
You are experient enced math educator and your task create traningg catz for improving model's skills salking math protrems, especially under the category sf skill category Jr.
Here are esamples of mestkkee the model made while selving probrimm requirig 'Il skill canegory JF
The moow was given a math probrem and reeponded incomecilyy. r's for math task crror math, Jask, emons fo)
Problem: math. bask, semortadd task. instanne innnuiioon
Model Hesponser (f math task encr predictor response
Correct Responso: ( math matt_aask error lask, httpraaaa_ssiaaiaaaaaaaaaaa instance ground_ htw._uu____aaa Jubol
%i)
propose hyporheses
training
improve 'Il category improve examples
any
case, propose hypotheses
you produce wildd IBON provided
desemption fields
"iriemed skill contice descripiion
under
what kind (problem,
model
"protiem" The math probtem
model adive. Eireure this valid Tex that property eaceped represertation string Pythnn
tchain, of thought step-ly-step expleration how model should sclve problem. Ensure this waldd LaTixx that
excxped for representation
atring Python.
"tnol, annwert: The thai answer to the problem as LaTex string. For example 117 or Ydrro) 122) "Imatieg & Hor & 4) Do not write serrence here, just the ansiver.
Prodiue (i num, hypothesss 3) hyporheses. For sash hypothesis and weak ckil, produce (f hum_data_speee spectications training data.
Figure 18: Example of prompt for the SKILL-LIST environment for MATH.
28


Reviews:
[{"review_id": "r8ZflFk3T7", "reviewer": "Reviewer_VQ9Y", "summary": "This paper introduces Gym environments for data synthesis, framing the problem as sequential decision-making. In these environments, actions correspond to data-generation plans, and states represent the performance summary of a student model. The paper implements environments for three tasks: visual question answering (VQA), math, and code generation. Each environment offers three state representations: open-ended, skill-list, and skill-tree. Additionally, it proposes an LLM-based policy for data generation. Experimental results demonstrate that the LLM can make strategically effective choices based on environment-state information.", "questions": "- Is it possible to implement a random-policy baseline where you randomly chose a set of (naturally collected) datapoints from a data pool? The no-state baseline has flavor of this baseline but LLM-informed decisions could be biased. \n- Is it possible to compare this approach with active learning, in which instead of doing data generation, you do data *selection* and ask models to generate only synthetic labels, but not synthetic inputs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Tackle a timely and interesting problem. \n- Provide the necessary infrastructure for the community to study the problem, opening up opportunities for future contributions. \n- Consider various data generation strategies,\n- Well-desgined experiments which demonstrate the effectiveness of the proposed approaches and conduct insightful analyses.", "weaknesses": "* The paper is currently dense and difficult to follow. The introduction includes excessive implementation details, which detract from providing a simple, high-level intuition. Using a specific task example to guide readers through the core concepts would make the paper more accessible.\n\n* The paper focuses solely on the data generation plan rather than a full, end-to-end data generation process. It relies on a fixed, off-the-shelf data-generation engine that cannot be modified. The authors should admit this limitation and discuss potential strategies for overcoming it.\n\n* The quality of the data-generation engine can impact both student performance and the data-generation plan itself. Current approaches do not take into account the data-generation engine capabilities in the design of the policy or the evaluation of the student. For instance, poor student performance might result from the engine producing low-quality data on a specific skill, which could prompt the policy to avoid querying the engine for that skill.\n\n* The learning procedure can be resource-intensive. The authors should report the time, cost, and computing resources used for the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GMsjHLXdOx", "reviewer": "Reviewer_c5nB", "summary": "This paper presents a modular system for automated data generation, designed to minimize the need for human annotations. The proposed approach employs a reinforcement learning-inspired methodology, decomposing the process into a sequence of action predictions (data generation policy) based on state information (feedback from model errors) in an iterative manner. The effectiveness of this approach is demonstrated through three diverse tasks, encompassing text, image, and code generation across different modalities.", "questions": "In the Experiments section, the authors mention that the baseline student model should not have been heavily post-trained so that there are rooms for further improvements. However, it would be beneficial to provide additional evidence and details to support the claim that the student's performance is improved due to the added data points rather than insufficient training. For instance, the training protocol involved a fixed 10-epoch training period; it remains unclear whether the model had reached convergence within this timeframe or if the introduction of new data points accelerated convergence. Further clarification on this aspect would enhance the overall validity of the results.\n\nAlso the result would be more sound if more quantitative and qualitative results for skill discovery is reported in this paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel and insightful perspective on the autonomous data generation problem, leveraging principles from reinforcement learning to conceptualize it as a sequential decision-making process. The authors provide a thorough explanation of this approach, the motivations behind and the underlying mechanics.\n\nThis paper proposed a modular framework/testbed that can be easily adapted to various tasks, showcasing its versatility and potential for widespread applicability. The authors demonstrate the effectiveness of their approach through experiments on 3 tasks of multiple modalities, including text, image, and code generation, yielding promising early results.", "weaknesses": "The experiment part should be conducted more thoroughly: specifically, creating a test set that incorporates newly generated data points from the data generation agent and reporting evaluation results for each retrained model over successive iterations would provide more comprehensive insights into the system's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3QgWgrJff", "reviewer": "Reviewer_rVo8", "summary": "The paper presents DataEnvGym, a framework designed to simulate environments for data generation agents. These agents iteratively generate synthetic data to address weaknesses in student models, aiming to improve model performance across tasks like mathematics, programming, and visual question answering. DataEnvGym provides various structured environments (Open-Ended, Skill-List, and Skill-Tree) where data generation agents create targeted training examples based on feedback from the student model, offering a dynamic approach to automated model improvement.", "questions": "- How does the performance of the data generation agents change over longer iterations? The paper truncates experiments when performance increases, but it would be insightful to explore whether performance plateaus or continuously increase over extended training.\n- Is the total training data allocation fixed in each environment, or does it vary dynamically? The methodology mentions rebalancing but lacks clarity on how these allocations adjust adaptively based on feedback.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Good contribution to automated data generation for model improvement.\n- Clearly written with structured sections explaining each environment type and experimental results.", "weaknesses": "- The paper should clarify early on that the focus is on synthetic data generation for training purposes, as this underpins the motivation for the approach.\n- Important related works on algorithms using feedback from training to generate the next training environments are missing [1, 2, 3, 4].\n- Lines 460 - 465, I believe there is a typo whereby it says that “each experiment is truncated once the performance consistently decreases for multiple iterations”. Should it be “increases”?\n- Repeated runs of experiments without confidence intervals will be valuable, especially since the variance of performance seems to be very high.\n\n[1] Sudhakaran, S., González-Duque, M., Freiberger, M., Glanois, C., Najarro, E., & Risi, S. (2024). Mariogpt: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36.\n[2] Todd, G., Earle, S., Nasir, M. U., Green, M. C., & Togelius, J. (2023, April). Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games (pp. 1-8).\n[3] Zhang, J., Lehman, J., Stanley, K., & Clune, J. (2023). Omni: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711.\n[4] Faldor, M., Zhang, J., Cully, A., & Clune, J. (2024). OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. arXiv preprint arXiv:2405.15568.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7XT4kLWV2f", "reviewer": "Reviewer_wuGW", "summary": "This paper introduces DataEnvGym, a novel testbed of teacher environments for developing data generation agents that iteratively improve student models by generating targeted training data. DataEnvGym frames data generation as a sequential decision-making task where an agent, comprising a data generation policy and engine, interacts with an environment that provides feedback from a student model. The agent's goal is to improve student model performance by generating training data based on student feedback (errors or weak skills). DataEnvGym offers multiple instantiations of teacher environments across three levels of structure: open-ended, skill-list, and skill-tree, each with varying levels of scaffolding support. Experiments across text and image-based tasks (mathematics, programming, and visual question answering) demonstrate that example agents within DataEnvGym can iteratively improve student model performance. Furthermore, the authors analyze the impact of state information, environment structure, and skill discovery quality on agent performance and student learning. The paper concludes that DataEnvGym, with its modular design and support for diverse tasks and student models, provides a valuable platform for developing and evaluating data generation agents, engines, and feedback mechanisms for automated model improvement. The code and leaderboard will be publicly released.", "questions": "Limited Evaluation of Agent Architectures: The paper primarily focuses on introducing the DataEnvGym environment, but the evaluation of data generation agents is limited to relatively simple baseline policies. Exploring more sophisticated agent architectures, such as reinforcement learning agents (e.g., using policy gradient methods, Q-learning) or agents incorporating larger language models for planning and decision-making (similar to the approaches used in Shimabucoro et al. (2024), would substantially strengthen the paper. A systematic comparison of different agent architectures in terms of their effectiveness in improving student models, their sample efficiency, and their computational cost would provide valuable insights and contribute to a better understanding of the challenges and opportunities in automated data generation.\n\nLimited Analysis of Skill Discovery Quality: The paper briefly discusses the impact of oracle skills on student performance but doesn't delve deeply into the quality of the skills discovered by the proposed LLM-based method. A more thorough analysis is needed to understand the strengths and limitations of the skill discovery module. This could involve quantitative measures of skill quality, such as measuring their coherence, coverage, and relevance to the target task, or qualitative analysis by human experts. Investigating how the quality of the discovered skills affects the performance of the data generation agents and the resulting student models would strengthen the paper's contribution. Exploring alternative skill discovery methods (e.g., clustering-based approaches, topic modeling) and comparing their effectiveness with the proposed method would further enhance the analysis.\n\nLack of Comparison with Existing Methods: The paper positions DataEnvGym as a novel approach for model improvement, but it lacks a direct comparison with existing methods like curriculum learning (Bengio et al., 2009) or active learning (Settles, 2009). Evaluating how DataEnvGym compares to these established techniques in terms of student model performance, data efficiency, and computational cost would provide valuable context and highlight the advantages of the proposed framework. This would also clarify the specific niche and contribution of DataEnvGym within the broader landscape of model improvement techniques.\n\nLimited Discussion of Scalability: The experiments in the paper are conducted with relatively small datasets and models. It's essential to address the scalability of DataEnvGym to more realistic scenarios involving larger datasets, more complex models, and a broader range of skills. Discussing the computational challenges and potential optimizations for scaling the framework to more demanding settings would strengthen the paper's practical relevance. For instance, how can the computational cost of LLM-based data generation be reduced while maintaining data quality? How can the skill discovery and agent training processes be optimized for larger datasets? Addressing these questions would provide valuable insights for future research and practical applications.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novel Problem: Automating data generation to improve models is a significant challenge with practical applications. This work directly addresses this problem with a novel approach.\n\nWell-Defined Framework: DataEnvGym is presented as a well-defined framework with clear components (trainer, evaluator, data generation policy, data generation engine) and different levels of structure (open-ended, skill-list, skill-tree). This structure makes the problem tractable and facilitates modular development and testing.\n\nMultiple Tasks and Domains: The inclusion of experiments across diverse tasks (mathematics, programming, visual question answering) and with different student models demonstrates the generalizability of the framework.\n\nPromising Results: The initial results showing improved student model performance across tasks and environments are encouraging and suggest the potential of this approach. The analysis of difficulty/rarity and training dynamics adds value.\n\nOpen-Source Release: The commitment to publicly releasing the code and leaderboard promotes reproducibility and encourages further research in this area.", "weaknesses": "Limited Evaluation of Agent Architectures: The focus is primarily on the environment itself, with less emphasis on the architecture and training of the data generation agents. While baseline agents are provided, more sophisticated agent designs (e.g., reinforcement learning agents, agents leveraging larger language models) and their systematic evaluation would significantly strengthen the paper. How do different agent architectures compare in terms of effectiveness and efficiency? Are there specific architectural choices that are particularly well-suited for this task?\n\nOver-Reliance on LLMs for Data Generation: While using LLMs for data generation is a reasonable starting point, it raises concerns about the quality and diversity of the generated data. Exploring alternative data generation methods (e.g., data augmentation techniques, programmatic data generation) and comparing their effectiveness with LLM-based generation would be valuable. How robust is the framework to the quality of the generated data?\n\nLimited Analysis of Skill Discovery Quality: While the paper briefly touches upon the impact of skill discovery quality, a more thorough investigation is needed. How does the quality of the discovered skills affect the performance of the data generation agents and the student models? What are the limitations of the current skill discovery method, and how can it be improved? Quantitative analysis of skill quality (e.g., measuring coherence, coverage, and relevance) would strengthen the paper.\n\nLack of Comparison with Existing Methods: While related work on knowledge distillation and model weakness discovery is discussed, there's no direct comparison with existing methods for model improvement. How does DataEnvGym compare to techniques like curriculum learning or active learning in terms of effectiveness and efficiency? Including such comparisons would better contextualize the contributions and highlight the advantages of the proposed approach.\n\nLimited Discussion of Scalability: The experiments are conducted with relatively small datasets and models. How does DataEnvGym scale to larger datasets and more complex models? What are the computational challenges associated with training data generation agents in more realistic settings? Addressing these scalability concerns is crucial for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)