Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments
Authors: Hongjin SU
Ruoxi Sun
Jinsung Yoon
Pengcheng Yin
Tao Yu
Sercan O Arik
Venue: ICLR 2025 Poster (2025)
Keywords: Data synthesis
Agent
Adaptation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A data-centric framework to adapt LLM agents to any given environments without human annotations

Abstract: Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis.   The abilities of existing LLMs at such tasks are often hindered by the lack of high-quality agent data from the corresponding environments they interact with.  We propose LEARN-BY-INTERACT, a data-centric framework to adapt LLM agents to any given environments without human annotations.   LEARN-BY-INTERACT synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summarizing or abstracting the interaction histories, a process called backward construction. We assess the quality of our synthetic data by using them in both training-based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld, and Spider2-V spanning across realistic coding, web, and desktop environments show the effectiveness of LEARN-BY-INTERACT in various downstream agentic tasks — baseline results are improved up to 11.1% for ICL with Claude-3.5 and 23.1% for training with Codestral-22B. We further demonstrate the critical role of backward construction, which provides up to 10.6% improvement for training.  Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that LEARN-BY-INTERACT will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.

Published as a conference paper at ICLR 2025
LEARN-BY-INTERACT WORK FOR ENVIRONMENTS
A DATA- FRAME- AGENTS IN REALISTIC
Hongjin Su 12 Ruoxi Sun
Jinsung Yoon Pengcheng Yin Tao Yu 2 Serean Ö. Arık
Google 2 The University of Hong Kong
ABSTRACT
Autonomous agents powered by large language models (LLMs) have the potential to enhance human capabilities, assisting with digital tasks from sending emails to performing data analysis. The abilities of existing LLMs at such tasks are of- ten hindered by the lack of high-quality agent data from the corresponding envi- ronments they interact with. We propose framework to adapt LLM agents to any given environments without human an- notations. LEARN-BY-INTERR
data-centric
synthesizes trajectories of agent-environment interactions based on documentations, and constructs instructions by summariz- ing or abstracting the interaction histories, process called backward construction. We assess the quality of our synthetic data by using them in both training- based scenarios and training-free in-context learning (ICL), where we craft innovative retrieval approaches optimized for agents. Extensive experiments on SWE-bench, WebArena, OSWorld and Spider2-V V spanning across realistic coding, web, and desktop environments show the effectiveness of downstream agentic tasks - bascline results are improved by up to 11.1% % for ICL with Claude-3.5 and 23.1% 1% for training with Codestral-22B. We further demon- strate the critical role of backward construction, which provides up to 10.6% im- provement for training. Our ablation studies demonstrate the efficiency provided by our synthesized data in ICL and the superiority of our retrieval pipeline over alternative approaches like conventional retrieval-augmented generation (RAG). We expect that LEAARN-BY-INTERAC will serve as a foundation for agent data synthesis as LLMs are increasingly deployed at real-world environments.
in various
INTRODUCTION
Pre-trained large language models (LLMs) offer great potential for assisting humans with various tasks in digital settings, such as editing images, performing data analysis, resolving software en-
gineering issues, and navigating commercial platforms (Xie al. 2023 2024 Yao al. 2022a
Jimenez et al. 2023). By streamlining these, LLM agents can greatly enhance human efficiency and productivity, allowing individuals to shift their focus toward higher-level, creative, and strategic en-
deavors. To explore this potential, many benchmarks (Jimenez et al. 2023 Zhou et al. 2023bb b) Xie
Cao et al., 2024 Koh al., 2024) and agentic Trameworks (Yang al. [2024] Zhan & Zhang 2023 Yang et al. (223) Gur et al. 2023 Chen et al., 2024a) have been established based on realistic digital environments, spanning web applications, code development, desktop computing, etc. However, current LLMs often fall short of expected performance in these tasks, consistently displaying significant gap compared human capabilities. As a result, they remain less practical and reliable for real-worls applications.
al. 2024
Efficient adaptation to new environments can be the key part of the performance improvements.
Prior works have explored various prompt-based approaches (Yao et al., 2022b Yang et al. 2024
Gur et al. 2023 Zhan & Zhang 2023), that are constrained by the capabilities of underlying foun-
dation models. Other studies training LLMs with human-labeled examples (Chen et al. 2023 2024b Li et al. 2020) on the other hand, come with the fundamental limitation of high annotation
costs when environments are considered. In particular, annotating agentic data be quite

---

Published as a conference paper at ICLR 2025
Synthesized data
Agent-environment interaction
Environments
Genarated task instruction by salt instruct: Upload CSV file Google Drive BigQuery
Instruction 1: Rapieate the Trajectory 10bst Ac12, Dos2)
LLM Agent
Code: ?
Obso: BieQuery Homepage.
Software:
Instruction n: Link CSV fie Trajectory lobso Obst3l
Actl: view dataset "demo"
Backward construction
Resources
10bs1, Ast2, 0bx2)
Tutorial and Decumentation
Obst: Dataset with into like creation time
Replicate the fellowing: In the datasel click
Filtered data
Training
Lowainng CSV
Act2: click button create table
In-context learning
1. Go to the eigQuery
create
<-
Obs2: Table creation page
Instruction
Agantic ratrievaa
select table source Google Storage, (wrong preaiction misaligned with instruction)
(Obso, Obs13)
Obs
Modee- based:
Rind csamples with similar
FAQ
Update instruction
Act
intant and trajectory
1. How create BigQuory table? Answer
jlign with oraicenory
Obs
Ooservation based: observation
0bs13: BigQuery table creeted.
......
Link CSV file Google Storage to Bigluerry
Act
with
Overview of the data synthesis and adaptation processes. Given an environment and stan- dard resources, we first leverage self-instruct to create diverse set of instructions. LLMs are then employed to complete these tasks, resulting in long trajectories of agent-environment interactions. We construct task instructions using LLMs for each sub- uub-trajectory. process called backward con- struction. The synthesized data are then filtered and used for both training and in-context learning, where we design agentic retrieval to retrieve demonstration examples based on information each step, using both model-based and observation-base approaches. See Appendix F for the complete data synthesis example and Algorithm| 2 for more details on agentic retrieval.
Figure 1:
difficult and expensive due to long-trajectory interactions with environments and specific domain expertise required. Few works have explored fully autonomous data construction pipelines towards self-adaptive agents that can efficiently learn new environments (Gulcehre et al. 2023 Aksitov et al. 2023).
In this paper, we introduce LEARN-BY-INTER.N a data-centric framework for LLMs to self- new environments, utilizing agent data synthesis via interactions. Intuitively, the effects of ac- tions executed in environments (e.g., the next webpage after clicking a button) serve as informa- tive demonstrations that help LLMs in future navigation. Inspired by this, we design LEARN-BY- INTERACT that first uses self-instruct (Wang et al., 2022b) to develop a variety of task instructions, referring to standard resources such as documentations and tutorials for a given environment. This covers most important scenarios that human users are interested in and avoids intensive prompt en- gineering to control the distribution and diversity of the generated data. We then collect diverse tra- jectories from interactions between LLMs and environments, illustrated in Fig. However, given the low performance of L.MSs in existing agentic benchmarks (Xie et al. 2024 Cao et al. 2024), is likely that large percentage of synthesized trajectories do not match with the instructions. To tackle this challenge, we construct new instructions by summarizing or abstracting each sub-trajectory, leveraging the strong summarization capabilities of L.MSs (Pu al., 2023 Liu et al. (023). We call this process backward construction. After obtaining synthesized instruction-trajectorr pairs and filtering low-quality ones, we apply it both training and ICL, where we craft innovative retrieval pipelines optimized for agents. Concretely, it consists of two parts: (I), model-based approach that leverages LLMs first write queries based on instructions, interaction histories and current obser- vations, and uses retrieval models to retrieve demonstration examples from synthesized data; (2). observation- baaeed approach that finds examples with the current observation appearing trajecto- ries (which indicates that the current state has been encountered in the data synthesis process).
Our comprehensive evaluations across four challenging benchmarks: SWE-bench (Jimenez et al. 2023). WebArena (Zhou et al. (023b), OSWorld (Xie et al. 2024), and Spider2-V (Cao et al. 2024)) highlight the efficacy of the data generated by LEARN-BY-INTERATT With ICL, both Gemini-1.. 1.5- pro (Reid et al. 2024) and Claude-3.5 3.5--3nn-eeee (Anthropic 2024) show consistent and remarkable im- provements - for OS World (Xie et al. 2024), our generated data nearly doubles Claude-3.5 Staude3355-oonnee -onne''ss''s baseline performance, increasing it from 11.4% to 22.5%. Furthermore, substantial improvements are observed by training models of varying sizes and architectures with our synthesized data. As
an example, Codestral-22BB: (Team 2024b) performance in WebArena significantly increases from
2

---

Published as a conference paper at ICLR 2025
4.7% to 27.8% after training. These results underscore the high quality of our generated agent data and its broad applicability across diverse agent environments.
Our extensive ablation studies reveal that backward construction not only increases the quantity of the synthesized data, but also improves its overall quality ($3.)). With data synthesized by LEARN- BY-INTERACT, observe significant improvements in both performance and efficiency during LLM inference ($4.1). Our empirical results demonstrate the superiority of the agentic retrieval in ICL ($4.2). We anticipate that this research will spark innovative developments in enhancing agent performance using LLMs and contribute to its wider-spread adoption in real-world application scenarios.
LEARN-BY-INTERACT
We introduce the LEARN-BY Y-INTERACT pipeline synthesize agent data in an autonomous way by leveraging interactions between LLMs and environments. We first formalize the agent canonical task [. and introduce the detailed synthesis ($2.) and filtering ($2.3) procedures. We then describe the application of the synthesized data in adapting LLMs in both training-free and training- -based settings (.44.
2.1 TASK FORMULATION
Given environment E and task instruction 7, the objective of agent A is to achieve the target G through multi-step interactions with E. At each step i, A predicts the next action a, based on the instruction / and the previous history environment E to get a new observation o₁. The interactions terminated until A predicts the action stop or the maximum number of steps m is reached.
(oo: a1 01, 022 ... 0i-1). which is then executed in the
2.2 AGENTIC DATA SYNTHESIS
The essential idea of LEARN-BY-INTERACT manifested in synthesizing environment- specific agent data with zero human effort. In Algorithm [ we show the overall pro- cess with pseudo-code. Given an environ- ment for a downstream application (such as vi- sual studio code), we first leverage commonly- accessible resources like documentation to generate diverse task instructions using self- instruct (Wang et al., (022b) (line 5). These resources are usually created by human experts to address common concerns and provide usage suggestions, e.g., how to navigate a website or operate software. Intuitively, such references often cover representative usecases of an ap- plication. Therefore the task instructions gen- crated conditioned on them could cover most
Algorithm Agent data synthesis
I: Input: LLM: Large Language Model; E: envi- ronment; Doc: standard resources like documenta- tion; N: the number of instructions generate per document; F: data filter.
Initialization:
= D:: synthesized data.
3: for d in Doc do
4:
I/ self-instruct generate N task instructions Instructions (d,
5:
Instructions
7:
E.reset)
8:
T =1 // initialize interaction trajectory
9:
while not E.finshhed) do
E.get_observaatons
(L, T += [o
12:
13:
end while
14:
T wappnd((EEpp.... B.geeee.eeeeeeettioe...
popular scenarios in the domain and avoids po- tentially unreasonable cases that may be of less value.
15:
// backward construction
16:
for in range(O, (1n) - 1) do
17:
for in range(i + 2, len((1))) do
18:
T T i r LLM(TT)
For each generated task, LLMs then aim to solve it, which results in long trajectory
19:
20:
D D.append((1' T'))
T = =(0p. 01, O1. an, 0n) (line 9-14 in Algo-
21:
end for
rithm II. To address the potential misalignment
22:
end for
between the instruction / and the generated tra-
23:
end for
jectories T, we introduce a novel mechanism,
24: end for
backward construction, to construct instruc-
25: D F(D) // Filter low low-quality data
tions based on trajectories (line 15-22 Algo-
26: Return: D
rithm I. Specifically, for each sub-trajectory
3

---

Published as a conference paper at ICLR 2025
T" (ot, ai+1, Oi+1, ....,........................... O.. 1 j we obtain two types of new instructions: (I). summarizations of trajectory steps; (2). abstractions of the trajectory purpose. In Fig. [ the sub- trajectory (Obs1, Act2, Obs2) is summarized into new task instruction that requires to replicate the Act2. The abstraction of the full trajectory updates the original task objective, which is no longer aligned with the generated trajectory due to the wrong prediction in the action 3. Overall, the LEARN-BY-INTERAS pipeline offers two notable advantages:
It corrects the potential misalignment between instructions and predicted trajectories by updating task objectives, which enhances the data quality as verified by the experimental results in $3.5
It maximizes the utility of each generated trajectory by crafting new instructions for each sub- trajectory. This results in a quadratic increase in the number of synthesized examples with respect to the steps in the sequence per generated trajectory. For given target dataset size, backward construction substantially decreases the necessary interactions, which is particularly valuable in scenarios where such interactions are challenging and costly to obtain such Robotics (Keipour) 2022).
2.3 FILTERING
To further enhance the data quality, we design the following criteria to filter inferior synthesized data: (1). Remove duplicate states: We remove duplicate (a, 0,) from T if (a,, o,)==(:-1,, 0(-1), which is potentially introduced by the invalid action or the environment error (inactivity). (2). LLM commit- tee check: We feed the generated instruction-trajectory pair (T', (") into committee of LLMs, and only classify it of high-quality all LLMS consider the trajectory coherent, natural, reasonable and aligned with the instruction. The listed criteria are all fully-autonomous and munoncclly--aplicabbe for filtering data synthesized in general agent scenarios. See Table 35 for our prompts used in LLM committee check.
2.4 ADAPTATION
After obtaining the synthesized data D, ap- ply it to both ICL and training. Given the unique characteristies of multi-round interac- tions with environments in agent settings, we design agentic retrieval (pseudo-code in Al- gorithm 2) to maximize the effectiveness of the synthesized data. Specifically, we propose two retrieval pipelines: Observationnbasee (line 5-14) and model-based retrieval (line 15-17). In Doservation-based retrieval, compare the current observation to the trajectory of each example in the synthesized data, where [I', [op, a1, OL.... an o,]]. J. If o matches one of the observations in i.e.,
Algorithm ICL. with agentic retrieval
I: Input: LLM: Large Language Model; E: envi- ronment; D: synthesized data; RM: retriever; I: task instruction; m1: maximum number of exam- ples from observation based retrieval; m2: max- imum number examples from model-hased re- trieval.
2: Initialization: H interaction history; R: re- trieved examples.
while not E E.finished(
4:
= E.get observation() // observation-based retrieval for i, t in D do
5:
6:
O;, then consider e as a helpful example to the current task. For the model-based retrieval, we lever- 9:
// iterate through the trajectory for 0r in do if 01=0 then Rappend 1
10:
age LLMS to first write queries based on the instruction, the interaction history and the cur-
11:
end if end for end for
12:
rent observation (line 16), and then employ re-
13:
trieval models to retrieve non-duplicate exam-
14:
H R[mm]]
ples (line 17). L.Mss are then augmented with
15:
// model-based retrieval
the retrieved examples to predict the next action
16:
9 LLM H)
(line 18). Refer to Table 36 to 39 for prompts
17:
+= RM (g. D, m2, R)
to write queries and predict actions.
18:
a = LLM H. R))
19:
H = [o,a]
Apart from using the synthesized data as demonstration examples in ICL, we further uti- lize them fine-tune models. For given generated example, convert it to the format of action prediction (Table 36). and prepare input-output pairs for supervised fine-tuning. More details on the experimental settings can be found in $3.
20:
end while
4

---

Published as a conference paper at ICLR 2025
Table 1: Statistics for the number of crawled documents, generated raw trajectories, examples Iestruction-traeccute pairs) and examples after filtering.
SWE-bench
WebArena
OSWorld
Spider2-V
Documents Raw trajectories
6,464
3,578
7,362
11,231
19,392
10,734
22,086
33,693
Examples
180,752
185,635
437,635
652,786
Filtered examples
101,523
109,276
103,526
125,683
EXPERIMENTS
3.1 BASELINES
We compare ICL with agentic retrieval to the following prompt-based approaches.
The vanilla prediction pipeline in each benchmark that includes the task instruction, interaction history and the state observation in the prompt. See more implementation details in Appendix
Baseline:
The conventional RAG pipeline that first retrieves from the resources like documentation based on the instruction, and augments LLMs with the retrieved content.
RAG:
Data distill: We follow same pipeline to synthesize data Algorithm except backward construction (replace line 15-22 with D.appnnd([, and follow Algorithm during the evalu- ation.
Reflexion (Shinn et al. 2024): :: A general framework to reinforce language agents through linguis- tic feedback from both executors and LLMs.
Language Agent Tree Search (LATS) (Zhou et al. 2023a): It integrates the combinatorial tree search into expanding ReAct (Yao et al. (022b) and combine agent online reasoning, acting and planning throughout the trajectory.
For the training-based evaluation, we primarily compare the data distillation, which also con- structs data from scratch and requires no human effort to annotate seed or preference data. Addi- tionally, we include the model performance before training as another baseline.
3.2 DATASETS
We consider 4 agent datasets that involve multi-roum tound interactions with realistic environments. They span diverse domains of code, web, computer desktop and professional software. Appendix[Cinn trates details of each dataset with examples.
SWE-bench (Jimenez et al. 2023) is an evaluation benchmark on realistic software engineering problems from realistic Github issues. We use the Lite version by default throughout the experi- ments.
Webarena Zhou et al. 2023b) evaluates agent capabilities to perform tasks in the web environ- ments such e-commerce, social forum discussion, and beyond.
OSWorld (Xie al., which involve diverse applications like terminal, chrome, etc.
2024) is an integrated environment for assessing open-ended computer tasks,
Spider2-V (Cao al. 2024) is multimodal agent benchmark focusing professional data
science and engineering workflows, which includes BigQuery, Airbyte and more.
3.3 SETTINGS
We synthesize one separate set of environment-specife data for each evaluated benchmark. Throughout the data synthesis process, employ the Claude-3.5-sonnet (Anthropic 2024) the generator model and both Gemini-1.5-pp (Reid et al.) (2044) and Claude-3.5-somee as the LLM committee for filtering low-quality data. For each document, sample three task instructions from
5

---

Published as a conference paper at ICLR 2025
other existing training-free approaches. SWE refers to SWE-bench, Web refers to WebArena and OS refers to OSWorld. The best results are highlighted in bold. We include more leaderboard results of SWE-bench and WebArena in Table
Table 2:
Comparison of LEARN-
Benchmark >
SWE
Web
OS
Spider2-V
SWE
Web
OS
Spider2-V
Approach
Gemini-1.5-po
Claude-3.5-some
Existing approaches
Baseline
13.3
17.9
4.9
8.3
26.7
31.5
11.4
7.5
RAG
13.7
19.5
5.1
9.1
27.0
31.8
11.7
7.7
Data distill
14.0
19.8
5.7
9.1
28.0
32.1
11.9
8.5
Reflexion
14.3
20.2
5.7
9.3
28.3
32.4
12.2
8.9
LATS
15.3
21.0
6.5
11.3
29.0
34.2
13.6
10.3
Ours
Learn-by-interact A over baseline
18.7
25.6
10.3
16.4
34.7
39.2
22.5
16.3
+5.4
+7.7
+5.4
+8.1
+8.0
47.7
+11.1
+8.8
LLMs. The statistics for generated raw trajectories, examples before and after filtering are shown in Table [ In Appendix E list document sources used for each benchmark. During ICL,, retrieve examples until the maximum length of LLMs and set an upper bound 5 for both model- m2 5 in Algorithm 2). We leverage Gemini-1.5.
based and observation-basel retrieval (m1 pro (Reid et al. 2024) and Claude-3.5-sonne (Anthropic 2024 D Codegemma-7BB (Team 2024a) and Codestral -2B (Tem 2024b) in the ICL evaluation, and tune Codegemma-7B and Codestral- 22B with LoRA Hu et al. 2021 to evaluate the data quality training sources. By default, we evaluating the trained model to avoid the confusion under- standing the effectiveness of our synthesized data in training. We include more detailed hyper- and machine information
do not include retrieval content
parameter settings (both existing approaches and in Appendix
3.4 EVALUATION
We follow the default evaluation metrics designed by the original benchmarks. In SWE- bench (Jimenez al. 2023), apply the generated patch program the repository codebase, and measure the agent performance by execution accuracy (pass@) I). In WebArena (Zhou et al. (023b), employ both LLM-based fuzzy match and string match that checks keywords in predictions. Slightly different from the original work that uses gpt-4-06 as the LLM judge, we use Claude- 3.5-sonner as a similar replacement. In OSWorl (Xie al.) 2024), we leverage the sample-specific evaluation scripts to assess the functional correctness of the task completion, which processes en- vironment states and checks if agents finish the task as expected. In Spider2-V (Cao et al., (024), we utilize file-based comparison, information-based validation, execution- based verification to de- termine whether task is successfully completed. All performance numbers throughout the paper shown the percentage of resolved instances with % omitted for brevity.
3.5 RESULTS
3.5.
TRAINING-FREE EVALUATION
We first consider LEARN-BY-INTERACE in the training-free setting, where the proposed methods can be applied to the commercial LLMs even with prediction-only API access.
Results on Table show marginal improvement of RAG compared to the baseline, which suggests limited effectiveness by simply concatenating standard reousrces to LLM prompts. By retrieving examples from distilled data, observe better performance compared to RAG, but still no more than 2% improvement over the baseline, which indicates that the distilled data tend to be noisy in the setting with multi-round agent-environment interactions. This highlights the critical role of
In the subsequent descriptions, Gemini refers Gemini-1.5 1.5-po, and Claude refers Claude-3.5-sonnee

---

Published as a conference paper at ICLR 2025
Table 3: Downstream task performance of models trained from data generated by Learning-by- interact and data distillation. We include the models results before training, where the synthesized data is used demonstration examples, and after training, where the synthesized data is used to train models.
Benchmark ->
Web
OS
Web OS
Web OS
Web OS
Model
Codegemma-7B
Codestral-223
Codegemma-77B
Codestral-221
Approach 4
Before tuning
After tuning
Existing approaches
Baseline
3.3
0.0
4.7
2.2
Data distill
4.2
0.0
5,8
2.7
6.2
1.4
10.2
5.4
Ours
Leatn-by-intrreet over baseline
7.6
3.5
9.9
5.4
17.9
6.5
27.8
11.7
+4.3
+3.5
+5.2
+3.2
+14.5
+6.5
+23.1
+9.5
backward construction, which corrects the misalignment between instructions and trajectories by curating new task objectives.
Both Reflexion and LATS consistently improve over the baseline across benchmarks, which demonstrate their general applicability to agent tasks. Using the data synthesized from the LEARN- DY-INTERACT, we can see a significant performance gain compared to other frameworks both Gemini and Claude. For example, in OSWorld, augmenting Claude with synthesized environment- specific data almost doubles the result compared to the bascline. This signifies the high quality of the generated data and the effecttivnness of the
framework.
3.5.2 TRAINING-BASED EVALUATION
We consider the data synthesized by LEARN-BY-INTERACT the scenario of LLM tuning, which is applicable to the LLMs with access to weight updates.
The results presented in Table 3 reveal that LEARN- BEERR-BBY-INE BY-INTERACT substantially surpasses both the baseline and data distillation, suggesting its capacity to generate high-quality training data that en- ables language models learn and adapt efficiently. We discover that utilizing our synthesized data for model training yields better results compared to using it as in-context learning (ICL. examples. notable instance is in WebArena, where Codestral-225 performance jumps from 4.7% to 27.8% when trained our synthesized data, while only showing 5.2% improvement in the ICL scenario. Remarkably, the Codestral- 22B model trained with our synthesized data even outperforms Gemini when the latter uses our data demonstration examples.
ANALYSIS
4.1 INFERENCE EFFICIENCY
We compare the efficiency of different pipelines at inference. We analyze the trade-off between downstream task performance and the required computational costs. We focus on measuring the number of LLM calls and consumed tokens per example, which are averaged across four evalu- ated datasets ($3.2) using Claude-3.5-sonnet. As illustrated in Fig. 2 while Reflexion and LATS demonstrate enhanced performance, this comes at the cost of significantly increased computational resources during inference. Specifically, LATS yields 2.5% improvement on average, but re- quires nearly four times used tokens per instance relative to the baseline. In contrast, LEARN-BY- INTERACT exhibits superior performance while utilizing fewer LLM calls and slightly more tokens compared the bascline. Thanks to the rich environment information stored in the examples of synthesized data, LLMs can potentially make better decisions and thus finish the task in fewer steps. This removes the performance-efffciengy trade-off during inference at the cost of data synthesis in

---

Published as a conference paper at ICLR 2025
Performance
LLM calls
Consumed tokens
29
250k
35
200k
30
25
25
150k
20
15
100k
21
10
50k
P7
Baseline
RAG
Data distill
Reflexion
LATS
Figure 2: Evaluation performance, the number of LLM calls and consumed tokens (per example) of various training-free pipelines during inference, which are all averaged across four benchmarks: SWE-bench, Webarena, OSWorl and Spider2-V.
Table 4: Model performance based on different retrieval paradigms. Observation-based and Model- based retrieval prove be particularly effective in agent tasks, whose combination (ours) gives the best results.
Benchmark +
SWE
Web
OS
Spider2-V
SWE
Web
OS
Spider2-V
Retrieval
Gemini-1..-pro 1.5-pro
Claude-3. -oadee-35--onne
No retrieval
13.3
17.9
4.9
8.3
26.7
31.5
11.4
7.5
Instruction-based
14.7
21.6
7.0
10.2
27.7
33.6
15.7
9.1
Observation-based
16.3
23.5
8.7
14.6
32.3
36.3
18.7
13.2
Model-based
17.0
24.3
9.5
15.4
33.7
37.2
20.3
14.5
Ours
18.7
25.6
10.3
16.4
34.7
39.2
22.5
16.3
advance and suggests that LEARN-BY-INTERACT particularly well-suited for real-world deploy- ment that demonds both low latency and high performance.
4.2 THE IMPACT OF RETRIEVAL
As mentioned in 22.4 employ both model-based and observation-base retrieval in our evalu- ation with ICL. We analyze their effectiveness by incorporating only one of them (skip line 5- in Algorithm 2 for model-based retrieval only and skip line 15-17 for observation-based retrieval only). In addition, we compare to two baselines: (1). no retrieval: LLMs predict each action in the zero-shot setting; (2), instruction-basse only use instructions retrieve synthesized data and apply the same demonstration examples in every action prediction throughout the trajectory.
The results presented in Table illustrate how various retrieval methods impact LLMs when us- ing the synthetic data as the retrieval source. Despite having access to the same example pool (except the baseline without using retrieval), there are notable differences different retrieval strategies, highlighting the crucial role of agentic retrieval in effectively utilizing synthesized data. Traditional Retrieval-Augmentt ploys instructions for retrieval, show the least improvement across four benchmarks and two LLMs. In contrast, the observation-baut approach proves particularly effective for agent-basee tasks, sig- nificantly outperforming the instruction-bass retrieval, for instance, achieving 4.4% absolute im- provement in Spider-2V when using Gemini. By leveraging task instructions, interaction history and the current observation, model-based retrieval demonstrates even better results compared to using the observation-base version. Ultimately, the most impressive scores are achieved by combining both model-based and observation- baaedd retrieval, which results in our agentic retrieval pipeline. These findings underscore the importance of carefully designing retrieval pipelines to maximize the potential of synthetic data and L.MS in agent scenarios.
performance across
Generation (RAG) methods, which only em-
4.3 DATA GRANULARITY
8

---

Published as a conference paper at ICLR 2025
Table 5: Effectiveness of synthetic data with various granularity. In general, short-trajectory data is more advantageous to both training and ICL, while mixing all of short, medium and long-trajectory data provides the best performance.
Benchmark
SWE
Web
OS
Spider2-V
Web
OS
Granularity +
Claude-3 3.5-sonnet
Codestral-22
Baseline
26.7
31.5
11.4
7.7
4.6
2.2
Short
28.7
33.3
14.9
10.3
13.5
4.9
Medium
28.0
32.5
13.8
9.5
12.6
4.0
Long
27.3
31.9
13.0
8.9
10.6
3.4
Short=Medium
30.0
34.4
15.7
10.7
14.6
5.7
Short+Long
29.3
33.9
15.2
10.5
14.4
5.3
Medium+Long
28.7
32.9
14.4
10.1
13.2
4.5
Short-Medium+Lonng
31.0
34.9
16.3
11.3
15.4
6.3
As mentioned in $2. we synthesize data by taking contiguous sub-trajectories from the full generation paths of LLMs, i.e. T" Ti j].
Cludee3.--somet Codegommo
Gemink-1.5pp Codastral 228 Codestral
which results in trajectories of diverse lengths in the synthesized data. We divide the syn- thetic data into three groups: (1). trajec-
30
tory steps < 5 (short); (2). 5 S trajec- tory steps < 10 (medium); (3). trajectory steps 10 (long), and leverage each group and their combinations free and the training-based process. To en-
Perforance
20
25-
both the training-
5-
sure a fair comparison, we constraint the data size in each group and combined group to 200M token: utilizing Su et al. (2022) for sub- samplin.. Table [5] presents the results. In both training-free and training-based eval- uation, L.Mss derive greater advantages from short-trajed trojt-trajectory data, demonstrated by its consistently superior performance compared to medium and long- trajeccccy data with Claude- 3.5-sonnet and Codestral-22B. This can be at- tributed the versatility of short-trajectoy data, which usually serves a sub-step or partial workflow in downstream tasks. The combination of any two data groups proves more effective than relying on single group, showcasing the complementary nature of diverse data sets. For instance, in Webarena with Codestral-221 -2BB, incorporating examples with both short and medium-lenge jectories shows additional improvement over using either one exclusively (14.6 vs 13.5 and 14.6 vs 12.6). This underscores the value of considering the trajectory length as unique dimension of agent data synthesis.
20k
80k
Synthesized
Figure 3: Scaling laws of the synthesized data. Compared in-context learning, tuned models achieves more significant improvements the data scales up. The performance is averaged across WebArena and OSWorld.
tra-
4.4 SCALING LAWS
We examine how the model performance improves as the synthetic data size scales up. Figure presents sets of results, with training-free (where Claude, Gemini, Codegemma and Codestral use retrieval augmentation without training) and with training- (where fine-tuned Codegemma and Codestral models are evaluated without retrieval). All results averaged across Webarena and OSworld due to the limit of computational resources. The findings indicate that both learning paradigms benefit from larger data, suggesting the synthetic data is diverse and high-quality. In the training-free evaluation, more substantial improvements are observed for larger models (Claude and Gemini) compared smaller ones (Codegemma and Codestral), possibly due the enhanced
We use the number of tokens measure the data size due to the fact that long-trajectory example may contain information compared to the short version.

---

Published as a conference paper at ICLR 2025
in-context learning abilities of larger models. Our analysis also reveals that for given amount of synthetic data, fine-tuning smaller models is more effective than using the data as demonstration examples during evaluation.
5 RELATED WORK
Various agents based on LLMs have been developed (Wang et al.
2024 Zhang et al. 2024; Shinn
al. 2024, Huang et al. 2022 Wang et al.] 2023a b). React Yao et al. 2022b) proposes to
synergize reasoning and acting in LLMs. By integrating Monte Carlo Tree Search (Kocsis &
Szeeesván 2006; Coulom 2006
Zhou al. (2023a) leverages LLM-powered value functions
and self-reflection (Madaan et al. 2024) to encourage proficient exploration and decision-
However, it comes with increased computational costs and relies on the premise that the environ- ment allows for state reversals. In contrast, LEARN-BY-INTERAN
removes such assumptions and improves both agent efficiency and performance by synthesizing high-quality data in advance.
Another line of research to improve agent models relies training on human-labeled exam- ples (Zeng et al. 2023] Yin et al. [023 Deng et al. 2024
Chen et al. 2024b Wang et al. 2022a)
or data distilled from LLMs like GPT-4 Chen et al.
2023 Zhao et al. 2024). AgentGen (Hu al,
2024) explores automatic synthesis of both environments and tasks and then leverages Dowt ward generate trajectory data. AgentTuning (Zeng et al. 2023) utilizes both existing datasets and self-instruct (Wang et al. (022b) to derive instructions and then samples trajectories from GPT- 4 (Achiam al., 2023). In contrast, LEARN-BY-INTERACE focuses realistic environments and generate tasks and trajectories using backward construction. Some other researchers are also ex- ploring ways to use data more efficiently with reinforcement learning (Ball et al., 2023 Schwarzer et 20.22 Nachhu
Nachum et al. 2018 Thomas & Brunskill Schwarrr et al. 2021)) Gulcehre et al. (2023) suggests using data created by an LLM's policy can enhance itself via offline reinforcement learning algorithms. AAsitovv et al. (2023) takes this further by combining with ReAct (Yao et al.) 2022b) to train agent models iteratively on experience trajectories. These typically require a reward feedback to enhance data quality. Our work, however, takes different approach by employing the backward construction to improve the data quality by aligning instructions and trajectories.
model as the scoring function LLM/execution--eeeeee
6 CONCLUSION
We introduce LEARN-BY-INTERACT, a data-centric framework to adapt LLM agents to any given environments without human annotations. Based on commonly-accessible resources like documen- taion, LLMs propose downstream tasks and complete them with multi-round interactions with en- vironments. We address the misalignment between instructions and trajectories by updating objec- tives with new instructions derived from trajectories. Additionally, we design innovative retrieval pipelines that leverage agent instructions, interaction histories, and current observations retrieve synthesized examples. Through extensive experiments, we demonstrate that the synthetic data from LEARN-BY-INTERACE significantly enhances model performance ICL and training. Compared with other leading approaches in agent tasks, LEARN-BY-INTERACT shows much better perfor- mance with lower latency and computational costs, which make it particularly suitable for large- scale deployment. Further analysis has also shown the superiority of B-RNN-B--NEERATT over the classical RAG. In future work, we plan to explore multi-modal settings and train general agent models widely applicable in realistic environments. We anticipate that LEARN-BY-INTERACC, will inspire future research to push the state-of-the-ee in this direction.
7 LIMITATIONS
Although BEARN-BY-INTERAC effectively synthesizes high-quality agentic data with trajectories, it requires lot of LLM calls in generation and filtering. We hope that future works will explore more efficient approaches to complete annotations without sacrificing quality. Additionally, LEARN-BY INTERACT leverages the environment-relate resources to generate instructions. In some scenarios, however, these resources may be incomplete or not available.
htaps:/wwww.faxx-ooowwrr..r.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint ar.ii:2303.087777 2023.
Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self- improvement for multi-step reasoning I1m agent. arXiv preprint arx1v:2312.10003, 2023.
Anthropic. Introducing claude 3.5 sonnet, 2024. URL https::/www.anteen
com/
news/clauee 3-5-sonnet]
Philip Ball, Laura Smith. lly Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-v: How far are mul- timodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10956, 2024.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint 2rXiv:2310.05915 2023.
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong. Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint wwx.i:24406011004 2024a.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangming Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.1 /288/, 2024b.
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2weh: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding. and pro- gram synthesis. arXiv preprint arXiv:2307. 12856, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint SXXv:2206.00685. 2021.
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, and Dongmei Zhang. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint arXiv:2408.0077
2024.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 9118-9147 PMLR, 2022.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
11

---

Published as a conference paper at ICLR 2025
Azarakhsh Keipour. Physical interaction and manipulation of the environment using aerial robots. arXiv preprint arXiv:2207.02886, 2022.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Pu Huang. Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv e-prints, pp. arXiv-2401, 2024.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020.
Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye. Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. On learning to summarize with large language models as references. arXiv preprint arXiv:2305. /4239, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-kke Advances in Neural Information Processing Systems, 36, 2024.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficiens hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Xiao Pu, Mingqi Gao, and Xiaojun Wan. arXiv:2309.09550 2023.
Summarization is (almost) dead. arXiv preprint
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05538 05530, 2024.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-cfficient reinforcement learning. Advances in Neural Information Processing Systems, 34: 16666-16629,,, 2021.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked 11m policies for web actions. In First Conference on Language Modeling, 2024.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975. 2022.
CodeGemma Team. Codegemma: Open code models based on gemma. arXiv preprint
arXiv:2406.11409, 2024a.
The Mistral AI Team. Codestral: Hello, world!, 2024b. URL
codestral/
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pp. 2139-2143 PMLR, 2016.
Guanzhi Wang, Yugi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 2305 2023a.
12

---

Published as a conference paper at ICLR 2025
Ruoyao Wang, Peter Jansen, Mare-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? arXiv e-prints, pp. arXiv- 2203, 2022a.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better Ilm agents. arXiv preprint wrXiv:2402.01038 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:22 12. 10560, 2022b.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, arXiv preprint arXiv:2302.0I 200220002..00... 5600, 2023b.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying Ilm-based software engineering agents. arXiv preprint arXiv:2407. 01400, 2024.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng. Yitao Liu, Toh Jing Hua, Jun- ning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the 2023.
wild. arXiv preprint arXiv:2310.108:
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 2024.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv: 15793, 2024.
Zhao Yang. Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang. Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023.
Shunyu Yao. Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-2005 20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Da Yin. Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. arXiv preprint arXiv:2311.06633 2023.
Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, et al. Swe-bench-java: github issue resolving benchmark for java. arXiv preprint arXiv:2408. 14354, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310. /2823, 2023.
Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309. 11466, 2023.
Jiwen Zhang, Yaqi Yu, Minghui Liao, Wentao Li, Jihao Wu, and Zhongyu Wei. Ui-hawk: Unleash- ing the screen stream understanding for gui agents. arXiv preprint, 2024.
Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang, and Gaoang Wang. Do we really need a complex agent system? distill embodied agent into a single model. arXiv preprint arXiv:2404.04611 2024.
Andy Zhou, Kai Yan, Michal Shlapentokh- Rothmmn,, Haohan Wang, and Yu-Xiong Wang. Lan- guage agent tree search unifies reasoning acting and planning in language models. arXiv preprint 2023a.
13

---

Published as a conference paper at ICLR 2025
Table 6: Top-10 results of SWE-bench from the leaderboard https:
www.sweeench..
com/
All the numbers are fetched on 2024-10-01.
Approach J
site
result
CodeStory Aíde Mixed Models
43.0
Honeycomb
RCtps:/ /oweeconnn...................... 8h7
38.3
AbanteAl MentatBot
ALtps://Grur htt.:://......eetttt...aaaarraa. ott.a///saooaoo/t////a//////aaaa
hetps: /tt.///ttttt.t.iieeeeem
38.0
Gru
35.7
Isoform
/ttp://www...... https://ww...8..oom
35.0
SuperCoder2.0
34,0
MarsCode
Bttp:://ww..arre0 com/
34.0
Lingma
01422
33.0
Factory Code Droid
https:/ This paper
31.3
AutoCodeRover
30.7
MMARN-BY-INTERNE (ours)
34.7
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv: 13854, 2023b.
A BASELINE IMPLEMENTATIONS
We follow the existing frameworks to set up baselines in each benchmark. In SWE-bench (Jimenez al. 2023), we follow the prompt styles of the Agentless (Xia et al., 2024) pipeline first localize suspicious files, then find classes and functions to edit. In WebArena Zhou et al. (023b), we follow the implementation of Step (Sodhi et al. 2024), which concatenates task objectives, action space descriptions, general instructions (e.g., output formats) and webpage observations in the prompt, and ask LMs predict the next action. By default, we the accessibility treef as the observation space. In OSWorld (Xie et al.) 2024) and Spider2-V (Cao et al. (024). we follow the original prompt style designed by the benchmark, which also concatenates task objectives, action space descriptions, general instructions and computer observations in the prompt. By default, we use the accessibility tree as the observation space for OSWorld, and use the set-of-mark for Spider2-X V due to the significant information loss of the accessibility tree in the original benchmark. See an example Table 22 and 23 for more details.
B COMPARISON TO TASK CES--SPECIIIC APPROACHES
In Table we compare LEARN-BY-INTRRACT to top-10 task-specific approaches (with open-sourced code) that may not broadly applied in agent scenarios for SWE-bench (Zan jet al.) (024) and WebArena (Zhou et al.) 2023b). All the information retrieved
on 2024-10-01 from the official leaderboard https:/ http:::://wwwwwwbenccc..c com/ and Lww.11LEEEBBBEBBEEEE
PF7LdyGU1f Sutdyyuuufuufbbb.oo wf2bb
To the best of our knowledge,
are the first to apply our methods in OSWorld (Xie et al. 2024) and Spider2-V (Cao et al. 2024).
C DATASET EXAMPLES
From Table to 21 we provide one example for each dataset with full instructions, interaction history with the environment.
D EXPERIMENTAL SETTINGS
We retrieve documents until the maximum length of LLMs for RAG and set upper bound number of 50 documents, where the retrieved documents remain unchanged throughout agent interaction trajectory because only instructions used the query for retrieval. For Reflexion (Shinn et al.
mozi http:///wweeeeeeeeeec/ccc//ccctt
14

---

Published as a conference paper at ICLR 2025
Table 7: Top-10 results of WehArena from the leaderboard at httpp:://cccs.oon/// spreadsheets/ /d/ 11801 - pF7LdyGU1E ui2b NWNB2O/edit?
gid-Otgid-0 All the numbers are fetched on 2024-10-01.
Approach
site
result
Jace.AI
bELps://www. jace.
57.1
WebPilot
37.2
AWM
35.5
Step
33.5
BrowserGym
23.5
Auto Eval
20.2
Tree Scarch
19.2
AutoWebGLM
http:.//xx.1..77//55440...8
18.2
gpt-4-0613
14.9
spt-4o-2024-05-13
13.1
LEARN-BY-INTERACTY (ours)
This paper
39.2
2024). use the maximum trials 3. In LATS Zhou et al. 2023a),
the number of gener- ated action depth limit 13, value function weight 0.8, following the original setting in paper with WebShop (Yao al.] 2022a). which is also agent task based website. By default, we www.../ouuunnnagtttlllaaeee 1
as the retriever for model-based retrieval con- sidering both size and the performance. We the temperature 0 throughout the experiments ensure better reproductivity of the experiments. During training, we the batch size 128, learning rate 0.00002, warmup ratio 0.03 and maximum length 8192, and tune the model for epochs. All experiments are conducted in H100 machines with 80GB memeory.
E DOCUMENT SOURCES
We use all the non-repeated python files in SWE-bench-Lite (Jimenez et al. 2023) as the document sources. Although may always find abundant documentations and tutorials for each envi- ronment, we believe that documentations in the same domain still have a good coverage frequent operations. For example, one subset of WebArena (Zhou al., 2023b) focuses on the navigation of the shopping website OneStopMarket, we use the Amazon documentation as good replace- ment. Regardless of the shopping websites, the frequent tasks usually include order change, product search, delivery checking, etc. Therefore, we use other documentations in the same domain to sam- ple task instructions when the exact version for the target environment not available. Concretely, we use the following sources for WebArena:
http://ooss.iiaaaccm/eeeuurrr
http://supporttooggl..cm/mmass
The following sources used for OSWorld:
htts://sagporroooog....cccmmmmme
http://wwww.iinnoorgtuuurriii
https:/ccoee www.:eiiiiiissssuiooooooosssssss
The following sources used for Spider2-V:
15

---

Published as a conference paper at ICLR 2025
https:/ddos..ettdbb.omm/
http::/ddocsaasttooomeriii
https://docsaaityyc..comm
http://aariyyee.commuuuurraa
http::/ssuppret...ppahcccoooooo/
http://wwww..cccaassccccooccov.
http://wwww.mmaaaasccccco//////
http://doosssroowlakcccoom//////
F SYNTHESIZED DATA EXAMPLES
30 provide complete example of data synthesis. To begin with, an LLM generates instructions based on standard resources like tutorials, documentations and FAQs: Upload CSV data in Google Drive BigQuery. (See prompt in Table It then attempts solve the task by predicting actions and collecting feedback from environments (interactions). This produces a long trajectory showing how LLMs try to achieve the goal.
From Table 24
However, it is not guaranteed that the trajectory successfully achieves the target. In our example, the selects the table source Google Cloud Storage,
LLM makes a wrong prediction in the action while the correct action should select "Drive" to align with the instruction that reuiqres to upload CSV data Google Drive. This results in wrong actions in the subsequent predictions, and the generated trajectory is not aligned with the initial instruction, which leads to noisy data in this case.
Instead of using the original cntruutiion-raaeectnn pairs for downstream training and in-context learning, we fix the mentioned misalignment by crafting new instructions for each sub-trajectory (backward construction). Concretely, we feed the generated trajectory into LLM prompts, and ask it to summarize the trajectory propose new task based on it. For example, the LLM updates the task objective to "Link CSV file Google Cloud Storage to BigQuery" after observing the trajec- tory, which makes the task instrucion and the trajectory aligned. Additionally, we also generate instructions for each sub-trajectory, which would increase the utility of a generated full trajectory. For instance, based on the sub-trajectory (observation 0, Action observation 1), the LLM gener- ates a new instruction: When is dataset "demo" created? In Table 31 and 32 instructions based on tab-traaecctoies.
we list more generated
G CASE STUDY ON FILTERED EXAMPLES
In Table 49 we demonstrate the representative synthesized examples that fail meet our de- signed criteria. The example in Table 45 is filtered because the trajectory shows detour in ac- complishing the goal, i.e. Action 1-6 are not necessary. The example Table 46 49 is filtered because it goes back and forth in states, i.e. repeat the actions of clicking "My Orders" and click- ing "View Order". We filter these low-quality examples to avoid their negative influences in the downstream applications.
16

---

Published as a conference paper at ICLR 2025
Table 8: SWE-bench example
Instruction
Please address the following Github issue.
### GitHub Problem Description ##### Ireee Value is not hashable. Description
Recently migrated from Django 3.0 to Django 3.1. In my code, add custom data- at- tributes the select widget options. After the upgrade some of those options broke. Error is Type@rrorunhashable type: "ModeeChoiceelerattor Value". Example (this one breaks):
def greate_option(self name, value, label, selected, index, subindex=None, attrs=None): context super(). cttptteeeeeeepeeeeeeeeee option(name,, value, label, selected, index, subindex, attrs) if not value:
return context
if value in self.show
elf.sho__filld: fields: This is dict 1: ['first_namee, "last camss__mmee] context[ attass ][ data-fields' = json.duuppsss coon.duup(seeffshoo. (ithhhwwiildssvalle]]]
However, working with arrays is not an issue:
def create_ preate_option(seff, name, value, label, selected, index, subindex=None, attrs=None): context super().create create saapper)ccrateeoopioonnnaa option(name,, value, label, selected, index, subindex, attrs) not value:
return context
value in allowed_values values: # This is an array [1, 2]
Observation (repository structure)
django/
setup.py init -py main_ py
shortcuts.py
forms/
init_pyy -py
utils.py
fields.py
renderers.py
forms.py
formsets.p py
widgets.py
boundfield.p py
models.py
db/
init -py
transaction.py
ut
models/
signals.py
constants.py
fields/
files.py
-iiit__ppp -py
related.py
17

---

Published as a conference paper at ICLR 2025
Table 9: SWE-bench example cont.
Action (Find related files to edit)
Based on the problem description and the repository structure, here are the most likely files that would need be edited fix the issue:
django/forms/moddll.py django/forms/ields..p
django//orms/widgetss. django/forms/ init
-py ppy
django//tbmoodebsfielld/// ittpa/eeeesiiedddiiii_________ init__yy .py
djangoddbmmdessfilllsssaaaaay
Observation (File skeleton)
### File: django/forms//oodlls..yy
def construct instance(form, instance, fields=None, exclude=None):
def model_toodieieieee _oadel_tt___le to_diiciinsttacee, fields=None, exclude=None):
class ModelChoicelleraaovVllee def iniit_ value, instance):
def str (self):
def ese other):
class ModeeChooiceFicldCChhcceeeol def get_ chofces(selff);
### File: django/formssfillds.sy
class Field:
widget TextInput Default widget to use when rendering this type Field.
hidden hidden_widget = HiddenInput Default widget to when rendering this "hidden"
def init (self, *, required=True, widget=None, label=None, initial=None):
def preparee_alueeeeeeee -ffpeee_ value):
def to pyttonssssss value):
def (sei, value):
def ran_uunadatoorrSSS value):
class ChoiceField(Fied)):
def validate(self, value):
def valid_ valuessee,,, value):
18

---

Published as a conference paper at ICLR 2025
Table 10: SWE-bench example cont.
Action 2 (Find related classes, methods to edit)
Here are the identified locations that need inspection editing to fix the problem:
django/forms/moddls.pyy
class: ModellChoicelleratovaaal
django/forms/models.yy
class: ModelChoiceeield
method: leggggetthhho__e _ce_cccciees
django/formssfeldds.p
class: ChoiceField
method: valid_valus
Observation (Classes and methods content)
### File djange/forrm//oodelspp
class ModeelChoieelleratovaaaa Value:
def init (self, value, instance):
velf.value value self.instance = instance
def str sstr (self: return strlself.valuue
def eq (self, other):
if iiiistancce ModeeChooiellerattrrallay)) other other.value return self.value == other
class Mode<ChoiieFaal(Chhieeeeell
def Leff__et____a choices(self):: return self. iterato((ee))))
### django/forms/fildds.yy class ChoieeFiiel(Fieed)):
def valid value(self, value):
text text_value value str(value) for k, v in self.choices:
isinstance(y, (list, tuple)):
# This is an optgroup, so look inside the group for options for k2, v2 v:
if value = k2 text value = str(k2):
return True
else:
if value = or value == str(k):
return True
return False
19

---

Published as a conference paper at ICLR 2025
Table 11: SWE-bench example cont.
Action (Edit code to address the issue)
Based on the issue description and the provided code segment, it appears that the 'Model- ChoicelreratorValue class is not hashable, which is causing the TypeError when trying use it as a dictionary key. To fix this, we need to make the ModelChoicelleratoraaale class hashable by implementing the hash method. Here's the proposed fix:
### djangofforms/models.yy ### Replace the following code
class boodelChhiiceteeatoovalauu
def init (self, value, instance):
self. value value
self.instance instance
def str (self): return vrlseef.vvluue)
def (seff other):
if isinstance ModeeChoiiellerattrraluuy> other = other. value
return self. value other
### with
class ModellChiicelleratoaaaau
def __iii_niiitttt value, instance):
self.value value
self.instance = instance
def (fssssssssllff return vtalse)f.valuu))
def eq (self, other): isinstance(other, ModeeChoicelterator Value): other other. value
return self. value == other
def hash hashaahhhhahe (self): return mash(self.valuee
Observation (Generated patch)
-iif- addjango/forrmm/mmddllssp
b/djanggo/ommmsmoodees.yy
index 5330801..55b3c8 100644
a/djangooffommsmnddel..yp
+++ bodjango/formm//oodeksp
@ @ -1 +1171,9 @ @ class ModeeCChoieelfeertttVaaeee:
other = other.value
return value other
+ def hash hasl_(self); return hash(self.
class ModelChoicelterator:
def init (self, field):
20

---

Published as a conference paper at ICLR 2025
Table 12: Webarena example
Instruction
Tell me the total cost of my latest cancelled order?
Observation
One Stop Market
Search entire store here..
Deauty Personal
Outdoore
Clething,
Home Kitchen
Office Products
Teals Home improvement
Health Hisumhall
Patto, Garden
Electronics
Phones
Video
Food
One Stop Market
Product Showcases
CocoswHi
Gingerbroad -Energe Energy
Orange
Dairy
Bueberry of
Candy Sprinites) Vellow Gold Red Hoyall Rose long Plowers Decorating Spenates,
$10.00
$19.36
$15.62
$16.07
Add to cart
AdS cart
AdE to Cart
te Cart
Action
click [4918] // click the button 'My Account'
21

---

Published as a conference paper at ICLR 2025
Table 13: Webarena example cont.
Observation
One Stop Markart
Search entire store here...
Beauty Personal Care
Outdoors
Cothing Shees
Home Oftchen
Office Products
Tools Home improvement
Health loussholld
Patto, Lawen Garden
Electronics
Phones Accemoriis
Video
Gourmet Food
My Account
My Account
My Orders
Dowmicedable Products
Account Information
Emima
You subscribed menoletier.
Address Btook
EBt
Account Information
Pastword
Address Book
Manage Addresses
Subscriptiere
Defaui Addeees
shipping Address
Emmna Lopaz
Drima Lopez
San Mateo Califoria, 94310
San Maseo, Califoriis 94010
Compare Products
6585551212
6505591212
Recendly Orderec
Recent Ordors
Vie
Bodrnat Outfits
000000170
5/1003
Erura Lopez
$365.42
Canceled
View Order
Rearder
Utternsosi Vulteria Guckled Taapr Cray Caramic Table Lamp
090000169
Pending
View
NOZE Rustic Cont Rack Well Mounted
000030108
50/23
Ererr Lopez
$2,004.99 99
Pending
View Order
Rearder
Sheff Hooiss. Hamging Greryway Organizer
Cup, Holding
090000187
5023
Enera
$1,006.99
Panding
View Order
Rearder
3/1123
Laguz
Complete
Voew
Action
Stop: The total cost of the latest cancelled order $365.42
22

---

Published as a conference paper at ICLR 2025
Table 14: OSWorld example
Instruction
Could you assist me in adding a new layer and naming it 'Square"? ?
Observation (Interface of the software GIMP)
X
Action
import pyautogui pyautogui. hotkey 'shift', 'ctrl', 'n') // shortcut to initialize a new layer.
Observation (Interface of the software GIMP)
Dhack
3
23

---

Published as a conference paper at ICLR 2025
Table 15: OSWorld example cont.
Action
import pyautogui
pyautogui. tywptlttttewwwwpwpeeeeeepe "Square") change the layer name to Square.
Observation (Interface of the software GIMP)
X
Ereate Newe
Fand
Action 3
import pyautogui
pyautogui.clice click( L096, 892) click OK to save the change.
Observation (Interface of the software GIMP)
x
41-41
24

---

Published as a conference paper at ICLR 2025
Table 16: Spider2-V example, cont.
Instruction
have established connection from Faker to local .csv file. Could you help me change the running schedule? hope it can be replicated at 18:00 pm every day.
Observation (Interface of the software Airbyte)
Aikayna
Connections
Creselon
zarpa 2011
saapl
27A011
Action
import pyautogui Local CSV"
// click the connection row with the name "Sample Data (Faker) +
Observation (Interface of the software Airbyte)
Gita medions
imatledl aoficee Taweeeaaettttt bustt
Sample Data (Faker)
Calu
Sctting
25

---

Published as a conference paper at ICLR 2025
Table 17: Spider2-V example, cont.
Action
import pyautogui pyautogui. pyautogui.click5500 (50033))))) // click the button Replication
Observation (Interface of the software Airbyte)
oedernmscci www.://teweeetteeeeeeooooooooo beeeerohoeeeeoeeeheeeeee
M
Sample Data (Faler)) Local
bactinge
21000 Ohiige Tamoery
Actwars the STVERE yOU 1910
Plane
wwtp://eeoooooooooooooooooo tokkost Aww.0 www.
Action
import pyautogui pyautogui.clice (lock(50004550 // click the panel Configuration
Observation (Interface of the software Airbyte)
M snvurma
Sample Data (Paker)
CSV
Dakup
Jos
Seddee
Sestric
Configanater
Dest conine
Deter
Actuaaa the wo Wort
205
Phene
26

---

Published as a conference paper at ICLR 2025
Table 18: Spider2-V example, cont.
Action
import pyautogui
pyautogui.c wlw.iicclciiiiie (270,430) reveal the dropdown menu of the schedule type
Observation (Interface of the software Airbyte)
M
Sample Data (Faler)) Locat CEV
barting
Denierte Smean Pefte
Deam
durgee sally
Actwate the stre tyaee
Prinay
Action 5
import pyautogui pyautogui.clice click( (200,565) select the schedule type Cron
Observation (Interface of the software Airbyte)
M snvurme
Sample Data (Paker)
Local CSV
Dakup
Jos Birrayy
Scdder
Sestric
Configanater
Eevinabte
vittat achamt
205
27

---

Published as a conference paper at ICLR 2025
Table 19: Spider2-V example, cont.
Action
import pyautogui pyautogui.clie clwck( (020,480) focus on the content of the Cron expression
Observation (Interface of the software Airbyte)
M
Sample Data (Faler)) Locat
barting
Und
Achuaa
Platy yoy
Action 7
import pyautogui
pyautogui.. cnautogui.hotkey(ctttt hotkey "a") // select all content the textbox
Observation (Interface of the software Airbyte)
M snvurme
Sample Data (Paker)
Local CSV
Dakup
Jos Birrayy
Scdder
Sestric
Configanater
Eevinabte
vittat achamt
205
28

---

Published as a conference paper at ICLR 2025
Table 20: Spider2-V example, cont.
Action
import pyautogui pyautogui. pww.ppreessssssse ("delete")) delete the default content
Observation (Interface of the software Airbyte)
oedernmscci www.huoteeerreeeeeeerooooooo
M
Sample Data (Faler)) Locat
barting
Centrrtin
Delest megie)
Action 9
import pyautogui
pyautogui. wr 18 ?") // update the content to align 18:00 pm in the instruction
Observation (Interface of the software Airbyte)
M snvurme
Sample Data (Paker)
Local CSV
Dakup
Jos Birrayy
Scdder
Sestric
Configanater
Eevinabte
vittat achamt
205
29

---

Published as a conference paper at ICLR 2025
Table 21: Spider2-V example, cont.
Action 10
import pyautogui
pyautogui. www.:ccccccccccoeeneee click the button save changes
Observation (Interface the software Airbyte)
dight [
M
Sample Data (Falber) Local
serangg
USC
Cest vetiraai Nawrrace
Acthata
indiaates
30

---

Published as a conference paper at ICLR 2025
Table 22: Observation space of Spider2-V.
Screenshot
kaftwee satp ouilalle doun Pleare dowebbadee iig aprise.
Jobe silemm.t
Sk Wima
Ooendov
Rura
A Highigh
Set-of-mark
Aabl Pars Chrcmianees Bracce
Sep 1741
Inallee end readhidom www.er/onnooootteoeeeooeooo coweiiaaapeeeeeiieeuuuuu..
A
Launchpad Rura
Type snop
INOV
Highte
1D Type
31

---

Published as a conference paper at ICLR 2025
Table 23: Observation space of Spider2-V. The accessibility tree suffers from significant information loss. Compared to the screenshot and set-of-mark shown in Table [2] the presented accessibility tree fails retrieve webpage information, and only shows the details of the desktop icons in the left panel.
208, 13] memu Chromium Web Browser ...
1463, 13] menu System
[35, 65] push-button Chromium Web Browser
[753, 81] label Please download waiting software updates.
[135, 109] label Home
[35, 133] push-button Terminal
[35, push-button Visual Studio Code ...
269 push-button Files
337] push-button Text Editor
[953, 370]
label Updated software is available for this computer. Do you want install it
now?
[35, 405] push-button LibreOffice Cale
463] table-cell Security updates
[1191, 463] table-cell 638.8 MB
335, 473] push-button LibreOffice Writer
9963, 486] table-cell LibreOffice
1191, 486) table-cell 23.4 MB
963, table-cell LibreOffice Calc
(191,509) table-cell 8.7 MB
9923, 524] toggle-button Details of updates s..
1963, 532] table-cell LibreOffice Draw
1191, table-cell 3.0 MB
335, 541] push-button Document Viewer
996,, 555] table-cell LibreOffice Impress
1191, 555) table-cell 1.3 MB
578) table-cell LibreOffice Math
11911 578] table-cell 673 kB
[35, 612] push-button Software Updater
9935, 660] label 1157.8 MB will be downloaded.
335, 680] push-button Trash
(671, 702] push-button Settings...
1054, 702] push-button Cancel
1176, 702] push-button Install Now ...
335, 884] toggle-button Show Applications
32

---

Published as a conference paper at ICLR 2025
Table 24: Example of data synthesis Bigquery
Instruction
Upload CSV data in Google Drive to BigQuery.
Observation 0 (Bigquery Interface)
Adbi Bee
Cet Brauser
Sce I733
cinaa leal grigle can guaryrpneen siociiaaa tne
(nectaiona 2000 on des wranng sctute purt accuan pretimaddceenss Coglle nd usay waunigonth thicee tirtttopuus
DINNSS
ACTiNil
Google Cloud
Hippery siae
Search 10 anse dael, products mome
Q Siarca
Exp lner
ADD
I
Welcome BigQuery Studio,
Siving Mocate
Pypeey exyco
canve
Ai WAROLLL
Reaetty socent
Try Google demo query
U0TA RETUETT
Try Thk vallo tauggh basos dalamaaee www.ereeetttttttteee Eypeeyy
lealory
CEETIN
Action 1
import pyautogui pyautogui.der www.bbbleCCCCc(3333.44)) // double click the dataset demo.
Observation 1 (Bigquery Interface)
Sep
veree to wewglggeggggego le puog ingeet 4564 Mieee betttee en shggain wejedt dtarittsien
Instrall bosts csstaaci des Adtust succuat
bogll out wease mork thrcae any attrysu
EIENIS
ACTIATS
Google Cloud
Weggery Vive
Ancon U5 mirr anoig dini poodurts
Arovk
trattad
Sovrng
Data set Info
un XOHAS
maa
Lodh
Dutiseet replica Info
MONIA
IOAA RETUDAG
Jos history
33

---

Published as a conference paper at ICLR 2025
Table 25: Example of data synthesis Bigquery
Action 2
import pyautogui pyautogui. (ow.dbbbeeCccc(0002221)) / double click the button CREATE TABLE.
Observation 2 (Bigquery Interface)
Dinenluniet
cakle
conall
minac luaiggeeee canp yuartypneee blgq soccccaaaaaaano novss seuttra dliga ungject mraneeeee
Google
Scurce
Destiration
Saray
34%
LS100P9 procis ban
Nalle 30
Schema
Partitoon and cluder settings
Action 3
import pyautogui pyautogui.clice www.ccccccccccce..... 1458,,77) // click the dropdown menu to select table source.
Observation 3 (Bigquery Interface)
Create tuble Biwwayyrrrrry Uigeerpp Gongle cleed mesule chosskm
Bing
Crecte toble
Bbgger
pe camch wwwppporeoooooeeeoooooooo reje 45641 bettte shogaaa my anjed
Google
aerc
Bosught Cheil
Boight gaile
Kaats Me0 strege
Scherma
Partttion and cluster settings
Dee
34

---

Published as a conference paper at ICLR 2025
Table 26: Example of data synthesis Bigquery
Action 4
import pyautogui
wrong prediction: click the table source Google Cloud Storage. // The correct prediction should select Drive align with the instruction. pyautogui.clice 302,331)
Observation 4 (Bigquery Interface)
obile hww.eeyyprceed--eoe.... soogla clead conale shranio
minec au www.lgggggggeooooooo conpl wwwp://aooooooooooooooooooo poaryypncecs wat.eetetttgeeoegoooeeeeee nww.-rrre iww.muuraaaaaaaooo ridigggaae sigarryygee--iiinnoeeee
Google
Source
Seerk
Lelen uoc ofe
Destination
Action 5
import pyautogui pyautogui.clic (etiicc(139,3444 // click BROWSE find file.
Observation 5 (Bigquery Interface)
Create cable -i www.wrgggggg........ wtw.eryy--ggeggggee..... igg.eepprgeee Goggle cload scescle chramam
tottee toble
Google
Exploeer
wenti
Jelac form bubee ww2.2tt
Destination
Schemn
35

---

Published as a conference paper at ICLR 2025
Table 27: Example of data synthesis Bigquery
Action 6
import pyautogui pyautogui.c
// click to find files under directory.
Observation 6 (Bigquery Interface)
cakle
conale chranion
mirac lualighec le canl yuartypneett sachiaa tro seussra www.p//www.....eo.eeeeeeeoooooo
rsturi
Chocss
Google
Source
Sarag
Han
Destination
Data
Action 7
import pyautogui pyautogui.cm pyautogui.doublcCCC CawuullcCicck12203358888 select the CSV file.
Observation 7 (Bigquery Interface)
Create tuble Biw.wryrrrrrrr hggeerp prsjes Gengle cleed mesule chrsskm
Bing
Crecte toble Bbggeen
be camch purrppgeeei 45641 2hsiggaaraaamamm mnjed
Google
Saki ViS
aerdi
select Nore arpptter
Destinution
10
Schema
Fieeme
Dee
36

---

Published as a conference paper at ICLR 2025
Table 28: Example of data synthesis Bigquery
Action 8
import pyautogui
pyautogui.clice www.ccccickkiacc (1000,915) // click to select file.
Observation 8 (Bigquery Interface)
cakle
conale chranio
minac dualggeece le can yoartypneeet bugq war.ecaattttnnno sesttra dlgga dug.pnyyyyyeee sajee--attttteeeeee
Google
Scurce
Adaciie
Saray
Destination
Aap
Schern
Action 9
import pyautogui pyautogui.c www.cccccccceeeccc 1000,668) // click to focus on the text box Table,
Observation 9 (Bigquery Interface)
Create tuble Biwwwry--rryry higpesp Gongle cleed mesil chrssum
Bing
Crecte toble Bbgqeen
pe camch wwwppegggggggroeeeeoooooo 45641 bettte hergammy anjed
Google
aerdi
sona
Destinution
Schema
Dea
winl
37

---

Published as a conference paper at ICLR 2025
Table 29: Example of data synthesis Bigquery
Action 10
import pyautogui
pyautogui. pyautogui.writef write (eeeee""tst"//""" // name the file "test".
Observation 10 (Bigquery Interface)
cakle
conale chranio
cinac be can yuartypneee bigq wac.ecatttttnnno sesttra didilg dug.geyyyyeeee wygageee--tttttrrrereee
Google
Scurce
Adaciie
Saray
Destination
Aap
Data
Scherna
Action 11
import pyautogui pyautogui.clice clw.ccccccccckk200 (425,726) // click to reveal dropdown menu of the table type.
Observation 11 (Bigquery Interface)
Create tuble Bigwwry--rryyy hig.eegeeeeeege Gongle cleed mesule chosskm
Bing
Crecte toble Bbggee
pe camchi puwrrrgregergrrereeoooo 45641 bettte sirgamy anjed
Google
aerdi
sona
Destinution
Dee
Aitrosic:
38

---

Published as a conference paper at ICLR 2025
Table 30: Example of data synthesis Bigquery
Action 12
import pyautogui pyautogui.clics cll.kccccckkkkcc (197,8811)) // select the table source external table.
Observation 12 (Bigquery Interface)
Dinenluniet Iraucer
cakle
conale chranion
cinac dualggeec le canl yuartypreeet bigq sochccaa tovus sesttra dliga waggggpuyyyy wagee--ttnnrnddeee
Google
Scurce
Adaciie
Saray
504
Destination
Aap
Pata
Eeplonath
Action 13
import pyautogui pyautogui.cm pyautogui.click3033.//. cl03((33,,)1)4) click CREATE TABLE.
Observation 13 (Bigquery Interface)
BigOvery- Migouery Google Clesd mesile Chrentes
Bing
ao tnw camch purcylinge Hegye y 4oee piggaam my mejet
Google
Q trantaet qary
BB dera
Jurch suguury
XITAR
Bak 1e13
2014, 17
Datiset replica
Dee
Neatory
39

---

Published as a conference paper at ICLR 2025
Table 31: Instructions generated from trajectory from Table 24 to 30
sub-trajectory
type
instruction
Observation 0
Action 1
New task
When is dataset "demo" created?
Observation 1)
Observation 1
Replicate the following: We are currently at the Google Cloud Console interface, specifically focused on BigQuery project. The browser window displays details of dataset named "demo" within BigQuery project. The interface provides information about dataset, including creation date, last modified time, data location (US), and other properties like default table expiry and rounding mode. On the left side of the sereen, there's s a navigation panel showing the Explorer view with the "demo" dataset selected. top of the screen shows Google Cloud header with project selection and search functionality.
Action 2
Replicate trajectory
Observation 2
The overall layout is characteristic of a cloud-based data management platform, with options to create tables, share data, and manage datasct properties. After taking the action click CREATE TABLE button, we go to the user interface for creating a table. The screen displays form titled "Create table" with various fields and options. The source section allows selecting table create from, while destination section includes fields for project, dataset, and table name. There's also a schema section and partition and cluster settings. The interface is part of the Google Cloud Console, evident from the sidebar on the left showing different Cloud services and project navigation.
Observation 4
Action 5 +
Observation 5
Action 6
Observation 6
New task
Select test.csv in the bucket test-1616 in Google Cloud Storage the table source.
Action 7
Obscrvation 7 +
Action 8
Observation 8
40

---

Published as a conference paper at ICLR 2025
Table 32: Instructions generated from trajectory from Table 24 to 30
sub-trajectory
type
instruction
Observation 8
Replicate the following: We are in the the interface for creating table in Google Cloud's BigQuery service. The page is divided into several sections. At the top, indicates the user creating table from Google Cloud Storage source, with CSV file selected. The destination section shows the project IDD and allows input for dataset
+ Action 9
Observation 9 + Action 10
Replicate trajectory
table name. destination table empty. The table type is set to "Native table". At the bottom, there's 's an option for schema detection, with buttons to create the table cancel the operation. The left side of the screen displays navigation menu for the Google Cloud Console, including options like Explorer and various project-related items. The overall layout suggests this is part a larger cloud data management analysis platform. After we click on the text box Table, we select and focus on the text box. We then type "test" into the box, which gives the table name. Except the textbox we are working the other parts of the webpage has not changed after clicking and typing.
Observation 10
Observation O + Action I
Observation
task
Link CSV file Google Cloud Storage to BigQuery
+ Action 2
Observation 13
Table 33: self-instruct prompts propose instructions based tutorials, documentations and FAQs.
[Documentation!
Based on the tutorial, examplify tasks that users frequently perform. User the following format to output:
41

---

Published as a conference paper at ICLR 2025
Table 34: Prompts summarize (sub-) Stuettrraecctories or propose new tasks based on the (sub- )trajectories.
Prompt
Below a trajectory to complete task.
Observation:
[Observation.)
Action:
|Action;+1] }
Observation:
OOservvatoon++11
Action: (Action;+2)
Action:
( }
Observation:
[Observation;]
Please write reasonable task instruction that is completed by the trajectory. Wrap the instruction with
Prompt
Below is trajectory to complete task.
Observation:
[Observation:]
Action:
Observation:
[Observation;4]]
Action:
(Action;+2]
Action:
(Action,-1]
Observation:
[Observation;)
Please summarize the trajectory about each observation and changes after each action. Wrap the summarization with
42

---

Published as a conference paper at ICLR 2025
Table 35: L.M prompts to filter low-quality data
Task instruction:
[instruction]
Below is the trajectory complete the task.
Observation:
[Oseervation:]
Action:
[Action;+1] }
Observation:
[Observation:+]]
Action:
[Action;+2]
Action:
(Actiony-1] }
Observation:
[Observation:]
Here are the criteria to indicate a good pair of the instruction and the trajectory: The instruction and the trajectory are aligned, which means the trajectory successfully accomplishes the goal in the instruction.
The trajectory is coherent, indicating that each action is logical based on its previous observation and the actions do not contradict with each other based on the task instruction.
The trajectory natural, meaning that the trajectory closely mimics real-world interactions and human user would possibly perform it when engaging in the environment.
The trajectory is reasonable, indicating that the trajectory finishes the task instruction using reasonable solution, e.g., using problem, not going back and forth states, etc.
complicate method, not over-simply the
Please answer yes if the task instruction and the trajectory satisfies all the criteria, otherwise, answer with no.
Table 36: Model inference prompts without external knowledge
SYSTEM MESSAGE:
(system message)
OBJECTIVE:
[task instruction]
INTERACTION HISTORY:
(interaction history)
OBSERVATIONS:
(observations)
Your REASONING and ACTION in the format:
REASON:
Your reason to choose specific action.
ACTION:
Your action
43

---

Published as a conference paper at ICLR 2025
Table 37: Model inference prompts with external knowledge
SYSTEM MESSAGE:
[system message)
ADDITIONAL INFORMATION FOR REFERENCE:
(external knowledge)
OBJECTIVE:
[task instruction)
INTERACTION HISTORY:
[interaction history)
OBSERVATIONS:
(observations)
Your REASONING and ACTION in the format:
REASON:
Your reason to choose specific action.
ACTION:
Your action
Table 38: Expected model outputs
REASON:
ACTION:
Table 39: Model prompts write query for retrieval
SYSTEM MESSAGE:
(system message)
Here is the final goal we want to achieve:
[task instruction]
To achieve the goal, we have done the following:
(interaction history] Now, we have observed: [observations)
To better finish the task, write query to ask for useful information, e.g. what kind of exam- ples or interaction history will be helpful to predict the next action.
44

---

Published as a conference paper at ICLR 2025
Table 40: OSWorld example (filtered)
Instruction
Sun numbers in the first column.
Observation (Interface of the software LibreOffice Calc)
Libroffincaac
File bdit
View
Insert
Pirmua
style
sheet
Data
Trals
Window
Help
Silve
11:t
Libreofftce evenbetter
Gelimvoked
Your donations sugpert divid conn MARTIT
Donate
12
Action
import pyautogui
pyautogui.clice (43,120 // click Tools.
Observation
File
Adit
Vitw
Insert
Foomat
styles
Sheet
Data
Toag
window
Help
spto
Sihve
119t
I
Optim..
livefftice evenbetter!
Autorqpu
Your donation support our wpr devide conmmniitty
Retect
teagofup
Donate
kadact
Ferma
Devlogment
Exterice Manage...
OHIAREE
Custoribe..
Optirn..
A1-F112
0
45

---

Published as a conference paper at ICLR 2025
Table 41: OSWorld example (filtered) cont.
Action 2
import pyautogui
pyautogui.coo (taptogguiicicc((5435535 (5335c5)))))) // click Solver.
Observation 2
Activites
Litreoffic caic
Sep 18:01
Student_Lovel Uill, Pienkubb Lireoiftee Cale
Insert
Styles
Sheet
Window
Solvw
Solver
Help make L lireiffice ewen better
Target cell
Get retioe
rytimiizereslttt
Moiium
Velueal
changing.st
Liilting
Cell referance
Operator
Vilue
16
Help
RosetA
optioss..
cosse
Solve
Bhet
Engio (36)
Action
import pyautogui (772,892)) // click Close.
Observation
Actiitton
Level
A
E10
Your doeation worldwide commnniity
Denate
17
46

---

Published as a conference paper at ICLR 2025
Table 42: OSWorld example (filtered) cont.
Action 4
import pyautogui pyautogui. (tt.geuiiicccc(0212226...) // click Data.
Observation
Studest Lww. Blaskak Librorttoc Cale
Vitw
Insert
Format
Styles
Sheet
Liga
Window Help
Sert.
PRL
Accendd
11
Sent Descrrding
18
E10
Sange-
Gatimoheo
donations support war divide
selcct lange
Refreskitarge
Dorate
Povtt Tible
Farm
Strear
XMI Scarce
Hullipe
Sealt
Action
import pyautogui pyautogui. (ooiggiii.mmooo(((555555)) // move to Calculate.
Observation
Student Lavel Slank siac Ulbroofti Cate
Edit View
Insert
Parmet
Styles
sheet
Tool:
Window Help
Sort.
Sort Acceninng
pt
Range.
donations support our canmeue
Range
Konge
Donate
Poot Teble
Caksinge
AdtoCalaaatt
Strema.
2011 Sourts..
Haltple
Statt
47

---

Published as a conference paper at ICLR 2025
Table 43: OSWorld example (filtered) cont.
Action
import pyautogui
caautoguicclick4488..... // click Sheet.
Observation
Stedent Level Blaskatee Liireoffie cale
Vitw
Insert
Style
Sigatt
Window
RRL
Iovert Dows
Istart columes
E10
Break
hlvoiffice
Geliovoked
donation support our
Page Brusk
Donata
Insert Sheet.. Ineart Sheal anded..
Ievert Sheet from Pil..
Eiternal Data... DeleteSheet.
f.
Oytle Saference Types
Nemed Rangoo and Exprcssions
Call Comments
Sireet
Novigata
Sheet Tab Color..
sheet swets..
Sheets
Eegits 1M1
Emage:
Action
import pyautogui pyautogui.clin click( 152,591) // click cell A4.
Observation 7
File
Edit
Viewi
Insert
Foommt
Stylec
Sheet
Data
Tbolk
Window
Help
Add
10pt
Your conatios support spr divide munitA
Donace
1
48

---

Published as a conference paper at ICLR 2025
Table 44: OSWorld example (filtered) cont.
Action
import pyautogui
pyautogui.clck((400.0 (180,(3)3))))3 // click the formula box.
Observation
Stadent, PRi Blank alse Lirreottie cal:
Vitwe
Insert
Format
Styles
Sheet
Tools
Window
Help
10
A4
Eeveiffice
Gelioviked
donatoon support our muntutt
Donate
f.
Meaes
Action
import pyautogui
pyautogui. write(" =SUM(AL:33))) // type the formula.
Observation
Stadent Jevel
Blank alse Libreomo Calc
Vitwa
Insert
Format
Stylec
Staee
Tools
Window
Help
RRL
SUM
Inpultie
Getioveked
donation support
Donate
fo
49

---

Published as a conference paper at ICLR 2025
Table 45: OSWorld example (filtered) cont.
Action 10
import pyautogui pyautogui, pre.ss"eeteerrre
Observation 10
Stadent Level Blank als Lireofffor Cal:
Viewa
Insert
Format
Styles
Sheet
Window
Help
FEL
10pt
A6
Livecifice
Getinvoked
donation support our munitt
Donata
f.
1
Meat
50

---

Published as a conference paper at ICLR 2025
Table 46: WebArena example (filtered)
Instruction
What are items ordered in the latest cancelled order?
Observation 0
MNAAOAE NARAA Spoidl Ietuans Eiresape
menccy Perserroes Spn'ts
One Stop Market
ineeed oegaresst scuaeioo
cimuodd uringe
muciiig Dalik Proucton
BAAM
di $15.00
Action
click [4918] // click the button "My Account" Observation
One Murket
Patis LaViE sarden
Snioky SoLithet
My Account
by
ASSIEE BOCK
170
Secerby
CHACOR
NOUS C1205
OVter 7951
1500
Sold soeeers 2eel with Licnns
Long
02000145
51

---

Published as a conference paper at ICLR 2025
Table 47: WebArena example (filtered) cont.
Action
click [4922] // click the button "My Orders".
Observation 2
Lppec
One Stopp Market
Beakins Recertedd
Fark Lewe's Sarder
My Orders
20103
1224
Lnty
Action 3
click [6357] // click the button "View Order". Observation 3
Staedty
Order # 000000170
Under may
Nourted sook
$40.0 SWess $40.00
Secendy Orbind
Oopueeet cagye Cop solling
2P 200g serdingiiing 10:0
wnk Tull Lang
52

---

Published as a conference paper at ICLR 2025
Table 48: WebArena example (filtered) cont.
Action
click [4922] // click the button "My Orders".
Observation 4
One Stopp Market
Beakins Recertedd
Fark Lewe's Sarder
My Orders
20103
1224
Lnty
Action 5
click [6357] // click the button "View Order". Observation 5
Staedty
Order # 000000170
Under may
Nourted sooi
$40.0 SWess $40.0
Secendy Orbind
Oopueeet cagye Cop solling
2P 200g serding 10:0
wnk Tull Lang
53

---

Published as a conference paper at ICLR 2025
Table 49: WebArena example (filtered) cont.
Action
click [4922] // click the button "My Orders".
Observation
One Stopp Market
Beakins Recertedd Fark Lewe's Sarder
My Orders
20103
1224
Action 7
click [6357] // click the button "View Order". Observation 7
Staedty
Order # 000000170
Under may
Nourted sook
$40.0 SWess $40.0
Secendy Orbind
Oopueeet cayye Cop sorling
2P 200g serding 10:0
wnk Tull Lang
54


Reviews:
[{"review_id": "uwixXSX81B", "reviewer": "Reviewer_Hzkc", "summary": "This paper aims to address the critical problem of data collection for training language agents. Annotating trajectory-level data in various environments can be quite expensive. To deal with this, this paper instead proposes a data synthesis pipeline that leverages the documentation available on the internet to generate high quality task instructions, and use the LLM to compose the corresponding trajectory for each instruction. Specifically, the error rate of directly generating the trajectory using LLM can be quite high. As a result, this paper proposes a novel scheme called backward construction to summarize the trajectory and refine the original instruction to make it align better with the generated trajectory. In addition, they also use LLMs to filter out low-quality data points. After obtaining the synthetic data, they use them for both fine-tuning and ICL in multiple different domains, including code agent, OS agent, and web agent. Experimental results show the effectiveness of their synthetic data.", "questions": "Generalizability of this method, for example, for the web domain, it's kinda cheating to use these sources of documentation? WebArena is essentially built on open-source alternatives of these sources. It might be interesting to explore removing the reliance on documentation, as it may not be strictly necessary; maybe you can just ask the LLM to propose task instructions based on the given environment?\n\nIn your dataset, is it possible for one data sample to be a sub trajectory of another sample?\n\nOn WebArena, why do you choose Step as your baseline method rather than more direct baseline used in the original paper of WebArena?\n\nTypos:\nline 44: desktop computing etc. -> desktop computing, etc.\nAlg 1: initilize interaction trajectory -> initialize interaction trajectory\nAlg 2: Langueg Model -> Language Model", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem addressed in this paper is highly significant and of great interest to the language agent community. Due to the scarcity of process/trajectory data available online, the community is eager to find efficient and scalable methods to obtain more trajectory-level data for training. The dataset collected in this paper might be a valuable resource to the community.\n2. This paper covers multiple domains and demonstrates promising results on all of them, which shows the effectiveness of the data collection process.\n3. This paper conducts comprehensive analyses, including scaling law of the training data and the effect of trajectory length.", "weaknesses": "1. A key issue with the data synthesis pipeline proposed in this paper is its generalizability. Specifically, the pipeline relies on a set of source documentation to generate meaningful task instructions, serving as the starting point of the entire process. However, the assumption that in-domain documentation will always be available may not hold in all cases.\n2. Related to the first point, the reliance on in-domain documentation might also set a ceiling for the size of the dataset. Also, the scaling law in this paper (i.e., Fig 3) suggests that achieving continuous gains becomes challenging once around 100k data samples have been collected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RjCWGFX6pd", "reviewer": "Reviewer_6Xyn", "summary": "This paper proposes learn-by-interact, which generates task-specific exemplars which are curated using backward construction which annotates the trajectory with an aligned objective instruction, and filtering using a committee of LLMs.\nThe results are evaluated on a wide array of benchmarks, SWE-bench, WebArena, OSWorld and Spider2-V, showing the effectiveness over strong baselines.", "questions": "- Can you show examples of what trajectories get filtered out?\n- Was LATS implemented by the authors for the benchmarks tested? As far as I'm aware the original LATS didn't evaluate on the benchmarks tested", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed approach for generating exemplars for ICL is novel and effective across several agentic scenarios.\n- The paper is well written and easy to follow\n- The experiments and ablations are very thorough, and validate the components of the proposed method well", "weaknesses": "- The discussion/details on filitering of synthesized trajectories could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LsZCfXhpg", "reviewer": "Reviewer_pZmd", "summary": "This paper presents a novel data synthesis framework to enhance agent performance. Contrary to the conventional forward data construction, the proposed backward construction generates instructions from interaction trajectories with the environment. The synthesized data can be used for training the generator or in-context learning. Experiments across four environments demonstrate the potential of this method.", "questions": "- As mentioned above, the proposed backward construction may have quadratic complexity. I note the relevant discussion in Figure 2, but it is unclear whether this figure applies to inference only or the entire training-inference pipeline.\n\n- On page 3, Algorithm 1 appears to lack a definition of the L() function presented in line 11. Does this function rely on the same LLM backbone as the other function, LLM(), mentioned above?\n\n- On page 5, Table 1, the drop rate is relatively high for OSWorld and Spider2-V, particularly for the latter, where fewer than 20% of synthesized samples are retained. This appears inefficient. Could the authors provide more discussion on this matter?\n\n- Following Question 3, could the authors assess the potential impact of this filtering rate on final performance? For example, if a less strict filtering rule is applied, retaining more samples, how would this affect overall performance?\n\n- In Algorithm 2 on page 4, what is the difference between the append() operation (line 10) and the += operator (line 19)?\n\n- (Minor) It seems that all evaluations lack the percentage unit (%) for accuracy.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In contrast to conventional data synthesis approaches, the proposed backward construction leverages unsuccessful trajectories, thereby improving data collection efficiency. This idea bears a high-level resemblance to the renowned reinforcement learning algorithm Hindsight Experience Replay, which is elegant and proven effective. The paper also provides comprehensive experiments covering performance, efficiency, and scalability.", "weaknesses": "However, as shown in Algorithm 1 (lines 16-21), the proposed backward construction has quadratic complexity concerning trajectory length, $O(\\text{len}(T)^2)$. This raises concerns regarding data collection efficiency and potentially higher computational costs than conventional forward construction. I am open to raising my score if the authors address the following concerns listed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmYvd6i4KK", "reviewer": "Reviewer_kEqP", "summary": "This paper proposes a data synthesis method named “LEARN-BY-INTERACT” that uses LLM to generate instructions and trajectories based on given environmental documents, and the synthesized data can be used in ICL and training scenarios to improve the performance of agents. Experiments conducted on 4 agent datasets validate the effectiveness of LEARN-BY-INTERACT.", "questions": "Refer to the concerns in “Weaknesses”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To the best of my knowledge, the backward construction mechanism is novel.\n2. The paper is well written.", "weaknesses": "1. Although the authors claim that the proposed  LEARN-BY-INTERACT can adapt LLM agents to any given environments without human annotations in both abstract and conclusion sections, but I think its application may not be very wide, it needs the environment to have corresponding documentation, and the LLM used to synthesize the data should be familiar with the environment, otherwise it is difficult for the LLM to synthesize valid instruction and trajectory. More discussion about the limitations of the methodology would make this paper better.\n\n2. There are many works that focus on using more powerful LLMs to synthesize data to improve agent performance, such as AgentGen [1] and AgentTuning [2], but this paper does not discuss or compare them.\n\n3. This method requires a lot of LLM calls to generate and filter the data, especially the backward construction phase, which seems costly. \n\n[1] AGENTGEN: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation\n[2] Agenttuning: Enabling generalized agent abilities for llms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)