WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
Authors: Zehan Qi
Xiao Liu
Iat Long Iong
Hanyu Lai
Xueqiao Sun
Jiadai Sun
Xinyue Yang
Yu Yang
Shuntian Yao
Wei Xu
Jie Tang
Yuxiao Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Web Agent
LLM Agent
Curriculum RL Learning
Online Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. 
However, existing LLM web agents face significant limitations: high-performing agents rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. 
This paper introduces WebRL, a novel self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. 
Our approach addresses key challenges in this domain, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. 
WebRL incorporates a self-evolving curriculum that generates new tasks from unsuccessful attempts, a robust outcome-supervised reward model (ORM), and adaptive reinforcement learning strategies to ensure consistent improvement. 
We apply WebRL to transform Llama-3.1 models into proficient web agents, achieving remarkable results on the WebArena-Lite benchmark. 
Our Llama-3.1-8B agent improves from an initial 4.8\% success rate to 42.4\%, while the Llama-3.1-70B agent achieves a 47.3\% success rate across five diverse websites. 
These results surpass the performance of GPT-4-Turbo (17.6\%) by over 160\% relatively and significantly outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2\%). 
Our findings demonstrate WebRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems.

Published as a conference paper at ICLR 2025
WEBRL: TRAINING LLM WEB AGENTS VIA SELF- EVOLVING ONLINE CURRICULUM REINFORCEMENT LEARNING
Zehan Qi Xiao Liu Iat Long long', Hanyu Lai', Xueqiao Sun', Jiadai Sun Xinyue Yang?, Yu Yang?, Shuntian Yao?, Wei x Jie Tang', Yuxiao Dong'
Tsinghua University
Zhipu AI
Shanghai Qi Zhi Institute
ABSTRACT
Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents face significant limitations: high-performing agents rely on expensive proprictary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WEBRL, a novel self-evolving online curriculum reinforce- ment learning framework designed train high-performance web agents using open LLMs. Our approach addresses key challenges in this domain, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. WEBRL incorporates a self-evolving curriculum that gener- reward
ates new tasks from unsuccessful attempts, a robust outcome-supervise model (ORM), and adaptive reinforcement learning strategies to ensure consis- tent improvement. We apply WEBRL to transform Llama-3.1 models into profi- cient web agents, achieving remarkable results on the WebArena- benchmark. Our Llama-3. 1-BBB agent improves from an initial 4.8% success rate to 42.4%, while the Llama-3.1-70B agent achieves a 47.3% success rate across five diverse websites. These results surpass the performance of GPT-4-Turbe (17.6%) by over 160% relatively and significantly outperform previous state-of-the-as web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demon- strate WFBRL's effectiveness in bridging the gap between open and proprietary LLM-- web agents, paving the way for more accessible and powerful au- tonomous web interaction systems. Code, model, and data will be available at https:// com SOO///HHUD//WbRI
INTRODUCTION
Large language models (LLMs) have exhibited not only superior comprehension of human language, commonsense reasoning, and knowledge acquisition, but also significant potential in complex plan- ning and logical reasoning. indicating their promising trajectory towards serving autonomous LLM agents (Liu et al. 2024b). A diverse array of applications for LLM agents has proliferated, en- compassing domains such code generation (Jimenez al.] 2024). database manipulation (Zhoul ct al. 2023: Gu et al.] 2024), and graphical user interface (GUI) interaction (Yang et al. 2023 Rawles et al. 2024.j Xie al. 2024). Among these, web agents powered by LLMs (Deng al. 2024 ZZeng et al.) 2024 Lai et al. 2024 Pan et al.] (024) have gamered particular attention due to their extensive application prospects and unique potential for fostering authentic autonomous intelligence within the digital ecosystem.
Notwithstanding these advancements, existing LLM web agents, regardless of their performance metrics or architectural paradigms, remain under-developed. High-performing LLM web agents predominantly rely on meticulously crafted prompts in conjunction with proprietary LLM APIs (e.g., OpenAl GPT-4) for web page comprehension and manipulation, which is both expensive and
"Equal contribution. Emails: ezh23@maii ils. www.sss.sssiee tsinghua. edu. ea. shawliu98gmail Work done when 7Q interned at Zhipu AI. Corresponding author: YD.

---

Published as a conference paper at ICLR 2025
WebRL Phase T
WebRI.
40
Interact
Rollout Agent Trajectories
Self- Evolving
Soccess Rate (2S)
Task Distance from CMU PIT car.
35
Clik [S)
Scroll Down
Back
Tasks
DigiRL
Web
20
0_a o Actor
M
Task Add red jacket to my listt.
Critic
Policy Update KL
ANR
Click 3)
Type [23) red jacket)
Click [14]
Filtered BC
SFT
Actor Confidence Filtered
Experience Replay Buffer
Experience Replay
Number
Figure 1: Overview of WEBRL. WEBRL is a self-evolving online curriculum reinforcement learn- ing framework for LLM-based web agents, yielding consistent continual improvements throughout the iterative self-evolution.
iineiintessive Conversely, open-source LLMs exhibit notable deficiencies their capability to function as proficient web agents, primarily due to the scarcity of decision-center data in both pre-
training and post-training periods. Despite recent endeavors (Lai et al. 2024 Pan et al., 2024) to train web agents on open LLMs via imitation learning, these approaches insufficiently leverage the inherently online nature web interactions and fail to yield consistent, continual improvements.
Challenges. In this work, we propose to train high-performance web agents based open LLMs within online environments, specifically utilizing WebArena (Zhou et al. (024a). Our investiga- tion has identified several critical challenges inherent to this task: 1) Insufficiency of training tasks: In contrast to offline datasets (Deng et al., 2024, Rawles et al. 2024) that facilitate agent training human aanunnatd oracle trajectories, online benchmarks such as WebArena (Zhou et al. 2024a) typically provide only a limited test set for evaluation purposes. This dearth of prede- fined training tasks significantly impedes the effective training of agents within these environments. 2) Sparsity and cost of feedback signals: The assessment of success for arbitrary web browsing tasks difficult in the absence of task-specific evaluation functions. Moreover, unlike tasks in certain GUI datasets (e.g., AITW (Rawles et al.) (2024) and WebShop (aao et al.) 2022)). those in WebArena are typically of long horizons, with oracle solutions averaging 10 steps. This characteristic introduces substantial sparsity in the available signals during online exploration. 3) Policy distribution drift in online learning: The absence of predefined training set necessitates online exploration, inevitably leading to distribution drift in the agent's policy. This phenomenon is likely to induce catastrophic forgetting and performance degradation over time.
and evaluation
The WEBRL framework. In response to these challenges, we introduce WEBRL, a self-evolving online curriculum reinforcement learning framework designed for training LLM web agents. To the best of our knowledge, this represents the first systematic framework enabling effective reinforcement learning for LLM web agents from initialization in online web environments. Through the application of WEBRL, have successfully transformed Llama-3.1-88 model into a proficient LLLM web agent, elevating its success rate (SR) on WebArena-LLite (Zhou et al., 2202aa] Liu et al. 2024c from an initial 4.8% to 42.4% across a diverse of five websites. Furthermore, when applied to Llama-3.1-70B we achieve a remarkable 47.3% SR, surpassing the performance of the most advanced proprietary LLM API (GPT-4-Turbo, 17.6% SR) by over relatively 160% and significantly outperforming previous state-of-the-ar web agents trained on open-source LLMs (Auto (AutoWebGLM (Lai et al. 2024). 18.2% SR).
Contribution:
The substantial performance gains from WEBRL can be attributed to several key architectural de- signs. To address the scareity of web agent training tasks, we have devised a self-evolving online curriculum that harnesses the trial-and-error process inherent in exploration. This curriculum is un- derpinned by a robust outcome-supervised reward model (ORM) that we have newly developed. In each training phase, novel tasks are autonomously generated from unsuccessful attempts in the pre- ceding phase, facilitating a progressive learning trajectory. To mitigate the policy distribution shift induced by curriculum-base based reinforcement learning, we incorporate KL-divergence term between the reference and actor policies into our learning algorithm, thereby constraining policy updates and promoting stability. We implement experience replay buffer augmented with novel actor
2

---

Published as a conference paper at ICLR 2025
fidence filtering strategy to ensure the fidelity of replayed experiences and prevent over- over-fitting to previously acquired knowledge.
In summary, work makes the following significant contributions to the field:
We introduce WEBRL, a novel self-evolving online curriculum RL framework for training LLM- based web agents. For the first time, it implements the infrastructure for RL in the WebArena environment, together with a strong ORM, to drive open LLMs to become capable web agents.
WEBRL advances the RL for LLM agent training by addressing key challenges including the scarcity of training tasks, sparsity of feedback signals, and distribution drift in online learning. The self-evolving curriculum and adaptive learning strategies allow the consistent continual im- provement of LLM web agents during iteration.
We demonstrate WEBRL'S substantial performance improvements over existing methodologies such AWR and DigiRI., achieving -fet----r-- results on the WehArena-L.ne benchmark. It surpasses the best proprietary LLM API and previously trained web agent on open LLMs by over 160% relatively. WEBRL has been adopted in AUTOGLM (Liu al. 2024a), a series of phone use and browser use foundation GUI agents.
2 WEBRL: SELF-EVOLVING ONLINE CURRICULUM REINFORCEMENT LEARNING FOR LLM AGENTS ON WEB
We present self-evolving online curriculum learning framework designed for training web agents, targeting the WebArena (Zhou et al. 2024a) environment. In this system, the agent continuously interacts with its environment collect real-time trajectory data. We train an outcome-supervisel reward model (ORM) to assess task success and use a KL-constrained policy update algorithm to prevent severe policy distribution drift during curriculum learning. A replay buffer is employed to retain knowledge and mitigate catastrophic forgetting. Furthermore, self- self-evolving strategy is inte- grated dynamically generate tasks that align with the agent's 's evolving capabilities. This approach enables the agent to improve incrementally, progressively handling more complex tasks. The overall training process can be found Algorithm T
Problem Formulation. We model the process of completing the web task as a finite-horizon Markov Decision Process (MDP), denoted by (S, A, R. T). Given a user instruction I, the agent is required to complete the corresponding task. The state is defined as the HTML content of the current web page along with the history of previous actions. The agent receives a reward of 1 upon successful task completion, and 0 otherwise. In the finite-horizon setting, the trajectory ends either when the task is accomplished or when the maximum number of interactions T is ex- ceeded. To explain represents the distribution over actions given the state 8 and the instruction T. The value function
method clearly, introduce the following notation. The policy T(||,,, 7)
V(sh, I) = F.z EE r(st, at, r) represents the expected cumulative reward from the state 8a under policy æ. The action-value function ((2,, 0t, I) is the expected cumulative reward for taking action on state st and following policy π thereafter: Q(st, ar I) = r(st, az) + V( I).
ORM Training. In the curriculum learning process, we need determine whether the correspond- ing instruction is completed based on the trajectory generated by the agent. Due to the lack of feedback from the environment, we train LLM as the oupeoie- reward model MORM to achieve this task success evaluation. MORM is utilized to assess whether the agent's rollout tra- jectory accomplishes a given task, providing a binary reward signal (0 for failure and 1 for success).
Similar to the approach in (Zhagg et al. 2024c), we configure MORM to output "YES" or "NO" to indicate whether trajectory successfully completes task, leveraging the learned knowledge from the language head of MORM- Given the limited context window of LLMSs and the typically long length of HTML documents, we adopt a strategy akin to (Pan et al. 2024), keeping the HTML of only the final state to the input. In addition, the historical actions of agents, which provide informa- tion about previous steps are also included. Thus, the input to the model consists of: the instruction I, historical actions, and HTML of the final state. We wrap these components into the prompt asking the model to determine whether the trajectory successfully completes the task described by instruc- tion T. To obtain the outcome, compare the probabilities of generating "YES" and "NO" from
3

---

Published as a conference paper at ICLR 2025
MORM- If the probability of generating "YES" is higher than that of generating "NO", the task is considered successful, and the reward is set to 1. Otherwise, the reward is set to 0.
2.1 SELF-EVOLVING NEW INSTRUCTION FOR CURRICULUM LEARNING
A typical challenge in training LLM web agents within WebArena is the scarcity of training tasks, resonating with the situation of developing real-world web agents. Although the recent work (Liu et al., 2024c) has curated a trajectory fine-tuning set for WebArena, only contains around 1K in- structions with oracle trajectories, far from enough for training strong LLM web agents. To address this limitation and drive continuous improvement, we employ a self-evolving curriculum learning strategy. This method generates new training instructions at each phase. As the phase progresses, the generated instructions become increasingly complex, allowing the agent's 's capabilities to improve gradually. We implement two-step process of generation and filtering, incrementally more challenging, while still being suitable for the agent's s current capability. During the generation step, we use the in-breadth evolving approach (Xu al. (023) to create new instruc- tions. We select instructions the model failed to complete in previous interaction phases as seeds for generating new instructions. Detailed prompts are provided in the Appendix [$ D To ensure that the generated instructions are both feasible in the target environment and aligned with the desired diffi- culty level, we first filter them using the trained critic. Specifically, we use the critic to evaluate each new instruction by considering its initial state. We select instructions with critic scores between 0.05 and 0.75, ensuring that only tasks meeting our difficulty criteria are retained. We manually review generated tasks and identify tasks that cannot be completed in WebArena. Based on these findings, we develop a prompt (Figure 20) and use GPT-4o to exclude infeasible tasks in WebArena automatically. The resulting set of instructions is used for interaction and training in this phase.
produce tasks that are
2.2 REINFORCEMENT LEARNING FOR LLMS IN ONLINE WEB ENVIRONMENTS
In each phase of curriculum learning, the model progressively encounters and learns a new set of tasks. Considering this setting. a major challenge here is avoid excessive policy distribution drift during each learning phase, which could lead to the catastrophic forgetting of previously acquired knowledge. Traditional approaches typically mitigate the issue by mixing data from different phases. However, in web agent tasks, intermediate steps do not receive direct process rewards, with only weak signals from the outcome of the final state. Consequently, even if an intermediate step is executed correctly, an error in later steps can easily lead to the final failure, resulting in misjudgment of the intermediate step and making it difficult to be reused. As a result, in this work, we primarily seek algorithmic improvements to address policy distribution drift more directly.
A potential solution comes from ideas in reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022), where the Kullback-Leibler (KL) divergence between two policies is constrained to mitigate policy distribution drift. By adapting this to our curriculum learning setup, aim to ensure that the policy in the current phase does not deviate
much from the policy in the previous phase, while still optimizing performance on new tasks. Let the policy from the previous phase be denoted as Tref and the current policy being optimized ma. The instruction distribution for the current phase is represented as p(T). The objective for optimizing Ty in the current phase can then be written as follows:
max (a 77
Σ (r(sz, at, 1) + B log mer |sc, 1)) + (7a()))
(I)
where 3 is a coefficient controlling the strength of the KI divergence constraint and H(mp) repre- sents the entropy of the current policy.
Following the work of (Rafailov et al. (2024)), the objective of eq. Caan be interpreted as a maximum entropy reinforcement learning problem. The optimal policy n² Tor this problem can be expressed as:
n" (arls,, I) Q (v (s..))))
where V* I) is the optimal value function, representing the expected cumulative reward under the optimal policy n². Q*( (st, 0t, T) is the optimal action-value function. The relationship between
4

---

Published as a conference paper at ICLR 2025
Q" and V is given by:
(5(s at, 1) 3 log Ter((a V (5t+1;;)); if 8:4-1 is not terminal ((86, 02; 1) + 33 log F 81, , I);
(st az, I)
(3)
8841 is terminal
Based on eq. and eq. we can derive:
B log " (1o18t,
r(st, ar 1,,1) (*(+++1, V (* (s (1,
Here, A" ag I) indicates the advantage of taking action a₂ in state s; compared the average reward expected in that state. Based on the condition, we can formulate the loss function of policy Tg as:
((TT) = Ey, ( To Ta(als, 1) A
(5)
Trer(a|s,1)
where v(s) represents the distribution of experience used for training. Note that our algorithm operates the off-policy manner. comprehensive derivation, thorough analysis, and detailed comparison with other RI. algorithms can be found in Appendix
What does the update do? To gain mechanistic understanding of the loss function, we analyze the gradient of the loss function, [(Te). The gradient with respect to the parameters 0 can be expressed as:
VoL(TT)) 2,3 as) log (alss T) Talr
Ve (og
(6)
update dircction
KL divergence constraint
sft loss
The gradient demonstrate the following attributions:
When the advantage (s, > O, action a is valuable, so its probability should increase. If To is lower than Tet. this increase will be amplified, especially as the gap between them grows. If To is already higher than Ner. the increase will be moderated to avoid excessive deviation.
When (s, a, T) O, the action is suboptimal, so its probability should decrease. my is lower than Tel. the KL. divergence constraint will limit how much it can be reduced to avoid large divergence. Tg higher than Tef. larger decrease will allowed.
The parameter 3 controls the strength of the KI. divergence constraint. Adjusting 3 can help fine-tune this constraint. For instance, increasing 3 can prevent unnecessary boosts in action probabilities when ref already assigns high probability to action.
Training a Reliable Advantage Estimator. A reliable advantage estimator is essential for effec- tive policy updates. We train value network V(s,, 1) and use Generalized Advantage Estimation (GAE) (Schulman et al. 2015) to compute the advantage. our setting, we only receive a binary reward (0 or 1) at the final step, with no intermediate rewards (i.e., intermediate rewards ef- fectively zero). Following recent approaches (Farebrother et al., (2024). we train the value network using a cross-entropy objective. The loss function for the value network V is defined as:
L(V) EE [(STT, ar, T) log V a, T) (1 r(ST, or, I)) log(1 - V(s, a, n)]
In line with (Bai et al. 2024), we focus solely on the next-step and final-step advantage estimators, since there is no intermediate reward.
A(se, Ot, I) A(r(st,
V(S+i1, Vss,, 1)) (1 (T(TST or, V(se, I)) (8)
where À is balancing factor that controls the trade-off between bias and variance in advantage estimation. We set À as 0.5 in our work.
Experience Replay Buffer with Actor Confidence Filtering. In addition controlling the policy distribution drift at the algorithmic level through KL, we also implement an adaptive replay buffer to alleviate knowledge forgetting at the data level. Specifically, trajectories (which can be sparse) from each phase in the replay buffer. During phase i, we use the actor from the last phase to compute the perplexity of all actions in the buffer. Actions with
only store those successful
5

---

Published as a conference paper at ICLR 2025
perplexity within the range of 1/0.95 to 1/0.5, along with their corresponding states, are added to the training data for the current phase. This filtering process excludes both over-familiar data and data that remains too challenging for the actor. Additionally, by storing only successful trajectories, we avoid the challenge of accurately estimating intermediate states for incorrect trajectories from previous phases.
EXPERIMENTS
3.1 ENVIRONMENTS AND BASELINES
Environments. The effectiveness of our method and the baseline models is evaluated using the WebArena environment (Zhou et al., (024a).. WebArena is particularly well-suited to our needs, as it provides a highly interactive platform that supports online learning. Additionally, WebArena encompasses variety of websites, including OpenStreetMap (Map), Reddit, GitLab, online store content management system (CMS), and OneStopShop (OSS), making it an ideal benchmark for comprehensively assessing model performance web tasks. In the original WebArena environ- ment, a total of 812 instructions are provided. Considering the cost of testing, we use 165 test cases from WebArena-Lite (Liu et al. 2024ch for evaluation.
Baselines. We compare WEBRL with several methods utilizing prompting techniques, as well as models trained with alternative methods. For proprietary models, select GIPT-4-Turbo-20244000
(GPT-4-Turbo) (Achiam et al. 2023) and GPT-44o In addition to AWM (Wang et al. 2024b) and WebPilot Zhang al. 20241). we also use the results of models under the simple prompt as base- lines. Details of the simple prompt can be seen in Appendix SS D
For the open-source models, we train Llama3. (Dubey et al. 2024) using various approaches as baselines. Specifically, we employ imitation learning, also referred to supervised fine-tuning (SFT), to train both the Llama3. 1-8B and Llama3. The training data derived from publicly available human-labeled demonstra- tions, sourced from the WebArena-Lite. In addition, we also explore several reinforcement learn- ing methods for comparison, including Filtered Behavior Cloning (Filtered BC) (Pan et al. 2024), advantage-weiihhted regression (AWR) (Peng et al. 2019) and DigiRL (Bai et (004). For WE- BRL. and the reinforcement learning-basee baselines, we utilize SFT-trained Ilama3. 1-8B as the initial model for the actor. For the critic, employ Llama3. 1-8B,, augmented with randomly initialized value head to serve as the initial model. The training details of WEBRL and baselines can be found in Appendix B
ORM. WebArena-Lite (Liu et al. 2024c provides training samples along with a corresponding reward function. We further enhance this set of data by introducing task rewrites, as well as mod- ifying certain data variables, such as place names and product names. We also make adjustments to the associated reward function. .MRRM trained using rollouts of WEBRL and part of baseline methods on this set of tasks, with evaluation results determined by the reward function.
3.2 MAIN RESULTS
Our main results, presented in Table IT show that Llama3.1-8D trained using WEBRL achieves an average accuracy of 42.4%, outperforming all baselines, including prompting and training al- ternatives. Notably, WEBRL excels in specific tasks such as Gitlab (46.7%) and CMS (54.3%), demonstrating its ability to address complex web tasks effectively. Reinforcement learning-based approaches outperform those based imitation learning, such SFT and Filtered BC, which tend to over-repeat certain actions. For instance, in CMS, SFT models often over-optimize "Scroll Down" actions. This over-optimize can cause the model become trapped local loops, thereby hinder- ing its ability to achieve the overall task objective effectively. In contrast, reinforcement learning mitigates this by using a critic to estimate the value of each step, optimizing for long-term rewards, hence enabling better handling of complex, multi-step tasks. WEBRL also consistently outperforms DigiRL, A key limitation of DigiRL is that performs policy updates on a fixed set of tasks, which can lead to the model learning only a subset of operations within these tasks due to sparse feedback. This may cause the model to converge to suboptimal solutions and limit its ability to fully explore its potential. WEBRL addresses this by employing self-evolving curriculum learning, adjusting task complexity based on the model's 's abilities, and promoting broader exploration and continuous improvement.

---

Published as conference paper at ICLR 2025
Table Task success rate (SR) of WEBRL and other comparison methods, evaluated on WebArena- Lite (Zhou et al. 2024a Liu et al. 2024c), a human-verified subset of WebArena (* denotes results on full WebArena taken from literature reporting). The best and second best models are highlighted.
Models
#Params Reddit
Gitlab
CMS
Map
OSS
Avg. SR
Prompting Proprictary L.Mss
GPT-4-Turbo
N/A
10.5
16.7
14.3
36.7
13.3
17.6
GPT-4o
N/A
10.5
10.0
20.0
20.0
11.1
13.9
Llama3. I-mmat.-Instrucct (Dubey ct al.
2024
8B
0.0
3.3
2.9
3.3
11.1
4.8
1lamm331-Instruce
(Dubey et al. 2024
70B
10.5
16.7
20.0
4.4
12.7
AWM +GPT-4-0613 Wang el al
2024b
N/A
50.9
31.8
29.1
43.3
30.8
35.5
WebPilot + GPT-4o"
Zhang et al
20241
N/A
65.1
39.4
24.7
33.9
36.9
37.2
Training Open LLMS
AutoWebGLM (Lai et al 2024
6B
9.4
15.0
28.6
24.8
17.1
18.2
GLM-4- Chat (LM et al 2024
9B
5.3
10.0
6.7
3.3
6.7
6.1
GL.M-4 + SFT (BC)
9B
47.4
13.3
31.4
23.3
13.3
22.4
GL.M- Filtered BC
9B
52.6
10.0
31.4
26.7
20.0
24.8
GL.M- AWR (Peng et
2019
9B
52.6
16.7
34.3
30.0
22.2
27.9
GLM-4 DigiRL (Bar et al.
2024
9B
63.2
30.0
34.3
26.7
26.7
31.5
GLM-4 WEBRL (ours)
9B
57.9
50.0
48.6
36.7
37.8
43,0
Llama3.1 1 SFT (BC)
8B
36.8
6.7
20.0
33.3
17.8
20.6
Llama3.1 Filtered BC
8B
52.6
20.0
31.4
23.3
8.9
23.0
Llama3.1 AWR (Peng al.
2019
8B
57.9
26.7
31.4
26.7
17.8
28.5
1 DigiRL. (Bai 2024
8B
57.9
26.7
17.8
30.3
1 WEBRL (ours)
8B
63.2
46.7
54.3
36.7
31.1
42.4
1 (Dubey et al. 2024
70B
10.5
16.7
20.0
4.4
12.7
Llama3. SFT (BC)
70B
52.6
20.0
20.0
26.7
13.3
23.0
L WEBRL (ours)
70B
78.9
50.0
54.3
40.0
44.4
49.1
3.3 SCALING EFFECT OF WEBRL
We further validate the effectiveness of WEBRL on larger-scale models by training Llama3. 1-70B using WEBRL. Due to cost constraints, we only conduct 4 phases of curriculum learning. The specific results are presented Table [ After training with WEBRL, Llama3.1 1-70B achieves an overall accuracy of 47.3%, reflecting 24.3% improvement over the accuracy achieved with SFT. This indicates that WEBRL scalable and can be effectively applied larger-scale models. Further- more, when comparing the performance improvement from Llama3. 1-8B to Llama3. 1-70B achieved through SFT, WEBRL demonstrates even greater performance gains as the model scale increases.
3.4 DISTRIBUTION ANALYSIS OF ERROR TYPES
We compare the performance of Llama -8B trained with WEBRL against bascline methods across different eπror types: "Fail to Recover", "Get Stuck Midway", "Stop at Wrong Page", and "Fail to Make Reasonable Attempt", as shown in Figure 2 WEBRL demonstrates significant advantages in reducing the "Get Stuck Midway" error, especially compared SFT and Filtered BC. The "Get Stuck Midway" error typically arises when the model gets trapped in a loop, repeatedly executing the same action without making progress. Reinforcement learning helps mitigate this issue by optimiz- ing each action while considering its overall impact on the task, enabling the model to make more effective decisions. Additionally, models trained with WEBRL demonstrate enhanced robustness in handling the "Fail to Recover" eитоr. Through curriculum learning, the model gradually learns how to adapt actions when encountering failures. For example, if the search query "Pharmacy near CMU within 20-minute walking distance" does not yield the desired results, the model learns to modify the query to "Pharmacy near CMU" and attempts the search again, rather than repeating in- effective actions. In addition, WEBRL exhibits the lowest error rate both "Stop at Wrong Page" and "Fail to Make Reasonable Attempt" errors, indicating the model trained with WEBRL has a more profound comprehension of the relationship between tasks and web pages. It better iden- tify the correct page needed to complete a specific task, reducing the chances mistakenly stopping on the wrong page navigating incorrect page.

---

Published as a conference paper at ICLR 2025
20- in Trajectories
30
SFT
Filtered BC
AWR
DigiRL
WebRL
Fail Recover
Stuck Midway
Stop Wrong Page Reasonable Attempt
Failure Mode
Figure 2: Distribution analysis of error types for WEBRL and bascline methods.
3.5 PERFORMANCE ON TASKS WITH VARYING COMPLEXITY
We further analyze the performance of WEBRL and base- lines across instructions of varying complexity, shown in Figure 3 Instruction complexity is measured by the number requirements in the task. For example, the in- struction "What are the top-3 best-selling products in Jan 2023" has two requirements: identifying the top-3 prod- ucts and specifying the timeframe, giving it a complexity level of 2. Our results show that WEBRL performs well across different complexity levels, particularly excelling more complex instructions. In contrast, while DigiRL. uses online learning, it struggles with higher complex- ity due to its focus on fixed set of tasks, limiting adaptability. This highlights the effectiveness of our self- evolving curriculum learning strategy, which progres- sively increases task complexity based on the model's ca- pacity, enabling better performance on challenging tasks.
SFT
Filtered BC
Complexity Level
AWR
DigiRL
3
WebRL
2
20
40
60
Success Rate (%)
Figure 3: Accuracy of WEBRL and baselines for tasks with different com- plexity.
3.6 PERFORMANCE ON TASKS WITH VARYING STEP REQUIREMENTS
We evaluate the performance of Llama3. trained using WEBRL and baseline methods, tasks with varying step requirements. To determine the required step count for each task, we ex- clude any tasks that no model completes and use the trajectory with the fewest steps the required step count for each remaining task. The results are shown in Figure It can seen that the per- formance of both the SFT and Filtered BC trained models shows noticeable decline the task length increases. This is likely because these models optimize individual steps without considering the cumulative impact, making them less effective long-horizon tasks. DigiRL-trainee model tasks but struggles with longer tasks (more than 10 steps).
improves performance on medium-lengee This limitation may stem from DigiRL's s online learning on fixed set of tasks. Even when the model executes intermediate steps correctly, it doesn't 't receive positive rewards if errors occur in later steps, making it harder for the model learn how to complete tasks that require many steps effectively. In contrast, WEBRL overcomes this issue with curriculum learning, progressively increasing task dif- ficulty, This approach enhances the model's ability to handle long sequences, leading to significant performance improvements on tasks requiring long-term planning compared to other methods.
3.7 ABLATION STUDY
We conduct an ablation study to evaluate the impact of the replay buffer. KL-constrained policy update algorithm, and the curriculum learning strategy on WEBRL. To assess their contributions, we compare WEBRL. with four altermative models: (1) WEBRL w/o replay buffer, where training uses only the current interaction trajectory, (2) WEBRL w/o KI,, where the policy is updated using REINFORCE with value function baseline (the gradient is [AA(A(s, a, V) log T (a|s, (, Z))) but retains the replay buffer, (3) WEBRL w/o KL & replay buffer, which uses neither replay buffer nor the KL-constrained policy update algorithm, and (4) WEBRL w/o CL, which ablates the curriculum learning approach, utilizing only the instructions generated in the first phase. The results, shown in Figure reveal that when the replay buffer is removed, both WEBRL w/o replay buffer and WEBRL w/o KI. & replay buffer experience worsening performance over time. This decline occurs because
8

---

Published as a conference paper at ICLR 2025
WebRi DigiR1
SFT
Piltered BC
AWR
WebRL w/o WebKL w/o KL replay bufler WebRL
100
WebRL w/o WebKL w/o replay butler
424%%
40
Suctess Rate (%6
80
Success Rate (%)
34.5%
35
60-
32.7%
30-
40
24.6%
20
20,5
20
20005%
7
Init
3
Number of Steps
Number of Phases
Figure 4: Accuracy of WEBRL. and base- Figure 5: Ablation study of WEBRL. replay buffer, lines for tasks requiring different steps.
Kl--conttrained policy update and curriculum strategy.
the models lose access earlier experiences and focus only recent data, leading to knowledge degradation. Comparing WEBRL, and WEBRL w/o KI., WEBRI. consistently performs better, due to the incorporation of KL--constrained policy update algorithm. When the replay buffer is not used, the KL-constrained policy update algorithm degrades more slowly than REINFORCE with a value function baseline. This is because the KL constraint better retains knowledge from past policy iterations by explicitly controlling the divergence between the current and previous policies. In contrast, REINFORCE with value function baseline more prone to overfitting the data from the current phase of training, leading to less stable performance over time. Overall, the KL-constrained policy update algorithm more effective at balancing the retention of past knowledge with the learning of new information. When comparing WEBRL to WEBRL w/o CL, both exhibit overall upward trend due to online learning. However, WEBRL w/o CL progresses more slowly and reaches lower performance ceiling because it operates within fixed task framework, whereas WEBRL generates new tasks that adapt to its evolving capabilities. This highlights the effectiveness of our self-evolving curriculum learning approach. Additional ablation studies further investigate the effects of curriculum learning are presented in Figure [2
The impact of B. We investigate the effect of 3 per- formance with and without the replay buffer, as shown in Figure 6 The study involves curriculum learning in single phase. First, when / is too small (c.g.. 3 = 0.01), performance declines regardless of the use of the replay buffer. This happens because a small & leads to weak control over KL divergence, causing the model overfit the current data. Second, without the replay buffer, per- formance initially improves as 33 increases but declines when 3 becomes too large, indicating that larger (c.g., 2 1) will overly restrict KL divergence, limiting the model's ability to update its policy and learn effectively. In contrast, with the replay buffer, performance stays high even at larger values. The replay buffer's historical experiences allow for more frequent parameter updates, supporting steady improvement despite higher 3.
wwoo replay buffer + seplay buffee
Soccess
26
-4
12
0.01
10
Value of
Figure 6: The impact of 8 of KL- constrained policy update algorithm on the model's performance.
The influence of perplexity. We analyze the impact of using perplexity to select data from replay buffer for train- ing. Various perplexity thresholds are tested the first learning phase, and the results are summarized in Table can be observed that training on data with very low per- plexity (range [1, 10/9]) leads to performance deteriora- tion. This suggests that repeatedly learning overly famil- iar data harms the model. Similarly, training exclusively on data with high perplexity (above 10/5) also degrades performance, likely due to the model struggling with unfamiliar data, causing a signifi- cant shift in policy distribution and hindering generalization. Optimal performance is achieved when training on data with a perplexity range of [10/9, 10/5], indicating that a balance between simple and complex data enhances model performance by focusing moderately difficult examples.
Table 2: The impact of perplexity in re- play buffer filtering of WEBRL..
[1,0]]
[1,
10
10
[00
po]
9
5
5
29.1
28.5
30.9
23.0

---

Published as a conference paper at ICLR 2025
Table 3:
Evaluation on output-supervised methods (baselines adopted from (Pan et al. Our ORM, without accessing proprietary GPT-4, performs the best among all.
2024
Our ORM (8B)
GPT-4
Captioner + GPT-4
GPT-4V
Test Dataset (%)
80.8
71.9
72.6
71.2
Rollout (%)
79.4
71.2
73.3
70.5
3.8 EVALUATION OF ORM
In the WEBRL framework, continuous improvement relies heavily on the performance of ORM, which plays crucial role in evaluating interaction trajectories guide the agent's 's learning. To assess ORM's effectiveness, we compare its performance with several baseline models, including GPT-4-Turbo using identical inputs of our ORM, Captioner + GPT-4-Turbo, and GPT-4V, both using the same prompts with Pan et al. (2024). We evaluate ORM and the baselines on two datasets: the WebArena-Lite test set and 100 sampled rollouts which are manually labeled. For the WebArena- Lite test data, we use its reward function outputs as labels. The results, shown in Table B indicate that while the baseline models achieve a consistent accuracy just above 70%, our ORM outperforms them with approximately 80% accuracy.
4 RELATED WORKS
Adopting LLMs as Agent. As LLMs advance, they extend beyond text generation (Zheng et al,,
2023 Zhao al. 2023) and reasoning (Zelikman al. 2024 Zhang et al.) (024d) to device con-
trol. Research follows training-free approaches that enhance LLMs via prompting (Yan et al. 2023
He et al. 2024 Zhang et al. 2024c) and system design (Liu al. 2023 Yang al. 2023 Wang
et al. 2024a Wu et al. 2024 long et al.
2024 Zhang et al. (2024a). but these remain limited by underlying LLM capabilities Chen et 2023 Zeng al. [023]] Xie et al.] 2024). Training- based
methods using imitation learning require costly expert demonstrations (Zhang & Zhang 2023 Gur
et al., 2024] Deng et al., 2024, Hong et al. 2024; Rawles et al., 2024 Zhang et al. 2024b). While
some use GPT-4 for demonstration generation (Chen et al.J [2023), they often focus individual
actions without considering long-term effects, limiting generalization (Ghosh et al. [021]] Bai et al.
2024.. Some approaches sampling estimate long-term effects (Lai et al. 2024 Putta al.
2024
while others leverage reinforcement learning (Carta et al.] [023]] Bai et al. 2024 Pan et al,,
2024
Tan al. 2024 Zhai al. 2024). Most rely on static task sets, hindering continuous im-
provement. We propose a dynamic task generation framework with a policy-update algorithm for ongoing enhancement.
Reinforcement Learning for LLMs. RL has gained traction in L.M training for preference op- timization (Ouyang et
2022 Casper al. (024) and reasoning (Hao et al. 2023 Pang et al., 2024). For device control tasks requiring sequential pociiionnmaking. existing research uses online learning methods like AgentQ (Putta et al. (024) with DPO (Rafailov et al. 2024b), or actor-critic architectures (Bai et al.) 2024, Zhou et al. 2024b) Zhai et al.) 2024), which we adopt. However, web tasks often provide only binary success/failure feedback after multiple interactions, potentially penalizing correct intermediate actions. Current research typically focuses on fixed task sets, lim- iting continuous improvement. We propose an autonomous curriculum learning mechanism that dynamically generates tasks based on agent skills, plus KL-constrained policy update algorithm and specialized replay buffer to reuse historical data and prevent knowledge forgetting during cur- riculum updates.
5 CONCLUSION
In this work, we introduce WEBRL, novel self-evolving online curriculum reinforcement learning framework for training LLM-based web agents. By addressing key challenges such as the scarcity of training tasks, feedback signal sparsity, and policy distribution drift, WEBRL enables continual and consistent improvement in agent performance within online environments like WebArena. Our approach demonstrates substantial performance gains, significantly surpassing existing state-of-the- art web agents and proprictary LLM APIs. These results highlight the effectiveness W WEBRL in advancing the capabilities of open-source L.LMs for web-based tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their suggestions in refining this work. This work is in part supported by NSFC 62495063, 624B2084, 62425601, and Tsinghua University (De- partment of Computer Science and Technology)-Sieeeeee Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things (JCIIOT). Wei Xu is supported by the National Key R&D Program of China 2023YFC3304802 and NSFC under Grant U2268202 and 62176135.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- girl: Training in- in-the-win wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406. / 1189, 2024.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference Machine Learning, pp. 3676-3713. PMLR, 2023.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2024.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.059115 2023.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprini 2rXiv:2407.21783 2024.
Jesse Farebrother, Jordi Orbay. Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.0395)) 2024.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in neural information processing systems, 14:25502-255111 2021.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2466. 12793, 2024.
Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, and Yu Su. Middleware for llm: Tools are instrumental for language agents in complex environments. arXiv preprint arXiv:2402.1 14672, 2024.
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. In The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: arXiv:1812.05905. 2018.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154-8173, 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint 1rX1122401.13919, 2024.
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
lat Long long, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, and Jie Tang. Openwebagent: An open toolkit to enable web agents on large language mod- els. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 72 81, 2024.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL
net/forumt iineiiidiiiEEE 8yNQM66
Hanyu Lai, Xiao Liu, lat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model- based web navigating agent. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. $295-5306 2024.
Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long long, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint 2.000241. 2024a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agenthench: Evaluating Ims as agents, In The Twelfth Inter- national Conference on Learning Representations, 2024b.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song. Shudan Zhang. Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagenthench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.00 06327, 2024c.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating aun-augmented autonomous agents. arXiv preprint arXiv:2308.0553 0059600 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Jayy Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.197:: 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 19 2019 2019.
12

---

Published as a conference paper at ICLR 2025
Pranay Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.071111 07119, 2024.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r' to q": Your language model is secretly q-function. arXiv preprint arXiv:2404. 12358, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02433. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1xXiiv1707006344 2017.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environments via reinforce- ment learning. In The Twelfth International Conference Learning Representations, 2024.
Guanzhi Wang, Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024a.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409007429, 2024b.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07455 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint 2rXiv:2304.12244 2023.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint (tt::23/1.07562. 2023.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20757. 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09622 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310. /2823, 2023.
13

---

Published as a conference paper at ICLR 2025
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision- agents via reinforcement learning. arXiv preprint arXiv:2405.10292 2024.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024a.
Jianguo Zhang. Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liang- wei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint 24..:::2022/55000 2024b.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thuuuhh for gui agents. arXiv preprint arXiv:2403.027133 2024c.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339, 2024d.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.1 15240, 2024e.
Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. arXiv preprint arXiv:2408. 159788 2024f.
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309. /436, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 32..3033........ 18223, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning. 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng. Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents, In The Twelfth International Conference on Learning Representations, 2024a.
Xuanhe Zhou, Guoliang Li, and Zhiyuan Liu. Llm as dba, arXiv preprint S.0iv:2300.05488. 2023.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer:
Training language model agents via hicrarchical multi-turn rl. In Forty-first International Conference on Machine Learning, 2024b.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF POLICY UPDATE ALGORITHM IN WEBRL
A.1 DERIVATION
By substituting eq. into eq. 2 can obtain:
x' (arlsts 1) B log Tna(ar st,
= r(st, ar, + V* ($141, V*()s) = As
(9)
n" (aels,, I) Ner(ae 1) exp (3 (5) at,
(10)
r is the optimal solution to our target (eq. 1 If vy is represented by function approximator, we aim make it as close as possible ..
arg min By (log Ta(als, - log lals, 1))²]
27 (11)
1 log (aglo [(alss,]) - log (ar 333 A As,
= arg min Ey
(glog
To(als, s, T T)
A as 27
arg min By
Hence, the loss function can be defined as:
((TT) Ev (8 To Ta(a|s, Tar(
A (0,,;
(12)
where y represents the distribution of experience used for training. For any given state-action pair, my is expected to match the target policy x" (als,, /). There no restriction on the data distribution indicating the algorithm can function effectively in an off-policy setting, Since A* is unknown, we estimate (s, (s, a, I) using eq. 7 and eq. 18 to train the policy wa. The feasibility of this approach
is demonstrated in Appendix A.2
Further Analysis. Although y can follow any distribution, achieving stable policy improvement often requires some control over v. The primary goal is to enhance the probability of actions that successfully complete the task and to address the deficiencies of the current policy Ty. Therefore, typically consists of data sampled from the current policy being trained and prior successful experi- ences.
Why not use KI. divergence to measure the distance between T² and mg. For eq. many studies use KL divergence to measure the distance between two policies. When KL divergence is used, the optimization goal of is:
arg min Ex-d(() [DLL (m1(-),, [- T))]
arg max E Eg d() Bawr (als.1) [log To (alss,
arg max Esnd(s) / Trer(als, I) exp( 3 A" a, I) log molals Da
(13)
arg max Egdd) Buwnnelale.t) [oggmoaass To(als 7) exp( A (s ('ss,,
where d(s) is a distribution state. The choice of using eq. instead of eq. 13 as the op- timization target can be explained as follows: (1) Eq. 13 imposes stronger restrictions on the training data. Specifically,, requires training data to conform to the distribution muf-
15

---

Published as a conference paper at ICLR 2025
When Net fails capture previous successful experiences, becomes difficult to sample these experiences from Tet. which in turn, can lead to further forgetting of those experiences. (2)
Succesa Rate (XC
MSB Logptob Divergenoe
35
Both eq. [1] and eq. 13 aim to approximate the distance be- tween distributions. The use of Mean Squared Error metric to measure the distance between policy distributions is also employed in Fujimoto & Gu (2021). (3) Eq. 13 is unable to decrease the probability of actions with negative advantage. When correct action is hard sample from Net. eq. 13 may
30-
25
Init 1
5
Number of Phases
Figure 7: Comparison of using
instead increase the probability of actions with negative advan- tage, compounding the challenge of sampling correct actions. We also conduct experiments to further validate the superiority of eq. 11 with the results presented
eq. 11 and eq. 13 as target.
in Figure In these experiments, eq. 13 is trained exclusively on data sampled from Trets while
eq. utilize both replay buffer data and sampled data.
When discount factor 7 is used. When considering the discount factor, eq g modified as follows:
(a. [st I) (a 1)
log
r(st, ae + "V" V = !I
(14)
This will not affect the subsequent derivation of the policy loss function (eq. 12 The loss function for the critic remains unchanged. However, the equation to calculate the advantage is modified as follows:
A( A(se,a,,)) = -=())=(((()))))))
+ ( +
(15)
In our experiment, we use the discount factor and set its value to 0.9.
We use eq. 7 and eq. off-policy manner.
g to estimate A* a, a, I). The training of critic (eq. also operates in an
A.2 PROOF
To prove the feasibility of replacing A" with an advantage estimator A based value network V. which is trained on data collected throughout the training process, we aim to demonstrate that using V to train the policy can lead to policy improvement.
The current policy denoted by Told The training dataset is composed of trajectories sampled using Vold well historical experiences stored in replay buffer. The behavioral policy associated with the dataset D is referred to as kμ. The corresponding value function and action-value function under Th denoted as 4TH and Q" respectively. Below, we demonstrate that minimizing the objective arg ming En[[[og,, (a, exp( Q (s..)) [)]] can lead to policy improvement. For brevity, we omit in the following proof.
Let the new policy Tnew Tnew (arls:) be defined as:
(QTH (Q% (8t,at)
V (st) (8+))
Tnew (aw (s:) = exp
(16)
3
can shown that: DKI. (Tnew (new) S DKI. (F Tnow). From this, we derive the following equality:
(=a(se,att) (*+
Bayss
log Tnew (aelse)))))
3 +
B
(17)
Qar Qur
(*
< Earr
log (
B
B
Since (st) depends only on the state s, and not on the actions, it can be factored of the expectation. We can rewrite the inequality as:
Eag Faee !QTH (st, ar) log Tnew (a:ls)) 2 V**(s))
(18)
where (st En-,, (a, (s,) QQ (0 a,) 3 log (a s;)] which is satisfied in maximum en- tropy reinforcement learning setting (Harrnoj al. 2018).
16

---

Published as a conference paper at ICLR 2025
Consider the soft Bellman equation:
Q (st, ar) r() + B log (ar((r)))) + V ($t+1)
I a() + +Blogg mer(a (t) + Eatt l ($t+1, ac+1) - 8 log Tnew ( ((+1)]
S Te (sc. ar)
(19)
where we iteratively expand Q by applying the soft Bellman equation and the bound provided
in Eq 18 Since Tnew is an improved policy over Ty (i.e.,
2 Q"x (8,, 4z)), and me is
trained elosely match new. it follows that me will also be an improved policy over Ta provided the approximation is accurate. Formally, we have:
( v Vima(d) (=*(s)
(20)
Since we only incorporate the correct trajectory from the historical data, we can assert that policy Ty will not perform worse than the old policy Toi-- Therefore, we have the relationship:
V (s) TH (s) VF (s)e(s)
(21)
The policy improvement process well-established. By iteratively performing policy evaluation and policy improvement, the policy Th can be made converge toward the optimal policy n² If the data in the replay buffer is not utilized, the same theoretical process holds. However, in this case, T and T are replaced by V and QFOM.
We conduct an experiment to evaluate the effectiveness of us- ing W instead of a in WEBRL training. In this exper- iment, the policy Ha is trained using data from two sources:
Success Rate (%)
40
replay butfer
-wio replay
35-
rollout trajectories during the current phase and past experi- ences stored in the replay buffer. For the critic, we compare two training settings: training with data from both rollout tra- jectories and experiences in the replay buffer versus using only rollout trajectories. The results, as presented Figure X re- veal that the model performs better when the critic is tramed with data from both the replay buffer and the rollout trajecto- ries. This observation underscores the significance of integrating experiences from prior successful trajectories into the critic's ' training process, which can lead to more effective policy improvement.
30
20
Init i
4
5
6
7
8
Number of Phases
Figure 8: Comparison of using VTH and V
A.3 COMPARISON
The comparison result between the policy update algorithm of WEBRL and other algorithms is shown in Table
Table 4: Comparison between WEBRL and different algorithms based on various criteria.
Algorithm Off-policy
Pair-wise Data
KL-constrained Target
Reduce Error Prob
WEBRL
DPO
PPO
AWR
Comparison with DPO (Rafailov et al., 2024b). DPO also solves for the optimal value of eq. [ obtaining the relationship between the optimal policy x² and the reward r. Rather than explicitly solving for the reward r, DPO employs contrastive learning by constructing paired data to build the learning target of ny. This approach allows DPO to bypass the need to estimate r. In contrast, approach explicitly fits the value function V to guide the optimization of Tg. making it converge towards the optimal policy x*. However, DPO requires pair-wise data, which is challenging to
17

---

Published as a conference paper at ICLR 2025
obtain in web tasks. This is primarily because implementing state backtracking web pages is difficult, making hard to collect outcomes of different actions on the same page state.
Comparison with PPO (Schulman al, 2017). Eq is also the target of RLHF. Prior work often employs PPO to optimize this objective. However, as an on-policy algorithm, PPO has low sampling efficiency and requires large amounts of data for stable improvement. This limitation makes unsuitable for environments such as WebArena and real-world websites, where interaction expensive and inefficient. Instead, our approach operates an off-policy manner.
Comparison with AWR (Peng et al. 2019). AWR is one of the widely used offline RI algorithms. The loss function of AWR arg max, En [log To(a)s /) exp(A(ss, 7)/3). Compared to our method, AWR has difffernnt optimization goal. Our approach explicitly minimizes the KL divergence between the reference policy Tef and the current policy Te. while AWR focuses on constraining the KL divergence between the current policy and the behavioral policy derived from training data (instead of the reference policy Tet)). Additionally, AWR does not directly reduce the probability of incorrect actions. In web tasks, sampled data often include unsuccessful traces. Ignoring such data wastes valuable information, while using risks increasing the likelihood of incorrect actions. In contrast, our method effectively reduces the probability of incorrect actions.
B TRAINING DETAILS
B. DETAILS OF WEBRL
The agent observation consists of three components:
User instruction: The instruction provided by the user.
Action history: A record the actions the agent has previously taken.
Webpage HTML: HTML content of the current webpage.
We process the HTMI., simplifying its structure and assigning distinct element IDs all clickable elements. This facilitates the model's ability to identify and indicate which specific element requires manipulation.
The agent actions are mostly similar to those defined in WebArena-Lite (Liu et al. 2024c), with some additional actions.
Click: Clicks an element with specific ID.
Hover: Hovers over an element with a specific ID.
Type: Types message into input box with specific ID.
Search: Types message into an input box with a specific ID and presses Enter to initiate search.
Press: Emulates a specific keyboard key combination.
Scroll: Scrolls the page up or down.
Sclcet.dropdown Selects an option from dropdown menu with a specific ID.
New- Opens tab in the current browser.
Tab. focus: Switches focus to browser tab at a specified index.
Close tab: Closes the current tab.
Goto: Navigates to specific URL.
back: Returns to the previous page.
Go forward: Moves to the next page available.
Exit: Terminates the operation, returns the response, and exits.
To provide more detailed information about the action, we include comments labeled with "# Ele- ment:" in the action, which deseribe the operated element. Similarly, we include comments labeled "# Note:", which quote relevant information from the current webpage that supports completing the instruction. An example of the agent's specific input and output shown in Figure The input is the
18

---

Published as a conference paper at ICLR 2025
observation, while the output specifies the action to be performed the webpage. The "element" argument identifies the target element for the action. More examples can be found in Appendix
The detailed training process of WEBRL is shown in Algorithm First, perform supervised fine-tuning using the WebArena-Lite training dataset. We then initialize the replay buffer and failure set by running the SFT-trained model instructions corresponding the WehArena-Liee training dataset. Subsequently, in each phase of the self-evolving curriculum reinforcement learning process, 500 new instructions that meet the filtering criteria are selected from those generated by GPT-4o. Both newly generated interaction data on these instructions and historical data with perplexity be- tween 1/0.95 and 1/0.5 from the replay buffer are used train the actor and critic. The amount of historical data used is limited to twice the size of the interaction data. The hyperparameters employed in WEBRL, are presented Table
B.2 DETAILS OF BASELINES
For RL-based baselines (except DigiRL., the interaction data from WEBRL'S first phase used, while DigiRI. is trained using the first-phase instructions an online learning setup. Hence, except for DigiRL, the other RL baselines fall under offline reinforcement learning.
We reproduce the same framework used in DigiRL within the WebArena environment. Specifically, use the same components, including the AWR method, the instruction-level and step-level value functions, and the replay buffer described in DigiRL. To apply DigiRL in WebArena-Lite, the main modifications we make are adjusting the data format to align with WEBRL and tweaking certain hyperparameters. DigiRL also conducts 8 rounds of interaction and training. The hyperparameters employed in those baselines are presented Table
Agent Input
Instruction: List where status Canceled and the recipient Jennifer White, ordering them by decressing value.
Round
user:
List orders where the status is Canceled and the recipient Jennifer White, ordering them by decreasing total value.
assistant:
Element: the Sales item the left sidehar, second from the top dofaction=" "Click", element="?" "2")
** Simplified
assislant:
Element: the 'Orders' link
dofaction="Clice element="4))
Round 2
chtml data- btt.bbboooo00000000000 7200122807200>>bbbdd data 1atps/0000122,9668"2".. data-bbox- (trps/s000000,,211"""" sul data-bbux-=002. "0,75,88,,666"
espan "d="I" besbbbbbo---- 0hoxx=11,,112,60111"""
spans.
middle content omitted ->
<button type-"button bbutoon sattabbox="548.110 18,,,000,11117>--- </buttono e/ ttt.////ll </ul>
<button id= "25" type- "button" data-bhox= "116,00=="767,,,,88889"" Clear all </buttons </dino content omitted
</ddvv> </ maino- </divs </bodys </htmb>
assistant:
Agent Output
Flemenn:: 'Clear
dolaction "Clic" element- (25")
Figure 9: The input and output format of WEBRL and baselines, where the input composed of task instruction (in green), action history (in blue), and HTML of the current webpage (in orange). The output (in red) the action taken on the current webpage.
B.3 DETAILS OF ORM
WebArena-Lin provides 1,186 training samples each comprising an instruction, a trajectory. and reward function. To train the MORM. first enhance the WebArena-Lite training dataset by rewriting the instructions. Then. we collect rollouts from all baseline methods on this augmented dataset. These rollouts, combined with the original WebArena-L ite training samples, form the com-
19

---

Published as a conference paper at ICLR 2025
plete training dataset for the MORM, with total of 12,200 samples. We the associated reward function to label these trajectories, identifying whether they successfully complete the task. These trajectories subsequently used to train the ORM, with the specific hyperparameters listed in Ta- ble O The prompt for MORM shown in Figure T5] MORM required to produce either "YES" "NO" as its output. To determine the evaluation result, we compare the probabilities assigned to "YES" and "NO" and select the one with the higher probability. We need to train MMRM be- cause new tasks that do have predefined reward functions will be generated in the subsequent self-evolving curriculum reinforcement learning process. We rely on the trained MORM to judge whether each task successfully completed.
Algorithm WEBRL.
Input: WebArena-Lite training set Do and corresponding instruction set Lo. Base model Musse. Replay buffer B, Failure set F, Reward model MMRM. Number of phases N, Number of instructions per phase K
Part 1. SFT Training
1: MSFT t perform supervised fine-tuning on model Mhnse with dataset Do
2: Droilou f- rollout trajectories of MSET on Io
3: Duccees: Diat 4- evaluate Drlloaur using WebArena-Lite reward function
4: B B+-D Do # Initialize replay buffer with successful trajectories
5: F -Dait # Initialize failure set with instructions of failing trajectories
Part 2. Self-evolving Curriculum RL
6: mi f- -MSTT Initialize actor/policy
7: V1 4- MSFT with a randomly initialized value head # Initialize critic
8: for n in I...N do
9:
In +0
10:
while size(I,) S K do
11:
I ggeneration f- instructions generated by GPT-4o with instructions from F as examples # instruction generation process
12:
Jūieer filter( generation) # instruction filtering process
13:
In - In U Intter
end while
15:
Drollout f- rollout trajectories of n. on I
Dauccess: Diat evaluate Droliou with MORM use MORM to label the rollout trajectories
17:
Dexperiance
experiences from B with perplexity computed by m, between 0.95 and 0.5
18:
I1+1. Vn+1 train(T Vn, Droloou U Dexperience). # use loss functions from eq. and eq.
19:
B
B U D success
20:
F
Dial
21:
end for
C OTHER QUANTITATIVE EXPERIMENTS
Figure 10 iillusstrates the performance variation curves of Llama3.1-8P trained with WEBRL on each website. can be seen that in all the sites except for Map, there clear upward trend. However, in the case of Map, there is an initial upward trend followed by a decline. We hypothesize that the final decline caused by a significant increase in OSS and CMS implementation, which creates a trade-off. This trade-off leads to performance drop in Map and a slight decline in GitLab.
Figure TT illustrates the error bars for WEBRL across each phase. We conduct four sets of exper- iments, each utilizing distinct random seed. The results from these four sets of experiments are presented in Figure TT Despite some fluctuations in accuracy, WEBRL. consistently demonstrates an overall upward trend. Furthermore, its accuracy consistently surpasses all baseline methods after completing the final phase training.
Figure 12 presents the success rate of WEBRL and DigiRL, both with and without the applica- tion of our self-evolving curriculum learning. The results indicate that incorporating curriculum learning significantly enhances the performance of DigiRL. However, its performance still remains below that of WEBRL. Conversely, when WEBRL is implemented without curriculum learning, its performance experiences notable decline, yet it still slightly surpasses that of DigiRI..
20

---

Published as a conference paper at ICLR 2025
Table 5: The hyperparameters we employ in WEBRL and baselines.
Method
Hyperparameter
Value
learning rate Ir scheduler type
le-5
cosine
warmup ratio batch size
0.1
SFT
128
training epoch
1
cutoff length
16384
learning rate Ir scheduler type batch size training epoch cutoff length filtering threshold
le-6
constant 128
Filtered BC
16384
70th percentile
actor learning rate actor Ir scheduler type critic learning rate critic scheduler type batch size discount factor actor training epoch
Ic-6 constant le-6
constant
AWR
128
0.9
1
critic training epoch cutoff length
16384
actor learning rate actor Ir scheduler type critic learning rate critic Ir scheduler type instruction value function Ir instruction value function scheduler type
le-6
constant
1c-6
constant le-6
constant
DigiRL
batch size
128
discount factor actor training epoch critic training epoch
0.9
1
instruction value function epoch
rollout temperature
1
replay buffer size cutoff length
10000
16384
actor learning rate actor Ir scheduler type
1c-6
constant
critic learning rate critic Ir scheduler type batch size discount factor actor training epoch critic training epoch rollout temperature replay buffer size cutoff length
le-6
constant
128
WEBRL
0.9
1
100000
16384
Figure 13 illustrates the error bars for both the baselines, DigiRL. w/ CL., and WEBRL. Notably, the performance of WEBRI. and DigiRL. with curriculum learning (CL.) shows a higher standard devia- tion, which can be attributed to the inherent randomness in task generation. Despite this variability, WEBRL consistently outperforms all baseline methods, demonstrating its effectiveness.
Figure 14 compares two settings of the replay buffer: one containing only past successful trajecto- ries and the other including all past trajectories, both successful and failed. The results show that including failed trajectories leads to slower and more volatile performance improvement. This issue arises because the advantage of an action in historical trajectories is estimated based on the trajec- tory's final reward. Including failed historical trajectories introduces potential problem: even if
21

---

Published as a conference paper at ICLR 2025
Table 6: The hyperparameters we employ train the ORM.
Hyperparameter
Value
learning rate Ir scheduler type warmup ratio batch size training epoch cutoff length
5e-6
cosine 0.1
128
16384
certain actions in a failed trajectory are correct, they are likely to receive a negative advantage when the trajectory's 's final reward is zero. This reduces the likelihood of selecting these correct actions, thereby hindering effective policy improvement and leading to degraded performance.
We employ variety of prompts to assess the stability of the curriculum learning strategy in WEBRL. Specifically, we uti- lize three distinct prompts: Prompt (in Figure T6). Prompt 2 (in Figure [7.. and Prompt 3 (in Figure [8]. The training results for these prompts, measured after eight phases of train- ing, are presented in Table 7 The results demonstrate that when using variety of prompts to generate tasks, WEBRL consistently delivers strong performance, highlighting the stability of our method.
Table 7: Performance of different prompts for task generation.
Prompt
Prompt
Prompt 3
42.4
40.6
43.6
Reddlt
Gelah
40
Success Rate (26)
50
Map
CAN
Number of Phases
Figure 10: Performance variation curves of Llama3.1-8B on each website under WebRL. training
D PROMPTS EMPLOYED IN WEBRL
The prompt used for MORM shown in Figure 15 We require the model to output "YES" or "NO" to determine whether certain trajectory successfully completes the corresponding task. Consider- ing the limited context size, only input the HTML content of the last state.
22

---

Published as a conference paper at ICLR 2025
Mean
WebRL WebRL w/o CL
DigRt - Dig RL. w/ C1.
Succoss Rate (56
Success Rate (%)
35-
35-
30-
25
20
Init
1
2
5
7
8
Number Phases
Number of Phases
Figure 11: Error bars of WEBRL for each phase.
Figure 12: Comparison of WEBRL and DigiRL with and without curriculum learning (CL).
40
WebRL w/o replay builer WebRL
WebRL failed
Seccess Rate (%)
Success Rate (%)
30
35
30-
20
25
10
20
Init
3
8
DipRi. W/CI
Number of Phases
Figure 13: Error bars of baselines, DigiRL w/ CL. and WEBRL.
Figure 14: Comparison of WEBRL and WEBRL with failed trajectories in the replay buffer.
The prompt for generating new instructions is presented in Figure 16 fails to complete successfully in previous phases as seeds for generating new tasks of similar diffi culty but with greater variety. The prompts shown in Figure 17 and Figure 18 represent alternative versions of the prompts used for task generation.
We use tasks that the model
The simple prompt we use test models including GPT-4-Turbo, GPT-4o. Llama3, 1-88---IBstrue and Llama3. 1--0..1------------ Instruct shown Figure 19 In this prompt, we define the feasible actions and provide illustrative examples. Additionally, we outline set of requirements in the "REMEMBER" part to guide the model's behavior.
The prompt
to filter the instructions generated by GPT-4o is presented in Figure 20 In this prompt, we provide a rule defining the feasibility of instructions across the five websites in WebArena.
E CASE STUDY
instructions generated by the self-evolving curriculum learning strategy across different phases. Although these instructions are grouped by phase, the instructions shown in phase i + phase increases, two types of augmentation occur for previously incomplete instructions. In one case, new instructions with similar task requirements or lower difficulty levels are generated. These new instructions help the model to successfully complete previously unfinishable tasks by allowing practice and build competence with the easier or related tasks first. For example, in phase 2, the instruction improves upon the phase 1 instruction by offering clearer task description and ex- plicitly requiring results in "yearly interval". This enhancement enables the model to complete the task successfully by removing ambiguity. Additionally, changing the description of the instruction can improve agent exploration. With the positive feedback from the clarified phase 2 instruction, the model can better understand and accurately perform the original phase I task as well. In another case, tasks with increased complexity and diversity are generated. This task complication facilitates continuous improvement the model's capabilities by challenging its performance boundaries.
Figure 21 presents
necessarily generated using the instructions from phase i seeds. As the
23

---

Published as a conference paper at ICLR 2025
System Prompt:
You expert in evaluating the performance of a website navigation agent. The agent is designed to help human user navigate the website to complete task. Given the user's intent, the agent's action history, and the final state of the screen, your goal to decide whether the agent's execution is successful or not. You must respond with YES or NO.
User Prompt:
The User Intent: (instruction]
Action History: (action history)
The Current Screenshot: (html of last state)
Figure 13: Prompts for MORM to assess the completion of Instructions.
User Prompt:
You are smart task creator for a website intelligent assistant. Your goal is to generate clear and practical tasks that the assistant can assist people with when they use (web) in their daily lives. These tasks should encompass wide range of possible instructions and questions that may arise when using [web) website.
Your need to draw inspiration from the Given Task# to create new tasks, These new tasks should belong to the same domain as the #Gvven Task# but more diverse. The difficulty level the #Created Task# should be similar to that of the #iven Task*, The #Created Task must be reasonable, understandable and realistic. '# Given Taski', Created Taskt', ; 'given task' and 'created task' are not allowed to appear in #Created Taske.
You need to make sure, as much as possible, that variable names in #Created Task#, like place name, username, and product name, are consistent with Given Taski. You need to avoid making up some new variable names yourself.
#Given Task#f
[task examples]
#Created Task#
Figure 16: Prompts for instruction generation.
Therefore, the process of instruction generation exhibits such pattern: for tasks that the model unable perform, analogous tasks are created provide incremental steps that facilitate learn ing how to accomplish that type of task. Furthermore, tasks that remain challenging for the model are also generated and undergo the aforementioned iterative process. Through this cycle of chal- lenge and refinement, the model's capabilities gradually expand, enabling it to handle increasingly complex tasks over time.
24

---

Published as a conference paper at ICLR 2025
User Prompt:
You are smart task creator website intelligent assistant. Your goal to generate clear practical tasks that assistant can assist people with when they use (web) their daily lives. These tasks should encompass a wide range of possible instructions and questions that may arise when using [web] website.
Your need draw inspiration from the #Given Task# create tasks. These new tasks should belong the domain the #Given Taski but be diverse. The difficulty level of the #Created Taski should be similar that of the #iveen Task#. The # #Created Task# must reasonable, understandable and realistic. * Givve Taske", *Createe Task#', "given task' and 'created task' not allowed appear #Created Task#.
"Guidelines:"
Use varicty of phrasing styles avoid repetitive expressions.
those in the provided task examples, such place names, usernames, subreddit name, and product names. Avoid inventing entirely new variable values.
Use variable values that
Maintain the same similar difficulty level the #Gveen Task#. Tasks can be slightly less challenging but should stay within reasonable range.
Before each generated task, first output the thought process.
#Given Task#
(task examples)
#Created Taskit
Figure 17: Prompts for instruction generation (version 2).
25

---

Published as a conference paper at ICLR 2025
User Prompt:
You are highly intelligent task creator for a website assistant. Your role is to generate clear and practical instructions or questions that users might need help with when interacting with the "[web]" service in their daily lives. The tasks you create should reflect wide range of possibilities and be realistic for users to request assistance with.
You will be provided with "Exxmmple Tasks" to draw inspiration from. Your job is to create new tasks within the same domain as the examples, ensuring diversity in phrasing, purpose, and scope.
#*я "Guidelines for Task Creation:**
1. "Extract Variable Values:"
Analyze the provided examples and extract the key variable values, such as names, places, usernames, subreddit names, product titles, or other specific terms.
Use these variable values when constructing new tasks. Avoid inventing entirely new values.
"Maintain Domain Difficulty:"
Ensure new tasks belong the same domain as the example tasks.
Match difficulty level the example tasks, allowing for slight variation (e.g. slightly easier harder)
while keeping them reasonable and realistic.
"Ensure Clarity and Varicty:"*
Format each task clearly using backticks (") for task descriptions.
Use varied phrasing to avoid repetitive expressions make tasks more engaging and dynamic.
"Avoid Explicit Reference to Examples:"
Do not include phrases 'Example Tasks" or "New Tasks" in your output.
Tasks should stand independently. without explicit references the examples the task creation process.
#** "Example T [aam_exxaples]
#a* "Steps Follow:"
1. **xttact Variable Values:**
List the key variable values from the example tasks.
2. *Cenerate New Tasks:"
Using the extracted variable values, construct set tasks within the same domain.
"OOtpuut *ormat:"
1. Extracted Variable Values:
Variable 1: [value]
Variable 2: [value]
2. New Tasks:
"Task 1 description"
"Task 2 description"
Figure 18: Prompts for instruction generation (version 3).
26

---

Published as a conference paper at ICLR 2025
System Prompt:
Setup
You are professional web browsing agent assistant that can fulfill user's high-level instructions. Given Simplified html of the browsed webpage each step, you plan operations in python-style pseudo code using provided functions, customize functions (if necessary) and then provide their implementations.
# More details about the code
Your code should be readable, simple, and only CONE-LIN-OF--CODE at time, avoid using loop statement and only use if-else control if necessary. Predefined functions are as follow:
def do(action, argument, element):
"""A A single browsing operation on the webpage.
Args:
one of the actions from ["Click", "Right Click", "Type", "Search", "Hover", 'Scroll Up "Scroll Down", "Press Enter", "Switch Tab", "Select Dropdown Option', "Wait"].
:param action:
:param argument: optional. Only for "Type", "Search", 'Switch Page", and "Select Dropdown Option", indicating the content to type in, page number(start from 0) to switch, or key to press. "Search" action is equivalent to "Type" action plus "Enter" key press.
iparam element: optional. Only for "Click", "Right Click", "Type", "Search", "Select Dropdown Option", and "Hover". Should be specific element id in the html.
Returns:
None. The webpage will be updated after executing the action.
def exit(message)):
""Ending the browsing process if the assistant think has fulfilled the goal.
Args:
:param message: optional. If user's instruction is a question, return assistant's answer in message based on the browsing content.
Returns: None.
def go backward(:
Go back the previous page.
def go forward():
Go forward the page.
Here some examples:
Element: the 'REPORIS' section the left sidebar
dofaction="Clie "Click", clement='7')
Element: the 'Period' dropdown, middle center
dolaction-"S.ee "Select Dropdown Option", , angument-'Monttee clement-"20
[part of all examples in the used prompt]
REMEMBER:
only "ONE-LINT-OF-CODDE" at time
Don't generate operation element that you do not see in the screenshot.
Use "# Element" to describe the element you choose in the html.
Use "# Note" to record information useful answer the instruction if needed.
1f you find yourself fallen into some sort of loop, try to use another method or change your action.
If you think page is still loading or still playing animation and you want wait while, use "Wait" action. action.
You are acting in real world, try your best not to reject user's demand. Solve all the problem you encounter.
If you think you didn't get expected webpage, you should try using more precise and locative description of the element.
You should "NEVER" try to use the browser's address bar at the top of the page to navigate. Your answer shouldn't i be in cude snippet format. Just write the function name and its arguments. If you use do function to perform "Click", 'Right Click", "Type", "Search", "Select Dropdown Option", and "Hover", the param element must not be None.
User Prompt:
Task Instruction: (instruction]
Action History: (action historyl
The Current HTML: [html of last state]
Figure 19: The simple prompt employed in baselines.
27

---

Published as a conference paper at ICLR 2025
User Prompt:
You are task filtering expert, you need to determine whether a given task is feasible or not. These tasks are primarily distributed across the following five platforms: MAP (OpenStreetMap)) Reddit, GitLab, CMS (online store content management system), and OSS (OneStopShop). You need to make judgments based the following criteria:
1. tasks in MAP:
Tasks that supported by MAP itself are feasible unless they involve the following goals, which deemed infeasible:
Viewing traffic flow, accidents, and road closure information.
Checking the weather conditions specific location.
Marking and saving favorite locations, such home, work, travel destinations.
Sharing real-time location.
Making reservations and bookings, such restaurant reservations hotel bookings.
Flight and train inquiries.
Viewing event locations and activitics, such concerts, exhibitions, and their details.
2. For tasks in Reddit: Tasks supported by Reddit is feasible.
3. For tasks in GitLab: Tasks supported by GitLab is feasible.
4. For tasks in CMS:
Tasks that are supported by CMS itself are feasible unless they involve the following goals, which are deemed infeasible:
Sending order information to customers via email.
Automatically generating e-invoices for customers.
Handling customer returns or refund requests.
Supporting profile updates.
Recommending products based on customer behavior.
3. For tasks in OSS:
Tasks that are supported by OSS itself are feasible unless they involve the following goals, which are deemed infeasible:
Filtering out discounted products.
Modifying the delivery address for a product.
Adding payment information such as credit cards, e-wallets, or bank transfers.
Displaying order status (c.g., pending payment, to be shipped, in transit, completed).
Providing order tracking functionality to view real-time logistics.
Offering online customer service to answer questions.
Supporting after-sales service requests, such as refunds or repairs.
You should evaluate the following tasks based on these rules and respond in the following format: Reason Answer (where the answer "[YES]" or "[NO]").
The website of the task is: (web)
The task (task)
Figure 20: The prompt used to filter instructions.
28

---

Published as a conference paper at ICLR 2025
lle Nanc" Jstall
Tatro the Thurhhos Dant
for orders with dittt
for the oedens with highest
Complesed that have the lnose transaciio values between April
leassay inn valuos that crecurscllly pending placed betwern March
Pundhase dealls Iox the voless with the higghst Irinsactun valuee that cummntly panding, placed bewwen March 1,
and May 30, 2012
Aprl 50
compune the legie Inansaction
Ageil 30, 2015.
values of ceders be oetwree March
Fath Order Quanity Tran the
2002, April 2005, those
report on bacberling products between Dcuiber Isi and Blst, year
cucrenily parding and that
Acquit Orider Quantiy from bec-eeling bees report between lanuary 1, 2022 and Janusey 31, with yoarly interval
Feich "Oneer
report besk willing products
betwoen Odtsber 2e and Sivt your
Phase
Phase
Phase 3
Phase 4
Figure 21: Examples of instructions generated in different phases under self-evolving curriculum learning.
F QUALITATIVE EXAMPLES
We list example of WEBRL each of the five sites in WebArena.
Which bustomer has completed the fiftt most number of orders in the entire hortey?
caseeaoorrig ASSIE MeriIarr Kioen NurOer' A dcjenboant demmmen
scwer oight Mesaad
Monter "Toa" momere comerrtiryy
Ar.
ISLK
aouulee
redt te Sian
P+.
P1
Figure 22: CMS Example.
29

---

Published as a conference paper at ICLR 2025
access to mr repo gimmeethat space, usernames 1111111111000
Figure 23: Gitlab Example.
Compane difirrence time for walking and driving roule from Randyland to Camegie Melion Universily
<1oherynaat COTTA00 esnera
Camnie
Dinam TW 0.
HeN VEEPOC COL THE C33
CCMII The So
Figure 24: MAP Example.
30

---

Published as a conference paper at ICLR 2025
count comments recevved downvotee than upvotes the use" who mode post Showerthouggg gitt
Figure 25: Reddit Example.
List products from living room fu furtnilura category by descending price
Conerct the Heme Noche Criplty th come
Comercin IN Fancur ConpTY
I Dennerr Kiing Foam mare TaNpO0
Becueel
Urw Stop Mather
Cha blop Nariet
Une 2000 Marlel
Lvnn Foors Furboure
Room
Figure 26: OSS Example.
31


Reviews:
[{"review_id": "qIyhYL7cIM", "reviewer": "Reviewer_8quU", "summary": "This paper proposes WebRL: a framework for training agents that can navigate the web using reinforcement learning. They apply their framework to train Llama-3 8B and Llama-3 70B agents that navigate the web, with strong performance on WebArena-Lite.\n\nThe overall process for training the agents is:\n\n1. Run the agent on an initial set of instructions from the WebArena-Lite training set, and score them with the WebArena-Lite reward function. Initialize the replay buffer with successful trajectories. Initialize the failure set with instructions from the failing trajectories. Train an outcome-supervised generative reward model (ORM) using these trajectories to predict whether a task has been successfully completed. \n2. Generate 500 new instructions using GPT-4o, prompting it to create instructions similar to the ones from the failure set, but more diverse.\n3. Filter the generated instructions. First filter them using the trained critic (value function), keeping instructions which the critic predicts are difficult but possible (between 0.05 and 0.75). Then filter through manual human review, to eliminate bad instructions.\n4. Run the agent on the instructions to generate a set of trajectories to generate a training set. Extend the training set with actions from the replay buffer whose perplexity is between 1/0.95 and 1/0.5.\n5. Update the agent policy on these trajectories using an RL algorithm that combines policy gradients, advantage estimation, and replay buffers.\n6. Add successful trajectories to the replay buffer, and add instructions from failing trajectories to the failure set.\n7. Go back to step 2 to begin a new phase.\n\nThe authors compare against a variety of baselines, including imitation learning and DigiRL, as well as a variety of ablations of their method.", "questions": "* Why do you develop a new RL approach, rather than applying PPO (or a similar algorithm) to the KL-regularized reward?\n* Why are the success rates in Figure 5 so low? Shouldn’t they be around 40% on average for WebRL, to be consistent with Table 1? Perhaps the denominator in Figure 5 was accidentally set to “all tasks” rather than tasks at the appropriate complexity level.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Training LLMs to navigate the web is an important area of research\n* The results in the paper appear to be quite strong.\n* The experimental results involve a good variety of baselines and ablations.", "weaknesses": "## RL algorithm\n\nThere are a few weaknesses with the RL approach (Section 2.1):\n\n* It’s not clear to me why the authors derive a new RL algorithm. I don’t see what benefits this would have over the approach of using PPO with the KL-regularized reward (as in [Ziegler et al](https://arxiv.org/abs/1909.08593)). In fact, I believe that the authors’ algorithm is nearly equivalent to REINFORCE with a value function baseline – the only difference I see is that in the authors’ algorithm, the gradient is multiplied by a factor of $\\beta$, which seems undesirable to me (ideally the scale of the gradient wouldn’t be sensitive to $\\beta$).\n* In addition, this new algorithm is an on-policy algorithm – but it is used alongside a replay buffer, which is naturally off-policy. Presumably an off-policy correction term would be needed.\n* Also, the math used to derive the new algorithm is flawed. Specifically, to derive Equation (5) from Equation (4), the authors rely on the result that the optimal policy in an RL environment is unaffected by potential shaping. However, this result does not hold in the KL-regularized RL setting, so this step is invalid.\n\n## Presentation\n\nI found the paper somewhat hard to understand – it was hard to tell what the various components were and how they fit together. Figure 1 is quite helpful in clarifying this with an architecture diagram, but I would recommend that the authors also include an algorithm description, similarly to the outline provided in my summary above. (Please do also let me know if my summary is wrong.)\n\nIt is somewhat misleading to say that the task distribution is “self-evolving” when it involves a manual step (L261):\n\n> Then, we manually check the instructions to eliminate instructions that are clearly unreasonable or impossible to accomplish.\n\n## Other\n\n**Single benchmark.** Since the authors only evaluate on a single task (WebArena-Lite) it is hard to tell which aspects of their setup are specific to WebArena and which are generalizable to any LLM control environment. Given the importance of browser control, and the diversity of tasks within WebArena, this is only a minor weakness. However, it could be addressed by applying the WebRL recipe to similar settings, such as Android (in which DigiRL was evaluated).\n\n**Task complexity results.** On lines 392-396 the authors say:\n\n> Our results show that WEBRL performs well across different complexity levels, particularly excelling in more complex instructions. In contrast, while DigiRL uses online learning, it struggles with higher complexity due to its focus on a fixed set of tasks, limiting its adaptability.\n\nHowever, this doesn’t seem to be supported by Figure 5, where it appears that both WebRL and DigiRL see significantly lower task success rates at higher complexity levels.\n\n**KL ablation.** The authors say they consider a version of WebRL without the KL regularization – however, they say that this involves running SFT. This seems like the wrong way to do it – instead, the policy should be trained with an RL algorithm, but without the KL regularization. This would produce a similar gradient as SFT, except that it would be weighted by the advantage function. The baseline reported in the paper is also interesting, though it should be called “WebRL-SFT” or something similar rather than “WebRL w/out KL”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcXrIeKVMk", "reviewer": "Reviewer_33ZD", "summary": "Authors present WebRL, a new framework to fine-tune LLMs for interactive web tasks with sparse rewards. WebRL leverages a curriculum learning setup to perform online actor-critic RL over generated task sets of increasing difficulty. The completion of these generated tasks is assessed autonomously using a learned reward model. Experimental studies are conducted on WebArena-Lite. There, authors showcase the performance advantages of their approach wrt prompt-based method and learning-based methods such as imitation learning or reinforcement learning.", "questions": "See weaknesses and the question about Digi-RL.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper tackles a very important topic.\nIt is well written, I enjoyed reading it.\nAuthors showcase an impressive performance gap over considered baselines.\nAuthors present an in-depth analysis of their approach, which is valuable", "weaknesses": "### Comparison to no-curriculum RL\n\nThe no-curriculum reinforcement learning baseline considered in this paper is Digi-RL. It is in my opinion the most important baseline. It would be good to better understand the differences between Digi-RL and WebRL. Is WebRL  == \"Digi-RL with a Curriculum Learning (CL) setup, a KL loss and a replay buffer\" ? If no, what are the differences ? What I mean behind this question is that ideally to propose a new CL method it is better to plug the CL method on a well known learner, rather than proposing a new learner AND a curriculum learning method.\n\n### Reliance on human-filtering for curriculum steps\n\nAn important weakness is that the curriculum learning process is not autonomous: generated new tasks are ultimately curated by a human. It does not mean the approach is not interesting, but it might be good to emphasize a bit more this limitation, which is only mentioned on line l.261\n\n### Decision\n\nDespite these issues I believe the paper is interesting enough to be accepted. I will go for a weak accept and look forward to the discussion to update my score.\n\n\n### minor/typos\n\nBe careful about citing journals/conferences and not arxiv, i.e. \"Agentbench: Evaluating llms as agents\" was published at ICLR 2024. Many arxiv citations in the current version\n\n\"(Rawles et al., 2024;Yang et al., 2023; Xie et al., 2024).\" --> always cite from oldest to newest\n\nl.162 \"progressive\" --> progressively\n\nl257 \"Detailed prompts are provided in the Appendix § A.3.\" --> Appendix A.3 is empty of text, although I managed to find the figures. Make sure to reference and briefly discuss each figure in the appendix text.0", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7JFObnaEkY", "reviewer": "Reviewer_gjpa", "summary": "This paper proposed WebRL, an online RL framework to train LLM-based web agents. It applies a curriculum learning for agents’ continual improvement with a trained outcome reward model. It applies a KL divergence to prevent the actor from deviating too much from the previous policy and a replay buffer to prevent catastrophic forgetting. Empirical studies show that WebRL outperformed baseline methods on WebArena-Lite benchmark and also has the ability to scale up.", "questions": "1. What is the performance of WebRL without curriculum learning and DigiRL with curriculum learning?\n2. In lines 77-79 “those in WebArena are typically of long horizons, with oracle solutions averaging 10 steps. This characteristic introduces substantial sparsity in the available signals during online exploration”. How can 10-step-long tasks be counted as long horizons in decision-making tasks? \n3. What is the clear definition of phase? \n4. Could you provide more analysis and insight into how RL mitigates the issue in lines 365-366?\n5. Why discount factor is not applied in the value function?\n6. In Lines 243-244, how to calculate the perplexity of the action?\n7. In lines 304-305, where do the data come from?\n8. What is the advantage of the loss shown in equation (6) compared to the standard loss function with KL?\n9. How is the generalizbility of WebRL. Can the trained agent be deployed to other tasks or even to the real PC?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. WebRL is an online RL framework, which greatly saves the need for dataset preparation in SFT.\n2. WebRL outperformed baseline methods on WebArena-Lite benchmark and also has the ability to scale up.", "weaknesses": "1. The authors claimed that WebRL is a self-evolving framework, mainly due to its curriculum learning module. However, the generated instructions need to be checked manually to eliminate unreasonable or impossible instructions. This manual check greatly weakens the term, self-evolving, which is one of the key contributions in this paper. It is also unknown to what extent this manual check impacts the overall training process.\n\n2. The experimental results are not conducted fairly. All the RL-based methods, except for WebRL are only trained on the generated 500 instructions in the first phase, whereas WebRL is trained iteratively on multiple phases. Training curves (steps VS success rate) and error bars are also not provided in the paper. The RL-based methods need to report the results with several seeds.\n\n3. The ablation study is not sufficient. In lines 342-346, the authors contribute the limitation of DigiRL to the lack of self-evolving curriculum learning, however, in the ablation study, the authors did not ablate the curriculum learning.\n\n4. Limited training details are reported in the paper and appendix, such as the hyperparameter for the experiments, observation and action space, how to train the ORM, how to use the replay buffer, etc. I am highly concerned with the reproducibility of this work.\n\n5. The writing needs improving, which is confusing and difficult to understand, especially in the introduction of ORM and curriculum learning, where curriculum learning should be introduced first in the second section. Due to the lack of examples, it is difficult to understand the difference of the generated instruction and the original tasks in the benchmark and the necessity of the ORM model.\n\n6. A number of RL for LLMs works are not mentioned in the related works, which are not limited to GLAM[1], TWOSOME[2], RL4VLM[3] and ArCHer[4].\n\n[1] Carta, Thomas, et al. \"Grounding large language models in interactive environments with online reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Tan, Weihao, et al. \"True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning.\" The Twelfth International Conference on Learning Representations.\n\n[3] Zhai, Yuexiang, et al. \"Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning.\" arXiv preprint arXiv:2405.10292 (2024).\n\n[4] Zhou, Yifei, et al. \"Archer: Training language model agents via hierarchical multi-turn rl.\" arXiv preprint arXiv:2402.19446 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)