How to Train Your LLM Web Agent: A Statistical Diagnosis
Authors: Dheeraj Vattikonda
Santhoshi Ravichandran
Emiliano Penaloza
Hadi Nekoei
Thibault Le Sellier de Chezelles
Megh Thakkar
Nicolas Gontier
Miguel Muñoz-Mármol
Sahar Omidi Shayegan
Stefania Raimondo
Xue Liu
Alexandre Drouin
Alexandre Piché
Alexandre Lacoste
Massimo Caccia
Venue: NeurIPS 2025 poster ()
Keywords: LLM agents
interactive tasks
MDP
reinforcement learning
reproducibility
compute allocation
generalization
statistical analysis
hyperparameters
curriculum learning.
Primary area: deep_learning
TL;DR: We provide a statistically rigorous guidelines for training interactive, multi-step LLM agents, exploring optimal compute allocation, generalization, and hyperparameter settings.

Abstract: Large language model (LLM) agents for web interfaces have advanced rapidly, yet open-source systems still lag behind proprietary agents. Bridging this gap is key to enabling customizable, efficient, and privacy-preserving agents. Two challenges hinder progress: the reproducibility issues in RL and LLM agent training, where results often depend on sensitive factors like seeds and decoding parameters, and the focus of prior work on single-step tasks, overlooking the complexities of web-based, multi-step decision-making.

We address these gaps by providing a statistically driven study of training LLM agents for web tasks. Our two-stage pipeline combines imitation learning from a Llama 3.3 70B teacher with on-policy fine-tuning via Group Relative Policy Optimization (GRPO) on a Llama 3.1 8B student. Through 240 configuration sweeps and rigorous bootstrapping, we chart the first compute allocation curve for open-source LLM web agents. Our findings show that dedicating one-third of compute to teacher traces and the rest to RL improves MiniWoB++ success by 6 points and closes 60\% of the gap to GPT-4o on WorkArena, while cutting GPU costs by 45\%. We introduce a principled hyperparameter sensitivity analysis, offering actionable guidelines for robust and cost-effective agent training.

How to Train Your LLM Web Agent: A Statistical Diagnosis
Dheeraj Vattikonda*.122.
Santhoshi Ravichandran Emiliano Penaloza
Hadi Nekoei1-2,6
Megh Thakkar*
Thibault Le Sellier de Chezelles Nicolas Gontier!
Miguel Muñoz-Mármol"
Sahar Omidi Shayegan 1.2.5 Stefania Raimondo
Xue Liu2,5
Alexandre Drouin Laurent Charlin Alexandre Piché
Alexandre Lacoste Massimo Caccia*.
Abstract
LLM-based web agents have recently made significant progress, but much of it has occurred in closed-source systems, widening the gap with open-source alternatives. Progress has been held back by two key challenges, first, narrow focus single- step tasks that overlooks the complexity of multi-step web interactions, and second, the high compute costs required to post-train LLM-based web agents. To address this, we present the first statistically grounded study on compute allocation for LLM web-agent post-training. Our approach uses two-stage pipeline, training Llama 3.1 8B student to imitate a Llama 3.3 70B teacher via SFT, followed by on- on-policy reinforcement learning. We find this process highly sensitive hyperparameter choices in setting where exhaustive sweeps are impractical. To spare others from expensive trial-and-error, to estimate effective hyperparameters. Our results show that combining SFT with on-policy RL consistently outperforms either approach alone on both WorkArena and MiniWob++ Further, this strategy only requires 55% of the compute to match the peak of pure SFT on MiniWob++, pushing the cempute-performance Pareto frontier and is the only strategy that can close the gap with closed-source models.
sample 1,370 configurations and use bootstrapping
Held-out Coals
Held-out Tasks
0.65
0.60
0.60
0.55
0.55
Succes Sate
8.50
0.45
0.45
0.40
0.40
0.35
0.30
10e18
15cl8
20e18
25c18
10c18 15e38
20e18
25e18
Compute (RLOPS)
Computte (PLOPs)
00. (9M)
OQ 7tb (Teacher)
10
branching Indes
Figure 1: Compute-performance frontier on MiniWoB++ (results averaged over two seeds). The blue curve shows pure SFT on teacher demonstrations. Warm-colored curves represent hybrid runs that branch off from SFT checkpoints and continue training with RL. Early transitions to RL push the Pareto frontier achieving higher success rates for the same compute and is the only approach able to achieve over 30% improvement on both held-out goals (left) and held-out tasks (right) closing the gap between open and closed-soure -ouucee models. See fig. for the corresponding plot with Qwen2.5 7B.
'Equal contribution "ServiceNow Research "Mila-Quebar Institute HHEC Montréal "MeGill University Unniveristé de Montréal
PPoytechnique Montréal
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language model (LLM) agents for web interfaces have advanced rapidly, but open-source systems still trail proprietary ones. Bridging this gap would allow organizations to train smaller, cost-efficient agents tailored to their needs while maintaining data privacy. Yet, despite impressive progress of open-source models in domains like math and code generation, advances in training web-capable LLM agents remain limited by mainly by the lack of attention to multi-turn, long-horizon tasks and the high cost and low reproducibility of current training pipelines.
Most research centers on single-step tasks like code or math domains with rapid feedback and simplified credit assignment which fall short of real- world web environments requiring sequential decisions and long-horizon planning. Recent benchmarks like WebArena [29]. WorkArena [TO], OSWorld (26). and The Agent Company [27] have exposed how brittle current methods become under delayed rewards, sparse feedback, and compounding errors. Addressing these settings demands training pipelines area directly tackle.
not just better agents, but reproducible, compute-
However, building such pipelines is nontrivial. Modern LLM post-training often involves a combina- tion of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), with performance sensitive to large number of interacting hyperparameters [T3]. Single-seed results are noisy and misleading, but exhaustive tuning and multi-seed sweeps (e.g. (2) remain infeasible for most labs due to the cost of LLM training. This makes it all the more critical to design pipelines that are not only effective but statistically robust and accessible within realistic compute budgets.
In this paper, we tackle these gaps by providing statistically driven diagnosis of training LLM agents for web- based, multi-step tasks. Specifically, we study how to allocate compute between expensive, high-quality off-policy demonstrations from a large teacher model and cheaper on on-policy rollouts from smaller student model. We analyze this tradeoff across two levels of generalization, held-out goals being tasks encountered during training but with novel goals and held-out tasks, which are entirely unseen during training.
To study this prrpuute-prformance trade-off we use two-stage training pipeline. First, a LLaMA 3.3 70B teacher model generates K successful trajectories to warm-start a smaller LLaMA 3.1 8B student via SFT. We then branch of various SFT checkpoints where training continues with an on-policy RL phase using Group Relative Policy Optimization (GRPO) [9] Our central objective is to determine the optimal compute allocation and hyperparameter mix for training web-based LLM agents. To this end, we run 1.370 training configurations, varying key hyperparameters and branching points between SFT and RL. We then apply bootstrap-based analysis to estimate the impact of different hyper-parameters on downstream performance and how they vary across branching SFT checkpoints. This data-driven approach enables us to identify important considerations to get the most out of each run. We use this method to show the optimal mix between SFT and RL in the MiniWob++ environment, achieving better task accuracy at a significantly lower cost, Additionally, we provide concrete recommendations compute allocations between SFT and RL on the more demanding WorkArena environment.
Putting things together, we show how our study yields several actionable insights. First, branching into RL early, but not immediately after SFT leads to better outcomes. This hybrid strategy consistently outperforms pure SFT and pure RL, and reaches the maximal performance of pure SFT while requiring only 55% of the compute, effectively pushing the compute-performance Pareto frontier. is also the only strategy that can close the gap with closed-source models. Second, curriculum learning and error log feedback help the less SFT warmup has been applied but can become counterproductive thereafter. Third, in GRPO, applying zero-advantage filtering consistently improve performance, while dividing the advantage with the standard deviation and wether to use an importance ratio are dependent on the amount of SFT warmup done. Fourth, decoding temperature is consistently critical, while learning rate and discount rate must also be carefully tuned.
These findings are significant for two reasons. First, they give smaller research groups reproducible, budget-aware playbook for pushing open LLM agents closer to state-of-the-ar without scaling model size. Second, they address slice of the broader reproducibility concerns recently highlighted in the RI community [3] [3, offering a template for rigorous, grruundd hyperparameter tuning and compute allocation in LLM agent training.
2

---

2
Background
This section consolidates the algorithmic ingredients used throughout the paper: (i) the MDP formulation of web-based language agents, (ii) SFT expert traces, (iii) GRPO for RL, and (iv) curriculum and normalization techniques that stabilize training.
2.1 Language Agents MDPs
We model each task as a Markov Decision Process (MDP) M = (S,A,,,,, 7). A state S, E S textual context-in
a prompt and action de A is textual response generated by the agent. Each action a consists of sequence of tokens 01, sampled autoregressively from the policy Ty (ar(s;) parameterized by an LLM sampled with temperature PLLM The environment then returns a scalar reward rN E {-1, 1} indicating task failure/success. In our setting we assume the environment dynamics are p(S (4) E P. We optimize the policy xy maximize the expected discounted return:
T
( = Er
Σn t=0
r'r
Here, y E [0, 1] is the discount rate, which controls the agent's preference for immediate versus future rewards and r = (So, ao, To. ........ ST) refers to trajectory sampled from the policy,
2.2 Off-policy boot-strapping via SFT
We first imitate stronger expert policy Te by minimizing the cross-entropy loss
LSFT (0) -ET
log (a 8t)
SFT offers a high-quality, low-variance gradient but is inherently off-policy which can lead to poor generalization [6].
2.3 Rejection Fine- Tuning
Similar to SFT, Rejection Fine- Fine-Tuning (RFT) optimizes the policy using successful trajectories, but in this case bootstrapped of the training policy rather than a larger expert
CarT(0) (0) -BT
I (r-=13
Σ log (a: st)
2.4 On-Policy Improvement with Multi-Turn GRPO
After the initial SFT warmup phase, we continue training using on-policy RL using GRPO as the optimization algorithm. Like REINFORCE [11. GRPO maximizes the expected return using policy gradient objective. However, GRPO introduces additional structure by leveraging per-goal advantage normalization and importance weighting, wherein each trajectory is associated with specific goal in our context, referring to the seed of the target task. For a given goal the croup-normalized advantage function is:
Tt. mean
Ar,g
std (..g)
where Rr,, (rr Gr, is the of rewards across the G goals at timestamp f. In addition to this similar to proximal policy optimization (PPO) [2] an importance-ratio is often applied to the GRPO objective:
n (ut 18
Tt.9 Ta. (az 8t)

---

where is the current policy and The is the behavior policy used to collect trajectories. Finally, hupped-minimune is also applied stabilize the training process. Putting things together the GRPO objective for our multi-turn setting is:
MT-GGRPO aa
min Clinn Le),))
Traditionally the GRPO objective includes KL penalty between the optimizing and reference policies which do not use early experiments showed they did not improve performance, slowed down training and required additional compute budget.
Zero-advantage filtering. Tokens with Ar.g = 0 contribute no learning signal yet still consume memory. Dropping them yields a constant effective batch size and modestly accelerates training [28]
2.5 Curriculum through Variance-Aware Boltzmann Sampling
To promote steady learning progress, we design a curriculum that prioritizes challenging tasks, neither trivial nor too difficult [22]. Specifically, select tasks according Boltzmann distribution centered around a target return Marget which specifies the desired performance threshold, encouraging focus partially mastered tasks, with temperature parameter PCur controlling the sharpness of the distribution, with lower values concentrating probability mass tightly around Pearget.
This sampling mechanism dynamically adapts the training distribution, concentrating learning tasks where the agent neither already proficient nor entirely unskilled. As a result, the agent avoids premature convergence easy tasks and prevents wasted effort on tasks far beyond its current capabilities.
Methodology
Our training pipeline consists of two sequential stages SFT followed by RL framed as resource allocation problem. We also describe our hyperparameter sweep strategy and statistical analysis protocol that consolidates hundreds of runs into reliable conclusions. We evaluate our recipe along two axes: compute cost, measured in FLOPss (using the formula from ( and model performance, assessed on both unseen training goals and held-out testing tasks.
Stage Supervised Fine-Tuning (SFT). We begin by generating Ne expert trajectories using large teacher model. Only successful trajectories are retained after filtering, and the corresponding (s, a) pairs, along with train-of-thougeee annotations, form the SFT dataset. Note that computing the cost of the SFT dataset includes both successful and discarded unsuccessful trajectories.
We then train a smaller student model for TSFT gradient steps. To explore the trade-off between SFT and RI., we branch off B times at fixed intervals along the SFT trajectory, yielding checkpoints at timesteps to J0, T Each checkpoint corresponds to student policy me(ts)) which initializes distinct RL phase. The total compute used up to each branching point, Fser(ts), includes both teacher inference and student training FLOPs accumulated over tú steps.
Stage 2 - RL Fine-Tuning, Each policy TT(),) is further trained using GRPO for TRL. steps. The compute cost of this phase, (TRL)) includes both the FLOPs required for data collection (online rollouts) and for student updates across all TRL steps. The total FLOPs for full training run starting from (fa)) is computed as:
FLOPs((t) = Fsrr(tt) + F
(5)
By varying to throughout SFT, we assess how shifting compute between expert-driven supervision and on-policy learning impacts final performance. This setup highlights the trade-off between the high compute cost of expert supervision and the lower-cost, but noisier, nature of on-policy learning.
3.1 Estimating the Uncertainty of the Hyperparameter Selection Process
Across the different SFT checkpoints, we sample 1,370 distinet configurations with ten varying hyperparameters (see Appendix L for details). Our objective is to study the effect of various

---

Algorithm Bootstrap Estimation of Hyperparameter Importance
Require: Set of training runs Resulta ts, evaluation metric column M, hyperparameter of interest H,
number of bootstrap iterations B
1: W = dict() )
Win counts per HP value Score samples per HP value
2: S def defaultdict (List)
(List)
3: Hvals
unique(Resulte)
>> Distinct values of the hyperparameter
4: weights = { h; //cooun for each h E H
5: for b = 1 to B do
6: sample, = [Rsumpl weights)
7: h" = argmax,, sample,
Get the winning HP value for sample,
8: W[h"] +=
9:
for each h in Hal do
10:
S[h]+ max sample, (smmlle, [smple = M][M
11: end for
12: end for
13: for each h in vals do
14: Compute 95% confidence interval from S(h]
15: end for
16: return W, S with confidence intervals
hy perpaaaraeeerss (HP) on the downstream success rate of our trained agents. This comes with two important considerations. First, if we change the value of, e.g., the batch size, and we want to know a bigger batch size better, the learning rate and other parameters need to be readjusted close to their optimal configuration (under fixed budget). Secondly, to account for noise, would need to restart the same experiment several times to avoid spurious conclusions. In practice, this is out of reach. For more computationally friendly approach, resort to bootstrapping the collection of trials over different HP configurations. The boostrap algorithm has many desriable properties such as being consistent and unbiased estimator for the sampling distribution of any data statistic without requiring parametric assumptions (see Appendix for details), thus allowing us to reliably obtain robust estimates of both the expected value and variance of the success rate given by any HP.
Bootstrapping the hyperparameter selection process. From the full set of 1,370 training runs, we perform bootstrap resampling by drawing individual runs (with replacement). For each resample, we identify the best-performing configuration and repeat this process 1,000 times. We also compute the fraction of times each hyperparameter value "wins", which serves as an estimated probability that it belongs to the global optimum. This procedure serves two purposes: to estimate the maximum relative improvement a specific spper-parameter provides while accounting for variation other parameters and to offer uncertainty estimates in the form of win-rate distributions between different configurations. addition, better understand how optimal hyper-parameters may change depending on the amount of SFT warmup we apply this analysis across various SFT checkpoints.
Balancing unequal coverage. Due to random search, some HP values were explored more than others. biasing the winner toward the larger groups. To correct for this, each is sampled with probability & 1/ /groupp size, approximating an equal compute budget for every HP value.
We provide detailed explanation of this procedure in Algorithm T describe how use this procedure to obtain the results reported Figure
Additionally in Appendix ! we
Experimental Setup
In this section, we describe the experimental setup used to validate our findings. We detail the models, benchmarks, action spaces, training framework, compute infrastructure, and evaluation protocols, ensuring that all components are aligned with the goals of studying compute-efficie reproducible training for LLM web agents.
Models. evaluate our approach using two teacher models, Llama 3.3 70B and Qwen 2.5 72B to generate SET traces, with Llama 3.1 8B and Qwen 2.5 7B acting the student model for fine-tuning.
5

---

All models operate with 16k token context window to handle the complexity of web-based tasks. For the LLama models we report results using GRPO, while for Qwen we report results using RFT as we found it more stable during training.
Benchmarks. Our experiments focus two benchmarks. The first is MiniWoB++, suite of 30 medium-horizon web interaction tasks, where we observe that optimal policies typically complete tasks in 2 5 steps. The second is WorkArena [10], more challenging benchmark of 33 enterprise knowledge-work tasks, where we observe optimal policies generally complete tasks in 3 to 10 steps. These benchmarks provide a representative spectrum of sequential decision-making challenges faced by interactive LLM agents. Both benchmarks are depicted in Figure
Observation & Action Spaces. MiniWoB++ provides raw HTML trees, whereas WorkArena leverages accessibility trees (AxTrees), which we truncate to 16k tokens to meet hardware constraints. The agent operates in dis- crete action space composed of high-level UI primitives: noop, fill (node, text), click (node),
SFT
SFT
LaMA 706
0.00
0.35
0.24
0.48
1.00
oreer welt
0.50
0.48
0.92
0.79
1.00
0.00
0.26
0.57
0.73
1.00
0.00
0.48
0.71
0.81
0.75
type (node, text), celicc_optionnee (node, option).
0.00
0.00
8.00
0.00
0:33
at-mens
0.00
0.06
0.19
0.00
0.25
scroll (node), and hover hover (node). This abstraction allows the agent interact ef- fectively with diverse web interfaces. All agents employ chain-of-thought prompting [25]. We also experiment with applying error log feedback, allowing the agent receive explicit error messages when it takes invalid actions (during both training and testing).
0.00
0.00
0.00
0:20
0.25
0.6
0.25
0.23
0.25
0.25
0.25
0.4
0.00
0.00
0.31
0.47
0.51
0.80
centesser
0.07
0.0
0.60
0.29
0.29
0.44
0.31
0.57
ondeviaederskaare
0.00
0.05
0.36
inpenceaiie
0.00
02
6. 1.00
0.00
0.09
Training Framework. To manage the train- ing pipeline, we BROWSERGYM {8 for orchestrating Chronium-based web environ- ments and structuring the agent's action space, while AGENTLAB
Anarage
Epoch
Epoch
Epoch
Figure Per-task performance SFT and SFT+RL agents on WorkArena. The Llama 3.1 8B model is initially fine-tuned for 4 epochs on trajectories from teacher Ilama 3.3 70B model. Training then continues either with additional SFT with GRPO fine-tuning up to epoch 20. The teacher model's success rate is also shown.
[8] handles agent design. Model fine-tuning is conducted with TORCH- TUNE 23 utilizing Fully Sharded Data Paral- lelism (FSDP) enable scalable training across multiple GPUs Given the high memory de- mands of long sequenec processing, we apply activation offloading and gradient checkpointing techniques, achieving approximately 40% reduction in memory usage.
Evaluation Protocol. Both environments are stochastic, and we control for randomness during training by fixing the task seed. Each task-seed pair is referred to "goal". The evaluation protocol assesses generalization at two levels: (i) performance on held-out goals within the training tasks, and (ii) performance on entirely new, unseen held-out tasks. We consider both settings important as they capture different aspects of generalization held-out goals test reliability on trained tasks, while held-out tasks test the model's ability transfer skills learnt in training to entirely new situations.
In both settings we use the the average task success rate as the evaluation metric, which reflects the agent's ability to generalize beyond its training distribution. To reduce the impact of evaluation noise during model selection, we apply a rolling average with a window size of on the held-out goals, and select the checkpoint with the highest smoothed score.
For our reported runs (SFT, RL, SFT + RL) we report the average of the non-smoothed scores at the selected checkpoint, aggregated over four runs chosen through the described model selection procedure based on our random search. For all other models, we report average scores over 100 independent runs.

---

Table I: Comparison of our method with baselines the held-out train and test splits of +iniWoB++ and WorkArena. Here are the averaged standard errors computed separately for each of the two top seeds.
MiniWoB++
WorkArena
Model
Held-out Goals
Held-out Tasks
Held-out Goals
Held-out Tasks
Claude-3.5-Sonne
70.5
70.4
52.5
70.0
GPT-4o
65.7:22
64,3:44
42.1
55.7=59
GPT-4o-Mini
56.2+22
66. +44
27.1+29
28.6=55
10mma-3.-70B-Imstruuc
57.0=22
65.2
25.0+228
32.9=56
ol-Mini
69.7.12.1
66.11444
53,8
68 68.6155
Llama- 111-4-0--Inste
65,9 12.4
65.2
39.2.124
58.6125
Llama-3.3-700 Instruct (Teacher)
63.2
61.9
36.0.12
44.0
Llama-3.1-8B Instruct (Student)
29.5
36,4
8,3
4.2-+1.2
Llama-3.1-88 RL (Ours)
43.5
43.5
8.0
11.5
Llama-3 3.B SFT (Ours)
55.2
56,7.22
28 18.4
26.4
Jlama-3.1-8B SFT+RL (Ours)
67.2
63.3
35.4
28.8
Qwen-2.5 2.B-72B Instruct (Teacher)
61.0.134
59.0.334
33.3
27.0.22
Qwen-2.5-7B Instruct (Student)
32.8:333
37.0=34
5.2
10.0=38
Qwen-2.5-7B RI. (1.rr))))
52.5
53.0g33
4.5
7.0=18
Qwen-2.5-7B SFT (Ours)
57.0
56.5 H:35
25.0+443
21.0+4.1
Qwen-2.5-7B SFT+RL (Ours)
65.0+34
61.5
32.5
25.0
Compute Allocation Protocol. We track the total floating-poin operations (FLOPs) consumed during both SFT and GRPO phases, following the procedure in Section 3 and Appendix [D] For RL branching, we start from the SFT run that (i) attains the highest SFT performance and (ii) exhibits stable learning, allowing us sample checkpoints regular intervals and cleanly study the effect of SFT warm-up. Because this run sits in the top 10% of all SFT trials by mea--undertthe-traiing (AUC) the mean success rate across all checkpoints of the run, serving as a proxy for overall training efficiency, match that selection pressure for RL by averaging the top two seeds out of ten RL trials, thus approximating the 90th-percentile of the RL distribution and ensuring a fair compute-aware comparison between strategies.
5 Main Results and Compute Trade-Offs
In this section, we present our primary findings and analyze the trade-offs between SFT and RL in terms of both performance and compute efficiency.
Performance Overview. Tummariis the results on MiniWoB++ and WorkArena. Combin- ing SFT with RL consistently yields the best performance among student models. Pure SFT and pure RI. each fall short, with RL from scratch particularly struggling due to sparse rewards and unstable learning dynamics. These results highlight the complementary strengths of expert demonstrations and on-policy fine-tuning.
On MiniWoB++, this approach not only maximizes student performance but also matches the teacher and significantly closes the gap with proprietary models. In contrast, WorkArena remains more challenging: while SFT+RL improves over SFT alone, student performance still lags behind the teacher and proprietary models, suggesting that stronger supervision or more effective RL strategies may be needed for complex enterprise tasks.
We observe that agent performance eventually saturates for both SFT and SFT+RI, especially on more difficult tasks. Further analysis of this saturation behavior is provided at the end of this section. Notably, Wren'ss test set can be easier than its training set for certain agents, which adds nuance to the observed performance gaps and complicates straightforward interpretation of generalization performance.
Compute-PPrformanne Trade-Off. Our analysis focuses on the trade-off between costly but high-quality teacher demonstrations and cheaper, noisier on-policy rollouts. examine this, we branch RL fine-tuning from SFT checkpoints sampled at fixed intervals along the training trajectory.
7

---

Figure I illustrates the compute performance frontier on MiniWoB++, where runs that combine SET with RL (warm colors) consistently outperform pure SFT (blue), pushing the Pareto frontier forward. Early branching into RI. unlocks substantial gains in compute efficiency, delivering stronger performance lower compute budgets. For example, SFT+RL. reaches the maximal performance of pure SFT on the test set (achieved at approximately 11 exaFL.Ps with pure SFT) using only around 6 exaFLOPs translating to roughly 45% of compute saved (11 vs exaFLOPs).
Notably, this is the only strategy that closes the performance gap with closed-source models like GPT-4o. The trend is consistent across both held-out goals and held-out tasks: warm-stare RL yields higher performance than either SFT or RL alone, reinforcing the importance of blending expert supervision with on- on-policy learning. These findings underscore the need to balance sample efficiency from demonstrations with the compute efficiency of on-policy learning, and the results generalize to Qwen2.5 7B (see fig.
Task Performance Saturation and Analysis Despite extensive post-training, agent performance on the WorkArena benchmark plateaus at around 40% after just 9-10 epochs. This stagnation appears to stem from the intrinsic difficulty of certain tasks (e.g sorting and filtering tasks) that even frontier models struggles to solve (see Figure 2). A per-task breakdown shows that while both SFT and RL agents gradually close the performance gap with the teacher model, with RL achieving a slightly higher final success rate, a significant portion of tasks (14 out of 33) remain completely unsolved. These failures are attributed to either the limitations of the teacher model or the sparsity of reward signals, both of which hamper the learning process. On-policy RL exploration proves ineffective in overcoming these challenges due to the lack of foundational skills and informative feedback. These findings underscore the need for more effective methods to address complex tasks under sparse reward settings. Additional per-task performance results for WorkArena and Miniwob are provided in Appendix C
Shortcut-Seeking Agents and Task Integrity During extended training found that giving agents full administrative privileges invites creative but problematic shorteuts. For instance, in WorkArena's 's "Order hardware devices" task, rather than navigating through the catalogue as intended, the agent simply edits its homepage to add a direct link to the ordering form. While this hack fulfills the success condition, it violates the task's spirit and, worse, the modifications persist across sessions. Other agents entering the environment later inherit the altered UI, leading to failures unrelated to their own policies.
In general, agents exploited admin rights to create or delete elements, reshaping pages in ways that saved clicks but destabilised the environment. These findings argue for a tighter sandbox enough access for exploration and accomplishment, but safeguards against permanent, inst side-effects.
6 Ablation and Sensitivity Analysis
We simulate re-running hyperparameter configurations and selecting the best-performing ones using the method described in Section 3.1 This done across three checkpoints: the base LLaMA 3.1 8B Instruct model and two warm-started variants with an additional 2.5 x 18 and 7.6 x 10 FLOPs of supervised tne-tuning. respectively, to assess variations across compute budgets. We evaluate the held-out goals and held-out tasks performance of 10 hyper-parameters across 1,370 runs. We report final held-out task performance to verify generalization in Appendix Ginding no significant deviations between held-out tasks and held-out goals parameters.
Figure displays our findings, which we summarize as follows. Curriculum learning is beneficial when starting RL from scratch but becomes detrimental after warm- warm-starting, likely because warm- started models already perform well on easy tasks, and curriculum forces them overfocus on harder ones. Error log feedback helps when there's no SFT but otherwise does not, likely because SFT warmup removes many common errors made by weaker models. decoding temperature of 0.25 consistently yields the best results, striking balance between exploration and exploitation; lower values led to under-exploration and were discarded. Grouped--rllate advantage helps only after SFT, while using raw rewards works better when starting directly from the Instruct model, possibly due to how advantage scaling interacts with the initial weights. Zero-advantage filtering improves training across most settings by ensuring batches focus informative updates. devviaiion
8

---

Instrcct
2.504EE FLOPA
PLOPS
2.3 POOPS
Fele
ride
Twe
Instruct
2.5e+la SIT FLOPs
7.60+ SFT FLOP
Instreet
2.5 511 FLOPs
+ 7. S FLON
200
Instruct
2.Sesi8 SIT FLOPs
7.fee28 SFT
2.5e+18 FudPs
7 FLOPs
622
51
975
27%
Instruct
2.Jesla
7.dex1e SPT FiOds
2.5ex33 RUOPs
7.5 FLOPs
622
79%
10%
Fele
24
24:00
Figure 3: Bootstrap analysis (n = 1000 samples) of hyperparameter optimization across different SFT compute budgets on training held out tasks. Each subplot examines different hyperparameter, including increasing SFT compute: the base instruct model (left), +2.5x101 SFT FLOPs (middle), and +7.6e 10 SFT FLOPs (right). For each hyperparameter-coopuue combination, the top panel shows relative reward performance with error bars indicating 95% confidence intervals, while the bottom panel displays win rates representing the percentage of bootstrap iterations where each parameter value achieved maximum performance. Results demonstrate that optimal hyperparameter values shift model pre-training compute increases, suggesting that hyperparameter selection should be adapted to the computational budget allocated to SFT.
normalized advantage, as noted by (17. seems aid performance under less finetuned models and decrease in value the more finetuning is done. Importance ratio correction and trust region, though standard, also hurt models with little or no SFT, likely because conservative updates slow down learning. In contrast, for models that start from stronger SFT checkpoint, these mechanisms can help stabilize training and avoid catastrophic updates. For the learning rate, the larger value (1e-6) generally worked better. Effective batch size of 512 appears to be generally be safe and robust choice in all our experiments. Finally, regarding the discount rate, values above 0.9 work well for models with little or no SFT warmup. while heavily warm warm-started models benefit from a lower rate around 0.5 likely because encourages the agent to optimize more aggressively on tasks already handles well. A further hypothesis is that the large deviation in optimal hyperparameters between the base and warm-start models could be attributed to the significant entropy reduction introduced by SFT (see Appendix Hb.

---

7 Related Work
Best prectices in deep RL. Building the recognition of reproducibility challenges and unstable RL training of LLM agents, recent studies have proposed best practices for training LLM agents using RL methods. Dang and Ngo [7] recommend leveraging high quality data, balancing easy and hard problems, and controlling length generation with cosine reward. Yu et al. [28) promote higher clipping in the GRPO loss promote diversity and avoid entropy collapse, dynamic sampling to improve training efficiency and stability, token level gradients for long CoT sequences, and overlong reward shaping to reduce reward noise. Roux et al. [20] introduce tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics. The proposed method (TOPR) allows the handling of both positive and negative examples in fully offline setting. More generally, Hochlehnert et al. [3] emphasizes the need for greater methodological precision, particularly concerning decoding parameters, random seeds, prompt formatting. as well as the hardware and software frameworks, to guarantee transparent and thorough assessments of model performance. These practices are essential for developing robust and reproducible agents.
LLM Agents trained with RL multi-step environments. Recent advancements have sought to bridge the gap in training LLM agents for multi-step environments, with approaches like WebRL [9] and SWEET-RL (30 demonstrating significant progress. WebRL employs self-evolving curriculum to address the challenges of sparse feedback and task scarcity, successfully enhancing the performance of open LLMs in web-based tasks [19]. Similarly, SWEET-RL RL introduces hierarchical structure that enables effective credit assignment over multiple turns, improving policy learning and generalization in collaborative reasoning tasks (30). These studies collectively illustrate the necessity of adapting RL techniques to accommodate the complexities of multi-step interactions, paving the way for more capable and versatile LLM agents, Similar in spirit our work, 4 propose an empiral study on the inference cost of trained LLM web agents. An extended version of the related work is provided in
Appendix F
LLM agents trained with RL in multi-step environments. Recent work has begun closing the gap for LLM agents that must reason over multiple steps. WebRL [9] introduces a self-evolving curriculum that tackles sparse feedback and task scarcity, substantially boosting open-source agents on web tasks. SWEET-RL (30) adds a hierarchical credit-assignment scheme spanning many turns, improving both learning stability and generalization in collaborative reasoning. Together, these studies underscore the need to adapt RL techniques to the intricacies of horizon interaction, paving the way for more capable, versatile agents. Similar in spirit, 4 provide an empirical analysis of inference costs for trained L.LM web agents. An extended discussion of related work appears in Appendix H
8 Discussion
Limitations. Our focus is on providing a comprehensive perspective on training an LLM-based web agent, studying compute trade-off's, hyperparameter selection, and analyzing failure cases. With this in mind, our results are limited to English-language web interfaces and Llama 3 models in the 8B-70B parameter range, where larger models may alter trade-offs.
Regarding our statistical method, it does not account for the lack of coverage from the random search. A more exhaustive search could discover configurations that would change the conclusions drawn in this study. We note also that a significant portion of the reported uncertainty is due to epistemic uncertainty that could be reduced by evaluating more configurations.
Conclusion. We present statistically grounded study on training LLM web agents, analyzing the trade-off between SET and RL. We perform random sweep across 1,370 configurations identify optimal hyperparameter choices, finding interesting variation across compute budgets. Using the bootstrap prescribed hyper-parameters we show that branching into RL early, but not immediately, after SFT achieves better performance-compee compute and closing the gap with closed-source agents. Our findings offer a reproducible, budget- aware blueprint for advancing open-source LLM web agents in complex multi-step environments.
trade-offs, matching peak SFT with 45% less
10

---

References
[1] Dgxc ogg.apacchhttiii.seesee//e/////n
URL
[2] M. Abdin, S. Agarwal, A. Awadallah, V. Balachandran, H. Behl, L. Chen, G. de Rosa, S. Gu- nasekar, M. Javaheripi, N. Joshi, P. Kauffimann, Y. Lara, C. C. T. Mendes, A. Mitra, B. Nushi, D. Papailiopoulos, O. Saarikivi, S. Shah, V. Shrivastava, V. Vineet, Wu, Youseñ, and
G. Zheng. Phi-4-reasoning technical report, 2025. URL htttps://arxii> org/ httgg/bb/2500 21318
[3] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 2021.
[4] M. Andreux, B. B. Skuk, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, M. Brunel, P.-L.. Cedoz, A. Chassang, M. Chen, et al. Surfer-h meets holol: Cost-efficient web agent powered by open weights. arXiv preprint arXiv:2506.00 02865, 2025.
[5] I.. Boisvert, M. Thakkar, M. Gasse, M. Caccia, T. S. D. Chezelles, Q. Cappart, N. Chapados, A. Lacoste, and A. Drouin. Workarena++: Towards compositional planning and reasoning- based common knowledge work tasks, 2024. URL http:::ttt//ttt.tt.. http://aaxvvvvv// org/ / 05291
[6] Chu, Y. Zhai, J. Yang. S. Tong, S. Xie, D. Schuurmans, Q. V. Le, Levine, and Y. Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. URL https://arxiv. org/ http//bb//221...0. http:/vww..rg/ab//511.1111 17161
[7] Q-A. Dang and C. Ngo. Reinforcement learning for reasoning in small Ilms: What works and what doesn't. arXiv preprint arXiv:2503. 19, 2025. URL https://t org/ ab//aa////// /503 16219
[8] T. L.. S. de Chezelles, M. Gasse, A. Lacoste, M. Caccia, A. Drouin, L. Boisvert, M. Thakkar, Shayegan, I.. K. Jang, X. Lù, O. Yoran, D. Kong, F. F. Xu, S. Reddy, G. Neubig, Q Cappart, R. Salakhutdinov, and N. Chapados. The browsergym ccosystem for web agent research. Transactions on Machine Learning Research, 2025, ISSN 2835-8856. URL http::t///teeeeeeiiiiiiee
Marty, R. Assouel,
nww.feruunmt 1d=5298fKGvv3 Expert Certification.
[9] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, Guo, J. Li, Wang, Chen, J. Yuan, Qiu, J. Li,
J. L. Cai, J. Ni, J. Liang. J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang.
L. Zhang, L. Zhao, L. Wang. L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li,
M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang,
R. Pan, R. Wang. R. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, Yu,
S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng,
W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. I. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li. X. Jin, X. Shen, X. Chen, X. Sun, X. Wang. X. Song. X. Zhou, X. Wang. X. Shan, Y. K. Li. Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang. Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma. Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong,
Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li,
Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, 7. Zhu, Z. Liu, 7. Li, Z. Xie, Z. Song, 7. Pan, Z. Huang, 7. Xu, 7. Zhang, and Z. Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025. URL httpp://arxiv.com org/ abs/ 12948
[10] A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, L. Boisvert, M. Thakkar,
Q. Cappart, D. Vazquez, N. Chapados, and Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.
11

---

[11] B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1-26, 1979.
[12] B. Efron and Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, May 1994. ISBN 9780429246593. doi: 10.1201/99804222445599 URL http://dx doi. 1ttp///x..o.ooo////..//////////. 1201 9301/978042924659
[13] A. Hochlehnert, Bhatnagar, V. Udandarao, Albanie, A. Prabhu, and Bethge. sober look at progress in language model reasoning: Pitfalls and paths to reproducibility, 2025. URL
org/ hhtggabb/25044. 07086
[14) W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13-30 Mar. 1963. ISSN 1537-274X. doi: 10. 111.1/2211499991.1000000000 URL http://dxx. www.....rg/111110000000002666110
10500830
[15] J. Y. Koh and et al. Visualwebarena: Evaluating multimaal agents on realistic visually grounded web tasks. In ACL 2024, 2024. URL https:://ccaaathollloeee org/2024. 2024. acl- 50
[16] E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https: //apxii////xvv org/ abs 1atbb//80...888
[17] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding rl-zero-like training: A critical perspective, 2025. URL https: wwwp://w..r./////////a//o hatgg//bbs/550.... 2503 20783
[18] S. Murty and et al. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild. arXiv preprint arXiv:2410.02907, 2025. Utt/tttss///axxi org/
abs/ /a10.210..m 02907
[19] Z. Qi. X. Liu, I. L. Iong, H. Lai, X. Sun, J. Sun, X. Yang. Y. Yang. S. Yao, W. Xu, J. Tang, and Y. Dong. WebRL: Training L.M web agents via self-evolving online curriculum reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevile net.eeeweeeeeeeeeeeeeeeeeeeeeeee wttt/forummtt ?ddoon/ JEqv
[20] N. L. Roux, M. G. Bellemare, J. Lebensold, A. Bergeron, J. Greaves, A. Fréchette, C. Pelletier, E. Thibodeea-- Laufer, S. Toth, and S. Work. Tapered off- off-policy reinforce: Stable and efficient reinforcement learning for 1lms. arXiv preprint arXiv:2503. 14286, 2025. URL https://
arxiv. org/ abs/ 2503. 14286
[21] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'99, 99, page 1057-1063. Cambridge, MA. USA. 1999. MIT Press.
[22] M. Thakkar, T. Bolukbasi, S. Ganapathy, S. Vashishth, S. Chandar, and P. Talukdar. Self- influence guided data reweighting for language model pre-trainnng. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023, URL / 1ett
[23] torchtune maintainers and contributors. torchtune: Pytorch's finetuning library, Apr. 2024. URL https///gtthuub.coo/ppyttorootrt wwwphhhhhhtttctuuteeeeeo
[24] M. Wainwright. High- DingennDmmssi Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, Feb. 2019. ISBN 9781 108498098 doi: 10.1017797811008227777 URL http://dx. doi org/ 10.
[25] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https
//arxiv. org/ abs /2bs/220...0 11903
[26] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Liu, Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
12

---

[27] F. Xu, Y. Song, B. Li, Y. Tang. K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, Jang, Y. Xie, S. Zhou, and G. Neubig. Theagentcompany: Benchmarking I1m agents consequential real world tasks, 2024. URL htrl.https:/// hatps://aaxxiv.ooo www.awwwwww//b///////a. ab1/2412...2./. 14161
[28] Q. Yu, Zhang, R. Zhu, Y. Yuan, X. Zuo. Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen,
C. Wang. H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W.-Y. Ma, Y.-Q. Zhang, L. Yan,
M. Qiao, Y. Wu, and M. Wang. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503, /4476, 2025. URL htttp:://atiiiii org/abs/////ooooooooo...ooeeeebt hts/bb//25053 2200 14476
[29] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng. T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents. 2023, BtRp:ttps://aatt
arXiv preprint arXiv:2307.1388
org/ abs abs/2307 13854
[30] Y. Zhou, S. Jiang, Y. Tian, J. Weston, S. Levine, S. Sukhbaatar, and X. Li. Sweet-rl: Training multi-turn IIm agents on collaborative reasoning tasks, 2025. URL : org/abs/ abs/ 2503. 15478
Broader Impact
Web-based LLM agents have the potential revolutionize markets by enabling cost-efficient and effective workflows. Our work focuses on making these agents accessible across a range of compute budgets, empowering only industrial labs but also smaller research groups and individuals to train their own assistants. This approach promotes data privacy and reduces reliance on costly infrastructure.
Despite their promise, web-based agents face significant challenges that limit their broader adoption. Issues such reliability, vulnerability to adversarial attacks, and limited access to proprietary data remain key obstacles to realizing their full potential.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide thorough empirical evidence of our claims in Section and
Section
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in Section
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not have theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss in depth our training protocol and evaluation scheme. All results should be reprodueible on the reported environments.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Current code tied with infrastructure which is difficult to open source.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss this in detail for our experiments
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our paper provides evaluations over multiple runs and reports their deviations. We also significantly analyze the impact of various hyper- parram-pr configurations.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our paper focuses on compute budgets and thus extensively discussess these details
14

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::////orrrp...cooooaa///haaa
Answer: [Yes]
Justification: We abide by the code of ethics
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in g
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release new models or data.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All our data environments, models and used code are open source and cited.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We do not have assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowd sourcing
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No humans where used in this study.
16. Declaration of LLM usage
Question: Does the paper describe the usage of L.LMs if is an important, original, or non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: While the our study focuses on L.Mss, did not use an LLM to design the experimental design, integrate new experiments do anything non-standard.
15

---

A Compute-peformance frontier for Qwen2.5 7B
1e65icut Goals Succems Rate Compute
He Thaks Success Compute
0938
2020
4030
6640
9:23
20:14
10003
14G23
20.0
2023
4020
00.0
20030
1263
14214
polle Stont Brdee Ladier yone rabing
Figure 4: Compute-per perfote-erformaccee frontier MiniWoB++ (results averaged over two seeds) for Qwen2.5 7B. The blue curve shows pure SFT on teacher demonstrations. Warm-colored curves represent hybrid runs that branch off from SFT checkpoints and continue training with RL. Early transitions to RL. push the Pareto frontier achieving higher success rates for the same compute and the only approach able achieve over 30% improvement both held-out goals (left) and held-out tasks (right).
B Compute Infrastructure
Our computational infrastructure comprises 8 x H100-80GB GPUs for expert data generation with the 70B model. For student model training, we allocate 2 x H100 GPUs for MiniWoB++ experiments and 4 x H100 GPUs for WorkArena experiments, reflecting the increased complexity of the latter.
C Extended Learning and Saturation Analysis
Challenges in ggent-Environment Interaction In this section talk about the general challenges faced by the agent to interact effectively with the environment.
Observation/action space mismatch: One of the important thing to note specifically in our web environment is the observation space which the agent uses a different from the action space. Multiple times, the agent can see the correct action in the AxTree but the action space, the icon is not visible and to make visible, the agent needs to scroll down and do the action. This mismatch causes huge problems [15]
The agent tries interact with items in the environment in ways that is designed. For example, the agent trying to fill in checkbox with value True while it should just click on it. (18]
UI Misuse:
Repeating actions: A common issue we observed is the repetition actions across multiple consecutive steps, often accompanied by verbose and redundant chains of thought. The agent frequently restates similar thoughts or re-executes the same actions unnecessarily, leading inefficiencies and sometimes getting stuck in loops. [18]
D Deriving Compute Cost
FLOPPs Estimation Methodology Flop calculations are based on model architecture, token counts, and average sequence lengths observed during training and evaluation.
FLOPs per Token
We estimate FLOPs per token using the following formula, adapted from nvidia benchmarking
FLOPs Spr token
(FLOPs Satto + FLOPSMLP + FLOPSembea) (1 + backward multiplier) (6)
16

---

SFT
Approach D: Continued SFT
Approach 1: On-Policy RL
LLaMA 708
Figure 5: Per task performance of SFT and SFT+RL agents MiniWob++.
SFT
Absrrcch Carl HARE SPT
Aparaht DAPURS
LLAMA 253
search
onder wpw.oeeacccccccooooooooooo.o coet-roth
623
0.47
3.12
3,20
1.20
onderiosd pro
600
E:30
131
131
1.20
0533
1.50
onder descrppeen hpmi p-c
111
1.11
6:3
3.0
1000
000
8:38
0.5%
1907
1032
043
0.2
0:33
335
123
1-4
023
0221
enter es hutee
3.50
3.00
0.33
Charcha chagg regarii
600
6:30
3.10
3.00
1.30
0003
0.50
0.00
600
0:30
0.83
310
2.00
1.50
003
0.8
0.00
The candee orooo ten lic
600
6:30
3.00
2.00
6:00
600
0.00
333
113
031
6:30
2.06
sn thange
$:301
310
116
5100
1933
wovavvteaaalagg tameus
8000
0:20
0.02
240
3.00
1.20
002
0:00
vider vaner hone
315
314
onate sibren
603
111
Average Train
603
Figure 6: Per task performance of SFT and SFT+RL agents on WorkArena.
17

---

Where:
FLOP Sattn
12 x (number of layers) x (hidden size)²
x (1 number query groups
sequence length
(7)
number of attention heads
hidden size
FLOPSMLP 18 (number of layers) (hidden size) x FFN
(8)
FLOPPS = 6 x vocabulary size x (hidden size)
(9)
On-Policy FLOPs (LLaMA-8B)
We compute the total FLOPs for each on on-policy epoch by summing the training and testing FLOPs:
FLOPStrin Ntrain X FLOPs
(backward=3)
(10)
per token
FLOPstes Nes X FLOPs
(hackward per token
(11)
FLOPs Sepoch FLOP + FLOPStcst Stest
(12)
Where Nusin and Nest are the number of tokens used for training and evaluation respectively. Sequence length S is measured per epoch from logged metrics.
Offiine FLOPs (Generation: I.L.MA-7B,, Training: LLaMA-8B)
Offline training includes two compute components:
Data Generation (LLaMA-70B, forward-only):
FLOPs Sgen N
FLOPs (70 backward=0)
(13)
per token
where Ngen avg seq len samples per epoch (from dataset metadata).
Training (LLaMA-SB, with backward pass):
FLOP Strain Ngen FLOPs (8B, backward=3)
(14)
per token
The total FLOPs per offline epoch is:
FLOPSgpoch FLOPs Sgen + FLOPS
(15)
All FLOPPs values are reported in exaFI.OPs by dividing the total FLOPPs by 1018
18

---

E Benchmark Descriptions
Drag all triangles into the black box.
Select 09/23/2016 as the date and hit submit.
Sort the numbers in increasing order, starting with the lowest number at the the list.
Copy the text from the 1st text area below and paste it into the text input.
Date:
$
Blandit quisquer
September 2010
Su
-12
Amet odio
aliquam.
$
Ultrices
ornare
$ -28
Submit
Submit
Submit
Baley
Ser vicence
New
Figure 7: Example tasks in MiniWoB++ 16 (top) and WorkArems 1T0 (bottom). MiniWoB consists of single-page simple tasks such as selecting a particular date and using a basic text editor, while WorkArena comprises multi-page complex tasks like filling forms and placing orders in enterprise environment.
F Extended Related Work
The Reproducibility Crisis in RL. The reproducibility crisis in large language models (LLMs) and reinforcement learning (RL) has garnered increasing attention, particularly due to the reliance on single seed results that distort the perceived performance of models. The reproducibility challenge" organized every year is a positive step towards addressing this. More concretely, Hochlehnert et al. [3] provide critical examination of how such practices undermine the reliability of published findings, revealing that many reported gains are sensitive to implementation choices, such random seeds and prompt formatting [13].
Bandit-domain RLHF with LLMs. Previous work RL for LLMs has predominantly focused on single-step tasks, which have shown effectiveness in mathematical reasoning and code generation 28 g 20). While these approaches exhibit promising results, they are limited in their applicability to real-world scenarios, which often require multistep decision-making capabilities. The narrow focus
https//ttpromll.org/
19

---

on bandit-style problems fails to address the complexities inherent in tasks that demand sequential interaction, highlighting a significant gap the current research landscape.
Interactive Agent Benchmarks. To assess the capabilities of LLM agents in more realistic envi- ronments, benchmarks such as WebArena [29]. WorkArena [0 5 the Agent Company [27]. and OSWorld [26] have been designed to evaluate agents on multi-step tasks across various domains. These benchmarks expose the limitations of current LLM agents, revealing that while they may perform well in controlled settings, their performance in practical applications remains subpar, un- derscoring the need for further advancements in agent robustness and generalization to multi-step planning.
Best practices deep RL. Building on the recognition of reproducibility challenges and unstable RL training of LLM agents, recent studies have proposed best practices for training LLM agents using RL methods. Dang and Ngo 7 recommend leveraging high quality data, balancing easy and hard problems, and controlling length generation with cosine reward. Yu et al. [28) promote higher clipping in the GRPO loss to promote diversity and avoid entropy collapse, dynamic sampling to improve training efficiency and stability, token level gradients for long CoT sequences, and overlong reward shaping to reduce reward noise. Roux et al. [20] introduce tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics. The proposed method (TOPR) allows the handling of both positive and negative examples in a fully offline setting, More generally, Hochlehnert et al. [13] emphasizes the need for greater methodological precision, particularly concerning decoding parameters, random seeds, prompt formatting, as well as the hardware and software frameworks, to guarantee transparent and thorough assessments of model performance.
LLM Agents trained with RL on multi-step environments. Recent advancements have sought to bridge the gap in training LLM agents for multi-step environments, with approaches like WebRL [19 and SWEET-RL 30 demonstrating significant progress. WebRL employs self-evolving curriculum to address the challenges of sparse feedback and task scarcity, successfully enhancing the performance of open LLMs web-based tasks [9]. Similarly, SWEET-RL introduces hierarchical structure that enables effective credit assignment over multiple turns, improving policy learning and generalization in collaborative reasoning tasks [30]. These studies collectively illustrate the necessity of adapting RL techniques to accommodate the complexities of multi-step interactions, paving the way for more capable and versatile L.LM agents.
G Test Set Hyper-Parameter Bootstrap Analysis
We overall find similar results between the held-out train and test tasks with respect to optimal hyper-parte ppper-parammters. While we see no large deviations, we find that some parameters such as curriculum learning from the instruct model and using error logs can have larger beneficial effect on the held-out testing tasks.
20

---

Instract
FLOP
7.6e+1s FLOPH
Instrct
2.55e 28 FLOPA
7..e+ea ILOPs
47%
12h
13%
16%
40%
13h
60%
50%
6.15
0.00
Instract
2.5ex
7.6o-11 FLOPs
2.5e+ FLOPA
7.6e+18 FLOP
A21-
Instract
2.5ee FLOPs
7.6o-11 SET FLOPs
Intrect
2.Se+it SFT FLOPS
7.Ge+18 SFT FLONs
0:3
042
50%
180%
70%
2.8ee18 PLOPS
100%
70%
T5%
6x.3 $4.47
Figure 8: Bootstrap analysis (n 1000 samples) of hyperparameter optimization across different SFT compute budgets on test held out tasks. Each subplot examines different hyperparameter, including increasing SFT compute: the base instruct model (left), +2.5e+18 SFT FLOPs (middle), and +7.6e+18 SFT FLOPs (right). For each hyperparameter-coopuur combination, the top panel shows relative reward performance with error bars indicating 95% confidence intervals, while the bottom panel displays win rates representing the percentage of bootstrap iterations where each parameter value achieved maximum performance. Results demonstrate that optimal hyperparameter values often shift as model pre- pre-training compute increases, suggesting that hyperparameter selection should be adapted based on the computational budget allocated to SFT.
H
Entropy
Figure reports log-entropy as function of training compute (FLOPs) across the training run.
21

---

0.5
0.0
Log Entropy
-0.5
-1.0
-1.5
2.0
-2.5
-3.0
0e18
Se18
10c18
Compute (FLOPs)
Figure 9: relationship between log-entropy and compute allocated to SFT warm-starting, We observe an initial sharp drop in log-entropy, followed by a period of stabilization and gradual decline. We speculate that this reduction in entropy, which makes rollouts more deterministic, may explain the difference optimal hyperparameters between the base and warm-started checkpoints.
I Random Search Space
We conduct a random hyperparameter sweep over 1,370 training runs over the following parameter configurations:
Decoding temperature (PLLM)): Sampled from (0.1, 0.25, 0.5, 0.75, 1) Curriculum learning: Enabled or disabled (True, False)
Curriculum mean (Hurga)): (0.25, 0.5, 0.75)
Curriculum Temperature (PCurr): ) (0.1, 0.3)
Discount rate: [0.5, 0.8, 0.9, 0.95, 0.98, 1.0]
Grouped-relative advantage: Enabled or disabled
Zero-advantage filtering: Enabled or disabled
Standard-deviation normalized advantage: Enabled disabled
Effective batch size: {64, 256, 512, 1024]
Learning rate: (1c-6, 5c-6, 5c-7}
Error log feedback: Enabled disabled
Importance ratio: Enabled or disabled
J Compute Allocation Hyperparameter Selection
For consistency, we select single set hyperparameters for all runs in Figure [ the bootstrap algorithm described in Algorithm [ where we use all the runs over all SFT checkpoints as input into Results. We then analyze results this bootstrap and use them to obtain the reported results.
so, we use
The optimal parameters given by this aggregate bootstrap are:
Decoding temperature PLLM: 0.25
22

---

Curriculum learning: False
Discount rate: 0.90
Grotped-relative advantage: True
Zero-advantage filtering: True
Standard-deviation normalized advantage: True
Effective batch size: 512
Learning rate: le-6
Error log feedback: True
Importance Ratio: False
K
Correctness of the Bootsrap
One of the main motivations for using the boostrap algorithm for our the vast amount of literature showing its desrable properties for estimating the sampling distribution of data statistics.
evaluation is
The most important attribute for the robustness of the boostrap is its ability to exploit the Central Limit Theorem (CLT) approximate the sampling distribution of a statistic. Using the CLT it can allow us to estimate its expectation and variability without parametric assumptions [12..
Specifically, in performs when paired with the best possible values of the other hyperparameters. The bootstrap algorithm yields unbiased estimate for target statistic given initial samples (full runs in our case). Thus, we choose the statistic interest to be the evaluation score M conditioned on the hyper-parameter of interst h assuming all other
case, want to evaluate, for each hyperparameter value h, how well it
g optimal:
(D) max (M 9): : E D. H'
(16)
Bootstrapping this yields an empirical distribution over Th, values, providing an unbiased estimate and confidence intervals for the maximum score achievable for h when other hyperparameters are optimal.
Additionally, there exists results showing the rate of convergence and consistency of the bootstrap algorithm which we directly exploit. It is well known result that as the number of training runs oc, the bootstrap estimate Tú, of the maximum achievable score for hyperparameter value becomes consistent. That is:
fn sup (M, : E E
(17)
This follows from the law of large numbers 24 assuming sufficient coverage of other hyperpa- rameters g. The rate of convergence depends on the tail behavior of the conditional distribution M H = h. Since our evaluation metric is bounded in [0, 1). the sub-Gaussing assumption holds automatically by Hoeffding's Lemma 14), ensuring concentration of the estimate. Under these conditions, the expected bias decays (LL/na), where n,, is the number of runs with H h.
These are well-known results, which is why the bootstrap is an extremely useful and simple algorithm for estimating any statistic of interest
L Train and Test Splits Used in the Experiments
We evaluate generalization by training only on the train split and reporting performance on held-out tasks from the test split. For the held-out goals metric, we instantiate goal variations using seed ranges [0, 2 g for training tasks (3 goals per task) and [0.9) g for test tasks (10 goals per task). Below we list exact task identifiers used in our experiments for both benchmarks. Task names are the registry keys from the respective environments.
23

---

WorkArena (Train: 24 tasks, Test: tasks)
Train:
workarena.sevvicnnvaal...eo
wedkattn..eviiccco....cccccccici
worfaree..eericeno..llleeeeeees
workmmen..eviccno..iilleesillll
wot.aaru..srricccooowodddsssssss
workaaee...eriiccooo.soooussssss
Test:
workaree.seevienno..rrateuuuur
wokkree..ssricnoooiiieeeidddddii
workaren.sericcooo..oottsssssss
wokkmrca.ssriccnno.mmmeccooooonn
wer.aaeusserrieeoovvddddaaaaaaa
MiniWoB+ (Train: 99 tasks, Test: 23 tasks)
Train:
ninivob.asending--uubeess
miniwob. anniwob.bisect-angh.
miniwob.boo bookboooko-ioooo -light
miniwob.choose-date
miniwob. chaeeb..hhoseeddteeeaayy
24

---

ainiwob.choose-date medium
miniwob.choose-date daldellladd
miniwob.choose-list
minivob.circle-centee
minivob.licc--bttoonesqcenne
minivobbdldk-cccckooo---ssttt
minivo.ciidd--ccckoooxs---rrarr
minivob.clickkccllaapitbee-
miniwoblicck-oollppibee22aaaa
miniwobdiidk--ollasiib--ddaaa
miniwob..lick-color
miniwob.click-dialog
miniwob.click-dialog-2
miniwob.click-link
miniwob.click-menu
minivob..lick-memu-2
minivob.dlick-srrollliii
miniwob.click-shee
miniwob.click-tab
miniwob.click-tab-2
minivob.clickktab-2hhar
t m-ddumm
miniwob.click-test
miniwob.click-test22
minivob.llick-estttrrnfffee
miniwob..lick-wideet
miniwob.copy-past
miniwob.copy-pastt-22
miniwob.count-See
miniwob.coumt-sides
miniwob.daily
aaiiwobddaily-caeendaa
miniwob.drag-Ba
miniwob.drag-cing
miniwob.drage
drwwob.raag-ubee
Miniwob.drag-iten
hariwob.drag-items..oo
miniwob.drag-shan
minivob.drag-shappss-
ninivob.drag-sort--nniii
Siniwob.draw-cing
miniwob.draw-
miniwob.emin emaiiiiboa
miniwob.email-iboo-.eee
miniwob. www.e//mmmmlooofoooeeeeee
25

---

miniwob.email
www.bbbemil-ibboo-fooweee
auniiwobemall--mbo---mpptrrtrt
miniwob. httpwobbmmmll--mbooooooooollllll
miniwobbrmaill-nnoo--rryyy
miniwob. aativoo.mmii--nbox--tttraarrrr
miniwob.cnter-date
miniwob.enter-text
minixobbenlee-exx-dynaniii
miniwob.enter-time
miniwob. sriiobb.inddgrretesst
miniwob.find word
miniwob. tiniwob.form-sequence
niniwob.form-sequeem.
miniwoo.genneaate-nnnne
miniwob.grid wrw.obb.rid-cooddinate
miniwob, wwwwwobggess-nnne
guess-number numberr
hnniwob.highligheee
miniwob.hot-cole
Jiniwob.ddentify-saapp
miniwob.login-use.
minivob.lognn-use--pooo
miniwob.mulee minivobbmullt-layoots
miniwob.multi-orderin.
ninivobbnavigate--ree
miniwob.oodderffor
seriwob.phone-beee
miniwob.read-taie
minivobb.ead-table-2
miniwobbresize-tettare
miniwob.right-angge
minlwob.ssrool-text
2
minivob.search-engine
minivob.sign-agreement
minivobssimple-algerra
minivob.social-media
minivob.social-mediaa
minivobssociallmediissoomo
minivor.tett-edior
minivob.tet-rransffom
Ainiwob.tic-tac-fe
26

---

miniwob. www.obbus--autocoompllo
hittwobb.scclloooomeeeeeeeetee
minivob.use-colorwwheed
miniwob.use-colorviie
miniwob.use-spinner
miniwob. vistaiiiisual
Test:
miniwob.buy-tie
miniwob.click-butte
minivob..lick-option
miniwob..lick-pie-nooee
miniwob.
himivob.email-imox--nlkaeee
miniwob.enter-text
miniwob.find-midpe
miniwob.focus-text
minivob.simple-arthneeiic
minivob.stock-market
miniwob.use-slider-2
minkwob.dicckcceeekboess
miniwobollsk-cheekboxee-llgg
miniwob.dlick-collapptte
miniwob.click-pic
minivob.Click-shades
minkvobelickk-ab-2eeaay
minivoobeeneer-password
miniwob..orm-seqqeee
miniwob.
hiniwwbbhighhighht--t--22
aniivobunicodde-tst.
miniwob.use-slide
27


Reviews:
[{"review_id": "13EhC7nHfp", "reviewer": "Reviewer_DhYQ", "summary": "LLM agent training is a field that has been substantially hindered by reproducibility issues and lack of transparency in both the RLHF-stage and the Agent-stage. This paper provides a statistically driven study of training LLM agents for web tasks.", "questions": "1. Some other works I've read seem to present mutually contradictory accounts of how important SFT before RLHF is. The general vibe I've gotten is that it may depend on the base model. Do you have any thoughts on this? Do you have intuition for if you'll see similar results if you use DeepSeek or Qwen?\n2. A key issue for deployment is the robustness of a model across a wide diversity of website structures. Arguably a 1% failure rate for a general purpose web agent is problematically high. Have you considered this or do you plan to assess this?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 1, "strengths_and_weaknesses": "This paper is clearly a step in the right direction and is far more transparent and statistically detailed than any other similar work I am aware of. The paper is relatively narrowly scoped to one model and one framework, but that's likely a de facto necessity because a more detailed analysis would be a mind-boggling amount of work. While this level of rigor would cause me to be critical in other areas, I don't fault the authors for balancing resource availability and detail, and praise them for raising the bar in this subfield.\n\nThe paper writing is pretty clean. On a conceptual level the work isn't very original, but that's a non-issue for a paper focused on providing systematic evidence of things that are vaguely known and often indirectly alluded to.", "was_revised": true, "final_justification": "I am already strongly recommending this paper for acceptance. The response from the authors is helpful but doesn't raise my score to a 6.", "rebuttal": "We sincerely thank Reviewer DhYQ for the thoughtful and constructive feedback, and for recognizing the transparency and statistical depth of our work. We are encouraged by the positive assessment regarding our reproducibility focus and overall impact on the LLM web agent training subfield.\n\n## General comments\nSince the submission deadline, we have significantly expanded and strengthened our results:\n* **Expanded the hyperparameter search for the bootstrap experiment (from 240 to 1,370 runs):** We increased the size of the random hyperparameter sweep by more than 5x. Importantly, the main conclusions reported in the original submission remain unchanged, demonstrating the robustness of our findings. We believe these results showcase that the proposed bootstrap technique is robust and sample-efficient, effectively finding the same conclusions even when scaling the number of runs 5x. The only minor update is that using standard-deviation normalized advantage appears moderately better than not using it when starting RL from the instruct model.\n\n\n* **Generality across benchmarks:** We performed additional WorkArena experiments. Our updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also get some stable runs for Pure RL, improving the held-out task performance by 7.5%.\n\n* **Generality across model families:** We ran experiments on Qwen2.5 and replicated the main conclusion: SFT+RL is the only method able to match or exceed teacher performance. This provides evidence that our findings generalize beyond the LLaMA family of models.\n\n* **Bootstrap reliability:** We discuss the consistency and unbiasedness of the bootstrap estimator as well as the statistic we are specifically interested in estimating. Additionally, we provide examples of the reliability of our estimate in terms of standard error as the number of runs increases, showing that even under pessimistic assumptions we can maintain a relatively small standard error.\n\n## Reviewer-specific comments\n\n*On a conceptual level the work isn't very original, but that's a non-issue for a paper focused on providing systematic evidence of things that are vaguely known and often indirectly alluded to.*\n\nWe appreciate this observation and would welcome suggestions from the reviewer on specific directions or techniques that could further improve the paper's originality. Our intention was to raise the bar of empirical rigor in this subfield, and we believe that systematically validating \"vaguely known\" insights at scale is itself a valuable contribution.\n\n*Some other works I've read seem to present mutually contradictory accounts of how important SFT before RLHF is. The general vibe I've gotten is that it may depend on the base model. Do you have any thoughts on this? Do you have intuition for if you'll see similar results if you use DeepSeek or Qwen?*\n\nGiven our new results, we find that SFT before RL is generally beneficial for both Qwen and LLaMA models. The contradictory accounts in prior work may stem from the fact that too much SFT can hinder learning in the RL stage, leading to diminishing returns. We observe this same phenomenon: after a certain amount of SFT, additional SFT provides less benefit.\nTo further address generality, we ran experiments with the Qwen2.5 model family. We performed a very restricted hyperparameter random search on MiniWoB++, only sweeping over learning rate, batch size, and sampling temperature (all other hyperparameters were fixed to those identified as optimal in the LLaMA bootstrap). The updated MiniWoB++ results are shown below:\n\n| Model                           | Heldout Goals     | Heldout Tasks     |\n|---------------------------------|-------------------|-------------------|\n|---------------------------------|-------------------|-------------------|\n| Llama-3.3-70B Instruct (teacher)| 63.2 ± 2.4        | 61.9 ± 2.4        |\n| Llama-3.1-8B Instruct           | 29.5 ± 2.3        | 36.4 ± 2.4        |\n| Llama-3.1-8B SFT                | 53.4 ± 2.5        | 55.6 ± 2.5        |\n| Llama-3.1-8B RL                 | 43.5 ± 2.5        | 43.5 ± 2.5        |\n| Llama-3.1-8B SFT+RL             | **66.3 ± 2.4**    | **62.9 ± 2.4**    |\n|---------------------------------|-------------------|-------------------|\n| Qwen-2.5-72B Instruct (teacher) | 61.0 ± 3.4        | 59.0 ± 3.4        |\n| Qwen-2.5-7B Instruct            | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT                 | 57.0 ± 3.5        | 56.5 ± 3.5        |\n| Qwen-2.5-7B RL                  | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT+RL              | **63.5 ± 3.4**    | **62.0 ± 3.4**    |\n\n\nWe were able to replicate the MiniWoB++ result that SFT+RL achieves the best absolute performance with a new model family, namely Qwen2.5. It is also the only method able to surpass the teacher. Interestingly, we could not improve the performance of the instruct model using Pure RL in this case. After a few steps, the model often failed to follow the correct output format, began producing repeated tokens, or switched to unrelated languages. Nevertheless, the conclusions drawn from the LLaMA experiment still hold.\n\n*A key issue for deployment is the robustness of a model across a wide diversity of website structures. Arguably a 1% failure rate for a general purpose web agent is problematically high. Have you considered this or do you plan to assess this?*\n\nWe agree this is a critical point. While human error rates can be higher for these tasks, autonomous systems are typically held to a higher standard. At present, no models achieve a 100% success rate on any of the reported benchmarks. As models improve, it is essential for the community to have access to high-performing open models that enhance data privacy and reduce costs, which is the main motivation of our work.\nWe also believe that it is important for models to be able to detect when they cannot complete a task and defer to a human or another system for assistance. This capability remains an open challenge for modern LLMs, and we believe that progress on this front will directly benefit web-based agents as well."}, {"review_id": "7nTlode4VC", "reviewer": "Reviewer_8JSX", "summary": "The paper provides guidelines and a recipe for training an LLM web agent with RL for compute-constrained practitioners to use, focusing on two benchmarks: MiniWoB++ and WorkArena. In the recipe, a larger teacher model generates reasoning traces for the agentic web task, which are then filtered for correctness and used to train a smaller student model with SFT. The student model then uses the rest of the compute budget to RL train itself on the task using GRPO. The authors provide guidelines for how much SFT to perform versus GRPO, enabling practitioners to strike a Pareto optimal balance. The authors also perform a statistical hyperparameter sensitivity study to recommend appropriate hyperparameters to use for this process.", "questions": "- Why is there an asymmetry between the two charts of Figure 4? I would like to see the analysis of WorkArena be at the level of the analysis on MiniWoB++. In particular, I would like to see the branching timing into RL training studied.\n- Why is RL training failing to be effective on WorkArena; in particular, why does standard RL training get 0% accuracy? I would like to see an adequate explanation and/or improvement in this area. \n- What is the effectiveness of the Boltzmann sampling strategy? How does it compare to a uniform baseline? If it is more effective, I would like to see a breakdown of what makes it more effective.\n- Is this recipe generalizable to models outside the Llama family? What about the hyperparameters? Evidence of this would support the generalizability of the method.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n## Quality\n- The experiments and ablations for MiniWoB++ go into an effective level of depth about choices to make during training web agents. In particular, the experiment design for when to branch out of SFT training is well-constructed.\n- The author's method of performing a statistical hyperparameter analysis appears effective for this resource constrained setting on aggregating information about the senisitivity of different hyperparameter choices.\n## Clarity\n- Many of the figures effectively communicate the results, such as the hyperparameter sensitivity analysis, and the left chart of Figure 4.\n## Significance\n- Considering the popularity of training web agents, providing a guideline for resource-constrained teams to train a successful agent is highly valuable to the community. In particular, the information on when to branch out to perform RL training will be valuable for teams to save their resources.\n## Originality\n- The authors' method of performing the statistical hyperparameter sensitivity analysis appears to be novel.\n- The Boltzmann sampling strategy for training curriculum appears to be novel.\n\n# Weaknesses\n## Quality\n- The RL training fails to improve model performance on WorkArena, with RL alone dropping train/test held-out performance from 9%/5% accuracy to 0% accuracy each, respectively. The drop to zero accuracy without an accompanying analysis indicates that either something may have been wrong with the training environment, or the training recipe does not generalize to benchmarks other than MiniWoB++. Either way, the generality of this training recipe has not been demonstrated, having only been successful on one benchmark.\n- The effects of the Boltzmann sampling strategy for training curriculum are not compared to any baseline, with no evidence provided about its effectiveness compared to uniform sampling.\n## Clarity\n- In Section 6: Ablation and Sensitivity Analysis, the hyperparameter analysis does not go into depth, and often states results without well-substantiated explanations. For example on lines 257-259, the paper states \"relative advantage helps only after SFT, while using raw rewards works better when starting directly from the Instruct model, possibly due to how advantage scaling interacts with the initial weights\", without explaining the basis behind the claim in the later half of the sentence.\n- The left and right charts of Figure 4 appear to represent the same analysis on the two different benchmarks, yet the left chart (for MiniWoB++) appears much more fleshed out than the right chart (for WorkArena). Additionally, there appears to be an asymmetry in the experiments with no explanation: why is there only one branch out of the SFT training curve for WorkArena, and why is the curve for the SFT warm-up different from the standard SFT curve, are different hyperparameters used?\n## Significance\n- The generality of these guidelines may be limited; the authors did not study whether their results generalize to new models or different compute budgets. Additionally, It is unclear whether the positive results are generalizable beyond MiniWoB++.\n## Originality\n- While appearing to be novel, the effectiveness of the Boltzmann sampling strategy on the training curriculum is not studied.", "was_revised": true, "final_justification": "I thank the authors for clearly addressing my concerns, and as such I am increasing my score. The most important factors for this were the addition of working experiments on WorkArena, which I find is critical, and demonstrating that the recipe generalizes to other model families. Together, these mostly address my concerns of generality. I do believe that at least a small version of the branching study (e.g. localized around where MiniWoB++ was optimal, since compute is expensive) would be beneficial for the final version of this paper, to see how consistent that is across tasks. This will help to maximize the impact of the paper.", "rebuttal": "We sincerely thank Reviewer 8JSX for the thoughtful and constructive feedback, and for recognizing the significance of our . We are encouraged by the positive assessment regarding our reproducibility focus and overall impact on the LLM web agent training subfield and have worked extensively to address your concerns.\n\n## General comments\n\nSince the submission deadline, we have significantly expanded and strengthened our results:\n* **Expanded the hyperparameter search for the bootstrap experiment (from 240 to 1,370 runs):** We increased the size of the random hyperparameter sweep by more than 5x. Importantly, the main conclusions reported in the original submission remain unchanged, demonstrating the robustness of our findings. We believe these results showcase that the proposed bootstrap technique is robust and sample-efficient, effectively finding the same conclusions even when scaling the number of runs 5x. The only minor update is that using standard-deviation normalized advantage appears moderately better than not using it when starting RL from the instruct model.\n\n* **Generality across model families:** We ran experiments on Qwen2.5 and replicated the main conclusion: SFT+RL is the only method able to match or exceed teacher performance. This provides evidence that our findings generalize beyond the LLaMA family of models.\n\n* **Generality across benchmarks:** We performed additional WorkArena experiments. Our updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also get some stable runs for Pure RL, improving the held-out task performance by 7.5%.\n\nThese results show that the SFT+RL approach continues to dominate in absolute performance, while Pure RL can achieve modest improvements with additional tuning. We will include this updated table and analysis in the camera-ready version.\n\n## Reviewer-specific comments\n\n*1. \"The RL training fails to improve model performance on WorkArena, with RL alone dropping train/test held-out performance from 9%/5% accuracy to 0% accuracy each, respectively. \"*\n\nWe performed additional experiments on WorkArena to address the reviewers’ concerns about generalizability. We were able to run many more WorkArena experiments after the deadline. We also fixed environment issues related to models breaking the WorkArena instances by using their admin rights to change the UI, which could lead to infeasible tasks. This allowed us to scale the experiment much more effectively.\nOur updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also obtain stable runs for Pure RL, improving the held-out task performance by 7.5%. Note that since the deadline, we changed the task split because we upgraded to a newer version of WorkArena, which explains some of the discrepancies with earlier results.\nWe have addressed this issue—our initial runs failed due to environment-related bugs that caused instability and crashes. Since fixing these issues, we have obtained stable RL runs and now report the updated results below.\n\n\nDue to character limits, we cannot insert the full table here, please see our response to reviewer `zHWC` for the full results.\n\n\n*\"2. Either way, the generality of this training recipe has not been demonstrated, having only been successful on one benchmark.\"*\n\nTo address generality, we ran experiments with the Qwen2.5 model family. We performed a very restricted hyperparameter random search on MiniWoB++, only sweeping over learning rate, batch size, and sampling temperature (all other hyperparameters were fixed to those identified as optimal in the LLaMA bootstrap). The updated MiniWoB++ results are shown below:\n\nAs above, due to character limits, we cannot insert the full table here, please see our response to reviewer `zHWC` for the full results.\n\nWe were able to replicate the MiniWoB++ result that SFT+RL achieves the best absolute performance with a new model family, namely Qwen2.5. It is also the only method able to surpass the teacher. Interestingly, we could not improve the performance of the instruct model using Pure RL in this case. After a few steps, the model often failed to follow the correct output format, began producing repeated tokens, or switched to unrelated languages. Nevertheless, the conclusions drawn from the LLaMA experiment still hold.\n\n*3. The effects of the Boltzmann sampling strategy for training curriculum are not compared to any baseline, with no evidence provided about its effectiveness compared to uniform sampling.*\n\nIn our ablation study, we compare curriculum learning via Boltzmann sampling against uniform sampling (which is what we use when curriculum learning is disabled). We find Boltzmann sampling helps when training from instruct models, where frequent failures can degrade performance. After SFT, however, curriculum learning tends to hurt, likely because the model already performs well on most tasks. In this setting, curriculum sampling causes the model to over-focus on the few very hard tasks, while continuing to train on tasks it already performs well on is still beneficial.\nWhile we do not claim to introduce this sampling strategy, our goal is to empirically investigate which hyperparameters and training techniques matter and how their effects vary across compute budgets. We agree that optimal curriculum structure remains an open question, being an active research area (e.g., in active inference)\n\n\n*\"4. The generality of these guidelines may be limited; the authors did not study whether their results generalize to new models or different compute budgets.\"*\n\nWe agree that further generalization is important, which is why we have now included results using Qwen2.5 (see above), where SFT+RL again achieves the best absolute performance. We are also actively conducting compute allocation experiments for Qwen2.5.\nRegarding compute budgets, we are unsure what the reviewer is referring to. Our compute allocation study already explores how to distribute a fixed compute budget across SFT, RL, and SFT+RL. If the reviewer meant something else, we would appreciate further clarification.\n\n\n*\"5. In Section 6: Ablation and Sensitivity Analysis, the hyperparameter analysis does not go into depth, and often states results without well-substantiated explanations. For example on lines 257-259, the paper states \"relative advantage helps only after SFT, while using raw rewards works better when starting directly from the Instruct model, possibly due to how advantage scaling interacts with the initial weights\", without explaining the basis behind the claim in the later half of the sentence.\"*\n\nWe appreciate the reviewer pointing this out and will revise the appendix to include deeper discussion. Our initial reasoning was that standard advantage normalization could scale updates pathologically, especially early in training, where naive reinforce (bounded in [-1, 1]) may be more stable.\nUpon further analysis, we believe the main factor is the interaction between reward baselines and the model's initial capabilities. Specifically, when starting from an instruct model, the baseline often cancels gradient updates on tasks the model performs well. Yet we find that positive updates even on such tasks are important for stabilizing learning. In contrast, SFT-trained models already acquire key skills (e.g., formatting, parsing HTML/AXTrees), so continuing to train on tasks they already solve yields diminishing returns. We believe this more accurately explains the advantage of naive reinforce in that setting.\n\n\n*\"6. The left and right charts of Figure 4 appear to represent the same analysis on the two different benchmarks, yet the left chart (for MiniWoB++) appears much more fleshed out than the right chart (for WorkArena). Additionally, there appears to be an asymmetry in the experiments with no explanation: why is there only one branch out of the SFT training curve for WorkArena, and why is the curve for the SFT warm-up different from the standard SFT curve, are different hyperparameters used?\"*\n\nWe agree this presentation could be confusing. We did not repeat the full compute allocation study for WorkArena due to resource constraints—it was meant to test whether MiniWoB++ conclusions hold in a different setting. We have clarified the manuscript to explain that WorkArena results are intended to validate the generality of the MiniWoB++ findings. We also revised the figure to use a format consistent with the MiniWoB++ version.\n\n*\"7. The generality of these guidelines may be limited....\"*\n\nAs noted above, we extended our experiments to WorkArena and to the Qwen2.5 model family, demonstrating that our main conclusions (e.g., that SFT+RL provides the best results) generalize across models and benchmarks.\n\n*Why is there an asymmetry between the two charts of Figure 4? I would like to see the analysis of WorkArena be at the level of the analysis on MiniWoB++. In particular, I would like to see the branching timing into RL training studied.*\n\nWe agree this would be valuable, but running a full compute allocation study on WorkArena would be very costly given the instability and runtime of its RL training. If this analysis is essential, we will commit to including it upon acceptance.\n\n*\"8. Is this recipe generalizable to models outside the Llama family? What about the hyperparameters?.\"*\n\nPlease see General Comment Section 2, where we describe how the hyperparameters identified via the bootstrap for LLaMA transfer successfully to Qwen2.5. This supports the generality of both the training recipe and hyperparameter recommendations.\n\n\nWe thank the reviewer for all their feedback which has greatly improved our work as a result. If there are any further clarifications we are happy to engage in further discussion."}, {"review_id": "Czi6KJkPyn", "reviewer": "Reviewer_zHWC", "summary": "This paper presents a statistical analysis of hyperparameter selection for fine-tuning the LLaMA3-8B model into a web agent. The training pipeline includes supervised fine-tuning (SFT) via distillation from LLaMA3-70B, followed by GRPO optimization. The paper suggest to (1) Branching into RL early (2) an optimal hyper-parameter configuration worked for training LLaMA3-8B into a web-agent (with temperature=0.25, no curriculum learning, zero-advantage filtering, etc.), (3) the paper propose a strategy for bootstrapping estimation of the important hyperparameter.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strength:\n1. The paper is clearly written and easy to follow.\n2. The proposed hyperparameter selection strategy is both efficient and potentially scalable for use in LLM agent training.\n3. The experiments, including ablation studies, are thorough and well-executed.\n\n\n# Weakness:\n1. Figure 2 is unclear. The task described in the figure is confusing and would benefit from a more detailed demonstration.\n2. All hyperparameter-choosing experiments are conducted exclusively on the LLaMA model and also all the corresponding insights (e.g. SFT + RL > RL alone, etc). It raises concerns about the generalizability of the findings to other LLMs, such as Qwen, DeepSeek, etc or other model size (1.5B, 32B, etc) \n3. The insight that “SFT + RL > RL alone” seems straightforward, but I am curious about the underlying reasons. Specifically:\n (1) Have you tried using test-time scaling inference to mitigate the Pass@1 bias in evaluation?\n (2) Why does pure RL perform worse than its SFT and SFT+RL counterparts? For example, in WorkArena (Table 1), RL applied to LLaMA3.1-8B-RL achieves a score of 0.0—worse than the base model. Is this because the web-agent task demands a specific reasoning format, such that without the prior structure introduced by SFT, rollouts receive zero reward and RL fails to learn anything meaningful? If so, it would be helpful for the authors to provide a qualitative analysis explaining this behavior.\n 4. I find the role of error log feedback unclear. Is error feedback incorporated during SFT/RL training in prompt, or is it only used at test time?\n 5. Why is a decoding temperature of 0.25 optimal for this task? This seems counterintuitive, as such a low temperature could lead to deterministic decoding and reduce rollout diversity. Could this explain the zero improvement of pure RL in WorkArena?\n 6. The hyperparameter configuration is also somewhat unclear. While Appendices E–F describe the search space, it's not obvious whether key hyperparameters—such as batch size and learning rate—are shared between the RL and SFT phases.", "was_revised": true, "final_justification": "I will raise the score to boardline accept, as most of my concerns have been addressed in the authors' rebuttal", "rebuttal": "We sincerely thank Reviewer zHWC for the thoughtful and constructive feedback, and for recognizing the clarity of writing, the scalability of our hyperparameter selection strategy, and the thoroughness of our experiments and ablations. We are encouraged by these positive remarks and address all concerns raised below.\n\n## General comments\nSince the submission deadline, we have significantly expanded and strengthened our results:\n\n* **Expanded the hyperparameter search for the bootstrap experiment (from 240 to 1,370 runs):** We increased the size of the random hyperparameter sweep by more than 5x. Importantly, the main conclusions reported in the original submission remain unchanged, demonstrating the robustness of our findings. We believe these results showcase that the proposed bootstrap technique is robust and sample-efficient, effectively finding the same conclusions even when scaling the number of runs 5x. The only minor update is that using standard-deviation normalized advantage appears moderately better than not using it when starting RL from the instruct model.\n\n* **Generality across benchmarks:** We performed additional WorkArena experiments. Our updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also get some stable runs for Pure RL, improving the held-out task performance by 7.5%.\n\n* **Generality across model families:** We ran experiments on Qwen2.5 and replicated the main conclusion: SFT+RL is the only method able to match or exceed teacher performance. This provides evidence that our findings generalize beyond the LLaMA family of models.\n\n* **Bootstrap reliability:** We discuss the consistency and unbiasedness of the bootstrap estimator as well as the statistic we are specifically interested in estimating. Additionally, we provide examples of the reliability of our estimate in terms of standard error as the number of runs increases, showing that even under pessimistic assumptions we can maintain a relatively small standard error.\n\n\n## Reviewer specific \n*1. Figure 2 is unclear. The task described in the figure is confusing and would benefit from a more detailed demonstration.\nWe understand that it might be difficult to get the gist of both benchmarks in that single figure.*\n\nWe extended the Appendix with examples of tasks for both benchmarks and intend to extend the main part of the paper upon acceptance using the extra allowed page.\n\n*2. All hyperparameter-choosing experiments are conducted exclusively on the LLaMA model and also all the corresponding insights (e.g. SFT + RL > RL alone, etc). It raises concerns about the generalizability of the findings to other LLMs (...).*\n\nTo address concerns about generality beyond the LLaMA family, we ran a very restricted hyperparameter random search of Qwen2.5 on MiniWoB++, taking into account the findings from the LLaMA bootstrap. We only swept over learning rate (LR), batch size (BS), and sampling temperature, fixing all other hyperparameters to those identified as optimal in the LLaMA experiments. From our sweep, we found that LR and BS are consistent with the results reported for LLaMA, with sampling temperature being higher at 0.75, which falls within the guidelines provided by the Qwen team.\nThe updated MiniWoB++ results are shown below:\n\n| Model                           | Heldout Goals     | Heldout Tasks     |\n|---------------------------------|-------------------|-------------------|\n| Llama-3.3-70B Instruct (teacher)| 63.2 ± 2.4        | 61.9 ± 2.4        |\n| Llama-3.1-8B Instruct           | 29.5 ± 2.3        | 36.4 ± 2.4        |\n| Llama-3.1-8B SFT                | 53.4 ± 2.5        | 55.6 ± 2.5        |\n| Llama-3.1-8B RL                 | 43.5 ± 2.5        | 43.5 ± 2.5        |\n| Llama-3.1-8B SFT+RL             | **66.3 ± 2.4**    | **62.9 ± 2.4**    |\n|---------------------------------|-------------------|-------------------|\n| Qwen-2.5-72B Instruct (teacher) | 61.0 ± 3.4        | 59.0 ± 3.4        |\n| Qwen-2.5-7B Instruct            | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT                 | 57.0 ± 3.5        | 56.5 ± 3.5        |\n| Qwen-2.5-7B RL                  | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT+RL              | **63.5 ± 3.4**    | **62.0 ± 3.4**    |\n\n\nWe replicated the MiniWoB++ finding that SFT+RL achieves the best absolute performance with a new model family, namely Qwen2.5. It is also the only method able to surpass the teacher. Interestingly, Pure RL did not improve the instruct model in this case: after a few steps, the model often failed to follow the correct output format, produced repeated tokens, or switched to unrelated languages. Nevertheless, the conclusions drawn from the LLaMA experiment still hold. We are actively working on the compute allocation study for Qwen2.5 and will include these findings in the next version of the manuscript.\n\n*3. “SFT + RL > RL alone” seems straightforward. (1) Have you tried using test-time scaling inference to mitigate the Pass@1 bias in evaluation? (2) Why does pure RL perform worse than its SFT and SFT+RL counterparts? For example, in WorkArena (Table 1), RL applied to LLaMA3.1-8B-RL achieves a score of 0.0—worse than the base model.*\n\nIt is not straightforward that SFT+RL forms a better compute Pareto front, given that expert data collection is much more costly (9x in our case). Regarding Pass@1 bias, there should be no bias because we model-select using Pass@1 and report Pass@1 on the test set. We are happy to run additional tests for the camera-ready version if the reviewer insists.\nOn Pure RL performance: the main issue is that RL applied to the instruct model struggles to operate effectively on challenging WorkArena tasks (starting at ~8% on training tasks). This results in numerous errors, including parsing issues, verbose rationales, clicking non-existent elements, and failure to follow output templates. We allocated additional compute to Pure RL on WorkArena and observed a 7.5% improvement on the held-out test tasks.\nWe also scaled up WorkArena experiments post-deadline and fixed environment issues where models could break WorkArena instances using admin rights to change the UI, leading to infeasible tasks. This allowed us to scale the experiments more effectively. Our updated results confirm that SFT+RL remains the best approach on WorkArena. Note that we changed the task split after upgrading to a newer version of WorkArena, which explains some discrepancies with earlier results.\nThe updated WorkArena results are shown below:\n\n| Model                           | Heldout Goals   (%)  | Heldout Tasks  (%)   |\n|--------------------------------|--------------------|-------------------|\n| Llama-3.3-70B Instruct (teacher)\t| 36.0 ± 2.4         \t| 44.0 ± 2.5 | \n| Llama-3.1-8B Instruct           \t| 8.0 ± 1.0          \t| 4.0 ± 1.0 |\n| Llama-3.1-8B SFT                \t| 28.4 ± 2.3         \t| 26.4 ± 2.2 |\n| Llama-3.1-8B RL                 \t| 8.0 ± 1.9          \t| 11.5 ± 2.3 |\n| **Llama-3.1-8B SFT+RL**         \t| **34.6 ± 2.4**     \t| **28.0 ± 2.2** |\n\nThese results show that the SFT+RL approach continues to dominate in absolute performance, while Pure RL can achieve modest improvements with additional tuning.\n\n*4. I find the role of error log feedback unclear. Is error feedback incorporated during SFT/RL training in prompt, or is it only used at test time?*\n\nError log feedback means that if the model encounters an environment error (e.g., clicking on a non-existent item), we return the error log to the model in the next step within its prompt. This is used during training.\n\n*5. Why is a decoding temperature of 0.25 optimal for this task? Could this explain the zero improvement of pure RL in WorkArena?*\n\nThe sampling temperature largely depends on the initial entropy of the model. While 0.25 may be low for some models, it falls within the recommended range for the LLaMA 3 family. The lack of Pure RL improvement in WorkArena was primarily due to environment errors, which we have since alleviated, as reflected in the updated results above.\n\n*6. The hyperparameter configuration is also somewhat unclear. Are key hyperparameters shared between the RL and SFT phases?*\n\nWe tune the hyperparameters for both SFT and RL phases. RL has considerably more hyperparameters to tune, so we focus there. SFT is more stable and works well with a wider range of settings. Specifically, we found that a learning rate of 1e-6 and batch size of 512 are generally good defaults for SFT and are also used in RL."}, {"review_id": "sAeXM12HJj", "reviewer": "Reviewer_jEdW", "summary": "This paper aims to fill the gap in reproducibility of training LLM agents with RL by providing a comprehensive training cookbook based on extensive experiments.\nThe cookbook includes suggestions on how to balance SFT and RL and also practical suggestions on hyper parameters selection.\nTo derive these insights, this paper fine-tunes Llama 3.1 8B and experiments with two agent benchmarks---MiniWoB++ and WorkArena. These two benchmarks are shipped with reward functions for RL training. To collect the trajectories for SFT, this paper directly uses a Llama 3.1 70B teacher model to generate them.\nConcretely, this paper finds the following things: 1. balancing SFT and RL can be more compute-efficient than pure SFT and pure RL; 2. curriculum learning and error log feedback become counterproductive once SFT warmup is applied; 3. settings that can lead to better performance in GRPO, etc.", "questions": "1. I think it is important to be clear about how generalizable the conclusions derived in this paper are, given that the purpose of this paper is to provide a playbook for pushing open LLM agents' performance forward.\n2. Could you justify the reliability of the bootstrapping method somehow? I am not very clear with your setting and may have some misunderstanding.\n3. Why do you choose to use Llama 3.1 70B to generate the training data for SFT? In practice, high-quality human trajectories are probably still necessary. As a result, models trained with synthetic trajectories may not reveal the capacity of SFT models in production.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. This paper has a clear motivation and tackles an important problem. The agent community wants to learn better practices of training LLM agents with RL.\n2. Providing such a training cookbook is heavy and requires a ton of effort.\n\n\n### Weaknesses:\n1. Though this paper has already made considerable efforts in training models with 240 configurations, it may not be sufficient for the purpose of providing a comprehensive cookbook. Specifically, this paper only considers a very limited family of models (i.e., Llama 3.1 8B/70B as the student and teacher model respectively). The conclusions may not generalize to other models. For example, for models like Qwen 2.5, which has probably included many SFT data during continual pre-training, will we still need SFT before RL? Also, will the conclusions hold if we use a different model to generate the SFT data? What if we use human-crafted SFT data? In that case, should we still spend the same amount of compute on RL?\n2. The bootstrapping method to obtain insights on hyper parameters may not be reliable. What if the original 240 configurations do not have a good coverage of good configurations?\n3. The conclusions on different benchmarks are not consistent. As a result, this paper may fail to convey a generally meaningful message to the community. For example, on MiniWob++, SFT+RL outperforms RL-only, while on WorkArena it is a different story.", "was_revised": true, "final_justification": "I have raised my overall evaluation after reading the response", "rebuttal": "We sincerely thank Reviewer jEdW for the thoughtful and constructive feedback, and for recognizing the importance of our work and the effort required to create a reproducible training cookbook for LLM agents. We address all concerns raised below.\n\n## General comments\nSince the submission deadline, we have significantly expanded and strengthened our results:\n\n* **Expanded the hyperparameter search for the bootstrap experiment (from 240 to 1,370 runs):** We increased the size of the random hyperparameter sweep by more than 5x. Importantly, the main conclusions reported in the original submission remain unchanged, demonstrating the robustness of our findings. We believe these results showcase that the proposed bootstrap technique is robust and sample-efficient, effectively finding the same conclusions even when scaling the number of runs 5x. The only minor update is that using standard-deviation normalized advantage appears moderately better than not using it when starting RL from the instruct model.\n\n* **Generality across benchmarks:** We performed additional WorkArena experiments. Our updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also get some stable runs for Pure RL, improving the held-out task performance by 7.5%.\n\n* **Generality across model families:** We ran experiments on Qwen2.5 and replicated the main conclusion: SFT+RL is the only method able to match or exceed teacher performance. This provides evidence that our findings generalize beyond the LLaMA family of models.\n\n* **Bootstrap reliability:** We discuss the consistency and unbiasedness of the bootstrap estimator as well as the statistic we are specifically interested in estimating. Additionally, we provide examples of the reliability of our estimate in terms of standard error as the number of runs increases, showing that even under pessimistic assumptions we can maintain a relatively small standard error.\n\n## Reviewer specific\n\n*1. Though this paper has already made considerable efforts in training models with 240 configurations, it may not be sufficient for the purpose of providing a comprehensive cookbook.*\n\nWe have increased the number of runs from 240 to 1,370 and found that our main conclusions remain the same. Please see the general comments above for details.\n\n*2. The conclusions may not generalize to other models.*\n\nTo address concerns about generality beyond the LLaMA family, we ran a very restricted hyperparameter random search of Qwen2.5 on MiniWoB++, taking into account the findings from the LLaMA bootstrap. We only swept over learning rate (LR), batch size (BS), and sampling temperature, fixing all other hyperparameters to those identified as optimal in the LLaMA experiments. From our sweep, we found that LR and BS are consistent with the results reported for LLaMA, with sampling temperature being higher at 0.75, which falls within the guidelines provided by the Qwen team.\nThe updated MiniWoB++ results are shown below:\n| Model                           | Heldout Goals     | Heldout Tasks     |\n|---------------------------------|-------------------|-------------------|\n| Llama-3.3-70B Instruct (teacher)| 63.2 ± 2.4        | 61.9 ± 2.4        |\n| Llama-3.1-8B Instruct           | 29.5 ± 2.3        | 36.4 ± 2.4        |\n| Llama-3.1-8B SFT                | 53.4 ± 2.5        | 55.6 ± 2.5        |\n| Llama-3.1-8B RL                 | 43.5 ± 2.5        | 43.5 ± 2.5        |\n| Llama-3.1-8B SFT+RL             | **66.3 ± 2.4**    | **62.9 ± 2.4**    |\n|---------------------------------|-------------------|-------------------|\n| Qwen-2.5-72B Instruct (teacher) | 61.0 ± 3.4        | 59.0 ± 3.4        |\n| Qwen-2.5-7B Instruct            | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT                 | 57.0 ± 3.5        | 56.5 ± 3.5        |\n| Qwen-2.5-7B RL                  | 32.8 ± 3.3        | 37.0 ± 3.4        |\n| Qwen-2.5-7B SFT+RL              | **63.5 ± 3.4**    | **62.0 ± 3.4**    |\n\nWe replicated the MiniWoB++ result that SFT+RL achieves the best absolute performance with a new model family, namely Qwen2.5. It is also the only method able to surpass the teacher. Interestingly, Pure RL did not improve the instruct model in this case. After a few steps, the model often failed to follow the correct output format, began producing repeated tokens, or switched to unrelated languages. Nevertheless, the conclusions drawn from the LLaMA experiment still hold. We are actively working on the compute allocation study for Qwen2.5, which is compute-intensive, and will include these findings in the next version of the manuscript.\n\n*3. Also, will the conclusions hold if we use a different model to generate the SFT data?*\n\nWe tried training LLaMA using Qwen as the teacher and vice versa. The SFT step still worked and gave comparable results. We have updated the paper with this finding.\n\n*4. What if we use human-crafted SFT data?*\n\nThis is a great question, but it is outside the scope of this work, which focuses on how to optimally allocate compute/FLOPs between SFT and RL.\n\n*5. The bootstrapping method to obtain insights on hyperparameters may not be reliable. What if the original 240 configurations do not have good coverage of good configurations?*\n\nAs discussed in the general comments, we increased the number of configurations to 1,370 and found that the main conclusions remained unchanged. This increase in coverage further demonstrates the robustness of our bootstrap-based analysis.\n\nWe will further clarify the reliability of our technique. The bootstrap leverages the Central Limit Theorem (CLT) to approximate the sampling distribution of a statistic, allowing us to estimate its expectation and variability without parametric assumptions [Efron & Tibshirani, 1993].\n\n**Goal:** We want to evaluate, for each hyperparameter value $h$, how well it performs when paired with the best possible values of the other hyperparameters. The bootstrap algorithm yields an unbiased estimate for a target statistic given an initial $n$ samples (full runs in our case). Thus, we choose the statistic of interest to be the evaluation score $M$ conditioned on the hyper-parameter of interest $h$ assuming all other hyper-parameters $g$ are optimal:\n\n$$\nT_h(D) = \\max \\{ M(h', g) : (h', g) \\in D, h' == h \\}$$\n\nBootstrapping this yields an empirical distribution over $T_h$ values, providing an unbiased estimate and confidence intervals for the maximum score achievable for $h$ when other hyperparameters are optimal.\n\n**Optimality and Reliance:**\n\nAs the number of training runs $n \\to \\infty$, the bootstrap estimate $\\hat{T}_h$ of the maximum achievable score for hyperparameter value $h$ becomes consistent. That is:\n\n$$\n\\hat{T}_h \\xrightarrow{p} T_h = \\sup \\{ M(h, g) : g \\in \\mathcal{G} \\}\n$$\n\nThis follows from the law of large numbers [Wainwright, 2019], assuming sufficient coverage of other hyperparameters $G$. The rate of convergence depends on the tail behavior of the conditional distribution $M \\mid H = h$. Since our evaluation metric is bounded in $[0, 1]$, the sub-Gaussian assumption holds automatically by Hoeffding’s Lemma [Hoeffding, 1963; Wainwright, 2019], ensuring concentration of the estimate. Under these conditions, the expected bias decays as $\\mathcal{O}(1 / n_h)$, where $n_h$ is the number of runs with $H = h$. All of these are well known results thus why the bootstrap algorithm is an extremely useful and simple algorithm to use to estimate any statistic of interest [Efron, 1979; Efron & Tibshirani, 1993].\n\n\n**References:** \n- Efron, B. (1979). Bootstrap methods: Another look at the jackknife. *The Annals of Statistics*, 7(1), 1-26. \n- Efron, B., & Tibshirani, R. J. (1993). *An Introduction to the Bootstrap*. Chapman & Hall. \n- Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. *Journal of the American Statistical Association*, 58(301), 13-30. \n- Wainwright, M. J. (2019). *High-Dimensional Statistics: A Non-Asymptotic Viewpoint*. Cambridge University Press. (See Chapter 2 on concentration inequalities.) \n\n\n*6. The conclusions on different benchmarks are not consistent.*\n\nIn the deadline version of the manuscript, SFT+RL outperformed RL-only in both benchmarks. In our updated results, SFT+RL remains the best approach in terms of absolute performance for both benchmarks and both model families.\n\n*7. Why do you choose to use LLaMA 3.1 70B to generate the training data for SFT?*\n\nIt does not scale as easily to use human-generated data; a compute allocation study would be infeasible in that setting. In practice, one often has access to a larger model that can collect higher-quality data to train a smaller, more efficient student model.\nScaling the offline training using human-generated traces is difficult, expensive, and time-consuming. We specifically restricted ourselves to the LLaMA family of models as they are open-source and there is no legal liability in using their traces to train other models. Benchmarks like WorkArena require domain knowledge and high-quality reasoning rationales, and prior work [1,2,3] shows that model-generated rationales can match or even exceed human rationales in some tasks. Since our student model is LLaMA 3.1 8B, we chose LLaMA 3.1 70B as the teacher.\n\n1. Gulcehre, Caglar, et al. \"Reinforced self-training (rest) for language modeling.\" arXiv preprint arXiv:2308.08998 (2023).\n2. Agarwal, Rishabh, et al. \"Many-shot in-context learning.\" Advances in Neural Information Processing Systems 37 (2024): 76930-76966.\n3. Wang, Yizhong, et al. \"Self-instruct: Aligning language models with self-generated instructions.\" arXiv preprint arXiv:2212.10560 (2022)."}, {"review_id": "oDTI2xByH5", "reviewer": "Reviewer_Kh9s", "summary": "The authors compare how to optimally allocate compute budgets on supervised fine-tuning (SFT) and reinforcement learning (RL, specifically GRPO) for fine-tuning LLM-based web agents on MiniWob++ and WorkArena.\nThey also do a large hyperparameter sweep and use bootstrapping to compare hyperparameter choices.\nThey find that it's worth spending compute on expensive teacher-generated traces for SFT before starting RL, but that adding RL will outperform pure SFT.", "questions": "1. Can you clarify how you use the Boltzmann distribution in curriculum learning?\n2. Why is your test performance better than your training performance? For example, Llama-3.1-8b-SFT and Llama-3.1-8b-SFT+RL are all stronger on the WorkArena test set rather than the training set. Wouldn't we expect the training performance to be stronger, given that the model saw other seeds of the same tasks during training? This is true for Llama-3.1-8b-SFT and Llama-3.1-8b-RL on MiniWoB++ too. \n3. Can you calculate how many trials were needed to get hyperparameters that were 90% as good as your final, optimal hyperparameters? That would be really helpful for understanding how many experiments other practioners should run in order to claim that they have near-optimal hyperparameters.\n5. How do you measure AUC of success rates?\n4. In Figure 4, right, why is the green line different to the blue line? Shouldn't SFT Warmup be the same as regular SFT? Also, what do the points/markers indicate in this graph?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n* The authors provide the first compute allocation curve. This graph (Figure 4, left) is awesome. It's a compelling and convincing visual argument that SFT+RL outperforms both pure SFT and pure RL. I cannot emphasize the value of this chart enough.\n* The authors do extensive hyperparameter analysis, using bootstrapping to report confidence intervals, across various amounts of pre-RL SFT. I am really happy to see an extensive statistical analysis of web agents; as the authors correctly note, the field of RL+LLMs seems to be rediscovering many of the deep RL reproducibility challenges.\n\nWeaknesses:\n\nThe biggest weakness is a lack of clarity in the writing.\n\n* How is reward defined in web tasks? On line 76, it says $r_t \\in [-1, 1]$. When does the model receive reward? Based on the writing around \"sparse rewards\", I assume it's only at the end of an episode. Does the model ever receive negative reward? Or is it really $r_t \\in {0, 1}$?\n* The curriculum learning via a Boltzmann distribution; what is the form of the Boltzmann distribution? I am not familiar with the distribution, and Wikipedia describes it as $p_i \\propto \\exp(-\\frac{\\epsilon_i}{kT})$ but the authors describe a target return $\\mu_\\text{target}$. How does the training algorithm select tasks? \n* Lines 132-134: \"Here the cost of $T_\\text{RL}$ steps $F_\\text{RL}(T_\\text{RL})$ is equivalent to the sum of FLOPs for one data generation step and one training step of the student model.\" Do you mean the sum of flops for $T_\\text{RL}$ data generation steps and $T_\\text{RL}$ training steps for the student model?\n* Line 237 says that 14 out of 33 WebArena tasks remain unsolved, but Figure 3 says 13/26 remain unsolved. Which is it?\n* In Table 1, there are +/- numbers with no explanation. Are these standard deviations over the top 2 runs?\n\nWhile I appreciate the statistical analysis of the hyperparameter sweep, I believe the conclusions are unlikely to generalize to other models or tasks due to the high variance in \"conclusions\". I would prefer the authors shift the messaging from \"we used a statistical procedure and arrived at these general conclusions for LLM RL training\" to \"we used a statistical procedure and optimized our hyperparameters accordingly; here's how you can use the same procedure.\" Specifically, it would be really useful to see how many of those 240 runs were needed to arrive at the optimal hyperparameters.", "was_revised": true, "final_justification": "The authors did additional experiments and clarified many of the confusing written aspects (graphs, tables, etc).", "rebuttal": "We thank the reviewer for their thoughtful and detailed review, as well as the positive comments regarding the value of our compute allocation curve and extensive statistical analysis. We address both the general improvements made since submission and each of the reviewer’s specific points below.\n\n## General Comments\nSince the submission deadline, we have significantly expanded and strengthened our results:\n\n* **Expanded the hyperparameter search for the bootstrap experiment (from 240 to 1,370 runs):** We increased the size of the random hyperparameter sweep by more than 5x. Importantly, the main conclusions reported in the original submission remain unchanged, demonstrating the robustness of our findings. We believe these results showcase that the proposed bootstrap technique is robust and sample-efficient, effectively finding the same conclusions even when scaling the number of runs 5x. The only minor update is that using standard-deviation normalized advantage appears moderately better than not using it when starting RL from the instruct model.\n\n* **Generality across benchmarks:** We performed additional WorkArena experiments. Our updated results confirm that the main conclusion from MiniWoB++—that combining SFT and RL achieves the best absolute performance—also holds on WorkArena. We could also get some stable runs for Pure RL, improving the held-out task performance by 7.5%.\n\n* **Generality across model families:** We ran experiments on Qwen2.5 and replicated the main conclusion: SFT+RL is the only method able to match or exceed teacher performance. This provides evidence that our findings generalize beyond the LLaMA family of models.\n\n* **Bootstrap reliability:** We discuss the consistency and unbiasedness of the bootstrap estimator as well as the statistic we are specifically interested in estimating. Additionally, we provide examples of the reliability of our estimate in terms of standard error as the number of runs increases, showing that even under pessimistic assumptions we can maintain a relatively small standard error.\n\n## Reviewer-Specific Comments\n\n*How is reward defined in web tasks? On line 76, it says . When does the model receive reward? Based on the writing around \"sparse rewards\", I assume it's only at the end of an episode. Does the model ever receive negative reward? Or is it really ?*\n\nThe model receives a positive (+1) or negative (-1) upon the completion of the episode based on its usefulness. We are sorry about the confusion around “sparse” rewards. By sparse, we meant that the only reward that the model receives is at the end of the episode, i.e., no intermediate rewards for partially completing the tasks. We adjusted the paper to clarify this in the Experimental Setup’s part on benchmarks.\n\n*The curriculum learning via a Boltzmann distribution; what is the form of the Boltzmann distribution? I am not familiar with the distribution, and Wikipedia describes it as but the authors describe a target return . How does the training algorithm select tasks?*\n\nBoltzmann is just a softmax. The target return is the mode of the distribution, e.g., if target return is 50%, tasks close to 50% success rate will have the highest likelihood to be sampled by the algorithm. We clarified the manuscript accordingly.\n\n*Lines 132-134: \"Here the cost of steps is equivalent to the sum of FLOPs for one data generation step and one training step of the student model.\" Do you mean the sum of flops for data generation steps and training steps for the student model?*\n\nThe reviewer is right that the sentence was confusing and was right about the correction. We updated the manuscript.\n\n*Line 237 says that 14 out of 33 WebArena tasks remain unsolved, but Figure 3 says 13/26 remain unsolved. Which is it?*\n\nIf you sum the 13/26 unsolved training tasks to the 1/6 unsolved test tasks, you get 14/33 unsolved tasks.\n\n*In Table 1, there are +/- numbers with no explanation. Are these standard deviations over the top 2 runs?*\n\nThe reviewer is right, and we are sorry for this omission. The +/- refers to standard errors. In line 202 we indeed say that the average is the two top seeds.\n\n*While I appreciate the statistical analysis of the hyperparameter sweep, I believe the conclusions are unlikely to generalize to other models or tasks due to the high variance in \"conclusions\".*\n\nTo address concerns about whether our conclusions generalize beyond the LLaMA family, we ran a very restricted hyperparameter random search of Qwen2.5 on MiniWoB++, taking into account the findings from the LLaMA bootstrap. We only swept over learning rate (LR), batch size (BS), and sampling temperature, fixing all other hyperparameters to those identified as optimal in the LLaMA experiments. From our sweep, we found that LR and BS are consistent with the results reported for LLaMA, with sampling temperature being higher at 0.75, which falls within the guidelines provided by the Qwen team.\nThe updated MiniWoB++ results are shown below in the Rebuttal of Reviewer zHWC, DhYQ and jEdW, due to space constraints.\n\nWe were able to replicate the MiniWoB++ result that SFT+RL achieves the best absolute performance with a new model family, namely Qwen2.5. It is also the only method able to surpass the teacher. Interestingly, we could not improve the performance of the instruct model using Pure RL in this case. After a few steps, the model often failed to follow the correct output format, began producing repeated tokens, or switched to unrelated languages. Nevertheless, the conclusions drawn from the LLaMA experiment still hold. We are actively working on the compute allocation study for Qwen2.5, which is compute-intensive. We will include these findings in the next version of the manuscript.\n\n*Can you clarify how you use the Boltzmann distribution in curriculum learning?*\n\nWe thank the reviewers for bringing this point up, and we will clarify the formula in the paper to allow for better comprehension. To clarify here, the Boltzmann distribution is a well-known family of distributions that we are sure you are familiar with. In short, this distribution gives rise to the softmax and logistic loss functions traditionally used in classification. For simplicity, you may think that we sample from curriculum learning using a temperature-annealed softmax function:\n$$\\frac{e^{\\mu_{\\text{task}}/\\rho_{curr}}}{\\sum_{\\text{task}}e^{\\mu_{\\text{task}}/\\rho_{curr}}}$$\nThus, we calculate the probabilities for each task using this function and use them to sample tasks during training.\n\n*Why is your test performance better than your training performance? For example, Llama-3.1-8b-SFT and Llama-3.1-8b-SFT+RL are all stronger on the WorkArena test set rather than the training set. Wouldn't we expect the training performance to be stronger, given that the model saw other seeds of the same tasks during training? This is true for Llama-3.1-8b-SFT and Llama-3.1-8b-RL on MiniWoB++ too.*\n\nThis is simply a feature of our random splits, which can randomly make the test set easier. For both benchmarks, the instruct model tends to do better on the test than on the train, but the tuned models close the gap, in line with the reviewer’s intuition.\n\n*Can you calculate how many trials were needed to get hyperparameters that were 90% as good as your final, optimal hyperparameters? That would be really helpful for understanding how many experiments other practitioners should run in order to claim that they have near-optimal hyperparameters.*\n\nDue to the CLT the standard error of the bootstrap estimate is bounded by a rate of $\\mathcal{O}(1/\\sqrt{n_h})$ thus we can make a comparison of optimality based on the standard error. For example, assuming a binary hyperparameter like applying advantage is evenly split across sampling groups, this gives us approximately $n_h = 120$ runs for each configuration from our initial 240 runs. To achieve a standard error that is only 10% less effective (i.e., 10% larger), we would need:\n $$\nn' = \\frac{n_h}{1.1^2} = \\frac{120}{1.21} \\approx 99\n$$\n\nThus, approximately 99 runs are needed for this hyperparameter to obtain a confidence interval whose width is only 10% larger than with 120 runs.\n\nTo further put this into perspective, since bootstrap exploits normality due to the CLT and since our scores are bounded in $[0,1]$, we can obtain performance bounds even under conservative variance assumptions.\n\nFor instance, assume $\\sigma = 0.25$ (which corresponds to a variance of 0.0625, and is still larger than typically observed in our experiments). Then:\n\n- With 100 runs:\n $$\n \\text{SE}_{50} = \\frac{0.25}{\\sqrt{50}} = 0.035\n $$\n- With 120 runs:\n $$\n \\text{SE}_{120} = \\frac{0.25}{\\sqrt{120}} \\approx 0.0228\n $$\n\nThus, even under the assumption of a high standard deviation in scores using even less than half our runs yields an estimate that is only ~15% worse than ours. While this may still be expensive for some labs, the purpose of our experiments are to take this into account and try to be able to provide robust recommendations. \n\n*How do you measure AUC of success rates?*\n\nThis is clarified in the new version. Essentially, it is the mean success rate across all checkpoints of the run, serving as a proxy for overall training efficiency.\n\n*In Figure 4, right, why is the green line different to the blue line? Shouldn't SFT Warmup be the same as regular SFT? Also, what do the points/markers indicate in this graph?*\n\nThis is because the SFT line is the best performing absolute run, which was not the one that learned the fastest (the one we branched off from for RL fine-tuning). We understand how this could be confusing. We are working on a better way to report some compute allocation insights on WorkArena for the next version.\n\n*N/A. You misspelled practice as \"prectice\" in your related work (and in the appendix).*\n\nThanks! Fixed."}]

Author rebuttal:

Meta-review: This paper makes a valuable empirical contribution to training LLM web agents by providing the first systematic analysis of compute allocation between supervised fine-tuning and reinforcement learning. The authors used extensive experiments to study resource allocation for SFT and RL for training LLM web agents. After rebuttal, most reviewers agree that this paper makes significant contributions to the community. Therefore, I recommend acceptance.
Decision: Accept (poster)