Thinking vs. Doing: Improving Agent Reasoning by  Scaling Test-Time Interaction
Authors: Junhong Shen
Hao Bai
Lunjun Zhang
Yifei Zhou
Amrith Setlur
Shengbang Tong
Diego Caples
Nan Jiang
Tong Zhang
Ameet Talwalkar
Aviral Kumar
Venue: NeurIPS 2025 poster ()
Keywords: LLM Agent
Web Agent
Reasoning
Primary area: deep_learning
TL;DR: We propose to scale the number of interaction steps for agents as a new axis of test-time scaling and develop a curriculum-based online RL algorithm for training agents to scale interaction.

Abstract: Test-time scaling in agentic tasks often relies on generating long reasoning traces ("think" more) before acting, but this does not allow agents to acquire new information from the environment or adapt behavior over time. In this work, we propose scaling test-time interaction, an untapped dimension for test-time scaling that increases the agent's interaction horizon to enable rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we situate our study in the domain of web agents. We first show that even prompting-based interaction scaling can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI, a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their interaction lengths during rollout. Using a Gemma 3 12B model, TTI sets a new state-of-the-art among open-source agents trained on public data on WebVoyager and WebArena. Case studies further reveal that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-action compute, offering new avenues for training robust and adaptive agents.

Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction
Junhong Shen 1.2* Hao Bai
Lunjun Zhang'
Yifei Zhou
Amrith Setlur
Shengbang Tong?
Diego Caples
Nan Jiang'
Tong Zhang
Ameet Talwalkar'
Aviral Kumar'
CMU
SSribe
UIUC
'U of Tronoto
UC Berkeley
TThe AGI Company
"YYU
Abstract
The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting bassid interaction scaling without any training can improve task success on web benchmarks non-trivially. Building this, introduce TTI (Test-Time Interaction), reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WeyVoeyger and WebArena benchmarks. We further show that TTI enables agents balance exploration and exploitation adaptively. Our results establish interaction scaling as powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents. Project page: http:://tes-time-nnteraee github. Code: https://github.com/ess iome-interactionTT1
murriculum-based online
Introduction
Recent advances foundation models have enabled a shift from static language models to interactive agents that perform multi-step tasks in dynamic environments like browsers [1-6], terminals [7], and the physical world [8-13]. These agents operate closed-loop settings where each action changes the current state of the world and affects future interaction with the environment. As a result, interactive agents must plan under uncertainty and adapt to failures real time to be successful. How can we build agents that succeed in such interactive settings?
Current post-training approaches produce reactive agents that respond to immediate observations but struggle with evolving or uncertain task dynamics. Methods like supervised fine-tuning (SFT) on expert demonstrations [14-18] or reinforcement learning (RL. with task rewards [19-23] typically train agents to predict single best action at each step. Even with test-time scaling, where agents
Equal contribution. Correspondence to: junhongr@andeev.omm.cco andrew.cmu.edu, haob2 illinois.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Compute Decomposition
Scaling Test-Tima Compute
Scaling Test-Time Interaction
7hat WobArena Evaluation
Find oheopest Boston on May 1S...
T.kar Pee Shop
Inseorth le leck lus
420
Fia
Clok
seraaionsiaing
200
hital
rig
pape
pages
Dook
Stape
Tragessry
Act More
Pach Stop, Thes ooggrr
More Stepi, Abrra Parabiliia
Figure 1: We propose a new-axis of test-time scaling for agents: scaling the number of interaction steps. Unlike traditional methods that emphasize longer reasoning per step, we show that acting more helps gain new information from the environment and improve task performance (detailed results of the left plot in Section 4.2).
prompted to "think" longer before prescribing action [24-26], they are still optimized to select the most effective action based on the agent's internal state. While sufficient for fully observable and stationary tasks, reactive policies based on the agent's internal estimate of the task state are often sub- optimal in partially observable (e.g., incomplete details visible on page) or non-stationary (e.g., fluc- tuating prices during flight booking) settings, where adaptive, inforintion-seekinng behavior is critical.
In this paper, we argue that instead of reactive "optimal" policies, agents should learn adaptive policies that can collect new information from the environment and adjust their behaviors on-the-fly. A pre-requisite for such adaptability is the ability to take more actions during deployment than those prescribed by expert trajectory. We therefore propose new dimension of test-time scaling: increasing the number of interaction steps of the agent. This allows agents to have sufficient context and time to attempt different behaviors. For example, in hotel booking task, an agent must first browse many listings to compare user reviews and check availability before selecting the best option. Interaction scaling is orthogonal to existing methods based on chain-of-thought (CoT), which emphasize deeper reasoning per step but do not support information -aiaaatinnggterringggg from the environment. This notion of information gain is unique to agentic tasks with partial observability and requires interaction, not merely larger per-step compute. For instance, an agent that reasons deeply about one selected hotel without interacting further may miss better options that show up only after exploration.
Although the idea of interaction scaling is conceptually straightforward, extending it to post-training and teaching agents to scale interaction autonomously presents key challenges. Without appropriate training signals, agents may overfit to exploratory behaviors like blindly clicking links but not making progress toward the actual task objective, wasting the additional steps. To tackle this issue, we propose to combine online RL with a curriculum that prescribes how to scale the interaction horizon, training agents that first learn effective exploitation before extending their horizon to explore.
We instantiate our approach in the domain of web agents, a widely applicable setting with well- established benchmarks. We first show that scaling test-time interaction via prompting the agent to "think and act again" " after it decides to terminate can already improve the task success rate from 23% to 2 28% WebArena [2] (see Figure 3 for details). While this increases trajectory length and the number of tokens generated, spending an equivalent amount of compute on conventional test- time scaling methods like forcing the agent to think for longer [27] running best-of-n [28-30) yields less than a 3% gain. These findings validate the potential of interaction scaling.
We then move beyond prompting and develop TTI (Test-Time Interaction), curriculum-baas based RL approach that trains agents to adaptively scale interaction by gradually increasing the rollout horizon. We scale TTI to large and diverse training sets (> 100K tasks across ~20 domains) by integrating an automated task generation pipeline [19]. TTI achieves sfate-of-the-arr performance among open-source agents trained on open data on both Web Voyager [1] and WebArena [2], using only a 12B Gemma 3 model, improving over the non-fine- con-fin-tumedd agent by 9% and 8%, respectively. Our analysis further shows that curriculum training enables adaptive exploration: agents lear to initiate new searches or backtrack in complex tasks, while following efficient paths in simpler ones.
In summary, we introduce interaction scaling as new dimension for test-time scaling of agents. We propose TTI to train agents by adjusting interaction horizon dynamically. Our results show that TTI yields strong empirical gains and offers promising directions for domains beyond web navigation.
2 Related Work
Scaffolded foundation models as web agents. Prior works use external control structures to scaffold foundation models via modular prompting [31. 6, 32-37], programs [38?- -41], or feedback
2

---

mechanisms [42-45]. These methods often rely on proprietary models like GPT-4 [46] or Claude [47].
Thus, progress is driven by designing better prompts and workflows for planning [48-51], self- correction [52], self-evaluation [53, 54], or by integrating external modules such as memory [55] or retrieval systems [56]. More recently, developing specialized agents has become a promising direction. ScribeAgent [14] first uses real-world data to demonstrate that simple fine-tuning can outperform most prompt-based agents. Prior work also builds automated data curation workflows [57-59] and distillation methods [18]. Despite these efforts, scaffolding methods do not enable agents to improve through interaction, and rely on fixed wrappers that lack adaptability across diverse environments.
RL training for foundation model agents. RL-based approaches enable agents to autonomously im- prove through interaction. Prior work has explored DPO [60]. actor-critic [20, 21, 61], or distributed sampling [62]. Pipelines like PAE [19] and Learn-Ba Learn-By-Inteen By-Interactt [63] support automatic task genera- tion, exploration, and labeling. However, most of these approaches lack mechanisms for test-time exploration, limiting the agent's 's ability to adapt its behavior over long horizons, especially under partially observable conditions. Our work addresses this limitation by scaling test-time interaction as an independent dimension, allowing agents to refine behavior while acting. purriculum-basee RL has been applied in Auto LutoWeGLMM [64] and WebRL [23], where curricula are based on estimated task difficulty derived from the complexity of LLM-generated instructions. While our approach also induces a progression from simpler to more complex behaviors, it does so by gradually increasing the allowed interaction horizon rather than relying on explicit measures of task difficulty.
Scaling test-time compute. Increasing test-time compute via best-of-n sampling [29]. beam search 165, 66], or verifiers [67-69] has shown to improve performance in reasoning-heav tasks. In non-interactive settings like math and competitive coding, recent methods train models to generate long CoT and scale reasoning internally [e.g., 70, 27, 71? 1. As for multi-turn interactive settings, most existing works simply integrate CoT prompting into the agent system to enhance per-step reasoning [e.g., 72, 51J. EXACT [73] scales up the search process for each action, GenRM-CoT [74] the number of verifiers, and Jin et al. [75] the number of agents. However, none of these efforts studies the benefits of scaling over the time horizon, where the agent can explore alternatives, backtrack, or gather more information before committing to certain actions. Our work extends this line of research by introducing test-time scaling of interaction. As we will show in our empirical results (Section 4.2), the benefits of scaling test-time interaction go beyond test-time scaling (or "rcasoning") within a given time step, because each extra step of interaction with the environment provides new information to the policy, whereas thinking for longer simply reorganizes information that the agent already has.
3 Problem Setup
We consider solving a web task as finite- horiz-n sequential decision-maine making process guided by a reward objective". Formally, the environment implements a transition function that evolves over time and provides an observation o, at step reflecting the current task state. The agent policy is parameterized by multi-modal foundation model that maps observation history 014-1 and action history d1 to the next action dz These histories allow the policy to represent rich, context- dependent behaviors enabled via interaction. We denote the environment horizon, or the maximum number of interaction steps allowed in the environment, as h. For each task, the actual interaction process ends when the agent issues stop signal or reaches the step limit h. Let hatop E (0, h] denote the actual number of steps taken. The agent receives a reward of for task success, and 0 otherwise.
Observation space design. Following [14, 19, 2], we consider an observation space consisting of the task goal, the URL, and a structured representation of the current web page that includes both the accessibility tree of the web page and screenshot augmented with a set-of- marks overlay [1], which assigns a unique identifier to each element that the agent can interact with. While the agent has access to all past observations in principle, doing so quickly exhausts the context window in practice, so we truncate observation history to the most recent three steps, similar to prior works [19]. However, the agent still has access to all of its past actions. We show in Appendix B an example observation.
click, type, scroll, go back, search (e.g., Google or Bing), and stop the task with an answer. Following Yang et al. [33], we do not consider compound actions like goto [url], as complex action spaces can hinder performance, For detailed definition of each action, see the agent prompt in Appendix C.I.
Action space parameterization. We adopt discrete action space with six actions:
'While this work centers on web agents, we believe the insights should generalize to other agent problem domains, and we hope future work will extend these ideas beyond web agents and web navigation.
3

---

4 Scaling Test-Time Interaction: A New Dimension of Agent Scaling
Prior methods for LLM test-time compute scaling usually scale the number of thinking tokens at each step [76? ? ? J. but this does not enable an agent to engage in longer interactions with the environment to collect new information. In principle, scaling the maximum number of interaction steps should allow the agent to employ richer behavioral strategies such as re-attempts, backtracking, and recovery. We will now verify this hypothesis via controlled experiments WehArena [2]. We will then build upon these insights to develop 17I, which trains agents to optimize test-time interactions.
Controlled experiment setup. We choose WebArena [2] testbed, primarily because it enables reproducible interaction with diverse domains (OneStopShop, Reddit, GitLab, CMS, and Open- StreetMap) and is equipped with ground truth evaluators. We randomly sample 62 tasks for testing and reserve the remaining 750 for online training (see Section 5.1). To ensure sufficient interaction, set a generous test-time limit of h 30, which is well above the average length of around steps required by most tasks [55, 77J. Note that experiments in this section are for analysis rather than bench- marking purposes. We will show more experimental results multiple benchmarks in Section 6.
To study the effect of increasing h, we use a simple prompting- based agent with Gemma 3 12B [78] base model, which observes the web page and outputs an action via a single model call. does not leverage any retrieval, verifiers,
Table 1: Results aver-
aged over three runs.
Prompt
Task SR (%)
or other external modules, ensuring any performance gains come solely from
Action Only
14.76
increased h but not auxiliary scaffolding. We also study whether it is ben-
23.81
eficial to prompt the agent to generate reasoning trace before acting (see Appendix C.I for the templates). As Table 1 shows, CoT prompting yields significantly higher task success rate (SR) than direct action generation, setting baseline of 23.8 18%%%%% on the test tasks. We thus adopt a simple chain-of-thought (Co'T) approach as the default prompting strategy.
4.1 Scaling Test-Time Interaction by Acting Longer
To study the impact of interaction scal- ing, we introduce a purely inference- time "check-again" mechanism: after the agent issues task completion, we explic- itly prompt it to reconsider its decision by "You just signaled task completion. Let's 's pause and think again..." We can extend re-checking to multiple checks. Detailed prompts are in Appendix C.4.
Task SR (Bars)
30
Avg Step (Dob)
0.4
20
0.2
10
0.0
Average
Reddit GtLab
Map
Shapping
CMS
Figure 2: Scaling test-time interaction by prompting the agent to "re-check" its answer.
As shown in Figure 2, prompting the agent to re-check not only increases the interaction length hato (line plots), but also improves the success rates on most domains (bar plots). When being asked to "check-again"" the agent either reaffirms its decision (e.g., "I previously stated the driving time was approximately 30 minutes....30 minutes seems plausible with typical traffic conditions. I'll stick with my previous answer.") or revises it upon reflection (c.g., "My apologies. Although the address book *displays* the desired address, the task requires me to *change* it.... should click button [24] to modify the address.") "). In particular, it changes its action ~5% of the time after double-checking. This highlights the potential of interaction scaling: when given sufficient time, the agent is likely explore alternatives before reaching an answer. The chances of the answer being correct could thus be higher.
However, we do observe that repeatedly prompting the agent re-check can sometimes lead to confusion or hallucination, causing it to revise correct answers into incorrect ones. This may explain the performance drop observed in domains like Map. This is perhaps inevitable limitation of scaling test-time interaction via prompting alone, akin to how prompting is not an effective way to even scale per-step test-time compute (see self-correction results for prompted models in ? J). We discuss this limitation further in Section 4.2 and address by training the agents explicitly.
4.2 Scaling Teme Interaction vs. Per-Step Test-Time Compute
Next, we examine the effect of scaling interaction compared to scaling per-step reasoning: Given a total token budget, should agents prioritize more interaction steps or generating longer reasoning traces at each step? To explore this, we study two conventional test-time compute scaling methods.

---

Per-step budget forcing, Following Muennighoft et al. [27]. we prompt the agent to "wait and think again" after generating an initial CoT, encouraging more intermediate reasoning before commits to an action. We vary the number of forced waits from 1 to 4. Despite trying various prompts (see Appendix C.4), our agent changes its actions only 15% of the time, and most of these changes involve reordering subtasks rather than exploring alternative paths or error correction.
Per-step best-of-n. At each step, we sample n E {3, 5, 7 candidate actions and select one via majority voting, similar to [14]. We did not scale n up further because best-of-n is compute-intensing for multi-step settings (n- h more expensive than the baseline per rollout), and we observe diminishing returns from sampling more actions, despite tuning sampling hyperparameters such as the temperature.
Finding 1:
Gaining new information via interaction beats thinking more a single step. Figure 3 (top)
Task Success Rate
0.30
Per Per-Stap Budget Fording Per-Step Beet-of- Intrratiin Scaling
plots the task success against total compute, measured by the number of tokens per trajectory in log scale. Among the three strategies, interaction scaling (green)
0.28
0.26
shows the steepest upward trend, achieving the highest success rate as the allowed token budgets increase. Bud- get forcing (blue) yields moderate gains but plateaus around 0.26. Despite incurring the highest cost, best-of- (orange) brings the least improvements, suggesting that repeatedly sampling actions per step is less effec- tive use of compute in interactive tasks.
0.24
0.22
6.5
7.5 Tokens Per Trajectory (log)
7.0
8.0
# Tokens Per Step
Per-Stap Budget Fording Per-Step Best-o-- Interaction Bealing
400
Compute Decomposition
A natural question that arises here is: how should distribute a bounded compute budget between running more interaction steps vs. reasoning longer? These two dimensions present different costs per unit, which may not be known apriori. Hence, illustrate how scaling the number of interaction steps and the number of to- kens per step affect performance and the total number of tokens individually. Figure 3 (bottom) decomposes total compute into tokens per step (y-axis) and steps per rollout (x-axis), so the shaded areas indicate the aver- age total compute per trajectory. Interaction scaling extends along the x-axis, while per-step reasoning scales along y-axis. We find that scaling across steps is more effective than scaling within steps in WebArena (Figure 3, top). likely because the former enables the agent to gather new information and enrich its context. This ability to query and observe external feedback is unique to agentic settings but not single-turn question-answering tasks. While standard per-step reasoning is constrained by the information already available at each step, our approach takes advantage of this dynamic interaction.
200
6
10 12
14
Steps Per Trajectory
Figure Top: Task success rale vs. total compute cost measured in tokens per trajectory, log scale. Bottom: Decompose compute into tokens per step and number of interaction steps (excluding the initial point for each approach).
Finding 2: Prompting alone insufficcentt for interaction scaling. While our results highlight the potential of scaling interaction, the "check-again" strategy only allows the agent to revisit its behavior upon task completion, does not enable to implement nuanced behaviors such as switching between exploration and exploitation in the middle of a rollout. We also experimented with combining interaction scaling with budget forcing and best-of-n (Appendix Table 6) and observe that simply increasing rest-time compute via prompting does not yield additive gains. In fact, the final downward trend in Figure 3 (top) suggests that asking the agent to re-check too many times or think for too long can confuse it and degrade performance. This shows the need for methods that train agents to optimize for best behavior when scaling test-time interaction, rather than naive prompting.
5 TTI: Curriculum-Basel Online RL for Scaling Interaction
How can we extend beyond prompting- based scaling to training agents that can effectively utilize interaction scaling? A natural starting point is to draw inspiration from current approaches for optimizing test-time compute [30, 71, 70] and extend these ideas to interactive settings. Specifically, we can run reinforcement learning (RL) with binary task rewards and longer task horizons. However, this approach sufficient? We first describe the key challenges in learning to scale test-time interaction, and then develop our approach to address them via curriculum learning.
5

---

Task Success Rate
All Domains
Harder Domain: CMS
Easier Domain: Reddit
Avg. Interaction
All Domains
0.60
0.03
0.25
0.2
0.10
0300
Training Step
Training Step
Max Rollout 5
Mox Rollout Interaction
Max Roliout Interaction 20
Figure 4: Online RL with different values of maximum interaction horizon. Left: success rates for different domains. "Harder" means lower success rate. Right: average rollout length (hssoo) on evaluation tasks,
5.1 Challenges of Training Agents with Fixed, Long Horizons
natural way to encourage the agent interact train at long horizons. To study this, we run the simplest form of REINFORCE [79] with binary rewards R(-), also known as online filtered behavior cloning (BC) or online STaR [20?] Only successful trajectories are retained, and the agent is updated by maximizing the log-likelihoon of actions conditioned on those high-reward rollouts:
arg max
wasks E Boos -
log (a oge, 7)
1
R(oo:h+ T) =1
reward
likelihood of trajectory
We use filtered BC as is stable throughout training (no negative gradient [? D. has been utilized previously for training agents [20], and is good "first-choice" algorithm for studying the role of interaction scaling. We scale the agent's horizon on WebArena, varying / E (5, 10, 203. Smaller ex- poses the agent only to exploitative rollouts that succeed within allowed time steps, while larger h also includes more exploratory rollouts. We use the non-test tasks for rollout. Details are in Appendix D.
As shown in Figure 4 (left), the agent trained with h = 5 learns quickly, likely because on-policy RL more sample-efficient smaller horizons, but it also quickly overfits, and performance decreases with more training (X-axis) This agent often terminates prematurely during evaluation despite being allowed to interact for much longer time. Conversely, agents trained longer horizons generally learn policies that are quite stochastic and learn significantly more slowly due to higher variance of the loss and credit assignment challenges due to longer horizons [e.g.. 80-82]. We manually inspect the trajectories and find that the = 20 agent tends to associate exploratory actions such as "going back" or "trying random links" with high rewards initially. Noisy credit assignment with h 20 slows learning, and only after several iterations do the agents begin to recover and produce more robust policies. The impact of horizon is domain- -deeendnttedd::: in complex domains requiring exploration like CMS, long-horizon agents outperform, while in simpler settings like Reddit, performance differences are minimal. We also note that the total number of tokens generated per action slightly decreases throughout training, so the training does not incentivize increasing the length of reasoning directly.
Importantly, although the interaction length increases expected for h 20 (Figure 4, right), noisy credit assignment and slower learning suggests that simply setting / to be large is insufficient to learn scale test-time interaction reliably. These observations motivate our core idea: rather than fixing the horizon throughout training, our approach aims to scale their interaction length dynamically.
5.2 Our Approach: Curriculum Over Interaction Horizon
To address these challenges, we propose TTI (Test-Time Interaction), curriculum -asdd online RL approach that trains the agent with short trajectories initially and gradually exposes it to longer ones. Existing curriculum learning methods in RL [e.g., 83-87] or web agents [23, 64] typically prioritize easier tasks before harder ones, often relying on predefined heuristics or external measures of task complexity. In contrast, we define curriculum progression in terms of the maximum number of steps an agent is allowed per trajectory. While interaction length can serve as a rough proxy for task difficulty, our approach does not require explicit labeling estimation of task complexity.
How do we design a curriculum over the interaction horizon h.? Ideally, the curriculum should allow the agent to first learn basic, "atomic" skills to solve easier tasks, then progressively tackle complex

---

ones via skill chaining and exploration. To enable this, we explored two strategies: (1) a conservative, additive increase in per iteration, giving the agent sufficient time to solidify core solving skills; and (2) a more aggressive, multiplicative increase, which assumes the agent can quickly acquire the basic skills and benefit from earlier exposure to exploratory behavior, Formally, for iteration
h; clip(hmin + i, Nimax) (Additive curriculum)
(2)
hy == clip(hmin i, hmax) (Multiplicative curriculum)
(3)
We store the rollouts in replay buffer and assign higher weights to more recent trajectories. The full pseudocode for TT/ and implementation details are provided in Appendix E.
Empirical insights. We instantiate these two strategies in WebArena, using the non-test tasks for online training. We set Amin to 10 and lmmax to 30, and apply the schedules on top of filtered BC. Evaluation results after 10 iterations shown in Table 2. Multiplicative curriculum outperforms the additive one, possibly because it exposes the agent to longer horizons early on and helps prevent from overfitting prematurely to shortcut behaviors like al always taking the shortest path. Based on these findings, we adopt the multiplicative curriculum as the default for TTI. Table 2 further shows that even with limited data (~700 training tasks), TTI outperforms fixed h = 20 in Figure 4 by nearly 3%, using 40% fewer training steps over 10 iterations. Next, we demonstrate that this advantage carries over to large-scale online RL training.
Table 2: Comparing various curricula. multiplicative curricu- lum produces the best success rate.
Schedule
Task SR (%)
Additive Multiplicative
29.50
32.25
Experiments: Scaling Up to Realistic Benchmarks
We now provide a comprehensive evaluation of TTI large-scale, realistic environments. More gen- erally, we demonstrate the effectiveness of training models to make good use of test-time interaction.
Synthetic task generation and evaluation. To enable large-scale training without training on the benchmark itself, we adopt synthetic task generation inspired by PAE [19]. Apart from prompting LLMs with seed examples or demos, we also deploy an exploratory agent to freely interact with websites and propose more grounded, diverse tasks. For evaluation, we leverage prompting- based verifier that uses action histories and screenshots to label rollouts, achieving 88.9% accuracy compared to WebArena's 's ground-truti evaluator. Details and prompt templates are in Appendix F.1. We evaluate agents on (1) Web WebVoyager [1] with 427 tasks across 13 domains (we replace Google search with Bing due to ReCaptcha issues); and (2) full WebArena [2] with 812 tasks.
Training. We obtain 128K synthetic tasks across diverse real-world domains for WebVoyager and 11K tasks for WebArena's 's self-hosted domains (data released with the code). We train separate agents for each benchmark avoid contaminating real-domain agents with synthetic environment data, as WebArena domains still have many differences from their real-world counterparts. We use Gemma 12B [78] as the base model, sampling 512 tasks per iteration for rollouts and updating with 512 successful on- policy trajectories. We apply multiplicative curriculum with Amin = 10 and max 30. We use vLLM [88] to sample rollouts and use DeepSpeed Zero 3 [89] with NVIDIA H100 GPUs for training. The evaluator is a Gemma 3 27B model, prompted to detect successful trajectories, which can then be used for the online filtered BC procedure. Other hyperparameters such as the number of iterations, learning rate, and the exact schedule of TTI can be found in Appendix F.3.
Comparisons. We evaluate zero-shot Gemma 3 12B and approaches that utilize a fixed horizon with l E {10, 303. We also compare to closed-source agents (e.g., those based on GPT-4 [46] and Claude [47]), open-weight models trained on proprietary data (e.g., UI-TARS [90]), and fully open-weight, open-data models (e.g., PAE [19]).
6.1 Web Voyager Results and Analysis
State-of-the-art open-weight, open-data performance. We report the overall task success rates (SR) on Web Voyager in Table and Figure ?? (left). The TTI -Tl-tneieed Gemma 3 12B achieves an average SR of 64.8%, setting a new state-of-the-art among open agents trained purely on public data. While previous methods such as UI-TARS achieves strong SR of 84.8%, they rely on private human-annotated data that remains inaccessible to the open-source community. In contrast, TT7 trained entirely on synthetic data and interestingly, this data is generated by the base model (Gemma
7

---

Table 3: Web Voyager results. TTI attains the best performance among open-weight agents trained on open- source synthetic data, Baseline results are taken from Zhou et al. [19]. Qin al. [90].
Model
Average Alliocipes Amazon Apple ArXiv Gatfub ESPN Coursera Cambridge BBC Map Search HuggingFace WolframAlpha
Proprictary Model
Clarde 3.7
81.1
Clande 3,5
50.5
50.0
68.3
60.4
46.5
58.5
27.3
78.6
86.0
36.6
58.5
30.2
44.2
66.7
OUA
87.0
Agont E
73.1
71.1
70.7
74.4
62.8
82,9
77.3
85.7
814
73.8
87.8
90.7
81.0
95.7
Open Model, Proprictary Noman-Annotatedd Data
U1-TARS-1.5
84.8
Open Model, Open Synthetic Data
LLaVa-348 SFT
22.2
688
26.8
23.3
16.3
4,9
8.6
26.8
67.4
16.7
12.2
23.3
20.9
38.1
PAE-LLaVa-7B
14.3
37.5
17.5
19.0
52.4
23.3
PAE-LLaVa.34B
33.0
22.7
53.7
38.5
25.6
14.6
13.6
42.9
74.4
39.0
22.0
18.6
25.6
42.9
Gemmaa 3 128
55.8
25.7
32.3
45.5
60.6
$4.8
69.6
56.3
69.6
65.6
54,8
72.7
66.7
61.1
Fixed
25.7
51.5
Fixed - 30
45.2
20.0
41.9
60.6
42.4
41.9
50.0
34.4
60.6
25.0
29.0
63.6
45.5
69.4
T77 (Ours)
64.8
57.1
48.3
69.6
66.6
45.2
56.3
46.9
85.2
81.2
66.7
72.7
75.7
79.4
3 12B) itself, meaning that our training protocol implements form of self-improvementt TTI also obtains the highest SR in 8 out of 13 domains, illustrating its efficacy.
TTI outperforms fixed-horizon via adaptive exploration. Table also shows that our curriculum approach outperforms fixed h = 10 baseline by 5.7% and fixed h = 30 baseline by 19.6% in average accuracy. To better understand the use of interaction within a training rollout, we plot the average number of interaction steps held-out validation set with 78 tasks in Figure 5 (a). Note that the agent trained with a rollout, while h = 30 quickly drifts into aimless exploration and executes a larger number of steps pre- maturely training, hindering performance. This aligns with our findings in Section 5.1 that simply running training at longer horizon may not be sufficient for obtaining effective interaction scaling. In fact, we find that when training with TTI, the interaction length of the agent's rollouts first decreases but then starts to increase as the maximum allowed horizon increases, indicating that an adaptive curriculum enables effective interaction scaling.
learns to continuously reduce the maximum number of steps spends in
Figure (d) shows that the task success rate also grows over time and correlates with the expanding horizon. While the average task success rates for TTI are better, we observe notable per-domain differences. Figure 5 (e) shows representative per-domain success rates. On domains like Allrecipes and Cambridge, TTI significantly outperforms Exxe-horizon and zero-shot approaches, improving success rates by 31.4% and 15.6%, respectively, likely because these domains are highly information- dense and benefit from extended exploration enabled by adaptive interaction scaling. However, in domains like Amazon and GitHub, TT7 underperforms the baselines. We notice that the base model already has strong knowledge about domain-specific terminologies (e.g., commit history, forks, stars) in these domains, resulting in high base performance. Inspecting the rollouts, find that instead of using built-in filters and sorting, TTI can engage in exploration behaviors such as initiating Bing searches or consulting external sites. This exposes the agent to noisy distracting information, reducing task success. We discuss these cases in Section 6.2.
Learning dynamics of TTI. To study how TTI enhances the "within-rollout" exploration capabilities of the agent, we measure the number of GoBack and Bing actions over the course of training. GoBack actions measure the number of retries the agent makes within an episode to get unstuck during exploration. Bing actions correspond to the number of times the agent attempts to seek information by moving to bing. com. As shown in Figure 5 (a. b. and d), the performance of TTZ improves substantially as the number of GoBack and Bing actions and the trajectory length grow.
Also note that the trajectory length and the numbers of GoBack and Bing actions begin to increase with TTI, once the maximum allowed horizon length is increased as a part of the curriculum schedule (this regime is shown by the green shaded area in Figure 5). In contrast, these quantities continuously decrease over the course of training for the run with lower number of maximum interaction steps (h 10). We also find that the trajectory length shoots up substantially for the run with h = 30 and this correlates with worse performance. Finally, as shown in Figure (c), as the agent's trajectory grows longer with TTI, the number of tokens appearing in per-step reasoning actually becomes smaller. This implies that our agent automatically learning to tradeoff interaction for per-step compute in order to attain higher performance, and perhaps prevents any issues with overthinking.
8

---

Trajectory Length (Centered)
Bing (Cantered)
(Centered)
Harizon
0.0
Difference (%)
heath-30
-5.0
Noe Wartren
-10.0
-15
Iter
TTI Average Success Rate
Rolative SR 193
Altrecipes
Relative 5R PM
Combril Dictisnary
Nax Horrinn
25
Success Rate (50
20
Iter
Belaatve SR (%)
Huggingtace
Amazon
200
-10
Iter
10
ster
10 Iter
Figure 5: Dynamics of TTI during training. The green area represents the phase where the maximum allowed interaction horizon is the largest (h 30), per our multiplicative schedule. All results are evaluated on held-out subset of Web Voyager, not on the training tasks. Average trajectory length, i.e. the average number of steps taken in trajectory normalized by the average length at the first iteration (iteration 0). b: Ratio of the sum of GoBack and Bing actions out of all actions normalized by the first iteration. e: The average number of tokens (CoT lengths) per action. d: Average task success rates for TTI. e: Per-domain success rates for TTI.
6.2 Case Studies: Strengths and Failure Modes
We conduct detailed case studies to analyze how TTI behaves across tasks and domains. These cases highlight both the strengths and remaining limitations of our approach.
Strength: Effective exploration in complex tasks (example visualized in Appendix F.4.. For complex, exploratory tasks that require information retrieval, 777 trains the agent to extend its interaction horizon through search and backtracking, thus gathering and comparing information before making decisions. For instance, when tasked to find the baking temperature an apple pie recipe with 4+ stars and 50+ reviews, our agent first selects recipe but encounters a pop-up it cannot dismiss due backend issues. It then tries another recipe but finds no baking temperature. Returning to the listing again, it correctly identifies one that meets all criteria. We also observe that such behaviors emerge progressively. In early training with shorter horizon, TTI-agent tends stick the first recipe it finds, keeps retrying it and saying "I remember seeing one with 613 ratings earlier" instead of seeking alternatives. Only after the training horizon becomes longer does it learn to explore and backtrack. This illustrates that when TT/ runs a curriculum over interaction length, it teaches agents to adjust their horizon within a task and shift from exploitation to exploration. contrast, training with fixed short horizon can make it difficult to develop such exploratory behaviors.
Strength: Strategic exploitation in simple tasks (Appendix F.5). For simpler tasks with clear, de- terministic paths (c.g., form filling or direct lookups), TTI-agent completes tasks efficiently without over-exploration. For example, when instructed find the "top trending open- project on ma- chine learning" in GitHub, the agent goes directly to the Open-Source menu, selects the Trending tab, and performs search. This shows that TTI balances exploration and exploitation based on task context.
Despite these strengths, we also observe characteristic failure modes that point to areas for improve- ment and may partly explain the agent's lower performance domains like GitHub.
over-reliance on resets (Appendix F.6.. When an action fails, our agent can reset the task by returning to the Bing search page rather than attempting domain-specifie recovery actions (e.g., revisiting menus, refining filters). This suggests the agent treats search as a universal fallback. We also observe repeated resets within the same trajectory, indicating lack of adaptive error recovery. While agents can extend horizons through both resetting and backtracking, the latter is often more natural. This highlights area where TTI could improve by guiding exploration more systematically and enforcing structure, e.g., by using dense rewards [71, 68].
Failure mode:
Failure mode: limited self- refffiiffiiaiin (Appendix F.7). We also observe that the agent can fail to verify its actions against the task goal. one case, the agent identifies a 2021 GitHub repository for task requiring one from 2022. While it explicitly acknowledges the mismatch, "It was created in

---

Table 4: WebArena results. For proprictary agents, we include the top 8 from the official leaderboard, Observe
that TTI attains the best performance an average, with especially better performance the CMS domain.
Method
Backbone
Average
Shopping
CMS
Reddit
GilLab
Maps
Pruprietary-Baaee
IBM CUGA [92]
61.7
OpenAI CUA
58.1
Jace Al [93]
57.1
ScribeAgen: [14]
GPT-4o Qwen2.5 32B
53.0
45.8
37.9
73.7
59.7
56.3
AgenSyymicttic [18]
3.5 Ilama 3.1
48.5
47.2
Leum Inneercc [63]
Claude 3.5 Sonnet
48
AgentOccan-Judge [33]
GPT-4
38,9
WehPilat [34]
GPT-4o
37.2
36.9
24.7
65.1
39.4
33.9
Fully Open-Source [Sef-Immrrvement)
Lean-byyllntereet [63]
Codestral 228
24.2
AgentTrek [37]
Qwen2.5 32B
22.4
AutoWebGLM M [64]
CharGLM1 M3
18.2
NNetnav [94]
7.2
7.4
Zeno Shot Baseline
Gemma 3 F2B
18.3
16.7
8.7
30.9
5.5
27.7
Fixed
Gemma 12B
23.8
28.4
26.0
13.2
34.7
Fixed - 30
Gemma 3 12B
19.0
25.7
9.7
29.8
8.7
28.57
TTI (Ours)
Gemma 12B
26.1
33,9
15.5
35,3
15.7
40.5
2021, not 2022, so it doesn't I meet the criteria", it still submits it as the answer. This implies limited self-verification ability and could be mitigated by longer, more deliberate per-step reasoning. An important next step is to enrich TTI with the cognitive behaviors that enable per-step reasoning [91].
6.3 WebArena Results and Analysis
We further assess TTI on the full WebArena [2]
(we only use the prompting-basee based verifier for training. but use the original benchmark evaluators for evaluation). As shown in Table 4 and Figure ?? (right), TTI obtains the highest performance among open-source agents trained entirely using self-improvement procedure on self- relf-generated data, without relying proprietary models for task completion distillation. While T77 improves over the zero-shot baseline by 7.8,, the gains are smaller than on Web possibly because: (1) WebArena tasks are more complex, as reflected in lower accuracies even for proprietary models, leading to fewer successful rollouts per iteration and slower learning: (2) Agents occasionally attempt actions that are valid in real-world domains but invalid in WebArena counterparts (for example, search works reliably on Reddit but fails in WebArena's Postmill due to environment bugs). More experiment details are in Appendix G.
Further scaling, While TTI equips agents with the ability adjust their interaction horizon during deployment, open question remains:
Taok Success Rate
0.35
ring
Can we further amplify performance by combining TTI with inference- time interaction scaling techniques such as re-checking in Section 4? To explore this, we apply the "check- again" strategy (Section 4) to intermediate TTI checkpoints. Due to the high evaluation cost associated with evaluating on full WebArena, we leverage the WebArena subset checkpoints obtained in Section 5.2. As shown in Figure 6, applying re-checking on top of TTI improves task success across various training stages. The benefits are more obvious in the early stage, when the agent has stronger bias to terminate prematurely. As training progresses, TTI encourages longer interaction that naturally incorporate behaviors like re-checking, reducing the benefit of explicit re-checks.
0.30
discussed
0.25
0.8
1.0
Tokens Trajectory (K)
Figure 6: We apply test-time re-checks TTI checkpoints.
Conclusion
In this work, we introduced interaction scaling agents. Through empirical studies on web agents, we validate that interaction scaling enables agents to explore and adapt dynamically, significantly improving task performance. We hope that this work opens new directions agentic reasoning and inspires broader applications beyond web navigation.
dimension of test-time scaling for interactive
Limitations and future work. Our experiments are limited to web environments; extending this method to other domains like robotics or open-world games requires further exploration. Besides, scaling interaction steps increases computational costs during both inference and training. Although our adaptive scheduling helps, more efficient handling of long interactions needed. In addition, our training relies simple behavior cloning; future work could incorporate more advanced RL methods like PPO [95], GRPO [70] to improve performance. Lastly, due to high compute cost, we only ran the full benchmark once per setting, limiting the ability quantify variance from policy, environment, and evaluator stochasticity. Future work should explore multiple runs or more robust evaluation.
10

---

Acknowledgments
We thank Jiayi Pan, Shanda Li, and Yuxiao Qu for feedback and informative discussions on an earlier version of this paper. This work was supported in part by the National Science Foundation grants IIS1705121, 120, IIS IIS1838017, IIS2046613, IIS21 12471, 1, the Office of Naval Research under N00014-24- 1-2206, and funding from Meta, Morgan Stanley, Amazon, Google, and Scribe. We specially thank Scribe, The AGI company, and CMU FLAME Center (Orchard cluster) for providing part of the GPU resources that supported this work. Any opinions, findings and recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies or employers.
References
[1] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL atttps://arxiv org/ hargg/aa///// ab0/2401. 299.119991..... 13919.
[2] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL. https:://// / /wpp//eeeeeeeeeeeee/eeeeeeeeeeee net./ forofoommmmmmm dioommmd
[3] Claude. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024. URL https://wwe anthropic. aathroppiccommmeetee hoom/neww//a http//eew///-mmeeeeeeee 5--5-mmoells- and computer-use
[4] OpenAL. Introducing operator, 2025.
URL
introducing- operator/.
[5] Magnus Müller and Gregor Zunič. Browser use: Enable ai to control your browser, 2024. URL https://gt thub. com/ http//bowwwerrrr wwwebrsoseeruusseeeeesessssseses
[6] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wemmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023.
[7] Claude. Your code's new collaborator, 2025. URL https://www, anthrropi. com/ cl claude-code.
[8] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowiez, James Tanner, Quan Vuong, Homer Rich Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. pi0.5: vision-languaag-action model with open-world generalization. 2025. URL https:tt////tt http:p/:///pppeeeemtiicsccoooooo www.seeeasssseeeeeeeeeeeeeee.aes cscholla org/CorpussDD 277993634.
[9] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large languanl models as decision-making agents via reinforcement learning. ArXiv, abs/2405.10292. 2024. URL https://api.. weeaanticscoolll. org/ II
[10] Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Chun Zhu. Emergence of pragmatics from referential game between theory of mind agents, 2021. URL https:
//arxiv. orgogrggggoo www/oor/////ba///eeeee abs///01...1.///////////////// 07752.
[11] Luyao Yuan, Dongruo Zhou, Junhong Shen, Jingdong Gao, Jeffrey Chen, Quanquan Gu, Ying Nian Wu, and Song-Chun Zhu. Iterative teacher-aware learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29231 29231-29245. Curran Associates, files/ ttteee/ppppr//2///11e..
Inc., 2021. URL https::s///ooeeeeiiigg hpee hww.://roooeeiinn....n...ee.epp neurips.cc. MBBo0fffabB8E0066611114444466542 Paper pdf.
11

---

[12] Weixin Liang. Junhong Shen, Genghan Zhang, Ning Dong, Luke Zettlemoyer, and Lili Yu. Mfxture-of-mamtee Enhancing multi-modal state-space models with modality- sparsity, 2025. URL https: httpps:///rrreeee //axivvvvvv...//i org/ htt//ab//// aaa/abb//2//00 /2001 16295.
[13] Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical usioonlanggaageaattion models. ArXiv, abs/2502. 2025. URL https:
/ /apii
org/Corpus ID: 276618009
[14] Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scale workflow data. ArXiv, abs/2411.15000 2024. https:ts://ap semanticscho lar. CorpussD 274192657.
[15] Xiang Deng. Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
attps://openrevies..neet
[16] Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: align then refine. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[17] Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-Ilm: Repurposing general-purpose Ilms for specialized domains, 2024.
[18] Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Le- andro Z. Agudelo, Peter Qian, and Tianlong Chen. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small Ilms. ArXiv, abs/2502.07942, 2025.
[19] Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran L. Li. model internet agents. ArXiv, 1ttp/2422.11194. 2024.
Autonomous skill discovery for foundation
[20] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. ArXiv, abs/2406. 1896, 2024.
[21] Hao Bai, Yifei Zhou, Erran I.. Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training device-control agents. ArXiv, abs/2502.15760, 2025.
[22] Junhong Shen and Lin F. Yang. Theoretically principled deep rl acceleration via nearest neighbor function approximation. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (11):9558-9566, May 2021. doi: 10. index. php/ httpp///AAA//aatleeeeee/e//e////
URL httptt///oojs aaai org/
[23] Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang. Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning, 2025. URL /ttps://arxxx org/ abs/2411111 02337.
[24] Claude. Claude takes research to new places, 2025. URL news/research.
[25] OpenAL. Introducing deep research, 2025. introducing deep- researh/..
URL http://pnnni..cmmnnmiaee
[26] Google Gemini. Gemini deep research, 2025. URL deep-ressearch/"hh1-
[27] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hanna Hajishirzi, Luke S. Zettlemoyer, Percy Liang, Emmanuel J. Candes, and Tatsunori Hashimoto. sl: Simple test-time scaling. ArXiv, abs/2501. 19393, 2025. URI. https://app semanticschollrr
org/CorpusID: 2767999993
12

---

[28] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM The Thirteenth International Conference on Learning Representations, 2025. URL https:
In
//openreview.o htt.:/nee/eooeeeeeeeee www.eeeeeeeiddiieeeeeeee derrmm??-VNcckkT.EH...
[29] Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. ArXiv, abs/2412.15287, 2024.
URL httpp://appp..... sem.a.iiccmatttccoooloos
org/ POO.pusID:274965000
[30] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test- time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
/ //openreview
[31] Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with human preferences. 2023.
[32] Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixi- ang Shane Gu, and Izzeddin Gur. Multimodal web navigation with -ontuuttiiooiinuneedd foundation models, 2024. URL btstps://arxiv..ot/ attpaa/...../eeeee 11854.
[33] Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: simple yet strong baseline for Ilm-based web agents, 2024. URL https: /arxivv..vvv//////////a///aaaal/ org/abs /2100.211..0 13825.
[34] Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024. URL https: org/ abs /2ab////2.... 15978.
[35] Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Towards foundation models for pde solving via cross modal adaptation. arXiv preprint arXiv:2403.07187, 2024.
[36] Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou Cat: Cdntent-adaptive image tokenization, 2025. URL https:
//arxiv. org/ wwwgabs//////ooooooooooooe.aaer/ /2000 03120.
[37] Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. ArXiv, abs/2412.09600 2024.
[38] Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents, 2024. URL htttps://axx: www.awwwwww...aa.a........weeeee /2400 16464.
[39] Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle to beat supervised baselines, 2024. URL http:: //tt::///aiiveeeee org/ has///b///// 201..0217777
[40] Junhong Shen, Abdul Hannan Faruqi, Yifan Jiang, and Nima Maftoon. Mathematical recon- struction of patient-specific vascular networks based on clinical images and global optimization. IEEE Access, 1800448-20611 2021. doi: 10. 10////CCESSSSSC/ 20DCCCSSSS222.... 13052501.
[41] Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, and Yu Su. Skillweaver: Web agents can self-improve by discovering and honing skills. 2025. URL https://api. semanticscholar. www.laarrrrrrooooooaeee orgg/orrpssD::277654000
[42] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.066. 244444044444 2024.
[43] Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
13

---

[44] Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-bench-360. Benchmarking neural architecture search on diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
[45] Yao Fu, Dong-Ki Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guide- lines for large language model agents. arXiv preprint wrxXv:2403.08975, 2024.
[46] OpenAl. Gpt-4 technical report, 2024. URL http://///aarv...rraa/////..0.. 087744
[47] Authropic. Introducing the next generation of claude, 2024. URL httpp://www.nthhrop.c. com/neus/claude 3-family.
[48] Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked Ilm policies for web actions. In Conference on Language Modeling (COLM), 2024. URL https:
wwwpxii..rggabb//31311 03720.
[49] Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. ArXiv, abs/2407.13032. 2024.
[50] Wenduo Cheng, Junhong Shen, Mikhail Khodak, Jian Ma, and Ameet Talwalkar. 1.2g: Repur- posing language models for genomics tasks. bioRxiv, 2024. doi: 12.1100/2222.11....627777 /www. biorxiv org/ content/earlly/222//.////// /tt////22221222/22///// 12. 627422.
URL https:
[51] Lutfi Eren Erdogan, Nicholas Lee, Schoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks, 2025.
[52] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal M. P. Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. ArXiv, abs/2409.1291 12917, 2024.
[53] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint wrXh:2247.01176. 2024.
[54] Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exempaa prompting with memory for computer control. In The Twelfth International Conference on Learning Representations, 2024. URL attps://openreviee
[55] Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint 2rXiv:2409.004422 2024.
[56] Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani- Tur, and Heng Ji. Infogent: An agent-based framework for web information aggregation, ArXiv, abs/2410. 19054, 2024.
[57] Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Bootstrapping agents by guiding exploration with language. arXiv preprint 40X1v:2403.08140, 2024.
[58] Shikhar Murty, Dzmitry Bahdanau, and Christopher D. Manning, Nnetscape navigator: Complex demonstrations for web agents without a demonstrator, 2024. URL https: /ttp///aivv////////////////aaaaa org/
[59] Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards internet-scale training for agents. ArXiv, abs/2502.06776, 2025. URL https: //api. semanticscholar
httpa//cccccc.....oooooooopppooo org/DorpusID::276449222
14

---

[60] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Ramesh Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. ArXiv, abs/2408.07199, 2024. URL. https:/ //api. semant comanticscholat org. Crr/CorpssID: 271865516.
[61] Yifei Zhou, Andrea Zanette. Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. ArXiv, abs/2402.19446 2024. URL https://apt semanti
Trrg/CoppuIDD.... 268091206.
[62] Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. ArXiv, abs/2410.148000 273501605.
2024. URL https://api sea.pi..mmmtticchhllaa.... wwwps/scchhaaaoooooooouooo
[63] Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arik. Learn-by- interact: data-centric framework for self-adaptive agents in realistic environments. ArXiv, abs/2501.10893, 2025.
[64] Hanyu Lai, Xiao Liu, Iat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang. Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5295 -300, 2024.
[65] Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters. ArXiv, abs/2408.03314, 2024. URL https: /api. wemanticsccooll.. org/ otg//orrpsaD::271779990 271719990.
[66] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis compute-optimal inference for problem-solving with language models. 2024. URL https: //app/apppp sewanti www.sacchow....caaeeeoeoeeeeeeee :t:://ooppiI::1100000000 : 271601023.
[67] Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/21 10.14168, 2021. URL https:
//api. //api.semanticschola. wtt.:/ccchoaaaaororrrroouuuaeees :B://oopp 239998651.
[68] Amrith Rajagopal Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. ArXiv, abs/2410.08 146, 2024. URL https: / //api. www.sssmmtticcchoo...o.o
arg/CorpusID.co 27323346.
[69] Amrith Rajagopal Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test- time compute without verification or rl is suboptimal. ArXiv, 12118 2025. URL
https://api. www.eticschoolie...
[70] DeL Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learn- ing. ArXiv, abs/2501. 2025. URL httpss:/appt semanticscholar. oonattrssshool..rroo/ooorppplo 275789950.
org/CorpusID:
[71] Yuxiao Qu, Matthew Y. R. Yang, Amrith Rajagopal Setlur, Lewis Tunstall, Edward Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning. ArXiv, abs/2503.07572, 2025. URL https://aapp semanti aschola org/ : 276928248.
[72] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https: org.abb//220..3622..
//arxiv. hwt.s/.rrgabb////////eee
[73] Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mets and exploratory learn- ing. ArXiv, abs/2410.02052, 2024. URL. https::///// //api. semanticscholar. semaantisccioolar.rrr//orrorpr org/CorpusID: 273098809.
15

---

[74] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408 15240, 2024.
[75] Can Jin, Hongwu Peng. Qixin Zhang, Yujin Tang, Dimitris N. Metaxas, and Tong Che. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. 2025. URL
https://api
semanticscholar, org/ Cotpporrrp:DD 2777811777777
[76]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300,, 2024.
[77] Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing program- matic skills for agentic tasks. 2025. URL htttt://appp semanticscholar. Irg/CorpusID: 27763423 6342866.
[78] Gemma Team. Gemma 3 technical report. ArXiv, abs/2503.19786, 2025, URL https:
//api. semanticscholar. httpr//iisc..o...ooooooorraaaaaa -T53
[79] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[80] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham M. Kakade, Igor Mordatch, and P. Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. ArXiv, abs/1803.002466, 2018. URL htaps://api.
semanticscholar. org :::p:/Corps:::::::000 4043645
[81] Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation. Neural networks: the official journal of the International Neural Network Sociery, 26:118-22 19, 2011. URL https:/ /tpp:/app..../// semanticscholat 2274728.
[82] Dotan Di Castro, Aviv Tamar, and Shic Mannor. Policy gradients with variance related risk criteria. arXiv: Learning, 2012. URL https://app semanticscholl .ogg CCrppsorp 3109162.
[83] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009, URL https://api.
semanticscholar org/CorpusID:8735 873344
[84] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Pe- ter Stone. Curriculum learning for reinforcement learning domains: A framework and sur- vey. ArXiv, abs/2003.04960, 2020. URL p/attps://ppl semanticscholar waweeorg/oooo Drg/CorpusID: 212657666.
[85] Xin Wang, Yudong Chen, and Wenwu Zhu. survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 44:4555-4576, 2021. URL https://apt
semanticscholar org/ Prrg/CorpusID:: : 232362223.
[86] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE Transactions Neural Networks and Learning Systems, 31:3732-3740, 2017.
URL https: www.seeeaassseeeeeeeeeeeeeeeeeee cichollarrcccrr org/ hegg/orrpusDD.. 8432394.
[87] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Neural Information Processing Systems, 2017. URI. https:tt//. //api. semanticscholee..........i org/ I
[88] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
16

---

[89] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020. URI.tttss:. / /appi semanticscholar
sea.aaniissccolaa..ooo/orrppaaaa www.Coorgg/t: 221111193.
[90] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv22501.12322 2025.
[91] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, nathan lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. ArXiv, abs/2503.01307, 2025. URL www.hhttp///pa.... semanticschon 276741915.
oroooogg///oo CorprrrD
[92] Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Aviad Sela, Asaf Adi, and Nir Mashkif. Towards enterprise-ready computer using generalist agent. ArXiv, abs/2503.01861, 2025. URL hittps://appp http/ppp..eaaatcccccol.aa..... org/ :eep:/orrpsI::::0000 276775684
[93] JaceAl. barena benchmark, 2024.
Awa 1.5 achieves
breakthrough performance on we-
URL https://ww/ jace. ai aippost/
on- an-webarena- www.aaaaaanaar......aaaaa.a
[94] Shikhar Murty, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised
learning of browser agents through environment interaction in the wild. 2024. URL https:
/ //api. sepaiiismmnttlachollar
[95] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. ArXiv, abs/1 707. 707.06347, 2017.
17

---

A Broader Impact
This work contributes the development of more adaptive and capable Al agents by introducing a new test-time scaling dimension focused on interaction rather than per-step reasoning alone. While this approach improves robustness and generalization in open-ended environments, it also raises important considerations. Increased agent autonomy can amplify both the benefits and risks of deployment in real-world systems. Moreover, agents capable of richer behaviors could be applied to sensitive domains (e.g., customer service, education, or automation workflows) where unintended actions could have large impacts. We encourage future work consider ethical safeguards, interpretability tools, and human-in -i-tu-iii-t---oop-- the-loop designs when deploying interaction-scut agents. Our experiments are conducted entirely in simulated environments, and we hope this work inspires further research on controllable and trustworthy agent behavior under realistic constraints.
B Observation Space Design
We use the screenshot accompanied with the web page's accessibility tree main observation. We study two versions accessibility tree. Rich accessibility tree is modified from the WebArena code and looks like:
[21]: RootWehArea 'Dashboard/ Magento Admin' focused: True; [O]: link 'Magento Admin Panel';
[1]: link "DASHBOARD"; [2]: link 'SALES'; [3]: link "CATALOG"; [4]: link "CUSTOMERS"; [5]
link "MARKETING:: : [6]: link 'CONTENT'; [7]: link 'REPORTS'; [8]: link 'STORES'; [22]: link
'SYSTEM'; [23]: link 'FIND PARTNERS & EXTENSIONS'; [24]: heading 'Dashboard'; [9]: link
"admin"; [10]: link "; [25]: StaticText 'Scope:"; [12]: button "All Store Views" hasPopup: menu; [13]:
link 'What is this?"; [14]: button 'Reload Data'
Simple accessibility tree is modified from the PAE code and looks like:
[1]: "Dashboard"; [2]: "Sales": [3]: "Catalog": ; [4]: "Customers": [5]: "Marketing": [6]:
"Content"; [7]: "Reports"; [8]: "Stores"; [9]: "admin"; [12]: <button> "All Store Views": [13]: "What is this?";
[14]: <button> "Reload Data"; [15]: "Go to Advanced Reporting"; [16]: "here";..
Rich tree contains more details such the HTML tag and attributes like required, hasPopup compared to simple tree. However, is much longer than simple tree and hence harder to optimize due to the increased context length. As simple tree gives more steady training dynamics, we mainly use for our experiments.
C Preliminary Test-Time Experiments on WebArena
C.1 General Prompts
General Prompt
Imagine you agent browsing the web, just like humans. Now you need complete task. In each iteration, you will receive an observation that includes the accessibility tree of the webpage and screenshot of the current viewpoint. The accessbility tree contains information about the web elements and their properties. The screenshot will feature numerical labels placed in the TOP LEFT corner of web elements th current viewpoint. Carefully analyze the webpage information to identify the numerical label corresponding to the web element that requires interaction, then follow the guidelines and choose one of the following actions:
1. Click a web element.
Delete existing content texthox and then type content.
3. Scroll up or down the whole window.
Go back, returning to the previous wehpage.
5. Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly, action should STRICTLY follow the format specified by one of the follow- ing lines:
Click [numerical label]
Type [numerical label] [content]
18

---

Scroll [up/down]
GoBack
ANSWER [content]
Some examples are:
Click [8]
Type [22] [Boston]
Scroll [down]
ANSWER [06516]
Key guidelines you MUST follow:
Action guidelines
Use either screenshot accessibility tree to obtain the numerical label. Sometimes the
accessibility tree captures more elements than the screenshot. It's safe to select these elements without scrolling
For text input, use Type action directly (no need to click first). All existing texts in the textbox will be deleted automatically before typing
Preserve text inside quotation marks exactly as provided by user
You must not repeat the same actions over and over again. the same action doesn't 't work, try alternative approaches
Use ANSWER only after completing ALL task requirements
Wrap content for Type and ANSWER with square brackets '[]
Do not add quotation marks for search queries Web navigation hints (hint)
Your reply should strictly follow the format:
Thought: Your reasoning trace. A good practice is to summarize information on the current web page that are relevant the task goal, then generate high-level plan that contains the sequence actions you probably need to take
Action: Based this reasoning, identify the single most optimal action. You should output in the format specified above (under "STRICTLY follow the format") After each action, you'll receive new observation. Proceed until task completion. Now solve the following task.
Task: (task_goal)
Current URL: [url]
Screenshot of current viewpoint: attached
Accessibility tree of current viewpoint: faccessibility_tree
Beyond the above CoT prompt, we also tried using a more complex prompt for the thought process. However, this does not lead to significant gain in downstream accuracy (see Table 5), but it could increase training and inference cost, so we did not use in the end.
Complex Prompt
Thought: You must analyze the current webpage thoroughly to guide your decision decision-making
Show your reasoning through these steps:
Summarization: Begin by understanding the page context identify what type of page you' re on (search results, form, article, etc.) and how it relates to your objective. Summarize important information on the webpage that might be relevant to task completion. Especially when the task requires to return some answers to a specific question, you should note down intermediate information that helps generate the answer.
Planning: Generate checklist of subtasks required for completion and cross-out the subtasks you've 've completed. Identify the next logical subtask.
Verification: Verify all information you't 'e entered far. Check that your inputs match requirements in terms of spelling and format (you should not change the user-specified information,
if there'n re grammar errors). Verify if any selections for dropdown items align with the task objective. Identify if there're necessary fields that have not been filled in. Note that if the last few steps are repeating the same action, there must be missing incorrect information.
19

---

Backtracking: If the task requires exploring multiple webpages (e.g., orders, posts, item pages, etc) to find out an answer, consider you need to issue GoBack and return to the previous web page.
Candidate Generation: After all the above reasoning, list the most relevant possible actions, evaluate pros and cons of each action, and finally select the most effective action progress task.
Action: Choose ONE of the following action formats:
-Click [numerical L_aaellaa_e____--- Click a specific element
-Type [numericall
[content] Input text into field SScooll [up/down] Navigate the page vertically
-GoBack Return to previous webpage
ANSWER [content] Provide final answer when task is complete
C.2 WebArena Prompts
Below are the content replacing "(hint)" in the general prompt.
General Hint
Always save progress through appropriate buttons (Save, Submit, Post, etc.)
-Always remember interact with dropdown options after expanding
CClerr filters before setting new ones
Reddit
Always save progress through uppropriate buttons (Save, Submit, Post, etc.)
Always remember to interact with dropdown options after expanding
Pay attention to words like "latest", "newest", "hottest" the task objective, which require clicking the dropdown menu and select "New" or "Topp with the correct time range
When selecting a subforum, you can either browse the dropdown menu in the "Submit" page navigate to "Forums" and check all subforums by clicking on "Next" to go over all pages. You must try to find a subforum that exactly matches your query. If there's no exact match, pick the most relevant one, ideally the subforum is about objects or locations contained in the given objective
"Trending" means "hot"
To find out all posts or replics from a user, click the user name and then click "Submissions"
"Comments"
CMS
Always progress through appropriate buttons (Save, Submit, Post, etc.)
Always remember to interact with dropdown options after expanding
Clear filters before setting new ones
Use date format: month/day/year (e.g., 1/1/16, 12/31/24)
When searching phone numbers, remove the country code
When searching product name, use single but not plural form
When the web page contains table, aggregate the rows with the same item
Shopping
Always save progress through appropriate buttons (Save, Submit, Post, etc.)
Always remember interact with dropdown options after expanding
Sort items by price by clicking the dropdown menu and set descendinn/asccending direction
When searching product name, use single but not plural form
If the objective requires only finding an item, stop at the item page without adding to cart
To find out the quality of product, search the item, click on review, and inspect its review
20

---

Click "Page Next" iterate over all orders
Since there's no way to filter order history, click "View Order" for every order within date range and inspect individually. If the condition is not met, go back
GitLaa
Always progress through appropriate buttons (Save, Submit, Post, etc.)
Always remember to interact with dropdown options after expanding
Clear filters before setting ones
-hee searching a repo gitlab, type only the project name after "/" in the search box
Map
Always remember interact with dropdown options after expanding
When searching for a place, remove prepositions like in/on/by/at. For example, use "star- bucks, craig street" instead of "starbucks on craig street". Put the city name the end
When there is no results shown after search, rephrase the address and try again
To find direction between two points, after entering the from and addresses, select the correct transportation (foot/bicycle/car) before clicking "Go"
When the given location not geological address, use your knowledge infer the address
C.3 CoT Experiments for Base Agent
To enable efficient rollout collection, spin up multiple Docker containers single GPU according to the official WebArena repository. We use the vLLM [88] engine for inference and apply the following inference hyperparameters for most of our experiments.
max _ex__eew___e foa__ne_ttkeenss 1024
max attached imgs:
temperature:
top_p: 0.95
We randomly subsample 62 test tasks for analysis purposes. Below the results of zero-shot agent vs CoT prompting. "CoT" uses the "General Prompt" in Section C.1. "Complex CoT" uses the "Complex Prompt" in Section C.I..
Table 5: Base agent results averaged over 3 runs WebArena subset.
Prompt
Task SR (%)
Action Only
14.76
CoT
23.81
Complex CoT
C.4 Scaling Trade-off Experiments
"Check-again" for interaction scaling. After the agent outputs the task-stop signal, we append the following prompts to the observation to induce it to check again.
Check-Again Prompt
Important: You returned an answer in the last step. Let's pause, check the web page, and think again. If you still think the task is finished, double-check your answer, revise it if need, and return final answer. If not, continue the task. Your output should still be in the same Prhtint....ction.... format.
21

---

Table Comparing different inference-time prompting strategies. Results averaged over rums on WebArena subset, All methods are applied once.
Inference- Time Strategy
Task SR
Baseline
23.81
Check-again
26. 14
Budget-forcing
24.81
Best-of-n
25.03
Check-again Budget-forcing
26.33
Chock-again Best-of-n
27.36
When applying multiple re-checks, we slightly vary the prompts such as "Before you finalize the answer, re-evaluate it in terms of the current web page-what exactly supports contradicts it?" or "Why do believe this answer is correct? What on the page justifies it? Could an alternative answer be better?" Please refer the code base for the exact prompt used.
Per-step budget forcing. Following [71], we use the phrases below to induce longer per-step thinking. The phrases are different to ensure that the model does not run into the scenario of endless repeating a phrase.
First time: Wait, let me think deeper.
Second time: But let me double-check.
Third time: But hold on.
Per-step best-of-n. We tried both selecting by log likelihood and majority voting, with the latter showing slightly better results.
Additional results for combined scaling, Beyond evaluating each scaling method separately, we also tried combining methods along different axes.
D Online Filtered BC on WebArena
We use the following hyperparameters to obtain the training curves Table During training, the vision_tower tower of Gemma 3 is kept frozen because it is frozen during pretraining. Other hyperparameters can be found in our code in the supplementary material.
num iteration: 10
actor eeroc_epochs: 1 # number of epochs to update the actor
rollout_size 512
num ham_upddate pondatessampleaoc per iteration: 512
Ir: le-6
optimizer: AdamW
scheduler: WarmupCosineLR
batch_size
grad aeccacccceooseeeee sternmsstess221
eval eval_horriee horizon: 30
E TTI Implementation
We provide the pseudocode in Algorithm 1. For the replay buffer, to encourage the agent to learn from more recent examples, assign weights based on recency when sampling rollouts to update the agent: for the k-th trajectory added to the buffer, its weight is & [D]
22

---

Algorithm 777: Filtered Behavior Cloning with Interaction Scheduling
1: Input: Agent policy Evaluator R, Environment &, Learning rate a, Replay buffer D, Interaction scheduler hyperparameters Amin- /mmax
2: Initialize policy from pretrained model
3: Initialize replay buffer D +- 0
4: for each episode do
5: Set interaction horizon h; <- get_schedule((, /emiu 1mmxx)
6: for each rollout to collect do
7:
Initialize environment: So
8:
for each / in [1, /.] do
9:
Observe current state 8t Predict action å, To(
10:
Execute action ág in environment
12:
Observe next state $t+1
13:
episode done then
14:
Compute reward R(ss, 0t)
15:
else end if Store transition: D + DU Ti, 84+1)
16:
17:
18:
19:
end for end for
20:
21:
for sample successful trajectory in D do
for f 1 hspp do
23:
Accumulate loss: L(0) + L(0) + CrossEntropy(roo(G as) 24: end for
25:
end for
Update policy: t- 0 (V)
27:
end for
F Web Voyager Experiments
F.1 Task Generator & Evaluator Prompt
Task Generator Prompt
You website exploration assistant tasked with discovering potential tasks websites. These tasks should be similar to a user-specified task and aim to complete some high-level goals such booking restaurants in website. Your goal is freely explore websites and propose tasks similar to given set of examples. For each iteration, you'll receive:
An observation with the webpage's accessibility tree
A screenshot showing numerical labels in the TOP LEFT corner of web elements You will then generate possible tasks while exploring the website. You should imagine tasks that are likely proposed by a most likely user of this website. You'll be given set of examples for reference, but you must not output tasks that are the same the given examples. The generated tasks must be realistic and at least require 3 steps to complete. It cannot be too simple.
## Response Format and Available Actions Your reply for each iteration must strictly follow this format:
Thought: Analyze the current webpage thoroughly to guide your exploration. Examine the webpage's structure, content, and interactive elements to identify potential tasks that users might perform this site. Decide whether you want to keep exploring output some tasks Tasks: If you think you are ready to generate some tasks, output them in the follow- ing format (note that different tasks are separated with double semicolons): GENERATE 2:takk2annswer2.
Action: Then, to continue with your exploration, choose ONE of the following action formats:
23

---

Click [numerical label] Click specific element
Type [numerical label] [content] Input text into a field
Scroll [up/down] Navigate the page vertically
GoBack Return to previous webpage
Examples:
Click [8]
Type [22] [Boston]
Scroll [down]
GENERATE [Find the company's phone number:(555) 123-4567;;Loccal the price of the
basic subscription plan:$19, 19999//mmmt)
Your final output should look like:
Thought:
Tasks: GENERATE [...] (this is optional, only generate when you are confident)
Action:....
## Critical Guidelines
###
Action Rules
Use either screenshot or accessibility tree to obtain the numerical_lane
For text input, use Type action directly (no need to click first)
Ensure proposed tasks are diverse and demonstrate different aspects of the website. The tasks must have diverse difficulty and require different number of steps (3-20) to complete. Tasks should clear, specific, achievable, and self-contained. cannot be too general, c.g., related to äny post, äny product; äny place: It must not depend any context actions that you have performed, i.e., you must assume zero prior knowledge when someone wants to complete the task
Your task should be objective and unambiguous. The carry-out of the task should NOT BE DEPENDENT the user's personal information such as the CURRENT TIME OR LOCATION
Your tasks should be able be evaluated ORJECTIVELY. That is, by looking at the last three screenshots and the answer provided by an agent, should be possible to tell without ambiguity whether the task was completed successfully or
Answers should be precise (e.g., exact prices, specific information, exact text)
Your should output both operational tasks (the goal is to complete some steps) and informa- tion retrieval tasks (the goal is to find some answer to return)
You must refer the examples given and mimic the complexity and task structure. See how these tasks are self-contained and realistic
Your proposed task cannot be a single action like click, type! Tasks like 'Determine the stand-alone task; "Uncheck Use system value' unacceptable because it not a complete task; 'Locate the total revenue for the last month' is unacceptable because 'last month' is ambiguous; After each action, you'll 'll receive a new observation. Continue exploring and generating tasks. Here're re some examples: (example)
number of uses for that term' is unacceptable because it is ambiguous
Current URL: [url}
Screenshot of current viewpoint: attached
Accessibility tree of current viewpoint: (accessibility |
Evaluator Prompt
You are an expert in evaluating the performance of web navigation agent. The agent is designed to help a human user navigate a website to complete task. Your goal to decide whether the agent's 's execution is successful or not. As an evaluator, you will be presented with three primary components assist you in your role:
Web Task Instruction: This is a clear and specific directive provided in natural language, detailing the online activity to be carried out.
Result Response: This is a textual response obtained after the execution of the web task. serves textual result in response to the instruction.
24

---

3. Result Screenshots: This is visual representation of the screen showing the result or intermediate state of performing a web task. serves as visual proof of the actions taken in response to the instruction.
You SHOULD NOT make assumptions based on information not presented the screenshot when comparing it the instructions.
Your primary responsibility is to conduct a thorough assessment of the web task instruction against the outcome depicted in the screenshot and in the response, evaluating whether the actions taken align with the given instructions.
NOTE that the instruction may involve than task, for example, locating the garage and summarizing the review. Failing to complete either task, such as not providing a summary. should be considered unsuccessful.
NOTE that the screenshot is authentic, but the response provided by LLM is generated at the end of web browsing. and there may be discrepancies between the text and the screenshots. Note that if the content in the Result response is not mentioned on or different from the screenshot, mark it as not success.
NOTE that the task may be impossible to complete, in which case the agent should indicate this the response. CAREFULLY VERIFY THE SCREENSHOT TO DETERMINE IF THE TASK IS IMPOSSIBLE TO COMPLETE. Be aware that the agent may fail because of incorrect actions, please do not mark it as impossible the agent fails because of its incorrect actions.
You should explicit consider the following criterion:
Whether the claims in the response can be verified by the screenshot. E.g. if the response claims the distance between two places, the screenshot should show the direction. YOU SHOULD EXPECT THAT THERE IS A HIGH CHANCE THAT THE AGENT WILL MAKE UP AN ANSWER NOT VERIFIED BY THE SCREENSHOT.
Whether the agent completes EXACTLY what the task asks for. E.g. if the task asks to find a specific place, the agent should not find a similar place. In your responses:
You should first provide thoughts EXPLICITLY VERIFY ALL THREE CRITERION and then provide definitive verdict on whether the task has been successfully accomplished, either as 'SUCCESS' or 'NOT SUCCESS'.
task 'SUCCESS' only when all of the criteria are met. If any of the criteria met, the task should be considered 'NOT SUCCESS'.
F.2
Agent Prompt
Web Voayagee
Imagine you are robot browsing the web. just like humans. Now you need to complete task. In each iteration, you will receive an observation that includes the accessibility tree of the webpage and screenshot of the current viewpoint. The accessbility tree contains information about the web elements and their properties. The screenshot will feature numerical labels placed in the TOP LEFT corner of web elements the current viewpoint. Carefully analyze the webpage information to identify the numerical label corresponding to the web element that requires interaction, then follow the guidelines and choose one of the following actions:
1. Click a web element.
Delete existing content a textbox and then type content.
3. Scroll up or down the whole window.
Go back, returning to the previous webpage.
5. Navigate Bing's 's homepage.
Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly. action should STRICTLY follow the format specified by one of the follow- ing lines:
Click [numerical_iee |lbbe]]
Type fnumerical_laber [content]
Scroll [up/down]
GoBack
25

---

Bing
ANSWER [content]
Some examples are:
Click [8]
Type [22] [Boston]
Scroll [down]
Bing
ANSWER [06516]
Key guidelines you MUST follow:
Action guidelines
The predicted action should be based on elements as long as it's accessibility tree OR screenshot. Sometimes, accessibility tree or screenshot captures more elements than the other, but it's fine to use either one.
2. To input text for search bars, need to click textbox first, directly type content. After typing. the system automatically hits 'ENTER' key.
3. When a complex task involves multiple questions or steps, select 'ANSWER' only at the very end, after addressing all these questions or steps. Double check the formatting requirements in the task when ANSWER. Always think twice before using "ANSWER" action!!!
4. When specifying the content for 'Type' and 'ANSWER' actions, be sure wrap the content with '[]'.
5. Use 'GoBack' to return to the previous state, use it when you find the previous action incorrect.
6. When you see pop-up page, you should immediately 'GoBack" the previous page. Use 'Bing' when you need to navigate to a different website or search for new information. Your reply should strictly follow the format:
Thought: Your reasoning trace. A good practice is to follow this format: -Obervvation summary: where
you now? list all elements that related to the task goal. e.g. you're trying to filter something out, list all filters visible.
Planning: what sequence of actions do you need take achieve the task goal? give high-level overview of the steps you need to take.
Possible actions: to achieve that plan, what are potential actions you need do immediately and what's 's their effect? List at least 3 actions and analyze each of them.
Action: Based on this reasoning, identify the single most optimal action. You should output in the format specified above ("...STRICTI. follow the format...")
After you issue an action, the user will execute it and provide new observation. Now solve the following task.
Task: (task _ook_
Current URL: [url]
Screenshot of current viewpoint: attached
Accessibility tree of current viewpoint: (accessibility_tree
F.3
Experiment Details
We the following hyperparameters to obtain the Web results.
num_iteration: 10
actor _poo__epohs: 1 # number of epochs to update the actor
rollout size: 512
rollout_size:
num_update_ pappddee_sample_pe_jierrrtoooo 512
le-5
optimizer: AdamW
scheduler: WarmupCosineLR
batch_size: 4
grad_ accum steps:
26

---

eval horizon: 30 # note that train horizon is different for different methods, but evaluation horizon is kept the same
27

---

F.4
Case Studies; Strengths 1
Task:
Locate recipe for American apple pie Allrecipes with rating of at least 4 stars and more than 50 reviews. Note the maximum lemperature mentioned in the Directions. Fully trained agent explores:
BREAKFAST
Hate 101 rest
Americaa Aupll
coo ececii viled dipiics
meccom harsp Tesspoonee ooeee wrr 30t
bo
arke
B0 poseciaai socrnia indees
28

---

Assi Grandrr
101 8.:
Frigh
uena
resoree
10 B Ber
Then opeine Soverig pas
aeceer pranoct pove
Bon
Best
R.. 8x Rry
the
singi Chanddy
treontmen Sereee rreating
29

---

stale-stage agent prefers exploitation:
sen leeut
cailt moal
Chosit Amerkan
Books
sever reting
pooati thskie
3.Bee lene lovir
lirel
Books
The
30

---

F,5
Case Studies: Strengths 2
Task:
Identify the latest top-tree trending open open-source sonree project in the category of 'Machine Learning' on GitHub, and check the number of stars it has received.
Build and ship software single, collaborative platform
Build and single, olatform
(penetiee mesogn des mederrggge toaer
dng
Tyor Thine men doaning
Adber howe
bcutsound of
deatl Toxsing
tite used
and
Heal! oig Wit cour Martine
Light
31

---

F.6
Case Studies: Fail Modes 1
Task:
On Apple's website, how many different types of keyboards are available when customizing your 14-inch MacBook Pro?
Mac
Modher's Day
2ourt to You Mautorn mulil
Buii Apole Mooiigeii
the Hihlights.
"Quar Wet dund MaccC
Haiiin Nou Brtepoeddy
Trngds
lavel
Veew cregh
32

---

the hrt thot lalerce compaeooo
Buciple
pich
2010 Doue!
thenciiv rook.
colccriiie DiN Pisice pa dio rutherto
Aca
33

---

Task: Which university maintains and manages ArXiv. Accessing the university's 's website from ArXiv, how many underegraduate students are currently at the university.
Pastart fed of SEPIC Scoer STi
Thi Serurd cnlvtry
poreg unpp tase noreel cortt and ingrsse
andecial COCO
nan dever salle
34

---

F.7
Case Studies: Fail Modes 2
Task:
Identify a new open-source project on GitHub related 'Al agriculture' that created in 2022. and note its main programming language and description.
single, collaborative platform
single, co llttform
Towel lee sacu wet Mer
cppurie spocttc lostert
Nphoot an ro sese anterrrn denn tiee
G Full WebArena Experiment Details
We use the following hyperparameters to obtain the full WebArena results.
num_ nin_iteration: 10
actor_epocha # number of epochs to update the actor
rollout_sizes 512
num update_sample per iteration:
per_iteration: 512
Ir: le-6
optimizer: AdamW
scheduler: WarmupCosineLR
batch size: 4
grad aeccamccceaepppee stepssttppp
eval_ horizoor:on 30
35

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contribution is to propose new axis of test-time scaling for agent settings and design effective methods for leveraging the benefits during online training. These reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
36

---

Justification:
See the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitionn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There is theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all training and inference configurations, hyperprameters, LLM prompts in the main text and appendix. We also include the code in the supplementary materials.
Guidelines:
37

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We open-source benchmarks and release code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Belliii/guudde//ooeeeue swn.bmiiisiooooiccyy..... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cts.sss.cc/uubic/guuuuu. gewds//////eeeeeee/o /CodeSubmian ss ierionnoolliii onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
38

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, see the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report numbers averaged three trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we specify the GPUs we used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
39

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //ttrr//n....peeeee httppp.......eooooo cc/public//eiceccccccccccc ctwteetiiiiiiieeeeiiieeeeeaeeeee
Answer: [Yes]
Justification: The paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Broader Impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We do not use data or train models with high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
40

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
41

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification:
We discussed how prompt and train LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https : /ttaas://neuuiie htwpppsss.cccooofeeeeeeeeeeeeeee wttp:/cceee//////2////// for what should or should not be described.
42


Reviews:
[{"review_id": "pMI17x2rCN", "reviewer": "Reviewer_nVXA", "summary": "The paper introduces test-time interaction scaling as a complementary axis to the usual “think-longer” compute scaling for agentic tasks. By allowing an agent to act for more steps before termination, the authors show that the agent can gain new external information by simply continuing to interact with the environment once again.  They further propose TTI—a curriculum-based online RL method that gradually lengthens the rollout horizon with a multiplicative schedule. Empirically, a Gemma-3 12 B model trained with TTI attains 66.5% success on WebVoyager and 26.1% on WebArena. Case studies illuminate how the agent learns to trade off exploration vs. exploitation and reveal residual failure modes.", "questions": "1. While the paper shows interaction horizon is increased with RL training under TTI curriculum, it is not clear how the agent learns to think longer during training. For instance, in Fig 4, after 400 steps training, the agent (h=20) stop increasing overall interaction horizon but still improves the performance. Could you clarify this and the saying in L248, and provide the thinking lengths?\n2. In Fig 6, the \"check-again\" strategy seems to be less effective to increase the trajecory lengths. Could you provide interaction times for the RL trained agent with \"check-again\" strategy? If it fails to improve RL (well-)trained agents, how could we further increase the performance by increasing interaction horizons? Also, please remove the centering in Fig 5 (left), and remain the original interaction times.\n3. Since the observation history is only recent 3 steps, how does the agent handle long-term dependencies when the interaction horizon is increased? And, under this setting, how does \"check-again\" prompting different to Reflexion [1]?\n4. In L127, the paper mentions that prior works \"usually scale the number of thinking tokens at each step\" without reference. What are the prior works? Could you compare current TTI with these prior works under prompting and RL training?\n5. **Open Question:** How should TTI be applied to other tasks when some tasks require high costs for interaction and with non-recoverable states (e.g., robotics, games, financial decision making)? If there is a place for thinking longer to be effective, how should TTI co-exist with thinking longer?\n\n*If the authors provide evidence addressing Q1–Q4, corresponding rating score could increase.*\n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning. NIPS 2023.", "limitations": "The authors acknowledge domain restriction to web-navigation and compute overhead of long horizons. However, the paper does not discuss the limitations of TTI in terms of generalization beyond web browsing. The application of overly long interaction horizons to crucial tasks with high costs for interaction and non-recoverable states (e.g., robotics, games, financial decision making) could be risky. Also, interacting with users in real-world applications could be a feasible scalable dimension. Overall, these limitations could block the applicability of TTI to many real-world tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Clear Writing**: The paper is well-structured, with a logical flow from motivation to method design to empirical results. Figures and tables are clear and informative.\n\n2. **Well-Designed Preliminary Experiments**: The curriculum design is methodologically sound, with careful ablations on the effects of prompting towards TTI and acting longer. Many experiment details are provided, which is helpful for reproducibility.\n\n3. **Detailed Case Studies**: The paper includes insightful case studies that illustrate how the agent learns to balance exploration and exploitation, as well as the types of errors it makes.\n\n### Weaknesses\n\n1. **Novelty of TTI**: Though the experiments are thorough with authentic RL curriculum, the core idea of scaling interaction horizons is intuitive and not novel. Besides a boarder demonstration of TTI in UI-TARS-1.5 [1], [2,3] have already shown that longer interaction horizons can improve agent performance substantially.\n\n2. **Limited Interaction Horizon**: The maximum interaction horizon of 20/30 steps is still relatively short for complex tasks. While the paper shows that TTI improves performance on current benchmarks, it does not explore how much horizons could be scaled further like [1].\n\n3. **Lack of Metrics/Baselines in RL Training**: The paper does not report how the thinking tokens change during RL training, along with the change of interaction times. Also, RL training to incentivize longer thinking (e.g., mixing reasoning data in agent RL) is not compared with RL training for TTI. This makes it hard to assess the effectiveness of the RL training in improving interaction horizons compared to improving thinking lengths. Most other open-source baselines are from other model backbones, limiting insights into the effectiveness of TTI.\n\n4. **Too Compact Layout**: The paper is quite compact, with many figures and tables crammed into a small space, and still large amount of details are buried in the appendix. It is recommended to summarize the case studies instead of showing all the depiction. Training details such as data generation pipeline should be described more clearly as a part of the method.\n\n\n[1] [UI-TARS-1.5 Blog](https://seed-tars.com/1.5/).  \n[2] CodeT: Code Generation with Generated Tests. ICLR 2023.  \n[3] Reflexion: Language Agents with Verbal Reinforcement Learning. NIPS 2023.", "was_revised": true, "final_justification": "I find it highly difficult to carry out additional experiments to further address my concerns. However, I do not believe this work deserves rejection, so I will leave my current score of 4 as it is.", "rebuttal": "Thank you for highlighting our work's clarity, careful experiments, and empirical insights. To address your questions regarding evaluation, **we add new results showing how per-step thinking changes during TTI training** and **new baselines that use RL to scale per-step thinking**. These results further showcase how TTI extends interaction and outperforms approaches that only scale per-step reasoning. Below, we elaborate on the findings and also discuss questions around related work, novelty, and broader applicability.\n\n---\n## Question 1\n> in Fig 4, after 400 steps, the agent (h=20) stop increasing overall interaction horizon but still improves the performance. Could you clarify this and the saying in L248?\n\nFig 4 shows that interaction length saturates after 400 steps for h=20 agent, yet performance continues to rise. This is not a contradiction, rather, the agent learns to use its interaction budget more effectively, choosing better actions and exploring more strategically (e.g., searching for the correct item instead of clicking randomly). This motivates why we want to train agents with longer horizons **during later stages of training**.\n\nAs for L248, it is meant to motivate why we want to use shorter horizons **during early stages of training**. By \"worse performance stemming from noisy credit assignment and slower learning\", we are referring to the initial phase of the h=20 agent (red dotted line in Fig 4 left), where the performance first decreases because h=20 agent has not yet grasped the basic abilities to solve tasks efficiently and associates random exploration moves with high rewards. Employing an adaptive horizon schedule via TTI can prevent the initial decrease and improve overall learning efficiency. We'll add this clarification to Sec 5.1.\n\n> provide the thinking lengths\n\nAs suggested, we add two sets of results to show how thinking token changes during RL training. We show tables here but will include figures in the paper.\n\n**For WebArena (Fig 4),** we compute the average token in the \"Thought\" section for h=5/10/20.\n|Per-Step Thinking Token|Train Iter 2|4|6|8|10|\n|-|-|-|-|-|-|\n|h=5|124.2|143.1|142.5|171.7|181.2|\n|h=10|130.6|129.4|128.9|118.1|117.8|\n|h=20|116.8|105.7|115.2|104.6|94.3|\n\nFor the h=5 agent, per-step reasoning increases over training, likely because the agent must solve the task within a limited number of steps, requiring more deliberate decisions. In contrast, h=20 agent relies more on extended interaction and shows a decrease in per-step reasoning. This aligns with our findings on large-scale WebVoyager (see below).\n\n**For WebVoyager results (Fig 5),** we show the following:\n|Per-Step Thinking Token|Train Iter 2|4|6|8|10|12|\n|-|-|-|-|-|-|-|\n|TTI|129.57|112.28|117.55|107.64|110.82|101.34|\n\nAs training goes, the agent's trajectory grows longer (see Fig 5 left), but the number of tokens appearing in per-step reasoning actually becomes smaller. This implies that our agent is automatically learning to **trade-off interaction for per-step compute in order to attain higher performance**, and perhaps prevents any issues with overthinking each step. \n\n---\n## Question 2\n\n> In Fig 6, the \"check-again\" strategy seems to be less effective to increase the trajecory lengths. Could you provide interaction times for the RL trained agent with \"check-again\"? If it fails to improve RL (well-)trained agents, how could we further increase the performance?\n\nAs suggested, we provide the interaction times for RL agent with check-again.\n|# Interaction Steps|No Re-check|1 Check|2 Checks|3 Checks|\n|-|-|-|-|-|\n|Before Training|6.03|8.24|10.67|12.25|\n|Iter 5|9.01|10.49|11.95|13.61|\n|Iter 10|10.53|11.33|11.84|12.5|\n\nAs you pointed out, the increase in interaction steps is smaller for well-trained agents than baseline agents. Fig 6 also shows that the benefit of check-again diminishes as the agent learns to self-reflect during TTI training. This shows TTI agents learn to internalize re-checking and interaction scaling behavior via curriculum RL. \n\nTo further increase interaction horizons for well-trained agents, we could consider new directions like hierarchical exploration (e.g., setting and pursuing sub-goals within a single rollout). These are future work beyond the current scope of TTI. We'll add a discussion to encourage exploring this in the community.\n\n---\n## Weakness 3 \n\n> RL training to incentivize longer thinking is not compared with RL training for TTI\n\nAs suggested, we add new baselines that use RL to scale per-step reasoning for the WebVoyager experiments. We tried two methods: (a) using a more detailed instruction for the \"Thought\" section, or (b) asking the agent to double check its action at each step by outputting two \"Thought…Action…\" blocks. We use a fixed horizon of 10.\n|Method|Success Rate(%)|Per-Step Thinking Token Before Training|After Training|\n|-|-|-|-|\n|Baseline (Prompt Only)|55.8|-|-|\n|Fixed h=10|59.1|131|110|\n|Fixed h=30|45.2|131|119|\n|RL for Scaling Per-Step Thinking (a)|60.5|131|197|\n|RL for Scaling Per-Step Thinking (b)|50.0|131|216|\n|TTI (Ours)|**66.5**|131|101|\n\nBoth (a) and (b) increase per-step thinking. (a) improves over the fixed h=10 baseline but still underperforms compared to TTI. This suggests that simply thinking more per step is not enough; **TTI's ability to scale interaction allows the agent to actively explore the environment and gather information critical to task success**, leading to stronger performance. Also, method (b) seems to hurt performance likely because it adds redundancy rather than improving exploration or decision quality. We’ll include the new results in the paper.\n\n---\n## Question 3 \n\n> observation history is only recent 3 steps, how does the agent handle long-term dependencies \n\nAs noted in Sec 3, the number of **web page observations** is kept to 3; our agent still has full access to **all past actions**. This action history can serve as a compressed task \"memory\". Also, TTI implicitly teaches the agent to revisit pages, backtrack, and gather missing information dynamically to improve long-term dependencies (see case studies in Sec 6.2).\n \n---\n## Question 4\n\n> L127 mentions prior works \"usually scale the number of thinking tokens at each step\" without reference. What are the prior works? Could you compare TTI with these prior works under prompting and RL training?\n\nTo the best of our knowledge, prior to submission, there were no **multi-step agent** works that systematically scale per-step thinking tokens during training (most focus on single-turn reasoning tasks like math). Recent work like Kimi K2 explores this, but it appeared much after our submission. That said, we already compare to baselines that scale thinking tokens:\n1. Muennighoff et al. for test-time budget forcing (see Sec 4.2)\n2. Claude, OpenAI models, and others trained for internal CoT scaling (see Tables 3, 4)\n3. New RL baselines for scaling per-step reasoning, which we include in this rebuttal (see response to W3).\n\nIf we are missing prior work, please let us know and we are happy to compare!\n\n---\n## Weakness 1\n\n> core idea of scaling interaction horizons is intuitive and not novel\n\nWhile we agree that TTI's core idea is intuitive, we view its simplicity as a strength, not a weakness. Crucially, our idea is still novel, as prior work (e.g., Reflexion, CodeT) has used longer interactions primarily at **test time**, either for evaluation or to inject human priors via prompting. In contrast, our goal is to build a systematic curriculum over interaction steps that **trains** agents via RL to acquire the ability to leverage longer interaction horizons to improve performance. For prior approaches, such ability is not explicitly optimized during training. While scripted retries or prompting tricks can still tackle a few extra tasks with longer test-time horizons, their agents **lack the capacity to adaptively scale interaction**, which TTI agents are capable of.\n\n---\n## Weakness 2\n\n> 20/30 steps is still relatively short for complex tasks\n\nWe note that a 30-step horizon is already 2-5x longer than typical task lengths in WebArena (~6 steps), Mind2Web (~8 steps), and UI-TARS-1.5 training data (~15 steps), and aligns with the standard 30-step limit used in most benchmark frameworks. This provides sufficient room for exploration while remaining compute-efficient. We agree that longer horizons may be necessary in other domains and we encourage researchers to adapt step limits to their specific settings.\n\n---\n## Open Questions\n\n> How should TTI be applied to other tasks that require high costs for interaction and with non-recoverable states? If there is a place for thinking longer to be effective, how should TTI co-exist with thinking longer?\n\nWe think the following strategies might be worth exploring:\n1. Simulated interaction or planning can be used to reduce the number of costly real-world steps. For example, we can develop hierarchical TTI that differentiates between cheap simulated interactions and expensive physical actions, allocating horizon accordingly\n2. We can also develop risk-aware curricula that gradually expose agents to higher-cost actions, e.g., in robotics and in several games, you have the option of \"going back and retrying\", meaning there is a possibility of developing risk aware curricula\n\nAkin to how long CoT reasoning is not broadly applicable to every domain, long interaction lengths are also not, but this does not undermine the need for long CoT reasoning models.\n\nFurthermore, our results suggest that longer per-step thinking is best suited for local optimization, validation, or final decision verification, whereas external interaction is more effective for exploration, gathering missing information, and adapting to partial observability. We believe an important future direction is *hybrid test-time scaling*, where agents learn to allocate their compute budget between internal reasoning and external action based on the environment’s cost and the agent's uncertainty."}, {"review_id": "ndPxmgQgee", "reviewer": "Reviewer_kwcd", "summary": "The paper introduces Test-Time Interaction (TTI), a curriculum-based online reinforcement-learning method designed to scale the interaction of LLMs. The authors apply TTI to fine-tune Gemma 3-12B for web navigation tasks on both WebArena and WebVoyager. By progressively lengthening the agent’s interaction horizon during training, the resulting TTI model ,on average, surpasses other open-weight baselines on the two benchmarks. The paper also supplies ablations on the components of TTI and exposes typical success modes and failure patterns.", "questions": "1. In the GitHub, ESPN, and Coursera domains in Table 3 the base Gemma 3-12B outperforms its TTI-trained counterpart. Do these domains share characteristics that explain the drop in performance when applying TTI?\n2. The variant trained with a fixed horizon of h = 10 outperforms both the curriculum schedule and the h = 30 variant on several domains in Table 3. Why is this intermediate horizon especially effective for the Amazon, ArXiv, GitHub, and Coursera domains?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- Sound experimental design.\n- Case studies effectively illustrate both capabilities and typical failure modes.\n- The paper shows empiirically that careful horizon curricula matter at least as much as model size in this setting.\n\n### Weaknesses\n- Adding per-domain training curves (success rate vs. training steps) would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We appreciate your recognition of the originality and significance of our work. To address your questions regarding  more visualizations, **we will add new per-domain training curves to the paper**. Below, we provide responses to each comment.\n\n---\n\n## Weakness 1\n\n> Adding per-domain training curves (success rate vs. training steps) would be helpful.\n\nThis is valuable feedback that would strengthen our analysis significantly! Due to space constraints, we prioritized aggregate results and included training curves for representative domains (e.g., Amazon, HuggingFace) in Fig. 5 (right). However, we have now added **per-domain success rate trends across training iterations for all WebVoyager domains** in Appendix F.9 of the updated version and would be happy to highlight these further in the main paper.\n\nSince figures cannot be shown in the response (and neither can links), we summarize key findings below. The domains exhibit distinct learning dynamics, which we broadly group into three categories:\n- Fast-learning domains (e.g., BBC, Apple): Plateau after ~5–6 iterations.\n- Moderate-learning domains (e.g., Allrecipes, HuggingFace): Improve steadily over 8–10 iterations.\n- Prone-to-overfitting domains (e.g., GitHub, Amazon): Performance may decline in later stages, as the agent learns to explore more despite the test tasks being simple and not requiring exploration (see our response to Q1 below for further discussion).\n\nThese differences appear to stem from domain complexity and the base model’s familiarity with each domain. We believe these insights are an important step toward understanding when and where TTI is most beneficial, and they point to promising future directions like tailoring curriculum schedules to domain difficulty or agent priors.\n\n---\n\n## Question 1\n\n> In the GitHub, ESPN, and Coursera domains in Table 3 the base Gemma 3-12B outperforms its TTI-trained counterpart. Do these domains share characteristics that explain the drop in performance when applying TTI?\n\nThis is an excellent observation. In our analysis (Section 6.2 and Appendix F.6), we found that these domains are likely to be better represented in the base model's pretraining data, so the zero-shot agent already performs pretty good. In these cases, TTI’s added exploration can occasionally lead to unnecessary detours, analogous to how reasoning models can overthink on simple tasks. For example, the TTI agent may query external search tools when a direct filter suffices (GitHub), or compare multiple courses when one stands out clearly (Coursera). These are **recoverable inefficiencies of our approach**, not failures of the method. \n\nWe view the performance in GitHub, ESPN, and Coursera positively and as motivation for developing a more adaptive system in the future. Akin to how reasoning models benefit from learning when to invoke long CoTs and when to produce a response  directly (referred to as “dual mode thinking”), we can improve TTI by dynamically allocating interaction vs. per-step compute based on task instruction features. We view this as an important next step of our work, that can build upon the training protocols we introduce or distill our agent.\n\n---\n\n## Question 2\n\n> The variant trained with a fixed horizon of h = 10 outperforms both the curriculum schedule and the h = 30 variant on several domains in Table 3. Why is this intermediate horizon especially effective for the Amazon, ArXiv, GitHub, and Coursera domains?\n\nThe fixed horizon of h = 10 performs well in domains like Amazon, ArXiv, GitHub, and Coursera because many tasks in these settings have short, structured workflows, e.g., \"find a paper by title,\" \"open a GitHub repo,\" or \"enroll in a top-rated course.\" In such cases, a longer horizon may introduce unnecessary exploration that distracts from the optimal path, similar to overthinking in CoT models.\nHowever, this is not a limitation of TTI, but **a reflection of its general-purpose design**. While fixed h = 10 happens to be well-matched to simple and straightforward tasks, TTI is trained to adaptively scale horizons across diverse domains. As shown in Figure 5, TTI matches or exceeds h = 10 on most domains and excels in information-dense or partially observable settings like Allrecipes or Cambridge, where longer interaction is essential.\n\nWe believe future extensions of TTI could incorporate domain-aware horizon control to better align with task complexity. This is a future direction we plan to explore."}, {"review_id": "OFEJAZQ2Kx", "reviewer": "Reviewer_GJf2", "summary": "The paper tackles the often-overlooked problem of scaling an agent’s interaction count rather than merely its per-step reasoning. A simple “re-check” prompt shows that allowing a Gemma-3-12B web agent to take extra actions lifts WebArena success rates. Building on this observation, the authors introduce Test-Time Interaction (TTI)—an online RL recipe that (i) filters trajectories to keep only successful ones and (ii) trains under a curriculum that gradually expands the maximum number of steps per episode. On the WebVoyager and WebArena benchmarks, a TTI-fine-tuned Gemma-3-12B sets a state-of-the-art among fully open-source agents.", "questions": "1. Can TTI be used in general tasks and environments where each action is risky or costly and, if so, what changes would it need?\n2. How can we keep extra actions from becoming too expensive or dangerous while still reaping the benefits of longer interaction?", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents a new scaling axis—allowing more interaction steps rather than just deeper per-step reasoning.\n2. It spells out the credit-assignment and gradient-signal problems that arise with long horizons and mitigates them through a horizon curriculum combined with reward-filtered updates.\n3. The proposed method delivers empirical gains on web tasks.\n\n**Weaknesses**\n\n1. Extending the interaction horizon is a double-edged sword: every extra step incurs latency, compute, and energy overhead, but—more critically—the agent is given many more chances to do something harmful or irreversible. Even in seemingly “safe” web tasks, longer roll-outs increase the odds of clicking fraudulent ads, submitting a form twice, buying unwanted items, or spamming an email thread. In real-world domains (finance, healthcare, robotics) a single stray action can be catastrophic or legally non-compliant. Yet the paper leaves these risks unaddressed, creating a substantial gap for practical deployment.\n2. The horizon-scheduling curriculum that tackles the training difficulty of longer interactions is still largely heuristic.\n3. Main results come from a single run, so robustness to random seeds remains uncertain at full scale.", "was_revised": true, "final_justification": "I appreciate the authors' efforts to address my concerns. However, I am maintaining my original score for the following reasons:\n\n- I am still not convinced that test-time interaction scaling is a general and promising direction.\n- I find that the proposed method is neither significant nor original, especially given the authors’ rebuttal stating that the curriculum heuristic they used is a well-established practice.\n- While the issue of the single-run experiment has been resolved, I believe this constitutes only a minor improvement compared to the aforementioned unresolved concerns.", "rebuttal": "We thank the reviewer for their thoughtful feedback and for recognizing the empirical gains of our method. To address your concern about the number of experimental trials, we now **include additional runs**, demonstrating that our results are statistically consistent. We believe that these results should help address concerns regarding the efficacy of our approach.\n\n## At the outset, we would like to first address your concern that TTI is impractical because \"extending the interaction horizon is a double-edged sword.\"\n \nAt a high level, a similar argument can be made about any expressive agent/policy architecture: using a larger model, more interaction steps, or longer reasoning chains, could in principle represent policies that increase the probability of risky behavior or incur higher latency and compute costs. However, recent advances in reasoning and agentic tasks have demonstrated that with the right training objectives and workflows, we can guide such expressive policies toward useful solutions rather than pathological ones, leading to improved generalization and performance scaling.\n\nWe developed TTI in the same spirit. **The RL-based training used in TTI explicitly discourages risky or degenerate behaviors** like submitting forms incorrectly or clicking fraudulent links, because these behaviors minimize reward  (see our RL objective in Eq. 1). Thus, TTI allows the model to benefit from extended interaction only when it leads to better outcomes. Case studies in Section 6.2 also confirm that TTI agents learn strategic behaviors like backtracking and verification, not random exploration. In fact, pretty much like how in closed-domain reasoning, using reasoning improves model safety (e.g., see deliberative alignment from OpenAI), we believe that deliberative behaviors that use more interaction steps should only help an agent detect risk early on through exploration and planning, while still enabling it to solve the task eventually. \n\nLike other test-time scaling techniques (e.g., long-chain CoT prompting in LLMs), **TTI does introduce some overhead, but in exchange, it yields significantly more robust and capable behavior,** especially on complex tasks where shorter horizons fall short. Indeed, just as long CoT traces have become widely adopted in LLM reasoning despite their compute cost, we view TTI as an analogous strategy for agentic settings, where interaction reveals new information critical for success. **We therefore believe it would be unfair to reject our work based solely on presumed risks or overhead, particularly in light of the benefits.** \n\nBelow, we address other comments in detail.\n\n---\n\n## Weakness 3: Multiple Runs\n\n> Main results come from a single run, so robustness to random seeds remains uncertain at full scale.\n\nThank you for raising this point. First, we want to clarify that **results in Section 4 and 5.2 are already averaged over three runs**, as we noted in the caption of Figure 1 \"results averaged over 3 runs.\" Since we only include the average numbers in the paper, we also provide the error bars below, which we will add to the appendix.\n|Method|Success Rate on WebArena Subset(%)|\n|-|-|\n|Baseline|23.81±1.6|\n|Fixed h=5|24.59±1.5|\n|Fixed h=10|25.19±1.2|\n|Fixed h=20|29.28±1.2|\n|TTI Additive Schedule|29.5±1.4|\n|TTI Multiplicative Schedule|32.35±1.3|\n\nFor large-scale RL training on WebVoyager and WebArena (Table 3 & 4), we note that the training tasks are over 100K, making rollout collection super expensive. Due to resource constraints, we thus prioritized domain and benchmark coverage over running multiple seeds. We note that recent work on web agents that are published in conferences, such as Learn-By-Interact (ICLR 2025), AgentOccam (ICLR 2025), WebRL (ICLR 2025), and Agent Workflow Memory (ICML 2025), all report single-run results on large-scale benchmarks due to the cost of training and evaluation. We agree that adding multiple seeds is valuable and will include this in the final version, though we expect our results and conclusions to be solid.\n\nFinally, several other reviewers (kwcd, 13jF) independently noted the consistency of performance across 20+ domains and 2 benchmarks. Besides, our results can be  reproduced with the released codebase and training details, and we plan to open-source checkpoints for verification. These all improve the reproducibility and robustness of our work.\n\n---\n## Weakness 2: Curriculum Heuristic\n\n> The horizon-scheduling curriculum that tackles the training difficulty of longer interactions is still largely heuristic.\n\nWhile our curriculum does not emerge from a theoretical derivation, **it is grounded in well-established practices from both RL and LLM training rather than merely being ad-hoc heuristics**. Specifically, \n\n**Horizon scaling** has been shown effective in prior work such as CodeT [1] and Reflexion [2], though none of these works have proposed a formal training mechanism. Our horizon-based schedule mirrors established RL curricula [3, 4] that gradually increase task difficulty to improve stability under sparse rewards.\n\nAs for the specific **curriculum schedule**, our use of a multiplicative curriculum (Eq. 3) is not purely heuristic because: (1) multiplicative schedule is very common in existing reasoning works; (2) we study several design choices such as additive schedule as ablations. Both of them outperform baseline agents and test-time-only strategies like budget forcing or best-of-n prompting (Table 2 and Figure 3). However, the multiplicative schedule is empirically more effective, so we adopt it for later experiments; (3) on WebVoyager,  TTI outperforms fixed-horizon baselines (Figure 4, 6). Such strong downstream results show the effectiveness of our curriculum design.\n\n[1] CodeT: Code Generation with Generated Tests. ICLR 2023.\n\n[2] Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.\n\n[3] WebRL: Training llm web agents via self-evolving online curriculum reinforcement learning. ICLR 2025.\n\n[4] AutoWebGLM: A large language model-based web navigating agent.  KDD 2024.\n\n---\n\n## Question: Applicability to Risky Settings\n\n>  Can TTI be used in general tasks and environments where each action is risky or costly and, if so, what changes would it need? \n\n> How can we keep extra actions from becoming too expensive or dangerous while still reaping the benefits of longer interaction?\n\nYes, TTI can be adapted to general environments where actions are risky or costly as future work. For example, we can introduce cost-aware curricula that assign higher weights to risky actions and optimize performance subject to total cost constraints, similar to safe RL frameworks [1].  We can also enforce risk-bounded exploration by limiting high-impact actions during early training, and gradually relaxing constraints.  For example, in robotics, TTI could first allow only low-cost actions such as camera adjustments, then gradually introduce manipulation steps, with irreversible actions gated behind explicit confirmation.\n\nAs for balancing longer interaction and the quality of each action, our findings in the paper already suggest a natural division of labor between internal reasoning and external action: internal \"thinking\" for planning and verification, and external interaction for exploration and information gathering. Thus, we can develop hybrid test-time scaling, where agents learn to dynamically allocate compute based on environmental cost and task uncertainty to achieve robust performance gains. We will add the above discussion to Section 7.\n\nThese extensions are important directions for applying TTI to more real-world settings, but are beyond the current scope of our work. We think it would be unfair to reject our paper based on the presumed risks in diverse environments.\n\n[1] A Review of Safe Reinforcement Learning: Methods, Theory and Applications, Gu et al., 2024."}, {"review_id": "7lH8WHHFvU", "reviewer": "Reviewer_13jF", "summary": "The authors investigate different ways to scale up inference compute for web agents. Interestingly, per-step inference compute  does not improve performance as much as longer interactions with the environment. Longer episodes allow the agent to collect more information to achieve its tasks. While it is possible to prompt an agent to perform more interactions, it is not obvious how to teach one using RL. Surprisingly, the authors show that a simple curriculum allow the agent to learn how to interact longer with a task to succeed.", "questions": "* Do you plan to train your agent with a more sophisticated method than online behavior cloning? Long horizon tasks such as web browsing could benefit greatly from a method that performs credit assignment such as PPO.\n* The drop in performance in Figure 4 as training steps increase is bizarre. Can you explain why the performance drops on all domains for interaction 5 and 10?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n## Simplicity\n* Despite the simplicity of curriculum learn and online filtered behavior cloning, the resulting agent performs favorably compare to the baselines.\n\n## Empirical performance\n* Figure 2 validates both the need to have longer episodes and the need for training, since performance improvement is task dependent.\n* Table 3 shows clear improvement of TTI over the baselines. The improvement is smaller for WebArena, but still positive.\n\n## The ablations are interesting\n* In section 4.2, it is shown that per step test time compute (best of n and budget forcing) do not work as well as interaction scaling.\n\n# Weaknesses\n\n## Presentation\n* The legend in Figure 6 is missing the circles.\n* What environment is Table 3? WebVoyager?\n\n## Compute cost\n* Compute cost is brought up constantly in the paper, would it be possible to include them?\n\n# Minor\nTypo: Comaprisons.", "was_revised": true, "final_justification": "Test time compute has been very popular in the recent months and this paper provide a fresh perspective on how it should be used for browser based environments. The method is simple and carefully evaluated, therefore I do recommend to accept the paper.", "rebuttal": "We sincerely thank the reviewer for the positive and thoughtful feedback. We're glad that you found our approach simple yet effective, and we appreciate your recognition of the paper’s originality, significance, and empirical contributions. To address your question regarding computational costs, **we add additional analysis on the token count, number of forward passes, and training time of TTI**. Below, we provide a detailed response to each comment.\n\n\n---\n\n## Weakness 2: Compute Cost\n\n> Compute cost is brought up constantly in the paper, would it be possible to include them?\n\nThank you for this suggestion. While we reported token-level compute costs in Figure 3 (top), we agree that a more detailed breakdown for large-scale training would be helpful. We update Appendix F.3 to include the following table for the WebVoyager experiments (Table 3). All settings use Gemma 3 12B. Statistics computed on the training set. Training time includes both rollout collection and model update performed on 4 H100s for 10 iterations.\n\n\n| Method         | Avg Total Tokens per Rollout | Avg Model Calls (# Steps) per Rollout | Training Wallclock Time (hrs) |\n|----------------|--------------------------|------------------------------|-------------------------------|\n| Baseline (Prompting Only)    |        810         |         10.8        |              -         |\n| Fixed h = 10   |     700       |            9.5      |       10            |\n| Fixed h = 30   |      760            |         14.1           |         25              |\n| TTI    |      640          |       10.5       |       20               |\n\n\nIn summary, TTI improves success rate while requiring fewer training compute than fixed h=30 baselines, making it both more effective and efficient in our current setup.\n\n---\n\n## Question 2: Fixed-Horizon Baseline Performance\n\n> The drop in performance in Figure 4 as training steps increase is bizarre. Can you explain why the performance drops on all domains for interaction 5 and 10?\n\nWe hypothesize that the drop in performance for short-horizon agents is due to:\n- Early overfitting: With smaller horizons, the agent initially learns to exploit short paths but quickly overfits to local patterns that do not generalize well on the held-out set.\n- Limited rollout diversity: These settings fail to expose the agent to exploratory trajectories or recovery strategies, leading to a plateau or even regression as training progresses.\n- Reduced data coverage: Since each rollout is shorter, fewer states are visited overall, reducing the diversity of successful examples stored in the replay buffer.\n\nWe will add discussion of these dynamics in Section 5.1. The drops in fixed-horizon training actually motivate our curriculum approach. TTI (Figure 6) shows more stable learning because its curriculum helps mitigate this collapse by exposing the agent to longer interactions progressively, improving both generalization and robustness.\n\n---\n\n## Question 1: Future Work\n\n> Do you plan to train your agent with a more sophisticated method than online behavior cloning? Long horizon tasks such as web browsing could benefit greatly from a method that performs credit assignment such as PPO.\n\nYes, we agree that more advanced RL algorithms such as PPO, GRPO, or Q-learning-based methods could offer improved credit assignment over online filtered behavior cloning, particularly for long-horizon tasks with sparse rewards. We chose to begin with a simple and robust baseline to isolate the effects of curriculum over interaction horizon, but we absolutely see value in applying policy-gradient-based methods or even value-based methods in future work.\n\nAs noted in Section 7, combining TTI with more advanced RL methods is a promising direction. In particular, using PPO could help stabilize training with longer horizons and enable more nuanced learning of exploration strategies. We plan to explore this in the future.\n\n\n---\n\n## Weakness 1: Presentation\n\nFigure 6 legend: Thank you for pointing this out. We have fixed the missing legend markers in the updated version.\n\nTable 3 environment: Table 3 reports results on WebVoyager (we now explicitly clarify this in the caption and surrounding text).\n\nTypo (“Comaprisons”): Fixed—thank you for catching this."}]

Author rebuttal:

Meta-review: This submission introduces a curriculum-based online reinforcement learning method (test time interaction, tti). Its goal is to scale the interaction of agents rather than only increasing per-step reasoning. The authors argue that longer interaction horizons allow agents to gather more information, adapt strategies, and achieve improved performance on complex tasks. Empirical evaluations on WebVoyager and WebArena show that a Gemma-3 12B model fine-tuned with tti achieves new sota results among open-source agents. Ablation studies and case analyses also show that tti enables adaptive exploration, backtracking, and dynamic re-planning. This helps provide a complementary scaling axis to traditional "thinking longer" approaches.

Strengths of this submission: 
1. reviewers kwcd, 13jF, and nVXA believe that tti demonstrates consistent gains over baselines on two challenging web benchmark. The experiments also cover ablations, case studies, and new baselines comparing per-step reasoning versus interaction scaling.
2. reviewers kwcd, 13jF like the technical idea presented. It identifies interaction scaling as an underexplored dimension for test-time scaling, showing its practical importance to existing approaches.
3. reviewers nVXA and kwcd consider that this submission is well presented.

Weaknesses of this submission: 
1. Safety and applicability concerns: reviewer GJf2 strongly believes that scaling interaction steps can introduce risks in real environments, such as harmful, risky, or irreversible behaviors, which are insufficiently discussed.
2. reviewer nVXA finds that the intuition that longer interaction horizons improve agent performance has been observed in prior works, and the contribution may be incremental.
3. reviewers GJf2 and nVXA also mention that the curriculum design for scaling horizons is largely heuristic, lacking theoretical grounding or cross-domain validation.

Despite concerns about safety, the paper provides a compelling contribution by establishing interaction scaling as a complementary and underexplored axis for improving agent performance. The experiments are relatively strong, with ablations, comparisons against strong baselines, and sota results on two benchmarks. While safety concerns raised by one reviewer GJf2 are valid for deployment in high-risk domains, they do not undermine the technical contributions and empirical insights of this work in the evaluated setting.

During the rebuttal, the authors added new analyses addressing several reviewers' concerns. They provided per-step thinking token trends, new RL baselines for scaling per-step reasoning, compute cost breakdowns, and per-domain training curves, which addressed questions from reviewers nVXA, kwcd, and 13jF. However, reviewer GJf2 maintained concerns about potential safety risks and the broader applicability of tti, arguing that longer interaction horizons could lead to unsafe behaviors in risky domains; the authors responded by clarifying that tti’s RL objective discourages harmful actions and discussing safe RL extensions and sandboxed environments, though these points did not fully convince reviewer GJf2. The other three reviewers expressed satisfaction with the clarifications and maintained their positive evaluations, supporting acceptance. 

All factors considered, this submission is given an acceptance recommendation. The major rational of this decision lies in 1) concerns about long-horizon reasoning producing risky behaviors do not necessarily imply a negative impact on the contribution of the proposed technique, and 2) the perspective provided in this submission is attractive though not perfect, and the ac does not find strong reasons to reject this submission.
Decision: Accept (poster)