Title: Thinking vs. Doing: Improving Agent Reasoning by  Scaling Test-Time Interaction
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Agent

Web Agent

Reasoning
Abstract: Test-time scaling in agentic tasks often relies on generating long reasoning traces ("think" more) before acting, but this does not allow agents to acquire new information from the environment or adapt behavior over time. In this work, we propose scaling test-time interaction, an untapped dimension for test-time scaling that increases the agent's interaction horizon to enable rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we situate our study in the domain of web agents. We first show that even prompting-based interaction scaling can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI, a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their interaction lengths during rollout. Using a Gemma 3 12B model, TTI sets a new state-of-the-art among open-source agents trained on public data on WebVoyager and WebArena. Case studies further reveal that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-action compute, offering new avenues for training robust and adaptive agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate different ways to scale up inference compute for web agents. Interestingly, per-step inference compute  does not improve performance as much as longer interactions with the environment. Longer episodes allow the agent to collect more information to achieve its tasks. While it is possible to prompt an agent to perform more interactions, it is not obvious how to teach one using RL. Surprisingly, the authors show that a simple curriculum allow the agent to learn how to interact longer with a task to succeed.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n## Simplicity\n* Despite the simplicity of curriculum learn and online filtered behavior cloning, the resulting agent performs favorably compare to the baselines.\n\n## Empirical performance\n* Figure 2 validates both the need to have longer episodes and the need for training, since performance improvement is task dependent.\n* Table 3 shows clear improvement of TTI over the baselines. The improvement is smaller for WebArena, but still positive.\n\n## The ablations are interesting\n* In section 4.2, it is shown that per step test time compute (best of n and budget forcing) do not work as well as interaction scaling.\n\n# Weaknesses\n\n## Presentation\n* The legend in Figure 6 is missing the circles.\n* What environment is Table 3? WebVoyager?\n\n## Compute cost\n* Compute cost is brought up constantly in the paper, would it be possible to include them?\n\n# Minor\nTypo: Comaprisons.\n\nQUESTIONS:\n* Do you plan to train your agent with a more sophisticated method than online behavior cloning? Long horizon tasks such as web browsing could benefit greatly from a method that performs credit assignment such as PPO.\n* The drop in performance in Figure 4 as training steps increase is bizarre. Can you explain why the performance drops on all domains for interaction 5 and 10?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the often-overlooked problem of scaling an agent’s interaction count rather than merely its per-step reasoning. A simple “re-check” prompt shows that allowing a Gemma-3-12B web agent to take extra actions lifts WebArena success rates. Building on this observation, the authors introduce Test-Time Interaction (TTI)—an online RL recipe that (i) filters trajectories to keep only successful ones and (ii) trains under a curriculum that gradually expands the maximum number of steps per episode. On the WebVoyager and WebArena benchmarks, a TTI-fine-tuned Gemma-3-12B sets a state-of-the-art among fully open-source agents.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper presents a new scaling axis—allowing more interaction steps rather than just deeper per-step reasoning.\n2. It spells out the credit-assignment and gradient-signal problems that arise with long horizons and mitigates them through a horizon curriculum combined with reward-filtered updates.\n3. The proposed method delivers empirical gains on web tasks.\n\n**Weaknesses**\n\n1. Extending the interaction horizon is a double-edged sword: every extra step incurs latency, compute, and energy overhead, but—more critically—the agent is given many more chances to do something harmful or irreversible. Even in seemingly “safe” web tasks, longer roll-outs increase the odds of clicking fraudulent ads, submitting a form twice, buying unwanted items, or spamming an email thread. In real-world domains (finance, healthcare, robotics) a single stray action can be catastrophic or legally non-compliant. Yet the paper leaves these risks unaddressed, creating a substantial gap for practical deployment.\n2. The horizon-scheduling curriculum that tackles the training difficulty of longer interactions is still largely heuristic.\n3. Main results come from a single run, so robustness to random seeds remains uncertain at full scale.\n\nQUESTIONS:\n1. Can TTI be used in general tasks and environments where each action is risky or costly and, if so, what changes would it need?\n2. How can we keep extra actions from becoming too expensive or dangerous while still reaping the benefits of longer interaction?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper introduces Test-Time Interaction (TTI), a curriculum-based online reinforcement-learning method designed to scale the interaction of LLMs. The authors apply TTI to fine-tune Gemma 3-12B for web navigation tasks on both WebArena and WebVoyager. By progressively lengthening the agent’s interaction horizon during training, the resulting TTI model ,on average, surpasses other open-weight baselines on the two benchmarks. The paper also supplies ablations on the components of TTI and exposes typical success modes and failure patterns.

STRENGTHS AND WEAKNESSES:
### Strengths
- Sound experimental design.
- Case studies effectively illustrate both capabilities and typical failure modes.
- The paper shows empiirically that careful horizon curricula matter at least as much as model size in this setting.

### Weaknesses
- Adding per-domain training curves (success rate vs. training steps) would be helpful.

QUESTIONS:
1. In the GitHub, ESPN, and Coursera domains in Table 3 the base Gemma 3-12B outperforms its TTI-trained counterpart. Do these domains share characteristics that explain the drop in performance when applying TTI?
2. The variant trained with a fixed horizon of h = 10 outperforms both the curriculum schedule and the h = 30 variant on several domains in Table 3. Why is this intermediate horizon especially effective for the Amazon, ArXiv, GitHub, and Coursera domains?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces test-time interaction scaling as a complementary axis to the usual “think-longer” compute scaling for agentic tasks. By allowing an agent to act for more steps before termination, the authors show that the agent can gain new external information by simply continuing to interact with the environment once again.  They further propose TTI—a curriculum-based online RL method that gradually lengthens the rollout horizon with a multiplicative schedule. Empirically, a Gemma-3 12 B model trained with TTI attains 66.5% success on WebVoyager and 26.1% on WebArena. Case studies illuminate how the agent learns to trade off exploration vs. exploitation and reveal residual failure modes.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Clear Writing**: The paper is well-structured, with a logical flow from motivation to method design to empirical results. Figures and tables are clear and informative.\n\n2. **Well-Designed Preliminary Experiments**: The curriculum design is methodologically sound, with careful ablations on the effects of prompting towards TTI and acting longer. Many experiment details are provided, which is helpful for reproducibility.\n\n3. **Detailed Case Studies**: The paper includes insightful case studies that illustrate how the agent learns to balance exploration and exploitation, as well as the types of errors it makes.\n\n### Weaknesses\n\n1. **Novelty of TTI**: Though the experiments are thorough with authentic RL curriculum, the core idea of scaling interaction horizons is intuitive and not novel. Besides a boarder demonstration of TTI in UI-TARS-1.5 [1], [2,3] have already shown that longer interaction horizons can improve agent performance substantially.\n\n2. **Limited Interaction Horizon**: The maximum interaction horizon of 20/30 steps is still relatively short for complex tasks. While the paper shows that TTI improves performance on current benchmarks, it does not explore how much horizons could be scaled further like [1].\n\n3. **Lack of Metrics/Baselines in RL Training**: The paper does not report how the thinking tokens change during RL training, along with the change of interaction times. Also, RL training to incentivize longer thinking (e.g., mixing reasoning data in agent RL) is not compared with RL training for TTI. This makes it hard to assess the effectiveness of the RL training in improving interaction horizons compared to improving thinking lengths. Most other open-source baselines are from other model backbones, limiting insights into the effectiveness of TTI.\n\n4. **Too Compact Layout**: The paper is quite compact, with many figures and tables crammed into a small space, and still large amount of details are buried in the appendix. It is recommended to summarize the case studies instead of showing all the depiction. Training details such as data generation pipeline should be described more clearly as a part of the method.\n\n\n[1] [UI-TARS-1.5 Blog](https://seed-tars.com/1.5/).  \n[2] CodeT: Code Generation with Generated Tests. ICLR 2023.  \n[3] Reflexion: Language Agents with Verbal Reinforcement Learning. NIPS 2023.\n\nQUESTIONS:\n1. While the paper shows interaction horizon is increased with RL training under TTI curriculum, it is not clear how the agent learns to think longer during training. For instance, in Fig 4, after 400 steps training, the agent (h=20) stop increasing overall interaction horizon but still improves the performance. Could you clarify this and the saying in L248, and provide the thinking lengths?\n2. In Fig 6, the \"check-again\" strategy seems to be less effective to increase the trajecory lengths. Could you provide interaction times for the RL trained agent with \"check-again\" strategy? If it fails to improve RL (well-)trained agents, how could we further increase the performance by increasing interaction horizons? Also, please remove the centering in Fig 5 (left), and remain the original interaction times.\n3. Since the observation history is only recent 3 steps, how does the agent handle long-term dependencies when the interaction horizon is increased? And, under this setting, how does \"check-again\" prompting different to Reflexion [1]?\n4. In L127, the paper mentions that prior works \"usually scale the number of thinking tokens at each step\" without reference. What are the prior works? Could you compare current TTI with these prior works under prompting and RL training?\n5. **Open Question:** How should TTI be applied to other tasks when some tasks require high costs for interaction and with non-recoverable states (e.g., robotics, games, financial decision making)? If there is a place for thinking longer to be effective, how should TTI co-exist with thinking longer?\n\n*If the authors provide evidence addressing Q1–Q4, corresponding rating score could increase.*\n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning. NIPS 2023.\n\nLIMITATIONS:\nThe authors acknowledge domain restriction to web-navigation and compute overhead of long horizons. However, the paper does not discuss the limitations of TTI in terms of generalization beyond web browsing. The application of overly long interaction horizons to crucial tasks with high costs for interaction and non-recoverable states (e.g., robotics, games, financial decision making) could be risky. Also, interacting with users in real-world applications could be a feasible scalable dimension. Overall, these limitations could block the applicability of TTI to many real-world tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}