GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
Authors: Haolong Yan
Yeqing Shen
Xin Huang
Jia Wang
Kaijun Tan
Zhixuan Liang
Hongxin Li
Zheng Ge
Osamu Yoshie
Si Li
Xiangyu Zhang
Daxin Jiang
Venue: NeurIPS 2025 poster ()
Keywords: GUI Environment
Large Vision Language Model
Multi-Turn Reinforcement Learning
Agent
Primary area: reinforcement_learning
TL;DR: We introduce GUI Exploration Lab, a flexible simulator for GUI agent navigation. Experiments show a staged SFT + RL approach (especially multi-turn RL) significantly boosts navigation and exploration capabilities.

Abstract: With the rapid development of Large Vision Language Models, the focus of Graphical User Interface (GUI) agent tasks shifts from single-screen tasks to complex screen navigation challenges. 
However, real-world GUI environments, such as PC software and mobile Apps, are often complex and proprietary, making it difficult to obtain the comprehensive environment information needed for agent training and evaluation. This limitation hinders systematic investigation and benchmarking of agent navigation capabilities.
To address this limitation, we introduce GUI Exploration Lab, a simulation environment engine for GUI agent navigation research that enables flexible definition and composition of screens, icons, and navigation graphs, while providing full access to environment information for comprehensive agent training and evaluation.
Through extensive experiments, we find that supervised fine-tuning enables effective memorization of fundamental knowledge, serving as a crucial foundation for subsequent training. Building on this, single-turn reinforcement learning further enhances generalization to unseen scenarios. Finally, multi-turn reinforcement learning encourages the development of exploration strategies through interactive trial and error, leading to further improvements in screen navigation performance.
We validate our methods on both static and interactive benchmarks, demonstrating that our findings generalize effectively to real-world scenarios.
These findings demonstrate the advantages of reinforcement learning approaches in GUI navigation and offer practical guidance for building more capable and generalizable GUI agents.

GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
Haolong Yan 1- Yeging Shen
Xin Huang
Jia Wang?
Kaijun Tan
Zhixuan Liang
Hongxin Li
Zheng Ge
Osamu Yoshie
Si Xiangyu Zhang?
Daxin Jiang?
Beijing University of Posts and Telecommunications
StepFun "Wseda University
Institute of Automation, Chinese Academy of Sciences
Abstract
With the rapid development of Large Vision Language Models, the focus of Graph- ical User Interface (GUT) agent tasks shifts from single-screen tasks complex screen navigation challenges. However, real-world GUT environments, such as PC software and mobile Apps, are often complex and proprictary, making it difficult to obtain the comprehensive environment information needed for agent training and evaluation. This limitation hinders systematic investigation and benchmarking of agent navigation capabilities. To address this limitation, we introduce GUI Ex- ploration Lab, simulation environment engine for GUI agent navigation research that enables flexible definition and composition of screens, icons, and navigation graphs, while providing full access to environment information for comprehen- sive agent training and evaluation. Through extensive experiments, we find that supervised fine-tuning enables effective memorization fundamental knowledge, serving as crucial foundation for subsequent training. Building this, single- turn reinforcement learning further enhances generalization to unseen scenarios. Finally, multi-turn reinforcement learning encourages the development of explo- ration strategies through interactive trial and error, leading to further improvements in screen navigation performance. We validate methods both static and interactive benchmarks, demonstrating that our findings generalize effectively to real-wor world scenarios. These findings demonstrate the advantages of reinforcement learning approaches GUI navigation and offer practical guidance for building more capable and generalizable GUT agents.
1 Introduction
Large Vision-Language Models (LVLMs) 29 38 drives significant advances in the development of GUI agents [31 44 28). At high level, GUI agent tasks [35) 12 45 46) fall into two main categories: single-screen task and screen navigation. For example, the task "Order an iPad from the Apple website with GUI-Agent engraving" screen task (e.g., entering the required information to place an order) and screen navigation (e.g., navigating to the iPad purchase page). As the grounding capabilities of current mainstream LVLMs 28 I8 continue to improve, tasks such as form completion and target icon selection no 13 43
be decomposed into subtasks such single-
Equul contribution. 'ooresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Start Page
Target Page
sigps
step4
steps
A. Transition relationships from start page larget page
B. Navigation map
C. SFT Memorize
D. ST-RL Generalize
E. MT-RL Explore
Figure 1: (A) Page transition sequences: green arrows show optimal paths, red arrows show detours. (B) Tree-structured navigation map with optimal (green) and redundant (red) paths. (C) SFT enables memorizing page transitions for basic navigation. (D) ST-RL generalizes to unseen paths, enhancing navigation. (E) MT-RL improves exploration and task success through environment interaction.
longer pose major obstacle to task completion. Instead, the main challenge for GUI agents now lies in navigating complex screens [31], which requires understanding the environment's 's navigation map and clicking the correct icons reach the target page.
To enhance the screen navigation capabilities of GUI agents, it crucial to have an environment that provides accurate screen layout information and inter-screen navigation graphs for training and evaluation. However, real-world GUI environments, such as PC software and mobile Apps, are often complex and proprietary. making difficult obtain the comprehensive environment information needed for these purposes [2 46 45). This limitation hinders systematic investigation and benchmarking of agent navigation capabilities. To address this, introduce GUI-Exploration Lab (GE-Lab)), simulation environment engine specifically designed for advancing GUI agent navigation, which screens, icons, and inter-screen navigation graphs can be flexibly defined, while providing full access to environment information for comprehensive training and evaluation.
Within the GE-Lab environment, illustrated in Figure it is straightforward to obtain both single- step transition knowledge, such as navigating from the current screen node to a new node by clicking specific icon, and multi-step transition knowledge, which involves reaching any target screen node from the current node through a sequence of actions (see Figure [[, B). Existing approaches based on SFT require large volumes of data and heavily depend on expert trajectories [34 20 5 31).
However, collecting high-quality trajectory data on real-world GUI platforms is costly [2]. and is challenging to assess the coverage of collected trajectories relative the full set of possible transitions available on the platform. In contrast, the GE-Lab environment enables low-cost scaling of trajectory data collection and facilitates systematic ablation studies. As shown in Figure [TC. SFT provides crucial foundation for screen navigation, suggesting that sufficiently large trajectory datasets can enable accurate navigation between arbitrary screens. In Figure [C. only nodes with degree represent training data that contains knowledge about two connected nodes. This does not imply that every node the actual state transition graph has degree of 1.
Nevertheless, SFT exhibits limited generalization to unseen environments 9 44
28 27 5). Recent
ST-RL methods, such UI-TARS [3] and UI-RI 28 leverage preference optimization and rule- based rewards to improve generalization. In the GE-Lab simulation environment, access to each screen's 's layout and navigation graph enables low-cost experimentation with various ST-RL methods. As shown in Figure DD, these methods demonstrate better generalization than SFT. ST-RL methods focus on correcting single-step actions, while real-world interactions often involve longer action chains. MT-RL, inspired by RAGEN 42] and VAGEN 40 offers advantages for training agents through interaction. GE-Laa provides the first GUT simulation engine for MT-RR. with interactive training and accurate reward signals, addressing practical RL challenges through curriculum learning

---

and data proportion adjustment. As shown in Figure MT-RL encourages exploration and further improves task success rates, Figure ITA B illustrate the agent's navigation process, including backtracking and successful task completion. Moreover, MT-RL, reduces dependence on annotated data by relying on instructions and interactive environments with reward signals. Moreover, we validate our methods on both static and interactive benchmarks, demonstrating that our findings generalize effectively to real-world scenarios.
To summarize, our key contributions are the following: (1) We present GE-Lab, a simulation envi- ronment engine that flexibly defines screens, icons, and navigation graphs, providing full access to environment information for comprehensive GUI agent training and evaluation. (2) Extensive experi- ments in GE-Lab reveal a clear training paradigm: supervised fine-turr fine-tuning memorizes fundamental navigation knowledge, ST-RR enhances generalization to unseen scenarios, and MT-RL promotes exploration and further boosts navigation performance while reducing reliance on annotated data. (3) Our systematic benchmarking clarifies the strengths of reinforcement learning in GUI navigation and offers concrete guidance for building more generalizable agents.
2 Related work
2.1 GUI Agents
In recent years, GUI agents experience significant evolution. Early methodologies predominantly rely on HTML structures and accessibility trees [2 substantial cross-platform compatibility challenges due potential information gaps across diverse software environments. The advent of LVLMs marks pivotal shift in this domain [14, 43 8). facilitating the integration of GUI comprehension and device control within unified frameworks. Current research in LVLM-powered GUI agents follows two primary technical trajectories: one leverages closed-source models with advanced prompt engineering [48]
52 25). yet these approaches encounter
21
39 while the other
enhances open-source models through fine-tuning for GUI interaction 34
20
5
13.. Existing
agents L3 31 28 18 exhibit robust visual grounding capabilities, effectively supporting basic
operations such as target icon selection. However, the principal challenge lies not in executing individual clicks but in navigating complex screen transitions based on task objectives. Addressing this navigation challenge, UI-TARS [31 extended fundamental grounding capabilities by introducing several GUI-specific competencies, including element description, GUI dense captioning, and state transition captioning, while incorporating task decomposition and trial-and-error strategies to learn inter-screen transition. Nevertheless, SFT necessitates a large volume of manually annotated real 13 31 and exhibits limited generalization unseen environments 27 51. Consequently, recent research on agents increasingly shifts towards RL approaches.
GUI trajectory data [34 20 5
To evaluate GUI agent performance, two primary categories of benchmarks are developed: static benchmarks, including ScreenSpot [8) ScreenSpot-v2 43 FuncPred [19], MoTIF (4, Refexp [3)
Llamatouch [50). VWB AG (24), and VWB FG [24] primarily assess task execution correctness through ground truth matching to evaluate grounding and planning accuracy; and interactive bench- marks AndriodWorld (35) and OSWorld [45), which enable agents to execute tasks within virtual machines and determine task success through components that evaluate actual task completion.
2.2 Reinforcement Learning
RL is a powerful paradigm for training agents to make optimal decisions through interactions with environment 3 10), enabling them to autonomously learn complex behaviors by maximizing
cumulative rewards. UI-TARS (31] employs Direct Preference Optimization (DPO) (32], leveraging
incorrect rollout actions paired with human-correcter actions for error correction. DeepSeck-RI demonstrates the efficacy of rulc-based RL for mathematical problem-solving Subsequently,
36
26
41
30
13
6
[1] 7 37 extende this framework RL, achieving significant improvements in
vision-related tasks such as visual grounding. UI-R1 [28] and GUI-R1 44 utilize Group Relative Policy Optimization (GRPO) [6). training with model rollouts and rule-based rewards guided by ground truth. However, these methods focus solely on single-step action correction, whereas real- world interactions are often more complex and involve longer action chains. Outside the GUI domain, several innovative agent systems [17 16 22 40) leverage interactive environments to train agents through reinforcement learning, thus enhancing their decision-maken and execution capabilities. Notably, RAGEN [2 and VAGEN [40] leverage MT-RL strategies to effectively enhance agents'
3

---

capabilities in environmental interaction and multi-step reasoning. These developments inspire our approach of combining MT-RL with the GE-Lab environment establish robust screen navigation capabilities for GUT agents, addressing a critical challenge in real-world GUT environments.
3 Method
3.1 GUI Exploration Lab
To address the challenges posed by complex engineering issues real-world GUT platforms that make them unsuitable for interactive environment training. we propose GE-Lab, novel simulation environment engine that generates diverse GUI environments for flexible agent training and evaluation. As illustrated in Figure [B, model the simulated environment as tree structure where nodes represent screens and edges represent clickable transitions between screens. Both the number of nodes and the connectivity of edges within the environment are fully customizable. Specifically, GE-Lab first generates a graph of the environment based on user-defined parameters, which determines the navigation relationships between different screens. Then, GE-Lab renders screen for each node of the graph, with each screen displaying n icons (where is the degree of the node), where clicking each
icon triggers screen transition. The icons appearing Internet and randomly sampled from an icon library, intentionally selected to be uncommon in typical applications and randomly named to prevent the model from leveraging prior knowledge about icon semantics, thereby enhancing the diversity of the simulation environment. During environment construction, both the selection and placement of icons are randomized, facilitating robust OOD lesting. In addition to these randomly assigned functional icons, each screen also contains "home" and "back" icons, enabling navigation the root and parent nodes.
collected from the
During training, the metadata generated by GE-Lab allows for the construction of screen navigation tasks. For instance, the page 0 node, task instruction such "From page_ to page 6" can be generated. Leveraging the navigation map, it is easy to synthesize trajectory data between arbitrary screens low cost, supporting both shortest-path and redundant-pait trajectories. Furthermore, agents can be trained interactively: the agent receives a image of the current screen and outputs an action, the environment executes the action to transition to a new state, and the environment can accurately determine whether the agent reaches the target screen, providing precise rewards. During evaluation, the simulation environment readily supports both static benchmarks and end-to-end interactive assessments.
3.2 Partially Observable Markov Decision Process Framework
3.2.1 Problem Formulation
The operation of GUI Agent involves step-by-step interaction with an environment. The GUI agent operates under conditions of partial observability, wherein it lacks access to the complete environmental state and must instead identify interactive regions based solely on the current screen state. Through the integration of historical states with present observations to inform decision- -aaiinnn processes, this navigation paradigm is appropriately formalized within the POMDP framework, defined by the components A, O, T), shown in Figure These components correspond to the task space Ut action space A, state space S, and observation space O. The probabilistic transitions between states governed by T S A P(S), which maps a state and action distribution over the next possible states.
Functionally, the GUI agent acts policy æ, mapping observations to actions O - A. Given a specific task u E Ut the agent executes a series of actions aiming for completion. At each time step using the current observation The environment then evolves to a subsequent state E according to the transition model T. Optionally, the agent may receive reward r R(s, a, s'), which is calculated based the reward function R: S x A
O, the agent's policy selects the next action E A.
3.2.2 Agent Protocol
Our agent receives three primary input components: task instruction, history, and current state. The task instruction defines navigation objectives such as "From page_20 221 to page_1511 151". The

---

Op
Gg
Pp-1
Og
Ft
ST-RL
Rollout
GUI Agent
Feadback
Task
Execute
d2
Rolout
GE-Lab
A. Framework
B. Reinforcement Learning
User:
Instruction: From page_ p5g History: step1: click home icon on p2ge_221, Observation: <images*
Assistant: " Explain: dick Hotel icon on page_ 0. Action: clck)635.65)*
C. Partially Observable MDP
Figure 2: (A) Overview of the interactive training framework: the GUI agent receives observations (o), actions (a), rewards (r), and interacts with the GE-Lab environment to receive feedback and execute actions. (B) Reinforcement learning: ST-RL operates on pre-constructed trajectories, while MT-RL extends multi-step online rollouts, enabling the agent to generate observation-acte sequences interactively within the environment. (C) The navigation task is formalized as a POMDP, where historical states and current observations jointly inform the agent's decisions.
history component maintains a record of previous observations the home icon on page_221"), providing temporal context for sequential decision making. The current observation a screenshot image of the GE-Lab environment, showing the unfamiliar GUT icons with which the agent interact. The GUI agent attends task instructions each step, makes decisions informed by textual representations of historical states and actions, and generates appropriate actions based the current screen state. This approach effectively mitigates task forgetting, reduces action repetition errors, and prevents state interference that would otherwise arise from maintaining excessively long sequence lengths containing complete historical screen states.
actions (e.g., "stepl: click
For action generation, our framework adopts ReAct 47 to formulate appropriate actions within the GE-Lab environment. This approach integrates reasoning and action generation by requiring the agent to produce an explanation of its intended operation along with a precise action command (e.g., "click Home icon on page_0 O; Action: click(635,66 (6",,)))",, which can be directly executed in the GE-Laab environment. In both MT-RL training and inference, we set a maximum number interaction rounds (e.g., 12). According to the agent protocol, with each image encoded 729 tokens, the current maximum context token length, which is the total of the instruction, history, and observation, less than 1 tokens.
3.3 Reinforcement Learning
3.3.1 Single-Turn Reinforcement Learning
The ST-RL paradigm, based on the GRPO (36], addresses the screen navigation challenge in partially observable GUI environments (as shown in Figure 2B). At each time step f, the agent observes o, E O and receives a task specification u U. The agent selects an action a, to interact with the environment, where the underlying state E S is partially observable. The policy is thus conditioned on the current observation, the history of past observations and actions, and the task:
(a lox:t 00.t-1, u),
(1)
where and do:--1 denote the sequences of observations and actions up to time t (the overline indicates pre-constructed trajectories), and a; the action generated by the model the current step via rollout. For ST-RL, oo.t and 0o.-- 1 pre-constructed trajectories, which correspond to the orange history part Figure 2 while a; is the action generated by the model at the current step. The objective is to maximize the single-step return:
J(0)= Bapa a.a [r(o,, a,, w)],
where r(.) composite reward that provides explicit supervision for GUT interaction, enhanc- ing both training efficiency and navigation performance. Specifically, the reward consists of four
5

---

Table 1: Models Performance on In D--Distributin, Out-of lil-of--Ditribution, and Interactive Benchmarks
OOD
Interactive
Model
Edge
Path
Overall
Edge
Path
Overall
Pass @
Pass
Non-tine-tuned Model
GPT-4o-2024-11- 1-22 (29
34.10
5.03
25.85
1.74
2.49
Claude-3. Claude-3.-Sounee 7-Sonnet [
21.77
1.92
21.52
0.43
0.61
Gemini-2.0-fin fo-faah-thinniing
15.05
5.33
8.80
0.36
0.52
Fine-tuned Model
Bwen2.5-VL-7BB.
94.82
99.76
98.89
64.55
41.76
55.45
14.30
20.86
Swvn2.5-VLL-7B--..
97.48
97.08
97.63
68.68
52.25
63.06
17.22
22.34
Qwen2.5- Owen2.5-VL-7B-MT-A.
72.60
57.77
67.33
69.86
52.35
63.25
17.47
25.16
PR
Tatlc
Figure 3:
Case Study. Left: Environmental navigation map. Right: Detailed visualization of the page transition flow from the initial to the final state.
components: (1) Action Type Rewart-evaluates the correctness of the action type; (2) Coordinate whether the predicted click coordinates fall within the target region; (3) Intent Matching Reward assesses if the selected icon matches the intended interaction; (4) Format Reward- valdddd-alidatts whether the generated action conforms to the required output format. The reward function formula is provided in Appendix A.4.
Accuracy
3.3.2 Multi-Turn Reinforcement Learning
The MT-RL paradigm extends the agent-enviromment interaction to multiple decision steps, as illustrated in Figure 23. Unlike ST-RL, where the observation and action sequences oo. and do.r-1 are pre-constructed, MT-RL generates these sequences online through interactive rollouts, i.e., {ok; ox) {ok,ax)
E(ra). where E(Ta) denotes the environment trajectory induced by policy ma. At each time step I, the agent observes or E O and selects an action d, E A according to its policy:
a+ To Oost: 00:t-1, u))
where E U denotes the task specification, and the sequences (0nas 40:t-1) are dynamically gener- ated during the agent's interaction with the environment. The objective MT-RL is to maximize expected final-step reward over the trajectory:
(() Eoo. [r(oe, 0t; u)]
where the reward function is defined in sparse, goal-based A2B reward: the agent receives reward of +1 if it successfully reaches the target screen node, and 0 otherwise. This sparse reward formulation encourages the agent efficiently navigate through the environment to accomplish the specified task. Notably, by providing reward only at the target state, this scheme promotes uninhibited exploration and enables the agent to learn a more comprehensive value function over the state space, facilitating the discovery of diverse and efficient trajectories for task completion.
4 Experiment
4.1
Environment Configuration
All training is conducted in Env-Base, a tree-structured graph with maximum depth of 7. Excluding system-defined edges representing the home and back, the environment's branching structure follows node distribution of [5,3,2,2,1,1000] at each level respectively. In Section 5.2 the Environment

---

Table 2: Performance Comparison of Methods across Tasks of Varying Difficulty
Path@1
Path@
Path@3
Path@
Path@
P
Path@7
SFT 99.71
51.16
19.55
8.52
3.13
2.15
0.31
Pass@1
ST-RL
99.71
59.73
27.57
14.01
4.59
3.38
0.83
MT-RL
98.10
52.93
26.31
13.64
6.63
4.17
2.92
SFT 100.00
74.15
36.04
19.75
6.71
5.04
P
ST-RL 100.00
70.07
37.84
23.77
7.52
7.02
3.39
MT-RL
100.00
66.67
43.24
24.69
13.01
8.11
8.33
Table 3: Performance Comparison of Methods across Out-of-Distribution Environments
SFT
ST-RL
MT-RL
Overall
Path
Edge
Overall
Path
Edge
Overall
Path
Edge
Env-Basc
55.46
41.76
64.55
63.06
52.25
68.68
63.25
52.35
69.86
Env-Image
32.91
7.38
63.90
33.99
7.42
69.68
34.17
7.54
70.64
Env-Name
32.01
6.77
61.24
32.13
5.95
64.77
32.74
6.65
65.90
Env-Position
36.19
12.22
64.55
40.31
17.34
68.81
42.79
20.72
70.51
Env-Noiss 41.18
21.87
57.50
42.33
23.55
57.90
44.56
27.05
60.55
Variants are utilized for OOD testing. Specifically, Env-Image, Env-Name, and Env-Position denote modifications to the icon image, icon name, and icon position within Env-Base, respectively. Env- Noise introduces noise icons into Env-Base.
Edge traversal tasks represent single-step transitions, while Path traversal tasks encompass multi-step navigation sequences. The current environment root node comprises five subtrees: two designated for SFT training, two for RL training, and one for OOD testing. Notably, the SFT training dataset also incorporates Edge data from all subtrees, including the Test subtree. This inclusion is intended to provide the agent with fundamental knowledge of the environment, which can be leveraged for path planning and prevent the model from making entirely random decisions. In addition, the SFT training set is augmented with grounding and understanding data for the icons used in the Env-Base, further enhaneing the agent's s ability interpret and interact with GUI elements. We employe Swwen2.5-VI--7B-Imae
2 as our foundation model for all experiments.
4.2 ST-RL Generalizes on Out-of-Distribution Tasks
Our experimental results, shown Table [ provide compelling evidence regarding the effectiveness of different model training strategies for page navigation tasks. Analysis of performance metrics across ID and OOD benchmarks reveals several significant findings. The non-fine-tuned models demonstrate poor performance across all evaluation metrics, with scores approximating random selection levels. This substantiates our hypothesis that general- purpose models without domain- specific training lack the requisite knowledge representations for effective navigation within simulated environments. In contrast, the Qwen2.5- --L-7B--------F exhibites remarkable improvements, particularly on ID benchmarks. This dramatic performance enhancement indicates the model's strong capacity memorize domain-specific navigation patterns. However, performance declines substantially on OOD benchmarks, suggesting limited generalization capability when confronted with novel scenarios out- side the training distribution. Notably, the ST-RI. approach (Qwen2.5-VI.s --L-..2.--VL----STRRRR demonstrates superior performance characteristics. While maintaining excellent benchmark results, it achieves significantly better generalization on OOD tasks (Overall: 63.06 vs. SFT's 55.45). This 13.7% relative improvement OOD performance validates our reinforcement learning methodology, which leverages trajectory -epeedde reward functions to better capture underlying navigation principles rather than merely memorizing specific action sequences. Furthermore, Interactive Benchmark results reinforce these findings, with ST-RL achieving superior Pass@) and @ scores (17.22 and 22.34, respectively) compared SFT (14.30 and 20.86). This consistent performance advantage across both automated metrics and interactive human evaluation confirms that while SFT effectively memorizes in-domain knowledge, the ST-RL approach yields substantially better generalization to novel scenarios, highlighting the advantages of RL for navigation tasks in dynamic environments.
7

---

(A) Stop Ace. vs Path Length
(B) Task Acc. Path Length
Puwq Success Rate Palh Length (D) Paang Succusa Rale va Path Length
SPT
SPT ST:RI RL
STRL
STRL MTRRL
ST:R0
MTFRI
MI RL
Succeas Robe
Male
Accurary
Eongth
Erngth
Path Length
Path Length
Figure 4: (A-B) Accuracy across different settings and difficulty levels within the static benchmark. (C-D) Success rates across Pass @N and difficulty levels within interactive Benchmark.
It Plmerttvv Stope
1C Rleaad Stansort Dovidt enyee Stow
Longth Ropward Stago
dooe-b--oo Coveh22
24
Air Pauss
Figure 5: (A) Heatmap visualization of performance in different OOD environments. (B-C) The impact of different SFT stages on ST-RL, showing performance (e.g., mean and standard deviation) respectively, (D) Results of action space modification counter reward hacking in MT-RL.
4.3 MT-RL Explores in Interactive Environments
According the results presented in Table T MT-RL presents a distinctive performance profile across evaluation metrics. Unlike SFT and ST-RI. which train on pre-synthesized trajectories, MT-RL employs interactive multi-round training, resulting in lower ID benchmark scores. This indicates MT-RL avoids overfitting to training distributions, instead developing more generalizable navigation strategies. On OOD benchmarks, MT-RL demonstrates comparable performance to ST-RL (Overall: 63.25 vs. 63.06), both significantly outperforming SFT (55.45), The most compelling evidence of MT-RL's effectiveness emerges Interactive Benchmark, where it achieves the highest scores (Pass@1 I: 17.47, Pass@5: 25.1 This superior performance in realistic interaction scenarios can be attributed to MT-RL's training paradigm, which aligns closely with actual deployment conditions. By learning through environmental exploration rather than static demonstrations, MT-RI. develops robust adaptive behaviors that translate effectively to real-world navigation challenges, enabling the agent to strategically explore and recover from suboptimal states. Figure 3 iillusttrats our agent's s capability to navigate complex screens, demonstrating strategic exploration and error recovery when deviating from the optimal path between page_54 and page_1l1 After encountering suboptimal state following the fourth click (834,907) the agent successfully explores alternative paths and backtracks when necessary, ultimately completing the task through multiple recovery steps.
5 Analysis
5.1
Performance Comparison on Tasks of Varying Difficulty
We conduct ablation experiments to evaluate the performance of our methods across varying task complexities. Figure A-B) shows the step and task success rates in static benchmarks, while Table and Figure C-DD presents Pass and P success rates in interactive settings. Across all scenarios, performance drops drastically as path length increases, confirming path length as a key indicator of task difficulty. In static evaluations, ST-RL and MT-RL consistently outperform SFT, especially for longer paths (5-7 steps). This suggests reinforcement learning improves generalization in complex tasks. On interactive benchmark, MT-RL shows even greater gains, beating other methods in Pass @ 1 and P 5 success rates. These results highlight MT-RL's alignment with real-world conditions, where iterative feedback and exploration allow agents to learn from failures and continually refine their strategics, leading to higher success rates.
8

---

5.2 Generalization in Different Out-of-Distribution Scenarios
We evaluate different approaches in various OOD environments to compare their robustness. Table 3 shows the detailed experimental results. Across all OOD settings, reinforcement learning-based methods (ST-RL. and MT-RL) consistently outperform SFT. Notably, the Env-Base setting, MT-RL and ST-RI. achieve success rates of around 63%, compared to 55% for SFT, underscoring better generalization in unseen scenarios. Altering icons or labels (Env-Image and Env-Name) causes significant performance drops (from around 63% to 32-34%: Path success dips to about 7%), highlighting how foundational knowledge about environment elements is crucial for navigation. In contrast, changing icon positions (Env-Position) moderate declines, and MT-RL still maintains higher rates (42-44% overall success, 20-27% on the Path metric), indicating its robustness in dynamic settings. The heatmap in Figurd5/A confirms these trends, showing MT-RL consistently outperforms ST-RL and SFT across all perturbations.
adding noise icons (Env-Noise) yields more
5.3 Influence of SFT on the ST-RL
To investigate the influence of SFT on the subsequent RL training phase, we first subject the base model to SFT for varying numbers of epochs. Subsequently, models from different training stages are utilized to initialize the ST-RL training. Figure 5 presents the reward and training steps for ST-RL training initiated with models that have undergone SFT for 1 to 5 epochs. It is clearly observable that models with different degrees of SFT training exhibit distinct learning trajectories and ultimately achievable reward levels during the subsequent RI. training. A core finding is that RL training initialized with models from early SFT stages achieves higher cumulative rewards. Notably, ST- RL-Base- 1posce, despite exhibiting lower initial rewards during RL training, demonstrates faster learning rate and subsequently stabilizes the highest reward level. Combined with Figure evident that models from early SFT stages possess diversity while retaining greater plasticity. This implies that the model has not yet formed overly entrenched biases regarding the data distribution, thereby enabling it to more readily adapt its policy in response to reward signals and engage in broader exploration during the RL phase. These findings indicate that more SFT is not invariably better, rather, trade-off exists. To achieve optimal outcomes in the subsequent RL training, the selection of a model subjected to a moderate extent SFT as the initialization point is paramount.
6 Generalization to Real-World
To validate the practical effectiveness of our approach, we conduct comprehensive evaluations on real-world GUI tasks to assess the generalization capabilities of agents trained in simulated environment. This section presents two complementary evaluation strategies: zero-shot evaluation on static benchmark to test transferability, and continual training experiments explore scalability with real-world data integration. The detailed specifications of our real-worke GUI benchmark and the continual training datasets are provided in Appendix A.5.
6.1 Zero-Shot Evaluation on Real-Worke Static Benchmark
Table 4: Performance on real-world static benchmark with extended action space.
Action
Total Number Success Number
Success Rate
CLICK
971
622
64.06%
COMPLETE
230
221
96,09%
WAIT
231
99
42.86%
TYPE
129
119
92.25%
SCROLL
8
62.50%
To demonstrate the practical applicability of our approach, we evaluate the generalization capabilities of agents trained in our simulated environment on real-world GUI tasks. To assess this, we construct a static benchmark by sampling 1,569 instances from several open-source real-world GUI datasets. The test samples cover broader action space: CLICK, COMPLETE, WAIT, SCROLL, and TYPE. Without any further fine-tuning, we evaluate our agent, which has not encountered WAIT, SCROLL, or TYPE actions during the training process, directly this benchmark. The results are presented

---

in the Table below, which show that the agent trained in the simulated environment demonstrates notable degree of generalization to real-world GUIs. Despite lacking explicit training on WAIT, SCROLL and TYPE, the agent still achieves non-trivial accuracy on these actions. This suggests that the agent possesses some degree of atomic competence in handling GUT elements beyond its training distribution, highlighting the generalizability of our approach and its practical implications for real-world deployment.
The results shown in Table f reveal several key insights: (1) The agent achieves strong performance on COMPLETE (96.09 6)).0%) and TYPE (92.25%) actions, suggesting effective transfer of fundamental GUI interaction skills. (2) Despite never encountering WAIT, SCROLL, or TYPE actions during training, the agent demonstrates non-trivial accuracy on these unseen actions, indicating emergent generalization capabilities.
(3) The moderate performance on CLICK actions (64.06%) reflects the complexity of real-world GUT element recognition, indicating that agents require additional real-world GUI knowledge to improve their understanding.
Table 5: Performance comparison on real-world grounding and interactive benchmarks after continual
training on real-world data.
Model
SS
SS-v2 43
FP
MoTTIF
Refexp
VWB AG
VWB EG
AW
Aver
8
19
4
33
35
24
24
Base (paper report)
84.70
Base (our report)
84.01
80.34
48.25
71.93
79.46
72.81
90.07
10.34
67.15
Continue-Train
84.91
84.43
59.50
68.30
72.13
67.96
93.70
12.06
67.87
SFT-COntinue--rran
85.06
85.06
59.30
80.47
83.19
68.93
92.01
12.93
70.87
ST-RI--Continuee Train
85.53
85.61
60.45
79.41
77.88
70.87
92.98
14.65
70.92
MT-RI-CCnntine-Trrain
86.08
85.61
59.40
78.18
81.77
76.70
92.98
15.51
72.03
6.2 Evaluation on Real-World Benchmarks with Continual Training
To further investigate the scalability of our approach, conduct continual training experiments by incorporating real-world data. Specifically, we continue training our agents (SFT, ST-RL, MT-RL) with 24k samples drawn from publicly available real-world GUI datasets. We then evaluate the resulting agents on comprehensive woaldworld grounding and interactive benchmarks.
As shown in Table our training framework consistently yields agents with strong generalization capabilities across both simulated and real-world GUT tasks. Notably, MT-RL outperforms ST- RI, which in turn outperforms SFT, maintaining the same performance hierarchy observed in our simulated experiments. These results corroborate our main conclusions: (1) MT-RI. training yields superior generalization compared to ST-RL training, and (2) ST-RL generalizes better than SFT. The quantitative results show that TTT-LLCoontiuu--Tin (72.03%), with particularly strong improvements on ScreenSpot (86.08%) VWB AG (76.70%) and Android World (15.51%). The consistent performance gains across diverse benchmarks validate the robustness of our multi-turn reinforcement learning approach.
achieves the highest average performance
Conclusion
In this work, introduce GE-Lab, novel simulation environment engine designed advance research in GUT agent navigation. GE-Lab enables flexible and modular construction of screens, icons, and navigation graphs, while granting comprehensive access environment information for both training and evaluation. Through systematic experimentation, we establish clear and effective training paradigm: SFT serves critical foundation by enabling agents memorize essential navigation knowledge; ST-RL further enhances generalization to previously unseen scenarios; and MT-RL fosters the development of robust exploration strategies, leading to superior navigation performance and reduced dependence on annotated data. Our extensive benchmarking not only highlights the strengths reinforcement learning approaches GUI navigation tasks but also provides actionable insights for building more capable and generalizable GUI agents. We believe that GE-Lab will serve as valuable resource for the community, facilitating future research and fostering progress towards intelligent and adaptable GUI agents.
10

---

Acknowledgements
This research is supported in part by the National Natural Science Foundation of China (Grant No. U23B2052 and No. 62495092), National Science and Technology Major Project (Grant No. 2021ZD0109003,, and National Key Research and Development Program of China (Grant No. 20232DD0121300).
References
[1] Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, March 2024. htt.tpp://ap...smmaiiiiiiie www.:l/rrrrrrooori......... 268232499
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. / 13923, 2025.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint
2016.
[4] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plum- mer. dataset for interactive vision-language navigation with unknown command feasibility. In European Conference Computer Vision, pages 312-328. Springer, 2022.
[5] Yuxiang Chai, Siyuan Huang. Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-amnotation expo dataset for mobile gui agents. arXiv preprint urXiv:2407.174:0 2024.
[6] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. RI-v: Reinforcing super generaliza- tion ability in vision-lagguag models with less than $3. https:///ithuut www.//om////eee Deoom//Dee-ennt...
R1-V 2025. Accessed: 2025-02-02.
[7] Zhangquan Chen, Xufang Luo, and Dongsheng Li. VisrI: Intention-driven visual perception via reinforced reasoning. arXiv preprint arXiv:2503.07522 2025.
[8] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401. 10935, 2024.
[9] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 17161, 2025.
[10] Paolo Dell' Aversana. The frozen lake problem. an example of optimization policy. 2021.
[11] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems, 36:28091-288 1010991-11110 14, 2023.
[12] Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.144 240..1337 2024.
[13] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world humans do: Universal visual grounding for gui agents. arXiv preprini 2rXiv:24 2024.
[14] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the JEFE/CVF Conference on Computer Vision and Pattern Recognition, pages (4281-14290, 2024.

---

[15] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749) 2025.
[16] Yucong Luo Mingyue Cheng Qi Liu Zirui Liu Shuo Yu Daoyu Wang Jie Ouyang, Ruiran Yan. Training powerful Ilm agents with end-to-end reinforcement learning, 2025. URL https //githu.. com http://uusssssst/eeeee Agent-R1
[17] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
[18] Hongxin Li, Jingran Su, Jingfan CHEN, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. Ul-pro: A hidden recipe for building vision-language models for GUI grounding, 2024.
[19] Hongxin Li. Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. Autogui: Scaling gui grounding with automatic functionality annotations from Ilms. arXiv preprint 20Xiv:250..01977, 2025.
[20] Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu. and Oriana Riva. On the effects of data scale on control agents, Advances in Neural Information Processing Systems, 37:921 00021130-2224 2024.
[21] Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408. I 1124, 2024.
[22] Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An open-source framework for building general ai agents, 2025. URL https://doi. org/10. 522.//eooo.. 15186407
[23] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint Xrvi 00...80022..222 2018.
[24] Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal Ilms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05995 2024.
[25] Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, lat Long long, Jiadai Sun, Jiaqi Wang. et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:24/ 1.00800, 2024.
[26] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0177 2025.
[27] Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint 2:XX::2408.00203,, 2024.
[28] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang. Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprini arXiv:2503.21620, 2025.
[29] OpenAI. Gpt-4o introduction, 2024. URL Accessed: 2024-11-15.
[30] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arxiv:2003.075366 2025.
[31] Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang. Shihao Liang. Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.
12

---

[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. 2023.
In Advances in Neural Information Processing Systems,
[33] Abhinav Kumar Rastogi, Blaise Aguera y Arcas, Chongyang Bai, Jindong JD Chen, Srinivas Ku- mar Sunkara, Xiaoxue Zang, and Ying Xu. Uibert: Learning generic multimodal representations for ui understanding. 2021.
[34] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In Advances in Neural Information Processing Systems, 36:59708-5999 2023.
[35] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14553. 2024.
[36] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint xx2v:2402..33300. 2024.
[37] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint ar arXiv:2504.07615. 2025.
[38] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
[39] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401. 2024.
[40] Kangrui Wang. Pingyue Zhang, Zihan Wang, Qineng Wang, Linjie Li, Zhengyuan Yang, Chi Wan, Yiping Lu, and Manling Li. Vagen: Training vlm agents with multi- -urtttuu reinforcement learning. https://mll- lab. notion. site/vaggen 2025. Notion Blog.
[41] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 10291, 2025.
[42] Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, 2025.
[43] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23311 2024.
[44] Xiaobo Xia and Run Luo. Gui-rl: A generalist rl-style vision- lssion-language action model for gui agents. arXiv preprint arXiv:2504. 10458. 2025.
[45] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei. et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems. 37:52040-5200 2024.
[46] Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024 2024.
13

---

[47] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. In International Conference on Learning Representations (ICIR), 2023.
[48] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.1377. 2023.
[49] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Ohain-of-action-thouu for gui agents. arXiv preprint arXiv:2403.02713, 2024.
[50] Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: faithful and scalable testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1-13, 2024.
[51] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. RI-zero's ss aha moment" in visual reasoning on 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025.
[52] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. /3854, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the paper's 's main contributions and scope. The experimental results presented in the paper provide evidence supporting these claims.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed in an appendix section A.2 titled
"Limitations".
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All necessary details for reproducing the main experimental results, including specific configurations and procedures, are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code and data are included in the supplemental material and will be released the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All experimental settings and details, including data splits, hyperparameters, and optimizer choices, are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results reported in the paper, including those in Table 1, Table 2, and Figure 4 (such Pass @ 5), obtained from multiple independent runs, providing appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details of the computational resources used for all experiments, including hardware specifications and runtime information, are provided in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. wwtpsuurrssscccpppppoooooo
/ http://wrwrppppcpppppp.//aaaaaaa
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and confirm that the research fully conforms to its principles and guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of the paper is provided in Appendix A.3.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The foundational model Qwen2.5-VL used in this work is properly credited and appropriately cited in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
[NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpps////eurpss.... /ctp////eeeeeeecc///22///////// for what should or should be described.
21

---

A Appendix
A.1 Experimental Details
This section details the experimental environment, dataset construction methodology, key training parameters, and evaluation setup employed in our research, aiming to ensure the reproducibility and transparency of our findings.
Compute Resources. All experiments are performed on nigh-performance computing cluster equipped with 16 NVIDIA A800 GPUs. The specific training duration varies depending on the model stage and task type. A typical training time for single SFT model is approximately 3 to 4 GPU hours. The training of ST-RL models generally requires about 48 GPU hours, while Multi-Turn Reinforcement Learning MT-RL models typically train for approximately 36 GPU hours.
Dataset & Benchmark. We construct and utilize multiple datasets to support various stages of training and evaluation tasks. Env-Base serves as the core environment for our experiments as shown in Figure O This is graph with a maximum depth of 7. Excluding yysem--dffied edges representing the home and back actions, the branching structure of the environment follows a node distribution of [5, 3, 2, 2, 1, 1, O] at each respective level. Its graph structure is partitioned into five independent subtrees, which are allocated according to a 2:2:1 ratio for the Supervised Fine-Tunie Fine-Tuning training set, Reinforcement Learning training set, and test set, respectively. Based on this partitioning, the data are further categorized into path data and edge data. Specifically, each subtree contains 12,439 path data instances and 274 edge data instances. These data are primarily utilized for training the model in graph structure understanding and fundamental navigation capabilities.
In the visual representation, the number of icons corresponds to the clickable areas that trigger page transitions, with the task completed by selecting the "complete" icon. At the root node, there are icons and clickable areas, allowing access to pages. For first-level child nodes, there are 4 icons (3 regular + "Back") and clickable areas, enabling navigation to pages. For other child nodes, which include "Home" and "Back" icons, the number of icons is NodeNum[i] 2, resulting in NodeNum[i] + clickable areas, allowing access to NodeNum(i] + other pages. On average, each screen contains more than 5 clickable icons.
We additionally generate datasets for specific vision and language understanding tasks: (1) Icon Captioning: This dataset comprises 2,320 instances and is used to train the model to understand icons and generate corresponding textual descriptions. instances, aimed at training the model to locate relevant icons within an interface based on textual instructions. To evaluate the model's practical performance in dynamic interactive scenarios, we construct dedicated interactive benchmark task set. This task set is formed by randomly selecting any two nodes within a reserved test environment to serve as the start and end points of a task, respectively. Ultimately, this process generates a total of 2,162 independent interactive test tasks. The test environment is strictly isolated from the training/validatto of the evaluation. The specific number of tasks, particularly the distribution of tasks according to different path lengths or types, is presented in T
(2) Icon Grounding: This dataset contains 2,320
environments to ensure the objectivity
Table 6: The Number of Interactive Benchmark Tasks
Path 137
Path @2
Path@3 222
Path@4
Path
Path@6 456
Path @7
Overall 2162
Number
147
324
492
384
Details of the Out-of-domain Test Environment. To systematically evaluate generalization capa- bilities, we construct comprehensive suite of OOD environments that challenge different aspects of visual grounding and navigation. As illustrated in Figure 7 our evaluation framework encompasses five distinct environment configurations: Env-Base (in-domain baseline). Env-Image, Env-Name, Env-Position, and Env-Noise. Following the hierarchical data partitioning strategy outlined in our dataset construction, one subtree reserved exclusively for OOD evaluation, ensuring zero overlap with training trajectories. The remaining subtrees are allocated for SFT and RL phases, constituting the in-domain training distribution. Beyond this natural distribution shift, we systematically construct four additional OOD variants through controlled perturbations of visual, semantic, and spatial modal-
22

---

Figure 6: The navigation map of Env-Base.
ities. Each environment preserves the underlying task structure while introducing specific domain gaps that probe distinct aspects of model robustness and transferability.
Env-Image introduces visual domain shift by systematically replacing icon appearances while pre- serving semantic labels and spatial configurations. As depicted in Figure each icon undergoes visual transformation (e.g., the human icon becomes a dog icon) through sampling from disjoint visual vocabulary, ensuring no visual overlap with the original environment. This perturbation simulates real-world scenarios such UI redesigns or cross-platts pross-platform adaptations where functional semantics remain invariant despite visual changes. Env-Name targets semantic grounding by altering textual labels while maintaining visual appearance and spatial layout. The transformation involves semanti- cally meaningful substitutions (c.g., "Human" - "Lady") that preserve categorical coherence while introducing lexical variations. This configuration specifically challenges the model's capacity for semantic reasoning and tests the robustness of language-viss alignment mechanisms. Env-Position evaluates spatial reasoning capabilities through global layout perturbations. Icon positions undergo unconstrained randomization subject to non-overlap constraints and screen boundary conditions, as illustrated by the repositioned elements in Figure 7 This environment assesses the model's ability to maintain spatial understanding beyond local neighborhood relationships. Eny-Noise introduces visual complexity through the injection of task-irrelevant distractor elements (Noise1, Noise2). These noise icons are strategically positioned to increase visual clutter while preserving the core naviga- tional structure, thereby testing the model's attention mechanisms and ability to filter relevant visual information.
Experimental Settings. Table 7 and Table 8 present the important hyperparameters used for SFT and RL training. Unless explicitly listed Table RL hyperparameters match the SFT settings in Table 7
Training Time and Sampling Efficiency We provide a comprehensive analysis of the training time and sampling efficiency across our three proposed methods. The SFT baseline requires 3-4 GPU hours for training on 60,864 samples, achieving the highest sample efficiency with minimal computational overhead. In contrast, our reinforcement learning approaches demonstrate different computational profiles: ST-RL requires approximately 48 GPU hours for training on 12,439 samples, while MT-RL completes training in 36 GPU hours using 2,162 interactive tasks. The computational trade-offs between methods reflect their distinct learning paradigms. While SFT exhibits superior sample efficiency for in-distribution scenarios, both ST-RL and MT-RL demonstrate enhanced
23

---

Human
Lady
Human
Histogram
Human
Noiset
Histogram
Histogram
Chart
Histogram
Hallowee
Hallowee
Ghost
Hallowee
Hallowee
Nolse2
Env-Base
Env-Image
Env-Name
Env-Position
Env-Noise
Figure 7: Out-of-Distributioo Environment Configurations: Env-Base serves as the in-domain base- line, while Env-Image, Env-Name, Env-Position, and Env-Noise introduce controlled perturbations targeting visual appearance, semantic labels, spatial layout, and visual complexity, respectively. Each environment maintains the same underlying task structure while introducing specific domain shifts to probe different aspects of model robustness.
Table 7: Hyperparameters for SFT.
Table 8: Hyperparameters for RL. config
config
value
value
learning rate
le-5
learning rate per device train batch size num train epochs num generations temperature top p top k
le-6
learning rate schedule
cosine decay
per device train batch size gradient accumulation steps warmup ratio
2
5
2
0.05
1.2
num train epochs max pixels
1
1.0
200704
8
exploration capabilities through increased environment interactions. Specifically, ST-RL achieves 99,512 total interactions compared to MT-RL's 89,939 interactions, with both methods using 8 generations per training round. This increased interaction volume enables more comprehensive state space exploration, leading to improved robustness in out-t--distribution and interactive scenarios.
Regarding the simulation environment, training framework employs efficient simulation envi- ronment that enables rapid policy iteration without real-time rendering overhead. The environment generates feedback by evaluating the predicted click position against pre-cached meta information, determining state transitions based on spatial overlap with interactive elements. For instance, when the agent predicts a click within the Home icon's bounding box from page 6, the environment transitions to page_0 O according to the cached navigation graph. This design provides significant computational advantages: while model inference requires approximately 13 seconds per sample, environment feedback generation incurs only millisecond-leves latency due pre-rendered screen caching. The simulation framework thus enables efficient policy learning without the computational bottlenecks typically associated with real-time GUI rendering, making our approach scalable for large-scale reinforcement learning training.
Table 9: Training Time and Sampling Efficiency Comparison for the Three Methods
Method
Dataset Size
Training Time
Interaction Count
Feedback
SFT
60864 samples
3-4 GPU hours
60864
Manually labeled
ST-RL
12439 samples
48 GPU hours
99512
Manually labeled
MT-RL
2162 tasks
36 GPU hours
~99993
Environment Feedback

---

A.2 Limitations
While the current version of GE-Lab provides a controlled and accessible environment for studying agent navigation, its interaction logic does not yet fully capture the diversity and complexity of real-world GUI interactions. In practical applications, user interfaces often feature broader variety of icons, substantial textual information, and various forms of visual noise. These elements, such as additional icons, text, or noise, can be readily incorporated into the current environment, and we plan to continuously enrich the simulation to better approximate real-world scenarios and enhance the generalizability of our findings.
Regarding the action space, we intentionally restrict the agent's 's available actions in the current version facilitate focused analysis of navigation strategies. This simplification allows us to systematically evaluate the impact of different training methods on navigation performance. Nevertheless, real- woold GUI interactions require a richer set of actions, including gestures such as swiping and gang-pressing. Expanding the agent's 's action space to support more complex interactions is an important direction for future work, enabling more comprehensive assessment of agent capabilities in practical settings.
A.3 Broader Impact
The advancement of GUI agents hinges critically on their ability to perform robust and generalizable screen navigation, which remains key bottleneck for real-world deployment. By systematically dissecting the navigation challenge, our work provides a clear roadmap for developing foundational agent capabilities. Specifically, we demonstrate that a staged training soarddig---ompiiing super- vised fine-tuning, single-turn reinforcement learning, and multi-turn reinforcement learning-enaen agents to first acquire essential knowledge, then generalize novel scenarios, and finally develop effective exploration strategies through interactive trial and error.
Our introduction of flexible simulation environment, GE-Lab, allows for controlled and compre- hensive analysis of agent behaviors, circumventing the complexities and proprietary constraints of real-world GUI platforms. This approach not only accelerates research progress by enabling reproducible benchmarking, but also lays the groundwork for agents to safely and efficiently acquire the core competencies required for interacting with complex environments. By abstracting and simplifying the environment, we are able to isolate and address fundamental challenges, ultimately guiding the development of GUT agents toward more general and autonomous intelligence (AGI).
From a societal perspective, the development of more capable and generalizable GUI agents has the potential to significantly enhance productivity and accessibility, automating routine digital tasks and enabling broader access to technology for users with diverse needs. However, as agents become more autonomous and capable of interacting with real-world systems, it is crucial to consider the ethical implications, such as ensuring user privacy, preventing unintended actions, and maintaining Our work, by focusing on simulation- based training and
transparency in agent spcisiinnmaking evaluation, provides a safer pathway for developing and testing agent capabilities before deployment in real-world applications, thereby helping to mitigate potential risks. We believe that these insights and tools will benefit the broader research community by informing the design of more capable, reliable, and generalizable agents, and by providing a foundation for future work on safe, ethical, and effective agent-enviromment interaction in both simulated and real-world settings.
A.4 Reward Function Design
To effectively guide the agent's learning process, we design four complementary reward functions that collectively form the complete reward signal R:
R(st, (t 81+1) = Ttype (ot, az) + Teoora(as, 8t) + Fintent (az, ex) + T'forrmat (ae, ex)
(5)
where of represents the reference action, and e, denotes the generated explanation. We employ equal weights for all four reward components based on empirical analysis of their behavior patterns. Specifically, the Type and Format rewards are consistently optimized early in training, achieving high means with low variance (0.98+0.03 and 0.95+0.05 respectively), effectively serving as syntactic constraints rather than meaningful optimization objectives where heavier weighting would not provide additional gradient information. Meanwhile, the Coordinate and Intent rewards exhibit strong correlation, as evidenced by our analysis of 1000 sampled interactions showing P(R_coord=1) R intent=1) 1) s 96.3% and P(R intent=1 RR 1 se 94.2%. This strong dependency between
25

---

these reward components reduces the necessity for differentiated weighting schemes, making equal weights a simple yet effective approach that avoids additional hyperparameter tuning.
The action type reward Ptype evaluates whether the action type generated by the agent matches the optimal action type. Our action space A includes two fundamental types: click and complete. Formally, it is defined as:
Action Type Reward
1,
if type(aa) type(a;) 0. if type((a) * type(a?)
Ttype (at, ai)
(6)
where type(a) extracts the type of action a. This reward encourages the agent to first understand the action type required in the task context, establishing a foundation for subsequent precise operations.
Coordinate Accuracy Reward The coordinate accuracy reward Feoord measures the spatial pre- cision of click operations. For click actions, determine whether the click coordinates (x; y) fall within the target area based on the bounding box of interactive elements:
{1, 1 if type(az) click and (x,, P(s:)
Fccord(at, s1)
1, if type(aa) complete
0, otherwise
where P(st) represents the set of points within the correct interactive region in state 8t. For complete actions, the default reward is they do not involve spatial coordinates. This reward mechanism encourages the agent to precisely locate UI elements, enhancing operational reliability.
The intent matching reward Fintent evaluates the exact name matching between the explanation e, generated by the agent and the actual UI element interacted with through action dz:
Intent Matching Reward
(xxactMatch(a, I[a,, s;)), Tintent (1t, ex) (Contains(ee, "target page"), type(a;) = complete
if type(a.) = click
where ExactMatch(es, I(ar, st)) returns if the explanation es contains the exact name of the interacted UI element and 0 otherwise. I(ae; 8t) represents the name of the specific icon or interactive element that is being clicked when action is executed in state 8t. Contains(e:, "target page") returns 1 if the explanation e explicitly indicates task completion, and 0 otherwise. For click operations, we verify exact name matching between the element mentioned in the explanation and the actual UI element clicked; for complete operations, we check whether the explanation properly acknowledges the successful completion of the task.
Format Reward The format matching reward format evaluates whether both the action a, and the explanation e generated by the agent conform to the format of their respective references, ot and et. This reward encourages the agent to produce actions and explanations that are not only correct in content but also consistent in structure and style with the optimal examples. It is formally defined as:
1, if FormatMateh(aa, at) and FormatMatch(ee. et)
format (at, (at,
Ct, oi eg)
otherwise
where FormatMatch(a,, a²) and FormatMatch((,,') e*) indicate whether the format of the generated action and explanation match those of the references, respectively (such structure, key fields, style). This reward term encourages the agent to generate actions and explanations that are not only semantically correct but also well-formatted and standardized, which benefits subsequent readability, usability, and consistency.
This multi-level reward R(st, (t,,,,,,,, 11+1) enables the agent to simultaneously learn correct action type selection, precise spatial localization, reasonable semantic understanding, and standardized explanation expression, thereby forming comprehensive navigation capabilities. Experimental results show that after adding the format matching reward, the explanations generated by the agent become more standardized, further improving the navigation success rate and generalization ability in the GE-Lab environment.
26

---

A.5
Real-Wool World Benchmark and Training Data
Evaluation Prompt for Real- Geal-World GUI Benchmark
You are Multifaceted Mobile Interface Assistant. Your responsibilities include:
Navigating a mobile phone interface to reach target page based user instructions, history, the current screen state.
Understanding icons identifyng their name or function based on their location on the screen.
Grounding icons by locating the coordinates an icon based on its name or description.
You will receive inpat typically includes:
Request: Specifies goal (navigation, understanding. grounding). might a complex
navigation or direct gustion/cemmand for icon tacks.
Task History (Optional, primarily for Navigation): Records previous steps.
Current Screen State: Represents the current sereen, an image (indicated by <inage>).
Based on user request and current screen state (and history applicable), you must first determine the type of task requested and then provide the uppropriate output.
Task Ty and Output Formats
General GUI Task
Goal: Reach target page step-by-step.
Typical Input: Multi-turn instruction, history, and state. scroen description and screenshot.
Possible Actions:
Tap specific element. Provide coordinates (x, y) relative to (0,0) top-left and (1000,100 1000) bottom-right system.
click:
type: Type text into an input field. Provide the string content be typed. Example: ("Teeee BBQ")
scroll: Scroll the screen. Provide the scroll distance. Example: SCROLL (5)
wait: Pause the exccution. Provide the duration wait in scconds. Example: WAIT(3)
complate: Task finished,
the target.
Output Format:
Explain: [Your brief explanation, e.g., 'click XXX icon on yyy page.", ?, "this is the target page. NtAction: (click(start_bae)) bareebbbbro//oottarrtrt (s.x))/boxx_ T"Pe("Text to type") SCROLL(6) or (3IT(3) or complate]
General Instructions
Carefully analyze the user request to determine
(Navigation, Groanding, Understanding).
Analyze
state (description image) thoroughly.
For actions involving covrdimates (cli the (0,0) to (1000, 1000) system.
Strictly adhere the specified output format for the determined task type. Use tab character ('v)) separator where indicated.
Evaluation Prompt for Real- World Grounding Benchmark
I want (gool_infio.. Please locate the target element should interact with. (with point)
To comprehensively evaluate the generalization capabilities of our approach, curate diverse collection of real-world GUI samples from multiple publicly available datasets. Our real-world evaluation benchmark mentioned in section 6.1 comprises 1,569 carefully sampled instances drawn
from four established GUI datasets: AITW (34. AITZ (49. AMEX (5). and Mind2Web [ These datasets collectively span diverse application domains, interface designs, and interaction paradigms, providing comprehensive testbed for cross-domain generalization assessment. The benchmark encompasses five distinct action types: CLICK, COMPLETE, WAIT, SCROLL, and TYPE, representing the fundamental operations in GUI navigation. As depicted in Figure g the collected samples exhibit substantial visual and functional diversity, including e-commerce interfaces featuring product listings and shopping carts, system settings with hierarchical menu structures, media applications displaying video content and playback controls, productivity tools with complex layouts, and social platforms incorporating feed-based content and interactive elements. Each sample annotated with specific instructions and target actions, ranging from simple element selection (e.g., "Click the YouTube icon") to complex multi-step operations (e.g., "Cancel all purchases over $200"), with instructions formulated in natural language that require agents to perform visual grounding, semantic understanding, and precise action execution. The GUI screenshots are sourced from the
27

---

original datasets, while the task descriptions and action annotations are generated by 4PT240-2024- 11-20 (29 based on the meta-information from the original datasets, followed by quality assurance conducted by human annotators. The prompt for this task test is shown at A.5 (Evaluation Prompt for Real-World GUT Benchmark). Notably, our simulated training environment only expose agents to CLICK and COMPLETE actions, making WAIT, SCROLL, and TYPE completely unseen during the initial training phase, which enables rigorous evaluation of zero-shot generalization capabilities.
To comprehensively assess GUI understanding and interaction capabilities, we evaluate our approach on eight established real-world benchmarks as discussed in section 6.2. Benchmarks are abbreviated as follows: SS (ScreenSpot) [8]. SS-v2 (ScreenSpot-v2) [4]. FP (FuncPred) [I9], MoTIF [] Refexp [33, VWB AG (VisualWebBench Agent Grounding) (24). VWB EG (VisualWebBench Element Grounding) (24] and AW (AndroidWorld)) [35]. Seven of these benchmarks (SS, SS-v2, FP, MoTIF, Refexp, VWB AG, and VWB FG) are static evaluation datasets that assess visual grounding capabiliiies by measuring the accuracy of locating target UI elements within screenshots given textual descriptions. In contrast, Worlo (AW) provides an interactive evaluation environment where agents perform complete tasks within Android virtual machines, with success measured by actual task completion rather than intermediate grounding accuracy. This diverse benchmark suite enables thorough evaluation across both fundamental grounding abilities and end-to-end interactive performance. For the continual training experiments described in Section 6.2, we utilize a subset of 24,000 samples from the same source datasets domain adaptation material, enabling our agents to bridge the gap between simulated and roaldworld GUI environments while maintaining the Iearned multi-turn capabilities. For a fair comparison, SFT-Continue- Train, Qwen2.5-VV S.ven..5-VI--B-. VL--B-STTRIL--oniieee ---.255--------niie Train, Qwen2.5-VL-77 WwI.:---VV--BMM--RR---O
Train, Train are trained on the same 24k dataset using 16 GPUs and identical hyperparameters, including global batch size of 256, a learning rate of le-5, a max length of 5120, and 2 epochs. The prompt for the grounding benchmark is AA.5 (Evaluation Prompt for Real-World Grounding Benchmark). and the prompt for the interactive benchmark is A.5 (Evaluation Prompt for Real-World GUI Benchmark).
Jouan
Hila
abioy
WL.
VLC. for Androil
Shopplnn cart
Suggened for you
9
Bee Moadh DeaG
SELLA
You right Mie
Instruttoo::
Insouction:
Instruction:
Inttrction:
Instruction:
Search Texas BBQ
Canoel all purchases over $200
Buy tarrs undar on homepage.
VLC cottware on phone.
Check out
Restaurants
Finels video YouTube.
Explsin: Tyring Tasan BBQ
Explain: Screll down view
Explain: Walting for
Explain:
Explain: Click
in the search box. Aution: TYPE: Toras BBQ)
products.
page load.
complets
app
Action: SCROLL/S)
Action: WAIT(3
Action COMPLETED
Action: BLCKK31,100
Figure& 8: Real-world Data with Expanded Action Space.
A.6 Training Stability
To evaluate the robustness and reliability of RL methods, we conduct a comprehensive analysis of training stability across multiple random seeds. As demonstrated in Table 10, ST-RL method exhibits remarkably stable training dynamics, with reward values demonstrating a consistent upward trajectory from an initial value of 1.30220.112 to final performance of 3.760=0. 175. Note that
28

---

since ST-RL and MT-RL have different specific iteration numbers, we use It-1 to It-4 to represent different training stages, with the specific correspondence being ST-RL (It 1005000 - 2000) and MT-RL (It_ 200 - It_ it_800). Notably, the standard deviations remain consistently below 0.2 throughout the training process, indicating minimal variance across different experimental runs. Similarly, our MT-RL approach shows substantial and stable improvement, with rewards progressively increasing from 0.394+0.081 to 0.745+0.079 The consistently low variance observed across all experimental configurations provides strong empirical evidence for the stability and reproducibility of our training methodology across different random initializations. Regarding convergence stability, our experi- mental results reveal that both ST-RL and MT-RL demonstrate smooth and monotonic convergence patterns, which is crucial for practical deployment scenarios. Specifically, the ST-RL approach shows steady improvement in evaluation scores, progressing from an initial performance of 55.13 to final score of 63.67, representing a substantial improvement of 8.54 points. Concurrently, the MT-RL. method achieves consistent performance gains, with evaluation scores increasing from 63.46 to 64, 40. Importantly, we observe that the standard deviations progressively decrease throughout the training process, indicating not only improved performance but also enhanced consistency and reliability. This convergence behavior suggests that our proposed methods successfully avoid common pitfalls in reinforcement learning such as catastrophic forgetting or unstable policy updates, thereby ensuring robust and predictable learning dynamics.
Table 10: Reward and Evaluation Performance Across Iterations for ST-RL and MT-RL (Mean + Std
Over Seeds)
Model
Type
It-1
It-2
It-3
It-4 3.760 + 0.175
ST-RL
Train
3.302 + 0.192
3.693 + 0.188
3.739 + 0.152
ST-RL
Eval
55.133 + 1.213
55.358 + 1.254
60.077 + 1.158
63.663 + 1.112
MT-RL
Train
0.394 + 0.081
0.402 = 0.083
0.614 + 0.102
0.745 0.079
MT-RL
Eval
63.462 + 1.089
63.494 + 1.193
64.185 + 1.062
64.402 + 0.988
Note: It-1 to It-4 correspond to: ST-RL (It_500 - 20000 MT-RL (It_200 800).
A.7 Reward Hacking in MT-RL
In the exploration of MT-RL, a specific phenomenon of reward hacking is observed, particularly in scenarios where the model possesses a diverse action space and supervision signals are relatively sparse. Figure illustrates the mean output length and reward for two MT-RL models. As indicated by the dashed line, the "MT-RL-w-Hack model exhibits significant reward hacking in the early stages of training, characterized by a substantial reduction in length alongside a rapid increase in reward. The fundamental cause of this hacking resides in the training mechanism and action space design inherent to MT-RL. Firstly, the primary supervision signals and reward calculations are concentrated on the output of the final turn in multi-turn interactions. Secondly, the model possesses at least two types of actions: "click" and "complete". Typically, task completion must conclude with a "complete" action, design that leads to shortcut effect. The model aims maximize short-term obtainable rewards and tends to select "complete" actions prematurely or excessively. Consequently, it does not genuinely learn how to solve the problem through effective "click" sequences but instead discovers "loophole" in the reward function. To address this issue, our strategy involves reducing action space diversity mitigate reward hacking. Specifically, "click" actions are predominantly retained, aiming to guide the model to focus on enhancing the accuracy and effectiveness of these interactions rather than seeking premature task termination. As shown by the solid line in Figure 5p. the "MT-RL-wo- Hack" model demonstrates a stable mean length and a progressive, genuine increase in reward. This indicates that unified action space enables the model to concentrate on core task actions in each decision-making turn, thereby learning deeper task logic through cumulative multi-turn interactions. These findings offer valuable insights for the future design of more complex MT-RL systems.
A.8 Case Study
To qualitatively assess the behavioral differences engendered by Supervised Fine-tuning (SFT), Single-Turn Reinforcement Learning (ST-RL.), and Multi-Turn Reinforcement Learning (MT-RL.), present 3 illustrative case studies. These studies are conducted within our GE-Lab environment,
29

---

where each task has maximum execution limit of 10 steps. The agent's 's interactions for each case under SFT, ST-RL, and MT-RL.
(a)
3
(b)
(c)
Figure 9: Case Study 1: Demonstrating Basic Navigation and Error Recovery. Task: From page_54
page 171. page_171.
Case 1 In the first case, the task involves navigating to specific target page. The SFT agent (Figure aa) exhibites a characteristic failure mode. It attempts an erroncous direct navigation from the root node to environment page not contain the target element. Subsequently, it becomes trapped, indicated by futile clicks on a blank area for the final three steps until the episode terminates. This behavior underscores SFT's reliance memorized trajectories and its brittleness when faced with deviations from seen data, a limitation our training paradigm aims to overcome.
The ST-RL agent (Figure b)) successfully completes the task by identifying and executing the shortest navigation path. This demonstrates the foundational generalization capability imparted by ST-RL, aligning with our contribution that ST-RL enhances generalization to unseen scenarios.
The MT-RL agent (Figure ()) also successfully reaches the target, but notably showcases its enhanced exploratory and recovery capabilities. After an incorrect transition from page_22 _2 to page 53, the agent adeptly recognizes its off-path state, executes "back" action return page_ 22, and then selects the correct subsequent navigation step. This ability to self-correct and recover from errors highlights MT-RL's promotion exploration and its capacity to build more robust agents, as states in our contributions.
Case 2 The second case presents similar navigation challenge. The SFT agent (Figure TO(a) again fails. It initially navigates to a familiar root page before attempting an incorrect direct jump unrelated environment page. As in Case 1, it concludes by repeatedly clicking blank area, exhausting the step limit. This reinforces the observation of SFT's limited generalization.
The ST-RL agent (Figure 10 (O) achieves the goal, albeit imperfectly. While identifies the shortest path, makes an initial incorrect click on an invalid area within page_8. 8. Due to GE-Lab's 's interactive design (where invalid actions result in state change), the agent remains page_8 8 and successfully selects the correct icon on its subsequent attempt, ultimately reaching the target. This shows ST-RL's capacity for in-state error correction.
The MT-RL agent (Figure (0(c)) demonstrates superior performance by navigating to the target via the shortest path with flawless precision in single attempt per step. This exemplifies the refined decision maaiiinnnmmkinggg and heightened navigation performance fostered by the MT-RI. stage, further corroborating our claims about its benefits.
30

---

(a)
Figure 10: Case Study 2: Navigational Precision and Efficiency. Task: From page_ 121 to page_ 180.
(a)
(c)
8
80
Figure 11: Case Study 3: Complex Navigation and Novel Path Discovery, Task: From page_51 51 to page_180.
Case 3 The third case involves more complex navigation sequence. The SFT agent (Figure II a)) initiates the task correctly for the first two steps. However, a single misstep on page_6 (clicking an incorrect icon) leads it down irretrievable and erroneous path, eventually terminating blank page after multiple invalid clicks. This highlights SFT's limitation to recover from navigational errors in longer sequences.
The ST-RL agent (Figure b)) also failes in this more challenging scenario. Despite successfully executing the first five navigation steps, it incorrectly branches off at page_8 8 and is unable to find the target page before reaching the maximum step limit. This suggests that while ST-RL improves generalization, its exploratory reach in complex, unseen situations may still be limited.
31

---

Remarkably, the MT-RL agent (Figure[[] (T) not only successfully reaches the target page but does so via novel 8-step path that is not present in any of the training data. A critical segment this path involves direct and accurate transition from page_51 51 to page_180 180. It demonstrates significant ability to generalize and discover entirely new and efficient solutions. Notably, these discovered solutions are sometimes longer than those typically encountered in training, which strongly supports our contribution that MT-RL promotes exploration. Furthermore, this capability indicates that MT-RL reduces reliance on exhaustive annotated data, the agent successfully discovers viable paths unseen during the SFT phase.
Collectively, these case studies, facilitated by the flexible GE-Lab environment, provide qualitative evidence for our proposed training paradigm. SFT agents tend to memorize and fail catastrophically upon deviation. ST-RL enhances generalization for moderately unseen scenarios. MT-RL significantly boosts performance, promotes robust exploration and error recovery, and enables agents discover novel solutions, thereby offering a clear path towards more generalizable GUI agents.
A.9 Performance Comparison on Tasks of Varying Difficulty
Table 11: Performance Comparison of Methods across Varying Difficulty in Static Benchmarks
Path@)
Path@2
Path@3
Path@4
Path@5
Path@6
Path@7
Overall
SFT 100.00
78.005
66.216
58.457
55.928
47.556
51.302
55.454
Step
ST-RL
100.00
83.673
75.338
62.346
59.621
60.338
59.766
63.06
MT-RL
100.00
79.819
72.860
64.691
59.383
60.62
60.514
63.252
SFT 100.00
43,537
15.315
8.333
1.626
1.316
0.0
12.766
Task
ST-RL MT-RL
100.00
57.823
24.775
13.889
3.049
2.851
0.0
16.189
100.00
52.381
22.072
14.506
4.268
3.07
0.26
16.003
To rigorously evaluate the performance of our proposed methods across tasks of varying complexity, we conduct comprehensive ablation experiments. Table [T shows the step and task success rates in static benchmark evaluations. We observe consistent trend across all experimental settings: the path length increases, the task complexity rises significantly, resulting in performance degradation across all methods. This phenomenon is uniformly present in all testing scenarios, confirming path length as a critical indicator of navigation task difficulty. Reinforcement learning methods (ST-RL. and MT-RL) demonstrate markedly superior performance compared SFT. Notably, longer path lengths (5-7 steps), ST-RL and MT-RL maintain relatively stable success rates, whereas SFT exhibits substantial decline. This indicates that reinforcement learning paradigms enhance model generalization capabilities, enabling more robust performance in complex scenarios.
32

---

A.10 System Prompt
System Prompt for Multifaceted Mobile Interface Assistant
You are Multifaceted Mobile Interface Assistant. Your responsibilities include:
1. Navigating a mobile phone interface to reach target page based on user instructions, task history, and the current screen state.
identifying their uame function based their location Grounding icons locating the coordinates
Understanding
based
description.
You will receive input that typically includes:
User Request: Specifies the goal (navigation, understanding, grounding). This might be complex instruction for navigation direct question/commmand for tasks.
Task History (Optional, primarily Navigation): Records previous steps.
Current Screen State: Represents
image (indicated by <image>).
Based on the user request and the current screen state (and history if applicable), you must first determine the type of task requested and then provide the appropriate output.
Task Types and Oulput Formals-
1. Task: Navigation
Goal: Reach a target puge step by step.
Typical Input: Multi-tum instruction, history, and state. screen description screenshot.
Possible Actions:
click: specific element. Provide courdinates relative left and (1000, battom right system.
complete:
Task finished, current screen the target.
Output Format:
Explain: [Your briaf explanation, e.g., 'click xxx icon yyy page.", "this the target page. VtAction::
only berettbbxo////otttaa
[click(start_B) (click start /> (x, <bbxx_end) or complete] # Include point
Task: Icon Grounding (Locating Icon)
Goal: Identify the coordinates of requested
Typical Input: User request like "Click on [ienn mameldescription) in the image"", screen image (<image>).
Action: Implicitly click (meaning "identify location") "
Output Format: The explanation is oflen implicit in the grounding request itself.
Action: click(start. bessstrt_bbore boz-c booeee/bo__eeeee start (> box_bboxee
Understanding (Identifying ing an Icon)
Goal: Provide
given
Typical Input: User request like "What the icon point (x, y) the image?", image (Cinage>).
Action: Provide textual information.
Output Format: Just the direct unswer text.
[Icon Name or Description]
General Instructions
Carefully analyze the user roquest to determine task (Navigation, Grounding, Understanding).
Analyze
state (description image) thoroughly.
For actions involving conedinates (c11 ck), the (1000, 1000) system.
Strictly adhere the specified output format for the determined task Use tab character ('t) separator where indicated.
33


Reviews:
[{"review_id": "v4mHx5289o", "reviewer": "Reviewer_DUjd", "summary": "This paper introduces GUI Exploration Lab (GE-Lab), a simulation environment for training and evaluating GUI navigation agents with reinforcement learning. The environment enables flexible compositionality of screens, icons, and navigation graphs, allowing researchers to generate large-scale, customizable benchmarks and detailed environment exposure for agents. Through extensive experiments, the authors benchmark three training regimes (SFT, single-turn RL, and multi-turn RL) using the new environment, showing that SFT helps agents memorize, ST-RL improves generalization, and MT-RL promotes active exploration and interactive recovery. The manuscript includes quantitative and qualitative analysis of these regimes’ performances under in-distribution, out-of-distribution, and interactive tasks, and investigates reward design and pretraining regimen effects.", "questions": "Overall, I prefer to be borderline; but, since a strictly neutral rating is not available, I lean slightly positive and therefore give a borderline accept. I will make my final decision after reading the authors’ rebuttal.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* GE-Lab is a meaningful addition to the field, providing a flexible simulation platform capable of modeling complex GUI structures, navigation graphs, and icon layouts, with controllable OOD scenarios for systematic evaluation.\n* The experimental setup and analysis are thorough, covering in-distribution, OOD, and interactive settings, with clear ablation studies and insightful discussions.\n* The paper is generally well-written and easy to follow, with clear figures and explanations.\n\nWeaknesses\n* Limited environment realism: Despite its flexibility, the simulated environment is essentially tree-structured and features a simplified action space that lacks the diversity and complexity of real-world GUIs. This limitation, which the authors themselves acknowledge, may reduce the generalizability of the findings and constrains the practical implications for deployed agents.\n* Missing real-world validation: Although OOD generalization is evaluated within the simulation (Table 3), the paper does not test the transferability of the trained agents to real-world GUIs or public benchmarks (e.g., Mind2Web). It remains unclear how the learned policies and training dynamics would translate outside the simulated setting.\n* Incremental novelty of simulation-based training: The idea of using simulation platforms to train and benchmark GUI agents, particularly in reinforcement learning settings, is not entirely novel, with existing platforms such as AndroidWorld already exploring similar directions. The paper would benefit from a clearer positioning relative to prior work in this area.", "was_revised": true, "final_justification": "The authors have addressed the main concerns raised in the review. Their additional experiments on real-world GUI benchmarks and further clarification of GE-Lab’s novelty provide support for the paper’s contributions. Initially, I felt the work was borderline but leaned toward a borderline accept due to the lack of a neutral rating option. With the clarification provided, I now believe the current score is appropriate and will maintain my positive rating.", "rebuttal": "We would like to thank the reviewer for their thoughtful feedback and positive evaluation of our work. We appreciate the recognition of GE-Lab as a valuable contribution to the field, particularly for its **flexibility in modeling complex GUI structures and controllable OOD scenarios.** We also thank the reviewer for **highlighting the thoroughness of our experimental setup and the clarity of our ablation studies**, which helped us assess the performance of different training regimes. The feedback encourages us to further refine our work and explore additional avenues for improving the generalization and interactivity of GUI navigation agents.\nBelow are our detailed responses to the Reviewer DUjd.\n\n1. **Weakness 1:** Limited environment realism. Whether the simplified simulation environment and action space reduce the generalizability of the findings and constrains the practical implications for deployed agents.\n  - Response:\n    * We acknowledge that the simulated environment used in our study exhibits limited diversity and complexity compared to real-world GUIs. However, we would like to emphasize that evaluating performance on real-world benchmarks is not the primary focus of this work. Instead, our goal is to validate the feasibility of using reinforcement learning for screen navigation tasks. By further adopting real-world icons and layouts inspired by actual apps within GE-Lab, the domain gap can be largely reduced. \n    * In fact, the agent trained in the simulated environment demonstrates a notable degree of generalization to real-world GUIs. To assess this, we manually constructed a static benchmark by sampling 1,569 instances from several open-source real-world GUI datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. The test samples cover a broader action space: CLICK, COMPLETE, WAIT, SCROLL, and TYPE. Without any further fine-tuning, we evaluated our agent, which had not encountered WAIT, SCROLL, or TYPE actions during the training process, directly on this benchmark.\n     **The results are presented in the Table 1 below. Despite lacking explicit training on WAIT, SCROLL, and TYPE, the agent still achieves non-trivial accuracy on these actions.** This suggests that the agent possesses some degree of atomic competence in handling GUI elements beyond its training distribution, highlighting the generalizability of our approach and its practical implications for real-world deployment.\n\n**Table 1:** Performance of the agent on real-world static benchmark with extended action space.\n\n| Action   | Total Number | Success Number | Success Rate |\n| ---------- | -------------- | ---------------- | -------------- |\n| CLICK    | 971          | 622            | 64.06%       |\n| COMPLETE | 230          | 221            | 96.09%       |\n| WAIT     | 231          | 99             | 42.86%       |\n| TYPE     | 129          | 119            | 92.25%       |\n| SCROLL   | 8            | 5              | 62.50%       |\n\n2. **Weakness 2:** Missing real-world validation.  Although OOD generalization is evaluated within the simulation (Table 3 in paper), the paper does not test the transferability of the trained agents to real-world GUIs or public benchmarks.\n - Response: \n    - We further investigated the generalizability of our approach by incorporating SFT on a subset of real-world data. Specifically, we continue training our agents (SFT, ST-RL, MT-RL) with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks. The results support the claim that our training framework yields agents with strong generalization capabilities across both simulated and real-world GUI tasks.\n    - **As shown in Table 2 below, the results demonstrate that our training framework yields agents with strong generalization capabilities across both simulated and real-world GUI tasks.** Notably, MT-RL outperforms ST-RL, which in turn outperforms SFT, in terms of accuracy. This observation mirrors the trend observed in our simulated environment as reported in the main text and reinforces our key findings: (1) MT-RL training yields better generalization than ST-RL training, and (2) ST-RL generalizes better than SFT. Additionally，we appreciate review's recommendation to test on the Mind2Web. While Mind2Web is a comprehensive bench for developing and evaluating web agents, its original version is primarily text-based, focusing on HTML and textual descriptions of tasks. Although a multimodal version of Mind2Web, which pairs HTML documents with webpage screenshots, has been released, its core structure and tasks are still fundamentally designed for web agents that operate on textual data. Visual perception of the GUI is a primary input channel in our agent, Therefore, the Mind2Web benchmark does not align with the architectural and data requirements of our multimodal GUI agent.\n    - We opt to conduct the experiments in simulation primarily due to the significant development cost associated with providing accurate reward signals in real-world environments. Simulation enables us to validate the proposed MT-RL framework in a controlled manner. This not only demonstrates the feasibility of our approach but also lays a foundation for future development of real-world environments, which would ultimately advance the capabilities of GUI agents.\n\n**Table 2:** Performance of the agent on real-world grounding benchmark and interactive benchmark.\n\n| Model                              | ScreenSpot [5] | ScreenSpot-v2 [6] | FuncPred [7] | MoTIF [8] | Refexp [9] | VWB AG [10] | VWB EG [10] | AndroidWorld [11] | Average   |\n| :----------------------------------- | :--------------- | :------------------ | :------------- | :---------- | :----------- | :------------ | :------------ | :------------------ | :---------- |\n| Qwen2.5-VL-7B-Base (paper report)  | 84.70          | -                 | -            | -         | -          | -           | -           | -                 | -         |\n| Qwen2.5-VL-7B-Base (our report)    | 84.01          | 80.34             | 48.25        | 71.93     | 79.46      | 72.81       | 90.07       | 10.34             | 67.15     |\n| Qwen2.5-VL-7B-Continue-Train       | 84.91          | 84.43             | 59.50        | 68.30     | 72.13      | 67.96       | 93.70       | 12.06             | 67.87     |\n| Qwen2.5-VL-7B-SFT-Continue-Train   | 85.06          | 85.06             | 59.30        | **80.47** | **83.19**  | 68.93       | 92.01       | 12.93             | 70.87     |\n| Qwen2.5-VL-7B-ST-RL-Continue-Train | 85.53          | **85.61**         | **60.45**    | 79.41     | 77.88      | 70.87       | **92.98**   | 14.65             | 70.92     |\n| Qwen2.5-VL-7B-MT-RL-Continue-Train | **86.08**      | **85.61**         | 59.40        | 78.18     | 81.77      | **76.70**   | **92.98**   | **15.51**         | **72.03** |\n\n3. **Weakness 3:** Incremental novelty of simulation-based training. The paper would benefit from a clearer positioning relative to prior work in this area.\n- Response: \n  * We have carefully surveyed several existing GUI-based environments—AndroidWorld (116 tasks), OSWORLD (369 tasks), and WebArena (812 tasks)—which target phone-use, computer-use, and browser-use scenarios, respectively. While these platforms serve as valuable benchmarks, the number of tasks in each remains relatively limited. This limitation stems from the significant engineering effort involved in designing tasks that support automated and accurate reward computation — often necessitating task-specific setups such as generating predefined files, which makes large-scale, diverse and scalable training infeasible. **To the best of our knowledge, no existing work has demonstrated multi-turn reinforcement learning training in real-world GUI environments, highlighting the practical challenges and the novelty of our simulation-driven approach.**\n  * In contrast to the above three platforms, GE-Lab, as a simulated environment, is capable of generating an unlimited number of tasks while also automating the provision of accurate reward signals. This is critical for scaling up multi-turn RL training and for exploring learning dynamics that are otherwise infeasible to study in constrained real-world environments.\n\n**Reference** (Due to space limitations, our reference list includes only the titles of the articles.)\n\n[1] Androidinthewild: A large-scale dataset for android device control[J].\n\n[2] Android in the zoo: Chain-of-action-thought for GUI agents[J]. \n\n[3] Amex: Android multi-annotation expo dataset for mobile GUI agents[J]. \n\n[4] Mind2web: Towards a generalist agent for the web[J].\n\n[5] Seeclick: Harnessing GUI grounding for advanced visual GUI agents[J].\n\n[6] Os-atlas: A foundation action model for generalist GUI agents[J].\n\n[7] Autogui: Scaling GUI grounding with automatic functionality annotations from llms[J]. \n\n[8] A dataset for interactive vision-language navigation with unknown command feasibility[C]\n\n[9] Uibert: Learning generic multimodal representations for ui understanding[J].\n\n[10] Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?[J]. \n\n[11] Androidworld: A dynamic benchmarking environment for autonomous agents[J]."}, {"review_id": "On9BihgjtU", "reviewer": "Reviewer_htJ2", "summary": "This paper introduces GE-Lab, an environment that composes navigation apps to train and test GUI agents. On this environment, they tested three types of training: supervised fine-tuning, single-step reinforcement learning and multi-turn reinforcement learning. Overall, they show how supervised fine-tuning is effective for in-distribution data, while single-step reinforcement learning is effective for out-of-distribution data. Finally, multi-turn reinforcement learning usually outperforms the previous strategies, especially in the interactive environment.", "questions": "- Can you show the difference in training time for the three methods?\n- Can you show the sampling efficiency difference for the three methods?\n- To improve the clarity, introduce the acronym before using them.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is well written, and the set of experiments is clear and informative. The paper is significant, although the environment is too simple for a real-case scenario. The work is original, even if prior works explored the use of multi-turn RL for agents. The experiments showed multi-turn RL generally improves performances; however they lack to analyze the efficiency in term of training speed and sample efficiency.", "was_revised": true, "final_justification": "I have raised my score, as they also assess the time efficiency of the 3 methods in the appendix. \nI recommend acceptance, as this work is valuable, although it doesn't present surprising results, it scientifically assesses the benefit in multi-turn RL.", "rebuttal": "We would like to thank the reviewer for their positive and constructive feedback. **We appreciate the recognition of the clarity and significance of our experimental setup, as well as the acknowledgment of the originality of our work in the context of multi-turn RL for GUI agents.** We also value the reviewer’s suggestion regarding the analysis of training speed and sample efficiency, and we have now included additional details on this aspect in the following sections of the revised manuscript. The feedback has been invaluable in helping us enhance the comprehensiveness of our work.\nBelow are our detailed responses to the Reviewer htJ2.\n\n1. **Question 1:** Can you show the difference in training time for the three methods?\n- Response: The difference in training time for the three methods.\n  * As mentioned in Appendix A.1 (line 478-482), A typical training time for a single SFT agent is approximately 3 to 4 GPU hours. The training of ST-RL agent generally requires about 48 GPU hours, while MT-RL agent typically trains for approximately 36 GPU hours.\n  * As stated in Appendix A.1, Dataset & Benchmark, the SFT dataset consists of 60,864 samples, the ST-RL dataset contains 12,439 samples, and the MT-RL dataset is based on online interactive training with 2,162 tasks. **The average training time per sample/task is shown in the Table 1 below.**\n\n**Table 1:** Training Time and Sampling Efficiency Comparison for the Three Methods\n\n| Method | Dataset Size  | Training Time | Average Training Time              | Interaction Count | Feedback                               |\n| -------- | --------------- | --------------- | ------------------------------------ | ------------------- | ---------------------------------------- |\n| SFT    | 60864 samples | 3~4 GPU Hours | 0.000049~0.000066 GPU Hours/sample | 60864             | Manually labeled                       |\n| ST-RL  | 12439 samples | 48 GPU hours  | 0.00386 GPU Hours/sample           | 99512             | Manually labeled      |\n| MT-RL  | 2162 tasks    | 36 GPU hours  | 0.0167 GPU Hours/task              | ~89939            | Environment Feedback |\n\n2. **Question 2:** Can you show the sampling efficiency difference for the three methods?\n- Response: The sampling efficiency difference for the three methods.\n  * For both ST-RL and MT-RL, the number of generations per round is 8. **Table 1 highlights significant differences in sampling efficiency across the three methods.** SFT has the lowest average training time per sample, indicating high efficiency for in-distribution data. In contrast, ST-RL and MT-RL show higher training times per sample or task, reflecting the additional time consumption required by reinforcement learning. However, ST-RL and MT-RL achieve higher interaction counts, suggesting that they are more effective in exploring diverse states and generating robust agent behaviors, particularly in out-of-distribution and interactive settings.\n\n3. **Question 3:** To improve the clarity, introduce the acronym before using them.\n- Response: About acronym.\n  * We will carefully review the acronyms in the final version to avoid any potential confusion for the readers."}, {"review_id": "o85h0zA3Jz", "reviewer": "Reviewer_VsGX", "summary": "The paper introduces GUI Exploration Lab (GE-Lab), a synthetic environment for GUI agent navigation. It constructs a graph where each page has icons that, when clicked, lead to other nodes (pages). The tasks are straightforward: navigate from page X to page Y. The authors test three training methods—supervised fine-tuning (SFT) to memorize paths, single-turn reinforcement learning (ST-RL) for generalization, and multi-turn reinforcement learning (MT-RL) for exploration. Experiments show MT-RL performs best in interactive settings. The paper claims these findings guide GUI agent development, but the reviewer finds the work lacks significance due to its oversimplified approach.", "questions": "**Does training in GE-Lab actually improve agent performance in real GUI environments?** If so, provide data or reasoning to support this. My main concern is that the conclusions drawn in this paper can also be drawn from prior work due to the lack of uniqueness from the GUI side. I will only be convinced and give higher marks if this environment is proven useful for enhancing model performance on realistic tasks or is sufficiently realistic for evaluation.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- **Originality**: The introduction of GE-Lab is a novel contribution, providing a controlled environment for GUI navigation research. The systematic comparison of SFT, ST-RL, and MT-RL training paradigms offers a structured approach to agent development, which is a fresh perspective in the field.\n- **Quality**: The experimental design is technically sound, with detailed results (e.g., Table 1 shows MT-RL achieving Pass@1 of 17.47 and Pass@5 of 25.16) and a clear methodology using Qwen2.5-VL-7B in a tree-structured environment (depth 7). The use of both static and interactive benchmarks adds rigor.\n- **Clarity**: The paper appears well-structured, with figures (e.g., Figure 1, Figure 2) and tables (e.g., Table 1, Table 2) likely aiding understanding. No clarity issues were raised in the critique, suggesting effective communication of methods and results.\n- **Significance**: The work addresses a relevant challenge in GUI agent navigation, a growing area with LVLMs. The benchmarking and training insights could inform future research, though practical impact is limited.\n\n**Weaknesses:**\n- **Quality**: The paper oversimplifies GUI interactions by reducing navigation to clicking icons, ignoring complex actions like scrolling, text input, right-clicking, or double-clicking (Appendix A.2 acknowledges plans to expand action space, but current work is limited). This simplification undermines the robustness of the findings for real-world applications.\n- **Significance**: The lack of evidence for transfer learning from GE-Lab to real GUI environments significantly reduces practical impact. Performance drops in OOD settings (e.g., Table 3, Env-Image: 34.17 for MT-RL vs. 63.25 in Env-Base) suggest limited generalization, aligning with concerns that synthetic environments may not translate to real-world improvements.\n- **Originality**: While GE-Lab is novel, the concept of synthetic navigation environments echoes prior work (e.g., text-based games), and the conclusions (e.g., RL enhances exploration) are not entirely new, reducing the paper’s distinctiveness.\n- **Clarity**: Although generally clear, the paper’s failure to address how its simplified model applies to complex GUI tasks may confuse readers expecting real-world relevance.\n\nAmmanabrolu, P., & Riedl, M. (2019). Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3557–3565, Minneapolis, Minnesota. Association for Computational Linguistics.\n\nOsborne, P., Nõmm, H., & Freitas, A. (2022). A Survey of Text Games for Reinforcement Learning Informed by Natural Language. Transactions of the Association for Computational Linguistics, 10, 759-776. https://doi.org/10.1162/tacl_a_00495\n\nGur, I. (2018). Learning to Navigate the Web. arXiv preprint arXiv:1812.09195.\n\nLi, Y., & Riva, O. (2021). Glider: A Reinforcement Learning Approach to Extract UI Scripts from Websites. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1420–1430.", "was_revised": true, "final_justification": "The additional experiments partially address my concerns about transfer learning and practical impact. The results in Tables 1 and 2 do demonstrate some effectiveness of GE-Lab for real-world GUI tasks, and I adjust my score accordingly.", "rebuttal": "We would like to thank the reviewer for the thoughtful feedback. **We appreciate the recognition of GE-Lab as a novel contribution to the field, and we also value the reviewer's acknowledgment of the technical soundness and clarity of our experimental design.** Regarding the practical impact, we have now elaborated on how the insights from our work can inform future research and practical applications in the following section. The feedback has been invaluable in helping us refine and strengthen our manuscript.\nBelow are our detailed responses to the Reviewer VsGX.\n\n1. **Weakness 1 Quality:** This simplification undermines the robustness of the findings for real-world applications.\n- Response:\n  * To address the concern about whether the simplification of GUI interactions might limit the robustness of our finding, **we expanded the action space to assess the agent’s ability to perform a broader range of actions.** Specifically, we sampled 1,569 instances from an open-source real-world GUI dataset (including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]) and manually constructed a static benchmark. The test samples in this benchmark cover the following 5 actions: CLICK, COMPLETE, WAIT, SCROLL, and TYPE, the last three of which were not included in the previous action space. We then evaluated the agent without any fine-tuning on this extended benchmark. As shown in Table 1, the agent achieved non-trivial accuracy on WAIT, SCROLL, and TYPE actions, despite not encountering these three new actions during the training process. This indicates that the agent has acquired transferable atomic skills for interacting with GUI elements beyond its original action space, supporting the potential applicability of our approach to more complex real-world scenarios.\n\n**Table 1:** Performance of the agent on real-world static benchmark with extended action space.\n\n| Action   | Total Number | Success Number | Success Rate |\n| ---------- | -------------- | ---------------- | -------------- |\n| CLICK    | 971          | 622            | 64.06%       |\n| COMPLETE | 230          | 221            | 96.09%       |\n| WAIT     | 231          | 99             | 42.86%       |\n| TYPE     | 129          | 119            | 92.25%       |\n| SCROLL   | 8            | 5              | 62.50%       |\n\n2. **Weakness 2 Significance & Question 1:** Whether the lack of evidence for transfer learning from GE-Lab to real GUI environments significantly reduces practical impact. & **Does training in GE-Lab actually improve agent performance in real GUI environments?**\n- Response: \n  * We further trained the SFT agent, ST-RL agent, and MT-RL agent on real-world datasets (including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]), and the accuracy results are shown in the Table 2. It is evident that **the agent's performance on the real-world dataset exceeds the baseline**, demonstrating that GE-Lab can improve agent performance in real-world GUI environments. We believe this can address the reviewer’s concern about the practical impact and the major question that the reviewer raised. \n  * As shown in Table 2, all agents trained in GE-Lab and then fine-tuned on real-world data outperform both the base model (Qwen2.5-VL-7B-Base) and the model that continued training directly on real-world data without GE-Lab pretraining (Qwen2.5-VL-7B-Continue-Train). Notably, **the MT-RL agent achieves the best performance across most benchmarks**, with the highest accuracy on ScreenSpot (86.08%), and Refexp (81.77%), as well as strong results on interactive benchmarks like AndroidWorld (15.51%). The average performance of the MT-RL agent is 72.03%, surpassing the other models in overall accuracy, reflecting its robustness.\n  This demonstrates that reinforcement learning in GE-Lab provides generalizable screen navigation capabilities that effectively transfer to complex real-world tasks. The consistent performance gains from SFT to ST-RL to MT-RL, as seen in both the individual benchmarks and the overall average, mirror our simulation findings. These results offer strong empirical support that the skills acquired in GE-Lab are transferable and beneficial when applied to real-world GUI environments, both in grounding and interaction settings.\n  * It is worth noting that the results above address both the concern regarding the significance of this work and the major question raised by the reviewer: \"Does training in GE-Lab actually improve agent performance in real GUI environments?\"\n\n**Table 2:** Performance of the agent on real-world grounding benchmark and interactive benchmark.\n\n| Model                              | ScreenSpot [5] | ScreenSpot-v2 [6] | FuncPred [7] | MoTIF [8] | Refexp [9] | VWB AG [10] | VWB EG [10] | AndroidWorld [11] | Average   |\n| :----------------------------------- | :--------------- | :------------------ | :------------- | :---------- | :----------- | :------------ | :------------ | :------------------ | :---------- |\n| Qwen2.5-VL-7B-Base (paper report)  | 84.70          | -                 | -            | -         | -          | -           | -           | -                 | -         |\n| Qwen2.5-VL-7B-Base (our report)    | 84.01          | 80.34             | 48.25        | 71.93     | 79.46      | 72.81       | 90.07       | 10.34             | 67.15     |\n| Qwen2.5-VL-7B-Continue-Train       | 84.91          | 84.43             | 59.50        | 68.30     | 72.13      | 67.96       | 93.70       | 12.06             | 67.87     |\n| Qwen2.5-VL-7B-SFT-Continue-Train   | 85.06          | 85.06             | 59.30        | **80.47** | **83.19**  | 68.93       | 92.01       | 12.93             | 70.87     |\n| Qwen2.5-VL-7B-ST-RL-Continue-Train | 85.53          | **85.61**         | **60.45**    | 79.41     | 77.88      | 70.87       | **92.98**   | 14.65             | 70.92     |\n| Qwen2.5-VL-7B-MT-RL-Continue-Train | **86.08**      | **85.61**         | 59.40        | 78.18     | 81.77      | **76.70**   | **92.98**   | **15.51**         | **72.03** |\n\n3. **Weakness 3 Originality:** While GE-Lab is novel, the concept of synthetic navigation environments echoes prior work (e.g., text-based games), and the conclusions (e.g., RL enhances exploration) are not entirely new, reducing the paper’s distinctiveness.\n- Response: \n  * Although the concept of synthetic navigation environments is not new, we believe that the contribution of this study is distinct in a number of important ways. \nFirst, prior synthetic navigation environments have largely been text-based. In contrast, this study is the first to construct a GUI simulation environment that supports interactive training and provides accurate reward signals. **This simulated environment enables us to validate the effectiveness of reinforcement learning methods**, providing a foundation for future real-world environment development and enhancing agent performance in real-world settings.\n  * Second, while it may seem intuitive that reinforcement learning can enhance the exploration capabilities of agents, **there are still several practical challenges, such as reward hacking in RL and the lack of progression during MT-RL training**. To address these challenges, we mitigated reward hacking by adjusting the training data proportions and improved the efficiency and stability of MT-RL through curriculum learning. The insights gained from addressing these issues in GE-Lab can be effectively transferred to real-world applications.\n\n4. **Weakness 4 Clarity:** Although generally clear, the paper’s failure to address how its simplified model applies to complex GUI tasks may confuse readers expecting real-world relevance.\n- Response: \n  * We further investigated the generalizability of our approach by incorporating SFT on a subset of real-world data. Specifically, we augmented our training pipeline (SFT, ST-RL, MT-RL) with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks. \n  * The results demonstrate that **our training methodology yields agents with strong generalization capabilities across both simulated and real-world GUI tasks.** Notably, MT-RL outperforms ST-RL, which in turn outperforms SFT, in terms of accuracy. This observation mirrors the trend observed in our simulated environment as reported in the main text and reinforces our key findings: (1) MT-RL yields better generalization than ST-RL, and (2) ST-RL generalizes better than SFT.\n  * We opt to conduct the experiments in simulation primarily due to the significant development cost associated with providing accurate reward signals in real-world environments. **Simulation enables us to validate the proposed multi-turn reinforcement learning framework in a controlled manner.** This not only demonstrates the feasibility of our approach but also lays a foundation for future development of real-world environments, which would ultimately advance the capabilities of GUI agents.\n\n5. **Suggested References:**\n\n* We will carefully review the suggested references and incorporate them to enhance our writing and references section.\n\n**Reference** (Due to space limitations, our reference list includes only the titles of the articles. For reference [9-11], please refer to our response to reviewer DUjd.)\n\n[1] Androidinthewild: A large-scale dataset for android device control[J].\n\n[2] Android in the zoo: Chain-of-action-thought for GUI agents[J]. \n\n[3] Amex: Android multi-annotation expo dataset for mobile GUI agents[J]. \n\n[4] Mind2web: Towards a generalist agent for the web[J].\n\n[5] Seeclick: Harnessing GUI grounding for advanced visual GUI agents[J].\n\n[6] Os-atlas: A foundation action model for generalist GUI agents[J].\n\n[7] Autogui: Scaling GUI grounding with automatic functionality annotations from llms[J]. \n\n[8] A dataset for interactive vision-language navigation with unknown command feasibility[C]"}, {"review_id": "TPl9ESQ1Hj", "reviewer": "Reviewer_JzUR", "summary": "The paper considers the pages in GUI as graph and apply single-turn and multiple-turn reinforcement learning to learn and explore the GUI environment.\nThe proposed approach enables the agent to learn structured navigation behaviors. The agent is also to emulate user actions across different task configurations.", "questions": "- How exactly is the OOD test environment constructed? What makes it OOD compared to the ID training set? Please clarify with specific differences, e.g., layout, content, structure.\n- For complex tasks, how long can the history string be? Is there a limit imposed, and if so, how is truncation handled?\n- In Equation (2), the reward function is composed of four parts, but none are mathematically defined. Please provide a formal definition of each component and explain how they are computed. And what is the weight factor, $\\lambda$, for each component?\n- RL is known to exhibit high variance. The readers would appreciate it if they can see the stability of the use of RL in this case, which can be demonstrated in Fig. 5 by adding shaded variance regions.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n- The author provides a simulation environment engine that can generate a GUI environment and proposes to try RL methods to explore it.\n\nWeaknesses:\n- The author adopted a randomized placement of the icons. However, this setting may not reflect realistic UI design principles. In real-life practical situations, GUIs tend to follow structured and intuitive layouts to enhance usability. Shouldn't it be more suited to learn icons with placement that is similar to modern-day UI design to complete its task to be useful?\nTraining agents on fully randomized interfaces, even for the sake of OOD robustness, may result in behavior that is less transferable to real-world systems. Maybe it would be more informative to consider a setting that maintains a plausible UI design and structure.\n- There is little description about the engine that generates the GUI learning environment for LLM agents. How the engine works is not clear; for example, what is the generated GUI environment like? how much information and available actions are on the visible page? how many other pages can one page reach on average?\n\n**nit**\n- line35: \"mobile apps\"\n- Fig.1(c) Is it true that all pages (the node) have only a degree of 1, i.e., they only have a distinct neighbor? The illustration is not very clear.\n- For Fig. 2, I would suggest a top-down, left-to-right ordering of the A, B, and C components. Current ordering makes the visual confusing.\n- line 108: typo \"extende\"", "was_revised": true, "final_justification": "My concerns regarding the strategy for generating the OOD settings have been addressed, and I now recognize the impact of this submission. I am updating my overall rating from 2 to 4, and the originality score from 2 to 4 as well. However, the current manuscript still suffers from clarity issues, including typos and dense methodological explanations. If the chairs are considering accepting this submission, I would recommend that a thorough revision be requested to improve clarity and readability.", "rebuttal": "We would like to thank the reviewer‘s constructive feedback. **We appreciate the recognition of our simulation engine as a valuable tool for generating and exploring GUI environments.** We also acknowledge the reviewer’s concern about the randomized icon placement and its transferability to real-world systems. As discussed in the following section, we attempt to address this issue by considering more structured UI designs in our experiments to maintain a balance between OOD robustness and practical relevance. Additionally, we have provided further clarification on the workings of the engine and its generated environments. The feedback has greatly helped us in refining these aspects.\n\nBelow are our detailed responses to Reviewer JzUR.\n\n1. **Weakness 1:** Shouldn't it be more suited to learn icons with placement that is similar to modern-day UI design to complete its task to be useful? Whether randomized icon placement affects generalization in real-world environments.\n- Response: \n  * The decision to randomly sample and place icons in the simulated environment is intentional and driven by the goal of promoting generalization. If the icons were not randomized, the agent might memorize which icons to click for specific tasks, making it difficult to evaluate the agent's memorization for the navigation map. Similarly, if the positions were not randomized, the model might overfit to specific data points, memorizing fixed positions and clicking on. These would hamper the evaluation of the agent's grounding ability. By contrast, **if the agent is still able to correctly complete tasks in the randomized setting, this would strongly demonstrate that it possesses a solid screen navigation ability.**\n  * Importantly, we have also validated the agent's performance in real-world GUI environments, where layouts tend to follow more structured, modern UI design patterns. **The results showed that the agent can successfully perform screen navigation in well-structured real-world settings.** Specifically, we continue training the agents of three methods with 24k samples drawn from publicly available datasets, including AITW [1], AITZ [2], AMEX [3], and Mind2Web [4]. We then evaluated the resulting agents on both real-world grounding and interactive benchmarks. As shown in Table 1, MT-RL achieves higher accuracy than ST-RL, which in turn outperforms SFT. This suggests that the agents trained in the simulated environment are able to generalize effectively to real-world benchmarks, encompassing both grounding and interactive tasks.\n\n**Table 1:** Performance of the agent on real-world grounding benchmark and interactive benchmark.\n\n|Model|ScreenSpot[5]|ScreenSpot&#8209;v2&nbsp;[6]|FuncPred[7]|MoTIF[8]|Refexp[9]|VWBAG[10]|VWBEG[10]|AndroidWorld[11]|Average|\n|-|-|-|-|-|-|-|-|-|-|\n| Qwen2.5-VL-7B-Base(paper report) |84.70|-|-|-|-|-|-|-|-|\n|Qwen2.5-VL-7B-Base(our report)|84.01|80.34|48.25|71.93|79.46|72.81|90.07|10.34|67.15|\n|Qwen2.5-VL-7B-Continue-Train|84.91|84.43|59.50|68.30|72.13|67.96|93.70|12.06|67.87|\n|Qwen2.5-VL-7B-SFT-Continue-Train|85.06|85.06|59.30|**80.47**|**83.19**|68.93|92.01|12.93|70.87|\n|Qwen2.5-VL-7B-ST-RL-Continue-Train|85.53|**85.61**|**60.45**|79.41|77.88|70.87|**92.98**|14.65|70.92|\n|Qwen2.5-VL-7B-MT-RL-Continue-Train|**86.08**|**85.61**|59.40|78.18|81.77|**76.70**|**92.98**|**15.51**|**72.03**|\n\n2. **Weakness 2:** How the engine works is not clear; for example, what is the generated GUI environment like? how much information and available actions are on the visible page? how many other pages can one page reach on average? Details of the GE-Lab engine and the simulated environment.\n- Response:\n  * GE-Lab is a novel simulation environment engine. Specifically, GE-Lab first **generates a graph** of the environment based on user-defined parameters, which determines the navigation relationships between different screens. Then, GE-Lab **renders a screen** for each node of the graph, with each screen displaying n icons (where n is the degree of the node). Clicking an icon triggers a page transition. The icons used in GE-Lab are randomly sampled from an icon library, enhancing the diversity of the simulation environment.\n  * The simulated environment used for our main experiments is Env-Base, as described in Section 4.1. **This is a graph with a maximum depth of 7**. Excluding system-defined edges representing the home and back actions, the branching structure of the environment follows a node distribution of [5, 3, 2, 2, 1, 1, 0] at each respective level. Figure 6 visualizes this graph structure. The graph is partitioned into five independent subtrees, which are allocated to the SFT training set, RL training set, and test set in a 2:2:1 ratio, respectively.\n  * In the visual representation, the number of icons corresponds to the clickable areas that trigger page transitions, with the task completed by selecting the \"complete\" icon. At the root node, there are 5 icons and clickable areas, allowing access to 5 pages. For first-level child nodes, there are 4 icons (3 regular + 1 \"Back\") and 4 clickable areas, enabling navigation to 4 pages. For other child nodes, which include \"Home\" and \"Back\" icons, **the number of icons is NodeNum[i] + 2, resulting in NodeNum[i] + 1 clickable areas**, allowing access to NodeNum[i] + other pages. On average, each screen contains more than 5 clickable icons.\n\n3. **Question 1:** How exactly is the out-of-domain (OOD) test environment constructed? What makes it OOD compared to the  in-domain (ID) training set?  The construction details of the OOD environment and its differences from the in-domain environment.\n- Response: \n  * As described in Dataset & Benchmark (A.1, lines 483-487), **one of the five subtrees under the root node is excluded from both the training and test sets, representing an OOD environment.** The path data from the other two subtrees are used for SFT training, while the remaining two subtrees contribute path data for RL training, which are considered in-domain environments. The OOD samples involve paths that were not encountered during the agent's training process.\n  * In addition, we have constructed other **more challenging OOD environments**. As described in Section 4.1, we generated Env-Image, Env-Name, and Env-Position by augmenting the icon’s image, name, and position, respectively. Furthermore, we introduced noisy icons to create the Env-Noise environment.\n  * In terms of the differences between OOD and ID environments, the icons on both types of screens are randomly generated and positioned at different nodes in the graph. As a result, the screen layouts, content, and structure of the two environments are distinct.\n\n4. **Question 2:** For complex tasks, how long can the history string be? Is there a limit imposed, and if so, how is truncation handled? Context string length constraints and truncation strategies.\n- Response: \n  * In both multi-turn RL online training and actual inference, **we set a maximum number of interaction rounds** (e.g., 12). According to our agent protocol (Section 3.2.2), the context token length, composed of the text from a single dialogue round and the current observation image, is less than 1k tokens, which is within the model's context length. Therefore, **no additional truncation is required.**\n\n5. **Question 3:** Please provide a formal definition of each component and explain how reward are computed. And what is the weight factor for each component?\n- Response:\n  * The reward function formula is **provided in Appendix A.4**. Thank you for your suggestion; we will add a reference to the section containing the details of the reward function in Section 3.3.1.\n\n6. **Question 4:** RL is known to exhibit high variance. The readers would appreciate it if they can see the stability of the use of RL in this case, which can be demonstrated in Fig. 5 by adding shaded variance regions. The variance in RL training metrics requires visualization to demonstrate training stability.\n- Response: \n  * Thank you for the reminder. **We will include the variance information in Figure 5** in the final version.\n\n7. **Nit 2:** In Figure 1C, is it true that all pages (the node) have only a degree of 1, i.e., they only have a distinct neighbor? The illustration is not very clear. Illustration of node diagrams in Figure 1(C,D,E).\n- Response:\n  * To clarify, in Figure 1C, **only nodes with degree 1 represent training data that contains knowledge about two connected nodes.** This does not imply that every node in the actual state transition graph has a degree of 1.\n  * Figure 1C visualizes all training samples as nodes with degree 1, indicating that the **SFT training for a single sample only memorizes knowledge of a one-step transition**. As shown in Figure 2A, SFT samples follow the data protocol of POMDP, and some of them contain not only one-step transition knowledge (corresponding to degree 1), but also historical trajectories in the history. \n\n8. **Nit 3:** For Figure 2, I would suggest a top-down, left-to-right ordering of the A, B, and C components. Current ordering makes the visual confusing.\n- Response:\n  * Thank you for the suggestion. **We will adjust the numbering sequence** in Figure 2 accordingly.\n\n9. **Nit 1 & Nit 4:** Typo.\n- Response:\n  * Thank you for your thorough review. We will make the necessary revisions and perform another round of checks.\n\n10. **Paper Formatting Concerns:** Reference formatting and authority verification.\n- Response:\n  * **We will thoroughly review all references, confirm their publication status, and ensure the quality of the journals in which they appear.** This includes addressing issues related to \"Improper citation of foundational RL concepts\" and \"Formatting errors in in-text citations.\" Due to space limitations, we will present the final references in the revised version.\n\n**Reference**\n\n Due to space limitations, please refer to our response to Reviewer DUjd for details regarding the references."}]

Author rebuttal:

Meta-review: This paper introduces GUI Exploration Lab, a simulation environment engine for GUI agents. This environment enables flexible GUI compositions with screens, icons, and navigation graphs. This not only allows scale-scale GUI synthesis but also creates controllable OOD evaluation. Experiments offer a systematic comparison of different training paradigms, including SFT, SF-RL, and MT-RL.   

During the rebuttal, the authors provided a real-world evaluation and training efficiency analysis, which further strengthened the work. I would encourage the authors to include the new results and analysis in the final version.
Decision: Accept (poster)