GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
Authors: Tao Liu
Chongyu Wang
Rongjie Li
Yingchen Yu
Xuming He
Song Bai
Venue: NeurIPS 2025 poster ()
Keywords: GUI agents
Multimodal large language models
Reinforcement learning
Primary area: applications

Abstract: While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, GUI-Rise, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks.

48 12 have reignited interest in agents GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
Tao Liu'*
Chongyu Wang!" Rongjie Li²
Yingchen Yu²
Xuming Hel Bai Song"
ShanghaiTTeh University "ByteDance
Shanghai Engineering Research Center of Intelligent Vision and Imaging
{liutao, wangchy, hexmmAahannhaaitehhhho edu.cn
lirj@bytedance.com com, yingchen001@e ntu sg, songbai site@gmail.com
Abstract
While Multimodal Large Language Models (MLLMs) have advanced GUI navi- gation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present reasoning-enhanee that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for fu- ture steps. Based on this framework, we train a GUT agent, GUT-Rise, through pseudo-labeled trajectories and reinforcement learn-
framework
supervised fine-tuning ing with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including history-aware objective, directly linking sum- mary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-ar results under identical training scenar-
data conditions, with particularly strong performance in out-of- ios. These findings validate our framework's 's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks. Code is available at ltttps://eeon22,. www.ane22...ggutboeee 1
Introduction
Recent advances in multimodal artificial intelligence
that can autonomously navigate Graphical User Interfaces (UUls. ls). By translating natural-language instructions into actions on screen, these agents hold the promise reshaping human computer interaction and strcamlining everyday workflows [8]
23 46 37). At the core of this progress are Multimodal Large Language Models (MLLMs) [18 17 which combine visual perception with linguistic reasoning to identify, interpret, and manipulate interface elements [14] 49). Despite this promise, deploying GUI agents real-world applications remains challenging: an effective agent must sustain coherent behaviour over multi-step interactions, continuously reason about the evolving interface state, and integrate its own history.
19
current agents remain far from reliable. Approaches that lean prompt engineering with proprietary LLMSs 46 52 are constrained by the frozen capabilities of the underlying model and
Although notable progress has been made in multi-step GUI navigation 40
49 6
37 19
such as GPT-4 [
cannot be easily adapted to new domains. Supervised Fine-Tuning (SFT) on open- source backbones
'These authors contributed equally to this work. I Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Structured Reasoning Subtask:
<Prognea category has selected timeframe "Last Laptoge &
Computees,
days" complett the task </Progress Estimation>
<Dedsion Ressoning: Selecting att 50 days" filter products those popular in the pasi serven daeys </Decision Ressoning
Action Prediction Subtask:
<Actiona: scttion CLOCK Value:: 'asst bosstion: (0.58, </Actions
History Summarization Subtaski
cHistory Summary- The user selected "Deskoop Computers" under "Compterrs & Office Equipment" and ser the time frame to the lest 30 days to siew recentt popuiar products.. </History Summanyo
History Summary:
The user cliced "Computors Office Equipment" and refonud to "Deshhop Computers.
Observation
Observation [+]
Action Dutpet: CUEX "Last 30 days
Step
Step t+1
Instruction: Open the list of prppiler products the denktop cotegory the
Figure 1: GUI-Rise agent framework overview. It introduces three-subtask framework that integrates structured reasoning, action prediction, and history summarization. At each step, the agent performs structured reasoning (progress estimation and decision analysis), predicts the next GUI action, and updates a compact history summary for the next iteration.
g 19 23 improves in-distribution accuracy, but often overfits to static instruction-action pairs and
fails generalize. A deeper obstacle is the need for long-horizon, sequential reasoning (24] 49
27]: effective decisions must depend on what the agent has already done and how the interface has
evolved. Existing systems encode history either (i) as action sequences alone 8 which omit
visual state and hinder progress estimation, or (ii) as full-screen screenshots [99] 23 50 which are computationally expensive and force severely truncated context windows. Consequently, unlike
humans who effortlessly integrate past observations and estimate task progress [14] today's S GUI
agents still struggle with coherent, long-term reasoning.
To address these limitations, we propose a reasoning-enhance GUI navigation framework that couples concise history summarization with explicit, structured reasoning. Based this framework, we train a GUI agent, GUI-Rise. GUT-Rise operates in a step-by-step cycle: it compresses the full interaction trace into a short textual summary. performs structured srain-of-thhughhe reasoning, and predicts the next GUI action, as illustrated in Figure] | This design imitates the way humans navigate interfaces, allowing the agent to maintain coherence and goal awareness over long action sequences. To achieve this capability, we introduce a reinforcement learning baanid training strategy where the agent interacts with a simulated environment develop adaptive, structured reasoning skills tailored to GUI tasks.
Specifically, GUI-Rise operates through three core subtasks at each interaction step: structured reasoning, action prediction, and history summarization. The agent first analyzes the current screen observation alongside previous interactions assess task progress, then predicts the optimal next action based on this analysis, and finally updates its interaction history to maintain an evolving context for future decisions. To enable this structured reasoning capability, we develop a two-stage training paradigm. The first stage bootstraps the model on a small, synthetically labeled dataset to establish basic reasoning and summarization skills, while the second stage adopts reinforcement learning in a simulated GUI environment to refine task-specifit reasoning strategies through interaction. Importantly, we employ Group Relative Policy Optimization (GRPO) with three complementary reward functions: 1) an action accuracy reward that evaluates prediction correctness, 2) a format reward that enforces structured reasoning through semantic tagging of components, and 3) a history summary reward that assesses the summary's 's quality for future decisions. This triad of rewards ensures the agent develops both precise action selection and robust reasoning capabilities while maintaining coherent, context-awas aware behavior throughout extended interactions.

---

We conduct extensive experiments on benchmarks including Mind2Web (9, AITW (30 and Mini- Wob [35], evaluating out-of-domain generalization in both offline and online settings. Our findings reveal that our proposed method yields substantial improvements in out-of-domain generalization under the same training data. In summary, our contributions are three-fold,
We propose a caasoning-enhancee agent framework that formally structures the reasoning and execution pipeline for GUI-based tasks, introducing novel history representation mechanism to improve decision accuracy.
We develop comprehensive reward combining action incentives with a specialized history summary reward, jointly optimizing for coherent reasoning and prog-eroeeessive outputs.
Extensive experiments demonstrate our model's ' state-of off-the-rr performance across multiple benchmarks, with particularly strong results in out-of -i-oof-dmminn scenarios, highlighting its superior generalization capabilities for real-world GUI navigation tasks.
2 Related Work
GUI Agent. In recent years, the rapid advancement of generative artificial intelligence
12
32
has made GUI agents prominent research focus. Early studies [15] 27
46,
49
52 primarily
focused the design of application frameworks, relying heavily the reasoning capabilities
of close-source large language models (LLMs) ( which often necessitated significant human
intervention during practical deployment. Subsequent researches [9 44 began to explore the
use of open-source LLMs [38] 11 for GUI navigation, typically by collecting interaction data and training task-specific agent models. Some approaches (39 22 further leveraged HTML document corpora, employing LLMs to perform structural parsing and long-context reasoning. thereby enabling
autonomous interaction with GUI environments. Recent works such as UI-TARs (29 leverage large-scale reasoning data, including GUI screenshots. to boost performance through task-specific fine-tuning, though this also poses challenges for generalizing reasoning to unseen domains. To address this challenge, our work introduces a structured reasoning Chain-of- Thouhht (CoT) that mimics human-lik, decision-making making through interpretable steps such progress estimation and decision reasoning, and further enhance generalization via reinforcement learning with tailored rewards.
Memory in GUI Agent. The memory mechanism [4] of intelligent agents is crucial in GUI tasks, storing necessary historical information. It's mainly divided into short-term memory (STM) and long-ferm memory (LTM) based on storage duration and functionality. LTM (36] 16 47 41 is persistent knowledge storage. It records global information like trajectories, environmental states, and user preferences from historical interactions, highlighting the importance of LTM in complex scenarios. STM focuses on the agent's 's immediate context for logical consistency during task execution. Early works like the action-oni oonly series [46] 16
g
9 used action sequence recording for task state 40 23 combined "actions and screenshots" within
perception. Later, studies such as Show UI [19 a fixed-length window for better history modeling. More recently, UI-Hawk (50) reduces visual history images to one-quarter of their original size and adopts visual token compression ratio of 16, improving computational efficiency and inference performance. However, tow-resonution images may lose fine-graing fine-griieed details, such as the states of small interactive buttons, compromising the integrity of historical information. Consequently, both forms of short-term memory (STM) still suffer from historical information loss, leading to inaceurate meaisi.n-making. Our approach integrates history summarization into agent reasoning, enabling efficient execution history representation without fixed window limits or external semantic processing, and improving memory efficiency and adaptability to complex tasks.
29
Reinforcement Learning for LLMs/MLLMS. Reinforcement Learning (RL) has emerged powerful tool for enhancing the capabilities of [LMs(T] and MLLMs(3). While Proximal Policy Optimization (PPO) (31) is widely adopted, its reliance on value networks incurs high computational cost and instability. To overcome this, GRPO (33] replaces value estimation with inter-group compar- isons, enabling more efficient and stable training, and has shown success in code generation (20 43
and mathematical reasoning 12 10]. Recent works (5 34 53) have further extended this algorithm multimodal tasks such as visual counting and grounding. Among them, UI-RI (25) applies GRPO to GUI navigation but is limited to single-step tasks. In contrast, explore the more complex
3

---

multi-step GUI navigation setting. introducing history summarization objective trained with RL to support long- monghhoizoon reasoning and better generalization.
Method
We present a prasoning-enhancee framework for GUI navigation that formalizes the interaction process through three core components: (1) structured reasoning, (2) action prediction, and (3) history summarization, Our MILLM-based agent, GUT-Rise, processes multimodal inputs-includdnn the current screen observation, user instruction, and interaction history-to generate coherent outputs comprising a CoT analysis, executable action, and updated context summary (Figure D The remainder of this section details our approach: Section 3.1 formalizes the GUI navigation problem, Section 3.2 presents the agent's architecture, and Section 3.3 elaborates on each subtask's ' design and integration.
3.1 Task Definition
We define general GUI navigation agent that executes natural language instructions by performing a sequence of GUI-level actions. At each time step t, the agent æ, parameterized by 0, observes the environment state s [o,, h,-1], and selects action a, based its policy:
= (u,
Here, O, E D denotes the visual observation of the GUI (e.g., a screenshot of resolution w x h), and h; E D represents the interaction history up to step 1-1. The agent outputs atomic GUI-level actions at,, such as clicking button entering text. A formal definition of the action space is provided Section 3.3 After executing at, the environment transitions to 8141, and agent receives scalar reward Tt:
(S,,, Ott, St+1)
The reward function R encodes both task task-specific progress and adherence to behavioral constraints, guiding the agent toward successful task completion.
3.2 Agent Architecture
GUI-Rise jointly processes visual inputs (GUI screenshots) and textual inputs (user instructions and interaction history). We adopt multimodal large language model (MLLM) with an encoder-decoder architecture to enable cision-conditioned reasoning and text generation. our implementation, we use the Qwen-VL series [88 3). though the framework remains compatible with other MLLMs of similar structure. We denote the model as Fo. which encodes the inputs and generates outputs via auto regresseeee decoding. At time step t, the model receives the user instruction u, the current screen o,, and the interaction history h
[c,, h au] V, Felu, O,, h;-1)
The screen o, is embedded as visual features, while u and h;-1 are embedded text. These are fused in the decoder, which generates a sequence V: D& of length L auto-regressively, ~ indicates that the output v, is generated from the policy model Fg. The output is parsed into: (1) CoT reasoning trace C₁ (2) an updated interaction history h, and (3) the predicted GUI actions at.
3.3 Agent Reasoning Framework
We now present our three core subtasks the agent performs at each interaction step. First, the agent engages in structured reasoning, where it processes the current input and previous history to form coherent understanding of the task. Next, predicts the appropriate action based on this understanding. Finally, the agent updates the history summary, integrating the new information for future decision-making. These subtasks ensure effective integration of historical context and support consistent, informed decisions throughout the interaction.

---

Structured Reasoning Subtask. To improve both decision quality and interpretability, we introduce a structured reasoning subtask that mirrors human cognitive strategies: first assessing task progress, then determining the next action. The output of this subtask Ct. is divided into two components: Progress Estimation and Decision Reasoning. As shown in Figure]] [ the agent estimates navigation progress by analyzing or and h, basing its reasoning on the current observation and prior actions. Based on this, the agent determines the next action, guided by u and prior decisions, ensuring alignment with the task objectives. This step-wise reasoning structure promotes both coherent mecision-making and interpretable intermediate reasoning.
Action Prediction Subtask. The action prediction subtask requires the model to predict a structured, parseable textual action based on reasoning outcomes. This textual action subsequently undergoes a parsing process to be converted into executable data format recognizable by the agent's 's execution module. Specifically, the predicted textual action a, is parsed into a structured, executable form at axaalue C), where: at E V is the action type (e.g.. "CLICK", "INPUT", value E V is the associated textual value (e-g., "Search Bar", "Date"); C (x (y E "ENTER") ); a R22 denotes the screen coordinates of the target element. Here, V and V are finite sets of valid action types and values, respectively. The finiteness of these two sets ensures the controllability of the agent's 's action space and the validity of each generated action.
Or = M(aa)) = (ar sype
History Summary Subtask. To sustain long-term coherence during task execution, the agent maintains a concise yet information-demt representation of prior actions and GUI states. At each step, it summarizes or, h and u into concise textual memory (Figure Dp. This updated hidden state h, allows the agent to transcend the constraints of individual step details, continuously track task progress over extended time horizons, and thereby formulate more targeted, context-aligned decisions for future steps. In comparison to approaches relying on raw visual data [19]. those utilizing paatuee-ppcce-compessseee historical screenshots [50] unprocessed action-only histories (8), these semantic summaries offer two key advantages: they provide clearer hierarchical abstraction and establish tighter grounding real-world task scenarios. Together, these strengths translate into more effective multi-step reasoning performance for the agent in complex task environments.
4 Training
To avoid ineffective supervision and local optima caused by sparse or low initial rewards, we design a two-stage training strategy for GUI-Rise. The first stage cold-start training employs supervised learning on pseudo-labeled data to establish solid initial policy and meaningful reward signals. The second stage reinforcement learning refines the model with reinforcement learning, enhancing adaptability. This approach ensures stable initial training and effective fine-tuning, resulting in improved performance and generalization. An overview is shown in Figure 2
4.1 Cold-start Training
In the cold-start stage, we aim to initialize the agent with essential skills in history summarization and structured reasoning. To initialize training, we use a stronger MLLM (e.g., GPT-4o-mini (52) to generate pseudo labels, consisting
textual summary h' E DJ and a structured CoT
Pseudo-label Generation. We employ a retrospective labeling strategy 29 to generate the agent's intermediate reasoning and history summary based known correct actions, creating accurate and goal-consistent pscudo-labels for supervised learning. For each trajectory, pseudo-labels are generated step-by-step using GPT-4o-mini. At the initial step (t = 0), we input observation Og, user instruction u, and ground-truth action ag"' producing a history summary húg and structured CoT ch as pseudo-labels. For subsequent steps (1 > 0), we include the previous step's 's summary h/- current
observation o₁, instruction and action ag' generating pscudo-labels hy and c/. This sequential process accumulates history through generated summaries (labeling details and prompts are in the supplementary material B.
Supervised Fine-Tuning
During cold-start training, the ground-truth action label at from trajectory annotations supervises final action prediction. These components are serialized into target token sequence: y T((c', a h']) E Itd Given the user instruction u, current observation or, and previous history summary h the agent is trained to autoregressively generate this sequence using
5

---

Observation
Instruction
History Summary
Action
Output:
Tags?
Input
Input
<Progress Estimation
Policy Model
Policy Model
4r.o
A
<Decision Reasoning >...
R!
Output
Output
<History Summaryp...
Gec
Ac.o
Ac.G
Action Validation
Action Reward
1
Loss
G Groups
ac+1.1
GRPO Optimizer
Policy Model
History Summary
Pseudo
Reward g
Labels
GPT
Actions
Future Simulation:::: Step to Rollout
Cold Start
Reinforcement Learning
Figure 2: Overview of the GUI-Rise training pipeline. The training consists of two stages: (I) supervised learning with pseudo-labeled labeled summaries and ground truth action trajectories to initialize reasoning, and (2) reinforcement learning with rule-based and model- based rewards to improve decision-making and generalization.
a standard token-level cross-entropy loss:
Lee
Σ log Poly, Yyg; h--1)
(4)
where Po(ys 1) denotes the probability assigned by the model's decoder over the vocabulary at position
4.2 Reinforcement Learning
After cold-start training, utilize RL further agent's policy. Specifically, we adopt the GRPO (3 algorithm, which enables reward optimization without human annotations by leveraging predefined task-specific reward functions. As illustrated Figure 2 guide the agent toward producing structured reasoning and correct decisions, we design three complementary reward functions: (1) format reward function Rí, enforcing structural correctness outputs; (2) action reward function R", providing binary feedback on action type, format, and location; (3) a history summary reward function Ra, measuring how well the generated summary supports accurate future actions. Together, these rewards guide the model to produce coherent reasoning traces and effective summaries for multi-step decision making.
To encourage structured output, we introduce a format reward function R/ based on predefined XML-style tags. To specifically promote structured CoT reasoning under this reward design, we separately assign different tags to the progress estimation and decision reasoning during the inference process. This tag-level decomposition guides the agent generate reasoning modular and logically ordered manner, thereby improving interpretability and promoting step- sigp-wie decision making. The expected output format consists of four tagged components, appearing the following fixed order: "<Progress sitiiaattonn../PPorrre Estimation>", "<Decision sissoniing..</DDccsiioi Reasoning>", " E<Actioos...</Actionns and "<Memory Summary>... munmarys..</Memorr Summarys". We define function, CheckTägs(yr.... that returns true if the output v strictly adheres to the prescribed tag sequence. We set the format reward rf,; I to if CheckTags(v... returns true, and 0 otherwise:
Format Reward.
{1 if CheckTags(vv.d)) true else
T = R² (vr,)) =
(5)

---

Action Reward. To assess the correctness and executability of predicted actions, propose composite action reward function R". Concretely in the function, the correctness and consistency of (4, is then assessed based on three criteria: (i) conformity to the required structural format, type (ii) consistency between the predicted action type at, and the ground-truth type, and (iii) spatial accuracy, determined by whether the predicted coordinates Ct,; fall within the bounding box b (ar pax .os + 1/2 E IR of the target UI element. Based on these criteria, the reward function is defined as (We add more action reward details in the supplementary materials B.2):
R. (axt,;; 02 be)
2,5
History Summary Reward. The purpose of this reward is to assess the quality of the history summary by evaluating whether it contributes to correct future actions. The design avoids training separate evaluation model by leveraging
model's own prediction behavior. Specifically, the i-th action o, is incorrect (i.e., r2.; 0), the history summary is considered invalid and receives no reward. Otherwise, the model performs additional & rollouts using h, as input to predict the next output Ve+1 Fu,u, oc+1, ir, with gradient backpropagation. The history summarization reward then computed by applying R to the predicted action å,+1 extracted from V441. This process can be defined as:
0, 1 1
if, = else
rhe (br,)
(7)
Rd (or0111 0441, bet1),
This structure rewards history summaries that enable accurate future behavior while directly boosting task success rates. By linking past summaries' value to subsequent action effectiveness, empowers the model to proactively learn and prioritize historically syntheses with tangible task relevance. Over time, this incentive drives the model to independently identify key contextual cues in historical data that improve outcomes, turning history summarization into self-improving loop for better task success rate.
Our total reward integrates assessments of both output format correctness, action accuracy and history quality, it can be formated as:
T1 r(. +
(8)
where X" and X are the weights for the action and history rewards, respectively. Based on this composite reward, we compute the advantage At,; via group-level normalization as below,
A T1.s - mean (fr
std std
and optimize the GRPO objective via policy gradient descent (More details, please refer to the supplementary materials B.3).
5 Experiments
We report comprehensive experimental results across various settings. Section 5.1 outlines the setup; Section 5.2 compares our method with state-of-the-an baselines in dom-o-- scenarios, while in-domain results are presented in Section 5.3 Section 5.44/oon55//eelluuttt zero-shot performance an online environment. Sections 5.5 cover the ablation study and an analysis of the history representation. Further experiments includinn sensitivity analysis, cold-start scenarios, impact by history representation and case studies are provided in the supplementary material C.
5.1 Experimental Setup
Datasets. Our experiments use three offline GUI navigation benchmarks and one online benchmark.
(i) Mind2Web (Web) (9), comprising 2,350 unique episodes across websites and an action space with
three action types. (ii) AITW (Mobile) (30), featuring an Android smartphone environment with an
action space of actions. (iii) GUTAct [6] benchmark that contains both web and mobile GUI
navigation tasks. (iv) Min (Online) 35 an online interactive environment with two action types.
7

---

Method
Base Model
Cross-Task
Cross- Website
Cross-Domaln
Step SR
Step
Step
Standard Serting
MindAct
55.1
75.7
52.0
42.0
65.2
38.9
42.1
66.5
39.6
GPT-4
41.6
36.2
35.8
51.1
30.1
37.1
26.4
OmnilPaseer
42.4
87.6
39.4
41.0
84.8
36.5
45.5
85.7
42.0
CogAgent
13
22.4
53.0
17.6
18.4
42.4
13.4
20.6
42.0
15.5
Qwen- VL-7B 2
Qwen-VL
15.9
86.7
13.3
13.2
83.5
9.2
14.1
84.3
12.0
SocCilek(S
28.3
25.5
21.4
80.6
16.4
23.2
84.8
20.8
ShowU1-2B
Qwen2-VL-22
Owen2-
37.7
39.9
88.6
86.4
33.2
37.2
36.0
41.6
83,5
27.6
35.1
39.4
36.3
86.8
81.8
30.7
35.2
VL-2B
GUI-Rise
45.5
84.8
38.8
35.4
46.5
39.7
Qwen2.5-VL-31
Qwen22.5
43.5
48.7
GUI-Rise
VL-3B
51.9
88.4 4
46.2
51.7
85.6
44.7
$3.0
87.0
47.6
Zero-Shot Setting
Qwen2-VL.
Qwen2-
14.3
22.7
18.1
ShowU1 2B
19
21.4
85.2
18.6
21.9
81.9
16.8
24.4
83.9
21.4
GUI-Rise
27.0
85.2
24.2
25.5
82.0
21.1
33.4
84.0
29.7
Qwen2.5-VI. 3L
Qwen2.5
22.2
87.1
20.1
22.5
84.3
17.0
24.8
84.3
22.8
GUI-Rise
VL-3B
29.4
87.4
24.8
26.1
85.7
22.4
35.1
84.8
30.1
Table 1: Out-of-domain evaluation results the Mind2Web benchmark. The table reports perfor- mance under two settings: (I) the standard setting, where models are trained on the Mind2Web training and evaluated on its test set; and (2) the zero-shot setting, where models trained the GUTAct training set and evaluated on the Mind2Web test set.
is used complement the offline benchmarks and evaluate performance real-time interaction. Further details for each benchmark are provided in the supplementary material.
Settings. To assess the generalization capability of GUI-Rise, evaluate it under both out-of- domain and in-domain scenarios [3] For out-of-domain evaluation, we conduct generalization performance testing on both mobile and web platforms. In addition, conduct zero-shot evaluation on the online platform MiniWob to evaluate the model's capability in handling dynamic environments. For in-domain evaluation, we train the model the training set of AITW and test the respective test set.
Evaluation Metrics. Mind2Web is evaluated using clement accuracy (Ele.Ace), operation Fl (Op.F1), and step success rate (Step SR) across three verified test pplits-test-taske test-website, and (est-domain-covett
variations in tasks, websites, and domains. On AITW, action accuracy (30) is used to measure the per-step success rate. For MiniWob, the success rate averaged over 50 random seeds per task and then aggregated across tasks
5.2 Out-of-Domain Evaluation
Mind2Weh. Given the inherently out-of-domain nature of the Mind2Weh test set, we adopt two generalization evaluation settings from ShowUl [19]: standard and zero-shot. In the standard setting, the model is trained on the Mind2Web training set and evaluated on its OOD test set. In the zero-shot setting, GUI-Rise, pretrained on the GUIAct dataset, is directly evaluated the Mind2Web OOD test set. Results are detailed in Table
In the standard setting, GUI-Rise with the Qwen2-VL. VL-2B 38 backbone achieves the highest step success rates (Step SR) across all splits, notably reaching 39.7 points in the cross-domain split. With the stronger Qwen2.5-VL VL-3B (3), GUI-Rise improves step SR by 1.2 and 3.5 points in the cross- website and cross-domain settings, respectively. the zero-shot setting, GUT-Rise significantly outperforms baselines, achieving 38.79 7877% performance improvement over the previous state-of-the-art ShowUl in the cross-domain split. These results demonstrate GUI-Rise's superior generalization in GUI navigation tasks.
To evaluate generalization on the mobile platform, we evaluate GUI-Rise on the AITW benchmark using the same zero-shot setting as in Mind2Web. As shown in Table GUI-Rise achieves substantial improvements over the previous state-of-the-art method ShowUl across all metrics, with relative gain of 50.7% in the overall performance metric. Notably, for the more complex WebShop
ATTW.

---

Method
Base Model
General
Install
G.Apps
Single
WehShop
Overall
In-Domain Setting
ChatGPT-CoTT
5.9
4.4
10.5
8.4
PaI.M2-CoT
39.6
OmniParser
48.3
57.8
51.6
77.4
52.9
57.7
SeeClick
Qwen-
54.0
66.4
54.9
63.5
57.6
59.3
VI.7B
Qwen2-VL-22 38
61.4
71.8
62.6
73.7
66.7
67.2
Qwen2-
ShowUl-2B 19
63.9
72.5
69.7
77.5
66.6
70.0
VL-2B
GUI-Rise
64.4
73,9
69.7
78.2
68.2
71.1
Owen2.5-VLL-3B 3
Qwen2.5- VL-3B
67.3
75.2
72.3
79.1
69.0
72.5
GUI-Rise
68.4
76.8
73.1
80.0
69.9
73.7
Zero-Shor Setfing
Qwen2-VL: 2L-22VL-2BB 38
31.0
46.9
40.2
19.4
36.5
34.7
Qwen2- VL-2B
ShowUU-2B 19
32.1
47.7
42.0
20.1
37.4
35.9
GUI-Rise
54.3
57.5
50.3
55.2
52.9
54.1
Qwen2.5 3 GUI-Rise
Qwen2.5-
35.5
43.1
41.7
35.0
39.3
38.9
VL-3B
56.4
59.0
52.3
59.7
52.7
56.0
Table 2: Evaluation results on the AITW benchmark. The table reports performance under two settings: (1) the in-domain setting, where models are trained on the AITW training set and evaluated on its test set; and (2) the zero-shot setting, where models are trained on the GUTAct training set and evaluated on the AITW test set.
Method
MiniWob(7S)
MiniWob(FT)
Android World
OSWorld'
SeeClick 8
19.5
Qwen2-Ve V 2B
38
20.8
66.8
0.0
ShowUl UT-2B
19
27.1
71.5
7.0
Infigui-2B [2T]
9.0
UT- UI-Tars-2B 25
6.5
GUI-Rise-2B
30.6
72.8
10.4
8.7
Table 3: Success rate (%) on online benchmarks. MiniWob(ZS) follows the zero-shot setting and MiniWob(FT) follows the fine-tuning setting used in ShowUl [19 and means results evaluated on the chrome-split of OSWorld.
task, GUI-Rise achieves +15.5 points improvement. We attribute this gain to its structured reasoning design, which enhances the model's understanding of complex environments and tasks in shopping interfaces.
5.3 In-Domain Evaluation
To evaluate the in-domain performance of GUI-Rise, conduct experiments the AITW bench- mark under an in-domain setting. As shown in T using the Qwen2-VL-22 backbone, GUI-Rise achieves an overall success rate of 71.1%, outperforming all prior baselines. Compared with the ShowUI-2B, GUI-Rise achieves consistent gains across all task categories, including a 1.6-point improvement in the complex shopping scenario (68.2% vs. 66.6%). With Qwen2.5. VL--3. GUI-Rise further exceeds baselines across all metrics, indicating that our two sub-tasks enhance execution stability and robustness multi-step interaction tasks.
5.4 Online Evaluation
To validate generalization and efficacy of our approach, we conduct comprehensive evaluations across suite of diverse GUI navigation benchmarks. Our model, GUI-Rise, consistently demon- strates SOTA performance (Table In the zero-shot scenario on the MiniWob benchmark, GUI-Rise achieves score of 30.6, showcasing its generalizability. Our GUI-Rise-2D model reaches 72.8

---

TST
SCoT HS
HSR
General
Install
G.Apps
Single
WebShop.
Overall
X
61.4
71.8
62.6
73.7
66.7
67.2
V
X
X
61.4
70.6
63.0
73.2
62.0
66.0
X
V
X
38.0
48.4
46.9
36.7
42.9
42.6
x
63.2
73.4
69.5
76.7
66.0
69.8
V
x
64.1
73.2
69.3
78.0
67.9
70.7
V
64.4
73.9
69.7
78.2
68.2
71.1
Table 4: Ablation study of GUI-Rise on the AITW benchmark. "TST" denotes the of two-stage training with RL; "SCoT" indicates the incorporation of structured Chain-of-Thoupe reasoning; "HS" means using history summary as input: "HSR" refers to applying the history summary reward during RL.
in the fine-tuned scenario, confirming the effectiveness our approach. More importantly, to test model's capabilities in long-horizon, multi-step scenarios that mimic real- real-world complexity, evaluated on the challenging Android World and OSWorld benchmarks. As these are online envi- romments, they feature significantly more dynamic changes, demanding high degree of execution context consistency. Agents must be able to summarize historical interactions and accurately under- stand its current progress within task. This is precisely the challenge our history summarization design is engineered to address. Our model outperforms the prior art on both the AndroidWorld and OSWorld benchmarks, demonstrating the efficacy of our design. By effectively compressing and integrating historical context, it empowers the model to make robust decisions in complex, multi-step tasks.
5.5
Ablation Study
To understand the contribution of key components in our model, we conduct an ablation study focusing on Stage Training (TST) which measn with/without RL, Structured CoT (SCoT), History Summary (HS) and History Summary Reward (HSR). These components enhancing the model's ability to execute tasks effectively, as shown in Table Using only two-stage training (SFT followed by RL) yields limited gains (Row 2), implying that small models may struggle with exploration or overfit easily. Training action with structured CoT reasoning via SFT alone (Row 3) causes drop in success rate, indicating the challenge of learning structured reasoning using SFT. Applying RL to structured reasoning (Row 4) substantially improves performance, confirming the benefit of step-by-step due to the lack of reasoning. Adding historical summaries (Row 5) brings minor gains, likely summaries may mislead the policy and hurt learning. To address this, introduce history summary reward (Row 6) reinforce semantic consistency. This yields the best overall results, showing that guiding the model toward coherent, goal-directed reasoning enhances policy learning.
critical for
Conclusion
We introduce reasoning-enhanced framework comprising three core sub-tasks. These sub-tasks enable the GUI-Rise model with robust contextual reasoning, action coherence, and historical integration capabilities. GUI-Rise achieves state-of-the-at success rates in GUI navigation tasks, excelling in out-of-domain (OOD) scenarios. These capabilities significantly enhance stability and generalization in complex multi-step interactions, highlighting the framework's potential for building high-performance, generalizable GUI agents.
Limitations. A limitation of our work is that while the model is evaluated in online environments, it trained entirely offlin.. This prevents the model from adapting to new scenarios or learning from its interactions in real-time. Future work could address this by exploring online learning methods, particularly by enabling the model to reflect on its successes and failures learn directly from live interactions.
10

---

7 Acknowledgments
This work was supported by NSFC 62350610269, Shanghai Frontiers Science Center of Human- centered Artificial Intelligence, HPC Platform of Shanghai Tech University, MoE Key Lab of Intelli- gent Perception and Human-Machine Collaboration (ShanghaiTech University).
References
[1] Josh Achiam al. "Gpt-4 technical report". In: arXiv preprint urXiv:2303.08721 (2023).
[2] Jinze Bai al. Qwen-VL. A Versatile Vision-Language Model for Understanding, Loculization, Text Reading, and Beyond. 2023. arXiv: 2308. 12966
[cs URI. https: /ttbis://arxiv.oom org/
abs abs/2308..0 12966
[3] Shuai Bai et al. "Qwen2. 5-vl technical report". In: arXiv preprint arXiv:2502.139) (2025).
[4] Davide Caffagni et al. "The Revolution of Multimodal Large Language Models: A Survey".
In: ACL (Findings). 2024.
[5]
Liang Chen et al. Vinci. RI-v: Reinforcing super generalization ability in vision-language models with less than $3.
[6]
Wentong Chen al. "Guicourse: From general vision language models versatile gui agents".
In: arXiv preprint arXiv:2406.1 1317 (2024).
[7] Zhe Chen et al. "Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024, pp. 24185-24199
[8] Kanzhi Cheng al. "Seeclick: Harnessing gui grounding for advanced visual gui agents". In: arXiv preprint arXiv:2401..0093 (2024).
[9] Xiang Deng et al. "Mind2web: Towards generalist agent for the web". In: Advances in Neural Information Processing Systems 36 (2023), pp. 28091-281 14.
[10] Evgenii Evstafev. "Token-Hungry, Yet Precise: DeepSeek Highlights the Need for Multi- Step Reasoning Over Speed in MATH". In: arXiv preprint arXiv:2501.18577 (2025).
[11] Aaron Grattafiori et al. "The Ilama 3 herd of models". In: arXiv preprint arXi::2407.21783 (2024).
[12] Daya Guo et al. "Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning". arXiv preprint arXiv:2501. 12948 (2025).
[13] Wenyi Hong et al. CogAgent: A Visual Language Model for GUI Agents. 2024, arXiv: 2312
08914 [cs cv] URL: https: / arxiv. org/ abs/ 2312. 08914
[14] Xueyu Hu et Os agents: A survey on mllm-based agents for general computing devices use. 2024.
[15] Jing Yu Koh et al. "Tree search for language model agents". In: arXiv preprint urXiv:2407.01477 (2024).
[16] Sunjae Lee et al. "Mobilegpt: Augmenting Ilm with human-like app memory for mobile task automation". In: Proceedings of the 30th Annual International Conference Mobile Computing and Networking. 2024, pp. 11 19-1133.
[17] Bo Li et al. "Llava-onevision: Easy visual task transfer". In: arXiv preprint arXiv:2408.03326 (2024).
[18] Zijing Liang et al. "A Survey of Multimodel Large Language Models". In: Proceedings of the 3rd International Conference Computer, Artificial Intelligence and Control Engineering. 2024, pp. 4054409
[19] Kevin Qinghong Lin et al. "Showui: One vision-language-action model for gui visual agent". In: arXiv preprint 24X14:2411.17465 (2024).
[20] Jiawei Liu and Lingming Zhang. "Code-rl: Reproducing for code with reliable rewards".
In: arXiv preprint arXiv:2503. /8470 (2025).
[21] Yuhang Liu et al. "Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection". In: arXiv preprint arXiv:2501 1.4555 (2025).
[22] Zhiwei Liu et al. "Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents".
In: arXiv preprint 2r8iv:2308.05500 05960 (2023).

---

[23] Quanfeng Lu et al. "Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices". In: arXiv preprint arXiv:2406.060 08451 (2024).
[24] Yadong Lu et al. "Omniparser for pure vision based gui agent". In: arXiv preprint arXiv:2408.0022) (2024).
[25] Zhengxi Lu et al. "Ui-rl: Enhancing action prediction of gui agents by reinforcement learning.. In: arXiv preprint arXiv:2503.21620 (2025).
[26] Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. "Coco-agent: A comprehensive cognitive milm
agent for smartphone gui automation". In: arXiv preprini arXiv:2402. 1941 (2024).
[27]
Runliang Niu et al. "Screenagent: A vision language model-driven computer control agent".
In: arXiv preprint arXiv:2402.07945 (2024).
[28]
OpenAl et al. GPT-4 Technical Report. 2024. arviv: 2303. 08774 [CS
URL: https
//arxiv..o org/ abs/ 08774
[29] Yujia Qin et al. "UI-TARS: Pioneering Automated GUI Interaction with Native Agents". In: arXiv preprint vrXiv:2501.12222 (2025).
[30] Christopher Rawles et al. "Androidinthewild: A large-scale dataset for android device control".
In: Advances in Neural Information Processing Systems 36 (2023), pp. 19708-59722)
[31] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprint
arXiv:/707. 17Xx:/7777.0637 (2017).
[32] Sandeep Singh Sengar et al. "Generative artificial intelligence: systematic review and applications". In: Multimedia Tools and Applications (2024), pp. 1-40.
[33] Zhihong Shao et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models". In: arXiv preprint arXiv:2402.03300 (2024).
[34]
Haozhan Shen et al. "VIm-rl: A stable and generalizable rl rl-style large vision-language model".
In: arXiv preprint arXiiv2504.07611 (2025).
[35]
Tianlin Shi et al. "World of bits: An open-domain platform for web-based agents". In: Interna-
tional Conference on Machine Learning. PMLR. 2017, pp. 3135-3144.
[36] Weihao Tan et al. "Towards general computer control: A multimodal agent for red dead redemption ii as a case study". In: ICLR 2024 Workshop on Large Language Model (LLM)
Agents. 2024.
[37] Junyang Wang al. "Mobile-agent: Autonomous multi-modal mobile device agent with visual perception". In: arXiv preprint arXiv:2401. (2024).
[38] Peng Wang et al. "Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution", In: arXiv preprint arXiv:2409.12 12191 (2024).
[39] Shuai Wang et al. "Gui agents with foundation models: A comprehensive survey". In: arXiv preprini arXiv:2441 1,04990 (2024).
[40] Xiaoqiang Wang and Bang Liu. "Oscar: Operating system control via state-aware reasoning and re-planning". In: arXiv preprint arXiv:2410. /8963 (2024).
[41] Zora Zhiruo Wang et al. "Agent workflow memory". In: arXiv preprint arXiv:2409.07429 (2024).
[42] Zhiyong Wu et al. "Os-atlas: A foundation action model for generalist gui agents". ". In: arXiv preprint arXiv:2410.2322 (2024).
[43] Tian Xie al. "Logic-rl: Unleashing IIm reasoning with rule-based reinforcement learning".
In: arXiv preprint arXiv:2502. 14768 (2025).
[44] Yiheng Xu et al. "Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction". In: arXiv preprint arXiv:2412.04454 (2024).
[45] Qiying Yu et al. "Dapo: An open-source Ilm reinforcement learning system at scale". In: arXiv preprint arXiv:2503. /4476 (2025).
[46] Chi Zhang al. " "Appagent: Multimodal agents as smartphone users". In: arXiv preprint
11X1v:2312.13771 (2023).
[47] Chi Zhang et al. "Appagent: Multimodal agents smartphone users". In: Proceedings of the
2025 CHI Conference on Human Factors in Computing Systems. 2025, pp. 1-20.
[48]
Duzhen Zhang et al. "MM-LLMs: Recent Advances in MultiModal Large Language Models". In: Findings of the Association for Computational Linguistics ACI. 2024. 2024, pp. 12401-
12430.
12

---

[49] Jiwen Zhang et al. "Android in the zoo: preprint arXiv:2403.02715 (2024).
for gui agents".. In: arXiv
[50] Jiwen Zhang et al. "UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents".
In: Preprints (Aug. 2024). DO1: 20944 preprritss022008 2137 v1 URL: https
//doi. org/ 10.
2137 .v
[51] Zhuosheng Zhang and Aston Zhang. "You only look at screens: Multimodal chain-of-action agents". In: arXiv preprint arXiv:2309. 111123399./1366 1436 (2023).
[52] Boyuan Zheng et al. "Gpt-4v (ision) is generalist web agent, if grounded", In: arXiv preprint arXiv:2401.01614 (2024).
[53] Hengguang Zhou et al. "RI-Zero's" Aha Moment" in Visual Reasoning on 2B Non-SFT Model". In: arXiv preprint arXiv:2503.05132 (2025).
A Datasets Details
A.1 Mind2Web
Mind2Web is pioncering dataset designed to develop and evaluate generalist web agents capable of performing complex tasks on any website using natural language instructions. features over 2,000 tasks across 137 websites and 31 domains, with 7,775 training actions, offering a diverse and realistic environment for training. Unlike simulated datasets, Mind2Web uses real-world websites, providing rich data like user interaction traces and webpage snapshots. A key strength lies in its three distinct test splits-crose cross-task, cross-wes websste, and cross-domain designed to rigorously evaluate generalization performance. The dataset's action space includes three core operations: CLICK, TYPE, and SELECT, capturing essential user interactions for navigating modern web complexities.
A.2 AITW
AITW is a comprehensive Android smartphone dataset featuring 30K instructions and 715K tra- jectories, collected using the Android Emulator: We follow the experimental setup of ShowUT[[9) dividing the data into five domains: Google Apps, Install, Web Shopping. General, and Single. The action space includes 12 actions: CLICK, TYPE, SELECT, SCROLL UP, SCROLL DOWN, SCROLL LEFT, SCROLL RIGHT, PRESS BACK, PRESS HOME, PRESS ENTER, STATUS TASK COMPLETE, and STATUS TASK IMPOSSIBLE, enabling diverse interaction analysis.
A.3 GUTAct
GUTAct is multi-scenario dataset designed enhance GUI agents' knowledge, covering web and smartphone environments. includes GUT navigation tasks split into three partitions: "web-single," "web-multi," and "smartphone,"" " with unified action space of 11 action types. The web dataset comprises 67K single-step and 5,696 multi-step instructions across 50 domains and 13K websites, while the smartphone dataset includes 9,157 multi-step instructions, totaling 67K training samples. Consistent with Show experimental setup, we pretrain on GUIAct for zero-shot experiments. GUIAct uses unified action space comprising eleven key actions: CLICK, HOVER, TAP, INPUT, SCROLL, SWIPE, SELECT TEXT, COPY, ENTER, SELECT, and ANSWER, enabling agents to interact with GUI systems across web and smartphone scenarios.
A.4 Miniwob
MiniWoB features 2000 open-ended tasks sourced from 137 real web environments. includes dynamic GUI environments, allowing validation of a model's adaptability to dynamic settings. Each task provides high-level instructions and action trajectories, enabling agents to perform low-level keyboard and mouse actions the Internet. The action space includes 2 actions: CLICK and TYPE.
13

---

B Training Details
B.1
Pseudo-Label Generation
We utilize retrospective labeling strategy to generate pseudo-labels for the agent's intermediate reasoning and history summary, ensuring accuracy and alignment with the intended goal using known correct actions. This process creates reliable pseudo-labels for supervised learning, generated step-by-step with GPT-4o-mini. The following are the specific prompts employed in our pseudo-label generation process, which demonstrate the structured input for each step.
_PRONPT_ SUMMPT_INGLE_WP WEB "o"You are an AI accistant designed to similate the model's reasoning process before executing given action in gut navigation task. Diven the instruction, current acreanshat, the previous history summary, the current action to be executed and thought, generate rigorous chain of thought. You must strictly follas thene reasoning steps: Comprehension (2) Decision Reasoning: Strategy Formllation
Progress Eatination:
Progresa Estination
History Sumnary: Update the history sumnary according action you
**# Output format:
<Progrese Estination>
(one or two sentence)
</Progrees Eetination>
<Decision Beasoning
... </Dacision Reasoning>
sentence)
<Bistory Sumaryy>
(one sentence)
</Bisstor Sumnary>
##AExample Input Output
Input:
Task Instruction: Find all eventa taking place in Neu York City during the nonth Septenber. Current Actien: {('action?: CLICK, "value": 'Apply', "position": [0.3, 0.68))
Previous History Sunnary: The first changed the location Nev York, then set the date Septenber 1, and set the and data to Septenber 30.
Output:
<Progress Estimation>
date range Septembar 1-30, the events displayed are still for Narch, indicating the need apply the date filter. </Progress Betination>
user has succeastully
location
<Decision Reasoning?
Clicking the 'Apply' button will confirn the selected date range (Septenbar 1-30) and refresh the avent York City during September.
listings shor only those occurring </Dacision Reasoning>
<Hiatory Summary>
The user changed the location New York, set the date range to September 1-30, and applied the filters to update the event listings.
</Bistory Summary>
#tAInput
Task Instruction: {_TASK}
Current Action: (_ACTION)
Thought: (_TROUGHT)
Previous History Sumnary: (_MMD))
***
B.2 Action Reward Computation
To evaluate both the correctness and executability of predicted actions, we introduce a composite action reward function Ra, This function assesses each predicted action 04 based three criteria:
Whether the output adheres to the required dictionary-style format: {'action":: "ACTION DACTION_TYPE", "value"" "element",, "position"" [x, y]} We define a function, CheckActionF, to check the predicted action. The function first checks the input is dictionary. If not, it returns False. It then verifies that the dictionary contains exactly the three required keys ("action", "value", and "position") ") and no extra keys, the function return true:
Action Structural Format Reward:
af R.th Rd (1 -(0
=
if CheckAction else
true
(10)
Action Type Reward: We evaluate whether the predicted action type o matches the ground-truth type. Specifically, check if the predicted value exactly equals the annotated action type for the
type
14

---

current step. If they match, the prediction is considered correct:
type R (2
(at, Li
type type
{1
type 1,
type
(11)
else
Action Position Reward: We evaluate whether the predicted coordinates Ct, fall within the bounding box b, = (zr v/_ pos .122) E IR' of the target UI element. Specifically, we check if the predicted point lies inside the rectangular region defined by by:
in by
pos
pec
(12)
else
The final action reward ri, is computed as weighted of the three components:
A
type
A
pos 1,
(13)
B.3 Reinforcement Learning Objective
In the second stage of our two-phase training, we adopt GRPO 33 GRPO improves upon traditional Proximal Policy Optimization (PPO) [3 need for separate critic model. time step t, the advantage A,, corresponding to the i-th output v; can be derived from Eq. (9) Then, the overall objective is:
45
surpasses traditional PPO by eliminating the
(
u. Ot, h )(,
bes (0)
(14)
Taa (Urii.)
u. Ot, 14-1, (5,
(0) = =B ou
C
lecal
min (bea)
(lip(b.s,00).1- -,---,,,++,) c))aa) - -BDKL |l Test))
(15)
where (u, Or, h,-1,;)) 04) is a question-answer pair from the data distribution D, is the clipping range of importance sampling ratio. To ensure stable policy updates, GRPO also introduces a KI- divergence regularization term that penalizes deviation from the reference policy distribution, and is a coefficient controlling the strength of the regularization. This formulation helps constrain policy updates, stabilizing training and encouraging consistency with previously learned behaviors.
C Experimental Analysis
C.1 Scalability Evaluation on Larger-Scale VL Models
Previous GUI-Rise evaluations only used small VL models (2B-3B params), limiting real-world relevance larger 7B-sc models are more practical for complex GUI tasks due to stronger reasoning. Thus, we tested @wen2.5-vV 5-VL--B (GUI-optimized 7B VL model) on AITW and Mind2Web under SFT. Tables and o show GUI-Rise-77 7B outperforms the baselines. On AITW, GUI-Rise- 7B boosts overall Step SR by 3.0% with the biggest gain in Install, proving cross-task GUI effectiveness. On Mind2Web, it achieves strong cross-scenario gains Step SR. experiments confirm GUT-Rise's scalability and boosts its practical value for GUI agents.
C.2 Impact of Cold Start
In this section, we analyze the impact of the first stage in our two-stage training framework: cold start pretraining. We compare two setups on the Mind2Web and AITW datasets: (1) Cold Start RL and (2) RI. Only. The initial model Owen2-VVL-2B. We exclude the history summary reward in this experiment, it depends future actions and is unrelated to cold start effects.
15

---

Method
Base Model
Cros-TTask
Cross- Website
Cross-Domain
Ele.Ace
OPF1
Step SR
Ele.Ace
OP.FI
Step SR
Fle.Acc
OPFI
Step SR
In-Domain Setting
Owen2.5-VL-7B
Qwen2.5-
54.5
90.5
50.5
53.5
88.7
46.9
51.1
89.1
46.8
Gui MuiiRRie-7B
VL-78
56.9
90.4
52.3
56.7
88.7
50.7
56.4
90.2
51.4
Table 5: Evaluation results on the Mind2Web benchmark with 7B models.
Method
Base Model
General
Install
G.Apps
Single
WebShop
Overall
In-Domain Setting
Qwen2.5-VL--77
Qwen2.55
68.6
77.4
76.2
80.0
68.5
73.7
Gui Ril-RRiee-BB
VL-7B
70.1
80.2
78.6
81.4
69.9
75.7
Table 6: Evaluation results on the AITW benchmark with 7B models.
Figure 3 Bandd4 the reward curves during training, revealing distinct patterns across datasets. On AITW, the action reward gap is minor first and narrows over time, stabilizing at difference of just 1.5 points. In contrast, on Mind2Web, cold start yields a sharp initial gain in action-type reward, while the action-positioo reward remains negligible without it. In fact, for the RL-only model, the position reward stays near zero throughout training, resulting in vanishing advantage estimates and ineffecttive gradient updates. Format reward improves rapidly on both datasets; within 30 iterations, the model consistently outputs responses in the correct format.
We attribute the differing outcomes to task complexity. Mind2Web's visually rich web interfaces (e.g., higher resolution, dense layouts) present a steeper learning curve than AITW's simpler mobile environments. Without cold start, the model fails receive meaningful reward signals early on, leading to ineffective learning.
Figure 3: GUI-Rise training process on Mind2Web benchmark.
C.3 Impact History Representation
Effective history representation is essential for efficient and accurate GUI navigation. We evaluate five Prrategies-Action--ee
(8) Action-Screenshot [19], ShowUl [1, GUI-Odyssey [23], and our proposed history summary representation to assess their trade-offs between navigation success and input token efficiency. Our experimental results shown in the Figure 3 our method achieves the highest navigation success rate with the lowest token footprint. Compared to vision-heavy approaches like ShowUI, it reduces visual token overhead while improving performance. Relative to GUI-Odyssey, it further shortens input length and enhances success rates. These results highlight the importance optimized history representation for enabling practical, high-performance GUT interaction in MLLMs.
16

---

Figure 4: GUI-Rise training process AITW benchmark.
Irput Tokens
Navigation (Step
1500
Navigation (Step. SR)
1400
33-
36-
Show!ll
12-
1200
31
Action Only
Action
1000
2000
3000
4000
OUi-Odbory
Outs
Gull-alowy
Our
Input Tokens
(b) Zero-shot navigation comparison hetween GUI- Rise and GUI-Odyssey in terms of input token cost.
(a) Zero-shot navigation comparison between dif- ferent history methods in terms of imput token cost.
Figure Impact by different history representation in GUI navigation Mind2Web benchmark.
C.4
Case Study
We selecte several samples from the test results of two distinct platforms to conduct a case study, As illustrated in Figure 6. the agent demonstrates the capability perform structured reasoning, analyzing the progress of the current task and reasoning about decisions through step-by-step analytical process. Additionally, 7 how the agent summarizes historical information across the entire trajectory, which enables coherent reasoning in future steps.
17

---

Step
Step t+1
Km
IMER
PRIVATE TRANSFERS
Inttarteer Duliin in
wwwllteeeeeeeee May beitee ahuli teaes
and
cPogres Latenaarn the sucondallly a / ('oudia" sahar kurae tha
ao dillt secrrsflly redeetee time decpectio number rocms chrggrn spadiy checcinnaee
-Dudtion Sasuunign-Cciirrrr
cana" fes
whin przzacing
untiing the crackrsu other micustry
Dbudiion Rassoring
Mictioe-factor:: 'UuCe batuel Dat infland, poution: J.L. /e/ttoe
<Aelieno actenn EUO,, iakee Theck andett" boottor 12.3 0.511 s/Actione
clittayy sumasapp ssal laga pr of looding bote tioked ulaciin
ettetry Sare meryhhh toeh instdd hote badk prooreo the Tymmal wibhitee dloting udldedd Mid'd fooe snad nor suulis vedicttss thes abhent us fed acturee ald Mili Summmnyy
des ritoon by dicing the in hnleed" inlcatin insition to ancomm accomndatooun
Figure 6: A case study from the Mind2Web dataset illustrates detailed reasoning across two consecu- tive steps.
Step
Step
Step
Step
Step 4
Google
Setupergll
Value Mone, partion: Hora!
Adtant CCUD. Yalu? Nows, postoon 13 EASE
"Whara the novilt Rod
Actoe:/ecton: TUOC Were Sooetor [E.2) (.H]]]
Actenslacton: TUOC halad Soee. joatiar 4342 6.0f
Hatory Summary The has celtdd the email sctup process and now on home screes, raady to atrees rews
Hatory Summany The has retur ned to the home screen and now attempting to accsss nows sourse frant by clicking on Google Chiomna ven.
History Summary: The user his entered the search sare Uneen in Dranft and now execating the search find lsbest
History Summany: The user has searched for noss firei and is now attwmpging to aed mineant artices by cldding on search suganitinn
Hiltory Summany: The narigattdd back the menu and hyped a query search for newe ami amrices relited to Bradi, and now soeculing search
Figure 7: A case study from the AITW dataset illustrates the detailed history summarization for an entire trajectory
18

---

TST SCoT
HS
HSR
General
Install
G.Apps
Single
WehShop.
Overall
Gain
61.4
71.8
62.6
73.7
66.7
67.2
2
V
61.4
70.6
63.0
73.2
62.0
66.0
-12
3
X
X
X
38.0
48.4
46.9
36.7
42.9
42.6
-24.6
V
63.2
73.4
69.5
76.7
66.0
69.8
+2.6
V
V
64.1
73.2
69.3
67.9
70.7
+0.9
6
V
64.4
73.9
69.7
78.2
68.2
71.1
+1.3
Table 7: Ablation study of GUI-Rise the AITW benchmark.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction provide detailed overview of the relation-
enhanced framework propose address the limitations of current GUI agents.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Conclusion Section 6 we analyze the limitations our work in online environments and outline directions for future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
19

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Guidelines:In the Method Section we provide detailed introduction to our work and offer clear definitions for each component.
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the Experimental Section 5 we provide a detailed description of the experimental setup and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
20

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A compressed archive containing the source code has been included in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guidee wwwsssoSooSSsissooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctttps.cc/pubbll httpppplwlpppppppppppppppppaaaep wtw.:iddde///eeedmbmmeeeee
ssioioolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the Experimental Setup Subection 5.1 the hyperparameter selection, optimizer, and other relevant information.
provide detailed description of
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
21

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our model requires large-scale data for training. Similar prior related work 19 8 26]. error bars were not reported due to the substantial computational overhead.
Our experiments used consistent seed ensure the reproducibility of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Experimental Setup Subection 5.1 we provide a detailed computer resources needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::/://auurpp...co.opoalltttp
Answer: [Yes]
Justification: We have reviewed and understood the Code of Ethics and have made every effort to ensure research complies with its requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
22

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research is primarily conducted in simulated environments for training and testing, without involving private information, thus having minimal societal impact beyond advancing the understanding of specific aspects of deep learning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model is trained and tested simulated environment, posing no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
23

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all external sources of assets and their licenses permit our use case.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppsateeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release code base with included readme files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects, so there is no need to address participant risks, disclosure, or IRB approvals.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use large language models (LLMs) for writing and editing purposes
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non nta-standard components.
Please refer to our LLM policy //npur:////urraie cc. for what should or should not be described.
25


Reviews:
[{"review_id": "V9RgIneNmz", "reviewer": "Reviewer_1riz", "summary": "This work presents a framework to train GUI agents through Cold Start SFT + Reinforcement Learning with GRPO. They predefine a structured reasoning template with 3 parts: structured reasoning, action prediction, and history summarization, which helps the agent to do better reasoning and issue correct actions. They evaluate the trained model on four different GUI navigation benchmarks and show that their method improves the base model a lot.", "questions": "1. Why do you design a separate history summary reward, but no separate rewards for progress estimation/decision reasoning\n2. Why do you choose to put the history summary after action prediction and prompt it for the next step rather than before the action prediction to be a part of structured reasoning?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The framework design (SFT + GRPO) is intuitive and has been proven effective on coding and math problems. This is a nice trial on GUI agent training.\n2. GUI-Rise improves the base agent on GUI navigation tasks\n\nWeaknesses: \n\n1. The online navigation evaluation part only uses MiniWob as the benchmark, which is a relatively easy environment. There are some harder online interactive environments like WebArena [1], OS-World [2] that are commonly used for GUI agents. Lack of evaluation on these complex online environments and comparison with other baselines on such environments would make the method less persuasive\n2. The improvement under the standard setting is minor on Mind2Web and AITW, which seems to indicate that, compared to vanilla SFT on the training set, your framework does not show much advantage. But you can argue and explain more about this result.\n3. I have a few questions about the framework design. See the Questions part\n\n[1] WebArena: A Realistic Web Environment for Building Autonomous Agents\n\n[2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments", "was_revised": true, "final_justification": "The authors explain in detail in response to my reviews and solve most problems. So I keep my rating of \"borderline accept\".", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n---\n\n### W1:\nWe appreciate your suggestions and have now included more realistic environmental results in Table 1-1.\nFor WebArena, it employs Axtree and SoM annotations for action execution, requiring agents to output element IDs instead of click coordinates. As our agent is a vision-based model that generates click coordinates, it is inherently incompatible with the evaluation framework of WebArena.  Instead, we conduct an evaluation on the Android World online benchmark, which consists of 116 tasks across 20 real-world mobile apps. Our agent achieves a 16% performance improvement over the previous 2B-parameter SOTA model, InfiGUIAgent, further validating its effectiveness in complex GUI navigation scenarios, with results presented in the table below. \n\nAdditionally, we evaluate our agent on the OS-World benchmark (chrome-split), with results reported in the supplementary material Table 3. Moreover, to align with prior works (e.g., ShowUI), we supplement our MiniWoB experiments under the supervised fine-tuning setting. Moreover, to align with prior works such as ShowUI, we supplement our MiniWoB experiments under the supervised fine-tuning setting.\n\nOverall, these results collectively demonstrate that our approach maintains state-of-the-art performance and strong generalization across diverse and challenging online interactive environments.\n\n**Table 1-1. Step success rate (%) of 2B-parameter models on online benchmarks.**\n\n| Method        | MiniWob | Android World | OS World |\n|--------------|---------|----------------|----------|\n| Qwen2-VL-2B  | 66.8    | 0.0            | --       |\n| ShowUI-2B    | 71.5    | 7.0            | --       |\n| Infigui-2B   | --      | 9.0            | --       |\n| UI-Tars-2B   | --      | --             | 6.5      |\n| GUI-Rise-2B  | **72.8**    | **10.4**           | **8.7**      |\n\n---\n\n### W2:\n\nWe agree with the observation that GUI-Rise yields a minor improvement in performance over a vanilla SFT baseline under the in-domain settings of Mind2Web and AITW. Similar trends [1] have been observed in other vision-language domains, such as Vision-Language Navigation, where the performance difference between RL-based methods and SFT remains small in in-domain scenarios. This is largely attributable to the strong fitting capability of SFT to the training data distribution.\n\nThe core of our work lies in improving the model's generalization ability in more challenging out-of-distribution (OOD) scenarios that are closer to real-world applications. When the evaluation scenario shifts from standard settings to OOD settings, the performance of vanilla SFT is substantially degraded. In contrast, the unique design of our framework helps the model better understand and adapt to unknown tasks, resulting in performance improvements of 31.5% (Table 1 in the manuscript) and 50.7% (Table 2 in the manuscript). This demonstrates the critical value of our work for building GUI agents that can operate reliably in diverse and dynamic environments.\n\n[1] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training\n\n---\n\n### Q1:\n\nOur reward design represents a trade-off: we introduce a dedicated reward specifically for the history summary subtask, while only two types of rewards, format reward and action reward, are designed for the other components. This choice balances model performance and training efficiency. Our framework can be readily extended to incorporate separate rewards for progress estimation and decision reasoning. However, we found that doing so significantly increases the computational complexity during training, while yielding only marginal performance gains. \n\nTo verify its specific impact in practice, we conducted a small-scale comparative experiment on the AITW benchmark. The experimental results show that if independent rewards are designed for all components, the overall training duration of the model will nearly double, while the resulting improvement in final performance is marginal (about 1.7% improvement). As a trade-off, we chose to retain a separate reward only for the history summary component to balance performance gains against training efficiency.\n\n---\n\n### Q2:\n\nWe generate the history summary after the action prediction to ensure that it reflects the most recent action, thereby providing a complete and coherent context for the next step action prediction. Placing the history summary before the action prediction as part of the structured reasoning has two main drawbacks:\n\n(1) The summary at the current step would lack information from the latest action prediction, resulting in an incomplete historical context, which could degrade the performance of subsequent reasoning and decision-making.\n\n(2) The history summary from the previous step is already incorporated as input to the current structured reasoning. Including the current step’s history summary before action prediction would introduce redundant information.\n\n---"}, {"review_id": "cTQxGs7bAs", "reviewer": "Reviewer_W4us", "summary": "This paper proposed a reasoning-enhanced framework that integrates structured reasoning, action prediction, and history summary. \nThe proposed GUI agent, GUI-Rise, is trained by a two-stage training scheme, including SFT and RL. The propose framework structures the reasoning and execution pipeline for GUI-based tasks, introducing a history representation mechanism to improve decision accuracy. The experimental results show that the proposed GUI-Rise achieves SOTA results on standard benchmarks.", "questions": "The current discussion regarding methods for encoding history lacks sufficient rigor. For instance, the assertion in Lines 34-36 states that \"Existing systems encode history either (i) as action sequences alone [8, 9], which omit visual state and hinder progress estimation, or (ii) as full-resolution screenshots [19, 22, 28], which are computationally expensive and force severely truncated context windows.\" This claim overlooks alternative approaches, such as those adopted by UI-Hawk, which utilize low-resolution history screenshots instead of full-resolution ones. Consequently, the presented motivation for new encoding methods may not be as persuasive as intended, as it doesn't fully account for the spectrum of existing solutions.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\nThis paper proposed a reasoning-enhanced framework. The experimental results show that the proposed framework is effective.\n\n## Weaknesses\n1. The core contributions of the proposed three subtasks primarily reside in prompt engineering. Consequently, the technical novelty and architectural innovation appear somewhat limited.\n2. The evaluation of the proposed history summary subtask is constrained to comparisons with raw visual or action-only histories. A more comprehensive analysis should include state-of-the-art approaches such as UI-Hawk [1], which utilizes low-resolution visual history images. This alternative method also achieves a commendable balance between computational efficiency and task accuracy, and its exclusion weakens the comparative rigor of the presented results.\n\n[1] UI-Hawk: Unleashing the screen stream understanding for gui agents", "was_revised": true, "final_justification": "The rebuttal addressed some of my concerns and I would like to change the rating to borderline accept after reading the rebuttal.", "rebuttal": "We appreciate the reviewer's feedback and address the main concerns below.\n\n---\n\n### W1:\n\nWe respectfully disagree with the claim that our core contributions mainly reside in prompt engineering. In fact, the effectiveness of prompt engineering depends on the model’s inherent capabilities. However, our experiments on Mind2Web demonstrate that smaller models (e.g., 2B/3B parameters) often fail to produce meaningful history summaries or accurate progress estimations. When directly prompting the model to output such information, we frequently observe vague or noisy history summaries, such as: *The user started planning a trip; The starting point was set to Braintree with a typo in the \"From\" field; Although they interacted with some additional elements, their effect on the progress is not explicitly clear.*\nThese noisy outputs negatively impact downstream reasoning, and in our experiments, such direct prompting caused a 20% drop in step success rate on the Mind2Web benchmark.\n\nOur key contribution to our method is not in designing handcrafted prompts, but in developing a structured reasoning framework that enables the model to autonomously learn to perform each subtask during training. We view this dynamic subtask learning framework as a core technical innovation that enhances the model’s reasoning capabilities, distinguishing it from conventional prompt engineering approaches.\n\nThis design leads to consistent performance gains across both offline and online benchmarks (as shown in Tables 1, 2, and 4 in the manuscript), achieving new state-of-the-art results. These results validate both the practical effectiveness and the technical significance of our method.\n\n---\n\n### W2 & Q1:\n\nWe appreciate the reviewers' comments and suggestions for improving the rigor of our statements and address their main concerns as follows.\n\nUI-Hawk reduces visual history images to one-quarter of their original size and adopts a visual token compression ratio of 16, enhancing computational efficiency and performance. However, low-resolution images may lose detailed information, such as small interactive button states, compromising historical data integrity. Furthermore, a key limitation of UI-Hawk is that it has only been evaluated on GUI-Odyssey and its self-proposed FunUI benchmarks. It has not been tested on mainstream benchmarks widely adopted in the GUI navigation field, such as AITW and Mind2Web. Moreover, a direct and fair comparison with UI-Hawk is not feasible, as its results are reported on a privately re-annotated version of GUI-Odyssey (GUI-Odyssey+), with no public code, checkpoints, or benchmark details available.\n\nRegarding Question 1, our statement was intended to address the broader category of methods that rely on history screenshots. Specifically, approaches like UI-Hawk that process these visual sequences inherently face a trade-off between preserving information integrity and computational efficiency. Our agent, GUI-Rise, leverages the history summary capability acquired through training to self-summarize historical task states and executed actions. This approach ensures the integrity of historical information while also reducing computational overhead. \n\nWe will expand the discussion of these trade-offs and related methods in the revised Related Work section.\n\n---"}, {"review_id": "lTW86MDIKF", "reviewer": "Reviewer_Dt5U", "summary": "This paper investigates the application of Large Language Models in GUI Navigation and proposes the GUI-Rise method. The core components include structured reasoning, action prediction, and history summarization. Moreover, GUI-Rise employs the GRPO method for policy optimization. The authors conduct experiments on three benchmarks to validate the effectiveness of their approach.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper is well-organized and clearly written, with figures and tables effectively illustrating the core concepts.\n\n2. The authors provide comprehensive experimental validation including Out-of-Domain, In-Domain, Online, and Ablation studies to demonstrate GUI-Rise's effectiveness.\n\n3. The authors include case studies in the appendix to enhance understanding of GUI-Rise.\n\n**Weaknesses**:\n\n1. The paper primarily uses GRPO for model optimization but does not sufficiently justify its necessity in the UI context. Specifically, the authors should discuss whether GUI-Rise could utilize alternative methods like PPO or DAPO for policy optimization.\n\n2. GUI-Rise is currently validated only on 2B and 3B models. Testing on larger models, such as Qwen-VL-7B, would strengthen the evaluation.\n\n\n3. Typos: in Figure 1, \\<Decision Reasoning\\> -> </Decision Reasoning>\n\n4. Typos: Notation inconsistency between equations (7) and (6), e.g., in (7), $\\mathcal{R}^a$ -> $R^a$", "was_revised": true, "final_justification": "The authors have addressed all my concerns, and I maintain my positive rating.", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n---\n\n### W1:\n\nThe reason we adopt GRPO lies in its training efficiency, as it avoids the computational overhead associated with critic models. In contrast, actor-critic methods like PPO require optimization of both actor and critic models, leading to significantly higher resource consumption.\n\nRegarding DPO, our preliminary experiments revealed its sensitivity to preference pair quality. For instance, using Qwen2-VL-2B, we constructed around 2,000 preference pairs on the Mind2Web dataset. The model trained with these preferences achieved step SR of only 28.5%, 25.1%, and 27.6% on the cross-task, cross-website, and cross-domain splits, respectively, while achieving 68.4% on the training preference pairs itself. This indicates a tendency for DPO to overfit in GUI-based tasks, thus compromising generalization performance.\n\nWe appreciate the reviewer’s suggestion to consider alternatives such as DAPO or other methods (as RLOO and REMAX). These methods share a similar principle with GRPO: they aim to obtain reliable advantage estimates without relying on a critic model. In particular, DAPO can be seen as an improvement over GRPO. Adopting these methods is indeed feasible. However, we would like to emphasize that the choice of training algorithm is orthogonal to our main contribution. Our work primarily focuses on proposing a framework that integrates structured reasoning, action prediction, and historical summarization to enhance GUI agent performances. An analysis of such optimization strategies would be a valuable direction for future work.\n\n---\n\n### W2:\n\nFollowing the reviewers’ suggestion, we additionally trained the larger Qwen2.5-VL-7B model on the training sets of AITW and Mind2Web, and evaluated it on their respective test sets. We ensured a fair comparison with Qwen2.5-VL-7B baseline by using the same training sets, as shown in Tables 3-1 and 3-2. The step success rate results, demonstrate consistent improvements over the baseline by 3% on AITW and 10% on Mind2Web. It further confirming both the effectiveness and scalability of our method.\n\n**Table 3-1. Step success rate (%) of Qwen2.5-VL-7B (SFT) and GUI-Rise-7B on the AITW benchmark.**\n\n| Model                | General | Install | G.Apps | Single | WebShop | Overall |\n|---------------------|---------|---------|--------|--------|---------|---------|\n| Qwen2.5-VL-7B (SFT) | 68.6    | 77.4    | 76.2   | 80.0   | 68.5    | 73.7    |\n| GUI-Rise-7B         | **70.1**    | **80.2**    | **78.6**   | **81.4**   | **69.9**    | **75.7**    |\n\n---\n\n**Table 3-2. Performance (%) of Qwen2.5-VL-7B (SFT) and GUI-Rise-7B on the Mind2Web benchmark.**\n\n| Model                | Cross-task         |                     |           &nbsp;&nbsp;           | Cross-website       |                     |             &nbsp;&nbsp;         | Cross-domain        |                     |           &nbsp;&nbsp;           |\n|----------------------|--------------------|---------------------|----------------------|----------------------|---------------------|----------------------|----------------------|---------------------|----------------------|\n|                      | Ele.Acc            | OP.F1       &nbsp;&nbsp;&nbsp;&nbsp;        | Step SR        &nbsp;&nbsp; &nbsp;&nbsp;    | Ele.Acc             | OP.F1      &nbsp;&nbsp;&nbsp;&nbsp;         | Step SR      &nbsp;&nbsp; &nbsp;&nbsp;       | Ele.Acc             | OP.F1       &nbsp;&nbsp;&nbsp;&nbsp;        | Step SR      &nbsp;&nbsp; &nbsp;&nbsp;       |\n| Qwen2.5-VL-7B (SFT)  | 54.5               | 90.5                | 50.5                | 53.5                | 88.7                | 46.9                 | 51.1                | 89.1                | 46.8                 |\n| GUI-Rise-7B          | **56.9**               | **90.4**                | **52.3**                | **56.7**                | 88.7                | **50.7**                 | **56.4**                | **90.2**                | **51.4**                 |\n\n---\n\n### W3 & W4:\n\nWe are grateful to the reviewer for their careful attention. These typos will be fully corrected in the final manuscript.\n\n---"}, {"review_id": "s2aTPvgl64", "reviewer": "Reviewer_hw3E", "summary": "This paper proposes a strategy for training multimodal LLMs on GUI agent tasks. The proposed strategy consists of two steps. In the first step, the authors augment existing GUI trajectories with reasoning traces generated from GPT-4o-mini, so that the model can learn a structured reasoning process. In the second step, the authors uses reinforcement learning (GRPO) to continue train the model. The reward is based on step-level action correctness, and is designed to enhanced the reasoning structure. The authors use Qwen VL models as the base model, and show improvement on several mobile/web GUI agent benchmarks.", "questions": "In order for me to raise my score, I would like to see if each stage of the proposed method (SFT+RL) can achieve improvement on a more realistic online evaluation benchmark.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-structured with clear motivation. The method is simple and clearly explained.\n- The reward design fits the proposed structured reasoning strategy. \n\nWeakness:\n- The proposed training method requires ground-truth trajectories. This makes it difficult to scale and generalize to more complex GUI tasks.\n- The proposed RL is done on a static dataset, and the reward is mostly based on step-level action correctness. Thus the RL stage may not add much value after SFT. Ideally, RL should be done in an online environment, where the agent can learn by interacting with the environment. \n- The evaluation benchmarks are mostly offline, and the only online evaluation is done on a small dataset with few baselines. In order to show that the proposed method can work on more realistic environments, the authors need to evaluate on more challenging online benchmarks, such as WebArena, OSWorld, etc.\n- The authors have mentioned online training and evaluation as future work. However, I believe that it is critical that a GUI agent can work in a dynamic and realistic environment, rather than just static datasets.", "was_revised": true, "final_justification": "Thanks the authors for the response and the online evaluation. I raised my score to borderline accept.", "rebuttal": "We appreciate the reviewer's feedback and address the main concerns below.\n\n---\n\n### W1:\n\nOur current training approach, which leverages ground-truth trajectories, can be viewed as a form of imitation learning. We frame our current work as a first stage toward addressing complex GUI tasks.  The primary goal of this stage is to establish a strong foundation by enabling the agent to acquire basic operational skills from a high-quality dataset. Without the guidance of ground-truth trajectories, learning from sparse rewards alone (especially in the early stages) would lead to inefficient reinforcement learning and significantly slower convergence.\n\nWe argue that exploring this foundational stage is itself an important and standalone research contribution. It provides a reliable starting point for building agents capable of tackling more complex and realistic GUI tasks.\n\nTo address scalability in the later stages, once the model has acquired essential skills through imitation learning, we further refine it using reinforcement learning within a dedicated simulator. In this environment, the agent no longer relies on step-by-step demonstrations; instead, it receives reward signals based solely on the final task outcome. \n\n---\n\n### W2:\n\nWe respectfully disagree with the assertion that \"the RL stage may not add much value.\" As shown in Table 1 on the Mind2Web benchmark and Table 2 on the AITW benchmark in the manuscript, our method achieves significant improvements in zero-shot task success rate by 31.5% and 50.7%, respectively. These substantial gains clearly demonstrate the added value of our RL stage beyond supervised fine-tuning.\n\nThat said, we agree that performing RL in an online interactive environment is a promising direction for future work, and we consider it complementary to our current setting.\n\n---\n\n### W3:\n\nThanks for your suggestions, we have added more realistic environmental results in the table below. \nFor WebArena, it utilizes Axtree and SoM annotations to guide action execution, requiring agents to provide element IDs rather than click coordinates. As our vision-based agent produces click coordinates, it is fundamentally incompatible with WebArena’s evaluation framework. Instead, we evaluate our agent on the Android World online benchmark, which comprises 116 tasks across 20 real-world mobile applications. Our agent demonstrates a 16% performance improvement over the prior 2B-parameter state-of-the-art model, InfiGUIAgent. Detailed results are presented in the table below, further confirming its efficacy in complex GUI navigation scenarios. \n\nAdditionally, we evaluate our agent on the OS-World benchmark (chrome-split), with findings reported in the supplementary material Table 3. Besides, to align with prior works such as ShowUI, we supplement our MiniWoB experiments under the supervised fine-tuning setting.\n\nCollectively, these outcomes highlight that our approach sustains top-tier performance and robust generalization across diverse, challenging online interactive environments.\n\n**Table 4-1. Step success rate (%) of 2B-parameter models on online benchmarks.**\n\n| Method        | MiniWob | Android World | OS World |\n|--------------|---------|----------------|----------|\n| Qwen2-VL-2B  | 66.8    | 0.0            | --       |\n| ShowUI-2B    | 71.5    | 7.0            | --       |\n| Infigui-2B   | --      | 9.0            | --       |\n| UI-Tars-2B   | --      | --             | 6.5      |\n| GUI-Rise-2B  | **72.8**    | **10.4**           | **8.7**      |\n\n---\n\n### W4:\n\nWe clarify that in the limitations section of the paper, we specifically referred to online training as a direction for future exploration, not online evaluation.\n\nThat said, we fully agree with the reviewer on the importance of evaluating GUI agents in dynamic and realistic environments. To this end, we have already conducted extensive online evaluations across diverse environments, including MiniWoB (Table 4 in the manuscript), OS-World (Table 4 in the supplementary), and Android World (Table 4-1), demonstrating the model’s effectiveness in dynamic and realistic settings.\n\nTo avoid such misunderstandings, we will revise the limitations section in the final version to more clearly distinguish between online evaluation and online training.\n\n---"}]

Author rebuttal:

Meta-review: The paper claims to provide a novel method for GUI use agents via RL (GRPO). Reviewers agree that the method convincingly works on the benchmarks chosen compared to baselines and design choices are well justified - i.e. correctness is sound. That said, the relative novelty of the method is rather overstated - there appear to only be two things that are new, the history reward during GRPO and the prompt that forms that state representation during RL training. Further, given such specific reward and prompt engineering, it remains unclear how performance would translate to other benchmarks. I will sustain the reviewers' overall suggestion that this paper has enough merits to outweigh such concerns and warrant publication if there is room.
Decision: Accept (poster)