Title: GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: GUI agents

Multimodal large language models

Reinforcement learning
Abstract: While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, GUI-Rise, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a strategy for training multimodal LLMs on GUI agent tasks. The proposed strategy consists of two steps. In the first step, the authors augment existing GUI trajectories with reasoning traces generated from GPT-4o-mini, so that the model can learn a structured reasoning process. In the second step, the authors uses reinforcement learning (GRPO) to continue train the model. The reward is based on step-level action correctness, and is designed to enhanced the reasoning structure. The authors use Qwen VL models as the base model, and show improvement on several mobile/web GUI agent benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-structured with clear motivation. The method is simple and clearly explained.\n- The reward design fits the proposed structured reasoning strategy. \n\nWeakness:\n- The proposed training method requires ground-truth trajectories. This makes it difficult to scale and generalize to more complex GUI tasks.\n- The proposed RL is done on a static dataset, and the reward is mostly based on step-level action correctness. Thus the RL stage may not add much value after SFT. Ideally, RL should be done in an online environment, where the agent can learn by interacting with the environment. \n- The evaluation benchmarks are mostly offline, and the only online evaluation is done on a small dataset with few baselines. In order to show that the proposed method can work on more realistic environments, the authors need to evaluate on more challenging online benchmarks, such as WebArena, OSWorld, etc.\n- The authors have mentioned online training and evaluation as future work. However, I believe that it is critical that a GUI agent can work in a dynamic and realistic environment, rather than just static datasets.\n\nQUESTIONS:\nIn order for me to raise my score, I would like to see if each stage of the proposed method (SFT+RL) can achieve improvement on a more realistic online evaluation benchmark.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
This paper investigates the application of Large Language Models in GUI Navigation and proposes the GUI-Rise method. The core components include structured reasoning, action prediction, and history summarization. Moreover, GUI-Rise employs the GRPO method for policy optimization. The authors conduct experiments on three benchmarks to validate the effectiveness of their approach.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1. The paper is well-organized and clearly written, with figures and tables effectively illustrating the core concepts.

2. The authors provide comprehensive experimental validation including Out-of-Domain, In-Domain, Online, and Ablation studies to demonstrate GUI-Rise's effectiveness.

3. The authors include case studies in the appendix to enhance understanding of GUI-Rise.

**Weaknesses**:

1. The paper primarily uses GRPO for model optimization but does not sufficiently justify its necessity in the UI context. Specifically, the authors should discuss whether GUI-Rise could utilize alternative methods like PPO or DAPO for policy optimization.

2. GUI-Rise is currently validated only on 2B and 3B models. Testing on larger models, such as Qwen-VL-7B, would strengthen the evaluation.


3. Typos: in Figure 1, \<Decision Reasoning\> -> </Decision Reasoning>

4. Typos: Notation inconsistency between equations (7) and (6), e.g., in (7), $\mathcal{R}^a$ -> $R^a$

QUESTIONS:
See Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a reasoning-enhanced framework that integrates structured reasoning, action prediction, and history summary. \nThe proposed GUI agent, GUI-Rise, is trained by a two-stage training scheme, including SFT and RL. The propose framework structures the reasoning and execution pipeline for GUI-based tasks, introducing a history representation mechanism to improve decision accuracy. The experimental results show that the proposed GUI-Rise achieves SOTA results on standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThis paper proposed a reasoning-enhanced framework. The experimental results show that the proposed framework is effective.\n\n## Weaknesses\n1. The core contributions of the proposed three subtasks primarily reside in prompt engineering. Consequently, the technical novelty and architectural innovation appear somewhat limited.\n2. The evaluation of the proposed history summary subtask is constrained to comparisons with raw visual or action-only histories. A more comprehensive analysis should include state-of-the-art approaches such as UI-Hawk [1], which utilizes low-resolution visual history images. This alternative method also achieves a commendable balance between computational efficiency and task accuracy, and its exclusion weakens the comparative rigor of the presented results.\n\n[1] UI-Hawk: Unleashing the screen stream understanding for gui agents\n\nQUESTIONS:\nThe current discussion regarding methods for encoding history lacks sufficient rigor. For instance, the assertion in Lines 34-36 states that \"Existing systems encode history either (i) as action sequences alone [8, 9], which omit visual state and hinder progress estimation, or (ii) as full-resolution screenshots [19, 22, 28], which are computationally expensive and force severely truncated context windows.\" This claim overlooks alternative approaches, such as those adopted by UI-Hawk, which utilize low-resolution history screenshots instead of full-resolution ones. Consequently, the presented motivation for new encoding methods may not be as persuasive as intended, as it doesn't fully account for the spectrum of existing solutions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a framework to train GUI agents through Cold Start SFT + Reinforcement Learning with GRPO. They predefine a structured reasoning template with 3 parts: structured reasoning, action prediction, and history summarization, which helps the agent to do better reasoning and issue correct actions. They evaluate the trained model on four different GUI navigation benchmarks and show that their method improves the base model a lot.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The framework design (SFT + GRPO) is intuitive and has been proven effective on coding and math problems. This is a nice trial on GUI agent training.\n2. GUI-Rise improves the base agent on GUI navigation tasks\n\nWeaknesses: \n\n1. The online navigation evaluation part only uses MiniWob as the benchmark, which is a relatively easy environment. There are some harder online interactive environments like WebArena [1], OS-World [2] that are commonly used for GUI agents. Lack of evaluation on these complex online environments and comparison with other baselines on such environments would make the method less persuasive\n2. The improvement under the standard setting is minor on Mind2Web and AITW, which seems to indicate that, compared to vanilla SFT on the training set, your framework does not show much advantage. But you can argue and explain more about this result.\n3. I have a few questions about the framework design. See the Questions part\n\n[1] WebArena: A Realistic Web Environment for Building Autonomous Agents\n\n[2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments\n\nQUESTIONS:\n1. Why do you design a separate history summary reward, but no separate rewards for progress estimation/decision reasoning\n2. Why do you choose to put the history summary after action prediction and prompt it for the next step rather than before the action prediction to be a part of structured reasoning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}