MobileUse: A Hierarchical Reflection-Driven GUI Agent for Autonomous Mobile Operation
Authors: Ning Li
Xiangmou Qu
Jiamu Zhou
Jun Wang
Muning Wen
Kounianhua Du
Xingyu Lou
Qiuying Peng
Jun Wang
Weinan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Mobile GUI Agent
Multi-Agent Framework
Hierarchical Reflection
Proactive Exploration
Primary area: applications

Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have enabled the development of mobile agents that can understand visual inputs and follow user instructions, unlocking new possibilities for automating complex tasks on mobile devices. However, applying these models to real-world mobile scenarios remains a significant challenge due to the long-horizon task execution, difficulty in error recovery, and the cold-start problem in unfamiliar environments. To address these challenges, we propose MobileUse, a GUI agent designed for robust and adaptive mobile task execution. To improve resilience in long-horizon tasks and dynamic environments, we introduce a hierarchical reflection architecture that enables the agent to self-monitor, detect, and recover from errors across multiple temporal scales—ranging from individual actions to overall task completion—while maintaining efficiency through a Reflection-on-Demand strategy. To tackle cold-start issues, we further introduce a proactive exploration module, which enriches the agent’s understanding of the environment through self-planned exploration. Evaluations on the AndroidWorld and AndroidLab benchmarks demonstrate that MobileUse establishes new state-of-the-art performance, achieving success rates of 62.9% and 44.2%, respectively. To facilitate real-world applications, we release an out-of-the-box toolkit for automated task execution on physical mobile devices, which is available at https://github.com/MadeAgents/mobile-use.

MobileUse: A Hierarchical Reflection-Driven GUI Agent for Autonomous Mobile Operation
Ning Li Xiangmou Qu²², Jiamu Zhou?*, Jun Wang', Muning Wen', Kounianhua Du', Xingyu Lou?, Qiuying Peng"! Jun Wang",, Weinan Zhang"
'Shanghai Jiao Tong University 2OPPO Research Institute {pengqiuying, wang
wnzhang@s tu. edu.
Abstract
Recent advances Multimodal Large Language Models (MLLMs) have enabled the development of mobile agents that can understand visual inputs and follow user instructions, unlocking new possibilities for automating complex tasks on mobile devices. However, applying these models to real-world mobile scenarios remains a significant challenge due to the long-houd ang-horizon task execution, difficulty error recovery, and the cold-start problem in unfamiliar environments. To address these challenges, we propose MobileUse, a GUI agent designed for robust and adaptive mobile task execution. To improve resilience in long-horizon tasks and dy- namic environments, we introduce a hierarchical reflection architecture that enables the agent to self-monito detect, and recover from errors across multiple tempo- ral scales-ranging from individual actions to overall task completion-while maintaining efficiency through
-aniiionn-o--emaaddd strategy. To tackle cold- start issues, we further introduce a proactive exploration module, which enriches the agent's understanding of the environment through self-planned exploration. Evaluations on the AndroidWorld and AndroidLab benchmarks demonstrate that MobileUse establishes new state-of- of-the-art performance, achieving success rates of 62.9% and 44.2%, respectively. To facilitate real- world applications, we release out-of-the -oo---o-ooooo toolkit for automated task execution on physical mobile devices, which is available at https:t///// h/ggttttt...... com/Madedgents. www.ldddddennttmmmeeeeeeeeeeeeee metiitss/ooolleesseeeeeeeeeeeeee
Introduction
Recent advances in multimodal large language models (MLLMs) have significantly enhanced the capability of Al systems understand and interact with visual environments. By jointly processing text and images, MLLMs such as GPT-4V (Achiam et al., 2023). Gemini (Team et al. 2024).
Claude (Anhhropic 2024), and Qwen-VL VL Bai et al. (0255) have demonstrated impressive performance
on range of lanion-language tasks, including visual question answering (Hu et al. 2023 2024).
image captioning (Bianco et al. 2023 Rotstein et al. 2024), and GUI control (Müller and Zunić 2024
Qi al. 2025). These developments open up promising opportunities for building general-purpose agents capable of interleaved visual and textual contexts perceiving and instruction following.
Among various platforms, mobile devices present promising option for building general-purpose intelligent agents due to their vast scope and versatility. On one hand, the wide variety of apps available mobile platforms provides expansive environment, making it an ideal domain for developing agents capable of addressing diverse user needs. Moreover, with only a touchable screen,
"Equal contribution.
i Work done during an interssiip at OPPO. Corresponning authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

user interactions on mobile devices are typically constrained to a small set of intuitive actions (e.g., click, swipe, type). making the action space relatively simple compared to desktops. On the other hand, building GUT agents specifically for mobile scenarios can address real user needs by automating repetitive or complex tasks, such as filling out forms, navigating multi-step settings, that would otherwise require significant manual effort. This is particularly valuable in contexts such as accessibility support, productivity enhancement, and digital assistance. With this in mind, we'd like to develop an effective mobile GUI agent-one that can serve as a viable solution for realizing general-purpose AL.
Several recent studies have explored the potential of building mobile agents with advanced MLLMs. significant body of work (Wu et al.) [024] Lin al., 2024; Xu et al.) 2024b; Qin et al. 2025) focuses on constructing high-quality datasets and fine-tuning MLLMs to develop specialized domain- adapted models for GUT interaction tasks. Meanwhile, frameworks such as AppAgent-v2 (Li et al. 2024a), MobileAgent-v2 (Wang et al.) 2024a), and Agent-S2 (Agashe et al., 2025) decompose mobile tasks into structured decision-making pipelines, leveraging the reasoning and compositional abilities of cutting-edge LLMs and MLLMs. Despite this progress, building robust agents for dynamic mobile tasks remains a significant challenge. (C1) Robustness in long-horizon tasks and dynamic environments: Many mobile tasks require long-horizon execution and are inherently complex, involving a sequence of interdependent actions. Coupled with the dynamic nature of mobile environments, where elements can frequently change (e.g.. due app updates or varying screen layouts), errors are inevitable during task execution. This necessitates that mobile agents be capable of detecting failures and recovering gracefully. (C2) Unfamiliar scenarios: It is common for mobile agents to encounter unfamiliar apps and interfaces, particularly in cold-start settings, which require strong exploration and on-the-fly learning capabilities to operate effectively.
To address these challenges, we introduce MobileUse, a GUI agent designed for robust execution and flexible error recovery in dynamic mobile operation tasks. 1) We equip MobileUse with hierarchical reflection architecture, which enables the agent to self-monitor, verify, and revise its decisions during the execution. Hierarchical reflection works from the microscopic step-level to the macroscopic task-level: It checks whether each action is in line with expectations, whether the process follows the correct track, and whether the goal is successfully accomplished. Besides, recognizing that excessive reflection on each step could increase the overhead and even be detrimental, we incorporate Reflection-on- -o--Deannd strategy. By selectively invoking reflection based on confidence score, hierarchical reflection achieves the unity of performance and efficiency. 2) Observing that many failures caused by unfamiliarity with the environment, enabling the agent to proactively interact with the environment prior to the task execution. It collects general knowledge on unfamiliar apps, helps the agent complete downstream tasks more efficiently and accurately in cold-start scenarios. As a whole, we build MobileUse as a multi-agent framework, including an Operator for task execution, a Progressor for progress summarization, Hierarchical Reflectors for thoughtful reflection, and a Proactive Explorer for knowledge accumulation.
design proactive exploration module,
We evaluate the performance of MobileUse on two dynamic Android benchmarks: Android and AndroidLab Empirical results show that Mobile achieves state-of ofate-of-theeart (SOTA) performance, with success rates of 62.9% and 44.2%, respectively. With comprehensive ablation studies and analyses, we highlight the effectiveness of hierarchical reflection and proactive exploration in solving complex tasks. To facilitate real- real-world applications, we release an out-of-the-box toolkit for automated task execution on physical mobile devices. With detailed documentation, users can easily connect their mobile phones to MobileUse and experience its capabilities as a powerful mobile assistant.
In summary, we propose MobileUse, a GUI agent with the following contributions:
Hierarchical Reflection. We propose hierarchical reflection, a novel architecture to support robust execution and flexible error handling in mobile operation tasks. The hierarchical reflection architecture enables the agent to detect and recover from errors only when nec- essary, operating across multiple fromm individual action execution overall task completion.
Proactive Exploration. We design proactive exploration module to improve robustness against unfamiliar environments. Rather than relying on explicit instructions, our method depends on the model's proactive exploration to accumulate common knowledge for better task execution.
2

---

SOTA Performance and Support for Real-world Application. We conduct comprehensive evaluations on two mobile benchmarks and demonstrate that MobileUse achieves state-of- the-art performance compared to strong baselines. To facilitate real-world applications, we release an out-of-the-box toolkit for automated task execution on physical mobile devices.
2 Related Works
2.1 GUI Agent
Autonomous agents have demonstrated significant potential in enhancing human tasks effectively. In digital environments, information involves multimodal information with text, images, and visual elements, This complexity poses challenges for language models, driving increased research interest in GUI Agents.
Models like GPT-4o (Hurst et al. 2024), Qwen2.5-VL. (Bai et al. 2025), and UI-TARS (Qin et al. 2025) integrate visual understanding capabilities, enabling end-to-end execution GUI automation tasks through natural language instructions. These models advance fine-grained visual perception, document parsing, object localization, and reasoning, laying the foundation for versatile GUI Agents. Similarly, AppVLMM (Papoudakis al. 2025), FerretUl-2 (Li et al. 2024b), and Aria-U1 (Yang et al. 20245) offer lightweight visual models, while a scalable data pipeline and pretraining on GUT tasks further enhance grounding and interaction ability. To better identify the interactive UI elements, V- Droid (Dai et al. 2025) parses the XML representation of the UI state. It utilizes an agent as a verifier rather than a conventional generator exccute appropriate actions. Advancements in multi-modal understanding, reasoning, and task automation have significantly enhanced the capabilities of GUI Agents (Anthropic] 2024). positioning them as transformative tools in human- human-mobile interaction.
2.2 Multi-Agent System
As research on automatic agents progresses, multi- systems are increasingly emphasized due to the inherent limitations of monolithic approaches in handling long-context scenarios involving multi-modal data. Single agents often struggle to meet the demands of planning, reasoning, and grounding tasks simultaneously.
et al.] (024). AutoGen((Wu et al.) (023). and CAMEL (Li et al. 2023) facilitate an orchestrator-agent framework, where the orchestrator plans and interprets user intent and generates instructions, while the assistant agent executes tool invocations. Mobile-Agent--i
Aagentic-One(Founmnn
(Wang al. 2024a) introduces reflection agent to evaluate operator performance and suggest corrective actions when deviations occur. AppAgent (Zhang et al. 2025), Agent-S2 (Agashe et al. 2024) enhances task decomposition by self-evolution module, leveraging tasks. These advancements highlight the
web knowledge and episodic memory for growing importance of collaborative mulli--gent architectures in complex task automation.
However, the current comprehension of the reflection module is still inadequate. Reflection at the action level (Wang et al., 2024a] Wu et al., (025) demonstrates limited utility in facilitating long- horizon tasks, while an excessive reliance on simplistic reflections not only prolongs processing time but also frequently leads to operational errors caused by hallucination-inddued assessments. In addition, existing agent frameworks exploring self-evolution modules typically extract task-related experiences from previously executed trajectories. Nevertheless, these experiences often fail to be effectively transferred to the execution of new tasks, limiting their adaptability and generalization.
MobileUse
3.1 Framework Overview
As illustrated Figure [ we build MobileUse as a multi-agent framework designed for robust execution and flexible error recovery in dynamic mobile operation tasks. It consists of two stages: Proactive Exploration stage enables the agent to familiarize itself with the new environment while systematically exploring and accumulating common knowledge. Autonomous Mobile Opera- tion stage incorporates the Operator, Hierarchical Reflectors, and the Progressor to execute user instructions in collaborative and feedback-driven loop.

---

Proactive Exploration
Mobile Environment
Browsey
search
Reflection on demond
Open
back
oction
Hierarchical Reflection
progressor
APPs
back
operator
uhual
reflection feedback
oction reflection
previous progress
Aevent trojectory
Rule bisee repect? action cycle?
trajectory reflection
Ardes
Crilc spree
Suemeryagee
Common Knowledge
summary
Impectoryy
D X
globol reflection
User-lnstruction
Autonomous Mobile Operations
Figure 1: Overview of the MobileUse agent. In the Proactive Exploration stage. MobileUse famil- iarizes itself with the new environment while systematically exploring and accumulating common knowledge. In the Autonomous Mobile Operation stage. given user instruction, in each step the Operator will observe the screenshot and output specific action. Then MobileUse will perform hierarchical reflection three different levels for robust task execution. Finally, the Progressor will summarize and update the current progress at the end of each step iteration.
Table I: Notations and corresponding descriptions.
Notation
Description
T
User instruction.
K
Knowledge collected through proactive exploration.
Kr
Knowledge related the instruction
E, O.
The Explorer, Operator, Reflector Progressor. Screenshot at step f. The superseript means the t-th step.
a'
The output generated The thought, structure and natural language representation an action, (ais ai o/)) The action type and parameters. ef, (diyper (quans))
the Operator at step f.
ais azy o' onges Gurams T
The feedback generated the Action, Trajectory, and Global Reflector step The feedback of hierarchical reflection at step r' (rá, ri, r.). The summarized progress generated by the Progressor step
p'
Stage I: Proactive Exploration. In this stage, the agent is guided to proactively interact with the apps and generate common knowledge K based on the execution trajectory. We do not provide user instructions for the agent complete specific tasks, allowing the agent to fully explore the environment without being constrained by specific tasks. More details are provided in Section 3.3
Stage II: Autonomous Mobile Operation. Given a user instruction I, firstly relevant knowledge K₁ will be retrieved to support downstream execution. In the I-th step the task execution, the Operator O serves the drcision-making agent to generate actions a' with I, Ky. current screenshot s", history actions -t, the feedback p and progress p² from last step. Formally,
a' = (aj, ag a O(T, K1,
where of a thought containing the reasoning process, at and a are structured and natural language descriptions of the action. See Appendix B for the full action space. After action execution, MobileUse will perform hierarchical reflection to generate reflection of three different levels (Section 3.2) Finally, the Progressor P maintains progress summary p' of the task execution.

---

Interaction Histories
Trigger Condations
Ropeated Screen?
Reposted Action?
Accumulated Emors!
terminate
Hicrurchical Reflection
Action Reflector
Trajectory Reflector
Global Reflector
Terdback Output
Figure 2:
Overview the hierarchical reflection architecture. The Action Reflector operates on single step to provide immediate feedback. The Trajectory Reflector operates the latest trajectory to ensure effective progress. The Global Reflector operates on the overall interaction history to validate the task completion.
updating after each step by integrating past progress, the latest action, and reflection feedback:
P(a"
This two-stage execution with the collaboration of different modules enables MobileUse to accumulate knowledge, interpret complex instructions, and adaptively self-correct, resulting in robust and reliable task execution.
3.2 Hierarchical Reflection
During the execution of complex mobile tasks, agents frequently encounter challenges such as incorrect grounding (Gou et al. (2025). hallucinations (Liu et al.] 2023). flawed planning (Agashe) al. 2025), and getting stuck repetitive behaviors (Agashe et 2024). These errors often go undetected in purely feedforward architectures and can accumulate over time, ultimately leading to task failure. To mitigate these issues, we propose hierarchical reflection architecture that enables the agent to self-monitor, verify, and revise its decisions across multiple temporal scales. This mechanism comprises three components: Action Reflector for immediate feedback, Trajectory Reflector for progress reflection, and Global Reflector for overall validation, illustrated in Figure 2
3.2.1 Action Reflector
When operating the device, mobile agent make various mistakes, such opening the wrong app, forgetting to activate the text box before typing text, clicking the wrong icon, etc. When errors occur, the sereen often does not change expected. The Action Reflector Raction is designed to provide immediate feedback after each action execution. At each step I, the Action Reflector will observe the screenshots before and after the action execution determine whether the action achieved its intended effect. To enhance perception and facilitate eтror localization, we design a Perception module Pe to pre-compute the visual difference between the two screenshots and highlight the changed regions using bounding boxes. The Action Reflection process can be formulated as:
r Raction (1, s Pe(s', s(+1), a²),
(3)
where rf is the generated feedback that describes the error and possible causes.
In the actual task execution process, observe that most of the actions made by the Operator correct. Therefore, invoking the Action Reflector at every step can introduce latency and unnecessary computation. Additionally, the Action Reflector make mistakes, which will mislead the Operator to wrong state or repeated error correction. To address this, we propose a Reflection-on-Demat strategy, which dynamically determines when trigger reflection based the action confidence.
Specifically, in step the Operator outputs a predicted action type "ype represented as token sequence (wi, . wK), where usually & E [1, Each token w; is associated with a model- assigned probability p(w.). We compute the confidence score at for the action as the mean log- probability of its tokens:
log p(w)).
5

---

Only when e' I 8, where 0 is predefined threshold, the Action Reflector is invoked to reflect on the current step.
Equipped with the Action Reflecor, MobileUse is capable of detecting and recovering from er- rors caused by one-step failure, such failed grounding, visual ballucinations, and incorrect UI understanding.
3.2.2 Trajectory Reflector
Only the Action Reflector is not enough for the robust and consistent execution of complex and long-horizon mobile tasks. Sometimes, the mobile agent executes each action correctly, but is not on the right path finish the user instruction. In other circumstances, although the Action Reflector recognizes the error in the current step and provides some suggestions, the agent is not guaranteed to fully recover from the potential cumulated errors, resulting in a repeated failure in the near future.
Here we design a Trajectory Reflector Ruajectory, which operates over the latest progress, and short history of recent actions (c.g., last 3-5 steps) along with the corresponding action-level reflections to evaluate whether the trajectory is coherent and progressing toward task completion. It is designed to detect error patterns that span multiple steps, such as deviations from the user instruction due to accumulated drift or repeated actions. Formally,
r = Rurajettory (I,''" ra', (a',
(5)
where rf is the generated feedback. p²11 is the progress summarized by the Progressor in the last step. ra will be empty if the Action Reflector is not invoked in the corresponding step.
The Trajectory Reflector is also called only when necessary to balance the efficiency and performance. Instead of computing confidence score, we define several trigger conditions for the Trajectory Reflector that focus on task progress and error patterns: (1) Repeated Actions, indicating that the agent is stuck or making ineffective decisions. (2) Repeated Screenshots, suggesting that the agent failing to make progress. (3) Accumulated Action-Level Errors, requiring broader analysis to assess whether the trajectory is deviating from the task goal. Once any of these conditions are detected, the Trajectory Reflector is invoked to analyze recent actions, helping the Operator stay on track and make effective progress toward fulfilling the user instruction.
3.2.3 Global Reflector
When the Operator believes that the user instruction has been completed, will terminate the task execution with specially defined action terminate. However, the task may not actually be finished. In long-horizon task that consists of several sub-goals, the agent may omit some important steps. In other cases, the Operator will terminate the task prematurely due to hallucination or high complexity of the task.
The Global Reflector Rglobul is designed to review the completion of tasks from global perspective. Invoked only upon task completion, the Global Reflector evaluates whether the user instruction has been successfully fulfilled based on the historical actions and the latest screenshots. If it determines that the task remains incomplete, will provide feedback to the Operator, who will be asked to continue finishing the task in the next iteration. Otherwise, the task will be terminated gracefully. Formally,
r = = Rgloball (a", (9)
(a', (()
if Spe = terminate,
(6)
where ri and of will be empty if the corresponding reflector is not invoked in the i-th iteration. In MobileUsse, we choose j max(0, 3) to balance the efficiency.
Together, these three-level reflection mechanisms form a hierarchical reflection architecture that allows the agent to detect and correct errors different scales. Each reflector provides timely feedback to the Operator, enabling it to revise strategies, avoid redundant behavior, and maintain alignment with the user instruction throughout the task.
3.3 Proactive Exploration
When interacting with mobile devices, the agent often makes mistakes due to unfamiliarity with the environment, such as not knowing the meaning of a certain icon's color, shown in Figure

---

11 A possible solution is to adopt the self-evolution mechanism (Agashe et al. 2024 Wang et al. Liu et al. 20256), which gradually accumulates experience when completing tasks. However, erroneous experiences can interfere with the model's normal execution of actions. On the other hand, the lack of common knowledge about interactions makes difficult for the model to correct complex operations through self-evolution.
2025
Proactive exploration is designed to enable the agent to mimic human exploration of new envi- ronments and discover the operational common knowledge of the environment. As illustrated in the left of Figure [ our method adopts a agent collaboration framework to guide the exploration process: (i) For each app. we guide the Action Agent explores unfamiliar regions of the current app environment without relying on task-specific instructions. (ii) After tra- jectory with n steps T; = (s', "e......""" ....a,, s") is collected by the Action Agent, the Summary Agent identifies valuable operation trajectories and summarizes them into reusable experiences K2 = summary as'..'....,,. a'',, s²+)), thereby establishing an automated pipeline for data explo- ration and knowledge generation. We constrain these summarized experiences to be as general and beneficial as possible. (iii) The Judge Agent monitors the exploration process to prevent redundant or ineffective behavior and provides corrective feedback when necessary.
Unlike the previously proposed task-driven exploration approach (Zhang et al. 2025). which relies on manually constructed tasks, our proactive exploration a aggnottoiticc and mostly autonomous method. By combining different agents, it can proactively and effectively collect general knowledge about UI layouts, navigation patterns, and potential states, which especially useful when dealing with scenarios involving exceed the boundaries of their pre-trained capabilities.
4 Experiments
4.1 Experimental Settings
Benchmarks. We evaluate the performance of MobileUse on two widely-used mobile benchmarks: AndroidWorld (Rawles et al. 2025) and AndroidLab (Xu et al., 2024a). AndroidWorld includes 1 116 tasks across 20 apps, with randomized parameters that yield millions of possible task variants, emphasizing generalization diverse instructions and UI states. Androidl .ab covers 138 tasks from 9 apps with fine-grat fineggriined success metrics and a structured evaluation pipeline. Both benchmarks provide controllable Android interaction environments, standardized task initialization procedures, and well-defined automated evaluation processes, ensuring consistency in evaluation.
Baselines. We compare our method against diverse set of recent baselines, grouped into two main categories. Single-agent models are end-to-end systems that directly map textual or visual inputs GUI operations, including general-purpose models such GPT-4o (Achiam et al. 2023).
Claude (Anthropic 2024), Gemini (Wang et al., 2024b), and Qwen2.5-VL (Bai et al. 2025). It also includes domain-specialized models such InfiGUlAgent (Liu al., 2025a). Aguvis (Xu
eet al. (2024b), V-Droid (Dai et al. 2025), and UI-TARS (Qin et al. (025). which are specifically trained fine-tuned various GUI operation datasets. Multi-agent frameworks like UGround (Gou et al. 2025), Aguvis (Xu et al., 2024b)) Aria-UI (Yang et al. (0224b), and Agent-S2 (Agashe et al. 2025) decompose the task into sub-components, usually combining powerful language models (e.g..
GPT-4o) with perception or grounding modules for improved decision-making.
Implementation Details. We use the open-source multimodal language model Qwen2.5-VL-722 Instruct (Bai et al., 2025) with temperature = for our base model. To assess the impact of model size, we also evaluate the performance with the Qwen2.5-VL. -t
Instruct and 32B model in Appendix D.2 Besides, we release the MobileUse Toolkit with open-source code and comprehensive documentation, which is detailed in Appendix H
4.2 Main Results
Table and Table [3] demonstrate that MobileUse outperforms existing mobile agents on the An- droid World and AndroidLab benchmarks, achieving 62.9% and 44.2% success rates, respectively. In AndroidWorld, MobileUse achieves a 3.4% performance improvement compared to the SOTA solution V-Droid. Compared to the best Multi-Agent baseline Agent-S2, which is built powerful close VLLMM Slaude337-Soomme and a carefully tuned vision model UI-TARS-72B-DPO MobileUse achieves 8.6% performance improvement. Compared to the single agent Qwen2.5-VL-72BB
7

---

MobileUse significantly increases the success rate by 27.9%. In AndroidLab, MobileUse achieves 5.9% performance improvement on the task success rate compared the strongest baseline V-Droid. It also achieves the best result on the sub-goal success rate. The RRR and ROR are secondary metrics reflecting each agent step's redundancy and rationality. Although MobileUse is not optimal in these two aspects, it just illustrates that our hierarchical reflection mechanism can reflect and correct errors with more execution steps, achieving better performance on the overall success rate.
Table Success Rate (%) the AndroidWorld benchmark.
Method
Model
SR
InfiGUIAgent (Liu et al. 2025a
InfiGUUAAent-2B
9.0
CogAgent [Hong et al 2024b)
CogAgent-9B-202244
9.0
Gemini WWag
U240
Gemini- 1.5-Pro
22.8
Aguvis Xu et 2024b
Aguvis-72B
26.1
Single
Claude Anthropic
20/24
Claude Computer-Use
27.9
Agent
GPT-4o (Hurst ct a..
2024
GPT-4o
34.5
Qwen2.5- VL. (Bai ct al. 2025
Swen2.5-VL-7728 -Istrut-nnttruct
35.0
UI-TARS (Qin al. 2025
UI-TARS-7BB-SFT
46,6
V-Droid Dai et al.
2025
V-Droid
59.5
M3A (Rawles et al. 2025
Gur-4Turbo
30.6
UGround (Gou et al. 2025
GPT-4o + UGround
32,8
Aguvis Xi ct at.
2024h)
GPT-4o + Aguvis-7B
37.1
Multi-
Mobile-Agent-vZ Waan al.
2024a)
Qwen2.5--VL-7B--nsstue
37.1
Agent
Aria-UI YYang al.
2024h
GPT-4o Aria-UI
44.8
AndroidCien (L.i et al.
20025
GPT-4o
46.8
Agent-S2 (Agashe el al.
2025
Claude-3.7-Sonnet + UI-TARS-72B-DDA
54.3
MobileUse (Ours)
Qwen2.5-VL-77BB-mm
62.9
Table 3: Results on the AndroidLab benchmark. SR, Sub-SR, RRR, and ROR represent Success Rate, Sub-Goal Success Rate, Reversed Redundancy Ratio, and Reasonable Operation Ratio, respectively.
Model
SR
Sub-SR
RRR
RORT
CogVLM2-ft (Hong ct al.
2024a)
11.59
16,06
57.37
85.58
Dwen2.5-VI--723 Instruct (Bai ct al.
2025
17.52
24.00
73.60
81.92
Gemini-1.5-Pro. WWan et
20244
18.84
22.40
57.72
83.99
Qwen2-7B-fi Yang al. 2024a)
19.57
24.40
77.31
92.48
LLaMA3.1-8B-II Grattationi el al. 2024
23.91
30.31
75.58
92.46
Claude-3.5-Somee (Anthropic)
2024)
28.99
32.66
113.41
81.16
GPT-4o (Hurst
[024
31.16
35.02
87.32
85.36
MutoGLM] el al. 22024
36,20
V-Droid (Dai et al. 2025
38.30
MobileUse (Ours)
44.20
50.01
74.40
88.50
4.3 Ablation Study
Table 4: Ablation study the AndroidWorld benchmark.
Method
Easy Tasks
Medium Tasks
Hard Tasks
Average SR
Base (Operator + Progressor) 65.6
41.1
13.7
49.5
+) Action Reflector
71.2
45.1
22.7
55.17
Trajectory Reflector
71.5
46.7
24.7
56.1
Global Reflector
70.5
52.8
58.6
Reflection-on Deneettionoo-Demaaddd
78.7
50.0
29.0
61.6
+) Proactive Exploration
83.6
47.2
26.3
62.9
To assess the impact of hierarchical reflection and proactive exploration on the agent's performance, we conduct a comprehensive ablation study on the AndroidWorld benchmark, as shown in Table
AutoGGLM and V-Droid didn't report the specific results metrics other than the success rate.

---

4
In addition to the overall success rate, we also report the success rate on tasks with different difficulty levels. We can see that each reflector hierarchical reflection architecture proven be effective, especially for medium- and hard-level tasks. The Reflection-on on--nnn strategy leveraged by the Action Reflector can further improve the performance on easy tasks, suggesting that it can reduce useless or even erroneous reflection to improve the robustness. After adding these components, the result achieves 12.1 % SR improvement, demonstrating the effectiveness of our hierarchical reflection architecture. Additionally, incorporating proactive exploration further improves the SR by 1.3%, and 4.9% on the easy tasks. Proactive Exploration provides an opportunity to interact with environment, providing valuable prior knowledge to complete the task. In Appendix F. we provide detailed case study to demonstrate how each module takes effect the actual task execution process.
4.4 Further Analysis
$2.5-
reflection
Namber of falted
$7.5
NvG. Reflecions
10
$5.0
10.0
30
$2.5-
Paanningg
morconation
Ievegeetion
Internction
droundinn
Otter
45.0
0.053
-0.01 40
W/ aarchiccl refiection
Threshold
Figure Confusion matris of the task com- pletion with and without hicrarchical refiee- tion on the AndroidWorld benchmark.
Figure Error type analyxis with and with- hicrarchical reflection the Android- World benchmark.
Figure 5: Performance wer.. different thresholds the Reflection-on-Demann strategy on the AndroidWorid benchmark.
Hierarchical reflection significantly enhances the agent's self-correction capability. As depicted in Figure 3] the hierarchical reflection mechanism successfully rectified 18 failed tasks, achieving 30.51% % correction rate with minimal misjudgment at 7.02%. These findings suggest that the hierar- chical reflection module not only plays pivotal role in the effective correction of task failures but also efficiently mitigates the risk of detrimental reflections. Consequently, the efficiency gained from successful corrections significantly outweighs the inefficiencies caused by occasional misjudgments, highlighting the modulc's positive impact on self-correction processes.
Hierarchical reflection reduces diverse failure modes. In the AndroidWorld benchmark, conducted a detailed error task analysis with and without hierarchical reflection. can be summarized into six failure types: Planning failures, Navigation failures, Interaction failures, Perception failures, Grounding failures, and Other failures, which are detailed in Appendix [ Figure 4 illustrates the impact of hierarchical reflection different types of task failures in the AndroidWorld benchmark. Without hierarchical reflection, failures are distributed across various categories such as planning, perception, navigation, and interaction, with relatively high counts. However, when hierarchical reflection is applied, there a noticeable reduction in the number of failures for most error types, particularly in perception and navigation failure tasks. This indicates that hierarchical reflection effectively improves task success rates by addressing specific failure modes.
Reflection-on-DDe -o--emanddd--D--eeeeeeee--e- strategy enhances both performance and efficiency. Here we evaluate the effect of the Dineectton-on-Demaale strategy leveraged by the Action Reflector. As shown in Figure 5 the threshold 0 = means performing Action Reflection in every step, while 0 -0.001, the
means disabling the Action Reflector. Notably, with a relatively large threshold number of reflections is reduced while the performance is further improved. This is attributed the strategy can avoid some erroneous reflections when the action than
fact that the Reflection-on-Demaan confidence score is high enough, shown in Figure TO When 0 -0.01, although 85% of the reflections are omitted, the decrease in the success rate is less than 1.5%. These results indicate that most of the performance improvement may be attributed some critical reflections. The Reflection-on-Dert DnDemd strategy effectively identifies and preserves these key reflections, enabling great performance while significantly reducing computational overhead.
4.5 Efficiency
To evaluate the running efficiency of MobileUse and the impact of each module, we calculate the average execution time for each task on the AndroidWorld benchmark, shown in Tables 5 and

---

Table 5: Efficiency comparison of different methods on the AndroidWorld benchmark.
Table Efficiency each hierarchical reflection
module on the Andro benchmark.
Method
Time (s Task)
SR
Module
Time (s Task)
M3A (sereen allylylloo tree)
234
25.4
Action Reflector (0 0)
90
M3A (a tree)
150
30.6
Action Reflector -0.01)
10.6
2obile-Agent-v2
319
37.1
MobileUse 0)
Trajectory Reflector
13.2
UsbileUse -0.01)
279
57.8
Global Reflector
7.7
Here, 0 indicates the deactivation of the Reflection-one -n--nmannd mechanism, meaning that the Action Reflection is performed at every step. We can find:
UsbileUse achieves both efficiency and performance gains. Under the same backbone (Qwen2.5- VL-72B), MobileUse with Ceflection-on-Demaane runs faster than Mobile-Agent-v2 and achieves a much higher success rate on AndroidWorld (>20%), demonstrating the effectiveness of our hierarchi- cal reflection design. Although M3A is relatively more efficient, this advantage is not sufficient to offset the performance gap compared MobileUse. Furthermore, M3A includes the al ally tree as input, whereas MobileUse relies solely on screenshots, which makes it more generalizable across wider range of mobile environments.
Hierarchical reflection adds minimal latency. With full reflection enabled, the Action, Trajectory, and Global Reflectors contribute approximately 25%, 3.5%, and 2% of the total time, respectively, strategy, the reflection overhead is significantly reduced
When applying our Reflection-on-Demaan to 10%, while the success rate drops by less than 1%, showing that unnecessary computation is effectively avoided without sacrificing performance.
Proactive exploration achieves cost-effective efficcincy. We set the total number of exploration steps to 100 per app. with a time cost of 19.5 seconds per step, allowing each app's s exploration to be completed within hour. Notably, exploration is conducted only once per app, and the knowledge accumulated is shared across multiple downstream tasks, making the exploration cost amortizable and enhancing its practicality in real-world deployments.
5 Limitations and Future Work
Despite the promising results of MobileUse, there still exists some limitations: 1) MobileUse relies on the strong cestruttio-foolowingg reasoning, and grounding capabilities of the foundational model, which may limit the generalizability smaller models or edge deployment. 2) The current proactive exploration module relics on the agent's inherent capabilities to explore the environment. 3) The tasks requiring human validation (e.g., biometric login, payment confirmation) not explicitly discussed.
In the future, we will try improve the performance MobileUse with smaller models by adjusting the agent framework or developing specialized rision-language models. To further optimize the proactive exploration module, we will design reward-oriented exploration process to improve the exploration efficiency. Additionally, we will involve human-in-the-loop security confirmations into our frameworks to ensure real-world deployment.
6 Conclusion
In this work, we introduce MobileUse, novel GUI agent designed robustly automate complex mobile tasks through hierarchical reflection architecture. Our approach addresses critical challenges in mobile environments, including Long-horizon task robustness, difficulty in error recovery, and cold-start adaptation. Extensive experiments on the AndroidWorld and AndroidLab benchmarks demonstrate that MobileUse establishes new state-of-the-art performance, achieving success rates of 62.9% and 44.2%, respectively. To bridge research and real-world utility, we release the MobileUse Toolkit-a lightweight, modular system for physical mobile device automation. This enables both practical deployment and future research extensions in mobile automation.
10

---

Acknowledgments and Disclosure of Funding
The Shanghai Jiao Tong University team is partially supported by National Natural Science Foun- dation of China (62322603). We also acknowledge the support of the SITU-OPPO Joint Lab on Artificial Intelligence.
References
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. Gradio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint arXiv:1906.022 190.00299,,, 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human, 2024. URL https: org/ 0tte/2410.08166
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang. Ang Li, and Xin Eric Wang. Agent s2: A compositional generalisst-peeiialis framework for computer use agents, 2025. URL https
wttp://.rraabba//0000/0 00906
Anthropic. Developing a computer use model. http:://www.aaturrpic.c.oom comppputenncmmueerrrerr. 2024. Accessed via https//wwwwaatttnpcccccc///////s computer-use.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
Simone Bianco, Luigi Celona, Marco Donzella, and Paolo Napoletano. Improving image captioning descriptiveness by ranking and Ilm-based fusion, 2023. URL /tttt:://txxii..t..//////t/ org/abs/2306
11593
Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. Advancing mobile gui agents: A verifier-driven approach to practical deployment, 2025. URL
https://arxiv.out org. httgg/bb//200 a220033../503..... 2503. 15937
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, al. Magentic-one: generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:241 1.04466, 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https:///penrrevie. wttprreeiiieeeeeeeftrreuee n Lttpr/rrrwreeeeeeeeooooooooooaan diw.moogiiiicc
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The llama 3 herd of models, 2024. https:/:://arxx org/
abs/2407.2172
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, et Cogylm2: Visual language models for image and video understanding. 2024a. https:t::///rxi wwweexiivv.rr//eeo
16500
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14281-14290, 2024b.
11

---

Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions, 2023. URL httpt:ttss//iii....... org hatg/abs/22/ / 09936
Wenbo Hu, Yifan Xu, Yi Li. Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: a simple multimodal Ilm for better handling of text-rich visual questions. In Proceedings of the
AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourfeenth Symposium on Educational Advances in Artificial Intelligence, AAAT22/AAA1244AAA AAAI Press, 2024, ISBN 378-1-57735-887-99 doi: 10. 1009/a/aaai. v3813.27999. URL https://dol. org/10. 1609/ 1609/aaai. v3813 27999
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shudan Zhang, Yuxiao Dong, and Jie Tang. Androidgen: Building an android language agent under data scarcity, 2025. httpbtts////xxi..rrgge 2504. 19298
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991 1919999110,,,9911111110210001111 2023.
Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions, 2024a. URL https://arxiv 2408 11824]
Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui 2: Mastering universal user interface understanding across platforms. arXiv preprini arXiv:2410. 18967, 2024b.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One lisio-lanquage-acttio model for gui visual agent, 2024. URL
17465
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang. Hanlin Zhao, lat Long long, Jiadai Sun, et al. Autoglm: Autonomous foundation agents for guis, 2024. URL https
//arxiv org/ abs/ 2411. 00820
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection, 2025a. WtRL:htts://arzi org/ htrg//b//25000 04575
Yuxuan Liu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, and Rui Yan. Mobilesteward: Integrating multiple app-ore app-oriented agents with self-evolution to automate cross-app instructions. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Data Mining V.I, KDD '25, page 893-893.440 ACM, July 2025b. doi: 10.114457699022447000771 URL http:///xx doi org/ 10. 1 1
Magnus Müller and Gregor Žunič. Browser use: Enable ai to control your browser, 2024. URL https://github wtt./oooo/ooeeeeee hrombrroweer-o www.uuueeuuuuoe baase//bowser-ue
Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, and Kun Shao. Appvlm: lightweight vision language model for online app control. arXiv preprint arXiv:2502.06395, 2025.
12

---

Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang. Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong Webrl: Training IIm web agents via self-evolving online curriculum reinforcement learning, 2025. URL
attps://arxiv.orrt abs
abt////s..... 2211 02337
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501. 122.2,.,.,. 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop, Wei Li, Folawiyo Campbell-A -Ajaa,, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International ntt/ferummieeeee
Conference on Learning Representations, 2025. https:/ 115yUQsrjC
Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leveraging large language models for enriched fused image captions. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5689-5700, January 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv. org/abs/2403.055330 2024.
Junyang Wang. Haiyang Xu, Haitao Jia, Xi Zhang. Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems. 37:2686-2710 10, 2024a.
Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual gui agent towards general computer control, 2024b. /ttpss//:///axx org/ abs/2412. 01268,
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint 2501 2025.
Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, and Ziwei Liu. Gui-reflection: Em- powering multimodal gui models with self-reftection behavior. arXiv preprint arXiv:2506.08012. 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang. Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang. Jiale Liu, et al. Autogen: Enabling next-gen 1lm applications via multi-agent conversation. arXiv preprint arXiv:2308.0811 08155, 2023.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL //tttttst//aattte //tpx/aaaivvv////a//aaa org//bs///////eoooo 2410223.
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng. Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents, 2024a. URL /ttp:///aaiii.v//.
/bwp///........... 24024
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024b.
URL htwph:ttp:////rt //axxiv org/ ab////22..2.2...///// 2212 04454
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li. Chengyuan Li, Dayiheng Liu, et al. Qwen2 technical report, 2024a. URL htttps://aat //axxiv
org/abs/2207 10671.
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions, 2024b. URL https://axxx wwwpeiii.oorrbooa. www.oosg/////e/eoooooo.ooooeeeee /2e...212.... 16256
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1-20, 2025.
13

---

A Broader Impacts
The proposed MobileUse framework advances mobile task automation through its hierarchical reflection architecture and proactive exploration module, offering significant positive implications:
Efficiency and Accessibility Gains. MobileUse automates repetitive or complex tasks (e.g., form filling, multi-step navigation), reducing manual input and cognitive load. This improves efficiency in professional workflows and enhances accessibility for users with physical or visual impairments by enabling seamless interaction via automated gestures and screen parsing.
Progress in Multimodal Interaction. By integrating vision-language models for real-time screen understanding and task execution, MobileUse advances the state-of-the-a in mobile GUI automation. It provides practical framework for applying multimodal models in dynamic real-world scenarios, fostering innovation human-Al collaboration frameworks for mobile platforms.
However, current GUI agent capabilities are still limited, leading to concerns about privacy and security. The automation of mobile tasks by intelligent agents necessarily involves access sensitive on-screen content, user inputs, and application states. Without careful design of data handling and permission control, such systems could increase the risk of privacy leakage, unauthorized actions, or exploitation by malicious actors. This highlights the importance of incorporating strong safeguards and transparent audit mechanisms.
B
Action Space
Table 7:
Action space.
Action Type
Parameters
Description
Perform a key event on the device using ADB syntax.
key
text
Examples include volume_up, power, clear.
click
coordinate
Click the screen at the specified (x, y) coordinate.
coordinate, time
Press and hold on the screen at (x, y) for a specified number of seconds.
long-press
coordinatel, swipe coordinate2
Swipe from the starting coordinate (x1, y1) to the end coordinate (x2, y2).
type text
Input text into the currently focused input box.
clear_text
Clear the content of the active input box.
system_button button
Press a system button: Back, Home, Menu, or Enter.
open text
Launch an app on the device by name.
wait time
Wait for a specified number of seconds.
take_note text
Save observed text the current screen for future use.
answer text
Answer the user query.
Terminate the current task and report whether it was a success failure.
terminate
status
C Failure Types
The failure types AndroidWorld benchmark with and without hierarchical reflection.
Planning failures, whether the agent produces action is incorrect, insufficient, early termination.
Navigation failures, where the agent struggles find a certain element or function, suggest- ing deficiencies in layout understanding and navigation.
Interaction failures, where the agent is unable successfully manipulate an element, reflecting lack of domain knowledge of GUI interactions.
14

---

Perception failures, where the agent is misunderstanding the text content on the screen or the function of the icon.
Grounding failures, where the agent produces inaccurate coordinates for the language description provided.
Other failures, the other types of failures, for example, incorrect answers.
D More Experimental Details
D.1 Experiments Compute Resources
Our experiments training-free frree and utilize two types of computational resources. The first type is the multi-modal large language model service, which we deploy using (L.LM(Wwon et al. 2023) on machine running Ubuntu 20.04 with 8 CPU cores, 100 GB of memory, and 4 A100 GPUs (each with 80 GB of VRAM). The second type of resource is used for running benchmark evaluations, which require CPU cores and 8 GB of memory. We note that in the absence of GPU resources, the multi-modal large language model service can alternatively be replaced by third-party APIs.
D.2 Experiments Different Model Size
Table Success Rate (%) of MobileUse with different model sizes the Android benchmark.
Model
Success Rate
MobileUse (Qwen2.5-VL--B--nsste
21.6
MobileUse Dowen2.5-VL-32B-Insmee
44.4
MobileUse (Qwen2.5- VLL-2BB- Instrrcct)
62.9
Here we change the backbone VLLM of MobileUse assess how model size affects the agent's performance, as shown in Table 8 When using Sween.55-V-77--Instt the success rate becomes very low. This is because the 7B model lacks the basic instruction-followiin ability. The hierar- chical reflection and proactive exploration mechanism are almost useless with the 7B model. The performance of MobileUse with the Qwen2.5-VL ---.55----32B-----ee Instructt model improves lot. Insufficient grounding and complex reasoning capabilities prevent it from achieving better performance. With the 72B model, MobileUse achieves the SOTA results. With accurate grounding and powerful reasoning ability of the foundation model, MobileUse can realize its huge potential for solving complex mobile operation tasks.
E MobileUse Toolkit
We develop the MobileUse Framework lightweight, modular, and pluggable toolkit. Based on Android Debug Bridge (ADB), this toolkit achieves seamless connectivity with a physical mobile device. Integrated with Gradio(Abid et al. 2019), provides a visual web interface where users can input commands through web browser to drive automated smartphone operations and monitor the execution process of the agent in real time. The MobileUse Toolkit enables end-users to achieve one- click smartphone automation and intuitively experience the capabilities of GUI Agents. Furthermore, its flexible design supports the activation, deactivation, and customization of reflection mechanisms at various stages of hierarchical reflection, offering researchers effective platform for experimentation and extension.
As shown in Figure 6 to use the Tookit, firstly users need to connect their phone to the computer via ADB. Next, install the MobileUse ToolKit on the computer and launch WebUl service, which can be opened with the URL. http::/117...0.1::00
in browser. After that, configure the VLM service on the webpage, enter task commands in the input box, and click run. At this point, the MobileUse agent will automatically operate the phone and display the execution process the webpage.
15

---

Precondition
Launch and Setting
200
Siee WeeLll server pythoe metilll, websd. Accoss the browser and setting VLM contguraatio.
computer computer USB cable.
Enable developer
Task Finished. The conteet for Hugo Parsina has been addedd succesalull.
Monitoring: Show the process of the agand real lime@houggt action, screanshal).
Input and Start "Cheate mew contest for Huga Pereins. Thair number is 11880744755
Figure 6: The illustration of MobileUse Toolkit automating mobile phone operations via ADB.
F
Case Study
Figure 7 8 and 9 show how the Action, Trajectory, and Global Reflector work to correct errors for robust task execution. Figure[OOshowe that the Reflection-on-Denaae strategy can avoid incorrect reflection. Figure IT shows how proactive exploration can provide useful knowledge to help with the task execution.
Goal: Record audio clip and save it with name "presentation [Gwr.m4a" using Audio Recorder app.
type(text presentati on (Gwr.m4a)
coorc(cood (540, 740))
clear text()
Audio Recordar
Action Reflector
Audio Secorter
Audio Recorter
The lust action did not meet the expectation. The text field was not cleared before typing "presentation fGwr, m4a." resulting in the text "Record- Ipresentation fGwr. m being entered instead of just "presentation iGwrm4a." " This indicates that the action of clearing the text field performed before typing the new name.
q
a
Figure 7: A case of the Action Reflection. The Operator doesn't realize that there a default name in the input box and enters the new file name directly. The Action Reflector detects this error and provides feedback to the Operator. In the next step. the operator successfully clears the text box and enters the correct file name based on the feedback information.
16

---

Goal: Add the following recipes into the Broccoli app: Recipe: Chicken Caesar Salad Wrup description: An ideal recipe for experimenting with different flavors and ingredients, servings:
snipetcoo dieate (54),
clicffoacdinnte- (540, 1543y)
chickCanardinat- (53, 1575))
click/coeddinnts (540, 15431)
clickrcocddinatc (53, 1575)1
Io0 (54
(94))
Trajectory Reflector
history operations are repeatedly attempting
close the clipheard and click
Servings fickl,
are not achieving intended
reall: The cligbosrd remnina
open, and the mser is unable
servings ... The user noads find diffirent upproach or troubleshoot the issue with clipheand proceed entering the recipe detail..
Figure 8: A case of the Trajectory Reflection. The Operator wants to click on the input box below, but due to the presence of the keyboard, the Operator is stuck in the wrong action loop. The Trajectory Reflector detects repeated actions and guides the operator in trying different operations. Finally, the Operator finds the new input box by swiping up.
Goal: Open the file haml in Downloads in the file manager; when prompted open with Chrome. Then navigate the to the bottom-right cell, by using the direction buttons.
click(coordinate-
terminate(statuss
clock(coreeinater
(198, 823))
success)
823))
coammpion
Global Reflector
The X is not in the hottom- right cell of the grid. The bottom-right cell is green, and the X is in the cell to the left of it. Therefore, the task of navigating the X to the bottom-right cell has not been completed.
Figure 9: A case of the Global Reflection. The operator doesn't 't complete the user's instruction and ends the task prematurely. The Global Reflector successfully detects this error and feeds back to the operator. Finally, the task is successfully finished.
17

---

Goal: Create a timer with 0 hours, 16 minules, and 35 seconds. Do not start the timer.
w/o Reflection-on-demand
coock(cooddinate-
click(cnordinate- (818, 826))
click(coordinate- (818, 1645))
(818, (106))
Action Reflector
The latest action was intended to tap the '3' key to the seconds to 35. However, the screenshot after the action shows that the timer 01 minute and seconds, which is not the expected result. This indicates that the action did not correctly set the timer to hours, minutes. and 35 seconds. The failure may be due to an incorrect tap or misinterpretation timer's 's inpur method.
O0.. 01
O0..
16.
O 1... 63.
2
2
2
4
5
6
9
7
8
00
0
the
w/ Reflection-on-demanee threshold -0.011
click(coordinate=
click(coordinate= (831, 837))
coock(cood (540, 1096))
terminate statuss success)
(831, 1116))
01
OO..
16.
0 1.. 63.
00.
1 6.. 35.
2
3
4
5
7
8
9
7
8
00
0
00
00
0
00
p
00.00031
p = 0.0004
p
= 0.,00366
=
0.00003
Figure 10: case of the Reflection-on -Demnnd strategy. When the Reflection-on-Denee Demand strategy is not used, the Action Reflector generates a wrong reflection, causing the Operator to make an incorrect action in the next step. When the R -Dmendd strategy is used, since the confidence score of each step is higher than the threshold, the Action Reflector is not triggered, and the Operator successfully completes the task.
18

---

Goal: What are my high priority tasks in Tasks app? Answer with the titles only. If there are multiples titles, format your answer in a comma separated list.
w/o Proactive Exploration
w/ Proactive Exploration
clickcueolirsee-
(415, (838))
answentent Reply t0 Emails, Water Pante)
ovipcfoounninsatt- (340, 1972).
coondinonen 1-90, 1088))
sooweflexx-Rovic performncc metris)
1
14.3
Experience
My Staks
IB
Masth Pick
Different colors checkboxes indicate varying levels priority or status, For example, red checkbox might signify high- priority task.
Arhes
ching
Meat
Figure 11: A case of the Proactive Exploration. Without exploration of the task app. the agent doesn't 't know which tasks are high priority and gives incorrect answers. After performing proactive exploration, the agent acquires knowledge about high-priority tasks, which helps him find and identify high-priority tasks during task execution and finally generates the correct answer.
G Examples of the Collected Knowledge in the Proactive Exploration Stage
Figure 12 13 and show the Knowledge collected from the Tasks, RetroMusic, and Camera apps.
Knowledge collected from the Tasks app.
1, Clicking microphone the pp main page does not sorm produce any visible changes the curment view. The task remains same, and no new elements or options appeac: This suggests that the microphone loon might be intended for voice input.or or commands but its effects
are not visually repreeented the cummnt display.
2, The Ag main page displeys a list of tasks with checkboxes for completion status. Different colors in the checkboxes indicate varying levels of checkbox might signify a high priority task, while yullow checkbox could indicate a medium- priority task, blue checkbox with checkmark signifies completed task.
priority or status. For example,
3, The task list includes dotailed descriptions each task, such "Send an update" for attending triniing session "Follow up with others' cleent follow-up. up. T These desctiptions provide additional context for the tasks.
4. The bottom the pp main page feature floating action button with a plus sign, which is likely used to add new tasks. Other icons the bottom, such the microphone and menu icons, offer additional functionalitess though their specific effects are not visually apparent in the current view.
5, the task creation section, clicking on the "Task name" field allows for the input of a new task name. The keyhoard appears, enabling the user type the task details. The task creation section also includes ophons for setting start date, due date, and prontty level, as well as adding location and tags. The prlorty levels are indicated by different colored circles: gray for low, blue for medium, yellow for high, and red for cuttaal.
6. In the location selection interface, elicking the red circular icon with location symbol centers the map on the current location. This action does not change the visual elements signifrantly but adjusts the map's focus to the user's curvent geographical position. The interface includes search bar for entering location names and prompt tn 'elecc this location. indicaling the user conffim the chosen location for further actions.
7, When "Missing permissions' dialog appears, indicating that location permissions are needed find the current location, clicking the "OK" button does not dismiss the dialog. This saggests that the A pp require explict permissionn handling for location services, and the must grant these permissions through the device's settings ora sabsequent prompt.
8, In the certificatte veriffcaatoon interface, the pp prompts the
trust an unknown certificate. The interface displeys the X 509 certificate details, including the issuer, validity period, and fingerprints. The user can choose to accrpt or reject the cortificate. Selecting the "A CCEP option does not change the interface, indicating that additional steps conditions, such manual verification the fingerprint, might required before the curtificato is trusted.
Figure 12: Knowledge collected from the Tasks app.
19

---

Knowledge collected from the Retro app.
1, The App main page provides options like History, Last added, Most playod, and Shuffle, each with distinct icons for quick access,
2. The Suggestions section displays a New Music Mix and other music options, allowing users explore new tracks.
3. The Recent artists section lists artists like George, enabling users access their music directly.
4. The Recent albums section shows albums such as Music, providing easy navigation to recent music collections.
5. The bottom navigation bar includes icons for For you, Music, and other catagories, facilitating seamless navigation within the pp-
6, The artist detail section for Georye includes options like Play all and Shuffle, allowing users to listen to the artist's music in
different ways. s. The section also displays the artist's album, providing direct access to their music collection.
7. The Song section under the artist detail for George lists individual tracks like "City of Stars," enabling users to play specific songs directy. The interface supports swiping to reveal more content, enhancing user interaction and exploration of the artist's music.
8, The artist detail section for G George includes a visual representation of the artist with a placeholder icun, providing clear and organized view of the artist's information and music options.
9, Clicking on a song title in the Song section under the artist detail for Goorge initiates play blayback of the selected track, as indicated by the now- now-play playing bar at the bottom of the pp main page. This feature allows users to start listening to specific song immediataly. 10, The album detail section for Music includes options like Play all and Shuffle, allowing users to listen to the album's music in different ways. The section also lists individual songs like Bright Lights' and "Chasing Shadows." enabling users to play speciftc tracks directly. The interface supports swiping to reveal more content, eabancing user interaction and exploration of the alb music.
11, The artist detail soction for Goorge includes a visual representation of the artist with a placeholder icon, providing clear and organized view of artist's information and music options. The interface supports swiping to reveal more content, enhancing user interaction and exploration
the artist's music.
12, The now playing ing interface shows the currently playing song. "City of Stars' by George, with progess bar indicating the elapsed time. It includes playback controls such as pause, skip, and shuffle, allowing users to manage their music playback. A dditionnl options like adding the song to a playlist or queue are also available, enhancing the user's music li listening experience. 13. The now pla-payyng interface also features thmee dot menu that reveals additional options for the song, including Playback Speed, Drive mode, Go to album, Go to artist, A to playlist, Save playing queue, Share, Set as ringtone, Clear playing queue, Details, Go to Lyrics, Equalizer, and Delete from device. These options provide users with extensive control over their music experience and management.
14, The Share option in the now-plas now-playing interface allows users to choose what they want to share, such as the audio file, a status update about the currendy playing song, or a story. This feeture enhances the user's ability to share their music experience with others. 15, After selecting the "Share story" option, the interface displays a preview of the story to be shared, including the song title "City of
Stars" and the artist name "George." The 'Share to Stories" button allows users finalize and share the story, providing a seamless way to share their current music experiecce.
16, The interface after clicking the back arrow from the "Share story" preview returns to the now- pow-playing interface, where users continue managing their music playback and access additional options. This ensures a smooth transition and maintains the user's control over their music experience.
17. The now plw-playng interface includes a lyrics section that can be accessed by clicking the lyrics icon. If no lyrics are found, a message indicating "No lyrics found" is displayed. This feature allows users to view synchronized lyrics while listening to the song, enhancing their music listening experience.
18, Swiping the now- www-payiig interface reveals more content, such as the progress bar and additional playback controls, enhancing the user's ability to manage their music playback and explore more options.
19, The now playing interface includes sleep timer feature that allows users to set specific duration for the music to play before stopping. Users can adjus jst the timer using a slider and choose whether to finish the last song before stopping. The "Start" button initiates the sleep timer, providing convenient way to manage music playback during sleep or rest peri ods.
20. After setting the sleep timer and retuming to the now playing interface, users can continue managing their music playback and
access additional options, ensuring seamless and uninterupttd music experience.
21. The now- playing interface includes "Now playing queme" section that displays the current song and upcoming tracks. Users can manage the queue by adding or removing songs, and there is "Clear queue" button to remove all songs from the queue. This feature provides users with control over their music play list and allows for easy queue management.
Figure 13: Knowledge collected from the RetroMusic app.
20

---

Knowledge collected from the Camera app.
1, The pp main page provides options for sclocting different modes, such as "MODE LIST" and "FILMSTRIP. Clicking "MODE
LIST" reveals menu with "Camera" and "Video" options, indicating the ability switch between photo and video modes.
2. The App main page includes
at the bottom, which likely to configuration on options for adjusting resolution, quality, and advanced foatures. 4. The "Z+" and "Z-" buttons at the top indicate functionality for zooming in and out eabancing the user's control over the view.
shortcut capbure photos videos directl.
3, A settings icon is visible, suggesting
5. The pp main page displays room scene with TV, bookshelf, and window, providing e visual context for the camera's view.
6, After clicking 'MODE LIST", the pp main page transitions to view where the camera is roady to capture images, with the
camera icon the bottom becoming active and ready for use. The three menu at the bottom right likely provides additional
options settings related to the current mode.
7, Clicking 'FILMSTRI transitions the pp
view, which appears be blank loading state, possibly indicating
section for viewing or managing captured images videos.
8, The three-de dot menu in the top right comer of the pp main page reveals "Details" option, suggesting additional information or
settings can be accessed from this menu.
Figure 14: Knowledge collected from the Camera app.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper's contributions and scope are clearly claimed in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
We include seperate "Limitations" section in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We release the code for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have detailed the experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We can't conduct significance experiments since most of the basclines are not open-source.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide detailed information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics lattps://neurpp:: http://pww...pcu..oocccccccccaaa
Answer:
[Yes]
Justification: We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No data or model is released in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We state the used asset in the README file of our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed document for our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We use Qwen2.5-VL-72B -nnstuutttt as backbone model.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M policy atte::e//eerrrps.cc/o/ffeeeeeeee for what should or should not be described.
28


Reviews:
[{"review_id": "G9GpAESUCD", "reviewer": "Reviewer_c8Yj", "summary": "The paper introduces **MobileUse**, a hierarchical reflection-based GUI agent designed for robust and adaptive task execution on mobile devices using multimodal large language models (MLLMs). It tackles three core challenges in mobile automation: long-horizon task execution, error recovery, and cold-start scenarios in unfamiliar environments. The framework consists of two key components: a Hierarchical Reflection Architecture with Action, Trajectory, and Global Reflectors that self-monitor across temporal scales via a \"Reflection-on-Demand\" mechanism, and a Proactive Exploration Module that enables learning from open-ended interactions without predefined tasks. MobileUse achieves state-of-the-art success rates of 62.9% on AndroidWorld and 44.2% on AndroidLab, and is accompanied by an open-source toolkit for real-world deployment.", "questions": "In addition to the weaknesses noted above, I have a few minor questions for clarification:\n\n1. Cross-App Knowledge Transfer: To what extent can the knowledge acquired during proactive exploration (e.g., identifying a red checkbox as indicating high priority) generalize across different app versions or to similar apps?\n\n2. Edge Deployment Feasibility: Considering MobileUse’s reliance on large-scale language models, is it doable for distilling or optimizing the agent to enable deployment on resource-constrained devices (e.g., mobile or edge hardware)?\n\n3. Privacy Concerns: Since MobileUse processes visual data directly from user screens—which may include sensitive information such as messages, financial details, or personal identifiers—what mechanisms are in place to ensure user privacy and data security?", "limitations": "Please refer to the Weakness and Limitation sections.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## **Strengths**  \n1. Effective Architecture\nThe paper introduces MobileUse, a well-structured hierarchical reflection mechanism that catches errors across different execution levels, offering clear advantages over flat, monolithic models. The Reflection-on-Demand strategy further enhances robustness by triggering reflection only when needed, striking a smart balance between efficiency and error recovery.\n\n2. Generalization and Adaptability\nThe proactive exploration phase helps the agent with prior knowledge of apps, enabling effective adaptation without relying on labeled data. \n\n3. Real-World Validation\nWith the release of the MobileUse Toolkit, the system demonstrates strong practical relevance by supporting real-device automation. The paper also includes thorough evaluation—ablation studies, confusion matrices, and failure analysis.\n\n## **Weaknesses**  \n1. Scalability and Accessibility Concerns\nMobileUse depends on large models (up to 72B parameters), with smaller variants (e.g., 7B, 32B) exhibiting significant performance degradation. This will make the model rely on internet access and limit the approach’s accessibility, practicality, and potential for deployment on resource-constrained edge devices (i.e., phones).\n\n2. Incomplete Evaluation of Generalization and Efficiency\nThe paper lacks an analysis of MobileUse's ability to generalize to entirely novel or unseen apps. It also ignores any discussion of computational cost or latency introduced by the reflection modules—critical factors for real-time use on mobile devices.\n\n3. Limited Adaptation: The system does not explicitly handle tasks requiring human validation (e.g., biometric login, payment confirmation), which are common in real-world apps. Additionally, while proactive exploration helps, the paper lacks in-depth discussion on how MobileUse handles app version changes, OS-specific UI variations, and different devices/systems.", "was_revised": true, "final_justification": "I have read through the authors’ rebuttal and other reviewers’ comments. While I agree that the method and novelty could be improved (based on Reviewer 1iG5, it currently feels more like an integration of existing methods). I remain concerned about the generalization ability across different mobile systems, devices, layouts, etc., as well as privacy issues (specifically, whether all processing can be done locally on-device). Overall, the paper is good, and the authors’ responses have addressed most of my concerns. I will keep my current rating.", "rebuttal": "Thank you for the thoughtful review and for highlighting the strengths of our work—especially the hierarchical reflection design, Reflection-on-Demand strategy, proactive exploration method, and real-world relevance. We address your concerns point-by-point below.\n\n---\n\n> Weakness 1: Scalability and Accessibility Concerns MobileUse depends on large models (up to 72B parameters), with smaller variants (e.g., 7B, 32B) exhibiting significant performance degradation. This will make the model rely on internet access and limit the approach’s accessibility, practicality, and potential for deployment on resource-constrained edge devices (i.e., phones).\n\nThank you for raising this important point. We agree that the reliance on large models presents challenges for edge deployment and have discussed it in Section 5. Here we'd like to make a further clarification:\n- **Limitations of Small Models**: As analyzed in Appendix D.2, smaller models often struggle with the complex reasoning and grounding capabilities. Our experiments show that while smaller models can execute simpler tasks, their performance degrades significantly on tasks involving fine-grained perception and complex reasoning.\n- **Industry Practice**: Currently, both academic and industrial GUI agents [1, 2] rely on cloud-based large models due to these performance gaps. **MobileUse aligns with this trend to explore the upper-bound of automation capabilities.**\n- **Ongoing Optimization**: We are actively developing a training pipeline to fine-tune smaller models using GUI trajectories. Early results are promising, and we plan to report them in future work.\n\n---\n\n\n> Weakness 2.1: Incomplete Evaluation of Generalization and Efficiency The paper lacks an analysis of MobileUse's ability to generalize to entirely novel or unseen apps. \n\nThank you for raising this important point. We would like to clarify that before introducing the proactive exploration module, MobileUse was evaluated in a **cold-start setting, without being deliberately trained or adapted to unseen apps** (e.g., 61.6% on AndroidWorld), demonstrating the core architecture's strong generalization ability.\n\nFurthermore, we observed in certain scenarios that even large MLLMs can struggle without prior exposure to common UI semantics. To address these shortcomings, we designed the **Proactive Exploration module to to improve robustness and adaptation to unseen app variants**.\n\n---\n\n> Weakness 2.2: It also ignores any discussion of computational cost or latency introduced by the reflection modules—critical factors for real-time use on mobile devices.\n\nWe thank the reviewer for highlighting this important point. We have conducted a detailed runtime breakdown, and our analysis shows that **the overhead introduced by our hierarchical reflection framework is marginal and well justified by the performance gain**.\n\nAs shown in the table below. The first MobileUse result corresponds to performing Action Reflection at every step, while the second uses Reflection-on-Demand (threshold = -0.01). We can find that:\n- **Hierarchical reflection adds minimal latency**: With full reflection enabled, the Action, Trajectory, and Global Reflectors contribute approximately 25%, 3.5%, and 2% of the total time, respectively. When applying our Reflection-on-Demand strategy, the reflection overhead is significantly reduced to 10%, while the success rate drops by less than 1%, showing that unnecessary computation is effectively avoided without sacrificing performance.\n- **Our method achieves both efficiency and performance gains**: Under the same backbone (Qwen2.5-VL-72B), MobileUse with Reflection-on-Demand runs faster than Mobile-Agent-v2 and achieves a much higher success rate on AndroidWorld (>20%), demonstrating the effectiveness of our hierarchical reflection design.\n\n| Method | Module | Time ($s$/task) | AndroidWorld_SR (%) |\n|---|---|---|---|\n| Mobile-Agent-v2 | Total | 319 | 37.1 |\n| MobileUse | Total | 359 / 279 | 58.6 / 57.8 |\n| MobileUse | Action Reflector | 90 / 10.6  | - |\n| MobileUse | Trajectory Reflector | 13.2 | - |\n| MobileUse | Global Reflector | 7.7 | - |\n\n---\n\n> Weakness 3.1: Limited Adaptation: The system does not explicitly handle tasks requiring human validation (e.g., biometric login, payment confirmation), which are common in real-world apps. \n\nWe appreciate the reviewer’s observation regarding real-world tasks that require explicit human involvement. We would like to offer several clarifications and forward-looking directions:\n- Current Scope: As GUI agents remain an emerging research direction, current efforts—including ours—**focus primarily on increasing robustness and success rates in autonomous mobile operation tasks** (e.g., navigation, information seeking, settings adjustment). These tasks represent a large fraction of real-world mobile usage and do not typically involve human-in-the-loop security confirmations.\n- In practical deployment, we envision two complementary strategies for handling such tasks within the MobileUse framework: (i) **Pausing and Deferring**: We can extend the operator’s action space to include a call-user operation, allowing MobileUse to recognize gated states and defer task execution until user validation is complete. (ii) **Specialized Scene Detectors**: We can integrate lightweight classifiers or on-device detectors to identify scenarios requiring human validation.\n- **Framework Flexibility**: Importantly, MobileUse’s modular design—with separable operator, reflection, and progressor modules—allows for seamless incorporation of either strategy without architectural modification. \n\n---\n\n> Weakness 3.2: Additionally, while proactive exploration helps, the paper lacks in-depth discussion on how MobileUse handles app version changes, OS-specific UI variations, and different devices/systems.\n\n> Question 1: Cross-App Knowledge Transfer: To what extent can the knowledge acquired during proactive exploration (e.g., identifying a red checkbox as indicating high priority) generalize across different app versions or to similar apps\n\nWe thank the reviewer for this insightful question.\n\n**The knowledge accumulated during Proactive Exploration is stored in a universal text format, allowing it to generalize across environments.** Taking the Chrome browser as an example, which is found in both AndroidWorld and daily usage: The exploration module summarizes operations such as searching, recommendation discovery, shortcuts to popular websites, information discovery, app downloading and installation, settings, and file opening. We find that most of the summarized knowledge remains consistent across different app versions and operating systems. Since the knowledge does not involve UI interface information, it is unaffected by UI changes that do not alter interaction logic.\n\nWhile there may be some incompatibility issues when transferring knowledge across very different apps, these challenges can typically be addressed with further exploration, adapting the learned experiences from similar apps.\n\nWe will revise the manuscript to further clarify the significance of our proactive exploration and its transferability.\n\n---\n\n> Question 2: Edge Deployment Feasibility: Considering MobileUse’s reliance on large-scale language models, is it doable for distilling or optimizing the agent to enable deployment on resource-constrained devices (e.g., mobile or edge hardware)?\n\nThank you for this valuable question. One of the strengths of MobileUse is its **modular and model-agnostic design, allowing the architecture to be adapted to smaller, more efficient language models for edge deployment**.\n\nTo this end, we are actively exploring model adaptation strategies to improve performance under limited resources. Specifically, we are constructing targeted trajectories and reflection data to fine-tune smaller MLLMs. Early exploration has shown promising results while offering significantly lower memory and latency costs. We see this as a promising direction and plan to present a more comprehensive evaluation in future work. We will also add a brief discussion on this in the revised version of the paper to clarify MobileUse’s potential for lightweight deployment.\n\n---\n\n> Question 3: Privacy Concerns: Since MobileUse processes visual data directly from user screens—which may include sensitive information such as messages, financial details, or personal identifiers—what mechanisms are in place to ensure user privacy and data security?\n\nWe take privacy very seriously. We'd like to clarify that MobileUse **does not store any user data in the cloud**. For development and testing, MobileUse supports running entirely within **Android emulators**, where no real user data is ever exposed. Besides, MobileUse is **compatible with both local and cloud-based deployment** of the underlying language model. Whenever feasible, we recommend local deployment to keep all computation on-device and avoid transmitting screen content to external servers.\n\nAs part of future work, we plan to further explore lightweight local models (as mentioned earlier), as well as integrate data encryption, secure computation, or even federated inference techniques to enhance privacy guarantees in real-world settings.\n\nWe emphasize that the code is **open-sourced**, and we welcome community review and improvement to further strengthen privacy safeguards. We will include a short section on privacy in the final version of the paper to make these points more explicit.\n\n---\n\n> Paper Formatting Concerns: The reference format does not follow NeurIPS.\n\nWe sincerely thank the reviewer for the careful reading and concerns regarding formatting. We will correct all formatting issues to align with NeurIPS standards in the revision.\n\n---\n\n[1] Ui-tars: Pioneering automated gui interaction with native agents.\n\n[2] Agent s: An open agentic framework that uses computers like a human."}, {"review_id": "GWBgNgZiat", "reviewer": "Reviewer_8sCX", "summary": "The paper presents MobileUse, a GUI agent designed for automating complex tasks on mobile devices. It introduces a hierarchical reflection architecture that enables self-monitoring and error correction during task execution, enhancing task success rates. Additionally, the proactive exploration module helps the agent accumulate knowledge in unfamiliar environments, addressing cold-start issues. The empirical results demonstrate state-of-the-art performance on benchmark tests, and the authors provide an out-of-the-box toolkit for practical applications.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel Approach: The focus on reflection rather than traditional planning is innovative, setting MobileUse apart from existing work that primarily emphasizes planning strategies.\n2. Unique Hierarchical Reflection Design: The hierarchical reflection mechanism is a fresh concept, addressing potential errors from single-dimension reflection by incorporating multiple levels of reflection granularity.\n3. Significant Results: MobileUse achieves state-of-the-art performance with a 62.9% success rate on the Android World benchmark, underscoring the effectiveness of the proposed method.\n\nWeaknesses:\n1. Impact of Proactive Exploration: For a fair comparison, the effects of the exploration phase should be isolated, as other methods do not involve prior familiarization with the environment. Without the exploration phase, the combination of the progressor, operator, and reflectors achieves a success rate of 55.2%, which mirrors existing frameworks without novel contributions [1]. Only the global reflector shows a substantial improvement with an additional 2.5%.\n2. Sensitivity of Reflection-on-Demand: While the success rate increase is notable (+3%), the Reflection-on-Demand strategy seems quite trick-oriented. Ablation studies indicate that the entire system is highly sensitive to the chosen threshold, which requires manual tuning on benchmarks. Additionally, this threshold may not be applicable across different models, posing limitations on its generalizability. Moreover, this approach is suitable mainly for open-source models, restricting its applicability in certain scenarios.\n3. Sensitivity to Model Size: MobileUse's performance is heavily dependent on the model size. It would be beneficial if the authors could provide an analysis of the capabilities lacking in smaller models, which contribute to the drop in performance.\n4. Typographical Errors: Upon reviewing the manuscript, I found a few typographical errors:\n  ○ In Section 3.1, \"pre-compute\" should be hyphenated as \"pre-compute.\"\n  ○ In the same section, the word \"superscript\" appears twice and could potentially be simplified or clarified.\n  ○ The phrase \"reflection-on-demand strategy\" could benefit from consistent capitalization throughout the document.\n[1]: Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration", "was_revised": true, "final_justification": "After reading the response and the other reviewer's reviews, I appreciate the efforts and additional experiments of the authors, most of my concerns have been addressed and I have no doubt on the experiment results while also recognizing it mainly improves upon existing multi-agent framework on reflection part, therefore I maintain my positive score.", "rebuttal": "Thank you for your detailed feedback and constructive comments on our manuscript. We appreciate your recognition of the novel aspects of MobileUse and the significant results presented. We would like to address your concerns and provide additional clarifications as follows:\n\n---\n\n> Weakness 1: Impact of Proactive Exploration: For a fair comparison, the effects of the exploration phase should be isolated, as other methods do not involve prior familiarization with the environment. Without the exploration phase, the combination of the progressor, operator, and reflectors achieves a success rate of 55.2%, which mirrors existing frameworks without novel contributions [1]. Only the global reflector shows a substantial improvement with an additional 2.5%.\n\nWe sincerely thank the reviewer for raising this important point. To clarify, Table 4 in our paper provides a detailed breakdown of the performance with and without the proactive exploration phase. **The success rate without proactive exploration is 61.6%, demonstrating a 6.4% absolute improvement** compared to the setting that only includes the progressor, operator, and single-step reflection. This clearly shows that our hierarchical reflection architecture contributes significantly to performance even in the absence of proactive exploration.\n\nWe would also like to emphasize the complementary roles of each reflection component:\n- Reflection-on-Demand and Action Reflection: The Reflection-on-Demand mechanism significantly improves efficiency by triggering reflection only when execution confidence drops, thereby reducing unnecessary computations and avoiding overcorrection of correct actions, achieving performance improvement while reducing the overhead. **Our designed Action Reflection combines the Reflection-on-Demand mechanism and perception module, which is significantly different from the traditional step-level reflection and has achieved a huge performance improvement**.\n- Trajectory Reflection: This module helps identify accumulated, latent errors and **ensures consistency throughout multi-step task execution**. This is especially important in real-world mobile scenarios, where GUI agents frequently fall into loops or repeated actions due to accumulated mistakes.\n- Global Reflection: This module **addresses hallucination and semantic drift** by enabling the agent to reassess its state from a global perspective. As reported, it alone contributes a notable +2.5% success rate gain, illustrating its impact.\n\nCollectively, these reflection modules work synergistically to enhance MobileUse's robustness and fault tolerance, which we believe **constitutes a meaningful and novel contribution beyond existing frameworks**—even before considering proactive exploration.\n\n---\n\n> Weakness 2: Sensitivity of Reflection-on-Demand: While the success rate increase is notable (+3%), the Reflection-on-Demand strategy seems quite trick-oriented. Ablation studies indicate that the entire system is highly sensitive to the chosen threshold, which requires manual tuning on benchmarks. Additionally, this threshold may not be applicable across different models, posing limitations on its generalizability. Moreover, this approach is suitable mainly for open-source models, restricting its applicability in certain scenarios.\n\nWe appreciate the reviewer’s comments regarding the Reflection-on-Demand strategy and its potential sensitivity to the threshold value.\n\nFirst, we would like to clarify that the range of thresholds shown in Figure 5 was deliberately selected to cover a wide interval so as to clearly illustrate the performance trend. In practice, the threshold offers ample flexibility: for example, when the threshold is lowered to -0.01, over 90% of reflections are filtered out, yet the overall success rate remains largely stable. This demonstrates that **our system is robust to threshold variation and not overly sensitive to specific tuning**. The choice of threshold can therefore be adjusted based on available computational resources or latency requirements, making it a controllable and practical design.\n\nMoreover, we respectfully disagree with the notion that the Reflection-on-Demand mechanism is “trick-oriented.” In fact, **it is built on a meaningful insight: Reflection, while useful, can introduce errors or inefficiencies when applied excessively**. Our strategy leverages the model’s internal confidence to selectively suppress reflection when the model is already confident, and retain reflection when the model is uncertain. This allows us to achieve a dual benefit in both performance and efficiency, by avoiding misleading corrections and focusing computational effort where it is most needed.\n\nIn terms of generalizability, the threshold is derived from token-level log probabilities, which are **inherent to the language model itself and independent of specific tasks or environments**. Therefore, while the optimal value may vary slightly depending on the model used, it is generally model-dependent but task-agnostic, and requires only once-per-model calibration in practical applications.\n\nLastly, we would like to point out that many closed-source models (e.g., GPT-4, GPT-4o) also expose token-level logprobs or similar confidence signals through their APIs. Thus, **our approach remains compatible with a wide range of both open- and closed-source models**, and is not fundamentally restricted in scope.\n\nWe will revise the manuscript to clarify these theoretical motivations, practical considerations, and compatibility aspects in more detail.\n\n---\n\n> Weakness 3: Sensitivity to Model Size: MobileUse's performance is heavily dependent on the model size. It would be beneficial if the authors could provide an analysis of the capabilities lacking in smaller models, which contribute to the drop in performance.\n\nWe thank the reviewer for pointing out the performance dependence on model size. In Appendix D.2, we already provided empirical results using smaller vision-language models—Qwen-2.5-VL-7B and Qwen-2.5-VL-32B—which achieved 21.6% and 44.4% success rates, respectively. We also included a specific analysis of the performance degradation from different aspects. Here, we elaborate further on the specific capability gaps we observed in smaller models that contribute to this drop:\n- **Instruction-Following Ability**: We found that Qwen-2.5-VL-7B often fails to follow structured instructions and struggles to produce actions in the correct format (e.g., missing keys in JSON actions), which is a critical requirement for acting as an agent. This limits its usability without significant post-processing or constrained decoding.\n- **Grounding Ability**: Both the 7B and 32B variants frequently exhibit grounding errors, where the model misidentifies UI elements or fails to map visual features to corresponding semantic entities. Since grounding is arguably the most essential skill for any GUI agent, this significantly impacts task success.\n- **Planning and Reasoning Ability**: Smaller models also lack strong task-level reasoning capabilities. They often fail to understand the global objective of the task or break it down into coherent subgoals, resulting in inefficient or incorrect action sequences. Moreover, their limited reasoning ability undermines the effectiveness of our reflection modules, which rely on the model’s capacity to infer causes of failure and plan recovery.\n\nWhile these limitations are inherent in current smaller models, we see promising directions for future work. Specifically, we plan to explore instruction tuning and reinforcement learning-based post-training to enhance small models’ alignment with GUI agent demands. We believe that with task-specific fine-tuning, even lightweight models can become viable components within the MobileUse framework, enabling broader applicability in resource-constrained environments.\n\n---\n\n> Weakness 4: Typographical Errors: Upon reviewing the manuscript, I found a few typographical errors: ○ In Section 3.1, \"pre-compute\" should be hyphenated as \"pre-compute.\" ○ In the same section, the word \"superscript\" appears twice and could potentially be simplified or clarified. ○ The phrase \"reflection-on-demand strategy\" could benefit from consistent capitalization throughout the document.\n\nWe sincerely thank the reviewer for the careful reading and detailed suggestions regarding writing and formatting. We will address all noted issues in the revision. We appreciate your attention to these details, which will help us improve the overall quality and readability of the paper."}, {"review_id": "M1thXFPlkj", "reviewer": "Reviewer_1iG5", "summary": "This paper introduces MobileUse, a GUI agent leveraging MLLMs to automate complex tasks on mobile devices. Addressing key challenges such as long-horizon task execution, error recovery, and the cold-start problem, MobileUse incorporates a hierarchical reflection architecture for dynamic self-monitoring and efficient error handling at multiple temporal scales. It further features a proactive exploration module, allowing the agent to adapt to new environments through self-guided exploration. Evaluations on AndroidWorld and AndroidLab benchmarks show MobileUse outperforms prior methods on most metrics.", "questions": "This paper claims that Proactive Exploration can help accumulate common knowledge, but its experimental results are not satisfactory. Proactive Exploration also incurs a time cost. If there is a domain gap between the environment explored and the experimental environment, can Proactive Exploration still be effective?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and clearly articulated, and the supplementary material provides detailed information about the experimental implementation.\n\n2. This paper achieves strong performance on both the commonly used AndroidWorld and AndroidLab datasets.\n\nWeaknesses:\n1. The technical contribution of this paper is limited. The main contributions lie in incorporating reflectors into the GUI automation framework and leveraging historical operation information to improve operational accuracy, both of which have already been explored in prior work [1,2,3].\n2. The paper is overly engineering-oriented. The method relies on multiple rounds of inference to achieve better performance at the expense of efficiency, which may negatively impact its practical applicability.\n3. Introducing three reflectors in the GUI automation framework inevitably increases inference time. A quantitative comparison should be provided, but such analysis is currently missing in the paper.\n4. As shown in the ablation results (Table 4), the model in the fourth row achieves better performance on Medium and Hard tasks, whereas Reflection-on-Demand and Proactive Exploration primarily benefit Easy tasks and can even significantly reduce performance on more difficult tasks. The paper lacks any explanation for this phenomenon. Since Easy tasks constitute the majority in the AndroidWorld benchmark, the full model appears strong overall, but these results suggest that Reflection-on-Demand and Proactive Exploration offer limited benefit for more challenging scenarios.\n5. The paper does not sufficiently describe the implementation details of Proactive Exploration—specifically, how the agent is guided to explore the GUI without explicit instructions, the duration of exploration, or the number of exploration steps taken.\n\n[1]. Wang J, Xu H, Jia H, et al. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration[J]. arXiv preprint arXiv:2406.01014, 2024.\n\n[2]. Agashe S, Han J, Gan S, et al. Agent s: An open agentic framework that uses computers like a human[J]. arXiv preprint arXiv:2410.08164, 2024.\n\n[3]. Yu X, Peng B, Vajipey V, et al. Improving Autonomous AI Agents with Reflective Tree Search and Self-Learning[C]//The Thirteenth International Conference on Learning Representations. 2024.", "was_revised": true, "final_justification": "Through the rebuttal, the authors have re-clarified the technical contributions of the paper, time overhead, and some module contributions. The subtle differences clarified between the proposed approach and existing methods do not constitute strong evidence of a technical contribution. Based on the experimental results provided by the authors, their method indeed introduces additional computational overhead compared to the baseline. Moreover, the authors explained that AndroidWorld has limitations on the maximum number of steps, suggesting that their method would not introduce excessive time overhead.    However, this reasoning is unreasonable because such constraints do not exist in real-world scenarios, which raises my concerns about the practicality of this method. The authors' rebuttal is detailed and addresses some issues, and the method demonstrates good accuracy on AndroidWorld, so my final score is 4.", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. We appreciate the recognition of our clear writing and effectiveness on dynamic benchmarks. Below, we address the main concerns point by point:\n\n---\n\n> Weakness 1: The technical contribution of this paper is limited. The main contributions lie in incorporating reflectors into the GUI automation framework and leveraging historical operation information to improve operational accuracy, both of which have already been explored in prior work [1,2,3].\n\nWe'd like to clarify that **reflection and exploration are broad scopes, and different works can focus on different aspects**. We declare the technical contribution of our work compared to prior efforts:\n\nOur work proposes a **real-time, hierarchical reflection mechanism** and a **pre-task proactive exploration module**, which together enhance robustness, adaptability, and generalization in dynamic mobile GUI environments. The prior works mentioned by the reviewer **differ significantly in focus and design**: [1] applies only naive single-step reflection, [2] performs post-task self-evolution relying on historical trajectories, and [3] uses reflection to assist tree search after tasks are completed. In contrast, our approach performs **online, structured, and selective reflection during execution**, and proactively builds **task-independent knowledge before execution begins**. These components are not only technically distinct but are also crucial for achieving high success rates in long-horizon, error-prone mobile scenarios, as demonstrated by our empirical results. \n\nWe hope these clarifications highlight the technical novelty and significance of our contributions.  We will further emphasize our technical contribution in the revised paper.\n\n---\n\n> Weakness 2: The paper is overly engineering-oriented. The method relies on multiple rounds of inference to achieve better performance at the expense of efficiency, which may negatively impact its practical applicability.\n\n> Weakness 3: Introducing three reflectors in the GUI automation framework inevitably increases inference time. A quantitative comparison should be provided, but such analysis is currently missing in the paper.\n\nWe appreciate the reviewer’s concern regarding efficiency and practical deployment. We have conducted a detailed runtime breakdown, and our analysis shows that the overhead introduced by our hierarchical reflection framework is marginal and well justified by the performance gain:\n\nAs shown in the table below. The MobileUse result corresponds to performing Action Reflection at every step, while the second uses Reflection-on-Demand (threshold = -0.01).\n- **Hierarchical reflection adds minimal latency**: With full reflection enabled, the Action, Trajectory, and Global Reflectors contribute approximately 25%, 3.5%, and 2% of the total time, respectively. When applying our Reflection-on-Demand strategy, the reflection overhead is significantly reduced to 10%, while the success rate drops by less than 1%, showing that unnecessary computation is effectively avoided without sacrificing performance.\n- **Our method achieves both efficiency and performance gains**: Under the same backbone (Qwen2.5-VL-72B), MobileUse with Reflection-on-Demand runs faster than Mobile-Agent-v2 and achieves a much higher success rate on AndroidWorld (>20%), demonstrating the effectiveness of our hierarchical reflection design.\n\n| Method | Module | Time ($s$/task) | AndroidWorld_SR (%) |\n|---|---|---|---|\n| Mobile-Agent-v2 | Total | 319 | 37.1 |\n| MobileUse | Total | 359 / 279 | 58.6 / 57.8 |\n| MobileUse | Action Reflector | 90 / 10.6  | \\ |\n| MobileUse | Trajectory Reflector | 13.2 | \\ |\n| MobileUse | Global Reflector | 7.7 | \\ |\n\n---\n\n> Weakness 4: As shown in the ablation results (Table 4), the model in the fourth row achieves better performance on Medium and Hard tasks, whereas Reflection-on-Demand and Proactive Exploration primarily benefit Easy tasks and can even significantly reduce performance on more difficult tasks. The paper lacks any explanation for this phenomenon. Since Easy tasks constitute the majority in the AndroidWorld benchmark, the full model appears strong overall, but these results suggest that Reflection-on-Demand and Proactive Exploration offer limited benefit for more challenging scenarios.\n\nThank you for the insightful observation. We would like to clarify this phenomenon and share our design rationale:\n- **Reflection-on-Demand achieves a trade-off between efficiency and robustness.** We observed that in some cases, especially on simple tasks, the reflectors could produce incorrect feedback due to hallucinations, leading to unnecessary or even harmful corrections. Our Reflection-on-Demand strategy filters out low-risk steps with high action confidence, thereby avoiding incorrect reflections while preserving essential ones. This mechanism is particularly effective on easy tasks, where the agent is already less prone to mistakes. As a result, this strategy not only improves efficiency but also enhances performance by reducing interference from spurious reflections.\n\n- **Proactive Exploration benefits tasks that appear easy but are actually hard and require prior knowledge.** While some tasks are labeled as “easy” by the AndroidWorld benchmark based on surface features like short length, they are in fact difficult to complete without prior exploration—for example, when interface semantics are unfamiliar. In such cases, our Proactive Exploration module is critical, as it allows the agent to acquire general knowledge about the app beforehand. This enables successful execution where agents without such context would fail. Thus, the observed gain in “easy” tasks actually reflects improved performance on structurally hard but short tasks.\n\nWe will revise the paper to provide detailed explanations of the ablation results and our future work. \n\n---\n\n> Weakness 5: The paper does not sufficiently describe the implementation details of Proactive Exploration—specifically, how the agent is guided to explore the GUI without explicit instructions, the duration of exploration, or the number of exploration steps taken.\n\nWe thank the reviewer for pointing this out and would like to provide more information: \n\n1) *Implementation details of proactive exploration*\n\nAs illustrated in Figure 1, our method adopts **a multi-agent collaboration framework to guide the exploration process**:\n(i) We guide the **Action Agent** explores unfamiliar regions of the current app environment without relying on task-specific instructions; (ii) The **Summary Agent** analyzes the environment states before and after actions, along with executed action information, to generate and update reusable knowledge; (iii) The **Judge Agent** monitors the exploration process to prevent redundant or ineffective behavior and provides corrective feedback when necessary. By combining these agents, the model can proactively collect common knowledge, which is especially useful when dealing with scenarios involving exceed the boundaries of their pre-trained capabilities. \n\n2）*Temporal and computational cost of the exploration module*\n\nIn terms of resource usage, we set the total number of exploration steps to 100 per app. The token and time cost per agent per step is as follows:\n\nToken cost per step: Action = 5,429; Summary = 8,135; Judge = 3,483.\n\nTime cost per step : Action = 3.4s; Summary = 11.5s; Judge = 4.6s.\n\nIt is worth noting that exploration is performed only once per app, and the accumulated knowledge is shared across multiple downstream tasks, making the cost amortizable and practical in real deployments.\n\nWe will provide these details in the revision.\n\n---\n\n> Question 1: This paper claims that Proactive Exploration can help accumulate common knowledge, but its experimental results are not satisfactory. Proactive Exploration also incurs a time cost. If there is a domain gap between the environment explored and the experimental environment, can Proactive Exploration still be effective?\n\nWe appreciate the reviewer’s concerns and would like to provide further clarification:\n\n1. *Irreplaceability of Proactive Exploration*\n\n**Proactive Exploration plays an essential role in certain tasks, where prior exploration of the app environment is critical**. Without this preliminary exploration, the agent would likely fail to complete the task. This is particularly true for apps with unfamiliar structures or dynamic content, where interaction knowledge must be gathered beforehand. As shown in our experiments, the Proactive Exploration module provides significant improvements in these cases.\n\n\n2. *Transferability of the exploration module*\n\nThe summary agent in the exploration module ensures that the accumulated knowledge is stored in a universal text format.  To in-depth discuss the transferability of knowledge, we have taken into account scenarios in real-world applications, including minor version updates, UI changes, similar applications, different operating systems, and cross-device situations.\n\nTaking the Chrome browser as an example, which is found in both AndroidWorld and daily usage: The exploration module summarizes operations such as searching, recommendation discovery, shortcuts to popular websites, information discovery, app downloading and installation, settings, and file opening. We find that most of the summarized knowledge remains consistent across different app versions and operating systems. Since the knowledge does not involve UI interface information, it is unaffected by UI changes that do not alter interaction logic. Incompatibility issues may arise only during knowledge transfer between similar apps, but these can often be resolved through further exploration and adaptation using the experience from similar apps.\n\nWe will revise the manuscript to further clarify the significance of our proactive exploration and its transferability."}, {"review_id": "OkRmTvaCk6", "reviewer": "Reviewer_Fa8z", "summary": "The paper proposes MobileUse, a hierarchical reflection-driven GUI agent for autonomous mobile task execution. The key contributions include a multi-level reflection mechanism for error detection and recovery, as well as a proactive exploration module to handle cold-start scenarios. Experimental results demonstrate that MobileUse achieves state-of-the-art performance on AndroidWorld and AndroidLab benchmarks. The approach is well-motivated and addresses important challenges in mobile agent robustness and adaptability.", "questions": "- In Eq. (3), does Pe(s^(t+1)) have only one input, s^(t+1)?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- This paper addresses an important issue in the GUI Agent scenario: how to enable models to possess reflective capabilities.\n- This paper conducts sufficient ablation experiments to demonstrate the necessity and effectiveness of the reflection mechanism.\n\n### Weaknesses\n\nI believe the main weaknesses of this paper lie in its methodology and experimental sections:\n\n- 1. The Stage I: Proactive Exploration section lacks novelty, as the idea of exploration for document collection has already been used in prior work such as AppAgent [1]. While I agree that enabling the model to explore the environment is necessary, the proposed method seems unrelated to enhancing the model's reflective ability. If the authors aim to improve the model's reflection through exploration, I suggest referring to work [2], which provides a compelling approach and implementation for achieving this. (I do not intend to ask the authors to compare with concurrent work, but I believe this particular work offers a valuable perspective.)\n- 2. Moreover, the experimental section suffers from issues common in many current works in the GUI Agent field, including unfair comparisons and a lack of implementation details. First, the evaluation protocol is unclear — it is not specified whether the evaluation was automated or manual. I recommend the authors provide more detailed implementation descriptions, as the information in the appendix is insufficient. Additionally, the range of baselines is limited; for example, End-to-End agents such as CogAgent [3] are missing. Furthermore, multi-agent frameworks like AppAgent [1] and Mobile-Agent-v2 [4] are not included in the comparison. These comparisons should also control for backbone consistency, whereas the authors present results based on Qwen-72B in the main paper, which complicates fair evaluation.\n\n[1] Appagent: Multimodal agents as smartphone users. CHI 2025.\n\n[2] GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior. Arixv.\n\n[3] Cogagent: A visual language model for gui agents. CVPR 2024.\n\n[4] Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. NeurIPS 2024.", "was_revised": true, "final_justification": "After the rebuttal, I still believe that this paper has issues regarding its novelty. First, as the authors themselves stated, autonomous exploration and reflection are two separate modules; however, the title of the paper only mentions reflection. I think that the extensive discussion and presentation of autonomous exploration diminish the perceived novelty of the reflection module and weaken the validity of the experimental results. Furthermore, numerous papers on reflection have already been proposed, as pointed out by Reviewer 1iG5, which further reduces the paper's originality. The authors have not clearly explained how their work differs from these existing studies. Therefore, I am inclined to give a score of 3 and do not recommend acceptance. **Finally, I kindly ask the AC to review the comments from Reviewer c8Yj. In my opinion, those comments are unreliable—being overly general and appearing to be generated by GPT.**", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. We appreciate the recognition of our motivation and sufficient experiments. Below, we address the main concerns point by point:\n\n---\n\n> Weakness 1.1: The Stage I: Proactive Exploration section lacks novelty, as the idea of exploration for document collection has already been used in prior work such as AppAgent [1].\n\nWe appreciate the reviewer’s concern and the opportunity to clarify this point. \n\nWe'd like to clarify that our proposed proactive exploration method has **significant and fundamental differences** with prior work, such as AppAgent [1]. The exploration in AppAgent is a **task-driven approach that relies on manually constructed tasks** and thus heavily depends on human knowledge and is constrained by the quality and diversity of the predefined tasks. Whereas our proactive exploration is an **automatic and experience-oriented multi-agent collaborative module without human involvement**, which is a significant advantage in extendability and is an important quality for achieving a super intelligent GUI agent.\n\nTo be specific, we highlight our proactive exploration in two aspects:\n- **Task-agnostic and fully autonomous exploration**: Proactive exploration is not given explicit tasks but instead proactively explores the environment to build general knowledge about UI layouts, navigation patterns, and possible states. In contrast, manually constructed task-oriented approaches are inherently limited by the diversity and complexity of the generated tasks.\n- **Exploration among collaborative multi-agents**: Multi-agent collaboration can more comprehensively address the demands of complex tasks, and we extend this idea to the exploration phase. i) We guide the Action Agent explores unfamiliar regions of the current app environment without relying on task-specific instructions; ii) The Summary Agent analyzes the environment states before and after actions, along with executed action information, to generate and update reusable knowledge; iii) The Judge Agent monitors the exploration process to prevent redundant or ineffective behavior and provides corrective feedback when necessary.\n\nWe will include a more detailed description of the innovative aspects of our proactive exploration in the revised paper.\n\n---\n\n> Weakness 1.2: While I agree that enabling the model to explore the environment is necessary, the proposed method seems unrelated to enhancing the model's reflective ability. If the authors aim to improve the model's reflection through exploration, I suggest referring to work [2], which provides a compelling approach and implementation for achieving this. (I do not intend to ask the authors to compare with concurrent work, but I believe this particular work offers a valuable perspective.)\n\nWe thank the reviewer for this thoughtful comment.\n\nFirst, we would like to clarify that in our framework, the proactive exploration module and the hierarchical reflection module are designed as **two relatively independent components**. Specifically:\n- The exploration module operates before task execution. Its goal is to allow the agent to proactively explore the mobile environment without predefined tasks, so it can accumulate common knowledge about diverse UI elements, layouts, and possible interactions, which is leveraged by the Operator to be familiar with the environment.\n- The reflection module works during task execution. It uses a hierarchical reflection mechanism to continuously monitor, detect, and correct errors in the agent’s behavior in real-time. This enables the agent to recover from mistakes and adapt to unexpected changes in the environment.\n\n**Both modules target the same overarching challenge: improving the robustness and adaptability of GUI agents in real-world mobile tasks.** The exploration module reduces the chance of failure caused by unfamiliar interfaces, while the reflection module ensures the agent can self-correct and recover when errors still occur during execution. Together, they form an integrated framework — MobileUse — that addresses different aspects of the robustness problem from both a proactive (pre-task) and reactive (in-task) perspective.\n\nWe also want to clarify that **the main contribution of our paper is the design and implementation of the hierarchical reflection mechanism**, which is why it takes up the majority of the paper’s technical discussion, analysis, and ablation studies. The exploration module plays a supporting role, providing additional gains by preparing the agent with broader environmental knowledge.\n\nWe sincerely thank the reviewer for suggesting GUI-Reflection [2]. We will cite this paper and clarify these connections more explicitly in our revised paper.\n\n---\n\n> Weakness 2.1: Moreover, the experimental section suffers from issues common in many current works in the GUI Agent field, including unfair comparisons and a lack of implementation details. First, the evaluation protocol is unclear — it is not specified whether the evaluation was automated or manual. I recommend the authors provide more detailed implementation descriptions, as the information in the appendix is insufficient. \n\nWe thank the reviewer for highlighting the importance of a clear and fair evaluation protocol. \n\nWe would like to clarify that we evaluate MobileUse on AndroidWorld [5] and AndroidLab [6], which are widely adopted standard benchmarks for GUI agent research [7,8]. Both benchmarks provide a unified testbed, including:\n- A consistent Android device environment with controlled settings;\n- Standardized task initialization procedures;\n- A well-defined automated evaluation process.\nSpecifically, after the agent finishes a task or reaches the pre-defined maximum step limit, AndroidWorld and AndroidLab automatically assess the result by reading the internal system states of the Android device to determine whether the task goal has been achieved. For certain tasks in AndroidLab, the benchmark also uses GPT-4o as part of the automatic evaluation pipeline. **No manual intervention is involved in any part of the evaluation process**, which ensures that the evaluations are fully automated and consistent across different methods.\n\nTo further support reproducibility and transparency, we have **open-sourced the entire benchmark setup and evaluation scripts** in our code repository under the `mobile_use/benchmark` directory. We also provide step-by-step instructions to help other researchers reproduce our results.\n\nIn the revised paper, we will provide more details about the evaluation protocol. We thank the reviewer again for this constructive suggestion.\n\n---\n\n> Weakness 2.2: Additionally, the range of baselines is limited; for example, End-to-End agents such as CogAgent [3] are missing. Furthermore, multi-agent frameworks like AppAgent [1] and Mobile-Agent-v2 [4] are not included in the comparison. These comparisons should also control for backbone consistency, whereas the authors present results based on Qwen-72B in the main paper, which complicates fair evaluation.\n\nWe thank the reviewer for this valuable suggestion on expanding our baseline comparisons.\n\nWe'd like to clarify that we have **included comparisons with several recent and strong End-to-End agents and multi-agent frameworks**, as shown in Table 2. For completeness, we have now added the results for CogAgent [3] and Mobile-Agent-v2 [4] on the AndroidWorld benchmark, as shown in the table below. Although CogAgent is fine-tuned for GUI automation tasks, it performs poorly on the AndroidWorld benchmark due to its insufficient grounding and reasoning ability.\n\nWhen comparing with Mobile-Agent-v2, we control a consistent backbone model. The result shows that our MobileUse achieves significant performance improvement compared to the Mobile-Agent-v2 framework, attributed to the hierarchical reflection architecture for error recovery and proactive exploration for common knowledge accumulation.\n|Method|Model|SR|\n|-|-|-|\n|CogAgent|CogAgent-9B-20241220|9.0|\n|Mobile-Agent-v2|Qwen2.5-VL-72B|37.1|\n|MobileUse|Qwen2.5-VL-72B|62.9|\n\nWe agree that backbone consistency is an important concern. Since different methods choose various backbones and many of them are not open-sourced, we present the best result of each baseline reported in their papers to reach a fair comparison. \n\nWe appreciate the reviewer’s careful attention to this issue and will incorporate these clarifications and additional results in the revised paper to make our evaluation more comprehensive and transparent.\n\n---\n\n> Question 1: In Eq. (3), does Pe(s^(t+1)) have only one input, s^(t+1)?\n\nThe Perception module takes as input both the screenshots before and after the action execution, i.e., it should be written as Pe(s^t, s^(t+1)) instead of only s^(t+1). We appreciate the reviewer catching this oversight and will correct this notation in the revised version of the paper.\n\n---\n\n[1] Appagent: Multimodal agents as smartphone users. CHI 2025.\n\n[2] GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior. Arixv.\n\n[3] Cogagent: A visual language model for gui agents. CVPR 2024.\n\n[4] Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. NeurIPS 2024.\n\n[5] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. ICLR 2025.\n\n[6] AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents. Arxiv.\n\n[7] Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. ICLR 2025.\n\n[8] AutoGLM: Autonomous Foundation Agents for GUIs. Arxiv."}]

Author rebuttal:

Meta-review: This paper introduces MobileUse, a hierarchical reflection-driven GUI agent designed for robust autonomous operation on mobile devices. The proposed system addresses three core challenges in mobile automation: long-horizon task execution, error recovery, and cold-start scenarios in unfamiliar environments. The key contributions include a hierarchical reflection architecture with multi-level self-monitoring and a proactive exploration module, which together enable improved resilience, adaptability, and state-of-the-art performance on AndroidWorld and AndroidLab benchmarks. An open-source toolkit is also released, demonstrating the practicality of the approach for real-world deployment.

In general, the paper is well written, clearly motivated, and introduces a novel hierarchical reflection framework that goes beyond existing single-step reflection methods. Reviewers appreciated the thorough experimental evaluation, the significant performance improvements over prior baselines, and the release of a toolkit that enhances the paper’s real-world relevance. The reflection-on-demand strategy and proactive exploration module were noted as innovative aspects that contribute to both efficiency and adaptability.

However, as pointed out by the reviewers, there are some concerns about the work. Specifically, the technical novelty may appear incremental relative to prior GUI automation frameworks, and the reliance on very large models raises questions about scalability and accessibility in resource-constrained settings. Reviewers also highlighted the need for deeper analysis of generalization to unseen apps, the computational overhead of the reflection modules, and the sensitivity of the reflection-on-demand mechanism to hyperparameters. Despite these concerns, the authors provided convincing rebuttals with additional experiments and clarifications, which alleviate many of the reviewers’ doubts.

Overall, the paper presents a strong empirical contribution with a novel hierarchical reflection design, and it achieves clear improvements over existing baselines while demonstrating practical applicability
Decision: Accept (poster)