SMAN-Bench: A Cross-System Benchmark for Mobile Agents under Single- and Multi-path, Ambiguous, and Noisy Tasks
Authors: Weikai Xu
Zhizheng Jiang
Yuxuan Liu
Pengzhi Gao
Wei Liu
Jian Luan
Yunxin Liu
Yuanchun Li
Bin Wang
Bo An
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal
mobile agent
offline evaluation
Primary area: datasets and benchmarks
TL;DR: A realistic and comprehensive benchmark for VLM-based mobile agents, with common, noisy and ambiguous trajectories.

Abstract: VLM-based mobile agents are increasingly popular due to their capabilities to interact with smartphone GUIs and XML-structured texts and to complete daily tasks. However, existing online benchmarks fail to obtain stable critical reward signals under dynamic environmental changes, and neglect the influence of noise components and interactive instructions. Offline benchmarks evaluate the agents through single-path trajectories, which stand in contrast to the inherently multi-solution characteristics of GUI tasks. To address these limitations, we introduce SMAN-Bench, a benchmark designed to evaluate agents under Single-path, Multi-path, Ambiguous, and Noisy task settings. We employ a slot-based instruction generation method to match templates with GUI trajectories from an existing, graph-structured, unlabeled mobile corpus. SMAN-Bench includes a common task split, with offline multi-path evaluation to assess the agent’s ability to obtain step rewards during task execution. It contains a noisy split based on pop-ups and ad apps, and a contaminated split named AITZ-Noise to simulate a realistic noisy environment. Furthermore, an ambiguous instruction split with preset Q&A interactions is released to evaluate the agent’s proactive interaction capabilities. Our evaluation covers mobile agent frameworks like AppAgent-v1, Mobile-Agent-v2, and Mobile-Agent-E, and includes both open-source and closed-source mobile fundamental models, as well as several multimodal reasoning models.

Published as a conference paper at ICLR 2026
SMAN-BENCH: A CROSS-SYSTEM BENCHMARK FOR MOBILE AGENTS UNDER SINGLE- AND MULTI-PATH, AMBIGUOUS, AND NOISY TASKS
Weikai Xu Zhizheng Jiang Yuxuan Liu Pengzhi Gao', Wei Liu Jian Luan', Yuanchun Li Yunxin Liu², Bin Wang', Bo An
Nanyang Technological University University of Electronic Science and Technology of China 3 "Gaoling School of Artificial Intelligence, Renmin University of China
"MiLM Plus, Xiaomi Inc. IIstitute for Al Industry Research (AIR), Tsinghua University 'Equal contribution
ABSTRACT
VLM-based mobile agents are increasingly popular due to their capabilities interact with smartphone GUIs and XML--tructured texts and to complete daily tasks. However, existing online benchmarks fail to get stable critical reward signals under dynamic environmental changes, and neglect the influence of noise components and interactive instructions. Offline benchmarks evaluate the agents through single-path trajectories, which stand in contrast to the inherently multi- solution characteristics of GUI tasks. To address these limitations, we introduce SMAN-Bench, benchmark designed to evaluate agents under Single-path, Multi- path, Ambiguous, and Noisy task settings. We employ a slot-based instruction generation method match templates with GUI trajectories from existing, graph- structured, unlabeled mobile corpus. SMAN-Bent -enchh includes common task split, with offline multi-path evaluation to assess the agent's ' ability obtain step rewards during task execution. It contains noisy split based on pop-ups and ad apps, and a contaminated split to simulate a realistic noisy environment. Furthermore, an ambiguous instruction split with preset Q&A interactions is released to evaluate the agent's ' proactive interaction capabilities. Our evaluation encompasses mobile agent frameworks such as AppAgent-vl, Mobile-Agent-v2. and Mobile-Agent--E and includes both open-source and closed-source mobile fundamental models, as well several multimodal thinking models. Our code and datasets available at wtt http:://tttttoooooeeeeeeeeeeeea/
INTRODUCTION
LLM- based mobile agents (Wang et al. 2023 Ding 2024) are increasingly popular due to their
capability interact directly with mobile Graphical User Interfaces (GUIs) and their potential to manage daily tasks autonomously. Unfortunately, L.LM-based agents cannot well comprehend the mobile GUT structure and widget functionality, based on text such as Visual-Hierarche
XMI., HTML, or Accessible- Visited Trees. Recent studies (Ma ct al. 2024 Zhang al.) 2024a) indicate VLMs can provide more comprehensive understanding of GUIs. This has led to mobile benchmarks replacing foundational models with VLMs, resulting in benchmarks for end-to-end mobile tasks
based on GUI pages (Wang et al. 2024c Xu et al. 2024a Rawles et al. 2024).
Existing VLM-based Agent benchmarks can be divided into two categories: (1) Online evaluation involves the agent executing operations on a real device based on the user's high-level instructions. These benchmarks directly determine the success rate by checking the widget values in the final GUI and allow agents to complete tasks through multi-path. However, due to the instability of the device environment, such as OS updates, APP updates, and user preference records, the step rewards of online benchmarks (Murthy et al. 2024 Deng et al. 2024 Wang et al. 2024c) are fluctuating and
Work done during the internship at XiaoMi.
"o An is the corresponding author.

---

Published as a conference paper at ICLR 2026
Table Comparison of SMAN-Bench other benchmarks.
# Inst.
Language
# Avg Steps
# Screen-
Path
Online Environ.?
Ambi. Noise
Interac.
Benchmarks
shots
Inst.
PIXELHELP
187
EN
4.2
800
Single
X
X
MOTIF
EN
4.5
21K
Single
X
AMEX
341
EN&CN
12.8
~ 104K
Single
X
X
X
SCREENSPOT
1,220
EN&CN
1
600
Dot
X
X
X
MOBILEAIBENCH
EN
Dot
X
X
AGENTBENCH
100
EN
20
2k
Multiple
GUI ODYSSEY
7,735
EN
15.4
Single
X
X
MOBILE- HOBILE-BENCH
832
CN
14,144
Multiple
X
X
MVISU-BENCH
404
EN&CN
Multiple
X
V
SPA-BENCH
340
EN&CN
*
Multiple
X
ANDROIDLAB
10.5k
EN
8.98
94.3k
Multiple
X
X
ANDROIDWORLD
116
EN
Multiple
X
SMAN-BENCH
12,856
EN&CN
7.28
4-8
Both
X
unstable. Meanwhile, task evaluation often depends on the process rather than the final page alone, as two failed tasks may represent very different completion progress, making purely nutcome-based comparison unfair across different agents. (2) Offline evaluation uses static datasets where the golden path is pre-executed generates the current action based on each step's GUI, instructions, and action history. and the action trajectory formulated as single-path. Although offline benchmarks (Chai et al. [024 [024|Cheng [hengg et al.) 2024, Rawles et al. 2023) are more practical for training, considering the diversity of agent task solutions, the agent's good performance may only represent good fit to the preferences encoded in the current benchmark annotations instead of handling multi-path solutions. This limitation causes some agents (Li et al. 2024b) that achieve exceptionally high performance on certain benchmarks to perform poorly in real-world scenarios, while also exhibiting overly simplistic decision-making. More critically, benchmarks such as MobileAgentBennch (Wang et al. (024c) and AutoDroid (Wen al. 2024) are constructed real devices and evaluated within Google apps using the Android Accessibility Service; these apps feature overly clean pages without task-irrelevant ads, buttons, and pop-ups. In real-world scenarios, users may not be able provide such precise and full instructions all at once (Wang et al., (2024)).
the device, with actions and screenshots saved offline. The agent
To address the above problems, we introduce new benchmark named SMAN-Bench with the following data and methods: (1) Instruction Annotation Method: To connect one instruction with several trajectories, we use the GIAS to construct 12k instructions with the unlabeled action sequences based on the random walk graph-structured corpus named Mobile3M (Wu et al.) (024a). Each instruction is generated via task templates and slot information based on the corresponding trajectories. (2) Offline Multi-path Evaluation: Combining the advantages of both online and offline environments, we propose multi-path evaluation approach. We allow the agent to execute in single-path manner and compare the result with the golden path. Alternatively, the agent is also allowed to perform action search within the graph corpus and accumulate step rewards, as does during online evaluation. (3) Realistic Noisy Environment: To explore the effect of noise, collect an additional sub-dataset named SMAN-Bench-Nooie
from apps that are heavily contaminated by advertisement noise. Several apps with substantial ads and pop-ups are specifically selected, including static pop-ups. dynamic video ads, and redirecting advertisement links. Additionally, also contaminate AITZ Zhang et al. (2024b) by inserting ads into original normal trajectories to build AITZ-Noise. (4) Active Interactive Evaluation: We also construct sub-dataset named SMAN-Bench- Bee---enncchheeeee mannch-mmiguuus, which allows agents ask when necessary during task execution. Full instructions are pre-constructed and then progressively simplified into ambiguous instructions through blot-based extraction. Questions and answers are built based on slot information and then assigned to the corresponding GUIs.
Our evaluation of general-purposs multimodal models is conducted on three agent frameworks: AppAgent-vl (Li et al., 2024c), Mobile-Agent-v2 (Ding 2024), and Mobile-Agent-E (Wang et al.) 2025c). Moreover, we include mobile-doman agents trained with continual pre- trainiiinigg..... SFT, and RL (Lu et al. 2025a) Luo al. 2025| Lu al. 2024 Qin et al. 2025). Finally. place special emphasis on examining how slow-thinking multimodal models perform in mobile scenarios, analysing whether the reasoning pattern is effective. Overall, work makes four main contributions:
2

---

Published as a conference paper at ICLR 2026
We construct cross-system benchmark named SMAN-Bench based on Mobile3M's graph structure corpus and propose a slot-based instruction generation method named GIAS.
We propose offline multi-path evaluation method and leverage slot-based key node annotations to enable stable assessment of step rewards.
We introduce MAAAN--Bench-Not noisy apps, enabling robust assessment under challenging environments.
support realistic noisy evaluation by collecting data from
We propose SMAN-Bench LeL---BenchAAmiigee Ambiguous to facilitate active interactive evaluation, where agents allowed to ask clarification questions during execution.
Scenario I: Common Task
Instructions are generated by GIPT-4o
Trajectories filtered from unbabeled corpux
traject. maitchng
Quality Verißication
Agent
Inatruction was mociles
Descriation
Instruction: Play the song "cry on my shoulder" on the music app. Evaluation 2: Multi path
Evaluation I: Single path
Trujectory I
instrcction mutched muliipl injecturis
Merge
Equinolont Auy made the finat page
Trajectory 2
and urr Afforent applications and fioa pages.
Trajectory 3
Key
page
LEATO
Figure 1: The overview of SMAN-Bench, the entire pipeline framework for data construction and filtering, and the distinction between single-path and multi-path evaluation methods.
RELATED WORK
2.1 MOBILE AGENTS
Large language models (Achiam et al., (023) emerge as autonomous agents (Li et al. 2024b) Wen
ctt al. 2023) in the mobile domain and garner considerable attention. With the rapid development of
Vision- Insion-language language models (VLMs), multimodal researchers build mobile GUI agents Yangg et al. 2025
Zheng et al., 2024) and multi-agent frameworks (Ding 2024 Li et al. 2024c| Wang et al. 2024b)
based on closed-source VLMs. Meanwhile, some researchers focus training agents with stronger
element grounding (Cheng et al. 2024 Hong et al. 2024 Wu et al. 2024b). page navigation NNu
al. 2024 [u al. 2024 Gou al. al.][ (022)) GUI understanding (Chai et al. 2024 You et al.) 2024
Baechler et al.] 2024 and task planning capabilities (Zhang et al. 2024c Nong et al. 2024 Xu et al.
2024b) based on open-source VLMs. In addition, Digirl (Bai al. 20224) and Distrl (Wang et al.
2024c) use joint online and offline reinforcement learning to enhance the generalization of mobile agents and mitigate performance degradation when facing APP updates and unseen APPs. Some researchers (Qinghong Lin et al., 2024) explore optimizing VLM structures. For example, Dorka (Dorka et al., 2024| Tan et 2025 Liu et al. [2025] Sun al., 2024 Xu et al. [025b) Huang et al. 2025a) optimizes the encoder by incorporating historical images and actions as input.
2.2 MOBILE AGENT BENCHMARKS
As shown in Table I AndroidEny Toyama et al. 2021 and MobilEnv (Zhang et al. 2023) are the first create LLM agent evaluation environments based reinforcement learning. Mobile Bench (Deng et al. 2024) and AppBench (Wang et al. 2024a) introduce online benchmarks combining API and GUI, while MobileAgentBench (Wang et al. 2024c) establishes the first fully automated
3

---

Published as a conference paper at ICLR 2026
multimodal benchmark for VLM- GUI agents. More offline benchmarks (Li al. 2020a
Burns et al. [021]] Murthy et al. 2024) are released, which are primarily categorized into GUI understanding and task-oriented. (1) For task-oriented benchmarks, AITW (Rawles et al. 2023) and
AITZ (Zhang et al. 2024b) create large-scale benchmarks based on Google apps, while AMEX (Chai
al. 2024) supplements these benchmarks by adding data for GUT understanding with similar app
types. ScreenSpot (Cheng et al. 22024), Mobile3M (Wu et al., (2024a). and GUTOdyssey (Lu et al.
2024) focus on more granular element grounding and task planning. (2) Rico (Deka et al. (017))
is the first -nnotttteeeed GUI corpus, followed by ScreenQA (Hsiao et al. 2022), Widget Caption (Li et al., 2020b). and Screen2words (Wang et al., 2021), which for Q&A, widget understanding, and page summarization. Subsequently, Mind2web (Deng et al.) 2023) incorporates additional GUI
data of varying sizes, and Meta-GUI (Sun al., 2022) provides tasks for multi-round dialogues.
Recently, more online benchmarks (Xu et al. 2024a Rawles et al. 2024: Chen et al. 2024a Wang
et al. 2025aa Huang et al. 2025b] Xu ct al. 2025c) have been released to evaluate agents in real
environments, such smartphones, PCs, and web browsers.
SMAN-BENCH BENCHMARK
3.1 MOBILE3M GRAPII-STRUCTURED CORPUS
Mobile3M (Wu et al. 2024a) is a large-scale Chinese&English mobile UI corpus constructed from 49 widely used third-party applications, each with more than ten million active users. The dataset was collected on Android emulators using Appium-based automated interactions and explored with BFS random walk and breadth- first strategies. Each UI page consists of both a screenshot and its corresponding XML document, while interactions such click, input, and scroll were recorded to build state transitions. In total, the dataset contains 3,098,786 UI pages and approximately 20 million interaction actions, organized into directed graphs, where nodes represent UI pages and edges denote transition actions. To reduce redundancy, a unique-page detection mechanism based on Action Space and pixel differences (BM25) (Robertson al. 2009) was applied, which improves exploration efficiency and naturally converts the exploration tree into graph structures. The dataset spans wide range of categories, including travel, lifestyle, and shopping, with relatively balanced distributions, ensuring diversity and representativeness. Unlike prior datasets that only provide isolated pages or chain-structured traces, Mobile3M's 's graph organization captures the complexity of real-world app interactions, supporting multi-path reasoning and graph-based modeling. More details are described in Appendix C
and the data distribution is shown in Figure 7 and Figure
AAG
Human Demonstration real Machine
Scenario 2: Noisy Task
Please help me find Beijing's 's air quality index for tomorrow Caiyun Weather.
Caijun Weather app was solected for nove collecttion
Cloving the ad directly king button related
bocaue hav addiiiional ads connent.
nod
Fiitiig jump.
Q&AAs are Pre- executed by GPT-4
Scenario 3: Ambiguous Task
will as JOM gRestions MecOSsers
Q: Help me find detailed information about my favorite singer.
The stnger's numms gender, nationaliy and style?
dhiukk the mitenion axcompsishad. interactine conffrmatiin dear the nnd has been complered RESESVACO
APP your Ieseet An MAM??
Cul Mali, Inlund,
Rock
Figure 2: The SMAN-Bench includes two other types of tasks: Noisy-split, and Ambiguous-split, and demonstrates the process of instruction generation and manual annotation for each task. In Noisy-split, the GUIs with orange shading represent noise.
4

---

Published as a conference paper at ICLR 2026
3.2 MULTI-PATH EVALUATION
In the multi-path evaluation setting, mobile agent can freely explore within the pre-executed and collected graph corpus, provided that the maximum step limit is not exceeded. As shown in Figure [ several discrete single trajectories are merged into a unified graph, where the merged nodes correspond to the key states of the current instruction. The merging criteria are mainly twofold: (1) Action space and pixel differences (BM25) thresholds (Wu et al. 2024a), which are used to identify the same page (e.g., g., the main interface of an app may differ slightly across sessions but is considered equivalent). (2) Consistency in button values in the Android XML/Accessibility, where pages are regarded as equivalent if buttons share the same values (Xie al., 2024) (e.g., searching the same keyword across different browsers produces pages that are merged into one key node). Since pre-execution cannot cover all possible search results, provide a predefined query pool for instructions, and any search beyond this pool is regarded as invalid. As shown in Figure noisy tasks allow the agent to return to the graph within limited number of steps. This contrasts with the single-path setting, where advertisements are directly closed, because not all ads are irrelevant to the current task.
3.3 DATA CONSTRUCTION
Generating common instructions from action trajectories. For the Mobile3M graph corpus, the key challenge is annotating instructions for each trajectory that closely align with the intended actions. Building Netscape (Murty et al. 2024)'s 's fine-tuning of web agents to eliminate redundant actions from action sequences, two key points for pairing trajectories and instructions are the Intent Understanding and the Slot Matching between different GUIs: (i) Using the intent behind actions reduces ambiguity, since coordinate-baste accurately. For instance, buttons with the same textview may differ in meaning: in Figure 9 the same "plus" button corresponds to adding "Hazelnut Latte" or "Cookie Mocha". (ii) Filling predefined templates with slot information provides reward signals at key nodes. As GUI agent tasks are inherently multi-path, single-path annotations with unstable preferences can cause performance drops on unseen tasks. Slot filling allows one template to match multiple trajectories that share key nodes, forming the basis of multi-path evaluation. Building on the above findings, we propose a slot-based instruction annotation method named GIAS (Generating Instructions From Action Sequences), which is shown in Figure I The whole process is as follows: (I) multi-path sampling based on fixed start and end GUIs; (2) GUI content annotation; (3) action intent inference; (4) extracting slot information from GUI changes; (5) filling instruction templates with the slot; (6) deduplication and simplification. To ensure device diversity, we expanded the evaluation to include additional Android and iOS systems, specifically iOS 18.5, HarmonyOS 5.0, and HyperOS 3.0.
actions without GUI pages cannot reproduce scenes
The whole process is as follows: (1) multi-path sampling based on fixed start and end GUTs; (2) GUI content annotation; (3) action intent inference; (4) extracting slot information from GUI changes; (5) filling instruction templates with the slot; (6) deduplication and simplifica- tion. The entire process is explained in detail in Algorithm [ Specifically, we choose paths that start from nodes with the same name in Mobile3M and end at homogeneous nodes with different names (Homogeneous nodes refer to pages whose similarity or the number of identi- cal UI elements exceeds the threshold (Lu et al.) 2006)). Considering its diversity, each trajectory includes least two different types of actions and minimises the proportion of intermediate homogeneous pages. Throughout the entire an- notation process, only the verification step involves a closed-source model, while all other steps are performed by open-source models without any human intervention. More details on GIAS are in Appendix [
I want 166B-51268 MacBook Pro M4 in the Midnight version.
1668 1668-5126B
MacBook Pro M4
Midnight version
(A) Online Evaluation
1668-51268
Promá
Midright T
No further checked
(B) Offline Multi-path Evaluation
Figure 3: Unlike Online Evaluation, offline multi- path evaluation checks both process and final GUT as reward signals.
Noisy app and ambiguous instruction data. OMAN-Bench-Nood is primarily derived from manual annotation and contamination in existing data: (1) For manual annotation, select apps from
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 GIAS Algorithm
Require:
Start Page, Po; End Page, Pt; Trajectory a; Page Description D; GUT Pages, 8; Action, a; Action Intent, 2'; Page Slot, C: Instruction, 4; Task, T:
Ensure: Prompt, P.; Few Shot Cases, Fs; Verified Flag. Q: Instruction Templates, 7i
1: Select
0, {8i 86z -66-111
8ig ~ Po:ts for j ....., ,
2:
for cach 8,, E 0, do
3:
for j 0: do
4:
Day & VLM Pt)
Get text descriptions from GUIs D. Get intent between two actions b. Get Page Slot based on pre-settings > Fill Templates with the slot
Titst1 &- Dayi P (88 - 81++1)
6:
Cha+t <- Day Da,
7: 1i + (Cea, Tia) ~ Uniform(r)
8: end for
9: end for
10:
each pair instructions (Ti, [5) in do
11: Sim 15) 2 r then
12: Discard 13 from I
Make sure there are no highly similar instructions
13: end if
end for
15: T.Q <- Veri (I,),Ps}
> for all Trajectories o, ensuring no redundant steps
16: return T
third-party markets; these apps contain unavoidable ads and pop-ups. When performing actions on these apps, we do not handle the following scenarios in advance: login, update, permission settings, ad pop-ups, and VIP subscriptions. In some instructions, to test the agent's s response in unexpected situations, we deliberately click on ad pages incorrectly to see if it can recover from divergent paths. All noisy steps and their redirect pages are additionally marked, making this subset adaptable for agents with a rollback mechanism as well. (2) For data contamination, we randomly insert at least one ad (randomly collected from the Google Store app) into AITZ (Zhang al., 2024b) trajectories, which is a high-quality subset of AITW (Rawles et al. 2023). For SMAN-Bench -ee---enncchbboooeee Ambiguous, we first construct the full instructions, annotate action trajectories, and remove slot information to build ambiguous instructions. Multiple sets of interactive Q& A are assigned to corresponding GUIs. For example, as shown in Figure 3 the full instruction is: want a 16GB + 512GB MacBook Pro M4 in the Midnight version.', ,, while the ambiguous instruction is: ' want to buy MacBook.' Information such as "16GB + 512GB", "Pro M4", and "Midnight version" treated as three slots, each assigned to the corresponding GUI for step reward. More details can be seen in Appendix E.7
3.4 DATA STATISTICS
The apps and categories in SMAN-Bench remain consistent with Mobile3M, comprising 15 categories and 49 apps, with each category containing at least the top three apps by download volume. As shown on the left part of Figure 4 QQMusic and Kugou account for over 70% of the monthly downloads in the music app category. which are selected as representative music apps in our datasct. The common split includes 12,854 instructions and 800 templates generated by GIAS, which are divided by action steps: simple tasks (1-6 steps) and complex tasks (7-15 steps). As shown in the middle part of Figure 4 there 9,620 simple tasks with average of 5.62 steps and 3,234 complex tasks with average of 8.21 steps. Figure 12 shows the task distribution. We strive for a difficulty balance, but some apps, like Netmail (an email tool), still have an imbalance because sending an email often entails multiple steps to complete essential information fields, resulting in a relatively long interaction process that cannot be easily classified as a simple task. From categorical perspective, shopping apps (DuApp) have a higher proportion of complex tasks compared to simple tasks. In contrast, since Baicizhan features a clean page and straightforward functionality (vocabulary learning), it allows constructing task instructions from templates with fewer slots. The noisy and ambiguous splits each contain 100 instructions, while these noisy data come from another 20 highly noisy apps and each trajectory in the ambiguous split includes at least 5 additional manually constructed Q&A. As shown in Table [6] pre-setting Q&A is strictly aligned with the missing slot information. The average trajectory lengths 12.74 for the noisy tasks and 7.53 for the ambiguous split. Furthermore, we randomly insert one of 150+ ads at a step within one of the 2,504 trajectories in AITZ. Each trajectory in AITZ-Noisy contains only single injected advertisement, whereas trajectories in Noisy split include at least five

---

Published as a conference paper at ICLR 2026
noisy steps, along with additional non-task-related pages such as authorisation, tutorials, redirection pages, and in-app purchase services. More details are in Appendix E.7
gl kiiking Pacch Rayori
Task Dww.ibttniittinnnn by Step Count (Pul va Sampledi
Duotbation Consirencyy FaB Sompoee
Figure 4: Download volume distribution of music APPs in one month, on the left. The distribution of tasks and their respective step counts, in the middle. The full distribution of apps and their sampled distribution, on the right.
4 EXPERIMENT
4.1 SETUP
Models. For SMAN-Bench common, noisy, and ambiguous splits, we evaluate agent frameworks
such as AppAgent-vl, MobileAgent-v2 and Mobile with different fundamental VLMs:
Intern VI.2-40B (Chen et al., 2024b), LLAVA-72B-NEXX
(Li et al. 2024a). Qwen2-VL-722 (Wang
ctt al. 2024d), Llama3.2-908 (Dubey et al. 2024), Qwen- VL-Max, GPT-4o (Achiam et al. 2023).
and GPT-4v. For open-source mobile agents, we use CogAgent (Hong et al. 2024). UGround-7B
(Giou et al. 2024). AS-Atlas-BB (Wu et al. 2024b). UI-Tars (Qin et al. [2055 Kimi- VL. (Team et al.
2025), DeepSeek-VL2 (Wu et al. 2024c), OpenCUA-328 (Wang et al. (0255), GUI-RI (Luo et al.
2025). UI-R1 (Lu et al. (025a). UI-ST (Lu et al. 2025b), and GUI-GI (Zhou et al., 2025). For
reasoning VLMs, we select Iv-thikign Qwen-QVO-plus, OpenAl o3, Claude S.aude-3.7-onnee, DeepSeek-RI(HTMLL and Doubao-Thinning-pre (Guo et al. 2025).
Settings. To reduce cost, only zero-shot evaluations done on a subset split of Random-800, which has a similar sub-split distribution to the full split. As shown in Figure 4 the distribution of step and app categories in Random-800 is fully consistent with that of the full dataset, and 800 instructions correspond one-to-one to 800 predefined instruction templates. For the simple and complex splits, we set the maximum number of steps to 20 and 25, respectively. about the next-step decision or the specific function of button are rejected; only supplementary information relevant to the instruction is allowed to be provided. Unlike AppAgent, we annotate the widget types using specific numbers (Figure
ambiguous tasks, direct queries
Metrics. We follow metrics proposed in MobileAgentBench (Wang et al. 2024ch and AUTO-UI (Zhang & Zhang 2023).
Success Rate (SR): Nsucoess/ Stucoess/Maastee MMasker judged by whether the agent reaches the final pages in multi- path evaluation or whether all actions correct in single-path evaluation.
Step Efficiency (SE):
where Sactual is the number of actual steps to complete a task, and Smin is the task's 's minimal annotated steps. This metric expresses whether the agent performs unnecessary or redundant actions in multi-path evaluation.
Step Accuracy (Step.Acc): Stp/Sgt. where Sip is the number of predicted actions that match the golden actions in single-path evaluation. This metric also reflects the step reward when the actions compared with the key nodes in multi-path evaluation.
TYPE Accuracy: Sup/SSt- where Stp is the number of predicted actions that match the type of golden actions. We use TYPE to check whether the action types correct.
7

---

Published as a conference paper at ICLR 2026
Table General VLMs with mobile agent framework results on SMAN-Bench Common, Noisy and Ambiguous splits. Type is used in the single-path, while SE is used in the multi-pale evaluation.
Medels
Cate.
TypetSE
Common-Simple
Step.
SR
TypetSSE
Commun- Cumplex
Step. Ace
Sit
Typet/SR,
Noisy Step.
Typet/SR, Steps
Ambignonn Thita
Single Apent Franewof::
Appltgent
Malti
1L-AWA-22--NEXT
Mulli
704
2.7
Qwm2-V11-738
Single
Muli
20.6
7.0
780
24-4
43.5
Dass-VL-Max
Single
Mului
20.5
12.6
7.5
77.1
24.3
90.3
48.5
Liama32 90B
Single
2.6
$7.0
24.5
69.7
11.2
85.4
15.5
GPT-4v
Single
Muin
29.7
3.0
6.0
29.4
1.5
72.7
13.4
88.6
20.5
GPT-46
Single
Mulli
90.4
61.8
19.8
18.5
79.2
4,4
61.7
50.6
11.5
16.5
72.3
18.2
94.7
33,9
Multi-gennt Franework: Mobrllegeeni-22
IniemVL2 4MB
Single
Mulni
$6.5
6.0
19.3
27.6
0.0
3.5
$0.7
4.4
32.6
26.8
0.5
3.5
64.3
9.1
0.0
76.0
16.3
0.0
Qwen2 VL-720
Single
Mulh
91.5
5.4
50.5
54.9
13.0
15.1
91.6
4.4
49.0
55.6
8.0
4.5
75.8
20.7
1.0
86.2
40.8
7.0
Qwe--VL-Max
Single
Mulli
74.2
5.4
20.6
17.0
3.0
4.5
68.8
4.3
24.8
12.3
2.0
3.0
66.5
4.2
0.0
66.6
9.0
0.0
Single
62.4
16.6
1.0
67.0
17.5
0.0
63.7
9.7
0.0
64.3
5.3
0.0
GPT-4v
Single
Mulii
90.3
6.0
22.9
17.8
3.8
3,4
90.6
4.5
25.3
11.8
0.5
0.0
62.5
12.6
0.0
91.0
15.6
0.0
GPT-4o
Single
Muin
91.9
4,9
53.5
37.6
23.5
13.5
92.3
4,2
56.3
50.5
17.0
17.5
77.1
25.5
2.0
91.6
39.7
12.0
Aulli-Aguuts Prameweek MihiliAggnt-
Single
Multi
Qwe-2-NL-72B
Single
32.7
Qwen VL
Single
Laama
Single
Mulli
4.2 DATA QUALITY VERIFICATION
Due to the random walk involved in the former, the instruction trajectories may include redundant actions, whereas the latter requires checking whether the noise is handled correctly and whether the quality of the Q&A is sufficient. Therefore, we design a data quality verification experiment shown in Table X and Table 9 Only 8% of the data constructed in the complex split are found be suboptimal. This is primarily due to more slots in complex instruction templates, which results in unnatural semantics after filling. Semantic alignment does not influence invalid steps between the instructions and the actual trajectories. Therefore, we only manually revised these instructions to ensure that the random-80k 800 subset meets the quality requirements.
Qwen2-VL-728
Llama3.2-VL-90B
GPT-40
Mapp Wwp.ActPaagee.
Figure 5: Performance on different backbones and agent frameworks.
4.3
MAIN RESULT
Agent frameworks. As shown Figure5Jan Table AppAgent-vl outperforms in single-patl evaluations, whereas Mobile/gent-vv2 excels in multi-path scenarios. This is because single-pate evaluation presets the correct historical action and guides agents to focus on the current page. In multi- path, agents' actions depend on their previous decision and pages, agents are more prone to being trapped in mistaken execution trajectories. MobileAgent-v2. v22ss reflection and expectation mechanism effectively mitigates this problem. With dynamic knowledge injection and planning, Mobile-Agen surpasses the other two frameworks on most metrics, though models with shorter context windows (e.g., Llama3.2 2-VI--90B) show performance drops due limited in-context learning.
8

---

Published as a conference paper at ICLR 2026
Table 3: Results on SMAN-Bench Common, Noisy, and Ambiguous splits. Agents include three Continuous pre-training mobile agents, RL-based mobile agents, and reasoning agents.
categories:
Models
Cate.
Type
Common-SSinpl Step. Ace
SR
Type
Common-- Complex
Step. Acc
SR
Type
Noicy Data Step. Ace
SK
Type
Ambiguuus Data
Step. Ace
SR
training
Agents
CogAgent-1 188
Single
75.6
20.9
13.0
62.3
20.8
6.0
57.2
16.2
1.0
72.4
30.6
11.0
UGround-788
Single
73.0
19.5
17.0
73.8
36.01
10.0
71.2
32.8
2.0
79.9
47.0
20.0
Ul-Tan--B-Jpp
Singlo
0S-Allas-8-ppro
Single
82.1
51.5
83.5
30.6
81.2
2.0
52.3
24.0
Kimi VL-A3B
Single
75.1
31.7
12.0
61.8
21.5
6.5
56.9
15.8
1.0
71.7
29.9
11.0
DeepSock-VLL VL2
Single
72.6
38.8
16.0
73.1
35.2
9.5
70.5
32.1
2.0
79.2
46.1
19.0
U1-Tars-22--dpe
Single
GUI-OWL-7B
Singlo
91.7
71.2
37.5
96.0
68.4
28.5
90.8
52.4
6.0
93.8
67.2
31,0
UI-TARS-1.55 1,5-7B
Single
98.2
72.2
39.0
97.1
77.5
38.5
98.0
67.3
15.0
99.0
78.4
42.0
OpenCUA-32BB
Single
98.0
73.1
39.0
97.4
76.2
38.0
96.0
65.8
13.5
98.2
79.2
43.0
Mobile Agents
UI-RI-3B
Single
76.5
42.7
18.5
74.8
39.1
L0.5
72.6
33.9
6.0
$0.9
47.8
21.5
GUI-RI-3B
Single
77.2
10.9
20.0
76.1
38.6
11:0
74.1
36.8
5.5
$2.1
48.6
22.5
LU1--G--3
Single
GUI-RI-7B
Singlo
92.8
62.7
30.5
94.2
61.9
22.5
92.7
58.1
5.5
90.9
64.3
28.5
U1-S1-7B
Single
95.7
65.8
33.0
97.5
65.1
25.5
96.1
61.5
8.5
94.2
67.5
31.5
Reasoning Mehile
GLM-4.1 GLM-L.Iv-hiining
Single
78.4
12.8
17.5
80.7
44.6
17.0
71.3
30.2
2.5
77.8
44.9
18.0
Owen Brenn-QVQ-phae plas OpenAl 23-2025-01-16
Single
90.7
48.8
24.5
89.9
52.8
18.0
88.1
44.4
6.5
92.2
64.4
29.0
Single
94.2
68.1
33.0
93.5
58.2
21.5
90.7
52.2
9.0
72.3
33.5
Clande
Single
Doubao-1 1.5: Thinking pro
Single
98.2
75.8
39.0
98.1
77.3
38.5
95.9
60.1
14.0
99.0
78.0
41.5
Clinde 4,3 Sonnet
Single
98.9
76.2
39.0
98L5
77.1
39.0
98.5
69.2
15.5
99.0
78.3
43.0
Backbones. As shown in Table The Qwen and GPT series models show superior performance in common tasks, while others perform relatively poorly (50-70% compared to 20-40%). Due the weaker instruction-followw
and grounding capabilities of smaller models, their overall performance limited. The differences planning ability across models are relatively minor, as reflected the comparable TYPE accuracy. For one model, the multi-path SR is generally higher than the single-path counterpart, further indicating that multi multi-path evaluation better reflects the model's 's true capability.
Mobile Agents. As shown in Table compared with agent frameworks, different mobile agents-despite variations their action spaces exhibit substantially stronger grounding capabili- ties, leading superior overall performance relative to purely framework- based approaches. At the same time, their reasoning outputs are more concise, and the average reasoning steps are significantly reduced to roughly one-fifth of the original pipeline. Notably, the RL-based GUI-GI-3B achieves performance comparable to the pretrained Att--tas----pro, highlighting the potential of online rein- forcement learning for improving generalization. General-purpose multimodal thinking models, such as Doubao- 1oobao---5-TTie S---hinking-Paa pro, perform on par with specialized mobile agents (e.g., OpenCUA-322 When such thinking models integrated into the above frameworks, no performance gains are observed, suggesting a potential equivalence between frameworks and inherent thinking patterns.
Table 4: Results on AITZ-Noise. Qwen2-VI. VI. and OS-Atlas are evaluated on AITZ and AITZ-Noise.
General
Google App
Install
Web Shopping
Total
Agent
Benchmark
Step. Ace
Noisy
Step. Ace
Noisy
Step. Ace
Noisy
Step- Ace
Noisy
Step. Ace
Noisy
Qwen2-V1-ce VL-7B
Norual
38.5
44.8
60.0
45.1
46,9
12.1
OS-Atlas-7B
Normal
41.9
46.4
60.5
46.3
48.6
Noise
38.8
21.8
41.7
19.7
56.4
23,5
43.8
23.6
45.1
21.7
4.4 NOISY TASK RESULT
Out-domain SMAN-Bench-NeN demonstrate that only few VLMs complete task. All VLMs exhibit declining trend in Step.Acc, particularly LLAVA and Qwen2-V VL. while this phenomenon is also observed in open-source agents, due to the absence of noise in their training data. Unlike AITZ-Noise, the ads in Noisy-App dynamic and variable, as shown in Figure 16 Specifically. these noises exhibit the following three features: (1) After the pop-up ad countdown ends, the ad disappears automatically, and the agent's
results. For noisy data, the results the third block of Table
more

---

Published as a conference paper at ICLR 2026
Table Ablation study on SMAN-Bench-Amiiguuuss
AppAgent (Full)
AppAgent (Ambig.)
MobileAgent (Full)
MobileAgent (Ambig.)
Model
Type
StepAce
Type
StepAcc
Type
StepAcc
Type
StepAcc
VT.2441.
76.7 (+5.9)
(47.4)
(+.0.))
16.3 (+8.0)
-ven2--1
82.9
41.5
(+8.3)
(32.0)22.)
80.3
38.5
86.2
40.8
Po-an-VL-Max
81.2
39.3
90.3 (49.1)
(+9.2)
57.6
3,3
66.6 (49)
(.0.7)
LJama3.22 2.2. V-L-999
67.9
7.6
85.4 (+17.5)
15.5
57.8
4.0
64.3 (+6.5)
8.3 (+4.3)
GPT-4v
72.8
15.9
(+.6()))588
20.5 (44.6)
84.6
13.9
91.0 (+6.4)
(+1.7)
GPT-4o
79.7
31.9
94.7 (+15)
(329
38.4
91.6 (+3.9)
39.7 (+1.3)
delayed instructions may cause accidental taps; (2) Some video ads cannot be closed during the early viewing stages; (3) The mis-taps caused by real ad noise may trigger app redirection.
Out-domain AITZ-Noise results. As shown in Table 4 the Step.Acc of Qwen2- VL and OS-Atlas decreased by average of 3.0% and 3.5% from AITZ normal to in-domain AITZ-Noisy. Given that the Noisy step accuracy is 17.4% and 21.7%, this indicates that open-source agents fail to learn the features of advertisements because few ads still exist in their training data. They exhibit almost no generalization capability on transferred noisy data, even when only the background screenshot changes. When agents become trapped a page unrelated the current task, they struggle to determine how to proceed next. More details can be found in Appendix D.2
4.5 AMBIGUOUS INSTRUCTION ABLATION STUDY
As shown in the far-right column of Table 5 all agents exhibit improved performance (by up to 17.5%) when supplied with more informative context through step-by-step Q&A. Ablation results demonstrate that the active interaction module can help agents effectively ignore irrelevant content in task instructions for the current step. This is because full instructions may affect the agent's ability identify tasks on the current page accurately. In contrast, ambiguous instructions with step-by-step Q&A help the agent better comprehend the page and execute more appropriate actions. However, it is worth noting that not all agents benefit from this mechanism. Weaker VLMs (c.g., Intern +5.9%) struggle generate effective questions, while stronger VLMM (e.g., GPT-4o, +3.9%) are already capable of effective planning and decision-making without additional support. VLMs with intermediate performance, such LLaMA3.2. VL-90B, benefit more from this mechanism (+17.5%). Meanwhile, end-to-end agents struggle to formulate and explore questions aligned with the predefined settings, highlighting a critical limitation and a promising avenue for future research.
CONCLUSION
this paper, we propose SMAN-Bench, a realistic and comprehensive mobile agent benchmark that includes common instruction trajectories, noisy app split, noisy contaminated split, and ambiguous instruction split. We also propose a novel slot-based trajectory annotation method without human evaluation, named GIAS, and an offline multi-path evaluation method, which assess the agent's ability to obtain step rewards more accurately. This benchmark provides foundation for evaluating and optimizing GUI agent studies focused on searching for multi-path solutions, noise robustness, and proactive interaction.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have rigorously refined our dataset to remove any elements that could compromise personal privacy, thereby guaranteeing the highest level of protection for individual data. All data annotations whom we paid $0.5 per step compensation and
were completed by crowdsourced volunteers, provided the necessary training. The human evaluation of our work was carried out through a meticulously randomized selection of IT professionals. This process ensured gender-balanced and educationally diverse panel, reflecting a wide spectrum of perspectives and expertise.
REPRODUCIBILIIT STATEMENT
All evaluation code, prompts, and datasets used in this paper are open-source. The code and data are mounted and stored on GitHub and HuggingFace platforms, respectively. The full experimental setup is detailed in Section 441 and Appendix Unless noted, all experiments use the same settings. Overall, these practices make our results reproducible.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. /
http:/wwwworg/b///3333//3 08774
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-languae model for UI and infographics understanding. In Proceedings of the Thirty-Thine International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp- 3058-3068. ijcai.org, 2024. URL hnttps://wwwy ijcai iacaai..o wttg/proceeding//2224/333
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, De- cember 10 15, 2024, 2024. URL / www.:nisssccpppppeoc (T964cabbefffffee002222200------
cc/ wwppapppppppppeppepppeeeeee wwwpss/appee/02/////// html
Wendong Bu, Yang Wu, Qifan Yu, Minghe Gao, Bingchen Miao, Zhenkui Zhang, Kaihang Pan, Yunfei Li, Mengze Li, Wei Ji, al. What limits virtual agent application? omnibench: A scalable multi-dimensional benchmark for essential virtual agent capabilities. arXiv preprint arXiv:2506608833, 2025.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer. Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments. ArXiv preprint, abs/2104.08560. 2021. URL httpss://axxi....tt httpss///xxxiv.or////////// 08560
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents, ArXiv preprint, abs/2407.17490 2024. URL /ttp:ttts///rr wrtp://////bb//...///// 17490
Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang. Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NewrlPS 2024 Workshop on Open- World Agents, 2024a.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. ArXiv preprint. abs/2412.052711, 2024b. URL https://arxivv wttpp/wwwwoog/b///222222222
11

---

Published as a conference paper at ICLR 2026
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, ArXiv preprint, abs/2401.10935, 2024. URL/https: //axxiv / abs/ abs/2401. 2401. 10935
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium user interface software and technology, pp. 845-80 445-854, 2017.
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. Mobile-bench: An evaluation benchmark for Ilmebased mobile agents, ArXiv preprint, abs/2407 0.09999,,, 2024. URL lttpss://axxii wtwp:/iww..../////a//aooooosss/t 220/ 00993
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December nips. wappappppppppppcpppppppppeee..ee htcpc/ppprrrfeee/eeeeeeee
10 16, 2023, 2023. URL 5950bf29001150eea4b bf bb-rrraattttt----bttttttttt---- DatinttttDttttttt__ttet Bew.eaanndd_ncccakka.cacam html
Tinghe Ding. Mobileagent:
enhancing mobile control via human- interaction and sop 04124
integration. ArXiv preprint, abs/2401 2024. URLLnttpss//rradv.orr//bb/// 04122
Nicolai Dorka, Janusz Marecki, and Ammar Anwar. Training vision language model model smartphone assistant. ArXiv preprint, abs/2404.08755, 2024. (tt./htps:://rxii.orr//bbb///
08755
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL.httss:/aarii org/ abs 2407 21783
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. ArXiv preprint, abs/2410.05243, 2024. WRL/hhtps://arxit http:/aawxxivrrrraaea//
orgoorgg///o abs/ 2410. 05243
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint arXiv:2505.07062. 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for GUI agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA,
USA, June 16-22, 2024, pp. 14281 119881114200 IEEE, 2024. doi: 10. 10.110//CVVP32223.0020.0333
URL https: org 10
2024 01354
Yu-Chung Hsiao, Fedir Zubach, Maria Wang, et al. Screenqa: Large-scale question-ansser pairs over mobile app screenshots. ArXiv preprint, abs/2209.00 08 199, 2022. URL https://arxiv.org)
abs/2209.081 08199
Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, and Bo An. Enhance mobile agents thinking process via iterative preference learning, arXiv e-prints, pp. arXiv-2505, 2025a.
Zeyu Huang, Juyuan Wang, Longfeng Chen, Boyi Xiao, Leng Cai, Yawen Zeng, and Jin Xu. Mvisu-bench: Benchmarking mobile agents for real-world tasks by multi-app, vague, interactive, single-app and unethical instructions. arXiv e-prints, pp. arXiv-2508, 2025b.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li. Zejun Ma, and Chunyuan Li. danaa--nxt-intereeaan Tackling multi-image. video, and 3d large multimodal models. ArXiv preprint, 0ttp/240700799555 2024a. URL https://arxiv... org/ 20880000082 07895
12

---

Published as a conference paper at ICLR 2026
Wei Li, Williim E. Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyama- gundlu, and Oriana Riva. On the effects of data scale on UI control agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet. Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, Decem- ber 10 - 15, 2024, 2024b. URL http: http::::/apppps...pp. wwt.:nisssccpppppppoeeeppoo $79f3ef3b4455446555 hat.:/44446..4e44444eeaaaaaaaaaa
cc/ wwwpcpppppeeapeeeeeeeeeeeeeeeeee htt.e///ppper//2///aaaeaee and an__aanndd_dddaareee Track.
Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions. ArXiv preprint, abs/2408. 18824, 2024c. httpp::////x org abs/aa////2/// 2408. 110988.11822 11824
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 10.18653/v1/2020 www.aacllmmnnaaam main.729. URL
Online, 2020a. Association for Computational Linguistics. doi: wwwptthhooloy...g/222222 maap:/pppr..orr/00...mmmmaa..... 729
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Bonnie Web- ber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 5495-5510, Online, 2020b. URL
Association for Computational Linguistics. doi: https://aclantthol.og org/ emnlp-main waalllpmman..mmm 443
Yijun Liang. Ming Li, Chenrui Fan, Ziyue Li. Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj. Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness. arXiv preprint 2rXhv:2504.10514 2025.
Xiao Liu, Bo Qin, Dongzhu Liang. Guang Dong, Hanyu Lai, Hanchen Zhang. Hanlin Zhao, lat Long long, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. ArXiv preprint, abs/2411 1. .08200, 2024. URL htth::htts//rrtts htttps://arxx..org//b//////1// 2411 00820
Yuxuan Liu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, and Rui Yan. Mobilesteward: Integrating multiple app-oriented agents with self-evolution to automate cross-app instructions. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD '25, pp. 993-893, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 979840071 110000122466 2456. doi: 10.1 10.1145/3690024.370991 URL httpp::///oi..og//0...... 3769171
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross- app gui navigation on mobile devices. ArXiv preprint, abs/2406.08451, 2024. URL https //arxiv.org/abs//22
08451.
Wei Lu, Stephen Robertson, and Andrew Macfarlane. Fielh-weighted xml retrieval based on bm25, In Advances in XML Information Retrieval and Evaluation: 4th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2005, Dagstuhl Castle, Germany, November 28-30, 2005. Revised Selected Papers 4, pp. 161-171. Springer, 2006.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21622 2025a.
Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, et al. Ui-sl: Advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543, 2025b.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision- lisiun-language action model for gui agents. arXiv preprint 2r04v:2504 10458, 2025.
13

---

Published as a conference paper at ICLR 2026
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. Coco-agent: A comprehensive cognitive milm agent for smartphone gui automation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 9097-9110, 2024.
Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awalgaonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, et al. Mobileaibench: Benchmarking Ilms and Imms for on-device use cases. ArXiv preprint, Abs/2406.10290, 2024. URL https
wwwxxiv.org/bb//2200 2406 10290
Shikhar Murty, Dzmitry Bahdanau, and Christopher D Manning. Nnetscape navigator: Complex demonstrations for web agents without a demonstrator. ArXiv preprint, abs/2410.02907,, 2024.
URL https://arxiv org/ abs//410..... 24100000777077
Runliang Niu, Jindong Li, Shiqi Wang. Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. In Proceedings of the Thirty-Third International Joinl Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp.
ijeai.org, 2024. htt/hhtss://ww/ ijcai.icornn. org/
proceedinggs/0224711
Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. Mobileflow: multimodal 11m for mobile gui agent. ArXiv preprint, 0e/22007004466 2024. URL
https://arxiv wwterrivvv.or///// org.orgg///.....o 2b0/ 04346
Yaoyao Qian, Yuanli Wang, Jinda Zhang, Yun Zong, Meixu Chen, Hanhan Zhou, Jindan Huang, Yifan Zeng, Xinyu Hu, Chan Hee Song, et al. Webgrapheval: Multi-turn trajectory evaluation for web agents using graph representation. arXiv preprint arXiv:2510. 2025.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang. et al. Ui-tars: Pioneering automated gui interaction with native agents. ArXiv preprint, abs/2501. 12326, 2025. URL /ttp:/ttss///r...or////ba///
12326
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent. arXiv e-prints, pp. arXiv-2411, 2024.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December ht1les/paper/222//aahh 2223//ash/
10 16, 2023, 2023. URL / nips. cc/ wwtppperrr.ipsscccppppeeeeeeee htwpaatssstttsadddnnnnnnnnrreeee Benchmarks html
and
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. ArXiv preprint, abs/2405.11573 2024. URL kttps://aaxiv.org/abbs//
14573
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-38, 2009.
Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. DetermLR: Augmenting LLM- based logical reasoning from indeterminacy to determinacy. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 9828-9862, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.523 URL httphhts///aaaaatllllo/o hww.llogy.ooggggg// wrg/2024/ aa10024.ccl--100 acl-llonnee 5317
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. META-GUL: To- wards multi-modal conversational agents on mobile GUI. In Yoav Goldberg, Zoritsaa Kozareva,
and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natu-
ral Language Processing, pp. 6699-6712, Abu Dhabi, United Arab Emirates, 2022. Associ- ation for Computational Linguistics. doi: 10.18653/w1/2022.ooom www.:/mmmmmmmmm44/.1101 m49..449. URL https! 7/aclannthllog.orgg/22
449
14

---

Published as a conference paper at ICLR 2026
Tao Tan, Quan Tu, Songhao Wu, Hongda Sun, Chuanqi Cheng. Weikai Xu, and Rui Yan. Luoling: An immersive cross- moosl interactive poem creation factory through multi-agent collaboration. In Companion Proceedings of the ACM Web Conference 2025, WWW '25, pp. 2911-2914, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400713316. doi: 10.1 00.114557701771633111844.
http:httss///oi.orrr 11411.11111113711111 3715184
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. ArXiv preprint, abs/2504.07491,
2025. URL htthhttss///xxittt wwtprriiivoogg/bbb... orggabs//5000/ 2504../25...000 07491
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. ArXiv preprint, 0bs/2105.15 13231, 2021. URL. httpt://////xiii http://orgg/////////////e wtgg/ab/////.....////// 3231
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2creen Au- tomatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 498-510, 2021.
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile using large language models. In Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (eds.). Proceedings Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 1-22:17.. ACM, 2023. doi: 10. 00.114553544444.3588888 URL https://dol. wrg/o...///00 11 11111115444445 3580895
of the 2023 CHI Conference
Hongru Wang. Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Pan, and Kam- Fai Wong. Appbench: Planning of multiple apis from various apps for complex user instruction. 19743
ArXiv preprint, 11.022400.194433 2024a. URL htttps://arxiv.org/g/a
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent--2: Mobile device operation assistant with effec- tive navigation via multi-agent collaboration. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024. 2024b. URL http::/paperss nips. cc/paper_ ptcpppper_flllsppppppp_e_eeeeppp ftwess/aperr//22/// h2024/ash/ 00220777700000111622222223322222
html
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang. Tianchen Min, Wei Chen, and Shoufa Chen. Mobileagentbenehh: An efficient and user-friendly benchmark for mobile Ilm agents. ArXiv preprint, abs/2406.08 84, 2024c.
http://axxivoorrr////////// 08184
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. ArXiv preprint, abs/2409. 2024d. URL httphttss///artt org/ / 2409.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang. and Kun Shao. Distrl: An asyn- chronous distributed reinforcement learning framework for on-device control agents. ArXiv preprint, abs/2410. 14803, 2024e. URL hnttps:/aaxxi..orr//////////////
14803
Wenhao Wang, Zijie Yu, Rui Ye, Jianqing Zhang, Siheng Chen, and Yanfeng Wang. Fedmabench: Benchmarking mobile agents on decentralized heterogeneous user data. arXiv preprint arXiv:2503.05144. 2025a.
Wenxuan Wang, Juluan Shi, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, and Michael R Lyu. Learning to ask: When Ilms meet unclear instruction. ArXiv preprint. Abs/2409.00557, 2024f. URL httph:tt//// http:p////aaaa..oo/aab//////////
Xinyuan Wang, Bowen Wang. Dunjie Lu, Junlin Yang, Tianbao Xie. Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu. Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents. arXiv preprint 25X0::550000012333 2025b.
15

---

Published as a conference paper at ICLR 2026
Zhenhailong Wang, Haiyang Xu, Junyang Wang. Xi Zhang. Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. aebite-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.1173: 2025c.
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Empowering Ilm to use smartphone for intelligent task automation. ArXiv preprint, 0tts/2308.15272.. 2023. httpp::////a httpp/////aaaii.oo/abbb/////////
15272
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang. Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Lim-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp. 013-557,22 2024.
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevim: A vision-language model for better intra-and inter-ui understanding, ArXiv preprint, 1445/2499.14188 2024a. URL https: orgg /a/////bs//...//////// 14818
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. ArXiv preprint, abs/2410.2322 2024b.
23218
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture--f--epern vision-language models for advanced multimodal understanding. ArXiv preprint, abs/2412.100002,, 2024c. URL
https://arxiv.org// abtp://a.22222..../ 2412. 10302
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, et al. Crab: Cross-envirommen agent benchmark for multimodal language model agents. In Findings of the Association for Computational Linguisties: ACL 2025, pp. 21607-21647, 2025a.
Weikai Xu, Chengrui Huang, Shen Gao, and Shuo Shang. llmedased agents for tool learning: A survey: W. xu al. Data Science and Engineering, pp- 1-31, 2025b.
Weikai Xu, Zhizheng Jiang, Yuxuan Liu, Pengzhi Gao, Wei Liu, Jian Luan, Yuanchun Li, Yunxin Liu, Bin Wang, and Bo An. Mobile-bench-ye A more realistic and comprehensive benchmark for vlm-based mobile agents. arXiv preprint arXiv:2505.I 11891, 2025c.
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng. Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. ArXiv preprint, abs/2410.24024. 2024a. URL httpss://arxivvoorr /
2410.24024
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android au- tonomous agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2144 216622 2025d.
Yiheng Xu, Zekun Wang. Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. ArXiv preprint, abs/2412.04454, 2024b. /tL/htts://arrxvvorr/// ab///////222222 2412.04454
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. ArXiv preprint, abs/2312. 2023. URL https
weexxiv.org/bss/// 13771
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweamgin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal 1lms. ArXiv preprint, abs/2404.05719 2024. URI. httt::::///riii.oo///////000010 05719
16

---

Published as a conference paper at ICLR 2026
Danyang Zhang, Hongshen Xu, Zihan Zhao, Lu Chen, Ruisheng Cao, and Kai Yu. Mobile-env: an evaluation platform and benchmark for Imm-gui interaction. ArXiv preprint, abs/2305. 04144, 144, 2023. URL httptstt/taaiii...tttaa org/ abs/ 2305. 08144
Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. Mobileexperts: A dynamic tool-enabled agent team in mobile devices. ArXiv preprint, abs/2407.03913. 2024a.
URL htttp:://arxiv.orr/bbb/////
03913
Jiwen Zhang. Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of www.aa-io---ttuuuueoeo for gui agents. ArXiv preprint, abs/2403.02713,, 2024b. URL https: /ttppp:://rre //ap:vv./a/ivv/v/aa www.oo/g///b////e 02713
Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation. ArXiv preprint, abs/2404.16054, 2024c. URL http:////aaiii.oogaa///////////
16054
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chainoof--ctiooooii agents. ArXiv preprint, abs/2309. 11436, 2023. htt.lttpss//ariioorrr/////211111
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision)) generalist web agent, if grounded. In Forty-first International Conference on Machine Learnngg, ICML 2024, Viema. Austria, July 21-27, 2024. OpenReview.net, 2024. URL BosLLnttss///peerrvieeeeeee////s piecKJ2D1B
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Weharena: realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13884 2023.
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-gl: Understanding rl-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505. 15810, 2025.
A MOBILE TASK FORMULATION
For mobile agents, there are four essential capabilities: (i) Overall Planning determine the action step sequences. (ii) Action Thought to produce an action description at each step (e.g. "open the flight detail page"), (iii) Element Grounding to identify widget (e.g. "[Click](rr the GUI, (iv) Action Reflection to determine whether the next GUI matches the expectation.
[x1, x2. U2),
â. E Click
ā:
{1,4, +
à. E Scroll
(I)
text,
à, E Type
Given a mobile screenshot S (e.g., a Ctrip screenshot on Android) and task T (e.g., "Book flight ticket from Chengdu Beijing Sep.15 55 for me." "), a GUI agent should generate sequence of executable actions. Specifically, at time step i, the agent should select an action a from the action space A, which includes three types of actions: (1) Click. (2) Scroll. (3) Input. Based on the current
environment observation S,, the action history P1 a2 .... d }. and the last step reflection fe-1, the GUI agent will generate plan P.:
P.
a
a(m) (Ô1, , fi
where P; represents the planning of the next n actions starting from the current step. The environment observation S₁ comprises HTML document tezt, and a mobile screenshot image,.
B
THE FULL RELATED WORKS
B.
MOBILE AGENTS
Large language models (Achiam et al., 2023) emerge as autonomous agents (Li et al. 2024b) Wen al. 2023) in the mobile domain and garner considerable attention. With the rapid development of
17

---

Published as a conference paper at ICLR 2026
vision-language models (VLMs), multimodal researchers build mobile GUI agents Yang et al.
2023
Zheng et al.] 2024) and multi-agent frameworks (Ding) 2024, Li et al. 2024c Wang et al.
2024b)
based on closed-soune source VLMs. Meanwhile, some researchers focus traiing agents with stronger
element grounding (Cheng et al. 2024| Hong et al. 2024, Wu et al. 2024b), ,, page navigation (Niu
et al.. 2024 Lu al. [024] Gou al. (024)) GUI understanding (Chai et al. 2024 You et al. 2024
Bacchler ct al., (2024) and task planning capabilities (Zhang et al. 2024c Nong et al. 2024 Xu et al.
2024b) based on open-source VLMs. In addition, Digirl (Bai et al.) (024) and Distrl (Wang et al.
2024e) use joint online and offline reinforcement learning to enhance the generalization of mobile agents and mitigate performance degradation when facing APP updates and unseen APPs. Some researchers (Qinghong Lin et al., (024) explore optimizing VLM structures. For example, Dorka (Dorka et al. (024) optimizes the encoder by incorporating historical images and actions as input.
B.2 OFFLINE MOBILE AGENT BENCHMARKS
As shown in Table 1 AndroidEnv Toyama et al. 202 and MobilEnv (hang al. 2023) are the
first to create LLM agent evaluation environments based on reinforcement learning. Mobile Bench (Deng al. 2024 and AppBench (Wang et al. (024a) introduce online benchmarks combining
API and GOI, while MobtleAgeenteench (Wang et 2024c) establishes the first fully automated
multimodal benchmark for VLM-based GUI agents. More offline benchmarks (Li al.] 2020a
Burns et al. 2021] Murthy et al. 2024) are released, which are primarily categorized into GUI
understanding and task-oriented. (1) For task-oriented benchmarks, AITW (Rawles et al. (023) and
AITZ (Zhang et al.) 2024b) create large-scale benchmarks based on Google apps, while AMEX (Chai ct al. 2024) supplements these benchmarks by adding data for GUT understanding with similar app types. ScreenSpot (Cheng et al. 2024), Mobile3M (Wu et al., 2024a), and GUTOdyssey (Lu et al. 2024) focus on more granular element grounding and task planning. (2) Rico (Deka et al. 2017) is the first non-annotated GUI corpus, followed by ScreenQA (Hsiao et al. 2022). Widget Caption (Li ct al. 2020b), and Screen2words (Wang et al. (02T), which is for Q&A, widget understanding. and page summarization. Subsequently, Mind2web (Deng al. 2023) incorporates additional GUI data
of varying sizes, and Meta-GUI (Sun et al. 2022 provides tasks for multi mouti-round dialogues.
B.3 ONLINE MOBILE AGENT BENCHMARKS
Recently, more online benchmarks (Xu et al. 2024a Rawles et al. 2024 Chen al. 2024a Wang cet al. 2025a Huang et al. 2025b) have been released to evaluate agents in real environments, such as smartphones, PCs, and web browsers. Existing OS-level benchmarks, such as OSWorld (Xie let al. 2024)). focus on open-ended, long-horizon computer-use tasks on desktop operating systems. AndroidWorld (Rawles et al. 2024). by contrast, provides 20 applications and 116 programmatically generated daily tasks within a controlled Android emulator, along with structured reward signals, enabling reproducible evaluation of mobile GUT agents. AndroidLab (Xu et al. (025d) further extends this line of work by operating in real near-real Android environments and constructing large families of tasks from composable atomic skills, thereby systematically assessing an agent's s generalization across task compositions and across applications.
B.4 GRAPI-BASED GUI AGENT BENCIMARKS
Recent efforts in evaluating GUI-based and web-oriented agents have produced several benchmarks that differ sub- stantially in task domains, instruction sources, graph- structured trajectory modeling. and reward assignment strategies. ColorBench (Liang et al., (025) focuses on long-horizon, cross-app search and query tasks driven by manually written instructions. It adopts VLM-based page-content descriptions and embeds them to compute similarity scores for node merging, while rewarding agents through sub-task verification of necessary actions in both online and mixed evaluation environments. WebGraphE- val (Qian et al., 2025). in contrast, targets shopping and web-retrieval scenarios within WebArena, where multiple
vipttp
Bruet
18
Figure 6: Categories and apps used in the Mobile3M Dataset

---

Published as a conference paper at ICLR 2026
Benchmark
Task Type
Instruction Source
Graph Node Merging Strategy
Key-Node Rewarding
ColorBench
Search & query; cross-app long- horizon tasks
Manually writ- ten instructions
VLM-based page de- scription; embedding similarity used merge nodes
Sub-task position; verifying necessary actions for reward Multiple agent trajecto- ries merged by action- sequence similarity
decom-
WebGraph
Shopping & re- trieval in web tasks
WebArena tasks
Final-an -nnwer correct- ness prop-aatee to earlier steps
CRAB
Calendar, map. web
Reverse DAG decomposition to generate instructions
Cross-device node alignment
Sub-task decompo-
sition; reward via checking environment state
OmniBench
Office, video editing, web
Reverse DAG decomposition
Trajectory-node merg- ing for instruction gen- eration
None; additional intent signals ensure data qual- ity
SMAN- Bench (ours)
Everyday tasks: music, travel, search, shop-
Online log fil- tering + GIAS- based synthesis
Action-space validity checking + page- content clustering
Slot-based matching to locate correct action and downstream key-node page; rewarding equiv- alent actions
ping; more app calegories
Table 6: Comparison of SMAN-Bench with existing graph-base and task-oriented GUI agent benchmarks.
agent trajectories for the same task are aligned and merged by comparing action-sequence similarity. Rewards for intermediate steps are back pack-propagated solely from the correctness of the final answer, enabling offline evaluation enhanced with LLM-as-judge. CRAB (Xu et al. (025a) extends to heterogeneous environments such as calendars, maps, and general web interfaces; its instructions are pro- duced through reverse DAG task decomposition, and node alignment is handled across devices. Rewards are obtained by verifying environment states after sub-tasks, and evaluation is conducted through cross-systedd online interaction. OmniBench (Bu et al. 2025) further broadens the domain to office automation, video editing, and complex web tasks. It also reverse DAG decomposition but merges trajectory nodes to generate high-quality instructions, supported by additional intent signals instead of explicit reward modeling. Its evaluation emphasizes sub-task-leve assessment. Overall, these benchmarks reveal a diverse design landscape- from instruction construction and trajectory graph fusion to reward propagation and evaluation modes- modes-bigghlihtin the need for more unified and robust frameworks for training and assessing long-hou Long-horizon multimodal agents.
As shown T compared with previous benchmarks, SMAN- (ours) targets more realistic everyday mobile scenarios, including music, travel, search, and shopping hhopping-sspaning broader set of apps. Instead of relying on handcrafted tasks or constrained trajectories, we construct instructions through online log filtering and GIAS-based synthesis. For trajectory consolidation, SMAN-Bench combines action-space validity checking with page-content clustering, enabling more reliable merging of semantically equivalent states. Reward assignment is also more fine-grained, slot-based matching automatically identifies the correct action and downstream key-node page, allowing both intermediate rewards and final-answer evaluation in fully offline setting. These designs make SMAN-Bench more scalable and realistic benchmark for long-horizon GUI agents.
C MOBILE3M DATASET
Mobile3M is a large-scale dataset designed to systematically explore and analyze the functionality of mobile applications through Ul-based interactions. It provides comprehensive representation of user interface (UI) elements, interactions, and app navigation patterns. Mobile3M is characterized by the following key features:
19

---

Published as a conference paper at ICLR 2026
Catggory
APP
Unique Nodes
All Nodas
Action Stegs
All Nodes(1)
Calegory
APP
Unique Kodes
All Nodes
Action Staps
All Nodes (%)
Tikiing
arjuke
57,286
190,107
1,334,428
6.13%
Raader
awokbooks
15,000
70,265
563,682
Liking
36
903.506
4.78%
Reader
QQReeler
477,500
Living
smertbume
12.585
12,961
300,816
1.39%
Renler
shustusheng
14,777
63,210
392,903
2,05%
Teare
ctip
63,109
187,079
122773006
6.00%
Ronder
pdizeader
495
1,507
5,211
0.08%
Trane
Quiar
42,462
161,000
1,213,015
5,20%
Social
xiaohongstu
40,324
B0,302
523,019
2.70%
Shopping
rpshhp
72,488
168,511
1,036 036
5.44%
zhiho
21,796
57,261
373,756
Stopping
SiaomiShop
21
90,770
755,718
3.22%
Social
QQ
7
20,600
141,969
0.66%
Stupping
dueg
18.925
30,926
223,379
1.26%
Educaiion
aurgellung
19.884
70,66]
507.146
2.28%
Transport
didi
12,706
81,865
637,100
2.78%
Eduration
Xiasyuaz
10,727
$6,806
393,395
1.83%
Tranport
calniao
20.503
73.133
480,223
2,36%
Education
Youdso
35,121
206,035
1.13%
Trensport
gaodeMap
319,377
Edication
Baitihhn
16,383
88,500
0.53%
Transpoct
BsidaMap
13,552
51.322
260,198
1.78%
Office
wgeOelice
11,156
73.739
105.061
2.58%
Browser
UCMobdle
40,518
50,220
615,069
2.88%
Office
Netmail
5,561
32,308
260,682
1.00%
Brwseer
36,016
70,282
401,348
2.27%
Office
banghuashun
6,410
30,722
163,297
0.98%
Brwwser
00Bmar
18,500
44,006
218,808
1.42%
0Qmail
1,580
4.597
0,05%
Brwwsr
224,804
1.23%
Video
bill
471,940
Syslem
24.759
105,461
624.941
3.40%
Valeo
qulive
12,497
22,603
99,677
0.73%
System
upiinsecure
8.997
12,691
379.925
1.38%
Video
kusishor
7,125
12.115
59,373
0.39%
System
Judazhi
2,773
32,474
219,804
1.08%
Picture
undroidesi
28,032
59.228
418,773
1.91%
System
gedownlonder
10.517
28,302
151,824
0.52%
Pictur
mixx
19,738
56,324
419,005
System
aicsltter
15,819
$0,005
0.51%
Health
medicinetalpe
15,046
83,832
347,690
2.71%
System
superranulto
630
1,369
5,444
0.04%
Health
keep
7,230
22,500
117,126
Music
xmmaloya
34,985
103,286
577,032
3.34%
oatheer
punowaather
25,252
73,283
610,685
Music
kageu
40,043
94,271
564,368
3,04%
Weather
doudwwothhr
1,056
3,904
19,330
0.13%
Music
OOnusic
5,545
17,589
64,211
0,57%
15
49
$98,334
3,008,786 736
20, 138.33
100%
Figure 7: The data distribution in Mobile3M.
(1) Scale and Diversity
Mobile3M includes over 20 million user interactions, covering 3 million screenshots and correspond- ing XML documents. These data are organized into directed graphs for 49 widely-used Chinese apps, where nodes represent UI pages, and edges capture user actions.
(2) Detailed UI Representation
Each UI page is described by both screenshot and an XML document. The XML documents provide detailed structural information, including UI elements (e.g., buttons, text fields), their hierarchical relationships, and layout properties such bounding boxes.
(3) Action Space
The dataset defines three fundamental user actions-click, scroll, and input-to to simulate real-world app interactions. Each UI page contains an action space derived from its interactable elements, facilitating comprehensive modeling of user behaviors.
(4) Graph-Based Organization
Mobile3M employs breadth-first search (BFS) algorithm to explore app functionality, representing the exploration results as graphs. This structure enables the identification of app workflows, the relationship between UI pages, and the possible transitions triggered by user actions.
(5) Efficiency and Optimization
To enhance exploration efficiency, Mobile3M incorporates a "unique page" mechanism that eliminates duplicates by comparing UI pages using combination of element and pixel- based similarity thresh- olds. This reduces the exploration space, prevents redundant actions, and avoids cyclic sequences, ensuring more diverse and meaningful data coverage.
(6) Balanced Action Distribution
The dataset emphasizes balanced representation of user actions by prioritizing underrepresented interactions, such as input. For example, random keywords are introduced for input actions, and scroll actions are executed in multiple directions to capture diverse app behaviors.
(7) Fask-OOiented Exploration
Inspired by APPAgent, the dataset leverages random walk algorithm systematically interact with UI elements and record transitions between pages. The exploration process captures action traces, enabling driven navigation and detailed understanding of app functionalities.
20

---

Published as a conference paper at ICLR 2026
D DATA ANALYSIS AND CONSTRUCTION
D.1 GIAS PROMPT
Input Format
You will be provided with series of user interaction histories, each consisting of a caption describing the current page and an action performed by the user.
Your Task
Analyze each action and the corresponding page caption to determine what action was taken on that page. Summarize these actions task description, which should be a request. For example:
"I want to see what VIP privileges are available."
"Help me find pants on sale."
"Tell me what items are in my shopping cart."
Important Notes:
1. The task description and the sequence of actions should have logical relationship.
2. The task description should be phrased as request, reflecting the goal of the actions taken.
3. Actions and captions should analyzed sequence to deduce the user's objective.
Output Format
"step-by-step description": "Provide series of interactions, where each entry corresponds to screenshot caption of the current phone screen and the action performed on that page."
"concise task": "Summarize the user's overall goal based on the step-by-step description."
Example
Caption 1:
This image shows a screenshot of a shopping application interface.
Action 1:
Ser)
Caption 2:
This image shows a screenshot of a shopping application interface. At the top, there is a search bar with the text "Skincare Set." Additionally, at the bottom of the page, there is navigation bar with options like "All Products," "New Arrivals," "Moisturizing. "Dry Skin," " "Niacinamide," and "Hyaluronic Acid." The current state is "All Products."
Action
Click[New Arrivals)
Caption
This image shows screenshot of a shopping application interface. At the top, there is a search bar with the text "Skincare Set." Additionally, there is a navigation bar at the bottom with options like "All Products." "New Arrivals." "Moisturizing," "Dry Skin." "Niacinamide," and "Hyaluronic Acid." The current state is "New Arrivals." Below are multiple product recommendations.
Action 3: Click(Ad)
Caption 4:
This image shows a product detail page. At the top, there is a pink banner that reads "Buy a set and get 13 items free," along with a product photo.
Output:
"step-by-step description":
21

---

Published as a conference paper at ICLR 2026
1. Click the "Skincare Set" product under the "Beauty" subcategory of " "Recommended."
2. On the Skincare Set search results page, click the "New Arrivals" tab.
3. On the product details page, click the "Ad" tab.
"Concise task" Help me find the latest skincare set that is on promotion.
New Input and Task
Now, based on the following input, please generate the "step-by-step description" and "concise lask":
(trajectory_.eecriipiion
D.2 DATA CONTAMINATION
The collected advertisements are shown in Figure TO We embed them into the normal dataset and applied background whitening. We ensure that the elements that should have been clicked on the current page are no longer visible after the contamination. When splitting the training and test data, the position of the embedded advertisements randomly assigned. However, the types of advertisements in the training data are largely consistent with those in the test data, and the same advertisements maintain consistent embedding positions.
Table 7: Qwen2-VL and OS-Atlas fine-tuned AITZ-Noise, AITW, AITZ and evaluation ATTZ-Nois((Out-omaiin) Metric "Noisy" means out-domain noisy step accuracy.
Agent
Training Data
Step. Ace
General
Noisy
Step. Acc
Google App Noisy
Install Step. Ace
Noisy
Web Shopping Step. Ace
Noisy
Total Step. Ace
Noisy
Normal Supervised tuning
Qwen2: VL-7B
ATTZ AFTZ
37.33
15.38
42.18
17.11
$4.68
20.45
42.06
17.14
43.84 45.16
17.46
OS-Atlas
38.81
21.79
41.61
19.73
56.37
23.57
43.71
23.57
21.62
In-domain Noise Supervised Fine runing
QwenZVL
AITZ Naisy
43.07
77.56
47.63
73.68
60.64
75.76
44.02
75.00
48.20
75.79
OS-Atlas
ATTZ. Noisy
44.92
82.05
49.64
76.32
79.55
78.57
79.56
Out-domain Noise Supervised Fine-tuning
Qwen2VI.
Naisy
49.90
OS-Atlas
ATTZ + Naisy
41.75
53.85
45.47
48.65
60.27
60.90
47.28
55.71
48.69
55.47
E EXPERIMENT DETAILS
F. BASELINE MODEL. DEMONSTRATION
AppAgent Below is the prompt used. We did not re-adapt adjust the prompt for different base models to ensure fairness.
I will give you the screenshot mobile app. clickable UI element is labeled with a letter's 'c' and the number <ui element> on the screen. The tag of each element is located
at the center of the this UI element is necessary part of procecding with larger task, <task <tassttaasssstse description>.
element. Clicking
which
In order to realize this larger task, you must first realize the current task ccurrent_tas__dest in current screenshot.
Your task is to describe the functionality of the UI element concisely in one or two sentences.
Notice that your
description the UI element should focus on the general function. For example, if the
element is used to navigate
to the chat window with John, your description should not include the name of the specific
person. Just say: "Clicking this will navigate the the chat window". Never include the tag of the
22

---

Published as a conference paper at ICLR 2026
12:16 1
O 4.40
Google
flights from Mexic
All
Flights
News
Images
M
M
Ad cwc.pattccccccpt.ccocooo
Ac
+
Cheap Boston Airline
C
Tupperware'
Tickets Book and Fly wit...
T
Tupperware
Save With Our Cheap Airline Tickets to Boston. Book Tickets Today from SOO..
S
Tupperware last
Tupperware to last
One Way Flights
Tupperware well-known brand that specializes in plastic containers
Tupperware is well known specializes containers
* Discover
Q Search
A Collections
Spen
Zone
AD top-up
GUT Sereenshot
Polluted GUI Screenshot
Figure 8: Contaminated datasets are constructed by inserting advertisements and whitening the background of the original GUT screenshots.
Pojin regestoo Contuaaion
vin lideyy
WIKIDATA
ceat
Welcome to Wikidata
Conmentyy sortel
hoe coaelagge lene with 18:2000b5 duta (emr that enpore
Gunder
Weee manalas?? alasing mesii
CRTEDENRI
BBI Vee
IB Lean durat
Ialedd curoo
unmer contiriale wit
Siottruaca maaoe ataccand
Figure 9: The red boxed represent identical graphical controls and identical textual controls, which can create ambiguity in the action history.
UI clement in your description. You can pronouns such as "the UI element" refer the element.
Listing 1: Click Document Template
will give you the screenshot a mobile app, the clickable UI element is labeled with a letter's 'c' and the number <eieeeeiieeee elemment> on the screen. The tag of each element is located
al the center of the element. Clicking this UI element is necessary part of proceeding with larger task,
which <task description>.
In order to realize this larger task, you must first realize the current task <current task desc>
in current screenshot.
Your task is to describe the functionality of the Ul element concisely in one or two sentences.
Notice that your
23

---

Published as a conference paper at ICLR 2026
MO ID
2
E 01 61
Tupperware'
LLC Formation &Bank Account
StanGiobal probien
Tupperware to last
Banking & Company Formations
businens non en demr
SUICK 1 CORR
Tupperware well-known braind that specialiaes in plastic containers
MuttiNattonal Service Provider- Extensive Client Base Covers Virtually Country.
Open
LLC Formation Bank Account BartDaaall ves the prbbler starting
Check Grammar Instantly Wrthe Engish ve willing kan Trynow
BIER 10 #333 S2 40:0: 18:0 1018
SarGGook
Gamneary
Open
18:8
Figure 10: A collection of pop-up ads, collected from Google service official apps, third-party market apps, and mobile apps in mainland China.
IIMAL PHREEAY
55
JUBALL POEN
120
of
RNF
Pdin FED
STER
LECCOO BOL
ASBANEN CRCB
ESTAVES
AREEI
E
37 20484
11E BELNETE
Origin GUI
Labeled GUI
Origin GUI
Labeled GUI
Figure 11: AppAgent GUI labeled method.
description of the UI element should focus on the general function. For example, if the UT element is used navigate
24

---

Published as a conference paper at ICLR 2026
the chat window with John, your description should not include the name of the specific person. say:
"Clicking this area will navigate the user to the chat window". Never include the tag of the
UI element in your description. You can use pronouns such as "the UI element" to refer to the element.
Listing 2: Click Documentation Template
documentation of this UI element generated from previous demos is shown below. Your generated description should be based on this previous doc optimize Notice that is possible that your
understanding of the function of the UI element derived from the given screenshots conflicts with the previous doc,
because the function a UI element can be flexible. this case, your generated description should combine both.
Old documentation of this UI element: cold_docs
Listing 3: Refine Documentation Suffix
You are agent that is trained to perform some basic tasks smartphone. You will be
given
smartphone screenshot. interactive clickable elements on the screenshot are labeled
with tags starting from "cl".
The interactive scrollable UI elements on the screenshot are labeled with tags starting from "sl
Thhe tag of cach
interactive element is located in the center of the element. Every screenshot I've given you
screenshot after
executing the correct action.
You can call the following functions control the smartphone:
click(element str)
10 This function is used to click an Ul element shown on the smartphone screen. "element" is tag assigned
UI element shown the smartphone screen. simple case be click(e5), which taps the UI element labeled with "c5".
13
14 2. input(text input: str)
15 This function is used to insert text input in an input field/box. text_input is the string you want
to insert and must
16 be wrapped with double quotation marks. simple use case can be text(" world!"), which inserts the string
17 "Hello, world!" into the input area on the smartphone screen. This function is usually callable when you see screenshot
about text inputing.
19
20 3. scrolleelement: str, direction: str)
21 This function is used to scroll an UI element shown on the smartphone screen, usually scroll
view slide bar.
22 "element" tag assigned an element shown on the smartphone screen. "direction" is string that
23
represents one of the four directions: up, down, left, right. "direction" must be wrapped with double quotation
marks.
25 simple use case can be swipe(s21, "up"), which scroll up element labeled with "s21".
26
27 <ui document>>>
28
The task you need to complete is to <task frass_ddscriptions. to complete this task you should perform current task
25

---

Published as a conference paper at ICLR 2026
Table 8: Quality verification results. Crowdsourced annotations verified by agent professionals.
Metric
Simple
Complex
Noisy
Ambiguous
Annotation Step
5.62
8.21
12.74
7.53
Evaluation Step
5.57
8.07
12.69
7.43
Win Rate?
96.0
92.0
100.0
100.0
SET
1.01
Table 9: App counts and proportions in full and subset datasets.
App
Full
Subsel
Full (%)
Sampled (%)
Maid
52
2
0.43
0.56
QQBrowser
3.72
anjuke
2773
82
23.11
23,16
hunduBrowser
682
20
5.68
5,65
bili
2658
78
22.15
22.03
kugou
943
28
7.86
7.91
medieinchelper
7.06
laplap
1570
46
13.08
12.99
vipshop
7.44
ximalaya
1151
34
9.59
9.60
<current task desc>>, Your past actions to proceed with this task summarized follows:
last acD>
50 Now, given the documentation and the following labeled screenshot, you need to think and
call the function needed to
31 proceed with the task. Your output should include three parts in the given format:
Observation: <Describe what you observe in the image>
33 Thought: <To complete given task, what the next step should do>
34 Action: <The function call with the correct parameters to proceed with the task.>
35 Summary: <Summarize your past actions along with your latest action in one or two sentences
Do not include the
36 tag in your summary>
37 You can only take one action at time, so please directly call the function.
Listing 4: Task Template
E.2 SAMPLED DATA QUALITY VERIFICATION
Ngg Swg.rtiiiionn
poterente
aimalaye
and entice bo
Figure 12: The task distribution chart is sorted by the number of simple tasks in descending order. The average steps for both simple and complex tasks in each app remain relatively balanced.
E.3 SAMPLING REPRESENTATIIVEESS
To construct a representative evaluation subset, we perform stratified sampling from the full dataset, with application-level proportions preserved. As detailed in Table 9 the sampled subset closely mirrors the original distribution across a diverse range of applications. Despite a substantial reduction in data volume, the relative proportions of key apps (e.g., anjuke, bili, taptap) remain nearly identical between the full dataset and the sampled subset (c.g. 23.11% vs. 23.16%, 15% vs. 22.03%). This strong alignment demonstrates the effectiveness of our sampling strategy in maintaining distributional
26

---

Published as a conference paper at ICLR 2026
Benchmark
Data Source
Music
Travel
Shopping
Office/Tools
Mobile-Benche
Deng
Online log filtering
93
92
98
at 2024
WebArena Zhou et al.
Human-written
88
90
(2023)
template-generated
AndroidWorld Rawles
Human-written
90
85
93
et al. 2024
template-generated
SMAN-Bench (ours)
Real logs template- generated
92
96
90
93
Table 10: Comparison of benchmarks, data sources, and category coverage quality.
fidelity. By preserving both frequent and less frequent app categories, the subset ensures that evaluation results remain reflective of real-world deployment conditions. Consequently. this sampled dataset serves compact yet reliable benchmark for downstream agent performance analysis.
E.4 HUMAN EVALUATION STUDY ON CONSISTENCY
To assess the consistency between our synthesized instructions and real user behavior, conducted an additional human evaluation study. The experimental setup is as follows: we invited four professional colleagues who frequently use music, travel, shopping, and office applications, respectively. Each evaluator was asked to assess 400 instructions sampled from these four domains. For comparison, we also provided them with instruction sets of the same categories from other existing benchmarks. The evaluation criterion was whether an instruction matches what they would naturally say when interacting with their mobile voice assistant. The results are shown Table 10
E.5 DISCUSSION
Table 11: Qwen2-VI, CogAgent, and OS-Atlas evaluated on AITZ-Noise. Metric "Noisy" means in-domain noisy step accuracy. More experiments be in Table 7
Agent
Training
Step.
General Noisy
Step.
Google App Noisy
Step.
Install Noisy
Step.
Web Shopping Nony
Supervised Fine- muning Settin((LakA)
CogAgent- 8B
ArTW(CoaT)
40.4
11.5
38.1
11.3
45.2
17.3
39.1
13.4
Qwen2-NL-7B
AUTZ(Caal)
Qwen2-V1-7B8
ATTZ-Noise
99.1
60.9
30.4
41.5
99.0
13.3
OS-Arlas-BB
AfTZ-Noose
46.2
99.0
18.7
50.2
99.5
21.3
62.4
100
33.0
44.8
99.0
17.3
Supervisod Fine-tuning Setting(Full)
Qwen2-VLLc
At-Aths-BB
AlTZ-Noise
47.2
98.0
19.0
47.1
99.0
22.3
66.7
99.0
38.0
$1.8
99.0
23.5
Solving in-domain noise through post-training. We are more focused on whether increasing the proportion noisy training data can address the in-domain noisy problem. As shown Table [T] Qwen2-VL, compared to the original AITZ training data, shows Step.Ace improvement of 3.7%, 3.2%, 10.0%, -0.3% and SR improvement 3.4%, 5.4%, 9.7%, -1.9% the four sub-tasks. At the same time, full parameter fine-tuning outperforms LoRA in overall results but performs slightly worse than LoRA on noise step processing (an average of 1. 1.5% lower). After training, the agent is able to correctly handle the vast majority of noisy steps (with an accuracy greater than 97%), demonstrating the effectiveness of training with noisy data.
Unusable noises. We exclude the keyboard-oceluste current evaluation protocol, VLM agents take both the GUI screenshot and the parsed HTML as input. The handwriting keyboard occludes part of the on-screen content in the GUI image, whereas the same content remains fully accessible in the HTML DOM, leading to inconsistent visibility signals and additional confounding. As shown in Fig. [3] in the dictionary app the words contour and trailer are visually blocked by the handwriting keyboard in the GUI screenshot, but they are still exposed in the parsed HTMI..
noise type from our evaluation. Under the
27

---

Published as a conference paper at ICLR 2026
E.6 BEHAVIORAL ANALYSIS OF GUI AGENTS.
To better understand the behavioral characteristics of current GUI agents, we conduct an analysis along three major dimensions: repeated loops, hallucinated interactions, and navigation inconsistency, These analyses cover both single-pane path and multi-pani settings and are evaluated across diverse set of frontier VI.M-based agents.
Analysis of Repeated Loops in Multi-Path Tasks. In our benchmark, single-path tasks follow pre- defined, deterministic sequence actions; therefore, no repeated loops can occur. By contrast, multi- path tasks allow free path selection, making repeated loops natural indicator of unstable decision- making or inefficient exploration. We adopt an n-gram n-gram-based based repetition metric to characterize these behaviors. For each trajectory, compute: (1) Repeat Length: average length of repeated action subsequences. (2) Repeat Count: average number of repeated subsequences. (3) Length-2 Count: frequency of minimal oscillations (e.g., "next-back").
Table 12: Repeated-loop statistics on simple tasks.
Model
Repeat Length
Repeat Count
Length-2 Count
Qwen2.5-7B
2.0845
1,6901
1.16
Qwen2-72B
3.0048
2.4105
1.9657
GPT-4o
0.6351
0.4844
0,2375
GLM-4.1
1.6714
1.2571
0.5667
Table 13: Repeated-logg statistics on complex tasks.
Model
Repeat Length
Repeat Count
Length-2 Count
Qwen2.5-7B
3.2210
2.4842
2.00
Qwen2-72B
4.8108
3.4241
3.42
GPT-4o
0,1194
0.0895
0.04
GLM-4.1
1.9142
1.2285
0.2766
Hallucinated Interactions. We further study interaction hallucination, where an agent predicts the correct action type but executes incorrect action instance. We quantify hallucination frequency
using:
Hallucination Ratio = Step Accuracy
Type Accuracy
A large ratio indicates difficulty grounding predicted actions onto the GUI.
Table 14: Hallucination analysis across simple and complex tasks.
Model
Cat.
Type S.
Step S.
Halluc. S.
Type C.
Step C.
Halluc. C.
CogAgent- 18B
Single
75.6
20.9
72.3
62.3
20.8
66.6
UGround-78 7B
Single
73.0
39.5
45.8
73.8
36.0
51.2
UI-Tars-7B-dae -B-
Single
75.3
41.8
44.4
75.9
37.8
50.1
7B-ppro
Single
82.1
51.5
37.2
83.5
50.6
39.4
Kimi-VL-A3B
Single
75.1
21.7
71.1
61.8
21.5
65.2
DeepSeek-VL.2
Single
72.6
38.8
46.5
73.1
35.2
51.8
UI-Tars-7BB-dpo
Single
94.3
64.2
31.9
96.0
63.5
33.8
GUI-OWL-7B
Single
94.7
71.2
24.8
96.0
68.4
28.7
U1-TARS-1.5-B
Single
98.2
72.2
26.4
97.1
77.5
20.1
OpenCUA-328
Single
98.0
73.1
25.4
97.4
76.2
21.7
Navigation Inconsistency. Navigation confusion evaluated using:(1) StepAce StepAce/RR SR: a high ratio indicates correct micro-actions but poor global navigation. (2) Path Efficiency: ratio between actual and optimal path lengths.
We report results multi-path tasks:
28

---

Published as a conference paper at ICLR 2026
Table 15: Navigation inconsistency analysis on multi-path tasks.
Model
SE
StepAcc
SR S.
StepAcc/SR
StepAce C.
SR C.
Intern BicemV12-a VL--40B
5.8
43.0
10.1
4.2
46.0
6.5
VVL-2B
5.2
62.8
20.6
3.0
58.9
7.0
Qwen-VI. VL-Max
67.6
12.6
5.3
9.6
GPT-4v
6.1
29.7
3.0
9.9
29.4
1.5
GPT-4o
5.3
61.8
19.8
3.1
61.7
16.5
22.00
10 R. 200+
22:08
22 LaD
1 B-9
BW
13 BIRS
606-6
himl: index-"0 "0" test- teatt'conntur" connoou"" resource id- eusoft endicrid/ist class- "andvoid. widget, TextView" class-"mndvoid,
Nov
Nov
make ourselves miserable we make oursehes strong The amount of work is the same.
make ourselves iibibrrbleee we make ourselvesaleeg Ste-emounts work is the samee
package= ewsfft
content-dese=""
checkable- "falte" checked- "false"
elickable- "false" enabledt "wvee
RI CU St (5 (57 0E 9L.
R8 St, BE2G 82TE 468821
focusable "Julse" focusedt- "fulse"
scrollable- "false long clickalic "Jolse" password- "Jalse" selected "false" bounds bounds-"]1144 "f114,18 (1114.816)/225,18666
pitch
picc
2120: Mil: 1:6: 101008 st
SUB) 0036 8:15 18
smodl index- "0" lest "trailer" id- "com ewsoft eudic: word' clays "undvoid, widget TexiView" package- "com ewsoft eulic"" content-desg-- checkable-"fais "fatee" checked- "false"
Affogato
Affogato
gaugo
clickable- "false" enabled "wreee
contour
focusable "Jolse" focused "falo "faloo"
0::0 N8:: W8th 02. B8
scrollable "false"" long-clickable- "Jolse"
A
F
G
K
Valler
paseword= "Jalse" selected "false"
bounds (iiiss-"111,1111111111 (1114,1900/233.200
O
x
128
Figure 13: A type of noise rendered unusable when the text and image information conflict.
E.7 TEST CASE STUDY
Common-Simple
2. Common-Complex
Figure shows a Common-Complex case of SMAN-Bench and the GIAS results are as follows:
1. On the "My Gold" page of the mobile app. click the "Category" tab enter the book category page.
2. On the category page, select the "Plot" category under the "Boys" tab.
3. In the plot category, select the "Return of the Strong" category to enter the list of books in this category.
4. In the "Return of the Strong" category, select the book "The First War God of the North."
5. On the book details page "The First War God of the North," click the rating 8.1 view the ratings and reviews.
6. On the review page, click "Must-see masterpiece" to view specific book review details.
7. Enter the comment "Science Fiction" on the book review details page and submit it.
Task: Help me find and evaluate a book called "The First War God of the North", view its ratings and related reviews, and add your own feedback under specific reviews.
29

---

Published as a conference paper at ICLR 2026
Help me find the current popular audiobook content and browse different audiobook categories.
Instruction:
SCT
WB
A
WAF
DE
EE
+
(
9-4
0
04
Step-by-step description:Inl. In1. In the main interface the music app, click the "A "Audiobook" option to enter the audiobook anea.!n2. In the audiobook page, swipe up to browse different audiobook content. |n3. Continue swiping in the audiobook page view more audiobook categories and content.in
Figure 14: Common-Simpll case.
Instruction: Help find and evaluate book called "The First War God of the North", view its ratings and related reviews, and add your own feedback under specific reviews.
APP-DD
B R-ARAANR
2
T002
Figure 15: Common-Complex test case.
3. Noisy Data
4. AITZ-Noisy
5. Ambiguous Data
30

---

Published as a conference paper at ICLR 2026
Instruction: Please help me find Beijing's quality index for tomorrow on Caiyun W Weather.
16%
Figure 16: Noisy Data test case. The red shadow in the GUI screenshot are advertisements, pop-ups, or tutorial noise steps.
Instruction: Search for flights from Mexico city to Boston
Google
G
Mattchhnn Moii
Cheapithrur Aetire
Coogle
dagle
Telett Book 2nd7
Tomeue
Boca Baak Fuee
Teppoweree
Ono Way flight
Tights
Ogms More
Hights
fighs
Cit 100
Choqp Bldl
Sine
Tights
totho frrm
Figure 17: AITZ-Noisy test case. The red shadow GUI screenshot in the trajectory is artificially inserted noise.
6. Slot- Moo--eemlltee Based Instructions From Table 17 instructions sounding "programmatic" or "mechanical" indeed exists, and is predominantly concen- trated in complex tasks with larger number of slots. Long and logically structured instructions are difficult to generate through templates, as they typically require coherent dependencies between earlier and later parts of the instruction.
we can observe that the issue of slot-based
31

---

Published as a conference paper at ICLR 2026
Table 16: Test cases study on SMAN-Bench-Ambiguoore
Ambiguous Instruction
Find information about movie
Q: Which app should be used?
A: Use Douban.
Q: Which app does this page belong to?
A: Douhan.
Q: which section should search?
A: Movies.
Q: Do you want browse specific rank-
A: Yes.
ing?
Q: For which time period?
A: Upcoming relcases.
Q: How should ranked?
A: By popularity.
Q: Which movie do you check?
A: The most popular upcoming movie.
Q: What information you need?
A: complete summary.
Full Instruction
Find the most popular upcoming movic on Douban
Ambiguous Instruction
Find midnight snack
Q: Which app should be used?
A: Use Ele.me.
Q: Would you like to filter by specific snack
A: Find new items from the nearest store that can
categories, speed.
Q: Any other conditions?
A: deliver within 30 minutes.
Q: Do you have specific price range?
A: No specific price range.
Q: Do you have preferred cuisine or taste?
A: No preference, just quick delivery within 30 minutes.
Q: Which section should you search?
A: Food delivery.
Q: What are the speed requirements?
A: Within 30 minutes.
Q: What are the distance requirements?
A: Nearest store.
Full Instruction
Find delivery item from the nearest store Ele.me that can deliver within 30 minutes.
Type
Slot
Slot-template based
LLM-vised
Human-annotated annotated
Quality
Easy case
Open "QQ Browser" for me.
Open the "QQ Browser" mobile
Open "QQ Browser" mobile app for me.
Good
app for me.
Normal
2
Set alarm for "tomorrow morning" "7:30".
Use the clock app alarm for "tomorrow morning" at "7:30".
Please wake me at "7:30 tomorrow morn- ing" using alarm, keep the phone sleep mode before that.
Fair
Difficult
4 Book a flight containing "this week". "Friday". "Beijing" to "Guangzhou".
Book a direct flight from Beijing to Guangzhou "Friday this week".
Book direct flight from Beijing to Guangzhou this Friday, prioritizing shortest flight time and secon- darily optimal price.
Poor
Table 17: Instruction complexity across slot count, generation method, and quality level.
7. Input words When collecting data, each app category is associated with predefined query pool containing 200 queries. These predefined inputs are aligned with the category and actual functionality of the corresponding app. Table [8 presents two illustrative (and slightly reduced) examples from these query pools.
F LIMITATIONS
Although multi-pati multi-path validation similar to that on online machines was achieved on MMAN-Bench. the diverse range of text inputs cannot be exhaustively covered, which differentiates it from online machines. Advanced agents such as AutoGLM (Liu et al. (024) and others deployed by smartphone manufacturers could not be tested due permission restrictions.
32

---

Published as a conference paper at ICLR 2026
App Name
Category Pre-defined Words
Ctrip
Travel
[[Beijing to Shanghai, Chengdu round trip Scoul], [Harbin, Beijing, Japan, New York. Chengdul, [Singapore 4-day tour, Tibet self-driving tour]. [Direct flights, Connecting flights], [High-end hotel chains, Budget hotels, Huazhu Club, Marriott, Sheraton], [Popular attractions, Folk culture, Itinerary planning, Food sharing), [Ancient town folk culture, Local festival events], [World heritage sites, National 5A scenic spots], [Ski resorts, Island vacation spots], [Food districts, Night mar- ket exploration]. [Hiking trails. Mountain camping sites]. (Family-friendly routes, Elder-friendly itineraries]. [Budget travel tips, Premium customized travel]. [Free- travel pitfalls guide, Public transport guide]. [New Year travel plans, Weekend short trips), [Local transfer schemes, Best sightsecing order), MMust-cat lists, Street food exploration], [Michelin restaurant recommendations, Scafood buffet picks], [Local breakfast guide, Night-market snacks collcction), |Café map, Dessert shop check- ins], [Specialty food rankings, Niche food sharing]]
QQ Music
Music
[Jay Chou songs, JJ Lin albums, G.E.M. hit singles]. [Faye Wong classics, Beyond classics, Mayday live versions]. [New releases, Top singles chart, Pop hits chart], [KTV hot songs, Bar singer selections]. [Dance tracks, DJ mixes, Electronic beats]. [Wedding BGMs, Love confession songs, Romantic BGMs), [High-quality loss- less, Spatial audio, Master tape qualityl. [Live versions, Remixes, Demo versions), [Instrumental versions, Pure music, Humming tracks), |Copyright versions, Exclu- sive tracks], [Song ID by listening. Melody-humming recognition], [Lyrics Keyword-based song search]. [Beat-based search, Mood-based search]. [Cover- image search, Audio-snippel search], [Official MVs, Live performances, HD stage videos), [Music short videos. Trending edit BGMs]. [Physical albums, Vinyl records, Digital album purchase], [Viral challenge BGMs, TikTok hits, Xiaohongshu trending music]. [Gaming battle themes, eSports BGMs]. [Study-focu white noise, ASMR white noise], [Children's 's songs, Farly education music), [Yoga/meditation BGMs, Relaxing natural sound effects]]
search,
Table 18: Pre-defined word groups for Ctrip and QQ Music in English.
G USAGE OF LLM STATEMENT
This paper utilized an LLM to improve the clarity and fluency of the text.
33


Reviews:
[{"review_id": "5VDydOW7bZ", "reviewer": "Reviewer_aiCQ", "summary": "The paper presents SMAN-Bench, a large, cross-system benchmark for mobile GUI agents that explicitly targets four real-world difficulties: Single-path bias, Multi-path execution, Ambiguous instructions, and Noisy screens (ads/pop-ups). It is built on top of the large graph-like mobile interaction corpus Mobile3M (49 popular apps, tens of millions of transitions), from which the authors automatically construct 12,856 bilingual instructions via a slot-based generator (GIAS). Each instruction can map to one or multiple valid trajectories, enabling both single-path and multi-path evaluation. Two special subsets—Noisy (ads, interstitials) and Ambiguous (underspecified user goals requiring clarification)—are added to stress-test current agents. Experiments on a wide range of mobile agents (AppAgent, Mobile-Agent variants, recent GUI-R1/G1-style models, and MLLM-based agents) show consistent drops on noisy/ambiguous settings and significant gains when multi-path scoring is enabled, indicating overfitting to a single gold path in prior work.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Multi-path as first-class citizen. Unlike benchmarks that match only one golden trajectory, SMAN-Bench merges multiple observed paths into a graph and scores hitting key nodes; this is much closer to how real mobile tasks have many valid sequences. \n- Noisy & ambiguous subsets. Explicitly injecting ads/pop-ups and instruction under-specification is valuable, because current agents often break exactly there and existing benchmarks seldom isolate this factor. \n- Scale and coverage. 12k+ bilingual tasks, 49 apps, 15 categories, and cross-system (Android, iOS, HarmonyOS/HyperOS) make it a broadly useful offline benchmark. \n- Good diagnostics. By reporting both single-path and multi-path scores, the benchmark can tell whether an agent failed because it “did the wrong thing” or because it “did a right thing that wasn’t the gold path.”", "weaknesses": "- Instruction naturalness. The GIAS is template/slot-based; although it gives nice coverage, some instructions may read slightly synthetic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0FMKHDcP5L", "reviewer": "Reviewer_FYRd", "summary": "This paper presents SMAN-Bench, a comprehensive benchmark for evaluating vision-language model (VLM)–based mobile agents across diverse and realistic settings. Built on the Mobile3M graph corpus, SMAN-Bench introduces an offline multi-path evaluation method that assesses agents across Single-path, Multi-path, Ambiguous, and Noisy task modes, providing both reproducibility and flexibility.\n\nThe benchmark features a slot-based instruction generation method (GIAS) for scalable annotation, and includes cross-system coverage across Android, iOS, HarmonyOS, and HyperOS. It also incorporates robustness and interaction evaluation, testing agents under ad noise and ambiguous instructions. Extensive experiments with open- and closed-source VLMs under multiple agent frameworks demonstrate that SMAN-Bench captures realistic challenges and reveals performance differences across reasoning, robustness, and multi-path capabilities.", "questions": "1. **Offline simulation mechanics**: When an agent issues an unrecorded action, does the simulator terminate or ignore it? Are any mobile emulators used during evaluation?\n2. **Slot template coverage**: Are slot values limited to those observed in Mobile3M, or can instructions include unseen values?\n3. **Cross-system alignment**: How are UI pages and key nodes aligned across Android, iOS, HarmonyOS, and HyperOS? Is the alignment manual or automatic?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Novel and well-motivated benchmark design**: The offline multi-path evaluation successfully bridges the gap between deterministic offline datasets and unstable online evaluations. By rewarding progress toward “key nodes” within the graph rather than requiring exact path matches, the benchmark achieves both reproducibility and multi-solution coverage.\n2. **Comprehensive task diversity and realism**: SMAN-Bench extends evaluation beyond task completion to robustness and interaction. The Noisy split (ads, pop-ups, redirects) and Ambiguous split (Q&A clarifications) emulate real-world conditions that are rarely tested in prior benchmarks such as Mobile-Agent-Bench or SPA-Bench.\n3. **Scalable annotation pipeline (GIAS)**: The GIAS method systematically extracts action intents and fills template slots to produce thousands of instructions without manual labeling. This approach reduces annotation cost and links one instruction to multiple trajectories, supporting multi-path evaluation.\n4. **Thorough experimentation**: The authors evaluate more than twenty models under three frameworks and analyze success rate (SR), step accuracy, and step efficiency (SE). Results reveal that multi-path evaluation correlates better with true capabilities than single-path evaluation, and ambiguous instructions enhance reasoning VLMs such as LLaMA and GPT-4o.", "weaknesses": "1. **Clarity and writing quality**: The paper is dense and often unclear about implementation details. For instance, the “offline simulator” and the definition of “key nodes” are spread across sections without a cohesive overview. Figures 1 and 2 contain too much information without step-wise guidance. A clearer explanation of evaluation loops and agent–graph interaction would aid reproducibility.\n2. **Evaluation protocol ambiguities**: It is unclear how actions not present in the Mobile3M graph are handled (for example, unseen coordinates or slot values). Does the simulator terminate, discard the action, or return an error? Similarly, the definition of the “predefined query pool” is not quantitatively specified, and the interaction between step limits and noisy tasks remains unclear.\n3. **Limited comparison with related graph-based benchmarks**: The paper focuses its comparison on Mobile-Agent-Bench and SPA-Bench but omits recent graph-structured evaluators such as ColorBench [1], WebGraphEval [2], CRAB [3], and OmniBench [4]. A discussion contrasting SMAN-Bench’s key-node reward with these graph-centric methods would clarify its unique contribution.\n4. **Under-explored cross-system analysis**: Although SMAN-Bench covers four OS ecosystems, the results mainly aggregate performance without analyzing per-OS differences or transfer difficulties. Understanding how models generalize from Android to iOS or HarmonyOS would strengthen the “cross-system” claim.\n5. **Slot-value and coverage uncertainty**: While the paper mentions a predefined query pool, it is unclear how slot values such as song titles or colors are bounded. If an instruction contains a slot not present in the dataset, the paper does not explain how correctness is judged. \n\n### **References**\n\n[1] ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks. https://arxiv.org/abs/2510.14621v1\n\n[2] WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation. https://arxiv.org/abs/2510.19205v1\n\n[3] CRAB: Cross-Environment Agent Benchmark for Multimodal Language Model Agents. https://arxiv.org/abs/2407.01511\n\n[4] OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities. https://arxiv.org/abs/2506.08933", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMfH3epEcz", "reviewer": "Reviewer_PBxb", "summary": "This paper constructs a large dataset of evaluating mobile agents with the combination of both dynamical and static evaluation strategies. The benchmark also contains common usage noises such as ads jumping. The authors claim they use DFS and BFS for searching trajectories, while not clear how to make sure the trajectory is reasonable and how to annotate the dataset, since the instruction are trajectories should be highly aligned. The evaluations are extensive on multiple LLMs.", "questions": "How to make sure the benchmark is reliable and consistent to common user using?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The articulation of current benchmark limitations is clear. The proposed mixed evaluation is reasonable. The constructed benchmark is quite diverse, including multiple diverse apps. The evaluation on multiple LLMs both with and without noises are appreciated.", "weaknesses": "The dataset synthesis process is unclear, especially if using rule-based or algorithm-based methods, how to make sure the trajectory is reasonable. Is human annotation needed to construct the benchmark? As for the innovation part of the benchmark creation, could authors clarify the difference from previous benchmarks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PenxpCCbpw", "reviewer": "Reviewer_frsK", "summary": "This paper makes a meaningful benchmarking contribution to the field of mobile GUI agents. SMAN-Bench addresses key shortcomings in existing benchmarks by introducing evaluations under multi-path, noisy, and ambiguous conditions, which better reflect real-world user interactions on mobile devices. The experimental analysis is extensive and compares numerous models and agent frameworks, providing valuable insights into current system limitations.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Slot-based automatic instruction generation from unlabeled mobile GUI corpora is interesting.\n\n2. The paper is well-written and easy to follow.\n\n3. The paper conducted extensive evaluation across many mobile agent systems.", "weaknesses": "1. The noise simulation includes ads, popups, and redirections, but does not cover other common mobile noise sources such as keyboard overlays, partial screen scrolls, background notifications\n\n2. The paper claims to support multi-path evaluation, but it is not clearly explained how all valid alternative paths are identified, annotated, or guaranteed to be complete. If the benchmark only includes a limited subset of possible valid paths, models that take an alternative (but logically valid) route may be unfairly penalized.\n\n3. While they report success rate (SR) and step accuracy (Step.Acc), the paper does not provide deeper agent behavior analysis (e.g., confusion navigation, hesitancy, repeated loops, hallucinated interactions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)