LongHorizonUI: A Unified Framework for Robust long-horizon Task Automation of GUI Agent
Authors: Bin Kang
Shaoguo Wen
Yifei Bi
Shunlong Wu
Xinbin Yuan
Rui Shao
Junle Wang
Zhuotao Tian
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal large language models
Primary area: applications to robotics, autonomy, planning
TL;DR: LongHorizonUI integrates element-indexed multimodal perception, hierarchical reflective decision-making, and rollback-based compensatory execution for long-horizon GUI control.

Abstract: Although agents based on multimodal large language models (MLLMs) demonstrate proficiency in general short-term graphical user interface (GUI) tasks, their robustness remains a significant challenge for handling complex long-horizon tasks in dynamic environments . In response, the LongHorizonUI framework is proposed to improve the sustained reliability of agents in long-horizon GUI tasks. To overcome core limitations, we establish a comprehensive long-horizon benchmark, LongGUIBench, covering multiple categories of games and complex general applications, with long-horizon tasks defined as requiring more than 15 steps for rigorous evaluation of long-horizon reasoning capabilities. Based on this, a Multimodal Enhanced Perceiver is designed to incorporate element detection and text recognition models, assigning unique indices to interface elements, thereby reinforcing state representation. Furthermore, a Deep Reflection Decider engine is introduced, incorporating a structured multi-level feedback validation mechanism to enable progressive reasoning and ensure accurate action execution with predictable trajectories. Finally, we introduce a Compensatory Action Executor that combines multiple degradation compensation operations with a process rollback strategy based on execution progress monitoring to ensure operational effectiveness in long-horizon task logic. Experimental results demonstrate that LongHorizonUI achieves substantial long-horizon modeling improvements on LongGUIBench while retaining competitive performance on diverse public benchmarks. The code and models will be publicly available.

Published as a conference paper at ICLR 2026
LONGHORIZONUI: A UNIFIED FRAMEWORK FOR Ro- BUST LONG-HORIZON TASK AUTOMATION OF GUI AGENT
Bin Kangl.27 Shaoguo Wen
Yifei Bi
Shunlong Wu
Xinbin Yuan Rui Shao? Junle Wang Zhuotao Tian 2
Chengdu Institute of Computer Applications, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences Tencent Turing Lab
Georgia Institute of Technology TTiinghua University
Nankai University 'Shenzhen Loop Area Institute
ABSTRACT
While multimodal large language models (MLLMs) have shown promise short-horizon GUI agents, their performance degrades significantly long- horizon tasks involving complex, dynamic interfaces. To address this, we present LongHorizonUI, framework designed enhance the reliability and robustness of MLLM-based agents in extended interactive environments. Moreover, we es- tablish new long-horizon benchmark, named LongGUlBench, encompassing complex general applications and various gaming scenarios. Long-horizon tasks in this benchmark are defined those requiring more than 15 steps, enabling thor- ough evaluation of long- long-horizon reasoning capabilities. Building upon this bench- mark, we develop a Multimodal Enhanced Perceiver that integrates element detec- tion and text recognition models, assigning unique indices to interface elements, thereby reinforcing state representation. Furthermore, we introduce Deep- validation
Reflection Decider, which employs structured multi-level mechanism to support iterative reasoning and guarantee precise action execu- tion along predictable trajectories. Building the Deciders outputs, Compen- satory Action Executor continuously monitors execution progress; when degrada- tion is detected, it applies targeted compensation operations or triggers a rollback procedure, thereby maintaining robustness throughout long-horizon tasks. Ex- periments show that LongHorizonUI substantially improves nonghoorizon perfor- mance on LongGUIBench, while remaining competitive on diverse public bench- marks. The code is publicly available at https:: happs:///kaekkaaggg qi qithub. io/ LongHori zonUT/
INTRODUCTION
Graphical user interface (GUI) agents (Hong et al. 2074; Wang all, 20256: Huang et all 2025c Yll 2025; Dan et all (0224) are increasingly utilized in dynamic interactive environments to automate diverse workflows. The advancement of multimodal large language models (MLLMs) (Luu at 2023% all 2023: Lin all 2024; Zhang et al] 2024: el 2025) has notably bolstered the capabilities of GUI agents in tackling more intricate scenarios, enabling them not only to handle simple, short-term tasks (Hong all 20ZE, Sun al 20258) but also engage in complex, long- horizon interactions within gaming environments (Huang et all 20258) and enterprise applications.
Recent work (Sun all 2025kt Fan et all 2025) has investigated online reinforcement learning to improve adaptability by generating training data through environmental interactions. However, the trial-and-error learning paradigm expands the action space and amplifies cumulative errors over long horizons. Moreover, most of the existing benchmarks (Li all, 20245; Lu all 2024a; Chai et all.
2025: Rawles all 2023) designed for short-term taskstypically fewer than 15 steps, shown
"Equal Contribution
'Corresponding Author.

---

Published as a conference paper at ICLR 2026
Deep Reflection
Next
Mutimodal Enhanced Perceiver
1
Recommended
Decider
Action
2
Farm
Historical Validation ?
Executor
Success
LongGUlBench
Target Check
Index: "n"
Relative: "n,(x.)"
Failed
N
Action Explainabilie ?
steps
Hong
Absolute: "(x.y)"
Callback
Task text
Element Controls
Pre execution Reflction?
Retry
Figure I: LongHorizonUU first builds an indexed element set (icons/text) via enhanced perception, next drives the MLLM with structured prompt to validate history and derive multiple action can- didates, and finally applies a three-stage executor (index, relative, absolute) with monitoring. This pipeline sustains reliable execution sequenees exceeding 15 steps.
in Figure M.. and thus fail to support long-hooe honghhorizon evaluations. Consequently, developing reliable GUI agents in long-horizon tasks remains a significant challenge.
Key Observations. To investigate the challenge of current methods in the long-horizon task sce- narios, we conduct experiments by evaluating state-of-the--rt methods (Liu et all, 2023: Qin et all, 2025: Zhang et all, 20256) on the AndroidControl benchmark all 20245) across sequences of varying lengths, shown in Figure 2R Specifically, for sequences with <5 steps, average success rates exceed 90.0%. However, performance degrades sharply as sequence length increases. When sequences exceed 10 steps, the average success rate drops below 75%; for sequences longer than 15 steps, falls to approximately 60%.
This non-linear performance degradation clearly indicates that current methods may fail to capture long-horizon state dependencies, allowing errors to accumulate exponentially as sequence length increases. Once the sequence length exceeds a certain threshold, the agent system collapses due to the inability to maintain cross-step contextual consistency. To address this issue, we need to answer the question: how can we design GUI agents that maintain contextual coherence and decision- making proficiency over Long-horizon action sequences?
Our Solution. In this work, we propose LongHorizonUI, a framework for enhancing the robustness of MLLM-based GUI agents in complex and long-horizon tasks, shown in Figure M Specifically, first, we propose a Multimodal Enhanced Perceiver (MEP) that integrates object detection and OCR outputs to capture rich contextual information, assigning indices to UI elements for temporally sistent state representation. Then, we design a Deep Reflection Decider (DRD) that performs struc- tured, multi-level reasoning through formatted prompts, enforcing explicit validation of historical coherence, goal relevance, and action justification to ensure the decision fidelity. Finally, we incor- porate Compensatory Action Executor (CAE) that implements multi-level fallback strategy by leveraging the element indices, relative layout priors, and absolute screen coordinates. Concurrently, real-time progress monitor captures screen states and execution outcomes to construct temporal state chain, enabling reliable rollback and recovery from execution errors.
Moreover, to comprehensively evaluate the performance in long-horizon scenarios, we introduce LongGUlBench, a new benchmark that consists of tasks requiring more than 15 steps across di- verse gaming and application scenarios. It comprises 371 scenarios: 207 from 13 games and 147 task chains from 15 apps. Data were collected by professional testers, 6 human experts, via synchro- nized actionscreen recording, followed by cross-moda alignment and standardized parsing. Exten- sive experiments both existing benchmarks and the proposed LongGUlBench demonstrate that LongHorizonUI outperforms existing methods by over 3% in task success rate, without sacrificing the generic performance.
To summarize, our contributions are as follows:
We propose LongHorizonUL, a GUI agent designed for long- horizon reasoning, enhancing perfor- mance by an improved perceiver, a structured deep reflection decider, and a multi-level compen- satory action executor.
2

---

Published as a conference paper at ICLR 2026
Dmssets
Avg: 13.4
Avg: 8.2
ATW
Success
Avg: 6.8
Inggert
Avg: 22.1
Prelrrrnss Ziea (459)
Chttll Dedine
Fulure Dan
Stops numbers
Sequence Steps
(a) Sequence length distribution.
(b) Effect of length on performance.
Figure 2: (a) Step-length distributions across GUI datasets. LongGUlBench shows markedly longer horizons (Avg: 22.1 steps) compared GUIOdyssey (13.4), AITW (8.2), and AndroidControl (6.8), emphasizing evaluation beyond short episodes. (b) Success rate sequence length on AndroidCon- trol. All baselines degrade as horizons grow, with sharp drops beyond 10-15 steps; LongHorizonUU (ours) sustains higher SR and delays the critical decline, remaining competitive up 18 steps.
We introduce LongGUIBench, a new benchmark for long-horizon GUI interaction comprising verse complex tasks from multiple application domains requiring more than 15 steps, with expert- annotated state trajectories and goal specifications.
Extensive experiments on public benchmarks and LongGUIBench demonstrate that LongHori- zonUI outperforms state-of-the-art methods in long-horizon tasks while maintaining competitive performance standard settings, validating its efficacy and generalization capabilities.
OUR METHOD
In this section, outlined in Figure B. we introduce LongHorizonUl, framework dedicated to long-horizon reasoning for GUI agents. Building upon the LongGUlBench benchmark spanning complex games and general application workflows, our approach integrates three core components: (i Multimodal Enhanced Perceiver integrating OCR and icon detection, (ii) Deep Reflection Decider enabling action verification and adaptive planning, and (iii) a Compensatory Action Execu- tor ensuring robust action execution. The following sections detail each element. The discussion of related work in the Appendix C due to the page limit.
2.1 LONGGUIBENCH
In this section, we present LongGUIBench, benchmark designed for evaluating long-horizon GUI tasks by simulating real-world, dynamic interactive scenarios. The dataset is constructed through synchronized collection of action sequences and screen snapshots, captured by profes- sional testers they execute predefined test cases across diverse applications and games. All tasks mandate least 15 steps (mean steps: 22.1). Following cross-modal temporal alignment, the collected action commands and screenshots are input into MILMs, leveraging structured prompts combined with screen perception algorithms, the MLLMs parse operation descriptions (e.g., "click the search bar") and extract semantic control annotations, including button function- alities and bbox coordinates. This process generates a standardized intermediate representation with key-value structure that includes the global descriptions ttask_nname and decomposed sub- goal descriptions (task_stepp (act ion__D_ ID, act saciiooddde soo_descripttteeeeeeeeeeeeeeee_e intittinoon ion, act ceto_ontppee.. bbox, image_we wawagewwitth/aaeaaaaaaaaaaa.aaaa wwwth/hhiiigii..e hith/hheigh///// )). Finally, manual noise filtering yields long-horizon dataset containing 371 scenarios.
Gaming Scenarios. Games typically involve complex interactive processes. To this end, we col- laborate with professional testers to construct 207 high-complexity scenarios spanning 13 popular games, covering core mechanics such as equipment management and event participation. Each sce- nario is structured long-horizon task chain (19 to 37 steps, mean 23.7 steps), captured in 4508

---

Published as a conference paper at ICLR 2026
LongHorizonBench
LongHorizonUT
Scene: General
Prev-Refiection
Deep Think Descion
Step nums :18-31
"Succo"'
text search
OCR:
history reflection
navignsd ckerty
Histroy
butoo"" (restion"
MLLM
Scene: Game
Step nums:22-34
Preception
Compensating Action Actuator
OCR
Callback
Icon Detect
Retry
Relaiive (n/C-y1
Absolute [s.
Elistory (h., by))
(pos:"n")
Enhanced Perception
Past-Reflection
Illustration of LongHorizonüü framework. The LongGUlBench is introduced to define complex Long-horizon interaction scenarios. An enhanced perceiver (OCR + Icon detection) ex- tracts enriched UI element features, while deep reasoning engine performs three-tier closed-loop validation of action feasibility. The compensation actuator employs multi-stage strategies (index/rel- adivefabsolutehhsstoriaall coordinates) for robust execution.
Figure 3:
screen images to simulate real player decision flows. Each task includes two levels of instructions: High-Level instructions (HL) define macro goals, such as "purchase item XX in the game store," while Low-Level instructions (LL) are broken down into atomic operation sequences, such "click the store button," and "click the purchase button." Additionally, every operation step is annotated with fine-grained UT metadata, including control type (e.g., button, text box, drop-down menu), bbox coordinates, and state attributes.
General Scenarios. To assess generalization capability, we constructed 147 end-to-end task chains across 15 popular apps, covering complete user workflows from trigger feedback. Each task quires 15-27 actions (mean = 19.5) and incorporates both abstraction levels: High-level instructions define global goals (e.g., 'Schedule 1.5-hour meeting starting 10 am on June 29th'). Low- level instructions specify atomic operations Launch Tencent Meeting; Click 'Schedule': Select 'Standard Meeting': Set duration). All steps are annotated with granular semantics, emphasizing complex interface behaviours (e.g., multi-level dropdown navigation, real-time input validation) to validate long-horizon GUI agents challenging workflows.
2.2 MULTIMODAL ENHANCED PERCEIVER
Accurately identifying and disambiguating interactive elements in context is key to enabling task automation in complex GUIs. To this end, we propose the Multimodal Enhanced Perceiver (MEP), which unifies icon detection, OCR recognition, and heuristic repair into an ID-centred abstraction layer, extracting actionable signals from evolving GUIs, inspired by prior work (Lu 20246).
Specifically, given a GUI screenshot S, MEP extracts visual elements through parallel perception modules: (i) An enhanced detector identifies interactive controls, producing E.i (id,, b,, c with id, a unique spatial tag. b; its bounding box, and c; the confidence from the detector head (sigmoid class probability). IDs serve stable anchors, robust small layout variations. MEP also highlights previously clicked elements. (ii) conventional OCR module extracts Exst (ty, b with t, the detected text and by the bounding box. To disambiguate composite controls such as icon + text, each E Emi is linked with its most relevant text via semantic binding function:
(idi, by Ubjr, lyr: ci), if IoU(b,, ZT,
2 Øle,, Frex))
(idd, by, 0, ci),
otherwise,
where j = arg max; loU(b, b,) denotes the text box with maximum overlap, and binding is ap- plied only when (b; by-) (Appendix 12.3).
4

---

Published as a conference paper at ICLR 2026
To mitigate missed detections of critical elements, such as close buttons on pop-ups, employ fallback template matcher that is activated when no elements are detected designated high tigh-priority areas Apririty (small normalized bands around pop-up corners and bottom bars where missing control can stall a trajectory). Upon activation, the module invokes a repair function R. over Apririty. leveraging template library T of canonical close/cancel, confirm/next, and back/home icons; high- similarity matches are inserted as new elements only in these regions to match and restore omitted key elements.
2.3 DEEP REFLECTION DECIDER
Current agent decision mechanisms (Nm et al 2024: Ki cf al
(IZE)) based on self-supervised
training paradigms exhibit limited long-horizon generalization due to constrained dataset diversity, while MLLMs-based mechanisms (Wang all, R024D), despite superior sequence modeling pabilities, suffer from cascading error propagation under dynamic interface shifts, compromising reliability in long-horizon task execution. To address this, propose the Deep Reflection Decider, as illustrated in Figure Al which implements a structured multi-level feedback mechanism to estab- lish triple closed-loop reasoning. This strategy validates goal rationality pre-execution and confirms environmental consistency post-execution, ensuring action precision and prediction credibility.
Specifically,
strictly defined JSON Schema (fields: historical_statur think, Execute_goo action, further details are provided in Ap- pendix E enforces structured three-tier reason- ing, where the first three fields implement reflection and the last two fields implement decision:
prompt: the 'Use' preset by
Excluction prev gooo
Baxc
imag, eoate shexty adeevedd tatmske gast
Import contents
costruct on
Provlde stup-ty thiaking presus, Aralys serves, reate overdt task entiveer prrvioas wap. Des next single
Think
(1) Historical Validation: the historical status validates UI state transitions (e.g., button activation, text input) via OCR/icon detection, establishing spatiotemporal verification loops. Failure flags trigger root-cause analysis upon detecting error dialogues or unresponsive elements.
Next_gooi
siift spoiific, inamediat
propestas
Action
dic coniorrrry partorictirs sctient
(2) Target Check: the import_contents contents field extracts screen-critical information, validating the MLLM's environmental comprehension via OCR/i- con detection. Task-goal consistency assessments retain high-relevance text while filtering noise.
Figure 4: Deep reflection and decision- making processes designed to validate prior actions and predict subsequent steps.
(3) Action Explainability: the think field requires the MLLM to sequentially analyze current UI states, failure causes (if any), and action localization rationale (c.g., "Button #12 has the highest interaction confidence"), with outputs culminating in executable goals (Execute_goal) that are translated into atomic actions (action).
Pre-execution Reflection. Before execution, each candidate action is screened for on-screen grounding and task entailment. We accept a for execution only if
H(s, a G. T) 1 ((ha(a) 9.] 1[K (duction) C K(T) =1
Here, G. denotes the UI elements at state st from the perceiver, T the global task description, and a a candidate action with (Execute_goal act ion) and description daction- Irg(a) is the target element of a, and (--) keyword extractor enforces that the action semantics are consistent with the task. In practice, if either the target element is absent from the current screen or the action semantics are not entailed by the task description, the action is rejected and brief revision step is triggered using available OCR/icon evidence; otherwise, the action proceeds.
2.4 COMPENSATING ACTION EXECUTOR
Current HILLM-driven agents face actioninstruction uncertainty: free-formal outputs lack a direct mapping executable screen coordinates, while dynamic UIs require real-time correction. To bridge
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Compensating Action Executor (single step)
Require:
Current state candidates A from DEEP REFLECTION DECIDER encoded as
index (posit ion: "?"), : relative (action "n, (ær v)
absolut (point: ((x,y) + + last committed snapshot (sc-1-Pt-))
Ensure: Exccuted (a", enc"", pp*) with
(SUCCESS, FAIL) (rollback on fail)
1: I1 f [index (position) "i"), relative (action: "n, (z, y)"
absolute (point: "(x,y) + +
> priority: index + relative - absolute
2: for each enc do
3: 3f E s.t. Encode(a) = enc then
4:
p RESOLVEPOINT(O, enc, 8t) centroid element-local map / screen map jitter
5:
EXECUTECLICK(P)
6:
($t+1.8) (LL, a,
7:
if = SUCCESS then
8:
return (a, enc, p. SUSCESS)
caller updates snapshot to ($t+1.P)
9:
else continue
10:
> degrade to next encoding
11: end if
12: end
13:
end for
14: RECORDFAILURE(S;, A); ROLLBACK(s; (1,11,
return FAIL)
this semanticphysical gap, we introduce the Compensating Action Executor (CAE), which adopt robust action pipeline with multi-stage compensation and progress-triggered triggered backtracking (see Algorithm I.
Compensating Action Execution. We first parse element indices (e.g., position: and se- mantic descriptions (e.g., Top Moments button) from the Deciders output, then resolve the target elements bounding box from the live layout, denoted B = (Imin, Umin, Tmax: (/max)... Normal- ized coordinates (Inorm Ynorm) are mapped physical pixels using device-aware scaling matrix sereen: Hscreen), ) with S (Znoem: Voorn) so that the same normalized command is
S = diag( WWsereen: mapped consistently to device-specific click locations across different resolutions.
To enhance operational robustness, we employ three-stage degradation policy consis- tent with our encodings index (position: "i"), relative (action: "n, (z. ")), and absolute (point: ((z,v) - =
(1) Index (centroid). Prioritize index-based execution at the element centroid Po of B; i.e., the midpoints of the intervals [Fminn and [yauin ymnax]
(2) Relative (in-box). If the attempt fails (5 0), draw click Pret uniformly inside B: sample Am, An (0, 1] and place the point using the box width w Xmax Tmin and height
Ymax Vmiin
(3) Absolute (screen) with jitter. Upon repeated failure, use absolute screen coordinates mapped from (x. y) and add bounded perturbation e (e.g., || I 5 px) to escape edge/occlusion cases; the base point defaults to the normalized target or Po when unspecified.
Post-execution Reflection. For each action instruction a at state in the stated priority order. After each attempt, the DEEP REFLECTION DECIDER performs state verification:
execute its candidates
it = Verify a, Pt, I++1) E {0,
(3)
where Pt is the click point computed from the current attempt and the resolved box B, and 1 the post-action screenshot. If Uz 1, we commit the step and update the snapshot to (s++1.P)) Pt). Otherwise, we degrade to the next candidate. When all candidates for a are rejected, we allow a few local re planning calls to DRD at the same state; if these still fail, we invoke Rollback( (1 Pi-1) to restore the last committed snapshot and continue execution. (see Appendix D for details and statistics).

---

Published as a conference paper at ICLR 2026
Table 1: Performance Comparison of Models on LongGUlBench Long-Horizon Tasks
Model Name
General-Low
General-High
Game Low
Game
High
Avg
TM
SR
TM
SR
TM
TM
Avg
Base Models
GPT-4o (OpenAl et all 2024)
87.5
20.8
75.0
4.2
91.6
23.9
85.9
3.7
49.1
Gemini2.5 (Comanici all 2025)
96.7
73.3
77.2
25.7
95.1
57.7
84.3
25.7
67.3
Qwen2.5-VL-7b (Bar all 2025)
82.7
73.1
29.3
92.4
72.8
68.9
27.4
67.4
GUI Models
OmniParser (Lueetal (IRA5)
90.0
83.0
79.3
35.6
91.8
61.0
70.4
20.1
66.4
AgenCCPM- (Zhang al 20256)
92.1
81.2
82.4
37.1
89.7
66.5
74.1
25.8
68.6
InfiGUI-RI (Luuctal all MIXE)
93.2
79.7
56.7
23.8
92.9
67.2
53.9
19.4
61,8
UI-TARS- 1.5 (Qin et all 20%
93.6
79.2
75,4
21.8
88.2
69.5
77.8
18.9
65.8
LongHorizonUI
93.5
85.3
78.0
52.3
93.8
83.9
79.7
52.1
77.3
Table 2: Grounding Performance Comparison on the ScreenSpot Benchmark.
Model Name
Mobile
Desktop
Web
Avg
Text
Icon
Text
Icon
Text
Icon
Base Models
GPT-4o (OpenAl (1223
30.5
23.2
20.6
19.4
11.1
7.8
18.8
Gemini2.0
84.0
Qwen2.55VL--T VL-7bb (Ba el ala 2023)
84.7
GUI Models
CogAgent (Hong et al. (024))
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SeeClick (Cheng et all
78.0
52.0
72.5
55.7
32.5
53.4
ShowUI (an (T25)
92.3
75.5
76.3
61.1
81.7
63.6
75.1
OmniParser ot 20221
93.9
57.0
91.3
63.6
81.3
51.0
75.1
UI-TARS (Qin et all 2025)
93.0
75.5
90.7
68.6
84.3
74.8
82.3
InfiGUI-RI (Li ctal (126
97.1
81.2
94.3
77.1
91.7
77.6
87.5
LongHorizonUI
95.6
86.9
96.8
81.4
93.5
90.9
90.4
3
EXPERIMENTS
IMPLEMENTATION DETAILS
To ensure fair evaluation across benchmarks, we select base models aligned with their architectures and configure consistent experimental settings. For LongHorizonBench, we adopt representative MLLMs (Comanici atall 2025) as the backbone to ensure stable reasoning in long-horizon tasks. All components operate without fine-tuning, leveraging pre-trained models for task execution and evaluation. Task-specific prompts are designed prevent ambiguity and enhance reproducibility (see Appendix D.I for more details.).
3.2 BENCHMARKS
We evaluate our model using the following benchmarks (i) LongGUInBench, our curated dataset of 371 complex GUI task trajectories spanning 28 diverse applications (including gaming, enter- prise systems, and creative tools) with an average trajectory length of 24.6 steps (max 37 steps) for evaluating long-horizon reasoning robustness; (ii) Screenspot for granular grounding capability as- sessment across multiple device types; and (iii) AndroidControl (Low/High difficulty tiers) (Lretall 20228) and GUI-Odyssey (Lu et all 20244) datasets to measure real- time navigation performance under dynamic interface constraints.

---

Published as a conference paper at ICLR 2026
3.3 MAIN RESULTS
Long-horizon Reasoning Capability. To systematically evaluate long-horizon reasoning capabil- ities, we conducted extensive experiments comparing LongHorizonUI with state-of-the-art meth- ods our proposed LongGUTBench benchmark, which features long-horizon tasks. As shown in Table M. LongHorizonUI achieves a step success rate (SR) of 85.3% for low-level instructions and 52.3% for high-level instructions in general scenarios, which represents improvements of 6. 6.1% and 30.5% over the state-of-the-ar method (UI-TARS-1.5) respectively, and significantly outper- forms all open-source models and GUI-specific training methods. more complex game scenarios, LongHorizonUI reaches a low-level instruction SR of 83.9% and high-level instruction SR of 52. %, which maintains a clear lead across all compared methods. These results validate the pro- posed LongHorizonUT's significant advantage in modeling long-horizon dependencies.
Grounding Capability. Table @ compares our LongHorizonUl framework with mainstream meth- ods on the ScreenSpot dataset, including base models and SOTA GUI agents. LongHorizon demonstrates consistent superiority across device subsets (mobile. desktop, web), achieving 90,4% average task success rate, surpassing all open-source models and outperforming the previous SOTA GUT framework (UI-TARS) by 2.9%. These results validate LongHorizonnUJs robust grounding ca- pability across diverse devices and scenarios.
Navigation Capability. To rigorously evaluate the navigation capabilities of our method, benchmarked LongHorizonUI against state-of-the-art approaches on AndroidControl (La et all 20Z4 and GUI-Odyssey (Lu all 2024B). As shown Table & LongHorizonUI achieves sig- nificant improvements in SR over both zero-shor models and GUI-specialized baselines. Compared Qwen2.5-VL-7B our method elevates SR by 6.4% on AndroidConttol-Hign and 6.1% on GUI- Odyssey. Moreover, LongHorizonU attains average SR gain of 2.3% over the strong GUI-R1-7B baseline. These results demonstrate that LongHorizoni not only significantly enhances planning robustness for long-horing tasks but also retains fundamental interaction capabilities for short quences.
Table 3: Performance comparison AndroidControl and GUI-Odyssey benchmarks
Model Type
Model Name
AndroidConttool..n
Android AndroidConttro-HHig
GUI-Odyssey
Avg
TM
SR
TM
SR
TM
SR
GPT--0
74.3
28.4
63.1
21.2
37.5
5,4
38.3
Base Models
Qwen2.5-VL-3B
62.0
59.3
47.8
38.9
37.4
26.7
45.4
Dwe72.5-VL-7B
83.4
62.5
68.7
47.1
55.6
34.4
58.6
OS-Atlas-4B
64.6
40.6
49.0
22.8
49.6
20.3
41.1
Os-Atlas-7B
73.0
50.9
57.4
29.8
27.0
49.8
GUT model
GUI-RI-3B
83.7
64.4
58.0
46.6
54.8
41.3
58.1
GUI-R 1-7B
85.2
66.5
71.6
51.7
65.5
38.8
63.2
Ours
LongHorizonUU
87.5
68.9
73.4
54.2
68.3
40.5
65.5
3.4 ABLATION STUDY
Effectiveness of Perception Components. Figure 5t reports an ablation study that isolates each perception module. Jointly using the refined icon detector and the OCR recognizer yields the highest accuracy and robustness. Removing the icon detector cuts fine-grained recognition, lowering the step-completion rate by 6.1%. Disabling OCR causes the same 2.3% drop and leads to frequent errors on icon-text composite widgets. Turning off the adaptive grid prevents the detector from scaling different screen resolutions, so microscopic elements on high-resolution displays are often missed. Together, these three modules supply the rich visual context required for reliable long- horizon modeling.
Effectiveness of Compensatory Actions. Figure 5H visually compares the different action modes, indexing instructions and step lengths, showing that indexing alone delivers an 81.4% task- completion rate, outperforming all other action modes. Adding compensatory actions top of
8

---

Published as a conference paper at ICLR 2026
Indkiing perfommacc
Compersation
85.00%
65.0
81.0
(16
Compa Companooae gain
$3..5
00.0-
77.5
71
74.37%
15.0
750
17.
715
Index ng
Abodlue Compensatory action
néadive
Nitoicall
730
(a) Perception components.
(b) Compensating actions.
Figure 5: Ablation analyses: (a) perception components; (b) compensating actions.
The user opens the "Henor of Kings" application, selects the "Animist" hero in Solo Practice mode, changes skin to Duke Lepus," and switches the skill to "Sprint," then closes the background apps.
Step2: "clickcs positionn 14)
Shep7 "click: "click"-"pessttoo"" 15)
Steck:"click: "clpcc":"""""s""iiiiii ("posiHón" 19)
"click": C' pppastiinn 40)
Stap6: positiion
Shepp pppcsition
Stepd: ('position:: 39)
Step24: "click" (ppssition 58)
Shep4: "click: ( posiiiion" :: 21)
Shep5: "click" ("position": 26)
...SStp23:"pper.. "rress key ("recent"
opp
Solo Practice mode
Changes hero's skin
Switches the skill
Closing
Figure 6: Case visualization of LongHorizonUI in gaming scenario.
indexing gives further gains by 1.2% (relative coordinates), 2.5% (absolute coordinates), and 3.9% (historical coordinates). These results confirm that compensatory actions complement indexing: by fusing historical spatial cues with fault-tolerant coordinate transforms, the executor remains robust even under dynamic interface disturbances.
3.5 CASE VISUALIZATION
As illustrated in Figure R. the LongHorizonUl agent executes a fully automated operation sequence in the Honor Kings scenario. Guided by indexing instructions, the agent achieves pixel-precise grounding on all target UI elements, including minuscule widgets (Step 3) and low-contrast compo- nents (Step 5). Notably, when confronted with a sudden pop-up interruption during Step 2, the agent dynamically detects and disables the interference source through its real-time perceptual module, subsequently resuming task execution without workflow disruption. This end-to-end workflow spans multi-step operations from application launch, skill switching. background process management, demonstrating LongHorizonUIs capability to maintain cross-step operational precision and dynamic disturbance robustness in complex task chains.
CONCLUSION
Summary. In this work, we present LongHorizonUl,, an innovative framework for long-horizon GUT tasks, featuring multimodal enhanced perceptron for precise capture of UI element states,

---

Published as a conference paper at ICLR 2026
three-tier closed-loop reasoning engine for action verification//recditiin,, and innovative multi- level compensator ensuring action execution validity. Demonstrating superior performance on Long- GUIBench (15-step tasks) and public benchmarks, it establishes paradigm for reliable long- horizon GUT tasks.
Limitations & Future Work. Despite achieving state-of-the-art performance without introduc- ing notable overhead relative to prior agents, LongHorizonUJI still inherits the latency of MLLM- dependent pipelines. Next, we will focus on model-level efficiencydditillatiin, quantization, and context-aware prompt compression.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, and Zeyu Cui et al. Qwen technical report, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqian Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502.13925 2025.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents, 2025.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024.
Gheorghe Comanici. Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3695-3706, 2022.
Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12): 7463-7474, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding novel environments via autonomous exploration, 2025.
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-fincte finetuned foundation models, 2024.
Wenyi Hong Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14281-14290, June 2024.
Junchao Huang, Xinting Hu, Boyao Han, Shaoshuai Shi, Zhuotao Tian, Tianyu He, and Li Jiang. Memory forcing: Spatio-temporal memory for consistent scene generation on minecraft. arXiv preprint arXiv:2510.031993 2025a.
Junchao Huang, Xinting Hu, Shaoshuai Shi, Zhuotao Tian, and Li Jiang. Edit360: 2d image edits to 3d assets from any angle. In ICCV, 2025b.
10

---

Published as a conference paper at ICLR 2026
Zhiyuan Huang, Ziming Cheng, Junting Pan, Zhaohui Hou, and Mingjie Zhan. Spiritsight agent: Ad- vanced gui agent with one look. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 11490-225000 June 2025c.
Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Guided point contrastive learning for semi-supervised point cloud semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6423-6432, 2021.
Bin Kang, Bin Chen, Junjie Wang, Yulin Li, Junzhi Zhao, Junle Wang, and Zhuotao Tian. Calibelip: Contextual calibration of dominant semantics for text-driven image retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 5140-5149 149,,111900009 2025.
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lan Chao. Dual-view visual contextualization for web navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14445-14454, June 2024.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 39648-39677, 2023.
Hanyu Lai, Xiao Liu, lat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 52955306. 2024a.
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi- supervised semantic segmentation with directional antext-aware consistency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1205-1214, 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024c.
Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, and Insik Shin. Explore, select, derive, and recall: Augmenting IIm with human-like memory for mobile task automation, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen vision encoders and large language models. In Proceedings of the Inter- national Conference Machine Learning (ICML), pp. 19730-19742. 2023.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala Divya Tyamagundlu, and Oriana Riva. On the effects data scale on computer control agents. arXiv preprint arXiv:2406.0367 2024a.
Yanda Li. Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions, 2024b.
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang. Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey large multimodal reasoning models. arXiv preprint arXiv:2505. 04921, 2025.
Bin Lin, Zhiyuan Ye, Shuyang Zhang, Jun He, and Dong Yu. Moe-llava: Mixture of experts for large vision languaee models. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition (CVPR), pp. 13245-13255, June 2024.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One aaooo--lnguage-acee model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 19498-19508. June 2025.
11

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Visual instruction tuning with large language models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5189- 5210, December 2023.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang. and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners, 2025.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li. Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey; A comprehensive dataset for cross-app gui navigation on mobile devices, 2024a.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024b.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based GUI agent, 2024c.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl A generalist r1-style vision-language action model for gui agents, 2025.
Xiaoliu Luo, Zhuotao Tian, Taiping Zhang, Bei Yu, Yuan Yan Tang, and Jiaya Jia. Pfenet++: Boost- ing few-shot semantic segmentation with the noise-filtered context-aware prior mask. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 46(2); 2023.
Zhenhua Ning, Zhuotao Tian, Guangming Lu, and Wenjie Pei. Boosting few-shot 3d point cloud segmentation via query-guided enhancement. In Proceedings of the 31stACM international con- ference multimedia, pp. 1895-19044 1904, 2023.
Runliang Niu, Jindong Li, Shiqi Wang. Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. In Proceed- ings of the Thiry-ThirdInternatinnd Joint Conference on Artificial Intelligence, pp. 64336441. International Joint Conferences Artificial Intelligence Organization, August 2024.
OpenAl, Josh Achiam, Steven Adler, and Sandhini Agarwal al. Gpt-4 technical report, 2024.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23641-23651 2023.
Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, and Jiaya Jia. Scalable language model with generalized continual learning. arXiv preprint (iXx:24444007477 2024a.
Bohao Peng, Xiaoyang Wu, Li Jiang, Yukang Chen, Hengshuang Zhao, Zhuotao Tian, and Jiaya
Jia. Oa-enns: Omni-adaptive sparse cnns for 3d semantic segmentation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21305-21315, 2024b.
Shangpin Peng, Senqiao Yang. Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level early intervention. arXiv preprint arXiv:2507.12455, 2025.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024.
Yujia Qin, Yining Ye. Junjie Fang, Haoming Wang, and Shihao Liang et al. Ui-tars: Pioneering automated gui interaction with native agents, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances Neural Infor- mation Processing Systems, volume 36, pp. 99708-59728, 2023.
12

---

Published as a conference paper at ICLR 2026
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training- free open vocabulary semantic segmentation. In European Conference on Computer Vision, pp- 139-156, 2024.
Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu. Mobilegui-rk. Advancing mobile gui agent through reinforcement learning in online environment, 2025.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis, 2025a.
Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with exploration. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 19477- 19486, June 2025b.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang. Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong. Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chao- jie Wang, Xinrun Wang, Börje F. Karlsson, An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control, 2024.
Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. Learning shape-aware embedding for scene text detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4234 4243, 2019.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050-10)0 1065, 2020.
Zhuotao Tian, Pengguang Chen, Xin Lai, Li Jiang, Shu Liu, Hengshuang Zhao, Bei Yu, Ming- Chang Yang, and Jiaya Jia. Adaptive perspective distillation for semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):1372-1387, 2022a.
Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. General- ized few-shot semantic segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11563-1153 1572, 2022b.
Zhuotao Tian, Jiequan Cui, Li Jiang. Xiaojuan Qi, Xin Lai, Yixin Chen, Shu Liu, and Jiaya Jia. Learning context-aware classifier for semantic segmentation. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 37, pp. 1438-2446, 2023.
Chengyao Wang, Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng. Hengshuang Zhao, and Jiaya Jia. Groupcontrast: Semantic-aware self-supervised representation learning for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1917-4923 2024a
Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. Declip: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp. 14824-14834, 2025a.
Junjie Wang, Keyu Chen, Yulin Li, Bin Chen, Hengshuang Zhao, Xiaojuan Qi, and Zhuotao Tian. Generalized decoupled learning for enhancing arXiv:2508. / 1256, 2025b.
dense perception. arXiv preprint
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. aobile-agent-22 Mobile device operation assistant with effective navigation via multi-agent collaboration. In Advances in Neural Information Processing Systems, volume 37, pp. 2686-2710, 2024b.
13

---

Published as a conference paper at ICLR 2026
Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang. Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. Gui agents with foundation models: A comprehensive survey, 2025c.
Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng. Xihui Liu, Kaicheng Yu, and Hengshuang Zhao. Towards large-scale 3d representation learning with multi-dataset point prompt training. In CVPR, 2024a.
Zhiyong Wu, Zhenyu Wu, and Fangzhi Xu al. Os-atlas: A foundation action model for generalist gui agents, 2024b.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, and Yian et al. Os-atlas: A foundation action model for generalist gui agents, 2024c.
Senqiao Yang, Tianyuan Qu, Xin Lai. Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.172400 2023.
Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adaptation. Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 23407-23415, 2024.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025.
Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. Mobile-agent-v3: Fundamental agents for gui automation, 2025.
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, al. Enhancing visual grounding for gui agents via self- evolutionary reinforcement learning. arXiv preprint arXiv:250S.12377 2025.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46()):5625-5644, 2024.
Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, Zhuotao Tian, Naiyan Wang, and Heng- shuang Zhao. Concerto: Joint 2d-3d self-supervised learning emerges spatial representations. In NeurIPS, 2025a.
Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, and Yesai Wu al. Agentepm-gui: Building mobile-use agents with reinforcement fine-tuning, 2025b.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
This is the supplementary file for our submission titled LongHorizonUUI: A Unified Framework for Robust long-horizon Task Automation for GUI Agent. This material supplements the main paper with the following content:
Motivation of LongHorizonUl
Related work
Additional Experiments
(I.I) Implementation Detail
Benchmarks
Parameter analysis
Prompts in Automated Pipeline
Output Format Structure Template
Visual Processing Template
Action Selection Protocol
Workflow Exception Handling
Qualitative Analysis
Additional Discussions
A THE USE OF LARGE LANGUAGE MODELS
In this work, large language models (LLMs) are used exclusively for polishing the writing and checking grammar. They are not involved in research ideation, experimental design, data analysis, or the formulation of conclusions. The authors make substantive intellectual contributions.
B MOTIVATION OF LONGHORIZONUI
To systematically assess the performance of state-of-the-aa UI agents on long-horizon interaction tasks, design step-length-drivee driven, multi-factor evaluation protocol that highlights the need for robustness at scale. We first compute the step-length distribution of the ANDROIDCONTROL test set (Figure Za)) and observe that more than 80% of the episodes contain fewer than ten actions, whereas sequences of ten or more steps, those that truly stress long-horizon reasoning, account for less than 20%. This imbalance suggests that average-case metries allow agents mask failures on long chains, motivating a dedicated benchmark for long- aor-horizon evaluation. We then simulate the execution success rate (ESR) as function of step length for five representative agents under the same distribution (Figure (b). UI-TARS-2B (Qin et al, (2025), InfiGUI-R--BB (Liu et all 2025), Qwen2.5-VL-7BB (Bai all 2025). and AgentCPM (Zhang et al, 20251) exhibit cliff-like drop after the ten-step threshold (ESR 50-70%). whereas LONGHORIZONUU remains nearly flat and sustains roughly 75% ESR between 16 and 24 steps. These results confirm that conventional agents accumulate uncorrected emrrrs on long chains, while the multimodal perception, reflective planning, and compensatory execution modules in LONGHORIZONUI markedly curb performance degradation. Finally, aggregating the mean ESR for sequences of ten more steps (Figure Ic)) shows that LONGHORIZONÜT achieves 73.8%, outperforming the strongest baseline, AGENTCPM, by approximately five percentage points, further substantiating its long-horizon robustness.
C RELATED WORK
Multimodal Large Language Models. In recent years, Multimodal Large Language Models (MLLMs) have emerged pivotal research focus in artificial intelligence due their capac- ity for unified model reasoning. Built upon conventional Large Language Models (LLMs), MLLMS incorporate vision encoders (e.g., ViT (Dosoviskkiy et all 2021), CLIP (Radford et all
15

---

Published as a conference paper at ICLR 2026
(a) Performance Degradation Analysis by Step Length
70
ESR (%)
LongHorizonUl Lead Over SO
50
Models
Long Horizon Threshold
UI-TARS-2B
InfiGUI-R1-38
Qwwnn2.5VVL-B
AgentCPM
LongHorizonUl
15
20
Step Length
(b) Test Set Step Distribution
(c)
Average ESR in Long Sequences (Steps 210
60
200
Sample Count
70
74.0%
ESR (%)
65
69.0%
67.9%
100
65.5%
62.9%
55
50
1011
LongHortzonUl
AganCPM
InfiGUI-R1388
UI-TARS-28
Owen2.5-VL-TB
Step Length
Figure 7: Further Analysis of Motivation. (a) Step-length distribution of AndroidControl test episodes; (b) Execution-success rate (ESR) vs. step length for UI agents; Mean ESR compar- ison for long sequences (10 steps). LongHorizonUI demonstrates sustained performance robustness on extended interactions.
2021)) to process image data, enabling cross-modal comprehension from static images to video se- quences. This architectural paradigm facilitates perffommreee systems such Qwen- VL (Bai etall 2025). GPT-4V et all 2024), and BLIP-2 (Ll 2025), which exhibit robust inter- active understanding dynamic multimodal environments. Recent efforts further extend MLLMs
to reasoning-intensive visual tasks such reasoning segmentation (Lai et all, 20246; Yang et al)
2023) and vocabulaay dense perception (Wang et all 2025::: Shao et all 2024 Kang et all, 2025), demonstrating the potential of language-guie -uuudddeggdddd visual reasoning. Meanwhile, token-efficient architectures like VisionZip (Yang et all, 2025) and unified domain adaptation methods (Yang et al), 2022)) advance the practical deployment of vision-language models. However, current models still lack fine-grained perception and can hallucinate (Peng et al, 2025), often yielding erroneous state predictions that constrain deployment in GUI agents and broader applications.
GUI Agent. Current research GUI agents primarily focuses input modalities and learning paradigms. Regarding input modalities, early LLM-based agents aLe 20221 Amiricial.. all 2024
Lai et all 2022R) typically relied GUI parsers to convert interfaces into text-based representa- tions via HTML parsing or screenshots. This approach lacked visual granularity. resulting in limited generalisation capabilities. The emergence of MILLMs (Wang all 20246 Kim all 2023
16

---

Published as a conference paper at ICLR 2026
Table 4: Grounding performance on ScreenSpotV2.
Model Name
Mobile Text Icon/Widget Text Icon/Widget Text Icon/ Icon/Widget
Desktop
Web
Avg
Base Models
GPT-4o (OpenAT 2022)
49.6
14.2
26.9
40.1
18.7
56.8
43.5
Gemini-2.5-PP (Comaniei stall (1225
63.5
42.1
70.8
49.3
81.7
84.2
68.3
Qwen2.5-VL. (Bar et 2025)
66.8
92.1
46.8
72.6
44.3
83.0
70.4
GUI Models
SceClick (Cheng (024)
78.4
50.7
70.1
29.3
55.2
32.5
55.1
OS-Atlas-4B
87.2
59.7
72.7
46.4
85.9
63.1
71.9
OS-Atlas-77 cl.. 20246
95.1
75.8
90.7
63.6
90.6
77.3
84.1
UI-TARS-7B
95.2
79.1
90.7
68.6
87.2
78.3
84.7
LongHorizonUl (ours)
94.5
80.6
94.3
72.9
91.5
83.3
86.2
ables agents to process visual inputs directly, achieving more intuitive interface comprehension. In learning paradigms, Supervised Fine-Tuning (Furuta al. 2024: Liet (0246) optimises models with domain-spectioo
data to enhance task-specific performance, yet requires costly annotations and struggles with generalisation to novel scenarios. Conversely, reinforcement learning (RL) (Shi all, 2025; Luo aLa 2025; Yuan ei all 2025) and step-wise preference optimization (Cai 20244)
improve decision efficiency through autonomous exploration, but face bottlenecks training sta- bility and reward function design. Additionally, scalable continual learning strategies (Peng all 20225) have been explored to maintain model capabilities across evolving task distributions. While these methods perform well in short-horizon tasks, current architectures struggle to maintain intent consistency across steps and lack precise historical state backtracking. Consequently. their reasoning capabilities remain confined to short-term tasks, making long-horizon task planning and execution a critical challenge.
D ADDITIONAL EXPERIMENTS
D.1 IMPLEMENTATION DETAILS
We adopt Googles Gemini-2.5 Pro as our core reasoning backbone due its advanced reasoning capabilities and high performance on complex reasoning tasks. The model is accessed via Google Vertex AI API with deterministic inference and maximum output length of 2048 tokens to en- sure reproducibility. All experiments run cluster of eight Tesla V100 GPUs under Ubuntu 20.04 LTS, using PyTorch 2.1 and CUDA 11.6; model serving managed by Ray Serve for scal- able, high-throughput inference. Prompt templates strictly follow JSON schema fields include historicall sitooricll__ without additional fine-tuning.
laciilaaaaaatllllllllllalllaaaaa stat us, think, and
structured multi-level reasoning
D.2 BENCHMARKS
Crounding-Centiinn Benchmarks: ScreenSpot Series. Accurate element localization is the foun- dation of GUI automation. ScreenSpot is a cross-platfonn grounding benchmark with over 1,200 Lanural- instructions spanning iOS, Android, macOS, Windows, and Web interfaces. Each instruction is paired with pixel-level bounding boxes and element-type labels (text, icon, or widget) and covers challenging scenarios such as icon-text composites and occluded controls. ScreenSpot-v2 (Wi ct all Mиzzh) further enhances robustness by adding 564 procedurally generated taskscreated via the JEDI synthetic pipeline with 4 million samplesto layout generalization across platforms.
Navigation-Cente Centric Benchmarks: AndroidControl & GUI Odyssey. Once elements can be liably located, agents must navigate within and across apps. AndroidControl (Lret ((225). the largest public mobile navigation corpus, contains 15,283 human demonstrations divided into low- difficulty single-app workflows (< 10 steps) and high-difficulyy cross-app tasks with real-time inter- ruptions (e.g., Select photo from Gallery Upload via Email). It evaluates agents comprehension of
17

---

Published as a conference paper at ICLR 2026
both high-level goals (Book a ride) and low-level operations (Tap Search). GUI Odyssey (Lu all 2022)) extends this to long-howe Long-horizon, cross-app navigation with 7,735 mission based episodes across 201 apps and 1,400+ app combinations. It injects dead-end paths to test backtracking and measures temporal efficiency through metrics like average path length and decision latency.
Long-Horizon Task Benchmark: LongGU/Bench. The ultimate test of GUI agent is executing extended multi-step workflows end to end. LongGUIBench comprises 371 complex task trajectories across 28 applications, split into 224 high-complexity game scenarios (1937 steps, mean=23.7) and 147 general productivity scenarios (1527 steps, mean=19.5), totaling 4,508 screenshots. Every task includes dual-level annotationsHHigh-Leww (e.g., Click the Store button; Select Buy)alongside control type, bounding box, and state metadata. A 42% layout-shift rate enables rigorous testing of historical-state verification and error-recovery mechanisms.
goals (e.g.. Purchase item XX) and Low-Level actions
D.3 PARAMETER ANALYSIS
Further Grounding Analysis. The extended evaluation on ScreenSpot-V2 (Table B) confirms our framework's 's robust grounding capabilities. where LongHorizonUl achieves competitive perfor- mance (86.2% avg) despite specialized UI-TARS models showing advantages in isolated recogni- tion. This apparent discrepancy stems from UI-TARS's specialization in static vision features while LongHorizonUI prioritizes dynamic actionability essential for downstream workflows. Crucially, our Multimodal Enhanced Perceiver's loU-based element fusion resolves 92% of mobile occlusion cases that degrade competitors (e.g.. 20.5% improvement over OS-Atlls-7he in low-contrast scenar- ios). Though UI-TARS-7B leads in desktop icon recognition (87.9% vs ours 72.9%), our unified representation reduces cross-device variance just 8.3% versus their 14.7%, validating our ap- proach's suitability for practical long-horizon operations where contextual adaptability outweighs pixel-level precision.
83.0
1338
834
88.2
0288
527
Success Rate (96)
Peak-83.99 83.9%
81
80
79
LongHorizenUi (SR)
78
0.5
0.6
0.7
0.8
0.9
IoU Threshold
Figure 8: loU Threshold Analysis for icon Elements.
Threshold-Sweep Experiment. To quantify how the detectors locality constraint influences down- stream control, we perform an experiment in which the IoU criterion for merging OCR text and icon boxes is varied from 0.6 to 0.9 (Figure R). When the threshold is too loose (0.6), false-positive matches increase, yielding an overall task-success rate (SR) of 82.7%. Tightening the requirement to 0.7 suppresses spurious pairs and raises SR to 83.5%. The best performance is obtained at IoU = 0.8, where LongHorizonUI reaches its peak SR of 83.9%. Pushing the threshold further to 0.9, however, makes the detector overly selective; missed matches propagate to action planning and drive SR back down to 83.1% %. These results confirm that an IoU of 0.8 provides the best balance between recognition precision and recall, and thus maximizes end-to-end success
18

---

Published as a conference paper at ICLR 2026
Output Format Structure Template: Defines the Mandated JSON Structure for Agent Output.
Historical status": "Success Failed Unknown Evaluate the previous action visually achieved its intended goal. Base this ONLY the screen image. Ignore the execution result status provided in the input.",
"Import_ Iimport_contents" "Output important contents closely related to userY's instruction on the current page. If there is, please output the contents. If not, please output empty string ".",
"Think": "Provide step-by-step thinking process. Analyze the current screen, relate the overall task the visual outcome the previous step ('evaluation Seppilatttoo_evee____ "ov_gl"""). Decide the next best "single* action. Explain your reasoning clearly, including why you chose the specific action and target (index coordinates). "evalluatonn _erooogo_eo_oollleee_ 'Failed', reflect on why and how the action addresses it.",
"Next_gooe goal": "Briefly describe the specific, immediate goal of the *next action" you are proposing in the 'action' field.",
"Action": ("action_name". //* dictionary of parameters the action */}}
Structured Agent Response Schema. Mandates a five-field JSON output format enforcing visual goal verification (Historical status), content extraction reasoning (Think), next-goal declaration, and parameterized actions.
Figure 9:
chain-of- -lain-of-thought
Action Command
Action Prompt: Tap the "Use" button
Deep Think & Descion
"Historical status": "Success",
"Import_content""" "The 'Recommended Preset' screen is now open, showing a list of presets.", "Think": : "I have successfully opened the 'Recommended Preset' screen. The next step is to tap the "Use' button for the preset by CIKE14. can see this preset in the list, and the 'Use' button highlighted with index 36. will click on it to apply this preset.",
contents":
"Next goal": "Tap the 'Use' button for the CIKE14
preset.",
"Action": ("click": ("position": 36)}
Figure 10: Structured Action Execution Example. Demonstrates agent output conforming to the five-field JSON schema: verifying historical success (CIKE14 preset), extracting relevant content, reasoning through actions, declaring next goal (tap 'Use'), ), and parameterizing the click command (position 36).
D.4 ROLLBACK FREQUENCY AND EFFICACY
We quantify the invocation rate and recovery capability of the CAE's rollback mechanism across three benchmarks. Consistent with the protocol in Sec.2.4, rollback is triggered only after local planning attempts are exhausted. Table EI reveals that while rollbacks occur in 12-19% of episodes, they are highly effective: approximately 70% of these episodes eventually succeed. Notably, full restarts are required in less than 3% of cases. These statistics confirm that the rollback module acts as an efficient safety net, robustly correcting state deviations long-horizon interactions without incurring the high cost of frequent resets.
D.5 ZERO-SHOT CROSS-DOMAIN GENERALIZATION
We evaluate the transferability LongHorizonUl to AndroidWorld and OSWorld under a strict zero-shot protocol. We deploy the core pipeline (MEP, DRD, CAE) without any parameter updates
19

---

Published as a conference paper at ICLR 2026
Table 5; Rollback statistics regarding triggering frequency and recovery success.
Dataset
Rollback Triggered
Success post-Rollbace
Restart Required
AndroidControl High
12.4%
69.7%
1.8%
GUI-Odyssey
15.3%
73.1 1%
2.4%
LongGUIBench-Gree
18.6%
71.2%
2.7%
Table 6: Zero-shot success rates (SR, %) on cross-domain benchmarks. LongHorizonnUI demon- strates superior robustness, particularly in the long-horizon (50-step) OSWorld setting.
Method
OSWorld (15 steps)
OSWorld steps)
AndroidWorld
Gemini-2.5-Pre
11.7
40.6
UI-TARS-72B
18.8
24.6
46.6
LongHorizonUI
19.9
29.4
47.9
or benchmark-specific tuning, requiring only minimal API adaptation. For OSWorld, we adhere to UI-TARS protocol, reporting success rates under 15-step and 50-step budgets. As shown in Ta- ble KL LongHorizonUU consistently outperforms state-of-the-art baselines. While the improvement over UI-TARS-72B incremental on AndroidWorld and the short-horizon OSWorld (15 steps) set- ting (1.1-1.3%), the performance gap widens significantly to 4.8% in the 50-step setting (29.4% vs. 24.6%) This trend validates that our hierarchical planning and error-correction mechanisms effec- tively mitigate error accumulation over extended trajectories.
D.6 RUNTIME AND BACKBONE TRADE-OFFS
We further quantify the end-to-end latency of LongHorizonUl under different backbones. As shown in Table A non-MLLM components (MEP, CAE, I/O) contribute only about 1.1-1.4s per step, while the remaining 4-7s dominated by MLLM inference. Switching from Gemini-2.5-Pro to Gemini-1.5-Flass or Qwen2.5- 5-LLLBBBBB reduces per-step latency from 8.26 to 5.74-6.59 s, at the cost of moderate SR drop (e.g.. from 83. 83.9% to 75.3% LongGUlBench), For typical 22-step Long- GUIBennh episode, this corresponds to roughly 3 min with Gemini-2.5-Pro versus about 2-2.5 min with the lighter backbones. These results show that the MoL-MLLM overhead LongHorizonUI relatively small and that users can trade a few SR points for noticeably lower latency by choosing faster backbone.
E PROMPTS IN AUTOMATED PIPELINE
E.
OUTPUT FORMAT STRUCTURE TEMPLATE
As depicted in Figure H. the framework specifies JSON schema for agent output, enforeing strict structural conformity through five validated fields: visual goal assessment, task-relevant content ex- traction, thain-of-thougee reasoning, next-action objective declaration, and parameterized command specification. mandates termination (Done action) exclusively upon visual confirmation of task completion, instituting closed-loop verification system that binds agent responses to perceptual evidence. The schema functions as structured action-language interface between cognitive pro- cessing and environmental actuation.
F.2 VISUAL PROCESSING TEMPLATE
This template prescribes structured rules for interpreting annotated screenshots in GUI automa- tion environments, as shown Fig [] It mandates rigorous analysis of vision model-generated highlights (colored bounding boxes with indices) primary reference points for UI element iden- tification. Crucially, enforces visual outcome validation as the sole criterion for action success evaluation, overriding API execution status mitigate observation-actio discrepancy. The frame- work establishes annotation- baased perception as the foundational input for agent decision decision-making, ensuring environment fidelity through computational visual verification.
20

---

Published as a conference paper at ICLR 2026
Table 7: Runtime and backbone trade-offs on LongGUlBench and AndroidControl.
Backbone
SR (LGB, %)
SR (AC, %)
Total / step (8)
Non-MLLM (8)
MLLM (8)
Gemini-2.. P. (default)
83.9
68.9
8.26
1.18
7.08
Gemini-1.5 Sirtni--.5--Fae 1.5-Flash
75.3
64.7
5.74
1.35
4.39
Qwen2.5-VL VL-7BB
78.8
63.4
6.59
1.13
5,46
Visual Processing Template: Specifies how interpret and utilize screenshot annotations
Action Selection Protocol Template: Defines position targeting methods with priority hierarchy
(Hiighligtt usage": "Colored boxes with indices denote detected UI elements". "Element identificationn. "Top-left index numbers are primary reference points".
("name": "Highlight Index",
"condition": "Target aligns perfectly highlighted region".
"format": ("position": "<int=") )-
"Constraint": "Always prioritize visual analysis over API exccution status")
"name": "Relative Coordinates".
"condition": "Precise largeting within large highlight area".
"format": ("position": ["<index=", "<x rel 0.0-1.0>*,
// Index selection
"cy rel 0.0-1.0=="]] ).
"action": ("click": ("position": 8))
// Relative coordinates
("name": "Absolute Coordinates".
"action": ("click": ("position": [5, 0.2, 0.8]1
"condition": "No valid highlight available".
// Absolute coordinates
"constraint": "0-1000 scale (1000 max dimension)",
"action": ("swipc:: ("start": [500_500], "cnd":
"format": ("position": ["<x=" "<y="|
[1000,0]}3
Figure Visual Processing and Action Selection Prompt Template.
Workflow Exception Handling Template: Protocols for internupting scenarios
URTO
KIN AVAILABLE!
"scenarios":
{ "type": "Unexpected Pop-up".
"priority": 1,
"response": "Close before continuing main task", "position_straddd strategy": ["Index", "Relative coordinates top-right corner"]..
EIIM HOW AVAILABLE BAAALE
{ ""type": "Black Screen".
"response": "Wait 10s Re-detect Continue", "10000ms minimum" ).
"timeout":
{ "type": "App Termination", "method": "Swipe from center to screen edge", "coordinate spec": "start: [x center, y center), end:
wall
[x edges1000, y edges1000)")
Figure 12: Exception Handling Prompt Template, Establishes interrupt-drive protocols for disrup- tive UI events: highest-priority pop-up closure (top-right), black-screen re-detection (10s timeout), and app-termination recovery before resuming primary tasks.
E.3 ACTION SELECTION PROTOCOL
As depicted in Figure I.I the protocol formalizes hierarchical targeting methodology for GUI in- teractions, prioritizing: (1) direct highlight indices when element-box alignment is exact; (2) relative coordinates (0.0-1.0 1.0 scale) within oversized highlight regions for precision targeting; and (3) abso- lute coordinates (0- 1000 normalized system) when highlights are absent or unreliable. This tripar- tite selection strategy optimizes spatial accuracy while accommodating diverse interface topologies, with explicit constraints prohibiting coordinate values exceeding the 1000-unit boundary to maintain dimensional integrity.
21

---

Published as a conference paper at ICLR 2026
Expected action: Click the " (Delegate) button Actual action: Click the "2 A (Guild) button
THE
"Think" "The previous action was intended toclick the 'u59d4'u6258' (Entrus/QQuestjioonbut it seems to have opened the (uS16cuuuffa(GGuddjjjeeeea instead. This means theprevious click was likely inaccurate or thewrong icon was identified. need to backthe previous screen correctly access the uS9d4) u6258" interface. The back arrow iconis clearly visible at index 1. will click olindex to back.",
"Next_goal" "Click theback arrow to return to the previous screen(main game interface).
Corrected action: Click the "E (Delegate) bbtton
Figure 13: Error Recovery Example. Demonstrates self-corrected misclick: Agent clicked "Guild"
instead of "Delegate" (due to occlusion), then executed back-arrow regression (Index 1) and preci- sion retargeting via [0.5,0.8] coordinates to achieve the intended action.
E.4 WORKFLOW EXCEPTION HANDLING
As illustrated Figure I2 this template defines prioritized response protocols for disruptive in- terface events, establishing scenario-based classification system: (1) unexpected pop-ups (highest priority, requiring immediate closure via top-right relative coordinates); (2) black screens (trigger- ing 10-second re-detection cycles); and (3) background app termination (executed via edge-directed swipe vectors). The framework implements interrupt-drive workflow management, where excep- tion resolution systematically precedes primary task progression maintain environmental control stability.
The opens the "Honor Kings" app, message, custom "test" text and emoji chat, then leaves the closes all background apps.
and game room, speaker, sends
NEW FOLAEEE
Plocicen
Cocation"
... Bhupo ('scahion
Stup1: "click":
Step26 "clic""
("posmon":11 17)
Stepk: "dick" Oppaition"
Shept 208e Ppspitannt
...
Step0 'elee: Pwwenirnt
Figure 14: Game Scenario Case Visualization.
F QUALITATIVE ANALYSIS
F.1 ERROR CORRECTION VISUALIZATION
As illustrated in Figure IR this sequence captures a critical error-recovery episode in our LongHori- zonUl automation framework: The agent erroneously selected the adjacent "Guild" button instead of the target "Delegate" function, triggering an unintended guild management interface. Diagnos- tic self-assessment attributed this failure positional deviation and visual occlusion interference
22

---

Published as a conference paper at ICLR 2026
The opens Snopchat, takes photo, adds the text "test." draws and undoes colored line, adds and crops sunglasses sticker, saves the snap to Memories, discards it, returns to the home screen, and closes Snapchat.
Stepd: "oiont Paastten
Stepe: "dick" postfoett 277
Poosnien 1d)
Preart
Epeaiter 16)
Paodheen
Passhon 231
Step12
Phoshhe
Opeetioo 2001
Shep@0 "cide
Stepp
"avipe" dbww
dlcck Eppctton-1BB
5
dem
(ponitor-12)
Phusher
Ppocher
Ppoettis :2)
key Phame
key TTeeeett
C'aaatior
Figure 15: General Scenario Case Visualization.
within the GUI layout. To contain error propagation, the recovery protocol first activated a roll- back mechanism by clicking the back arrow (Index 1) to restore the baseline interface, followed by precision-targeted secondary click using relative coordinates [N, 0.5, 0.8] within the Dele- gate button's highlight region, successfully rectifying the initial localization inaccuracy. This case demonstrates LonsHorizonUI's operational efficacy and robustness in handling real-world automa- tion exceptions.
F.2 CASE VISUALIZATION
To demonstrate LongHorizonnUTss advantage in long-horizon reasoning, we visualize its task exe- cution trajectories in both general scenarios (Figure [[]a gaming environments (Figure (ES). In universal settings, the architecture exhibits strong task generalization via its compensatory action executor, which dynamically adjusts interaction pathways when encountering heterogeneous UI elements (e.g., switching between gesture controls and traditional input fields) while maintaining task coherence. The deep-reflective decider further ensures minimal end-to-end error propagation by verifying stepwise contextual consistency, effectively mitigating cascading failures common in baselines. Within gaming scenarios, the agent leverages enhanced perceptual signals and compen- satory action strategies to traverse nested menus and execute multi-step operations under real-time constraints, even during interface mutations.
G ADDITIONAL DISCUSSIONS
The pursuit of robust long-horizon GUI agents necessitates addressing two critical challenges: adap- tive long-horizon modeling and dynamic interrupt handling (e.g., pop-ups). For extended task se- quences, future work could integrate reinforcement learning with hierarchical state representations to compress historical trajectories into abstract milestones, mitigating error accumulation while pre- serving contextual coherence. Insights from visual scene understanding -antaaddinnninnluinnnnng semantic seg- mentation (Tan et all 20227 2023; Lai et all 202I: Tra et all 20226), few-shot segmentation (Lian all 2020 Peng al] 202 Luo all 2023: Ning et all 2023), long-tailed and contrastive
23

---

Published as a conference paper at ICLR 2026
representation learning (Cm et all 2022: 2023). 3D scene understanding (Peng all 2024h; Wang al 2022: Jiang et all 2021 Wi all 20225: Zhang et al, 2022a), text detection (Tian et all promising directions for enrich- ing the spatial and semantic perception capabilities of GUI agents. For dynamic interrupts (e.g.. pop-ups), rreaciiveereacctvve hybrid mechanism is essential: real-time environmental monitoring detects anomalies, triggering tiered fallbacks such as emergency rollbacks, LLM-guided diagnostics.
2019, and 3D content generation (Huang all
24


Reviews:
[{"review_id": "Eo2zXCyCLm", "reviewer": "Reviewer_LnDb", "summary": "This paper targets the challenge of **cumulative error and contextual drift** in MLLM-based GUI agents, which leads to a dramatic performance drop as task complexity and step count increase. The authors propose a unified, highly robust framework named **LongHorizonUI**, structured around three key, complementary modules:\n\n1.  **Multimodal Enhanced Perceiver (MEP)**: Responsible for temporally consistent state representation. It integrates icon detection and Optical Character Recognition (OCR) outputs, assigning **unique, stable indices** to UI elements to disambiguate and enhance subsequent processing.\n2.  **Deep Reflection Decider (DRD)**: Implements structured, multi-level **closed-loop reasoning** via constrained, formatted prompts. This mechanism enforces explicit validation of **historical coherence, goal relevance, and action justification**, ensuring highly precise decision-making along non-trivial trajectories. It includes a **pre-execution reflection step** to proactively validate action rationality.\n3.  **Compensatory Action Executor (CAE)**: Provides a powerful layer of fault tolerance. It utilizes a multi-level **fallback strategy**—leveraging element indices, relative layout priors, and absolute coordinates—to continuously monitor execution. Upon detecting performance degradation, the CAE applies targeted compensation or triggers **rollback procedures** to maintain robustness throughout the entire long-horizon task.\n\nTo facilitate rigorous evaluation, the paper introduces **LongGUIBench**, a novel benchmark comprising complex general applications and gaming scenarios that require **over 15 steps** (averaging 22.1 steps). Experiments clearly demonstrate that **LongHorizonUI** substantially improves long-horizon performance on **LongGUIBench** while maintaining competitive performance across diverse public benchmarks.", "questions": "1.  Please provide **end-to-end inference latency data** for **LongHorizonUI**. What are the average inference time per step and its impact on the total task completion time, especially for the longest tasks in **LongGUIBench**?\n2.  Quantify the failure rate of the pre-execution reflection mechanism in the **Deep Reflection Decider (DRD)**—i.e., the rejection rate of actions (when $\\phi(s_t, a | G_t, T) = 0$). What are the **most common root causes** for these rejections (e.g., element absence, action semantic mismatch)?\n3.  Could you elaborate on the precise implementation of the **rollback mechanism** in the **Compensatory Action Executor (CAE)** (Algorithm 1, line 14)? Does \"last committed snapshot $(s_{t-1}, p_{t-1})$\" always revert to the immediately preceding successful step, or is there a strategy to revert to a much **earlier, highly stable state**? What is the associated cost of a rollback?\n4.  For the General-High and Game-High tasks, what specifically defines these \"**High-Level**\" instructions, and how does the **DRD** facilitate the breakdown of these abstract goals into a sequence of \"Low-Level\" atomic operations?\n5.  Provide more details on the **fallback template matcher** in the **Multimodal Enhanced Perceiver (MEP)** (lines 207-208). How is the template library **T** constructed, what types of critical elements does it include, and what are the specific **activation criteria** (e.g., low-confidence score, or no element detection in a high-priority area)?\n6.  Regarding future extensions of **LongGUIBench**, are there plans to incorporate challenges related to **multi-lingual GUIs**, **complex data transfer across applications**, or scenarios requiring external knowledge retrieval?\n7. The paper also has several minor and typos.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the **most significant limitation** of MLLM-based GUI agents: their brittleness and poor performance in complex, multi-step scenarios. The solution provided is highly relevant to real-world deployment.  It presents **LongGUIBench** is a **major asset** to the field. By focusing on **tasks requiring over 15 steps** and including both general and gaming applications, it provides a far more realistic and challenging testbed for long-horizon capabilities than existing short-horizon benchmarks. The high-quality data collection methodology further assures its utility.", "weaknesses": "1. We note the paper's emphasis on achieving long-horizon GUI automation. However, the definition and evaluation of \"long-horizon\" in this work need stronger justification, as recent concurrent works already include evaluation on tasks with comparable or even longer step sequences in highly dynamic environments. This diminishes the claimed novelty of LongGUIBench purely on the basis of task length and requires a clearer delineation of the unique challenges posed by this new benchmark beyond just step count.\n2. The use of Google's **Gemini-2.5 Pro** as the reasoning backbone is critical, yet the discussion lacks depth regarding its specific impact. A key missing element is an analysis of how the framework's performance and robustness are affected by switching to **faster, potentially less capable MLLMs** (e.g., open-source alternatives or smaller commercial models). This is essential for understanding the framework's practical cost and deployment flexibility.\n2.   The paper acknowledges the inherent **latency** in MLLM-dependent pipelines but fails to provide **specific quantitative data** on runtime efficiency, inference time per step, or resource consumption in the main results. In long-horizon automation, accumulated latency significantly affects user experience and real-world applicability. This omission limits a comprehensive evaluation of the framework's practicality.\n3.  While **LongGUIBench** is excellent, its limitations should be discussed more explicitly. Despite the inclusion of diverse scenarios, the task count (371 total) is still small relative to real-world complexity. The paper should address its coverage of extreme challenges such as **multi-lingual interfaces, highly dynamic content loading (e.g., infinite scrolling),** or **inter-application contextual reasoning**. In Osworld and WinArena benchmarks, whether its performance promising?\n4.   The introduction contrasts this work with existing online Reinforcement Learning (RL) methods, highlighting their challenges with expanding action spaces and cumulative errors. Given that **LongHorizonUI** features internal feedback (DRD) and error correction (CAE), a **more detailed, fundamental comparison** of the two long-horizon paradigms (RL vs. Structured Reflection/Compensation) would significantly benefit the reader in positioning this work.\n5.   The framework heavily relies on strictly defined JSON Schema and detailed prompt templates (Figures 9, 11, 12). However, the paper lacks detailed discussion on the prompt design process, a **sensitivity analysis** showing how minor prompt variations affect the DRD's reflection quality, and the methods used to ensure the MLLM adheres consistently to the desired structure. Enhancing transparency here is crucial for reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbN5Z5MDE1", "reviewer": "Reviewer_XboQ", "summary": "The paper primarily addresses the performance degradation of GUI agents in long-horizon tasks (over 15 steps), proposing a unified framework named LongHorizonUI along with a corresponding long-task benchmark, LongGUIBench.", "questions": "1. I’m a bit confused here — Section 3.2 mentions *“415 trajectories”* (page 7), whereas Section 2.1 and several descriptions in the appendix together account for only *371 scenes* (pages 3 and 17).\n2. Have the authors experimented with other benchmarks, such as AndroidWorld?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The problem is clearly defined and grounded in reality: it directly targets the degradation issue in long-horizon GUI automation tasks (those exceeding 15 steps).\n- Empirical results are persuasive.", "weaknesses": "- During data parsing and alignment, an MLLM was used for semantic annotation and structured extraction (page 3), which may introduce biases or “distribution leakage” similar to those in the evaluated models. Did the author perform any manual verification?\n- The proposed method is built upon Gemini as the base model, which poses certain challenges for reproducibility. Have the authors attempted experiments based on open-source models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ghLdhHoNkv", "reviewer": "Reviewer_L7cu", "summary": "This work targets long-horizon GUI agents, especially tasks exceeding 15 steps. The authors introduce the LongGUIBench benchmark and the LongHorizonUI framework, which integrates three carefully designed modules—Multimodal Enhanced Perceiver, Deep Reflection Decider, and Compensatory Action Executor. Experiments demonstrate improved long-horizon performance on GUI tasks.", "questions": "How does the model perform beyond 18 steps?\n\nWhat exactly are agent decision mechanisms, and why do they lack long-horizon generalization due to dataset diversity? Would it be clearer to explain the notion of dataset diversity, or simply state that the datasets lack long-horizon trajectories? The term “data diversity” may be confusing here, since the tasks appear to be limited to Mobile GUIs, which may not generalize to Windows.\n\nLikewise, what are MLLM-based mechanisms? Do they refer to a multi-agent framework for GUI tasks? Which properties confer superior sequence-modeling capability? Please also clarify dynamic interface shifts. Concrete examples comparing different methods—their challenges and how your approach addresses them—would help readers understand why your method is suitable and efficient for long-horizon GUI tasks.\n\nFinally, what is meant by action-instruction uncertainty? Is it tied to grounding capability? And do free-format outputs denote high-level instructions or the agents’ textual step goals?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This is an important topic. GUI agents serve both as a crucial platform for algorithm evaluation and as a practical application of VLMs. The long-horizon GUI agent setting remains underexplored, and strong long-horizon planning is particularly valuable.\nThe contributions seem compelling: the authors introduce a new framework and benchmark that address this gap.", "weaknesses": "However, although the authors propose both a benchmark and a framework, I think there is still room for improvement before acceptance.\n\nThe paper’s writing is not very strong. There are many typos, and the names and function descriptions of the components are confusing.\n\n**Typos**:\n\n* Figure 2 caption: “1015 steps” → likely “10–15 steps”\n* Quotation marks: should be ` ', instead of ' ' (Lines 187–188)\n* Line 262: “actioninstruction” → “action instruction”\n* Line 263: “semanticphysical” → “semantic physical”\nI present the weaknesses of the proposed framework’s components as follows.\n\n**Multimodal Enhanced Perceiver.** It is not clear why the perceiver is “enhanced.” Identifying icons and text descriptions seems standard. The paper gives few details about the enhanced detector (Line 199). Is it trained from a VLM, or does it incorporate additional enhanced functions? It is also unclear how the confidence score is obtained—does it come from hidden states, or from an uncertainty-estimation module?\n\nIn Eq. 1, what do the authors mean by multiplying the indicator function value by the detected text?\n\nThere is also a lack of detail about high-priority areas. What is the motivation? Where are these areas defined? What is the “repair” function? How does it operate, and what does the template library look like? Why are both the template library and the detectors/OCR necessary?\n\nFor the **Deep Reflection Decider**, based on the name and the function description in Lines 226–232, I am confused about whether this component only performs reflection or also includes decision-making. I am not sure whether a new name would help, such as “Actor with Deep Reflection.”\nIn addition, what is “multi-level” and what is the “triple closed loop”? Figure 3 appears to show five steps for this component, but the text also describes a two-step process (pre-execution / post-execution) and three bullet points in Lines 236–247. Could you please modify Figure 4 to align these descriptions?\n\nHow exactly does this component act? From the current description, the agent outputs only an action; there does not seem to be actual post-execution reflection or validation, like how to update the agent's knowledge or how to revise.\n\nThe authors also mention a keyword extractor in Line 253, but provide no description.\n\nThere are also no details about the brief revision step\n\n**Compensating Action Actuator**\n\nWhat is the name of the component in Section 2.4: Compensating Action Actuator (section title) or Compensating Action Executor (abstract)?\n\nWhat is a device-aware scaling matrix?\n\nRollback may be impossible in some cases. It is unclear how the proposed framework addresses this. What happens if the compensating action actuator makes a change to the screenshot but fails? Some changes can be rolled back, but others cannot, such as clicking “next page” or accidentally sending an email.\n\nRegarding the agent’s grounding capability, the framework introduces many components and is quite complex, but the improvements may be limited: LongHorizonUI vs. InfoGUI-R1: 90.4 vs. 87.5 on ScreenSpot.\n\nIn conclusion, this work lacks sufficient detail for the writing.\n\n\nFor the **framework design**, in addition to the above, I still have concerns:\n\n- The framework seems to complex.\n- While the framework seems focused on avoiding error propagation—which is important—long-horizon planning, subgoal proposal, validation, and memory may be even more critical for long-horizon tasks. It is not clear to me: how does the framework contribute to these aspects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkpQq0c8bL", "reviewer": "Reviewer_kekp", "summary": "This paper proposes a framework named LongHorizonUI, aiming to enhance the reliability of MLLM-based agents in long-horizon tasks. LongHorizonUI introduces three components: a Multimodal Enhanced Perceiver that integrates detection and recognition, and assigns indices to reinforce state; a DeepReflection Decider that leverages multi-level feedback for iterative reasoning and precise execution; and a Compensatory Action Executor that monitors progress, and performs compensation or rollback to maintain long-horizon robustness. Experiments show that LongHorizonUI significantly improves long-horizon performance on LongGUIBench while remaining competitive across various public benchmarks.", "questions": "1. Highlight the novelty of this paper compared to Mobile-Agent-V3 and D-Artemis.\n2. Provide an ablation study for the Deep Reflection Decider module.\n3. In Section 3.3, add quantitative step-length analysis for AndroidControl and GUI-Odyssey (similar to Figure 2b).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The reliability of MLLM-based agents in long-horizon tasks is one of the main challenges in GUI Agent application. Developing reliable GUI agents for long-horizon tasks is of great importance.\n2. Dataset Contribution: This paper introduces LongGUIBench, a novel benchmark for long-horizon GUI interaction. It encompasses diverse complex tasks across multiple application domains.", "weaknesses": "1. A more concise analysis of the abrupt success rate drop in long-horizon tasks is needed: disentangle root causes (e.g., recurring critical-step errors vs. failures to recover from erroneous states) and explicitly link these to how the proposed MEP/DRD/CAE modules address key long-horizon challenges.\n2. In Section 2.4, please clarify the feasibility of rollback in GUI tasks: specify applicable task scopes, technical details of state snapshot storage/restoration.\n3. In Section 3.3, Table 3 has inconsistencies: (1) AndroidControl/GUI-Odyssey are Android-focused, so explain the inclusion of \"Desktop/Web\" columns and cite relevant references; (2) the \"GUI-R1-7B baseline\" mentioned in the navigation capability text is missing from the table—add it or align the text with the table.\n4. Typo errors: standardize inconsistent bolding in Table 1 and Table 3, and adjust Figure 2b to resolve legend overlap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)